pmxbot IRC Log Viewer

[01:30:08] <azneita> what are other options for having a secondary node catch up with the master when master is being written to fairly often?

[01:31:15] <azneita> have done initial sync but it took too much time and was already far behind when it finished

[01:32:28] <context> ok so... mongo, using 99% of disk, removed 'the' collection to rebuild it in question. how free disk space so other people stop crying about it

[01:32:32] <context> i cant repairDatabase()

[01:33:38] <joannac> azneita: filesystem snapshot?

[01:33:44] <joannac> context: why can't you repair?

[01:33:47] <joannac> context: replset?

[01:34:30] <context> not enough diskspace to do repairDatabase()

[01:34:45] <context> if i recreate the collection and .compact() will that possibly work?

[01:35:35] <context> err, i guess compact is not a function :X

[01:36:30] <context> no, no replica or anything cause these people are retarded

[01:36:58] <azneita> @joannac: wer're not using filesystem snapshots i'm afraid

[01:37:24] <azneita> yeah, replSet

[01:37:44] <joannac> context: spare disk, mount it somewhere, --repairPath?

[01:38:31] <joannac> azneita: i dunno, how fast can you copy files?

[01:38:59] <joannac> azneita: is there a downtime window or a less-busy time window?

[01:39:42] <context> will mongo 'auto reuse' the space that was freed?

[01:39:47] <context> or its just sitting there, wasting space ?

[01:40:00] <joannac> context: repair or resync is how you get space. you could try turning it into a replica set and syncing to a new node, then swapping over maybe

[01:40:12] <joannac> that seems like more effort than just mounting another disk though

[01:40:18] <context> yeah

[01:40:25] <joannac> context: yes it will, it's sometimes difficult though

[01:40:46] <joannac> because for most people, documents get bigger

[01:41:17] <context> scary.

[01:41:19] <joannac> which means a document that went from 1kb to 1.1kb gets moved, but a new document inserted which is 1.05kb can't fit in the space either

[01:41:40] <context> disk used is 186G, there is only 3.3G of data. you'd think at least two of these 2G files could be removed....

[01:41:47] <joannac> :o

[01:42:06] <joannac> i think you should've been repairing days/weeks ago :)

[01:43:27] <context> ... yeah

[01:43:35] <joannac> go buy a disk :)

[01:43:41] <context> well... it tipped the point today

[01:43:46] <context> i removed the collection, and was rebuilding it

[01:43:55] <context> didnt expect this crap

[01:44:06] <azneita> joannac: we're reserving downtime window as last option

[01:45:06] <context> still.... the db has like 93 2.1G files, it cant figure out how to remove three of them and repair then.....

[01:45:13] <Boomtime> azneita: you are at your last option, by leaving an emergency to escalate into catastrophe you need to do whatever you must now

[01:45:32] <Boomtime> oops, sorry, that was for context, that might not apply to your case

[01:45:53] <context> i understand that.

[01:46:22] <context> guess i get to look up this repairpath crap :/ any helpful url's ?

[01:46:32] <joannac> context: there's no concept of "3 of them" - the data could be anywhere

[01:46:50] <joannac> the data could be spread out such that removing any single file would result in data loss

[01:46:57] <context> so... you are saying 3.3G data, 1.8G index, and all 93 files are being used?

[01:47:06] <joannac> perhaps, I don't know

[01:47:10] <context> you cant rebuild using the 3.3G of available space left ?

[01:47:21] <joannac> no

[01:47:36] <joannac> for a start, all your data + indexes takes up more space than 3.3GB

[01:49:49] <context> ok... done worrying about that

[01:50:08] <context> mongod --repair --repairpath .... ; do i do that WITH mongod running or kill it first?

[01:50:32] <context> and im assuming: "repair using repair path; remove old db crap; move repairpath files to original location" ?

[01:50:49] <context> thats how it works im assuming

[01:52:14] <joannac> yes

[01:52:27] <context> shutdown first or not?

[01:52:36] <context> and assuming i want to do --config with that as well

[01:52:42] <context> AND run as mongod user :x

[01:52:43] <context> sorry

[01:53:29] <joannac> if you're going to run it like that, then yes...

[01:53:46] <joannac> Run mongod directly from your system’s shell. *Make sure that mongod isn’t already running*, and that you invoke mongod as a user that has access to MongoDB’s data files. Run as:

[01:53:52] <joannac> emphasis mine

[01:54:58] <context> k

[01:54:59] <context> thnx

[02:01:03] <context> ok... when was --repair/--repairpath added

[02:01:37] <context> nm its there

[02:02:30] <context> really wish this thing was printing more

[02:02:45] <context> it didnt even print a "starting repair"

[02:05:08] <context> shouldn't this like create a file in the repairpath, or... log /anything/ at all

[02:05:17] <context> or just be silent for 10 minutes

[02:05:57] <joannac> ow are you running it?

[02:06:18] <joannac> I just ran mine with --dbpath and --repair and I'm getting output

[02:08:54] <context> well, i did -f /etc/mongod.conf --repair --repairpath ...

[02:09:03] <context> am guessing -f overrides everything else or some shit? :-/

[02:09:42] <joannac> what's in the config file?

[02:10:09] <context> i dont know

[02:10:13] <context> its what it always runs with

[02:11:19] <context> ok, attempting to run again with just --dbpath --repair --repairpath

[02:12:18] <context> "you must use a repairpath that is a subdirectory of dbpath" ...

[02:12:24] <context> when using journaling

[02:12:36] <context> this gets better and better

[02:13:25] <Boomtime> wow, that doesn't seem to be documented

[02:13:30] <Boomtime> :(

[02:13:41] <context> YEY !@#

[02:14:01] <context> even with repairpath, it failed due to "not enough disk space"

[02:14:04] <Boomtime> link time

[02:14:18] <Boomtime> use a symlink

[02:14:36] <context> 23G isnt enough for 6G of data apparently

[02:16:30] <joannac> wait, how much space do you actually have?

[02:16:37] <joannac> in your new disk

[02:40:19] <hydrajump> just looking at rs.status() is it normal that the number of operations after the optime is so high "optime" : Timestamp(1413427150, 136),

[02:40:46] <hydrajump> it just means that those 136 operations have not been written to disk?

[02:41:03] <joannac> no

[02:42:11] <joannac> the first part is seconds since the unix epoch

[02:42:17] <joannac> the second part is a counter

[02:43:16] <joannac> it just means the last activity was in that second, and there were 136 operations done in that second

[02:44:10] <joannac> I have no idea whether it's normal for you to have that many operations

[02:45:20] <hydrajump> ok so when I compare the optime between two members I just need to be concerned with the first part?

[02:47:05] <joannac> well, no, you should be concerned with both parts

[02:48:19] <hydrajump> what I'm trying to do after chatting with Boomtime yesterday is write a backup script and the part I'm struggling with right now is finding out which of my two SECONDARY members has the lowest optime lag with the PRIMARY and than I'll use that SECONDARY as the member to db.fsyncLock() and snapshot.

[02:49:59] <joannac> okay

[02:55:20] <hydrajump> joannac: If you could please take a look at my script https://gist.github.com/anonymous/2579afd65e3c37bcd555 and maybe give me some pointers

[03:01:10] <joannac> Sorry, I don't really have the time to review scripts

[03:06:21] <hydrajump> ok maybe you can just give me some pointers on my backup method.

[03:06:48] <hydrajump> I just came across this `rs.printSlaveReplicationInfo()` command. It will output the lag for each secondary member.

[03:07:33] <hydrajump> Maybe it's sufficient to use the output from this command to determine which SECONDARY member to lock,flush and snapshot?

[03:14:16] <joannac> if you can parse that, sure

[03:14:22] <joannac> and assuming you know which the primary is

[03:41:37] <hydrajump> after looking at how `rs.printSlaveReplicationInfo()` is implemented if I understand the code the lag is calculated by taking the current time - the optime of each secondary member. I don't see that it takes the number of operations in the last second being accounted for in the calculation.

[04:23:16] <clu3> hi guys, is it possible to export a collections with a query? like export a collection where {type : 1} for example?

[04:26:20] <Boomtime> clu3: http://docs.mongodb.org/manual/reference/operator/aggregation/out/

[04:26:44] <Boomtime> aggregation pipeline $match -> $out

[06:18:57] <e_m_fields> Hola amigos.

[06:19:25] <e_m_fields> Anyone live?

[06:19:34] <Boomtime> it's true, we are Unicode compliant

[06:19:46] <e_m_fields> :-D

[06:20:01] <LouisT> who is this we you speak of?

[06:21:43] <e_m_fields> Newbie: MongoDB + PHP question: How do I query for a second-level element? (I'm not sure terminology, should call it a nested document or field ?)

[06:21:53] <e_m_fields> Example: (My data):

[06:23:49] <e_m_fields> { name: 'element_1' , date: '10-12-2014', subelements: [ {name: 'subelement_1', time:'10:15:00'}, {name: 'subelement_2', time:'10:16:00'} ] }

[06:24:18] <e_m_fields> With the PHP driver, how would I get an array of 'subelements'?

[06:25:20] <e_m_fields> ie: Something like "$cursor = $collection->find('subelement')"

[06:25:45] <Boomtime> you are conflating two concepts

[06:25:52] <Boomtime> you need to do two things:

[06:26:06] <Boomtime> 1. query for the document that you want to find

[06:26:07] <e_m_fields> That sounds like a likely thing: :D

[06:26:14] <Boomtime> 2. access the array of that document you are interested in

[06:26:58] <e_m_fields> Okay. So, 1) I'm expecting to walk through the list of all the top-level documents.

[06:27:41] <e_m_fields> ie, find a way to query for every "element_1"-level document.

[06:28:24] <ArTeS> is there any java code example for Java Driver Concurrency ?

[06:28:48] <e_m_fields> was doing this with "$cursor = $db->collection->find()"... Is this the wrong way to start here?

[06:32:13] <Boomtime> e_m_fields: find() is good to use

[06:33:33] <e_m_fields> Okay. So this will give me an array with each top-level document. From there, how do I go about accessing the array of the sub-level documents?

[06:34:43] <e_m_fields> foreach ($cursor as $c) { $c->find() .... something here, to give me the 'subelements' array}

[06:35:25] <e_m_fields> This is the sorry level where my grasp of MongoDB goes 'pffftt'.

[06:42:44] <Boomtime> that isn't mongodb you have trouble with now, you're in PHP land there

[06:43:08] <Boomtime> wait, you have the find() inside the loop, what?

[06:43:24] <Boomtime> find() returns a cursor, you then enumerate that cursor to get each document matched

[06:43:46] <Boomtime> btw, if you know only one document will match, you can use findOne() to just get that and skip the cursor bit

[06:48:31] <e_m_fields> Boomtime: okay, I just changed code to use findone: Successfully returned the 1st document. Err, maybe I'm in PHP land fuzzies here, not MongoDB issues: But to access the "subfield" array of the document, I should start ... where?

[06:49:08] <e_m_fields> aka, $document = $db->findone() ; $subfield_array = ...

[06:57:36] <Boomtime> e_m_fields: have you just tried $document->subelements?

[07:01:59] <e_m_fields> -_-

[07:02:32] <e_m_fields> Boomtime: Nope. attempting now.

[07:03:15] <samdevel> Hi.. I m unable to connect to mongodb server in my linux server from my windows machine using robomongo.

[07:04:32] <e_m_fields> Boomtime: Okay, PHP vomits: We're attempting to get property of a non-object. I think I'm almost there: $element['subelements'] seems to work.

[07:07:36] <e_m_fields> Booyah.

[07:09:12] <e_m_fields> Boomtime: Seem to have PHP vomiting correctly now. :) It was $element['subelement']['subsubfield'] syntax we're looking for. Thanks for the responses, G.

[07:14:21] <Boomtime> samdevel: can you connect to your mongodb via the mongo shell?

[07:19:51] <samdevel> How do I check it.

[07:20:46] <Boomtime> you use the mongo shell to connect to your mongodb

[07:21:13] <Boomtime> http://docs.mongodb.org/manual/tutorial/getting-started-with-the-mongo-shell/

[07:28:32] <samdevel> Its connection. How to permanently start mongod ?

[08:38:22] <cheneydeng> any ony trid to run function with static variable throught $where when execute find?

[09:11:05] <PirosB3> Hi all!

[09:11:23] <PirosB3> my mongoDB is doing 99% CPU usage, with read-only operation

[09:11:34] <PirosB3> what way would you suggest me to diagnose these propblems?

[09:17:16] <KieranG> Hi guys - I have been tasked to find a way to store an arbitrary XML document properly in MongoDB, currently I am converting it from XML -> JSON -> BSON, which all seems to work fine, no information is lost whatsoever, however when I try and insert it into the database and try to retrieve it, it seems to lose any elements, including the root node, that are containers for other elements - Does

[09:17:16] <KieranG> anyone have any ideas how I can get around this?

[09:34:40] <glontu> hi

[09:34:56] <glontu> i'd like to try and migrate one of my projects to mongodb but i have 1 question

[09:36:10] <glontu> i have an object A that has a set of objects B and another reference to a particular B from that same set. Does that sit right with mongodb ?

[09:36:24] <glontu> is it a special case or is it normal usage ?

[09:37:28] <glontu> and do i need to hold the second reference to the particular object from the set ( i was doing it for performance concerns ) or is it enough to just put a special mark on one of the objects from the set and use that when i fetch ?

[09:43:22] <arisalexis> i want to model a chat messaging system can anyone help with a few questions?

[09:44:41] <arisalexis> "chat" document gets appended "messages" in an array. could be a lot of them. should i normalize this or not? i read document growth is bad

[09:50:48] <arisalexis> nvm i will model it with norm

[09:50:58] <GIjsNelissen> hi all! i have a map reduce with input: 50k docs, emits 50k, reduces to 510 and then outputs just 1

[09:51:19] <GIjsNelissen> i want it to output 510 results. Can anyone have a quick look at the map reduce functions ?

[09:51:22] <arisalexis> another questions is can i have a key like "aris:alexis" and anotherone "aris:makis" and then query all keys for "aris:*"?

[09:52:21] <GIjsNelissen> the functions are on the mongodb google group https://groups.google.com/forum/#!topic/mongodb-user/nkfLYuIpA9Q

[11:25:56] <kPb_in> Hello All.. I have a mongodb 2.4.9 running production(1 primary, 2 secondaries).. All as running well until sometime ago mongo suddenly become very very slow.. some basic queries took 30 minutes to run.. What could be the issue?

[12:35:37] <ssarah> hei guys, the build that is installed through this http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/ doesn't come with --sslMode available ?

[15:11:25] <goel88> Hi. I am trying to enable sharding on mongodb databases for some benchmarking purposes using YCSB. I am trying to run mongod on three different nodes and this part I am success fully able to do. Now in the second part I am trying to mongo to one of those three nodes which has already mongod running. Running mongos erratically throws me the following error sometimes -> “Failed to connect to 1 reason: errno:111 Connection refused”

[15:12:25] <goel88> while sometimes it is able to connect successfully. What can be the possible reason for it ? It would be great if anyone could help in this regard

[16:19:31] <hydrajump> Looking at this output from my replica set https://gist.github.com/anonymous/034e9bfab6ccb005674b it looks as if the secondaries are only 1 sec behind the primary, but the "syncedTo" says 255 secs. Is that supposed to be milliseconds?

[16:49:37] <culthero> What happens when you have max_size on a shard set (lets say 20gb), the shards are full, and you are still inserting.. do the new queries fail?

[16:50:10] <tscanausa> max size?

[16:51:30] <MxxCon_> hello folks. I'm trying to figure out if it's possible to monitor(and maybe manage) existing mongodb installs using MMS service?

[16:51:47] <MxxCon_> it seems like everywhere it talks about making new deployments

[16:52:09] <MxxCon_> unless i'm misunderstanding terminology here

[16:52:12] <tscanausa> culthero: it depends.

[16:54:14] <culthero> tscanausa: I am ingesting data at a maximum rate of 4.8gb an hour (including indexes), but it is usually ~500mb to 1gb an hour. 4.8gb an hour can go on for hours, but isn't common.. From what I understand I will need to account for these scenarios in code, not in mongo configuration to manually delete old chunks of my shards are getting too full

[16:55:14] <tscanausa> you can use ttls

[16:55:15] <culthero> I am expiring old data using a TTL index, but either I have a commonly 25-30% disk utilization and less historical data, or I run out of disk space and need to run around resizing vps's and whatnot

[16:56:12] <tscanausa> if you were not using a sharded collection like it sould like you shouldnt be that you can use a capped collection

[16:56:53] <culthero> Unfortunately I have conflicting interests in the database, one is to ingest and index the data, and second is to search it quickly (using a fulltext index on the largest collection)

[16:57:00] <tscanausa> max size if for chunks, and a server can have many chunks, if you fill up a chunk the chunk wont move then then you are SOL

[16:58:03] <culthero> with databases that get outside of working set, if my index gets bigger then memory querys on fulltext thrash the disk. Even on speedy SSD's searches that contain better then 20% of the whole index bring the application to its knees

[16:58:08] <tscanausa> unsharded is always the best way to go if you can.

[16:59:00] <culthero> IE my database needs to read through like a 50gb index.. and since there are no range predicates, I have no good way of chunking this.. unless I start storing additional things like hour - day - month - second - minute etc

[16:59:35] <tscanausa> so if you dont want that then you need to manage in code.

[17:00:25] <culthero> Yeah. I only have myself to blame, read a couple of blogs on High scalability and think "sure, I can do that!" .. (4 months later still dealing with hardware limitations, ballooning cost and no actual solution) :)

[17:00:42] <culthero> well I have somewhat of a solution, it's just a lot less sexy then I had hoped

[17:01:33] <tscanausa> culthero: I luck out by hand managing my "shards".

[17:01:50] <tscanausa> similar to how one would mysql shard

[17:02:10] <culthero> oh just get an email alert if you're running into i/o cpu disk space issues?

[17:02:28] <hydrajump> is it ok to use mongo shell version 2.6.4 with older versions of mongod, e.g. 2.4.11 or should the mongo shell and mongod always be the same version?

[17:02:48] <tscanausa> hydrajump: try to keep them the same

[17:03:00] <culthero> well thanks tscanausa:

[17:04:39] <hydrajump> tscanausa: ok thank you.

[17:19:35] <tristanp> im a little unsure of how to install mongodb on mac. i put the binaries in ~/dev/mongodb/bin, and then i created ~/dev/mongodb/data/db/

[17:20:01] <tristanp> but apparently that's not correct, as mongod claims /data/db/ is nonexistent

[17:20:13] <cheeser> brew install mongodb

[17:20:23] <tristanp> trying to do it manually without brew

[17:20:40] <cheeser> unzip, edit mongod.conf, start mongod

[17:21:21] <tristanp> where is mongod.conf

[17:21:32] <cheeser> should be in etc

[17:21:45] <tristanp> the unzipped directory has a readme, bin/, gnu license, third party notices

[17:22:07] <cheeser> no etc?

[17:22:13] <tristanp> no etc file

[17:22:16] <tristanp> or directory

[17:22:31] <cheeser> well, you can always pass config parameters to mongod directly

[17:22:44] <tristanp> i would rather just use the default

[17:22:59] <cheeser> then make /data/db and give yourself write perms

[17:22:59] <tristanp> which is data/db, but im not sure from what root

[17:23:09] <cheeser> there's only one root

[17:23:23] <tristanp> i did make a data/db, but did i make it in the right place? see my first message for where i put it

[17:23:32] <cheeser> not data/db. /data/db

[17:24:58] <tristanp> what is the difference between data/db and /data/db. either way i am putting it at ~/dev/mongodb/data/db/ right

[17:25:16] <cheeser> the first is relative to the current dir. the second is absolute.

[17:25:29] <cheeser> that's not how unix paths work

[17:25:39] <tristanp> absolute goes from what then

[17:25:41] <tristanp> ~?

[17:25:42] <cheeser> the root

[17:25:45] <cheeser> /

[17:29:26] <tristanp> ok, figured that part out. i must use sudo to create /data/db/ though, does that mean my permissions are not set up correctly?

[17:29:59] <cheeser> that path will be owned by root. you'll need to change it

[17:31:28] <tristanp> i can't make it owned by mongod?

[17:31:36] <cheeser> is that you're user?

[17:31:42] <cheeser> *your* user...

[17:33:00] <tristanp> ohh im reading a bit more and it seems to me that having the db in /data is not such a good idea

[17:33:15] <tristanp> but then i have to run mongod with arguments every time...

[17:33:46] <cheeser> or set up a mongod.conf

[17:33:58] <cheeser> or script your start up. you have options.

[17:34:18] <tristanp> actually i should learn how to change the read write permissions for /data/db

[17:34:25] <tristanp> is it complex

[17:34:39] <cheeser> no. it's pretty basic unix.

[17:35:29] <tristanp> ok did it

[17:35:36] <tristanp> i just needed to know the chown command

[17:37:28] <tristanp> cheeser: thanks for the help, looks like i got it runnin

[17:37:34] <cheeser> np

[19:36:38] <dreinull> maybe I'm looking in the wrong places. On a VPS with only localhost access necessary, is there anything I need to do in order to secure my db?

[19:37:30] <dreinull> do I need users or am I just fine with full access from my local app that accesses the db?

[21:43:39] <Trindaz> mongodb driver for node.js. I can't see anything in the docs about how to chain find commands (e.g. will find().limit(5).toArray() work?)

Log file Viewer

Help | Karma | Search:

#mongodb logs for Thursday the 16th of October, 2014