PMXBOT Log file Viewer

Help | Karma | Search:

#mongodb logs for Monday the 14th of July, 2014

(Back to #mongodb overview) (Back to channel listing) (Animate logs)
[03:37:33] <fluter> hi,
[03:37:46] <fluter> what's the recommended memory for mongodb?
[03:37:55] <fluter> it is using a huge memroy on my setup
[03:38:56] <fluter> 3517 mongod 20 0 36.8g 4.0g 4.0g S 0.7 6.4 0:05.86 mongod
[07:27:44] <VeeWee> Mongod won't start. I tried repairing, removing lock file, killing old process and removing pid file. No logs are written. Any ideas?
[07:32:03] <kees_> permissions on the log/pid/data files VeeWee?
[07:34:40] <VeeWee> switched them to mongod user, but they got switched back to root for some reason
[07:35:25] <kees_> prolly because you ran it as the root user to test it once or twice?
[07:38:48] <VeeWee> Could be ... Can't start the process with the mongod user though.
[07:41:24] <yruss972> Hi, I'm trying to understand the serverStatus output. Is it normal for recordStats.local.accessesNotInMemory to be as high as 2666252 ?
[07:52:33] <VeeWee> kees_: got it ... emptied the log directory a while ago *facepalm*
[07:53:13] <rspijker> yruss972: it could be… how long has the process been running?
[07:56:46] <yruss972> rspijker: "uptime" : 19928429 (not sure what units those are)
[07:57:26] <rspijker> yruss972: seconds
[07:58:01] <rspijker> so that’s ~ 230 days
[07:58:26] <yruss972> :? kind of hard to believe its been up so long
[07:58:39] <yruss972> uptime
[07:59:04] <yruss972> in server status it gives you the uptime of the process or the server?
[07:59:11] <rspijker> process
[07:59:32] <rspijker> http://docs.mongodb.org/manual/reference/command/serverStatus/
[08:00:13] <rspijker> so, basically. That number for local.accessesNotInMemory is the amount of times mongod had to fetch a page that wasn’t in memory for the local DB
[08:00:49] <yruss972> yeah- I read it but I'm just not sure how bad it is
[08:00:56] <rspijker> that’s not all that strange. The local DB is written to every time you write to the DB (since the oplog lives there)
[08:01:03] <yruss972> I have some other dbs with high numbers also
[08:01:24] <rspijker> are you seeing bad performance?
[08:01:36] <yruss972> I'm seeing memory issues
[08:01:51] <yruss972> server is using too much ram, starting to swap
[08:02:14] <yruss972> and the vsize is about 6 times the mapped memory size
[08:02:23] <yruss972> only about 60 connections
[08:03:40] <rspijker> which mongod version and which OS?
[08:03:53] <yruss972> 2.4.9 on SmartOS
[08:04:09] <yruss972> if I mongodump the whole db, I get 105MB
[08:04:14] <rspijker> There was someone in here the other day as well which reported very large vsize…
[08:04:22] <rspijker> also on SmartOS
[08:04:22] <yruss972> it was me ;)
[08:04:25] <rspijker> ah :)
[08:04:37] <yruss972> problems went away for the weekend
[08:04:50] <yruss972> but resurfacing - I guess with traffic
[08:05:20] <rspijker> well… those numbers being high (accessesNotInMem) are a logical result of the server running out of resident memory
[08:05:33] <rspijker> if it’s swapping, then, by definition, it’s accessing stuff not in memory
[08:05:46] <rspijker> the quesiton is, why is it swapping in the first place
[08:05:59] <yruss972> I don't understand how I could be running out of memory when all my DBs dump into 105MB of files?
[08:06:05] <rspijker> from what you’re saying the amount of actual data in the system isn’t very large
[08:06:20] <yruss972> driving me nuts
[08:06:30] <rspijker> how much ram does the machine have?
[08:06:34] <yruss972> we do have a lot of replicas (about 9 replicas)
[08:06:36] <yruss972> 4G
[08:06:44] <yruss972> each machine has 4G
[08:08:08] <rspijker> what does mongostat show?
[08:10:12] <yruss972> https://gist.github.com/anonymous/a1cbe839fd1d40258e71
[08:10:45] <rspijker> hmmm, your resident size is larger than ram
[08:10:58] <rspijker> are you sure you’re dumping everything when you get the 105 MB?
[08:10:59] <yruss972> yeah- happened this morning
[08:11:14] <rspijker> I’ve got to go into a meeting
[08:11:20] <rspijker> will be back later
[08:11:26] <yruss972> I ran mongodump with no options
[08:11:44] <yruss972> thanks
[08:38:50] <shambat> is there some way to limit the amount of ram mongodb uses?
[08:44:33] <sflint> shambat: there isn't a configuration in mongod for that
[08:44:37] <sflint> why are you trying to limit it?
[08:46:21] <shambat> sflint: I want to use mongodb as a db instead of storeing data in huge json files, not for speed, but to save memory usage. I'm using the data in a python script, and the json files are quite large, taking up a lot of memory
[08:46:46] <shambat> so I'd rather query a db than load in the whole json file into memory
[08:47:06] <JT-EC> mongodb is not the best choice if you're trying to cut down on memory usage.
[08:47:10] <sflint> sure....then why do you want to limit the RAM mongod is using?
[08:47:50] <sflint> how big is huge? shambat?
[08:48:07] <rspijker> yruss972: how large is you local db? (By default, mongodump doesn;t dump the local)
[08:48:22] <shambat> sflint: the files are 250 MB
[08:48:30] <sflint> for a single json?
[08:48:36] <sflint> you can't store that in mongodb
[08:48:44] <sflint> 16MB file document limit
[08:49:02] <shambat> so not gigantic, but when they're loaded into mamory, my VPS with 4GBs of memory crashes due to full memory
[08:49:20] <shambat> sflint: not a single json
[08:49:29] <shambat> well, it doesnt have to be
[08:49:29] <sflint> oh
[08:49:36] <yruss972> rspijker: db.Status() shows "dataSize" : 2723693564
[08:49:43] <shambat> the file I store it in now is 250 MB
[08:49:59] <sflint> oh well that is nothing
[08:50:06] <rspijker> yruss972: so.. that’s about 2.5GB?
[08:50:30] <yruss972> yeah- is that normal?
[08:50:35] <sflint> shambat: when you say loaded into memory?
[08:50:38] <sflint> what do you mean?
[08:50:45] <sflint> are you running a query?
[08:51:02] <yruss972> seems like a lot for such a small DB
[08:51:10] <rspijker> yruss972 it can be. It’s a capped collection (well the most important on in there, the oplog, is) and it defaults to 5% fo free FS space on 64bit linux
[08:51:37] <yruss972> I have 124G on the fs
[08:51:42] <rspijker> if you have an active system, the oplog could live mostly in memory
[08:52:13] <yruss972> oplog is writes and updates?
[08:52:18] <yruss972> it's not so active
[08:52:22] <yruss972> backend for a cms
[08:52:39] <rspijker> ok
[08:52:58] <sflint> yruss972: oplog is every operation
[08:53:09] <sflint> lol
[08:53:37] <sflint> writes,updates....possibly index creation
[08:53:44] <sflint> but mostly writes,updates
[08:54:49] <yruss972> sflint: "The oplog (operations log) is a special capped collection that keeps a rolling record of all operations that modify the data stored in your databases." ?
[08:54:57] <sflint> yeah
[08:55:00] <sflint> indexes too
[08:55:04] <yruss972> so reads aren't included
[08:55:08] <sflint> correct
[08:55:13] <yruss972> we don't update so much
[08:55:23] <yruss972> editing pages in backend, etc.
[08:56:32] <shambat> sflint: sorry I was afk there ... right now I have nothing in mongodb. I have a python script that does json.load(myfile), which is very demanding on memory. I was hoping that storing the data in mongodb would allow me to just query the database instead of loading the whole thing to memory, thusly preventing the server to crash due to lack of memory
[08:57:01] <sflint> you are correct
[08:57:12] <sflint> you will just page fault
[08:57:18] <sflint> but that is okay
[08:57:27] <rspijker> yruss972: can you run this script:
[08:57:29] <rspijker> var c=0;use admin;db.runCommand({"listDatabases":1}).databases.forEach(function(x){db=db.getSiblingDB(x.name);c+=db.stats().dataSize;})print(c);
[08:57:38] <sflint> if you use an index on the data you will be fine....then the query will be fast as well
[08:58:11] <shambat> sflint: so mongo will use disk if memory is getting full?
[08:58:28] <rspijker> shambat: no, mongo itself won’t
[08:58:47] <rspijker> it uses memory mapping, so it will always use memory
[08:58:57] <rspijker> the kernel will swap pages in and out as required
[08:59:02] <shambat> I see
[08:59:05] <yruss972> SyntaxError: Unexpected identifier
[08:59:22] <shambat> so it will page fault, but then swap and try again with success
[08:59:37] <rspijker> yruss972: that’s odd….
[08:59:56] <yruss972> missing a ; I think
[09:00:01] <yruss972> before the last print
[09:00:01] <sflint> shambat: it will pull all it can into memory
[09:00:11] <sflint> then it will pull from disk...these will cause page faults
[09:00:18] <yruss972> nope :?
[09:00:31] <sflint> but not a huge concern if you don't have a lot of traffic to the databse
[09:00:40] <rspijker> yruss972: yes
[09:00:48] <rspijker> yruss972: try this one instead: var c=0;db.adminCommand({"listDatabases":1}).databases.forEach(function(x){db=db.getSiblingDB(x.name);c+=db.stats().dataSize;});print(c);
[09:01:11] <shambat> sflint: this is a script that runs hourly and will add data and compare stored data
[09:01:14] <yruss972> 2840483772
[09:01:38] <rspijker> yruss972: hmmm, 2.6GB
[09:02:15] <rspijker> and most of it will be memory mapped twice (for reading and writing)
[09:02:38] <yruss972> I added indexsize here also: 2869901020
[09:02:38] <rspijker> maybe smartOS isn’t the best in terms of handling memory mapped files
[09:03:27] <yruss972> do you think limiting the oplog size will help- most of that seems to be the local db?
[09:03:40] <Derick> no, don't do that
[09:03:46] <sflint> shambat: hourly is not bad
[09:03:58] <sflint> if it ran 100 times a second that would be bad
[09:04:19] <yruss972> Derick: you mean me?
[09:04:34] <sflint> shambat: also if you write your queries correctly and use projection you won't return the entire set of data
[09:04:36] <sflint> only the working set
[09:04:55] <Derick> yruss972: yes
[09:06:39] <sflint> yruss972: are you using sharded clusters or just replica sets?
[09:06:55] <yruss972> sflint: no sharding- just replicas
[09:07:05] <yruss972> we only have 100MB of data :?
[09:07:20] <sflint> then you should never have an issue
[09:07:28] <sflint> how much RAM?
[09:07:30] <sflint> 4G?
[09:07:36] <yruss972> yep- per machine
[09:07:57] <sflint> and what is the error you are getting?
[09:08:01] <sflint> what OS is this on?
[09:08:19] <sflint> sorry if you already said...
[09:08:42] <yruss972> no errors, just using 5G+ of RAM and vsize of 119GB on SmartOS
[09:08:57] <yruss972> but the 5G+ is more than the 4G RAM
[09:09:33] <sflint> this from mongostat?
[09:10:11] <yruss972> mongostat, prstat, etc
[09:10:29] <sflint> where are you seeing 5+ GB of Ram being used?
[09:10:31] <yruss972> mongostat output here: https://gist.github.com/anonymous/a1cbe839fd1d40258e71
[09:10:38] <yruss972> both
[09:11:44] <sflint> i think that is common on mongostat
[09:11:48] <sflint> i don't know why
[09:11:51] <sflint> but look at this
[09:12:04] <sflint> 284g 572g 37.2g
[09:12:12] <sflint> mapped vsize res
[09:12:16] <sflint> 284g 572g 37.2g
[09:12:44] <sflint> that is on a machine with
[09:12:44] <sflint> Mem: 264477632k total, 261714052k used, 2763580k free,
[09:13:09] <rspijker> in an active system it is normal and desirable to have almost all RAM utilized
[09:13:10] <sflint> yruss972: were you just curious why or you are seeing performance issues?
[09:13:17] <rspijker> he’s seeing swapping though, which is causing problems
[09:13:19] <yruss972> I wouldn't mind except we really don't have that much data
[09:13:25] <yruss972> and the swapping
[09:13:29] <rspijker> yruss972: what does sar -B show?
[09:13:37] <rspijker> does SmartOS have sar?
[09:13:38] <yruss972> no sar
[09:14:10] <yruss972> what does sar -B show
[09:14:20] <yruss972> I can probably find an equivalent
[09:14:20] <rspijker> paging stats
[09:15:16] <yruss972> vmstat -p: https://gist.github.com/anonymous/9c34449959e01fac045e
[09:17:24] <yruss972> I restarted some of the other machines in the cluster last week for similar issues (left this one to try to find the problem)
[09:17:59] <yruss972> they are all showing 2.3g resident, 29g vsize, and 8.8g mapped
[09:21:03] <rspijker> I’m more interested in swapins and swapouts
[09:22:22] <JT-EC> vmstat -S
[09:25:59] <yruss972> not much there
[09:42:56] <JT-EC> yruss972: The issue seems to be that SmartOS zones don't have absolute memory limits and mongo doesn't seem to be able to cope very well with that. SmartOS lets you take more memory but swaps it and it kills performance as you're finding.
[09:43:41] <JT-EC> Because of the soft mem limits and (from the docs) "MongoDB automatically uses all free memory on the machine as its cache" this tends to screw SmartOS users.
[09:44:04] <JT-EC> Not sure how Joyent make this all work.
[09:44:44] <yruss972> JT-EC: any idea how mongo determines how much free memory there is?
[09:44:54] <Derick> yruss972: it doesn't care, it just allocates more
[09:45:16] <Derick> it leaves it to the OS to manage the swapping
[09:45:50] <yruss972> ok- but I still don't understand why it should be trying to take more memory when I only have 105MB of data :?
[09:51:16] <JT-EC> yruss972: Look at what I quoted from the docs - "all free memory" - doesn't mention anything about your data size only mongo's greed.
[09:52:09] <JT-EC> So you could try ulimits to fix this.
[09:58:00] <nylon> lo
[09:58:51] <nylon> anyone know why text search data that contains periods have strange behavior on search results?
[09:59:06] <nylon> e.g. "1234.67"
[10:01:29] <nylon> never mind, found the answer
[10:38:53] <dragoonis_> Looking to effectively do upserts on my collections, is this a popular thing for mongoDB?
[10:39:27] <dragoonis_> I need a composite unique index of "survey_id, category_id, results_date (ISODate)) to upsert upon
[10:39:55] <dragoonis_> s/composite/compound
[10:46:21] <_Nodex> dragoonis : if it works for your app then I don't see why not :)
[10:49:04] <dragoonis> I had another mongoDB question about indexes, i just made it now - maybe you have some feedback for me here too? :) http://stackoverflow.com/questions/24734811/hiow-to-mongodb-ensure-index-upon-collection-creation
[10:51:48] <_Nodex> do it in your app
[10:52:04] <_Nodex> you can "getIndexes()" to see if it exists else run it
[10:54:43] <dragoonis> _Nodex, ok
[11:00:04] <dragoonis> fun times..
[11:00:06] <dragoonis> > db.company_category_score_months.ensureIndex( { "survey_id" 1, "category_id": 1, "results_date": 1 }, {unique: true} )
[11:00:06] <dragoonis> 2014-07-14T05:56:37.717-0500 SyntaxError: Unexpected number
[11:00:23] <dragoonis> found it
[11:07:56] <phr3ak> hello, do I have to use cman/rgmanager if I want HA? currently it is not clear for me how mongod could work in HA?
[11:09:44] <_Nodex> cman?
[11:09:49] <_Nodex> rgmanager?
[11:09:52] <phr3ak> redhat cluster
[11:11:41] <phr3ak> so if i set the database server ip in our application and if the node will be unreachable how to know the application the ip address of the working node? how the HA works?
[11:13:05] <phr3ak> is there any virtual ip support in mongodb?
[11:13:35] <_Nodex> your mongos takes care of that iirc
[11:13:50] <rspijker> if you have just a single replica set (no mongos, no sharding)
[11:13:54] <_Nodex> http://docs.mongodb.org/manual/reference/program/mongos/
[11:13:55] <rspijker> then you just use a connectino string
[11:13:58] <rspijker> *conneciton
[11:14:06] <_Nodex> or that ^^
[11:14:09] <rspijker> as in, you pass it multiple URLs, primary, secondary, etc.
[11:14:44] <rspijker> if you pass a single mongos, then you’re just moving your single point of failure
[11:14:57] <rspijker> unless you just run the mongos on your app server, of course
[11:16:32] <phr3ak> if i understand well in the application we have to set more than one node in the connection string?
[11:16:53] <_Nodex> yes
[11:17:07] <_Nodex> string1;string2;url3;...
[11:17:13] <phr3ak> thank you!
[11:18:34] <_Nodex> sorry it might be a comma
[11:19:11] <_Nodex> http://docs.mongodb.org/manual/reference/connection-string/
[11:19:46] <_Nodex> apparently semi colon is supported for backwards compatibility
[11:20:09] <Derick> _Nodex: not supported everywhere though
[11:22:15] <_Nodex> :/
[11:23:04] <Derick> (the ; that is)
[11:23:13] <Derick> i think for new implemenations, we weren't supposed to support it
[11:23:23] <Derick> hence, the PHP driver does only support ,
[11:23:26] <_Nodex> docs prolly need an update
[11:23:38] <_Nodex> they imply that's it's left there on purpose
[11:24:04] <Derick> _Nodex: you're misreading it
[11:24:15] <Derick> it's for *arguments* instead of &
[11:24:30] <Derick> so:
[11:24:31] <Derick> ?replicaSet=test&connectTimeoutMS=300000
[11:24:32] <Derick> or:
[11:24:35] <Derick> ?replicaSet=test;connectTimeoutMS=300000
[11:25:12] <Derick> hosts can only be separated by ,
[11:28:14] <phr3ak> i don't know what is that arbiter but someone told to me we want to use this redhat clustering for the arbiter...
[12:33:04] <backSlasher> Hi.
[12:33:04] <backSlasher> we are expiriencing really slow queries (>3s) with a collection that holds medium (<10MB) files.
[12:33:04] <backSlasher> I don't have a very high lock percentage (~10%-20%) but I do see a very high write queue on my disk, once every ~20s.
[12:33:04] <backSlasher> AFAIK it's no the data flush because the console says that they are one minute apart. "accessNotInMemory" is lower than 1 for the relevant collection.
[12:33:05] <backSlasher> Any thoughts about what to check next?
[12:34:31] <cheeser> do an explain on your query. make sure you're using an index.
[12:35:19] <backSlasher> it's an upsert based on id, and idhack is true for these queries, so index is probably not the issue
[12:35:57] <cheeser> better to check than assume
[12:36:14] <cheeser> and are you running 3s queries or 3s upserts?
[12:38:23] <backSlasher> "idhack" in profiling means that the query is using the native "_id" index
[12:38:34] <backSlasher> and the 3s are upserts
[12:40:38] <dragoonis> I'm talking the mongodb query log, how do I identify if the query being ran has hit an index or not ?
[12:40:57] <dragoonis> s/talking/tailing *
[12:41:15] <dragoonis> I added some indexes but unsure how to see if they're being hit or not
[12:41:55] <backSlasher> dragoonis, first enable profiling http://docs.mongodb.org/manual/tutorial/manage-the-database-profiler/
[12:42:02] <dragoonis> backSlasher, done
[12:43:33] <backSlasher> dragoonis, actually my bad. just run the query with "explain"
[12:44:03] <dragoonis> didn't know you could do that here :D
[12:44:05] <dragoonis> only in mysql
[12:44:10] <backSlasher> dragoonis, http://docs.mongodb.org/manual/reference/method/cursor.explain/#cursor.explain
[12:45:13] <dragoonis> got it explained, what field here identifies if an index was hit
[12:45:34] <dragoonis> I got this output: https://gist.github.com/dragoonis/c84a1f79ea5b4dad0052
[12:46:00] <dragoonis> I assume indexOnly ?
[12:46:49] <backSlasher> dragoonis, I think according to this
[12:46:52] <backSlasher> dragoonis, http://blog.mongohq.com/explain-explain-understanding-mongo-query-behavior/
[12:47:07] <backSlasher> dragoonis, that "basicCursor" means no index - just scanning
[12:48:07] <backSlasher> cheeser, any thoughts about my problem? Any way to trace mongo's writes to disk?
[12:48:27] <dragoonis> backSlasher, ok got it, i added an index on results_date field and then queried on it.
[12:48:32] <dragoonis> I got this: "cursor" : "BtreeCursor results_date_1",
[12:48:44] <dragoonis> which is good, but I have "indexOnly" : false,
[12:48:50] <backSlasher> dragoonis, this means that it used the index. It should also scan less objects
[12:48:50] <dragoonis> Does this imply I have more work to do ?
[12:49:20] <dragoonis> The docs on indexOnly said ... "indexOnly is a boolean value that returns true when the query is covered by the index indicated in the cursor field."
[12:49:36] <dragoonis> and the only condition I have is results_date, so it should fully cover this index?
[12:49:42] <dragoonis> > db.company_category_score_months.find( { results_date: { $gte: new Date(1404169200000), $lte: new Date(1406761200000) } }).explain()
[12:49:55] <backSlasher> dragoonis, why are you not using range?
[12:50:18] <dragoonis> backSlasher, will a Range give me an indexOnly trigger?
[12:50:25] <dragoonis> (set value to true)
[12:50:45] <backSlasher> dragoonis, dunno but it tells mongo more specifically what you're trying to achieve
[12:50:58] <backSlasher> dragoonis, try using a direct comparison (A==B) and then see if indexonly lights up
[12:51:10] <dragoonis> backSlasher, can you give an example of a==b ?
[12:51:20] <dragoonis> you mean results_date == Date(1404169200000) ?
[12:51:29] <backSlasher> dragoonis, {results_date: new Date(1234567)}
[12:51:31] <dragoonis> ok
[12:51:46] <backSlasher> ofc it wouldn't serve the same results, but just for the experiment
[12:52:22] <backSlasher> I think indexonly is when mongo doesn't have to do any sacnning, and in your case you'll always have to scan something because you have no specific number, only a range
[12:52:47] <dragoonis> > db.company_category_score_months.find( { results_date: new ISODate("2014-07-01T00:00:00Z") }).explain()
[12:52:57] <dragoonis> found the 2 matched rows, but indexOnly is false
[12:53:16] <backSlasher> dragoonis, well don't know then... try by id? :)
[12:54:56] <dragoonis> same result - https://gist.github.com/dragoonis/d171cbd9cb69f27610a4
[12:56:49] <backSlasher> maybe try only fetching the id like find({id:1234546},{id:1})
[12:57:17] <dragoonis> ah, yea
[12:57:18] <dragoonis> that's it
[12:57:22] <dragoonis> > db.company_category_score_months.find( { _id: "53c3b5cb34d6199c7e8b465b" }, {_id:1} ).explain()
[12:57:28] <dragoonis> "indexOnly" : true,
[12:57:52] <dragoonis> So indexOnly is when it only returns fields that are fully indexed..
[12:58:40] <backSlasher> I think "only when the fields requested are in indexes used to find the records"
[12:58:54] <backSlasher> in short, irrelevant here
[12:59:01] <dragoonis> yep, i understand it now. So I need all fields back of course so indexOnly will always be false for me
[12:59:41] <dragoonis> Is having a BTreeCursor good enough for indexing (quickly scanning documents)
[12:59:45] <dragoonis> ?
[12:59:50] <backSlasher> it means that it used an index
[13:00:23] <backSlasher> the real thing to note is to compare "n" (number of objects returned) to "nscanned" (or something similar), which is how many objects mongo manually compared
[13:00:42] <backSlasher> you have to keep the ratio high, meaning that mongo doens't scan objects you don't want
[13:01:24] <dragoonis> n = 2 and nscanned = 2 .. so we're good here :)
[13:01:44] <dragoonis> backSlasher, thanks for your time and helpful explanations!
[13:05:50] <backSlasher> dragoonis, sure thing. I actually came here seeking help for my own problem... :-)
[13:07:16] <dragoonis> We're all at diff points in the food chain
[13:07:30] <dragoonis> I can now help people with indexing questions a bit :)
[13:07:43] <backSlasher> :-D
[13:17:33] <backSlasher> we are expiriencing really slow upserts (>3s) with a collection that holds medium (<10MB) files. They have "idhack", meaning the native "_id" index is used.
[13:17:33] <backSlasher> I don't have a very high lock percentage (~10%-20%) but I do see a very high write queue on my disk, once every ~20s.
[13:17:33] <backSlasher> AFAIK it's no the data flush because the console says that they are one minute apart. "accessNotInMemory" is lower than 1 for the relevant collection.
[13:17:33] <backSlasher> Any thoughts about what to check next?
[13:21:00] <alanhoff> Hey guys, I'm willing to receive a JSON and use it as the a query param inside find(), is this ideia ok?
[13:46:11] <Cactusarus> can anyone help me figure out if I can use browseify with mongodb in node.js?
[13:49:12] <_Nodex> eh?
[13:49:36] <_Nodex> mongodb is a database, not a front end tech
[13:49:48] <Cactusarus> oops
[13:50:09] <Cactusarus> I Saw the npm package, I'll install seperatley
[14:28:30] <uehtesham90> hey, i want to make a copy of a collection in a database....there are two options: using copyTo() and cloneCollection....im not sure which is a better option
[14:34:09] <obiwahn> http://paste.debian.net/109545/ -- Is there a faster/better way to delete the homework with the lowest score?
[14:36:53] <uehtesham90> im doing the same course...lol
[14:37:18] <uehtesham90> @obiwahn....i used python though
[14:38:07] <obiwahn> how did you do it?
[14:38:49] <obiwahn> i just think it is best to do stuff you can do in the db inplace instead sending stuff over the driver
[14:39:05] <pseudo_> I have a 3 node cluster that "ran out" of disk space. I am getting "Can't take a write lock while out of disk space" errors. However, each machine in the cluster has at least 900MB of disk space left. Why am I getting these messages still and how can I force mongo to delete some data?
[14:39:10] <uehtesham90> https://www.irccloud.com/pastebin/UkSxMYFP
[14:39:28] <uehtesham90> @obiwahn u can have a look at my code in the above link
[14:40:04] <rspijker> obiwahn: you probably have the quickest way
[14:40:37] <rspijker> pseudo_: after a while, mongod will allocate disk space in chunks of 2GB
[14:40:38] <cheeser> findAndModify would be atomic
[14:41:16] <rspijker> pseudo_: to get mongod to return disk space to the filesystem, either remove the files and resync from another node in the replica set, or run repairDatabase
[14:42:24] <pseudo_> rspijker: so if i remove the files directly from the filesystem, i can run repairDatabase and all the other data should be fine?
[14:43:42] <rspijker> pseudo_: no… you EITHER remove the files and sync back from another replica set member, OR you run repairDatabase
[14:43:49] <rspijker> for repairDB you don’t have to remove anything
[14:44:11] <rspijker> it will just rewrite the files, ‘unfragmented’
[14:44:52] <rspijker> however, if your DB files are already fairly unfragmented, which can happen if you don;t have many removals or updates. Then repairDatabase won’t give you much space back.. but then again, neither will anything else. Because you just have that amount of data
[14:45:41] <pseudo_> yeah, repair database didn't give a whole lot. i dont think i had a single removal/update
[14:45:51] <obiwahn> uehtesham90: lowest_score = min(entry['scores'][2],entry['scores'][3]) - ok works but not really elegant you could have matched for homework
[14:46:34] <pseudo_> each node in the cluster is short on disk space and the only place with 2gb of data is /var/lib/mongodb. is it safe to delete files straight from here?
[14:46:46] <obiwahn> :P on the other hand i have wasted too much timer over it:P
[14:48:15] <rspijker> pseudo_: if reapirDB doesn’t give you back any space, neither will a deletion
[14:48:34] <rspijker> apparently, you just have that much data in your system
[14:48:42] <pseudo_> rspijker: i have a 98GB database that I want to completely drop. I just don't have enough disk space to perform the drop operation
[14:48:46] <rspijker> or, it’s distributed really uncomfortably and you have a lot of overhead
[14:49:01] <rspijker> o, then you can just remove the file :)
[14:49:05] <rspijker> files*
[14:49:15] <rspijker> just, shut down the db first
[14:49:18] <pseudo_> okay, it won't screw up metadata or anything?
[14:49:42] <rspijker> as long as you remove the correct files, no...
[14:49:55] <pseudo_> also, should i stop the whole cluster and remove it all at once? will the cluster be able to come back online if they all go down?
[14:50:11] <rspijker> easiest to test with 1
[14:50:16] <rspijker> bring down a secondary
[14:50:18] <rspijker> remove the files
[14:50:19] <rspijker> bring it back
[14:50:23] <rspijker> drop on that secondary
[14:51:51] <rspijker> if something does go wrong, at least you’ll have an intact copy of your data elsewhere
[14:53:14] <pseudo_> it looks like when I try to bring the node back online, it tries syncing the data but can't
[14:53:22] <pseudo_> the mongod process just stops
[14:54:15] <rspijker> with what message?
[14:56:05] <pseudo_> rspijker: http://pastebin.com/Rpwb5243
[15:00:52] <rspijker> pseudo_: which files did you remove?
[15:00:57] <rspijker> and, which are still there?
[15:01:56] <pseudo_> rspijker: i just restored the files that i deleted. i took away the 0th file from the packets database. i am going to try removing a different one.
[15:02:20] <rspijker> I thought you wanted to drop the entire thing?
[15:02:45] <pseudo_> rspijker: i do, but i was just trying to free enough disk space to perform the drop command in mongo
[15:03:01] <rspijker> well… there is a packets.ns file
[15:03:10] <rspijker> the namespace file
[15:03:10] <pseudo_> yes there is
[15:03:25] <rspijker> it keeps track of where the extents are
[15:03:28] <rspijker> among other things
[15:03:40] <rspijker> so… might very well be that it just checks all of the files are present
[15:04:05] <rspijker> making it kind of an all or nothing thing…
[15:04:09] <pseudo_> so if i delete ALL the packets.* files, including the ns file - perhaps it will come back online wihtout the database?
[15:05:38] <rspijker> correct
[15:05:56] <rspijker> it should, really...
[15:06:51] <dragoonis> Derick, ping
[15:07:05] <Derick> hi
[15:07:26] <dragoonis> Missing documentation on what 'multi' implies here: http://docs.php.net/manual/en/mongowritebatch.add.php
[15:08:17] <pseudo_> rspijker: alright, it seems to be back up and running. thanks for the help!
[15:08:24] <rspijker> np
[15:08:48] <dragoonis> Derick, I believe upsert and multi need a 1 line description of each. I'm happy to add this now if you can approve my wording of 'upsert' and you tell me what 'multi' means :)
[15:09:13] <Derick> multi means that one update statement can update more than one document
[15:09:40] <og01> If i want a replicaset with only 2 instances, how can I ensure that both instances become primary when network connectivity goes down?
[15:10:05] <og01> or is that the normal situation anyway
[15:10:46] <dragoonis> Derick, ok
[15:10:57] <Derick> og01: you can not do that
[15:11:02] <og01> basicly I want both sites to continue to function even after network connectivity fails between the two
[15:11:03] <Derick> it would be a disaster for data consistency
[15:11:20] <og01> we dont mind handling the roll back data by hand
[15:11:29] <og01> Derick: if you have a different recomendation?
[15:11:46] <Derick> for doing what exactly?
[15:12:26] <rspijker> og01: would picking 1 to always become the primary be good enough?
[15:12:31] <og01> we have two sites, one mongo instance in each, and a service running at each site collecting data,
[15:13:00] <og01> if network connectivity fails between the two, we want both sites to continue to accept writes
[15:13:17] <og01> i understand that there can be conflicts in that scenario
[15:13:38] <og01> rspijker: what would happen when the secondard gets a write opperation?
[15:14:11] <rspijker> it wouldn’t accept it
[15:14:44] <rspijker> there’s no way to do it through the regular replication/failover mechanism that I’m aware of
[15:15:08] <rspijker> maybe if you did some scripting around detecting it and then spinning up an arbiter or something
[15:15:08] <og01> http://docs.mongodb.org/manual/core/replica-set-high-availability/
[15:15:14] <dragoonis> Derick, if I understand this right, 'u' is the updated data and 'q' is the query used to pull this record out ?
[15:15:16] <rspijker> but I’d advise strongly against that :/
[15:15:55] <rspijker> og01: yeah… look at the election document. It requires a majority
[15:23:10] <dragoonis> Derick, how does this look? https://gist.github.com/dragoonis/cc39aeb7b5cde622ced9
[15:24:22] <Derick> that doesn't do anything?
[15:24:28] <ejb> Is there a way to specify that id's should be strings not ObjectIds? Meteor uses strings so it doesn't play nice with documents that I've loaded via mongoimport
[15:31:20] <saml> ejb, doc json can contain _id field and mongoimport would respect that
[15:32:08] <ejb> saml: hm, ok. How can I generate good ids to put in my json file?
[15:33:37] <saml> let mongodb generate one
[15:33:43] <saml> but your problem is meteor
[15:33:57] <saml> you can copy whatever meteor is doing to generate ids
[15:34:09] <saml> or, don't use garbage like meteor
[15:36:00] <ejb> saml: hater
[15:45:29] <Nodex> lol
[16:34:31] <jParkton> Im having a couple of simple problems connecting to my mongodb via node I was wondering if someone could help point me in the right direction?
[16:34:45] <jParkton> https://gist.github.com/jparkton/1f10f53f4dcb18e570c8
[16:35:01] <jParkton> ignore the .find() I know thats off
[16:36:49] <jParkton> Im trying to find my collections and return them I looked here http://runnable.com/UW3vaGTkq498AACC/query-for-records-in-mongodb-using-mongodb-native-for-node-js but its not working in my case maybe I am missing some package?
[17:04:10] <og01> jParkton: you seem to have various race conditions in your code
[17:05:14] <jParkton> racial code?
[17:05:17] <jParkton> wut
[17:05:49] <og01> im not 100% familiar with nodejs' driver, but it looks like to be that your connection isnt ready before you try to use it
[17:05:55] <og01> give me a moment
[17:06:00] <jParkton> og01: Im pretty new to this whole thing, I guess the issue is in my getCollection there is no find method?
[17:06:07] <jParkton> ahh
[17:08:12] <og01> jParkton: yest perhaps look at this tutorial: http://mongodb.github.io/node-mongodb-native/api-articles/nodekoarticle1.html
[17:08:39] <og01> jParkton: see how the connect method takes a callback which provides a db object
[17:08:58] <jParkton> yes
[17:09:02] <og01> you then use that object to access the collections and other opperations
[17:09:09] <jParkton> right
[17:09:18] <jParkton> like I did db.databaseName
[17:09:45] <jParkton> and I can get that but if I do db.myCollection.find() I get an undefined
[17:10:16] <og01> getCollection in your example isnt returning anything usefull
[17:10:26] <og01> so collection is undefined
[17:10:58] <og01> infact its difficault to return anything meaningful at that point unless you use something like a promise
[17:11:08] <jParkton> I keep hearing this word
[17:11:14] <og01> which word?
[17:11:19] <jParkton> promises
[17:11:30] <jParkton> I need to google some things it seems lol
[17:11:41] <og01> right i can give you a short lession on event loops an async/non-blocking operations
[17:12:21] <jParkton> ok
[17:13:31] <og01> nodejs (and/or javascript in general) runs an event loop. this loop is native to nodejs (in some languages it must be implemented). This loops allows the programmer to setup watchers, listeners and fire events
[17:13:55] <jParkton> yes
[17:13:59] <og01> operations you perform in the event loop should be non-blocking, that is, code execution should never pause and wait
[17:14:38] <jParkton> true
[17:14:38] <og01> instead it should setup callbacks/events that occur some time in the future (my description here isnt excellent, but hopefully you follow)
[17:14:50] <jParkton> yes
[17:15:24] <jParkton> I need to setup something in there to just find my collection and return it and stop executing
[17:15:27] <jParkton> I think?
[17:15:48] <og01> no but i can understand your thinking
[17:16:18] <og01> so methods on MongoClient (and other places), take a callback (function) that gets called some time in the future with the results
[17:17:37] <jParkton> right
[17:17:52] <jParkton> Im making a simple login / logout app
[17:18:11] <og01> so with the connect command you only have access to the db object once the callback is called... you cant reliably use this object, and you can't just set it to a global variable because you wont actually know when its ready
[17:18:12] <jParkton> so I need to somehow setup a backend so I am not hard coding it all in my exposed files
[17:18:29] <og01> really what you end up with is a load of nested callbacks....
[17:18:33] <jParkton> but I am quite nubly
[17:18:34] <jParkton> lol
[17:18:45] <jParkton> yeah
[17:18:52] <og01> sorry if i go slow, it can be difficault to describe, and i want to make sure you get the whole picture
[17:18:56] <jParkton> the ones on runnable run
[17:19:03] <jParkton> mine dont so Im doing it wrong somehow
[17:19:32] <og01> MC.connect(url, function (err.db) {db.find({...}, function (blah) {....}}
[17:19:45] <og01> it can become a mess
[17:20:20] <og01> also the return from function calls/methods become meaningless
[17:21:00] <og01> now, with this understanding, you can work around things, you can set flags on variables when things are ready, setup timers and work wround things...
[17:21:16] <og01> delays can be setup and all sorts to help bodge things into working
[17:21:26] <og01> but this is not a good way to go around things
[17:21:36] <og01> you can pass arround callbacks, and such
[17:21:42] <og01> and this is what you should do
[17:22:07] <og01> so getCollection in your example should take a callback
[17:22:18] <og01> which it should call when the db is ready
[17:22:25] <og01> etc
[17:22:50] <jParkton> ok let me try this again
[17:22:51] <og01> im not explaining myself well at this point... sorry but do you think you follow
[17:22:57] <jParkton> I think I get why it was failing
[17:23:16] <jParkton> I think so, but Ihave been wrong before lol
[17:24:00] <og01> MongoClient.connect() completes immediatly - it does not block, but the function you provide to it is called some time in the fucture when the function is complete
[17:24:26] <og01> but by that time you've already left your function (connectDatabase or getCollection)
[17:25:09] <og01> now I like promises, but some people I work with do not, i personally think some people simply fail to understand them.
[17:25:31] <og01> as promise, is the promise of a result some time in the future.
[17:26:20] <og01> it is an object you can store, and you can call promise.then(function(db) {...do something with ...});
[17:29:15] <og01> it makes it so you can rebuild a more familiar flow to your code, you can return a promise of your results, be it successfull or failure, it makes returning a value usefull again, and also provides the equvilient of a thrown error via rejection possible
[17:30:11] <og01> I think you should perhaps grasp how to pass around callbacks through your code first
[17:30:32] <jParkton> https://gist.github.com/jparkton/1f10f53f4dcb18e570c8
[17:30:40] <og01> and then later perhaps read up about promises, to see it that makes your life easier in the future.
[17:30:43] <jParkton> I at least have my collection now lol
[17:31:59] <og01> yes, let me show you something based on your example though
[17:32:22] <jParkton> ok
[17:37:06] <og01> jParkton: http://0bin.net/paste/Eamh-ruAuGKYtSwX#xajgiczFiyAvbBR9-FZgiXeNEvBtyY3koT9f1pQKf2y
[17:38:12] <og01> jParkton: not tested, but i was trying to show how you can pass a callback to get the results from non-blocking async functions
[17:38:29] <jParkton> right
[17:38:33] <jParkton> that makes sense
[17:38:45] <jParkton> now to start thinking like that and I am set
[17:38:47] <jParkton> lol
[17:39:02] <og01> let me show you how you might handle your db object
[17:39:12] <og01> because this example could open many connections
[17:39:18] <og01> and that isnt what you want
[17:39:35] <jParkton> I need to make it so I can query my db
[17:39:47] <jParkton> from my login page
[17:43:20] <og01> jParkton: http://0bin.net/paste/UnPq653TpdUyGl8Y#bkokqhmbfzunt+6KRCGIqNvOVhbGbRFpmUxV1UsPlaB
[17:43:40] <og01> take a look at this (again untested) example
[17:43:59] <og01> please note that i wouldnt write code like this, but it is an example for you to learn from
[17:44:49] <og01> in this example getDatabase reuses the same db object every time it is called
[17:45:10] <og01> if it isnt already created, it connects stores the db object and calls the callback
[17:45:33] <og01> if it already is created then it calls the callback with the already existing db object
[17:46:41] <og01> as for your querying from your login page, you'd need to know your webserver/webframework and have an understanding of HTTP and potentially XHR/AJAX
[17:47:24] <og01> perhaps lookup how to write a RESTfull interface in express or some such
[17:48:33] <og01> jParkton: i have to go in 5 min, so any final questions ask now
[17:48:46] <og01> in anycase i hope i've helped more than hindered :)
[17:53:52] <jParkton> lol
[17:54:00] <jParkton> I think I have enough to chew on
[17:54:05] <jParkton> but thank you
[18:15:45] <apetresc> Hey, guys; I'm having trouble getting scons to respect the targets I tell it to build. Basically, even if I say something like "scons tools install --prefix=/whatever" it will insist on building and installing *everything*
[18:15:50] <apetresc> mongod, mongos, etc
[18:16:13] <apetresc> How do I convince scons to only build a subset of the targets? I only need to build the tools and/or client, not the server
[18:17:06] <ranman> apetresc: most of the tools compile the server as a prereq
[18:17:24] <apetresc> ranman... oh :(
[18:17:31] <apetresc> ranman, so there's no way to only build the client?
[18:17:47] <apetresc> Or at least only *install* the client, even if it does get built as an intermediate step?
[18:17:56] <ranman> apetresc: do you mean the C++ driver or the shell?
[18:17:58] <apetresc> It's not like `mongo` requires `mongod` to be present to execute
[18:18:03] <apetresc> The shlel
[18:18:04] <apetresc> *shell
[18:19:01] <ranman> apetresc: let me take a look
[18:19:13] <apetresc> Thanks so much :)
[18:38:26] <rickibalboa> Hi, I have a strange situation where a mongo query performed from a nodejs process hangs and ends up timing out. The database connection is made, as soon as I run any find() queries they hang, but they work if I use mongodb shell / robomongo.
[18:38:37] <context> with the aggragation framework can i $project an embedded documents field (this is an array of documents not a single embedded document)
[18:38:43] <context> im thinking i might have to mapReduce it
[18:40:03] <apetresc> What do you mean, context? Like, you want to only project some of the documents in the array, or the entire field (which happens to be an array)
[18:40:26] <context> well im doing a count based on the embedded documents
[18:41:07] <context> primary { blah []secondary }, many secondary, i want to project fields out of every embedded doc, some records only have one in the array, some have more
[18:41:53] <ranman> apetresc: scons mongo -j 8 or something like that seems to just build the shell for me
[18:43:28] <apetresc> ranman, not me :( I create a brand new directory (/Users/apetresc/tmp/mongo) and run exactly: `scons mongo install --prefix=/Users/apetresc/tmp/mongo`
[18:44:24] <apetresc> And then `ls ~/tmp/mongo/bin` returns everything... mongo, mongod, mongos, etc
[18:45:12] <apetresc> I feel like maybe even if the `mongo` target only builds the shell, the `install` target accidentally depends on `all`, perhaps?
[18:47:15] <ranman> apetresc: seems to be the case, I don't really want to go through the sconstruct atm though :/, my recommendation scons mongo -j8 then copy the build artifacts to wherever you want them to be
[18:47:54] <apetresc> ranman, sigh, I see :( If I track this down and submit a patch for it, there's no reason it wouldn't get pulled, right?
[18:48:15] <ranman> I doubt it would get pulled actually
[18:48:20] <ranman> install is an alias, not a target
[18:48:29] <ranman> and it's probably depended on my something
[18:48:51] <apetresc> So... what's the point of having individual fine-grained targets if it's by-design impossible to install any of them?
[18:49:02] <apetresc> (Other than `all`)
[18:49:45] <ranman> I don't think install was intended to be used that way -- nor do I think mongodb is meant to be installed on machines that way -- better to use the packages
[18:50:04] <ranman> if you're just doing development then you rarely need to "install" anything you just need to run tests
[18:50:47] <apetresc> No, but what if you're building it from source on a machine that just needs a client? You can't envision any scenarios where someone would want to talk to a mongo server without BEING a mongo server?
[18:50:49] <ranman> scons is an interesting beast
[18:51:17] <apetresc> Well anyway, I think I'll just workaround it by manually cp-ing the binaries out
[18:51:47] <ranman> apetresc: there are packages for the just the client, why are you building from source ?
[18:51:49] <apetresc> But for the record, I don't understand why installing a target doesn't install just that target :P But to be fair, I don't know too much about scons
[18:51:51] <ranman> apetresc: it's not that at all
[18:52:34] <apetresc> ranman, in this particular case it's because I'm trying to modify the Homebrew recipe for Mongo to support a --no-server option
[18:52:43] <ranman> apetresc: it's that scons and master and all of these things are for development of mongodb itself -- if you're using them your results are not guaranteed -- when most people deploy a database the want a battle tested release and package
[18:52:44] <apetresc> Homebrew insists on building from source, they don't take binary packages
[18:53:02] <ranman> apetresc: homebrew has a bottle for mongodb
[18:53:07] <apetresc> Right
[18:53:12] <ranman> ranman: I maintain that recipe, I'll look into it
[18:53:13] <apetresc> But they don't have a bottle for just the mongodb CLIENT
[18:53:27] <apetresc> If i want to run the mongo shell on my mac, I have to install the mongo server
[18:53:27] <ranman> that's a good point
[18:53:41] <apetresc> Oh cool, you maintain it? :)
[18:53:57] <apetresc> I was gonna put this patch in as a PR, i guess this is more direct
[18:54:21] <ranman> apetresc: which is an additional 300mb or so :/
[18:54:22] <apetresc> Yeah, it would be cool to have a --server-only flag for the mongodb brew
[18:54:33] <apetresc> errr
[18:54:35] <apetresc> --client-only
[18:54:54] <apetresc> Or even just two separate recipes, mongodb-server and mongodb-client
[18:56:13] <context> nm. i needed $unwind
[19:00:39] <ranman> apetresc: if you're interested in trying to fix it I think the bug is in here somewhere: https://github.com/mongodb/mongo/blob/master/src/mongo/SConscript
[19:01:38] <apetresc> ranman, thanks :)
[19:02:08] <ranman> apetresc: I don't have a lot of time to look into it right now but you have a good point
[19:22:34] <Gargoyle> Are there any channels for the university courses - can't see anything obvious. If there are no other suggestions, I'll be hanging out in #mongodb-m102 for the next seven weeks.
[19:33:45] <coreyfinley> My document has a field that is an array of dates where the given object meets some boolean requirement. so given a date range, could anyone help me craft a query that can find objects that have at least one date in the array that is within the given date range?
[20:17:16] <lfamorim> Hello! Why Mongo locked my database on indexRebuilder operation? I'm crying ... ;(
[20:17:18] <lfamorim> really
[20:17:41] <lfamorim> I have a severe outage because of it
[20:19:39] <kali> lfamorim: well, for this kind of operation, you usually perform them by taking replica out of the replica set one at a time...
[20:22:22] <lfamorim> kali, my server are standalone =(
[20:22:58] <lfamorim> the server come in this condition after a background index finished
[20:22:59] <kali> in production ? this is a disaster waiting to happen
[20:23:21] <lfamorim> kali, mongo need a replica always?
[20:23:36] <patricklewis_> Hey I have question does anyone have a sec to try and help me out
[20:24:09] <kali> any critical database needs a replication or some sort.
[20:24:33] <patricklewis_> More or less a way to find the next document in the DB
[20:24:55] <patricklewis_> meaning if I am viewing information on a document and I want to go to the next document how can I do so?
[20:25:26] <patricklewis_> a great example would be viewing a portfolio of web work. Each project is a document
[20:25:42] <patricklewis_> and I need to be able to find the next and previous documents
[20:25:55] <kali> next and previous according to what ?
[20:25:55] <patricklewis_> so that way I can navigate using arrow icons
[20:26:14] <patricklewis_> so if I am viewing a project and I want to view the next one based on when it was added
[20:27:24] <kali> patricklewis_: you need to have something to materialize this "added" operation. if you're using standard ObjectIds as _id, you're lucky, because they have a built-in timestamp you can use
[20:27:44] <patricklewis_> yep I am using object ids
[20:28:02] <patricklewis_> btw i just started using mongodb so Im using the wrong terminology let me know
[20:28:42] <patricklewis_> example.com/portfolio/view/YxvFYNYQyRyvNXH8e
[20:28:46] <kali> just look for a document with { _id : { $gt: [your current document id] } }
[20:29:04] <patricklewis_> where YxvFYNYQyRyvNXH8e is the document id
[20:29:41] <kali> mmm i'm not sure you get what i mean
[20:30:29] <kali> patricklewis_: mongodb (and bson) comes with a ObjectId type
[20:30:56] <kali> that's what i'm referring to
[20:31:11] <patricklewis_> Basicly I am building a site/webapp using meteor js. I and the trailing part of the url is a parameter telling the app what document to pull the information from
[20:31:29] <patricklewis_> so in this case YxvFYNYQyRyvNXH8e is my _id
[20:31:40] <patricklewis_> however I would like to find the next document
[20:31:47] <kali> well, it does not look like an ObjectID
[20:31:56] <kali> so the objectId trick will not work
[20:32:09] <kali> you need to add a timestamp to your document model
[20:32:13] <patricklewis_> hmm.. I guess meteor does it differently
[20:32:17] <kali> but i don't know how to do that in meteor
[20:32:38] <patricklewis_> I do, how would I find the next document based off of the time stamp?
[20:33:43] <kali> just look for the smaller value of the timestamp strictly higher than the one you're on
[20:34:43] <patricklewis_> how do I do so in mongo?
[20:36:20] <kali> find({ timestamp : { $gt : [mine] }}).sort({ timestamp:1 }).limit(1)
[20:36:38] <kali> or whatever will work with meteor
[20:50:03] <cozby> question... if an application is supplied one mongo url replica endpoint is that application aware of the other replica sets?
[20:50:20] <cozby> or does one need to supply ALL replica set endpoints for the application to be aware of them
[20:57:08] <kali> cozby: one is enough when everything is right. the driver will contact the one it know about and get the full list from there
[20:57:56] <kali> cozby: but if the one you're passing it does not work when the client starts, then you're in trouble. so it's better to provide the driver every replica you know about
[20:58:16] <cozby> cool thanks, make sense
[21:13:26] <sflint_> question for anyone : how does mongo choose which secondary to read from when using secondaryReadPreference?
[21:14:15] <sflint_> does it use the ping time?
[21:14:42] <sflint_> i know for "nearest" mongo will using ping time for the nearest node
[21:30:23] <lfamorim> kali, there was a possibility when I put rs.add({my-master-instance}), this replica remove all master information?
[21:51:18] <maxmanders> Hi all!
[21:51:33] <maxmanders> I know every mongodb configuration is different, but can anyone offer advice on useful values from mongostats to use as alerting thresholds?
[21:51:49] <maxmanders> I.e. if queues reads/writes are > X then potential problem; if > Y then significant problem etc?
[22:03:49] <joannac> maxmanders: what does your normal pattern look like?
[22:04:51] <maxmanders> joannac: I think unfortunately my answer will be 'give it a shot and find out' - I'm looking to put forward some useful indicative suggestions as part of a management offering. Would immediately go for MMS, but in this case it's not an option.
[22:05:11] <joannac> WHy not? Data securty?
[22:05:29] <maxmanders> I just hoped there might be some indicative useful figures / ratios to look out for - but I really don't have all the details that I'd like to offer for a decent answer (own worst enemy)
[22:05:49] <maxmanders> joannac: among other things - not a reservation on my part at all, but client governance requirements
[22:06:52] <joannac> well okay
[22:07:10] <joannac> you probably don't want queued reads or writes at all
[22:07:27] <joannac> because it means you're not keeping up with the load on the database
[22:08:06] <maxmanders> So as a starter-for-ten, nominally these values should be zero, and anything non-zero is worth investigating, and further tweaking as we understand more about the application and its behaviour?
[22:08:14] <joannac> but of course, you don't provision for your absolute peak load, so you will occasionally get spikes of queued readers and writers
[22:08:21] <joannac> yeah
[22:08:24] <maxmanders> joannac: makes sense
[22:08:57] <maxmanders> so as reads or writes increase, would increased queue length necessarily inform a decision to add more replica sets to spread the load?
[22:09:59] <joannac> as in, more shards?
[22:11:09] <maxmanders> Er yeah :-\ iiuc scaling sharded replica sets can offer better performance, but premature sharding can be counter productive over having entire data sets on multiple replicas?
[22:13:16] <joannac> yes, assuming your data is easily separated into multiple replica sets
[22:14:01] <maxmanders> So unless there's a need to share to spread writes across different replica set members, would I be right in saying it would be good to start with having the entire data set on multiple replica set members? *with your assumption above*?
[22:14:18] <maxmanders> Newer tech to me, still trying to get it right in my head :-)
[22:14:24] <maxmanders> s/share/shard
[22:17:25] <joannac> the first part of your sentence doesn't make sense
[22:17:48] <joannac> actually, none of your sentence makes sense
[22:18:17] <maxmanders> Hehe sorry, trying my best :-)
[22:18:21] <joannac> you can have: replica set A with all your user data, replica set B with all your invoicing data
[22:18:37] <joannac> (multiple replica sets with different data sets on each)
[22:18:49] <maxmanders> Ah okay!
[22:19:07] <joannac> or you can shard and have replica set A with half your invoicing and user data, and replica set B with the other half
[22:19:25] <maxmanders> Thank you!
[22:20:02] <joannac> no prob
[22:20:49] <maxmanders> I'd misread http://docs.mongodb.org/manual/core/replica-set-members/
[22:21:42] <maxmanders> And thought that one or more schema were present on a primary member, and were made highly available through other secondary replica set members.
[22:22:56] <maxmanders> Anyways - you've cleared things right up - and I've been awake too long - so thank you again, very much appreciated.
[22:32:34] <wc-> hi all, im having a lot of trouble creating an admin user on a local mongo server
[22:32:49] <wc-> i have auth=true in the config
[22:32:52] <wc-> as well as a replica set defined
[22:33:04] <wc-> when the server starts up (with an empty data dir)
[22:33:13] <wc-> i initiate the rs, it becomes primary
[22:33:21] <wc-> (also im connecting via localhost)
[22:33:28] <wc-> when i try to create the user i get permission denied errors
[22:34:10] <joannac> did you create the user admin?
[22:34:44] <wc-> oh looks like i might have just had some luck
[22:34:48] <wc-> i added a user with the userAdminAnyDatabase permission
[22:34:56] <wc-> instead of dbadminanydatabase or something like that
[22:35:11] <joannac> yes
[22:40:34] <wc-> im running this on ubuntu, installed mongo via apt, it seems like the init.d script isnt stopping mongo
[22:40:47] <wc-> it will start it, but i have to send a kills ignal to the process id to get it to stop
[22:41:01] <wc-> ubuntu 12.04 lts HVM aws image
[22:41:37] <wc-> never seen this before
[22:43:37] <wc-> ah i think i know, i might have the pid specified in a non-standard location in the mongod.conf
[22:43:42] <wc-> that would mess up the upstart script right?
[22:47:57] <ashutosh> hey
[22:48:15] <ashutosh> I am facing this error "exception: too many namespaces/collections"
[22:53:08] <ashutosh> @all can any one help with
[22:53:09] <ashutosh> I am facing this error "exception: too many namespaces/collections"
[22:53:20] <joannac> ashutosh: http://docs.mongodb.org/manual/reference/limits/#namespaces
[22:53:46] <ashutosh> @joannac i have increased namespaces to 512
[22:54:33] <ashutosh> we have around 200,000 Collections
[22:54:50] <joannac> ashutosh: i don't know what that means. what's 512?
[22:55:00] <ashutosh> 512 MB
[22:55:01] <joannac> and why do you have so many collections?
[22:55:13] <ashutosh> before it threw error it was 256
[22:55:37] <ashutosh> sharejs creates the collection
[22:55:44] <ashutosh> and we have lot of data
[22:56:27] <wc-> hmm im getting a ton of these entries in mongo’s log
[22:56:28] <wc-> ClientCursor::staticYield can't unlock b/c of recursive lock n
[22:56:29] <rasputnik> is there a downside to sticking each collection in its own db? trying to work around the db level locking
[22:56:35] <wc-> and the query is too big
[22:56:52] <rasputnik> wc-: damn yeah see that thousands and thousands of times a day
[22:57:00] <wc-> am i missing an index?
[22:57:24] <rasputnik> wc-: that can cause it
[22:57:31] <joannac> ashutosh: erm okay. well increase it again?
[22:57:33] <valonia> When i update an object, it changes _id and moves to the bottom, is there a way to fix that?
[22:57:45] <joannac> I'm dubious you need that many collections
[22:57:54] <joannac> valonia: that's a new object
[22:58:14] <valonia> joannac: but its there a way to keep the same _id?
[22:58:24] <joannac> valonia: update instead of making a new object
[22:59:53] <valonia> joannac: i am updating, so :/
[22:59:53] <ashutosh> @jonnac we already updated it from 128 -> 256 then again to 512
[23:00:00] <joannac> rasputnik: in my experience it won't help unless that's actually the bottleneck
[23:00:07] <joannac> valonia: what's the update?
[23:00:23] <joannac> ashutosh: like i said, i doubt you actually need that many collections
[23:00:33] <wc-> is there some permission i can add to a user that is just a global admin
[23:00:35] <wc-> can do anything
[23:00:37] <wc-> anywhere
[23:00:40] <rasputnik> joannac: we spend 180% of our time in lock contention according to MMS :)
[23:00:43] <ashutosh> @jonnac yes we do need it seriously
[23:00:57] <valonia> joannac: this is an example of the code
[23:01:00] <valonia> $db->update(array('Code' => $_SESSION['user']['Code'], 'Username' => array('$regex' => new MongoRegex("/^$names\b/i"))), array('$set' => array('Position' => $i)));
[23:01:01] <joannac> wc-: 2.4? 2.6?
[23:01:04] <wc-> 2.6
[23:02:19] <joannac> wc-: "root" is closest, but i think it also needs backup/restore
[23:03:25] <ashutosh> @jonnac is there any setting or limitation on number of collections?
[23:03:37] <joannac> ashutosh: yes, the page i linked you
[23:04:31] <joannac> valonia: test just the find part of that update
[23:05:31] <joannac> rasputnik: and it's not disk bound?
[23:06:34] <rasputnik> joannac: everything is disk bound - but a db level lock doesn't sound like a good thing, we're pretty write heavy
[23:06:48] <joannac> rasputnik: well, test it and see
[23:07:26] <rasputnik> joannac: yeah it's on the timetable for tomorrow. just wondering what the tradeoffs are
[23:07:55] <joannac> storage, authentication
[23:08:12] <ashutosh> @jonnac when trying to delete any collection it throws "exception: too many namespaces/collections"
[23:09:10] <ashutosh> is there anyway to force delete?
[23:12:22] <ashutosh> @anyone when trying to delete any collection it throws "exception: too many namespaces/collections"
[23:12:28] <joannac> ashutosh: ugh, you've hit SERVER-3570
[23:15:40] <ashutosh> @jonnac - anyway to recover from this?
[23:18:20] <ashutosh> @jonnac would doing mongodb repair help?
[23:20:36] <ashutosh> the log is full of "warning: hashtable namespace index long chain"
[23:24:03] <temple3188> Hi guys I was hoping someone could help me with a complex (to my knowledge) query with mongodb
[23:28:03] <temple3188> I have the following collections in my db: https://gist.github.com/christemple/5cd17d9277308e760ad1
[23:33:52] <temple3188> I have updated the gist with what I am trying to achieve and can not figure it out :(
[23:33:54] <temple3188> √
[23:33:57] <temple3188> https://gist.github.com/christemple/5cd17d9277308e760ad1
[23:37:58] <ashutosh> stopping a repairdatabase, does it have any impact ?