Log file Viewer
#mongodb logs for Wednesday the 3rd of December, 2014
[00:13:39] <hicker> Hi Everyone :-). When I use model.create({user: mongoose.Types.ObjectId(req.body.userId)}) to create a new document, it generates a new ObjectID instead of storing the object id passed in the request :-(
[00:17:58] <hicker> I'm passing an object id string. Yeah, I want to store a reference to the user doc
[00:18:59] <hicker> Storing just the object id string gives me a validation error, saying it needs to be cast to an ObjectID
[02:36:25] <harttho> Is there a way to detect if a program is connecting directly to a replica set/shard rather than going through mongos?
[02:46:37] <appledash> Since actually figuring out how to use mongo I've been finding all sorts of applications for it with stuff I've previously been using SQL or just a (very huge and slow) flatfile for
[03:07:04] <harttho> A follow up question, we're running into an issue where one of our mongo shard thinks that the chunk size is 1K, when it's in reality much larger. Anyone encounter anything similar to this or know where to investigate? I can share a log momentarily
[03:08:04] <harttho> Tue Dec 2 17:10:34.352 [conn3406773] warning: chunk is larger than 1024 bytes because of key { _id: { date: new Date(1406851200000) } }
[03:14:27] <harttho> We have other collections with the same ID scheme, (which we know is less than ideal), and we don't see that specific error
[03:31:34] <blizzow> I have 10Gigabit ethernet cards on my routers, and replica sets. I'm running a 7.1GB mongorestore from another host with a 10GbE card. The restore has 7130219438 records and it's taking upwards of an hour to do the restore. When I look at mongostat on my router, the says the netIn is peaking around 10m. Does anyone have an idea why my routers aren't taking data faster?
[03:36:37] <Boomtime> your situation sounds strange though, there must be a bottleneck somewhere - apparently it isn't the network or destination disk
[03:37:06] <Boomtime> check top on both ends, obviously check iotop too to ensure the disk isn't simply waiting the entire time
[03:37:30] <Boomtime> (because waiting will show up as no activity while actually being a bottleneck)
[03:49:57] <Boomtime> mongos can totally be a bottleneck - consider that any information sent to it must be re-transmitted for storage
[03:51:08] <Boomtime> finally, it must also do some processing on the incoming data to determine their destination
[03:56:19] <blizzow> My routers are 4 CPU cores, 4GB RAM, 2GB swap, and 10GbE cards. iftop -i eth0 shows a rate of about 80Mb/s. nethogs reports something similar.
[03:58:55] <Boomtime> i actually suspect you are CPU bound, despite having spare cores - certain operations are serial when performed on the same database, which unfortunately, importing/restoring does one DB at a time
[04:00:18] <Boomtime> you'll see a lot of discussion around mongoimport/mongorestore not being very fast - they "get the job done" but hardly in an efficient way, the DB can be used much better than either of those tools utilize
[04:01:38] <blizzow> ouch. Another router to run the mongorestore through? Or set one up on the host that's running the mongorestore?
[04:02:50] <Boomtime> this is a common implementation, put the mongos close to the app (on the same host)
[04:28:48] <CipherChen> hi all, i'm configuring how rs init sync since primary only keep capped oplog and the older oplogs have already been dropped.
[04:29:57] <blizzow> hrm, set up a router on my host where mongorestore is running from. in iotop it seems to hover around 2-4M/s Read rate.
[04:37:11] <Boomtime> blizzow: yeah, restore has to get confirmation back from the server, the smaller your individual documents the worse that ratio will be
[04:54:47] <CipherChen> we want to auto sync a primary from another primary, acted as: mongosync --from-source primary1 --to-source primary2
[04:56:41] <joannac> also, you want to put data on a new node, and have that new node be a primary for a different replica set?
[04:58:03] <arussel> I'm moving data throught mongoexport/import, when looking at the data through the mongo shell, what was before 1444259592914 became NumberLong("1444259592914"). Any reason for that ?
[04:58:45] <joannac> "Do not use mongoimport and mongoexport for full-scale production backups because they may not reliably capture data type information. "
[04:59:29] <CipherChen> since mongodb use oplog to keep refresh among its slaves, how do i replay those oplogs at other rs?
[05:00:18] <joannac> you just said you want to have a different replica set... so why should oplogs at replica set 1 be replayed at replica set 2?
[05:00:20] <arussel> joannac: I'm using it to extract production collections to my local machine to test some functions.
[05:01:19] <CipherChen> joannac: i want to make sure that the two rs data are exactly the same, so i want to do oplog replay myself
[05:03:02] <CipherChen> since secondary may fall down when secondary can't reach primary for a long time
[05:03:43] <joannac> if the secondary can't reach the primary, how will your solution help? how does it get the oplog then?
[05:05:33] <CipherChen> we hope to replay the oplog until the network recover, just dont want the secondary to turn DOWN/RECOVERING...
[05:08:25] <joannac> also, why do you anticipate long periods where your 2 servers can't see each other?
[05:08:38] <CipherChen> I mean the network exception, and long time later the network just recovered, and B has already been down.
[05:09:46] <Boomtime> if the network is down then data does not transfer, no matter what clever software you write
[05:11:13] <CipherChen> what we want is that: host B dont be down/recovering even when network is down
[05:13:01] <Boomtime> i think i understand the problem you are trying to fix, but your solution is incredibly complicated considering it has a simple fix
[05:19:13] <joannac> CipherChen: please use pastebin.com next time, so as not to paste multiple lines in chat
[05:22:57] <CipherChen> my plan is to build multi center and do the data consistency ourselves, is that possible?
[05:26:05] <Boomtime> so you need an oplog that is long enough to cover the time the two machines are not in communication
[05:29:32] <CipherChen> a data set contains the older data, when replay the oplog at the image, we can get the newest db.
[05:30:07] <CipherChen> that is how hdfs doing it, so i think since mongodb have oplog, it probably have image things
[05:32:05] <CipherChen> if not so, what happens when rs.add() a node, the primary dont keep the oplog all the time
[05:33:11] <joannac> and then reads oplog entries from when it started taking a copy, until the current time
[05:35:32] <joannac> to achieve what? if you secondary goes into DOWN or RECOVERING? yes, that's the correct action to take
[05:37:33] <joannac> CipherChen: we are going around in circles. I do not understand the use case you have and why it is not served by running a normal replica set across multiple servers or data centers
[05:38:56] <joannac> CipherChen: if you could explain apecifically why a replica set does not work for your environment, that would be good
[05:46:45] <CipherChen> joannac: thanks, i'll try the increase oplog size to solve the secondary DOWN problem first
[06:08:12] <joannac> do the classes, play with mongodb, try and break your replica set / sharded cluster
[10:00:17] <zzz> Hi, is there any possibility to set up mongodb to use id' s only from the [0-9] range, and not from the [0-9a-f] ?
[10:01:53] <kali> zzz: http://docs.mongodb.org/manual/tutorial/create-an-auto-incrementing-field/ for instance
[10:02:20] <zzz> kali: i meant from the db side, so if i insert a document, the generated id should contain only 0-9 characters automatically in my case.
[10:59:38] <varadharajan> Hi, Is it possible to restrict particular user to log in from a particular IP when using a mongodb instance configured with Challenge/Response as the auth mechanism?
[14:30:20] <samgranger> Hey guys - lets say I have a collection of users, containing first and last names
[14:31:37] <samgranger> but I can't get it fully functional, it only finds first names or last names, but if I type in John Doe, it doesn't work - any ideas?
[14:41:38] <GothAlice> samgranger: If you have both a first, and a last name… couldn't you just explicitly query for them explicitly? Why the or?
[14:42:02] <GothAlice> samgranger: User.find(firstName="Bob", lastName="Dole") — will find you only records whose names are "Bob Dole" in the literal sense.
[14:43:01] <GothAlice> (Also, if you're using regexen or $where in MongoDB, you're probably coming at a problem sideways. Both have important negative impacts on your queries.)
[14:43:53] <GothAlice> samgranger: Bad logic is worse than no logic and simply presenting separate first and last name fields to end-users. ('Cause splitting will fail.)
[14:44:47] <GothAlice> If that's how you want to query it (always together, not sorted on last name, etc.) then yes.
[14:45:53] <samgranger> Let's say i'd like to order on last name - what's best in that case? Having a full name, first name and last name field? Seems a little dirty?
[14:46:39] <GothAlice> Are you utterly stuck with presenting users a unified full name field of some kind?
[14:48:52] <GothAlice> If you need to sort on last name, then you need last name by itself. That means it would make sense to have the first name by itself. (The split is usually "family name" and "surname" — since one can have multiple of either as in my multiple family name example.) This would then dictate presenting split first/last name fields to users of your system for the purposes of data entry (to match the data structure) and querying.
[14:49:13] <samgranger> Well, I just want users to find other users - just by typing in their name, preferably still having the names split in mongodb, just stuck on how best to query ie. Alice Bevan McGregor if Alice Bevan is in first name and McGregor is in the last name
[15:23:27] <markotitel> I cannot find config option with which I can log mongodb queries to a file. Is that actually possibile, or I need to read it from a db?
[15:31:00] <winem_> markotitel: also not very familiar with mongodb but db.setProfilingLevel(2) should do what you want
[15:32:28] <markotitel> winem_, yes I have set that, I am thinking possibly at the moment I dont have slow queries because of empty base. Although I set slowms is set to 1ms
[16:28:03] <pulse00> hi all. i've run into a "no file left on device" error on a development mongo instance. is there a way i can just wipe all data manually and "reset" the db? db.dropDatabase() doesn't work anymore because "Can't take a write lock while out of disk space"
[19:10:49] <ss_> so if i have a field called tags which is an array. how would i find list of articles that contain that tag
[19:14:53] <ehershey> You only have to use $in if you want to specify an array of values to filter by
[19:15:00] <ss_> ok so i need to get the most recent articles that has that tag.. my current query is db.collection.find({tags: "tagname"}).sort({ 'timestamp': -1 })
[19:26:57] <cheeser> hahuang65: not afaik. you can dump a single collection or all of them. but nothing in between
[19:51:53] <hahuang65> why would something report as "already sharded" when I try to shard it, yet it's not show in sh.status()?
[20:02:19] <hahuang65> it keeps saying some of the collections are already sharding when I drop a database and recreate it
[20:10:16] <lfamorim__> Hello! How cold I remove only N number of records with remove? Example, 2 records of 3, or 10 of 20.
[20:40:27] <harttho> "errmsg": "exception: collection's metadata is undergoing changes. Please try again."
[20:49:22] <decompiled> strange issue, on a cluster's configuration server I can authenticate against the mongod running as the config server, but can't auth against the mongos running on the same host
[21:44:36] <GothAlice> harttho: If no-one answered your question, certain operations (like building an index in the background) will prevent other administrative operations from happening on the collection until they finish.
[21:52:48] <decompiled> joannac, they did at some point, but another admin screwed things up. I have a backup of the admin database, but only tried to restore it to 1 of the config servers. Should I restore it to all three?
[22:01:34] <decompiled> Good call. I've made backups of files I've changed, but will just do a general backup before anything else and try to restore the admin database on all 3
[22:38:12] <varesa> My mongod service has stopped starting, any ideas? http://i.imgur.com/lTVM1nQ.png http://i.imgur.com/1exoMjv.png
[22:38:52] <varesa> sorry for screenshots instead of paste/whatever, I was running those commands in a VM console so no easy text copying
[22:45:04] <varesa> I also tried setting SELinux to permissive, so it shouldn't be a wrong context/etc.
[22:46:38] <joannac> also it looks like this http://stackoverflow.com/questions/18591664/mongod-2-4-not-running-fedora-19-freeze
[22:57:46] <varesa> I just need to remember to remove that overriding file when it gets fixed in the repos and possibly gets some other changes :)