Log file Viewer
#mongodb logs for Thursday the 28th of June, 2012
[00:19:45] <dstorrs> libbyh: check ping on machine, check firewalls, verify you can ssh in, verify mongod running on target machine
[00:20:40] <libbyh> i think my problem is a firewall issue. ping OK, mongo via SSH ok, some ports OK. no response on 27017 or 28017 even though they're in my iptables
[00:21:25] <libbyh> and show listening in netstat. i could connect when i was in the same building as the server but not from home. (not sure how much overlap btw IP of server and IP when I was able to connect.)
[00:34:43] <libbyh> thanks, dstorrs and jY. think my mongo config is ok and will go work on firewall.
[01:31:08] <dnnsmanace> hello, i am trying to set up a db structure where users get sent a message when they are inactive
[01:31:20] <dnnsmanace> i am trying to think of the best way to do this without parsing every user constantly
[01:32:16] <dnnsmanace> if they haven't uploaded anything in 30min, i want them to get a notification
[01:33:15] <dnnsmanace> i guess i am trying to think of the logic behind this in my app, doesnt have to be db
[01:33:16] <dstorrs> every time they take an action, canccel any existing idle_timer, do the action, then set a 30min idle_timer.
[01:33:32] <dstorrs> if an idle_timer pops, have the client send a notification to itself (update a view, whatever)
[01:35:55] <rossdm> if they are uploading via email then how can they have any sort of session that requires activity < 30 mins?
[01:36:41] <dnnsmanace> it requires a kind of continuous communication, so i want to track the time to the last email they sent
[01:38:00] <dstorrs> have a collection "last_seen" structure is: { _id : username, time : epoch / ISODate / whatever }
[01:40:30] <dstorrs> once a minute, do a 'find' for users in time >= 30_mins_ago. mail everyone you find. use a capped collection so you don't need to worry about storage
[01:46:12] <dstorrs> hm. actually, one correction. you need to also not keep pinging them after they get active again.
[01:52:57] <dstorrs> if you use a capped collection you may run into an issue with update, not sure. I know you can't remove from a CC
[01:57:07] <dstorrs> try creating a collection, adding an index, then look in log to see what's there.
[03:03:29] <lizzin> why doesn't `show dbs` from within the mongo client show collections created by my liftweb app using net.liftweb.record?
[03:07:16] <lizzin> instead several files have been created in dbpath/. such as collection.0, collection.1, and collection.ns
[03:16:36] <Ahlee> lizzin: appears your collection name is collection, those files are the preallocated files for the collection namespace
[03:17:09] <Ahlee> lizzin: You should be able to run db.collection.find().foreach(printjson) to see what's being inserted
[03:18:41] <Ahlee> Does this mean I didn't get this field indexed, or it didn't get inserted? from my log file: Wed Jun 27 23:04:40 [rsSync] ibp.logging Btree::insert: key too large to index, skipping ibp.logging.$message_1_background_ 1362 { : "SurVo - survey():
[03:22:07] <lizzin> Ahlee: you're right, i changed the name of the collection to 'collection. the real name is 'phonebook'
[03:22:15] <lizzin> Wed Jun 27 22:18:47 TypeError: db.phonebook.find().foreach is not a function (shell):1
[03:23:54] <Ahlee> lizzin: so what's the question - you're looking for why the collection name is created under the db?
[03:25:12] <lizzin> Ahlee: i was hoping to add documents to the collection from within my app and then connect to the db via the mongo client to verify things are going as expected
[03:26:07] <Ahlee> lizzin: ok, so there are dbs, and collections in the db, you've issued use <db_name>, db.<collecton>.count()
[03:40:16] <Ahlee> yeah, that's what happened on my system when I just did it, I wound up with a database foobar, a collection in foobar database named foobar, with a single document consisting of an ObjectID, and name: "ummyeah"
[03:46:44] <AAA_awright> What's the exact rule on . in property names? I have several documents with Object-hashtables of {"url": {information...}} and that doesn't seem to be a problem, except recently
[03:48:06] <dnnsmanace> whats the best way to compare ISO-8601 date and a date like this Thu, 28 Jun 2012 03:42:06 GMT
[03:50:06] <dstorrs> dnnsmanace: convert them to a single format. compare. enjoy a milkbone in a commie-free world phase
[03:50:10] <AAA_awright> dnnsmanace: Use a date-time library, most of them should be able to convert arbritrary strings to dates... Javascript has the Date object, look at the documentation for that for the speciics
[03:51:14] <dstorrs> dnnsmanace: as a thought -- I personally have just been forcing everything to UTC before insert and storing it as epoch values.
[03:51:59] <Ahlee> dnnsmanace: I'm having issues with date comparisons. Namely, issues with javascript's ISODate and Date
[03:59:19] <AAA_awright> { "_id" : ObjectId("4ebcb47a8bfb992231000001"), "menu" : { "http://magnode.org/Menu_MainMenu" : { "title" : "Main Page", "weight" : 0 } } }
[04:00:00] <AAA_awright> Somehow, MongoDB happily stored my key names in the database without any domain restrictions whatsoever
[04:06:45] <AAA_awright> Should it be possible? idk, I just want to know what's the behavior that inserted that key into the database
[04:07:55] <AAA_awright> What's the exact rule on . in property names? I have several documents with Object-hashtables of {"url": {information...}} and that doesn't seem to be a problem, except recently
[04:13:38] <AAA_awright> Guest1231231: If you're going to be rude don't bring it into private messages. As an author of a program, I need to know the behavior of MongoDB so I can code accordingly for my program. I'm asking for that behavior, since appearently it's not documented. Got it?
[04:15:13] <AAA_awright> Ahlee: As far as I can tell, no. It's the actual key name. Unless MongoDB collapses sub-objects into parents like {a:{b:1}} -> {"a.b":1}, but I wouldn't expect that
[04:15:58] <Ahlee> AAA_awright: Sadly, I can't speak definitively on that. I believe that "a.b" is a short cut for {a:{b:1}}
[04:18:18] <AAA_awright> Ahlee: There's nothing I can do to make it spit out {"a.b.c":1}, it looks like it's only used in queries
[04:22:09] <deoxxa> AAA_awright: {"a.b.c": 1} means exactly what it looks like - it would translate to pseudocode of `if (a.b.c === 1) { ... }'
[04:24:21] <AAA_awright> I've never had a problem inserting documents with properties containing "." until about 3 days ago
[04:26:38] <Max-P> Is it possible to update an associative array with a key that doesn't exist in an existing document? I need to {$set: {'folder.my_user_id': 'INBOX'}} but it seems to fail when the field is not already set =/ Thanks
[04:27:02] <Guest1231231> i tend to ignore questions from people that can not read other persons answers if they are perfectly correct
[04:27:54] <AAA_awright> I've been using Javascript, specifically, mongolian for Node.js which uses (used?) the mongodb driver in turn
[04:30:08] <AAA_awright> Guest1231231: You asked "do you have a point?" before any other person even acknowledged my question. Again, rephrased, do you have something useful to contribute?
[04:30:30] <deoxxa> looks like it's never been valid, but there's been inconsistent driver support for stopping you from doing it
[04:31:07] <deoxxa> it doesn't *technically* get rejected at the database, but only to allow maximum flexibility for drivers implementing weird crap, by the looks of things
[04:33:49] <AAA_awright> I'll have to take a closer look at the driver to see then, now I'd guess that queries aren't sent as-is
[04:38:24] <AAA_awright> Having full range keys would be nice... Storing URIs is a major use-case I would think
[04:39:55] <deoxxa> well, it's a bad enough idea that drivers actively try to stop you from doing it
[04:42:54] <AAA_awright> There's no reason not to attach this information to a document, it makes sense to have an Object of { URI : {...} }
[04:44:13] <AAA_awright> I can think of other use cases, storing link-level data about all the URLs you reference on a page. You want to know where they are, what version of the page they're linking to, etc.
[05:31:50] <ferrouswheel> and it brings everything to a crashing halt. seems like a retarded design that background=False is default.
[05:35:38] <ferrouswheel> issue here: https://jira.mongodb.org/browse/SERVER-3067 seems like a big issue, i'm not the only one it's brought down a site for.
[08:28:47] <LambdaDusk> Hi, I have seen that mongoose uses "Buffer" as a type for fields, I guess it's a BSON byte buffer... what is the use of that data type?
[08:32:55] <kali> assumptions are made on string by mongo and the driver. they have to be utf8 encoded for instance
[08:33:25] <kali> so if you try to store an arbitrary byte array in mongo as a string, you'll get an error
[09:02:58] <Gargoyle> If I have shutdown 2 nodes from a 3 node RS. Can I force the remaining note to be Primary?
[09:10:08] <Gargoyle> got one of them back online, and the one I want to keep was promoted back to PRIMARY (I did originally shut down the secondaries).
[09:14:53] <Gargoyle> http://www.mongodb.org/display/DOCS/Reconfiguring+a+replica+set+when+members+are+down
[10:29:07] <c2c> Hello, I have a question I can't find in doc : How can I find() documents that doesn't have a field 'blah' in a collection
[10:52:44] <neil__g> @NodeX I have a P-S-S replica set and I do a query on one of the servers. $res->count() returns the correct result count, but if I foreach() over $res it never enters the loop. Seems to be related to the secondaries being out of sync by an hour or so.
[11:42:29] <kali> ron: no. i just mean i can see some parts of the data migrations where tools could help, but it's not really a mongodb issue imho
[11:48:36] <pilgo> kali: I just created a date created attribute for my collections and am wondering how I would migrate the existing documents to have a date. Any tips?
[11:49:47] <kali> pilgo: if the collection is small, just iterate over it, get done with it and forget about it
[11:49:50] <NodeX> pilgo : do.foo.update({date: {$exists:false}},{$set : {date:"Somedate"}},false,true);
[11:50:07] <kali> pilgo: if the collection is big, fix the data in the collection every time you read it
[11:52:38] <pilgo> Am I guaranteed to get the same order of documents if I sort on a date field and they all have the same date?
[11:56:39] <balboah> is there a way to check consistency of a database that is synced in a replicaset? For example to see that some data that should be there isn't?
[11:58:14] <ron> pilgo: sorting should be done on specific fields. how they are stored in the database shouldn't matter (that much).
[11:59:32] <balboah> kali: only because I've moved data files around and forgot some of it before I started it again
[12:06:32] <ojii> hi, does anyone know a workaround the bug in pymongo that prevents it from talking to remote (authenticated) databases?
[12:46:33] <ojii> does anyone here have a solution for http://stackoverflow.com/questions/9191136/pymongo-fails-to-work-with-multithreading?
[14:44:09] <cemalb> Hi. I have a replica set on EC2 w/ a primary, secondary and arbiter. We restarted all 3 individually and now get the error "not master or secondary, can't read" when trying to query the primary or secondary. rs.status() shows "loading local.system.replset config (LOADINGCONFIG)". What should be my next step in getting these back online?
[14:45:12] <cemalb> Also rs.initiate() gives "local.oplog.rs is not empty on the initiating member. cannot initiate."
[14:47:39] <kchodorow_> ~50 lines from around the "loading local.system.replset config (LOADINGCONFIG)"
[14:56:46] <sharp15> how detailed is mongodb's permissions/access tracking? for instance. can i give a group of people access to a database and mongo will note who created (and when) which documents?
[14:58:09] <kali> cemalb: everything clear on the hostname front ? because "replSet error self not present in the repl set configuration" sounds like the replica can not find itself in the set
[14:59:21] <cemalb> Yeah, I was wondering about that. The hostname is mw-rs1-data1 which is in there though
[15:02:48] <kali> cemalb: can you check also that "host" in db.serverStatus() is consistent with what you expect ?
[15:08:08] <kchodorow_> cemalb: can you start a mongo shell on mw-rs1-data1 by running: "mongo mw-rs1-data1:27017/test"
[15:09:36] <augustl> need a field in my db, "api token", some kind of UUID would do. Should I just generate my own, or does mongo have something built-in that ensures uniqueness etc?
[15:18:44] <cemalb> kchodorow_, kali: Welp, it was in fact a problem with the host name. It was mapped to the wrong IP. Everything's working now. Thanks for the help!
[15:54:46] <JoeyJoeJo> I've got documents that include lat/lon. How can I find all documents in a collection that are some distance from a given lat/lon?
[16:00:00] <Ahlee> Does this mean I didn't get this field indexed, or it didn't get inserted? from my log file: Wed Jun 27 23:04:40 [rsSync] ibp.logging Btree::insert: key too large to index, skipping ibp.logging.$message_1_background_ 1362 { : "SurVo - survey(): <snip>
[16:00:38] <drudge\work> JoeyJoeJo: check out http://www.mongodb.org/display/DOCS/Geospatial+Indexing
[16:01:43] <JoeyJoeJo> drudge\work: I was just reading that actually. I have my lat and lon stored as db.collection.lat and db.collection.lon. Do I need to store them in one field?
[16:03:31] <JoeyJoeJo> In that case I need to wipe all the data in a collection so I can re-import it properly. How can I do that?
[16:23:54] <JoeyJoeJo> drudge\work: Thanks for your help. I can now run the geoNear command and return correct results
[16:57:15] <souza> I have the following bson in C language (http://pastebin.com/ZzXtdPSM) , it must return one result from mongodb, but it doesn't returns anything, and i've no idea how to fix it, thanks!
[17:03:41] <multiHYP> {"date":"20120628", "items":[{…},{…},{…},{…},{…},{…}]}; I want to find out the length of items array.
[17:05:22] <multiHYP> I was reading that before coming in here and asking that question. sorry, i don't know how to ask it more clearly.
[17:06:17] <multiHYP> no, on the example above I would do: db.collec.find({"date":"20120628"}).pretty();
[17:07:10] <diminoten> is there a way to query for the count of documents in a collection which have arrays containing a certain value?
[17:12:02] <multiHYP> migration is a pain, I don't know if its worth combining 3 collections into 1 that would fit with my current new model. analytics is always relevant though.
[17:43:20] <halcyon918> hey folks… I've got a question about rep sets… if I set the WriteConcern to REPLICAS_SAFE, and one replica gets the update, but the others do not… does Mongo rollback the save on the first replica so it's as if it never was saved to the first replica?
[17:51:06] <kali> stefania: when a replica is primary and get splitted, it will rollback the last few write he got before becoming aware he was splitted
[17:51:49] <kali> halcyon918: as far as i know, this is the only case that can lead to rollback: resycing a split replica set
[18:01:11] <dstorrs> hey all. I just connected to my develop data and all data is gone. Yet, db.coll.totalSize() has not changed.
[18:02:03] <dstorrs> we're trying to see if the vanishing data was something we did or a Mongo issue. what could cause it aside from developer error?
[18:03:25] <dstorrs> also, am I correct that when a remove() is done, the disk-state of the the data is marked 'deleted' (much like 'rm filename'), but it does not actually change the totalSize? for that you would need to run repairDatabase to compact, yes?
[18:09:35] <souza> Hello guys, i'm having problem to compare dates with mongodb and C, this is my output http://pastebin.com/59yuTimT It was returning all services in my database, but must to return only the services that have date less than "1340906127000" (date in milliseconds), and the last record, have a date (in milliseconds) 1340906187000, and didn't must be returned, someone know how can i fix it?
[18:14:43] <fomatin> I'm trying to use mongorestore to restore a database from S3, but I'm getting a seg fault error
[18:16:57] <Ahlee> Does this mean I didn't get this field indexed, or it didn't get inserted? from my log file: Wed Jun 27 23:04:40 [rsSync] ibp.logging Btree::insert: key too large to index, skipping ibp.logging.$message_1_background_ 1362 { : "SurVo - survey(): <snip>
[18:18:32] <souza> dstorrs: i'm about two hours in this problems, it's too weird, because it doesn't retrieve that last record
[18:19:21] <fomatin> this is what i'm getting after running mongorestore http://pastebin.com/hDZQNCSA
[18:19:46] <dstorrs> souza: if I'm reading this correclty, I think the query you're building is '{vm : { $lt : last_date}}'
[18:20:58] <souza> dstorrs: i just want to get all documents that have the "last_date" $lt the value informed.
[18:21:06] <dstorrs> like I said, your code is useless to me understanding what you need. I really need to see the data.
[18:21:25] <dstorrs> post two documents from your collection and I can probably give you the query you need.
[18:21:55] <dstorrs> but if you want what you said, then I think you just need to change "vm" to "last_date" is your query.
[18:22:55] <souza> dstorrs: that the "vm" object is a subobject from other service called "service", and this service has a array of "vm's"
[18:27:20] <halcyon918> kali: sorry for the late response about the replica set stuff… but does that mean that the primary will keep the write even though the other replicas didn't get it? I'm confused about the implications of a REPLICA_SAFE partial save
[18:29:15] <dstorrs> souza: ok, so for each 'Services' object the 'vm' key has only one entry under it, yes?
[18:30:40] <souza> i have an array of vms, but i want to get all services with only the vms that have a date $lt the date informed
[18:30:49] <dstorrs> if it's always exactly one, this will work: db.Service.find({ 'vm.last_date' : { $lt : THE_DATE_YOU_CARE_ABOUT } })
[18:32:24] <souza> dstorrs: Ok, that i'm doing this in C language, but my query it's something like this, but it returns a date $gt yet
[18:32:52] <dstorrs> if it gives you grief about 'last_date' sometimes being there and sometimes not, you can do this: db.Service.find({ $and : [ $exists : { 'vm.last_date' : true }, 'vm.last_date' : { $lt : THE_DATE } ] })
[18:34:53] <souza> dstorrs: my problem not is in exists or not, my problem is that my query it's retrieving a greater than that value, must i want only the less than
[18:37:42] <souza> dstorrs: like in this paste http://pastebin.com/59yuTimT you can se in the first line the time in milliseconds, and the last record have a value greater that one.
[18:39:47] <dstorrs> souza: please check that your query actually matches what I posted. Because what I saw in your code did not.
[18:40:59] <dstorrs> you were doing { 'vm' : { $lt : DATE }} when you needed {'vm.last_date' : { $lt : DATE}}
[18:45:59] <souza> dstorrs: i changed my code to this http://paste.org/51095, but it doesn't works yet, it returns the vm with date higher
[18:47:19] <dstorrs> souza: stop working in C. Connect to the Mongo shell. Get a query that works and returns what you want. Then translate it to C.
[18:47:53] <dstorrs> I am not familiar with the C driver, but I am willing to bet money that you have still not accurately translated what I told you to use.
[18:48:54] <dstorrs> becaue as far as I can see, your query is now this: { vm : { $lt : 'vm.last_date' : Date }} which is not even syntactically correct
[18:52:46] <souza> db.Service.find('vm.last_date' : { $lt : 1340908730000 }] }); This? it was returning error. :(
[18:54:14] <dstorrs> souza: I'm trying to be nice about this, but I'm on three hours sleep and I'm starting to feel like I'm talking to a wall, because I keep posting the exact query and you keep not using it.
[18:55:04] <dstorrs> One last time. Do this: db.Service.find({ 'vm.last_date' : { $lt : 1340908730000 } })
[18:55:20] <dstorrs> do it in the shell. not in C. it if returns the right thing, then translate to C.
[18:58:46] <dstorrs> thanks for helping Derick. maybe that was the concept I was not getting across to him.
[19:01:19] <dstorrs> Derick: I don't know if that would have been better or not. it seems that he had never really RTFM'd and had no mental model of Mongo queries. I suspect he was copy pastaing
[19:04:51] <souza_> dstorrs: thanks for help, i don't get it yet, but i think that i have a way now!
[19:06:19] <dstorrs> souza_: if you haven't done so, I strongly recommend sitting down and reading absolutely everything under http://docs.mongodb.org/manual/#for-developers
[19:06:44] <dstorrs> it will take several hours. It will save you several days over the next few weeks.
[19:18:07] <souza_> i have to create a Date object and pass in the query > db.Service.find({ "vm.last_date" : { $lt : date } });
[19:35:02] <FerchoDB> Hi, We're doing some tests with TTL collections. We are trying the example on blog.mongodb.org but it doesn't remove the document at 30 seconds, it lasts much more before removing it
[19:44:35] <dstorrs> FerchoDB: if you don't mind me asking, why are you using a dev version? does it have something specific you need?
[19:47:20] <FerchoDB> dstorrs, it's ok, we are just making some documents and investigation about new features in MongoDB
[19:59:57] <FerchoDB> MongoDB 2.1.2 lasts 44 seconds to remove 30 second TTL documents. Is 15 seconds an accepted bias?
[20:00:52] <php10> so i'm trying to build an aggregated stats collection. takes every combination of the hit/sale metadata such as user, site, lander, geoip country, geoip region, http referrer, etc. and then has a bunch of counters raw hit, index hit, unique hit, leads, etc. i need to query any of these fields and then be able to break down by any of these fields. is storing all of these combinations the best
[20:00:53] <php10> way to do this? it seems that after even a day of traffic collection the collection becomes massive
[20:01:33] <spikie> I use doctrine mongodb odm, and when I try to make a map reduce with inline option, mongo doesn't return me an'results' as expected, but a 'result' <collection_name>.... Some one could help me
[20:02:06] <php10> but it seems that in order to keep any of the fields associated in order to breakdown the data, the only way i can aggregate are on unique combinations
[20:15:46] <FerchoDB> now MongoDB 2.1.2 lasts about 90 seconds to remove 30-second TTL documents. ¿? is this because it's still unstable or because I'm missing something?
[20:18:17] <charnel> hi when I try to connect to mongolab with the uri I am getting unable to authenticate user mongodb which is the protocol in the url. Anyone knows how to set the url
[20:26:20] <dnnsmanace> hi i am getting a dup key error even though the key should be the unique id generated on creation of document
[20:26:21] <dnnsmanace> MongoError: E11000 duplicate key error index: dblblnd.convos.$id_1 dup key: { : null }
[21:18:11] <Ahlee> I had no problems with it 'catching up' when I copied over to a node that fell too far behind
[21:32:14] <tystr> I'm setting up our replica set on aws and want to make sure I have all the bases covered with regards to backups
[21:33:35] <tystr> not sure how that would perform with millions of documents in a collection, though
[22:12:13] <tomlikestorock> Is it better to store fewer fat rows with a lot of data in a row field, or is it better to split that field up into individual rows and store many smaller rows in a collection?
[22:26:16] <kenneth> php> =d('9b000000027575696400290000003637383533336230333162353535316261666432653966396433303564393762303030303030303000026d616369640029000000623766363932616661656564376639303639646563623236653137333236663839343436653462350002757569642d616476002100000036353133336262643863363234616438623535643536333235393838333837300')
[22:26:28] <kenneth> =d('9b000000027575696400290000003637383533336230333162353535316261666432653966396433303564393762303030303030303000026d616369640029000000623766363932616661656564376639303639646563623236653137333236663839343436653462350002757569642d61647600210000003635313333626264386336323461643862353564353633323539383833383730000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
[22:26:29] <kenneth> 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000')
[22:29:28] <algernon> I assume it is so, because the bson has a length specifier in the beginning, and the php driver ignores any junk past that length
[23:06:43] <php10> question for php mongo users. trying to increment a field in an embedded document. driver is building the $inc as follows: ['$inc']['sums']['hits'] 1 but i'm receving "Modifier $inc allowed for numbers only" ... note: i'm doing this on upsert. it works with ['$inc']['hits'] 1
[23:08:04] <kenneth> algernon: well, i'm more curious why the C lib is generating a bunch of extra junk and making my payload significantly bigger