Log file Viewer
#mongodb logs for Wednesday the 29th of August, 2012
[00:24:12] <daniel____> hi, every body. i'm newbie for mongo. and i wanna learn something here with you. thanks
[02:57:05] <crudson> macBigRig: use dot notation to index nested attributes, but note that you will get entire documents back, not just the bit of the array that matches, unless you do some extra work (depends what you really want to do). The second question is simple enough with .aggregate()
[02:59:38] <macBigRig> curdson, looking for syntax examples for indexing. As for selecting, what kind of extra work is needed to get that output. Again any example is greatly appreciated.
[03:12:28] <crudson> macBigRig: this will give you an array of items that fits your second Q: db.c.aggregate({$unwind:'$PS'}, {$project:{CR:1,'PS.ETB':1,'PS.DRB.HIC':1,'PS.PTB':1}}, {$unwind:'$PS.PTB'})
[03:46:59] <hadees> how terrible is it to use a string as the _id ? I'm creating a versioning system for my collection and so I want the history table to have the _id: <objectid>:<version>
[04:02:20] <crudson> hadees: Not terrible at all. "If your document has a natural primary key that is immutable we recommend you use that in _id instead of the automatically generated ids." source: http://www.mongodb.org/display/DOCS/Object+IDs
[04:16:48] <tyl0r> Is it worth adding an arbiter to a 3 node replica set? Does that mean if 2 servers go down (for whatever unlikely reason), the 3rd will be elected to be the Master for writes?
[07:40:52] <jsmonkey123> If I have a collection called Foo and that collection stores multiply documents that has a property called boo that has an array with strings. If I want to remove one of the strings. Is the proper method to use update? or is it remove and specify what property/entity to remove?
[08:54:12] <makka> I'm having trouble debugging a mapreduce - is this the right syntax for launching a mapreduce from the shell with a subset of documents in the collection (those with foo == bar) : "db.coll.mapReduce(map, reduce, {out:"test"}, {query: { "foo" : "bar" })"
[09:07:20] <rantav> I just posted my question to the forum https://groups.google.com/forum/?fromgroups=#!topic/mongodb-user/NClTihy_PPk
[09:13:55] <rantav> insert query update delete getmore command flushes mapped vsize res faults locked % idx miss % qr|qw ar|aw netIn netOut conn time
[09:14:54] <rantav> The overall query time was 87000 millis, so the mongostat doesn't cover the entire query lifetime
[09:19:58] <Gargoyle> Eg. Wed Aug 29 11:17:17 [conn35034] warning: virtual size (82655MB) - mapped size (78108MB) is large (4265MB). could indicate a memory leak
[09:20:10] <rantav> b/w I have another query, in which the index does have ranges last, and it's still awefully slow
[09:23:20] <NodeX> you changed the index on 1.8 million documents already ? .... I would imagine the index is still building
[09:28:37] <rantav> and in this pastebin you can see that the ranges are last on the index (It's a bit different query)
[09:34:44] <rantav> NodeX I swapped end/start dates in the query, no good, still slow :( took 75 secs this time
[09:44:46] <kali> rantav: don't worry about the weird value for the isodate, i've seen that before, it's how the shell show the "max value" for a date
[09:46:45] <rantav> But is there a way to sum up all indexes sizes from all collections? Not just index size of a single collection?
[09:52:32] <rantav> although this time it might make sense, b/c it's not fully indexed (there's no index for just serviceId+start)
[09:53:15] <kali> your index should be fine for queries with a service value + a range (or a half-range) on start
[09:53:56] <rantav> kali, here's the explain for a query with only one half-range http://pastebin.com/9zZ3Vb0x
[09:56:41] <kali> rantav: you have a two dimensional range here. btree indexes are good at single dimension queries
[09:58:35] <rantav> it wasn't like 70 seconds like in the case of two half-ranges, but it was something like 20 seconds
[09:59:49] <kali> rantav: if you run it again, it's the same ? we're not wasting time on an atypical run ?
[10:01:36] <rantav> maybe ebs is slow, but again, if the index is in ram, that shouldn't make a diff, right?
[10:13:06] <rantav> kali, I can say one thing for sure - when using two full ranges, it's faster than using two half ranges
[10:14:34] <rantav> I think 15-20 sec for a query that's indexed is still unacceptable, just for the record...
[10:22:03] <rantav> nodex, yeah, I will, but still 15 or 20 seconds for a query is lousy… I can't stop there...
[10:26:42] <NodeX> I have collections in excess of 140M and run range queries that take < 1s, not sure why yours would differ
[10:27:32] <rantav> could it be b/c the compound index has elements? Or in other cases (other indexes) 4 or 5?
[10:28:09] <rantav> nodex, kali mentioned that indexes with more than 2 elements are hard on btrees… I wouldn't know...
[10:31:06] <NodeX> maybe it's more efficient to query the range to a tmp collection then query that?
[10:33:42] <rantav> it's an analytical app that keeps data about what users are doing along the session
[10:35:03] <NodeX> my approach to realtime (or near as) is every minute process X documents from a history collection (this stores every interaction with my app)
[10:35:40] <NodeX> I group by session id / user id and process say 5k docs at a time which keeps my queries and processing time nice and low
[10:37:03] <rantav> I mean, if the data is already there and reading is slow, does it matter if it was inserted in batches or not?
[10:37:25] <NodeX> I read 5k docs at a time for example and process it in my app, I dont range query anything
[10:41:13] <NodeX> rantav : my method simply grabs the data, if I wanted to query for a range I query on the already processed data
[10:42:05] <NodeX> for example I have a report that's got 1 days worth of reads/writes or w/e inside it and a date on it, this is one document and inside it lay nested objects with grouped/reduced actions from a user
[10:42:35] <NodeX> no, the session collection is your live data, you grab 5k docs from it and process them into a smaller collection, then the next minute you grab the next 5k
[10:43:34] <NodeX> In one of my apps I need users as a whole so I save the net result of what all users did ... i/e 50 people searched for Blah.... yours needs individual users
[10:44:00] <rantav> what I do is, a backgroud job processes the events from user activity, each event is a single user action
[10:44:04] <NodeX> it doesn't depend on any POV it's the correct model. You NEED a live collection where the data is dumped
[10:44:23] <rantav> and that backgroud process calculates session data from the stream of user activity
[10:44:31] <NodeX> it's analoguous to message queues, they grab soem data from a job server, process them and continue
[10:45:01] <NodeX> it's the most efficient way of doing it for mine and many apps that require analytics
[10:45:25] <rantav> ok, but the things is that I don't know in advance which start/end parameters I'm going to get
[10:46:27] <NodeX> you can model the data in such a way that after processing it's still queryable ... only in Smaller chunks
[10:49:59] <rantav> anyway, for the record, I found two ways that I can improve my query, even without changing the data model
[10:50:17] <rantav> one, is to use a full-range for the start parameter (the end param doesn't matter)
[10:50:48] <rantav> another is to limit(). I didn't realize this could be such a huge factor, but it is...
[10:53:28] <rantav> so you're thinking something like groups of session object in resolution of hours, or something along those lines?
[10:55:23] <NodeX> you might want to have a per user doc with a d/m/y stamp and store that 24h in embedded
[12:14:35] <vak> Are there any standard commands to re-arrange documents in a collection according some "master" index? my collection doesn't fit in RAM, so, I'd like to minimize seek operations at least for uch a master index.
[12:25:49] <vak> NodeX: why to delete others? I need them too... I just wuold like to phisically store data according to one index as much as possible. Kind of defragmentation if you like
[12:29:10] <kali> vak: are you aware that depending on what you're doing, chances are the documents are stored in insertion order ?
[12:31:00] <vak> NodeX: my collection is OLTP-like. No deletes. So, no "reclaim space" issues. I'm talking only about oder
[13:06:05] <Bartzy> but I have a problem - the 2nd query will be much more effective with uid_pid, since Monog will only have to look in the index in the uid, then all the PIDs that that user liked are under the same btree "branch"
[13:06:25] <Bartzy> And... If I am - is it worth to have both "pid" index and "uid_pid" index, just for that ?
[13:11:26] <[AD]Turbo> when logging with syslog, what is the facility that mongod uses? some of the local1/2/3/4/5/6/7 ?
[13:17:19] <Bartzy> NodeX: I read that several time, as well as reading the whole MongoDB in Action book
[13:20:38] <NodeX> "If you have a compound index on multiple fields, you can use it to query on the beginning subset of fields. So if you have an index on "
[13:21:16] <Bartzy> <Bartzy> but I have a problem - the 2nd query will be much more effective with uid_pid, since Monog will only have to look in the index in the uid, then all the PIDs that that user liked are under the same btree "branch"
[13:29:56] <NodeX> I am just about to re-release one of my apps and was wondering whether to upgrade before or not
[13:39:49] <ranman> hmm, I just worked with Akshay to push those last night but I'm looking at it now and they're not out
[13:44:24] <ranman> kk, so I'll get the packages updated later today we found a small error in our packager script
[13:46:06] <NodeX> which is funny because in ENglish being "apt" at somehting means being good/correct
[13:54:59] <ansolas> would i just have to store the filename with teh user infos and the picture itself in a separate grid?
[13:58:04] <NodeX> so the file gets stored and the relevant info (perhaps UID / name of user or w/e) inside the meta
[13:58:33] <ansolas> ok so i have to use 2 collections , one for the user info and one for the files ?
[14:00:31] <ansolas> something else, I found teh following in the skip() doc: Range based paging provides better use of indexes but does not allow you to easily jump to a specific page.
[14:01:32] <NodeX> I dont have problems skipping upto a few thousand but I save a cache of the total results first time
[14:03:53] <NodeX> skipping say from 7-14 probably wont be bad becuase they will probably be seqential on disk
[14:07:25] <NodeX> for example Mongo is very fast but I cache the last say ten blog posts in Redis to save a call to mongo
[14:30:22] <NodeX> Our model is made possible due to our resiliant infrastructure and scalability thanks to products like MongoDB from our friends at 10gen "
[14:49:58] <ramsey> Derick: I was just having some bit of frustration over DBRef not appearing to work in find() anymore, like my notes from 3 months ago suggest. I thought I recalled seeing examples in the docs, but I can no longer find them.
[14:50:47] <ramsey> Derick: I'm not claiming that you or anyone removed them. I just can't find them. :-)
[14:51:24] <emocakes> Our model is made possible due to our resiliant infrastructure and scalability thanks to products like MongoDB from our friends at 10gen "
[14:51:54] <kali> Our model is made possible due to our resiliant infrastructure and scalability thanks to products like MongoDB from our friends at 10gen "
[14:54:50] <ramsey> Derick: my notes showed something like this: db.assets.find({ catalog : DBRef("clients", ObjectId("4fafca158f4852ebb394f756")) });
[14:54:57] <ramsey> Derick: but that kept giving me this error: error: { "$err" : "can't have undefined in a query expression", "code" : 13629 }
[14:55:01] <NodeX> considering original plus rewrite are mongo based I thought it'd be nice to thank 10gen
[14:55:22] <ramsey> Derick: I worked around it, though, with this: db.assets.find({ "catalog.$id" : ObjectId("4fafca158f4852ebb394f756") });
[14:57:31] <ramsey> Derick: this is what I'm referring to: http://docs.mongodb.org/manual/applications/database-references/#dbrefs
[14:57:57] <NodeX> emocakes : since I decided to never use an RDBMS again I have not encountered one problem that I cannot get around
[14:58:06] <Derick> the db ref contains a field called $id, but it still points to the normal _id field
[14:58:33] <NodeX> I have built many many apps for many different purposes and not ran into a single problem that could not be overcome in a day or two
[15:00:16] <NodeX> I seldom joined things, I mostly embedded serialized data from day 1 knowing it was faster to query than to join
[15:01:15] <Derick> ramsey: I think you're confusing a reference stored compared to how to query them
[15:01:21] <ramsey> Derick: it points to a document in the clients collection that has an embedded document called "catalog" with an _id property
[15:01:53] <Derick> I don't think you can just query with: { catalog : DBRef("clients", ObjectId("4fafca158f4852ebb394f756")) }
[15:02:19] <ramsey> Derick: so, what is the proper way to query for documents that contain a particular DBRef?
[15:03:35] <ramsey> In the way we're using them, I don't see the point, either, because we started using them and then chose not to dereference them
[15:31:36] <eka> sorry on the http://downloads-distro.mongodb.org/repo/ubuntu-upstart/dists/dist/10gen/binary-amd64/
[15:35:57] <eka> spaam: yes... old news... can't upgrade cause the package is not there for debian/ubuntu :(
[15:45:04] <vak> info about 2 days delay is just my date comparison from these two links: http://downloads-distro.mongodb.org/repo/ubuntu-upstart/dists/dist/10gen/binary-amd64/ and http://dl.mongodb.org/dl/linux/x86_64 Do you see any error in conclusion?
[15:51:20] <bjori> vak: if you don't provide an _id the driver will generate it for you, and if the driver is broken, the server will
[15:54:52] <vsmatck> If the user doesn't provide _id it's better to let the server generate it rather than the driver IMO. Less stuff to send over the network.
[15:55:38] <algernon> the advantage of the driver generating it is that if you later want to use the _id for other stuff, you'll already have it at hand, without an extra network round trip
[15:56:01] <ramsey> Derick: here's something weird that I just noticed... it appears that the MongoDBRef class in PHP is creating structures in the DB that look like this: { "$ns" : "collection", "$id" : "1234..." } rather than what the docs say, which is this: { "$ref" : "collection", "$id" : "1234..." }
[16:02:57] <eka> ramsey: are you using anything that wraps the driver? like an ODM or something like?
[16:13:14] <ramsey> Derick: I see what's going on now. I'm using MongoHQ, and it appears that they are displaying through the interface $ns in place of $ref (in a DBRef object), but when I use the shell to access the DB directly, I see $ref for the same exact object, so it's not actually stored in the DB as $ns
[16:16:18] <ramsey> Derick: you can see what I mean here: http://www.evernote.com/shard/s4/sh/4ca017cb-249b-4646-bc4b-7983db6184c9/539198e667a53cc286601eabdb3b17de
[16:17:25] <NodeX> Get:1 http://downloads-distro.mongodb.org/repo/debian-sysvinit/ dist/10gen mongodb-10gen amd64 2.2.0 [52.0 MB]
[16:22:44] <Derick> * Starting database mongodb start-stop-daemon: unrecognized option '--interleave=all'
[16:27:05] <mog_> I'm looking for a way to both provide a modifier in an upsert query and use a new object if the object isn't found. Is that doable?
[16:30:15] <tyl0r> Do I need to have an arbiter to get the last remaining node (in a 3 node replica set) to become the Master?
[16:53:34] <eka> ranman: using ubuntu here... did an agt-get update then an upgrade for mongo and it's working fine
[17:03:53] <eka> ribo: I alway check from version to version... if 2.1 says normal and then 2.2 says normal... so nothing special to de
[17:10:23] <mog_> is there a way to obtain this behaviour: db.col.update({x: 1}, {$inc: {a: 1}}, {x: 1, a: 1, b: 0, c: 0}) where the third argument object would be inserted in the upsert request if it wasn't found in the first place?
[17:27:10] <sssilver> Hello everyone. I have a BinData field in my collection. Is it possible to use a BinData value as a condition when selecting from it? Thanks
[17:30:33] <IAD> sssilver: try to create a new md5 field for binary data in documents. i'm not sure about using binary data as filter value
[18:54:24] <coopsh> Hi, I noticed that all members of a replica set are in state SECONDARY after I ran a stepdown on the current primary
[19:01:40] <Honeyman> Hello. A weird question: there is no conditional sorting of the output in any form yet, isn't there? I was trying to perform something like db.messages.find({}, {}).sort({user: {'$eq': 'JohnDoe'}, time: 1}) but seems no hope for it...
[19:02:27] <Honeyman> That is, in this case, kind of - 'give me all messages ordered by time, but, if there are messages by JohnDoe, they'll go first'.
[19:08:34] <sssilver> I'm trying to understand why for _id it returns a nested structure like this: { "$oid" : "<id>" }, and how can I make it flat, getting rid of $oid
[19:10:47] <Honeyman> sssilver: haven't the result been already preprocessed by something like json_util ?
[19:11:56] <sssilver> Honeyman: I'm doing a return json.dumps(classes, default=json_util.default) , yes
[19:12:40] <sssilver> I wouldn't just go and write an extra argument, working with a library for the first time
[19:13:28] <Honeyman> Funny, I guessed the problem, without even knowing about the existence of json_util two minutes later
[19:14:48] <Honeyman> Reformulating my question about conditional sorting. Well, since in fact I need only the ONE result, then... $or does NOT guarantee the order of execution, right? Isn't there anything that guarantees?
[20:03:13] <vak> just wanted to say thanks to 10gen. After upgrading one part of my calculation takes 30 min instead of 2 hours 30 min. Well, 5x speed up is a nice thing :)
[20:07:47] <bjori> coopsh: a rs.stepDown() should force elections.. but if your overall network is slow it could take few seconds..
[20:32:02] <coopsh> bjori: network is not slow. all instances run on my desktop. secondaries are in sync. no writes happen while I tested
[21:43:05] <Gargoyle> Is it possible to "tag" an query in code to get it to put something in output.log ?
[21:54:13] <leandroa> hi.. I've been using a .js script on 2.0.4, and it worked fine.. now (since 2.0.7, and even 2.2) I noticed that when I do ObjectId('….').toString(), it returns the entire string, instead of the 24 digits id as string.. is there a better way to do that?
[22:06:44] <thomasthomas> HI ALL! I am trying to .find({transactions:undefined}).count() and failing any explanation?
[22:12:41] <thomasthomas> leandroa & Gargoyle: thanks but false is returning 0 and true is returning 25327 (all), I've seen transaction equal to undefined when I'm querying.
[22:15:57] <thomasthomas> db.orders.find({transactions: {$type: 10}}).count(); came back with 0 also =[
[22:44:05] <thomasthomas> I have a root level field that is equal to an array sometimes and null others. How can I select all orders when this field "transactions" is null?
[22:49:05] <thomasthomas> I finally figured it out! .find({"transactions.0":{$exists:false}}).count();
[22:49:45] <thomasthomas> db.orders.find({"transactions[0]":{$exists:true}}).count(); comes much more natural ...