Log file Viewer
#mongodb logs for Thursday the 21st of June, 2012
[00:24:24] <matthewvermaak> ah, nvm, thats effectively being treated as an or condition across the embedded documents, which matches, and then the positional is set to the last matching criterion
[01:37:54] <kenny> any tips on why db.auth("","") will fail right away, but db.auth(user,password) and show collections hangs forever?
[04:13:12] <dstorrs> I have a document that looks like this: { _id : 'foo', pages : [ {}, {}, ...] } What is the query I need to find the length of the 'pages' array?
[04:15:05] <kobold> dstorrs: you can't; fetch the document and count it, or store the length in a field and use that instead
[08:10:55] <carsten_> rumors said: you should have database background building database applications
[08:28:17] <NodeX> http://www.infoworld.com/d/open-source-software/red-hat-releases-nosql-database-enterprise-java-196020 <---- interesteing, I wonder what it's like
[08:30:49] <mkopras> hey guys, I got question about particular query http://pastie.org/private/vx6yybunxkxmzoleau1vw can someone look ?
[08:36:26] <mids> mkopras: db.bbb.find({ steps : { $elemMatch : { name: "aaa", date : 1340020000 } } })
[08:39:40] <mids> NodeX: not sure how dot notation would work in this example; but you can also do db.bbb.find({ steps : {name: "aaa", date : 1340020000 } })
[08:42:57] <mkopras> thx guys, I was not aware how to search over collections properly, these example will make it more easier now :)
[08:43:14] <remonvv> Sincerely doubt there's a difference. An $elemMatch unpacks to the equivalent of the dot notation route.
[08:45:53] <remonvv> mids is right. Pretty sure I've made this mistake about 10 times in here by now.
[08:45:54] <NodeX> if there are 2 I mean ... your way only returns one of two possible matches which would not allow for a sort
[08:46:44] <NodeX> but the query is not correct because 2 documents match that criteria so two should be returned
[08:46:55] <mids> my understanding is for the query to return the document with both attributes matching in the same subdocument
[08:47:41] <remonvv> No, there's a semantic difference between two fields having to match for a single array element and either field having to match for any but not necessarily the same element in the array.
[08:48:12] <remonvv> With all that said though, who would read the dot notation version and not assume the $elemMatch case?
[08:48:54] <remonvv> Right, but that's where I have a problem. I think it's a bit of a query syntax design booboo
[08:50:13] <NodeX> either way ... with both documents having the same values for the feilds then either works
[08:50:57] <remonvv> Ah, no I was just wrong. Ironically enough I probably suggested the exact opposite at some point as well.
[08:52:41] <NodeX> http://news.cnet.com/8301-1009_3-57457470-83/code-crackers-break-923-bit-encryption-record/
[08:55:10] <NodeX> it beats the old record hands down though which means encryption needs to get better else Moores law will nullify SSL very very soon
[08:55:25] <remonvv> You have to wonder what comes first. Household quantum computers that make the research useless or conventional computers that start cracking this stuff in any sort of practical speed.
[08:56:36] <remonvv> Well, effort to bump up encryption bits to something higher is endlessly easier than speeding up brute force decryption. I somehow doubt this'll ever really get relevant in day to day life.
[09:04:00] <JamesHarrison> Hi - relatively new to MongoDB, working on a site which makes fairly heavy use of $nin to filter out content - a query like {$orderby: {created_at: -1}, $query: {tag_ids: {$nin: ['some-tag']}, hidden_from_users: {$in: [None, False]}}} appears to completely not use an index set up for created_at-1, tag_ids 1, hidden_from_users 1. What am I doing wrong?
[09:05:33] <JamesHarrison> Tried to, but it's not making the query performance any better, which makes me think I may be missing something more fundamental with my query structure.
[09:12:01] <JamesHarrison> Not seeing it in currentOp and it appears to be built; I didn't do it as a background thing to start with
[09:14:44] <JamesHarrison> That query type is currently showing at about 150ms avg, some (with many nin tags) up to 1.5 seconds, which is a bit high for an 8-core dual xeon box with 16 gigs of ram, SSDs and only 13,000 objects (38MB) total to look through
[09:15:41] <NodeX> I didn't know $nin doesn't use indexes .. is this intentional or going to be fixed?
[09:16:50] <remonvv> It can use an index but a) it almost always results in very little performance gain and b) only hits it for relatively simple indexes.
[09:17:47] <JamesHarrison> http://pastie.org/private/sdrapb9k0helolvav7omga are the current sets of indexes on that
[09:18:03] <remonvv> Unless almost all of your records actually match the $nin you wont notice much of a difference
[09:19:10] <remonvv> Seperate index for each fields usually means a lack of understanding about indexes ;)
[09:19:35] <JamesHarrison> and most of those indexes work great, tbh, heavy read, little write on this collection I should mention
[09:19:50] <JamesHarrison> we're doing a very large number of different queries against this collection, yeah.
[09:20:29] <remonvv> Anyway, which index does it say it hit for your problem query, if any? pastie explain()
[09:20:54] <NodeX> JamesHarrison : seocndly ... 1. The sort column must be the last column used in the index. <-- from the docs
[09:22:08] <remonvv> Well, since it can only use 1 index for the entire query (sort included) it just might
[09:29:52] <JamesHarrison> okay, set it up in the shell, more readable: http://pastie.org/private/kuyslzwv2ae0utetg1ltvg
[09:30:07] <JamesHarrison> that's after adding an index "tag_ids_1_hidden_from_users_1_created_at_-1"
[09:35:15] <JamesHarrison> Okay, so nin is inefficient in terms of index performance when >50% of the data doesn't match any of the $nin query
[09:37:45] <JamesHarrison> well, it doesn't pick up on it, and if I hint to explicitly specify that query, http://pastie.org/private/u82xqwmedtpax4r85nt9na
[09:38:48] <JamesHarrison> I'm still confused as to why it's failing -this hard- with a limit() in there
[09:40:05] <JamesHarrison> This should surely at least be: sorting by created_by, then for the resultant datasets, getting at least 100 with first query filter, paring down with second filter (both with indexes) and doing that again till it hits 100
[09:41:09] <remonvv> Well if it would do that (it doesn't) it would have to sort the entire dataset before doing anything else.
[09:42:28] <JamesHarrison> and the latter is also hitting a small percentage (2-5% maybe, varies per user)
[09:43:40] <remonvv> first index field doesn't help much at all and thus the candidate set is huge, the $nin that follows is slow (regardless of index) because it has to scan a huge candidate set. Once that has all happened MongoDB has to sort the full set in memory (scanAndOrder: true) and then return the first 100.
[09:45:17] <JamesHarrison> Yeah. Okay, so for the problem we have here (filtering images tagged with a set of tags (there are 10,000+ tags and growing), filtering hidden images, and sorting) what's the right approach in MongoDB terms?
[09:45:43] <JamesHarrison> Doing a quick query to find the image IDs we want to exclude, then excluding them and sorting?
[09:46:01] <JamesHarrison> (though I guess $in is going to suck pretty hard performance-wise too...)
[09:47:04] <remonvv> Well, $in is faster technically and conceptually is usually used to select a small subset from a larger set.
[09:48:36] <remonvv> if it's null it doesn't exist and will fail an equality test, so simply {hidden_from_users:{$ne: true}} will do
[10:00:59] <NodeX> [10:46:29] * ifesdjeen (~ifesdjeen@109.144.18.169) Quit (Remote host closed the connection)
[10:10:24] <remonvv> Inform me about how I fixed all your problems and changed the way you view life.
[11:22:48] <JamesHarrison> remonvv: sorry, real life intervened - good stuff, I'll have a play, thanks to you and NodeX
[11:52:57] <ahmeij> Hi! I'm new here, and have a question that is probably an education issue, but I cant find an answer. I have a mongo replica set, 3 equal rackspace cloudservers, with a significant load, where 2 servers run with a load of 20 (the master and 1 slave) and the 2nd slave is ideling at load 0. However mongostats seems to indicate the second slave is processing just as much queries as the first slave
[11:54:16] <ahmeij> I dont think this is typical behaviour, but maybe it points to a common configuration mistake?
[11:57:13] <carsten_> driver issue where the driver can not balance the load properly across the slaves and master?
[11:58:27] <ahmeij> all connections come from the default mongo driver for ruby, I have tried to investigate that a bit, however mongostats indicates all queries are split over the 2 slaves and all updates go to the master (as expected)
[12:10:34] <remonvv> ahmeij, are all members up to date? one of the possible explanation is that your idle member is behind which results in faster queries (less data to query on). That's a bit far fetched though.
[12:25:45] <mklappstuhl> I want users to be able to setup own "content structures" where they enter a name and add a set of properties... then they should be able to save this "form" and "instantiate" it later and it should be saved with actual data
[12:26:19] <mklappstuhl> It's kind of similar to what you can do in some CMS where you can define custom content elements
[12:39:16] <npx> So what's up with the V8 port? Is it going to be possible for Node.js to share a libv8 instance with MongoDB?
[12:49:09] <npx> it would remove the need for a client library and just let me write Mongo queries directly
[12:49:40] <npx> which I can sorta do, it's probably a bad idea for them to share a context for a dozen reasons, it was just the first thing that came to my mind
[12:50:18] <remonvv> That, and I don't think your premise is correct. You'll need a library of some sort to convert your queries to the MongoDB line protocol.
[13:02:41] <lazymanc> hi, I've just installed mongodb from the 10-gen repo on ubuntu 12.04 (VM) and it's now sat there idling at 15-20% cpu - is that normal?
[13:15:15] <lazymanc> remonvv: was wondering if it was swapping to virtual disk because of limited memory (completely new to this so just guessing)
[13:15:43] <lazymanc> Gargoyle: I mean mongodb wasn't doing anything, it was just started but nothing is accessing it as yet
[13:16:49] <Gargoyle> lazymanc: I have a 3 node replica set running on virtual hardware idling at 4% (When I say idling, the could be 1 or 2 users + mms agent)
[13:20:47] <Gargoyle> What would be "a lot" of connections for a mongod server replicating to 2 more nodes, and running 2 web apps (in testing, so only a handful of users / apache threads)
[13:25:19] <Gargoyle> Another one just to leave hanging round the channel - Has anyone got an ubuntu 12.04 init.d script for the mms-agent ?
[13:37:08] <remonvv> Most drivers follow a rather odd pattern of establish new connections until they run out of allowed connections from a single host.
[13:38:05] <remonvv> Gargoyle, it's not a worrying amount of connections but you can control the maximum amount of connections per client if you feel it should use less.
[13:45:12] <Gargoyle> Well, if it's not a worrying amount, then I'll let it be - at this moment in time, I really have no idea what a sensible value would be!
[13:48:20] <wereHamster> what is the difference between { $pull : { field : {field2: value} } } and { $pull : { .field.field2': value } } ?
[13:57:44] <BlenderHead_625> Hi guys. I'm trying to $pull all elements from an array where the first element of the arrays (in the array) I want to $pull have a given value. I tried {'$pull': {'bookmarks.0': ...}} but that removed the first element of the array. I want to removed the entire array from its parent array if its first element matches the criteria. Any pointers?
[14:01:55] <BlenderHead_625> What I want is {'$pull': {'bookmarks': {0: 'match'}}} but it won't let me do that.
[14:08:41] <BlenderHead_625> there must be a way to match array elements like it's possible to with {}
[15:22:45] <JoeyJoeJo> I'm setting up a new MongoDB and as of yet I don't have much data, but I plan on having more and more as time goes on. Can I set up one shard initially and add more as needed?
[15:27:38] <JoeyJoeJo> Is it a good idea to run my app server, mongos and configdb all on one server?
[15:30:44] <remonvv> Well, it's definitely a good idea to have your mongos local to your app server in almost all cases.
[15:33:43] <JoeyJoeJo> Ok, so one last question. I'm planning on getting 4 Hi-mem large EC2 instances. One will be my app server, mongos and configdb. The other three will be my first shard. Does that pass the sanity check?
[15:35:57] <JoeyJoeJo> I don't have an exact answer to that yet, but I'd say much more reading than writing. Data will come in large bursts every few days or so, but I assume clients will be looking at the data at any time
[15:36:36] <remonvv> Ah right. So assuming those 3 machines form a replica set and your queries are all slaveOk=true then that should be fine.
[15:41:35] <remonvv> It's easier for you to read up on it than for me to explain it in a few sentences ;)
[15:42:10] <remonvv> Sharding is basically a horizontal scaling strategy whereas replica sets are mostly a durability/availablity feature. With that said a replica set does scale up reads.
[16:08:29] <rguillebert> I would like to unittest my python webapp that uses mongodb, do you know if there's a mock version of mongodb (or an in-memory version of mongodb) ?
[16:09:19] <remonvv> I'm not aware of any python mongodb mock tools and MongoDB cannot run in memory only mode atm
[16:11:14] <rguillebert> erf, that would be a nice thing to have, for sql you can use sqlite (especially if you're using an ORM)
[16:29:35] <drockna> im trying to start a mongod in a openvz read only container. What folders do i need to make writable for mongod to work?
[16:33:19] <pulse00> hi all. i'm trying to dig into document oriented databases, coming from a rdbms background. when the domain model is a relation between book and author (many-to-many), how would this be stored in a document oriented db? have a document "author", which then contains an array of books?
[16:51:55] <kali> pulse00: an author collection, a book collection, most likely. then you can deal with the relation in either side or both side.
[16:52:33] <kali> pulse00: the important thing with doc-oriented DB is not the static description of your model, but rather the way you will need to read it and access it
[16:53:41] <pulse00> kali: yeah, that's what i was asking myself. when i want to show a paginated list of all books sorted by release date, having the book as a "inner" document of the author looks odd from a rdbms perspective.
[16:59:28] <kali> if you embeds author ids in book, you can also use an index on the author array. that way you don't have to denormalize and maintain the relation both ways
[19:06:06] <MrJones98> hello - i have an issue understanding some semantics but may be specific to the pymongo driver
[19:47:33] <Zelest> I thought of having a collection with embedded docs (one to many) where the content basically is "one domain/site and multiple pages" .. however, I wish to be able to select a single "page" within a document as well as sorting them to request the "oldest" page within that document.. is this possible?
[19:48:35] <Zelest> like { _id: 'domain.tld', pages: [ { uri: '/', date: ... }, { uri: '/about', date: ... }, { uri: '/contact', date: ... } ]
[19:50:44] <dstorrs> Zelest: it's probably possble, but I'm not clear on the goal. what are you trying to achieve?
[19:51:52] <Zelest> I'm writing a crawler and I plan to use mongodb to store the data on domains and pages within these domains.. and I wish to keep that in the same document to avoid excessive lookups.
[19:53:01] <Zelest> not sure on the most optimal solution for this.. and I use find_and_modify to fetch the pages to crawl (in order to make the crawlers concurrent)
[19:53:16] <dstorrs> ok. I would recommend profiling with embedded vs separate collections before assuming this will be optimal, but it sounds plausible at least.
[19:54:35] <Zelest> if I have 10 threads fetching "the oldest urls" .. i want to get the 10 olders, right? :P
[19:54:39] <dstorrs> better solution is to throw out an optimistic update() where you set some attr on the item (e.g. "owner : my_id") then do a find()
[19:56:06] <dstorrs> I know because I'm currently building a web crawler that uses that strategy. :>
[19:57:41] <Zelest> but yeah, ruby is lovely.. sadly I'm stuck with php at work so ruby is a nice change. :-)
[19:58:00] <dstorrs> I *strongly* recommend profiling before you go too far down the embedded or referenced path.
[19:59:10] <dstorrs> FYI, if you find it write-only, I'd be willing to be you haven't looked at it in ~5 years.
[20:01:08] <therealkoopa> If I have a list of embedded documents, is there a way to run a query to find all of the embedded documents up to a certain object id? For instance, [1, 2, 3, 4, 5]. Grab me everything up to objectid 3903 which is the 4th
[20:01:35] <dstorrs> First would be a "vertical" stack of pages, like so: { site: 'foo.com', page : 'http://...', owner : 'none', lock_expires: 0 }
[20:03:14] <dstorrs> then, when you want to query it, you do: db.coll.update({ site : 'foo.com', owner : 'none' }, { $set : { owner : my_id(), lock_expires : time() + 10 } )
[20:04:28] <Zelest> that's almost how I use it now.. but with code (http code) set to 0 if "in progress" ..
[20:04:54] <dstorrs> the second way is to have generally the same structure, but "stripe" them horizontally, like so { site : 'foo.com', pages : [ { owner : 'none' ...} ]}
[20:06:06] <dstorrs> In profiling, I found that up to ~50 pages, I was getting better performance with embedded docs than a straight collection, but above that it flipped.
[20:07:33] <dstorrs> also, watch out for the "16M per doc" limit if your max number of pages is very high, or if the page objects are large
[20:08:41] <dstorrs> sure, but if that metadata document is large-ish -- say, 1k -- and you have 1,000 pages per site, that means the entire site document is ~1M.
[20:09:18] <dstorrs> You can't retrieve just one page from the array, you need to pull the entire pages array over the wire. => slow
[20:10:15] <dstorrs> even if 99.9% of your sites will have only a dozen pages, you need to account for the one Googlebomber with 50,000 pages on his site. That blows the doc limit.
[20:12:50] <Zelest> i use it on my mini-crawler on httpdb.org .. and it can easily reach up to 40-50 sites a sec
[20:12:56] <dstorrs> oh and, before you think of it -- don't even think about using one collection per site. http://www.mongodb.org/display/DOCS/Using+a+Large+Number+of+Collections
[20:13:46] <dstorrs> I was getting about 100 jobs a second with FaM, vs something like 5000 with update(). but don't take my word for it -- do your own profiling.
[20:14:51] <dstorrs> it's possible it was an error on my part -- poor / missing index, inefficient query, etc.
[20:21:25] <Zelest> regarding indexes.. if I want to find a document where site: is "foo.com" and the path: is "/robots.txt" .. i do want to use an index to find /robots.txt, but I don't want to create a full index on the path:, only the ones which is /robots.txt.. is that doable?
[20:48:15] <kenny> I'm looking to create trigrams on an object. Would it make more sense to store them directly on each object, or in a separate collection with a list of matching objects? (there'll be a lot of overlap)
[20:53:09] <kenny> (the problem I'm trying to solve is fast regex searching across around 25G of data)
[21:06:26] <therealkoopa> Any idea why a console.log(change) gives me an object that has a property called name that's set, but if I do change.name it's undefined?
[22:14:34] <jonwage> I'm trying to figure out what causes that in our test suite. I watch db.serverStatus() and we're not using all the connections
[22:15:50] <jonwage> possible that I issue some background operation or something that then blocks later tests?
[22:18:21] <ahmeij> Hi, can anyone please help by explaining where i need to start looking to solve the following?: query sr_production.profiles ntoreturn:1 nscanned:144381 scanAndOrder:1 nreturned:1 reslen:1340 6861ms
[22:19:48] <ahmeij> I've been looking all over but I cant seem to find one responsible query (although it should be somewhere I guess :) )
[22:39:54] <dstorrs> ahmeij: note that, as jY implied, count() is always slow. That's an infrastructure problem -- Mongo uses uncounted BTrees for its indices, so a count is always a table scan
[22:41:27] <ahmeij> However all reads go to slaves which wont allow me to check the db.system.profile collection
[22:42:27] <ahmeij> dstorrs: I think I found my culprit, a relative new query via mongoid, however I cant determine the exact query format yet, as to build the best index
[22:55:52] <kenny> dstorrs: I'm not sure if full text search is what I'm trying to do. Googling now. I'm trying to do fast full regex searching on at least one text field of around 500 bytes.
[22:56:54] <dstorrs> I don't know if it fits your use case, but you may want to at least consider using something like Solr that's purpose-built for text searching
[22:57:39] <dstorrs> you can probably make it work with Mongo, but it may end up being cheaper and easier with something that's dedicated to the job
[22:58:41] <dstorrs> no worries. There's other things like it too. That's just the one that comes to mind first.
[23:06:36] <ahmeij> jY / dstorrs: I found this query: { "ts" : ISODate("2012-06-21T23:00:16.906Z"), "op" : "query", "ns" : "sr_production.profiles", "query" : { "$query" : { "identities.service" : "service2", "identities.uid" : "24h24kh8" }, "$orderby" : { "_id" : 1 } }, "ntoreturn" : 1, "nscanned" : 145262, "scanAndOrder" : true, "nreturned" : 1, "responseLength" : 6037, "millis" : 8465, "client" : "xxx", "user" : "" }
[23:06:55] <retworda> So Im changing _id in one of my collections. Have been running this, but its taking forever. Have I made a mistake? The docs Im inserting arent being picked up by the find are they? db.coll.find().forEach(function(d) { // remove d, edit it, reinsert it })
[23:07:39] <ahmeij> But it does have an index: "v" : 1, "key" : { "identities.service" : 1, "identities.uid" : 1 },
[23:12:48] <ahmeij> 200k records, query with an index, resulting in either 1 or 0 records takes 8+ seconds :( if anyone has a clue (see above) please help
[23:13:58] <ahmeij> I'm so new ro mongo performance investigations that i keep forgetting the basic stuff, thnx
[23:23:37] <retworda> Pls ... http://pastie.org/4129175 Is there any chance the function is interfering with the set of documents being iterated over? (the inserts making the set endless?)
[23:24:03] <ahmeij> briljant, the application server load just spiked becasue it can finally move at a decent speed :)
[23:24:07] <dstorrs> ahmeij: This may or may not be useful to you. It was to me: http://kylebanker.com/blog/2010/09/21/the-joy-of-mongodb-indexes/
[23:28:34] <Zelest> dstorrs, if I run an update.. and a find afterwards.. is it ever possible that the update won't apply until after the find has been done?
[23:31:47] <ahmeij> Zelest: In my experience, certainly with slaves but even without, if you have async writes (default?) the read of a just written record can return the previous version, we swithced to savemode for all writes becasue of that
[23:33:23] <dstorrs> ahmeij: no worries. We all are. I dthink there are only two or three people in the channel with really significant experience.
[23:35:32] <ahmeij> does that mean that if you get a high enough experience level through use you will automatically be hired by 10gen ? :P
[23:36:28] <dstorrs> I don't know of a non-legacy tech company that ISN"T desperate for good people right now
[23:37:10] <ahmeij> True, however strange I find it that a company like for example github would need 50+ developers
[23:37:56] <dstorrs> A successful company always has more projects they would like to do than hands to do them.
[23:38:21] <dstorrs> One of the biggest things that made google successful was that they got started right after the last tech boom. There was a huge glut of available talent in the Valley,
[23:38:50] <ahmeij> I think to be succesful in the long run it is important to keep focussing on the core business, --> Google is a nice exception though
[23:38:54] <dstorrs> That way, they weren't available to the competition and Google could expand fast
[23:40:02] <dstorrs> If you're a business with a future, your "core business" generally has a lot of room in it. I run a startup, and I could easily find work for 50 good devs / sysadmins / designers / etc
[23:41:31] <ahmeij> I run one too, and while we could have a few more, I'd think 5 would be sufficient :) however once we have 5 our goals would probably expand faster etc
[23:42:27] <ahmeij> Same issue, we need a bit more cash to grow the software a bit faster to be more actractive to VC's, and that completes the chicken and egg problem
[23:43:31] <ahmeij> anyway, since it is close to 2am. and thanks to you guys my production issue is resolved, it is time to get some sleep
[23:44:10] <ahmeij> I'll be back, I hope to learn way way more since we are fully running on mongo now :)
[23:51:03] <mmlac> Hello everyone. I am struggling with loading my custom Mongoid::MyMongoidModule with rails standard autoloading, so I was wondering what is the best practice to extend Mongoid? Right now I create lib/mongoid/mymongoidmodel.rb
[23:51:58] <mmlac> and in that, standard module Mongoid module MyMongoidModule end end. Normal autoload fails (Expected to define MyMongoidModule at load_missing_constant)