Log file Viewer
#mongodb logs for Tuesday the 5th of June, 2012
[00:07:38] <radicality> Hi. Does anyone here have any experience/tutorials using amazon's elastic map/reduce for mongodb ? I can't find even the simplest example :S
[00:22:25] <acidjazz> [Tue Jun 05 00:20:48 2012] [notice] child pid 23477 exit signal Segmentation fault (11)
[00:46:55] <acidjazz> Tue Jun 5 00:45:24 [conn2] bravo.fs.chunks Assertion failure false db/key.cpp 409
[01:31:32] <sebastiandeutsch> Hi I'm running a mongodb with the following stats: mapped 20.1g vsize 40.9g index 2.4g - queries are getting slow lately. I have server timeouts etc. Are there any parameters I can tweak beside the default config?
[01:34:21] <sebastiandeutsch> dstorrs: no but we need to identify docs by string (and we have 8mio of them)
[04:34:43] <OsamaBinLaden> whats up with so many posts on HN, hwere ppl are moving away from mongo ? :P
[06:01:57] <hdm> quick question, trying to add a generic MR script for counting unique fields, this seems to blow up: emit(eval("this." + fname), { c : 1 }); in the map routine, fname is defined further up in the same script
[06:41:30] <henrykim> philnate: I don't need more bigger size of bucket size. I am just wondering the fact.
[06:42:14] <henrykim> The situation I met is I set url as a shard-key. the average of size of it is about 70 bytes.
[06:43:10] <henrykim> but, the performance of mongo shards was getting slower and slower after 1 or 2 hours later
[06:44:39] <dstorrs> henrykim: FYI, I wouldn't use anything <v2.0 http://blog.mongodb.org/post/10126837729/mongodb-2-0-released
[07:09:44] <henrykim> dstorrs: for example, currently, I got 100 billion urls now. If I changed them to md5 values, it is all different?
[07:11:04] <henrykim> dstorrs: md5 has 128 bits array size to keep a hash value. I am sure that it's enough size to keep my datas. but, it's algorithm. now, I have no idea about that.
[07:16:26] <philnate> as it's only a hash algorithm there may be collisions, so you may end up having two or more URLs with the same hash, that can happen with every hash algorithm, so you have to live with that.
[07:16:57] <philnate> Although as we mentioned it's quite unlikely to happen you may encounter that two URLs hash to the same value
[07:25:27] <philnate> you just filled your db and then when the memory was filled it started to decrease?
[07:27:14] <henrykim> search-ddm-test1.daum.net:27018 0 0 0 0 0 3 0 38.1g 77.5g 4.71g 0 0 0 0|0 2|0 186b 4k 474 rs_1 M 20:13:55
[07:28:20] <philnate> could you pastebin a few more lines together with a header line? and please tell what your queries are
[07:34:43] <philnate> so some minor improvement could be to remove http:// from your id, as this isn't given you anything
[07:39:01] <philnate> so I may missed it, but can you name it what happend when the performance degration started. When/What did you monitor where you encountered this problem?
[07:39:29] <philnate> Did you just monitor the normal system doing daily business, or was it importing those data, or whatever
[07:41:04] <philnate> btw from what I saw from your mongostat, it looks like only two queries where sent to the server
[07:42:45] <henrykim> from mongostat updates/queries data, I draw the performance graph. I can monitor the total performance is about to half of the max performance after 1 hour.
[07:43:34] <henrykim> this system is storing all documents from our services. we need to store it permanently.
[07:43:52] <philnate> maybe, but you have to consider the other values as well, if no queries enter your system you may see less queries executed
[07:44:41] <henrykim> yes sure, currently I am testing the performance of mongodb. in a real service, we need to ensure several indexes more.
[07:50:45] <philnate> can you show some lines around the position where performance got down, so something around 100-200 lines at most
[07:51:53] <philnate> basically without looking at your stats, you'll have two problems with random inserts as soon as you hit the memory limit. It will decrease to some extend.
[07:54:07] <henrykim> the performance at this time is over 3600 TPS ( sum of all master nodes performance)
[07:58:33] <philnate> hmm the paste shows me that this isn't a insert only scenario, else there wouldn't be some getmore entries, or am I wrong?
[08:00:07] <philnate> additonally you have somewhat many query commands as well, not sure if those are from sharding (and rebalancing)
[08:02:41] <philnate> so whats the test1? why is this server getting such a bunch of query and update statements? Is this a mongoconf server?
[10:31:52] <ub|k> i was wondering what is the best way to express a many to many relationship in mongodb
[10:32:36] <ub|k> i was using a list of DBRefs in each document, but the fact that i.e. mongokit has problems with it makes me think that maybe i am doing something terribly wrong
[10:33:08] <ub|k> i want to know for each speaker all the talks he gave and for each talk who were the speakers
[10:33:44] <NodeX> Personaly I would store what you need to know quickly in an embedded array inside the talks
[10:35:56] <NodeX> "talks.speaker":"bob" ... where your talks collection looks somehting like this ... {speakers : [{speaker:"bob",date:"blah"},{speaker:"john",date:"blah"}...]}
[11:27:04] <tonyk> do slow read queries (for example a sort on an unindexed column) block other read queries on that collection?
[11:49:07] <ub|k> NodeX: still about my earlier question... the problem with your approach is that i will end up having several copies of the same speaker lying around, if (s)he's got more than 1 talk
[11:52:46] <NodeX> you store the uid and the name (because that doesn't change) and the rest in another collection
[12:01:08] <Saby> is it possible to map a formula to a field in my collection, so that the value of that field gets calculated based on the formula
[12:07:29] <Saby> So, I will need to perform the calculation in my code. Is it possible in any way through a hack or something ?
[12:14:30] <tonyk> is count() with no arguments fast enough to be used in place of say storing count on a doc by itself?
[12:20:45] <ledy> after playing with db.largecollection.find( { $or : [ { identifierX : "sha1_hash123" } , { identifierY : "md5_hash456" } , { identifierZ : "another_hash789" } ] } ) i checked the indexes that mongodb prepared automatically.
[12:21:53] <ledy> in addition to the "single" ensureIndex for the identifiers, there is a identifierX_1_identifierY_1_identifierZ_1 now and performance is down :-(
[12:22:40] <ledy> any ideas how to explain to mongodb that it could be enough to use the indexes for the single IDs because i do not have $and, but $or queries?
[12:41:09] <kali> kelye: afaik, you can force mongodb to use a given index, force it to perform a table scan, but not forbid the use of one given index
[12:41:58] <kali> ledy: have you consider performing three queries instead of one and mixing the result in application land ?
[12:52:45] <ledy> kali: strange, after removing the triple-index, it has not been created again. now it is using the single indexes.
[12:53:45] <ledy> What query do you suggest me to use with the find? <i>{ $or : [ { identifierX : "sha1_hash123" } , { identifierY : "md5_hash456" } , { identifierZ : "another_hash789" } ] } </i> OR better 3 * db.find for any single identifierY/Y/Z and merging the results on my own?
[12:55:17] <kali> ledy: i don't remember how $or are treated by the optimizer... but if you perform three simple query, at least, i'm sure you'll get the right index for each query :)
[12:56:02] <kali> ledy: http://www.mongodb.org/display/DOCS/Indexing+Advice+and+FAQ#IndexingAdviceandFAQ-Oneindexperquery.
[13:10:06] <ledy> i'd prefer to let mongodb do the job with one query including all three identifiers in the $or... i hope $or is the "lazy operator" for this statement. so mongodb can stop on its own after first match when i use findOne...
[13:40:21] <leecher> Hey guys, I'm trying to use find to locate all documents that do not have the filed "deleted_at" .. give me a tip on how to do that?
[13:40:56] <rexxars> leecher: http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-%24exists
[14:11:39] <Derick> ledy: http://meta.stackoverflow.com/questions/5234/how-does-accepting-an-answer-work
[15:36:23] <tonyk> kali: I dont know why count(*) is "slow" on SQL servers, should be an easy problem to solve, right?
[18:20:59] <linsys> yes you can replicate your data across continents but you asked for master-master
[18:24:08] <linsys> a replica set is a set of mongodb servers with one master or PRIMARY while the rest are secondaries or slaves
[18:24:20] <linsys> if the primary fails one of the secondaries takes over the function of the master
[18:25:30] <NodeX> it kind of is master/master replication because they both have a full copy of the data
[18:25:33] <linsys> http://www.mongodb.org/display/DOCS/Data+Center+Awareness#DataCenterAwareness-Tagging%28version2.0%29
[18:27:20] <NodeX> http://www.mongodb.org/display/DOCS/2.0+Release+Notes#2.0ReleaseNotes-Datacenterawareness
[18:27:36] <NodeX> "You can now "tag" replica set members to indicate their location. You can use these tags to design custom write rules across data centers, racks, specific servers, or any other architecture choice."
[18:29:01] <linsys> That is for devs to code that they want to make sure a write goes to two racks or 3 specific servers etc..
[18:30:11] <NodeX> it was not for you.. you obvisouly know how mongo works so it can't be for you ;)
[18:35:11] <souza> guys i have a non tecnical question, what is more recommended using mongodb, i can use three schemas, and make relations among them, or create only an schema and create arrays inside this schema, what's more recommended? Thanks
[18:37:53] <souza> NodeX: i don't understand, i think that's the same think, but showed in differents ways.
[18:38:43] <linsys> souza: It depends very much so on your data model... If the single json document will only grow to a certain size you might want to imbed. If you want to create a chat service and user Joe: is going to have 100s or even 1000s of imbeded conversations in a single json document its better to create a new collection called chat and reference Joe's conversations from that collection. Each conversation its own document
[18:41:54] <souza> linsys: in my case i'm using this only for tests, i want to login determinated system, and all my actions generate a log, the i'll have a "object" user and another "log"
[18:43:58] <linsys> Not sure I understand exactly.. sounds like the actions could grow and grow. If this is a test you can make it as simple as you want but if the actions are going to grow in an unlimited manner once you go to prod you might want to break them out into their own collection
[19:03:38] <linsys> spicyWith: I am actuaing using Fabric to do ec2 deployments working on that right now
[19:04:46] <spicyWith> linsys: ah cool - used to use fabric as well. just discovered cloudformation which seems very powerful to describe infrastructures - having some trouble attaching EBS volumes to ubuntu though
[19:08:05] <nicholasdipiazza> why is mongodb so much better for documents with plenty of internal structure versus a small fixed size?
[19:09:28] <durre_> if I have two "entities" (Domain & Position). I want to find all positions for a certain domain. should I "link" to Domain with the id, or should position contain the domain name to letting me do the query? what's "the mongo way"?
[20:11:03] <dstorrs> it so happens that the string representation of the one on the "broken" box is 900 bytes
[20:11:38] <dstorrs> whereas the other one is 2613 due to data diffs / filesystem diffs / version diffs / whatever
[20:14:09] <dstorrs> I very much doubt that length is 'broken', except perhaps in the sense that sharks are broken as paintbrushes :>
[20:15:11] <dstorrs> so, clarify for me what exactly you're trying to do? page through a collection <=900 elements at a time?
[20:19:04] <dstorrs> that's all the paging control you'll ever need. just store the '900' (or whatever) in an env var / config / code etc
[20:20:56] <dstorrs> speaking of configs / env vars, has anyone else seen this? http://www.12factor.net/
[20:25:03] <dstorrs> (the '...' should be read as'or the database handle, which must be stored in DB and then referenced by state stored in the....)
[20:27:50] <dstorrs> db.COLL.find(NNN).limit(MMM) means "create a cursor in collection COLL that is restricted to at most 900 rows and then will return 'undef'"
[20:28:45] <dstorrs> I don't know why you keep trying to apply .length() to it. I can't even tell what you think it's going to do, which is why I'm having trouble helping.
[20:29:15] <dstorrs> step back and lay out for me exactly what you're trying to achieve. I'll help if I can.
[20:30:25] <dstorrs> ayakushev: I'll say it one more time, then I give up. Please step back and lay out for me exactly what you're trying to achieve. I'll help if I can.
[20:35:49] <dstorrs> you and fjay have mentioned the number '804' twice now. The first time was when he said you were doing this: db.profile.find().limit(900).length()
[20:36:21] <dstorrs> you then came in and said you were doing this: db.profile.find().limit(900).toArray().length
[20:37:36] <dstorrs> just to eliminate the obvious -- in the mongo shell, do db.profiles.count() to verify record total
[20:38:45] <fjay> but... on another box we are using sharding.. and it doesnt show the issue ;) perhaps from sharding after the fact
[20:39:10] <dstorrs> but I suspect that what's happened is that on one box there are actually >900 docs resident locally
[20:42:15] <ayakushev> but, as fjay mentioned, copying the whole db to a different place fixes the world
[20:43:19] <dstorrs> sorry guys, I got nothin'. All I can say is try to narrow it to that one record and then throw it in JIRA so it can be fixed in next release.
[21:31:36] <nicholasdipiazza> If I have doc = {"inner":[{"innerId":1, "name":"nick"}, {"innerId":2, "name":"fred"}, ..., {"innerId":100, "name":"jeff"}, ]} How can I update everyone's name to "Todd"
[21:36:12] <nicholasdipiazza> i am having trouble knowing what operations i can perform on the document variables themselves
[21:38:32] <dstorrs> nicholasdipiazza: as to the updating...you could write a loop, you could map/reduce, you could do it in app code, you could replace all of those records with a single one -- it all depends on what you're trying to achieve
[21:39:10] <nicholasdipiazza> how can i update that document (without using db.save or update) form mongo console
[21:39:56] <dstorrs> erm...why are db.save and db.update verboten? If you're trying to change data from the console, that's kinda how you do it
[21:40:35] <dstorrs> also, step back farther. what does this collection represent? why are you using embedded docs? why the update?
[21:40:37] <nicholasdipiazza> i'm used to treating a JSON object in Javascript... where if I loaded an object with an array of strings, i could loop through and update those strings with a for loop
[21:41:32] <dstorrs> you can do the exact same thing in mongo. The diff is that if you want the data to persist, you need to save it back to disk
[21:42:07] <nicholasdipiazza> i'm stomped how to do that sort of stuff after getting an object with find
[21:42:28] <nicholasdipiazza> I've found my object... how I want to update a bunch of the array values stored in that object, then update it
[21:43:35] <dstorrs> nicholasdipiazza: I'll save you some time -- if someone else wants to help you with this, more power to both of you. But I'm not comfortable doing someone else's homework, so I'm going to bow out.
[21:47:35] <dstorrs> nicholasdipiazza: good for you for finding the answer afterwards. I'd start looking here: http://docs.mongodb.org/manual/reference/javascript/
[21:48:15] <dstorrs> you may also want to look at this: http://docs.mongodb.org/manual/reference/javascript/#cursor-methods
[21:49:21] <dstorrs> at each step ask yourself "if I were doing this in a JS script, how would I do it?" then back that up until you've turned the mongo environment into essentially the same script
[21:49:22] <nicholasdipiazza> why are these api docs not easier to get online? i was looking for something like this with no luck
[21:53:23] <dstorrs> nicholasdipiazza: as to finding the docs, try this: http://lmgtfy.com/?q=mongo+shell ;>
[21:56:28] <dstorrs> nicholasdipiazza: if you're at all interested in Mongo, I STRONGLY suggest setting aside 3-4 hours and reading all of docs under http://docs.mongodb.org/manual/
[21:57:03] <modcure> lets say i fire up mongodb. I run a query(get documents from a collection). mongodb would need to fetch this from vritual memory(on disk but faster than non virtual memory). this would cause a page fault since its the first time the data is being access?
[21:57:11] <dstorrs> ramsey: you know, I've never seen any of Derick kchodoro_ or tychoish actually active on this channel. have you?
[21:58:07] <dstorrs> or hey, I just realized. kchodorow must be Kristina Chodorow -- shiny! we are the in presence of greatness.
[21:59:00] <eph3meral> so I'm basically absolutely new to mongo querying, I've worked a bit with Mongoid and I did the basics of this like, last year... plenty familiar with SQL and JavaScript themselves... so anyway...
[21:59:10] <eph3meral> i've come across this issue here while reading http://www.mongodb.org/display/DOCS/Updating#Updating-%24set
[21:59:11] <dstorrs> modcure: they typically suggest that you keep your working set small enough to fit in RAM
[22:00:01] <eph3meral> db.dashboards.update( {_id: "4fce7c96b29a7b141c000001"}, { $set : { creator_id : "4fce7b10b29a7b12c7000001" } } )
[22:00:13] <dstorrs> eph3meral: step back for a second and tell us what you're trying to achieve at a high level
[22:00:35] <modcure> dstorrs, when mongodb loads. it maps the data files to virtual memory(on disk but faster than regular block on disk). then i run a query . mongodb would need to fetch the data from virtual memory into memory? this would be consider a page fault?
[22:00:55] <eph3meral> dstorrs, i know what you mean, but there is not much higher of a level than this
[22:01:52] <eph3meral> dstorrs, it's unicorns, and it's for fun, because I'm curious and I want to know how to do this
[22:04:34] <dstorrs> kchodoro_: I write in Perl and have been using your MongoDB driver. It is great -- thank you.
[22:04:53] <modcure> dstorrs, since mongodb maps the files to virtual memory on disk. this would mean I would need double space for my data? one for disk and one for virtual disk?
[22:05:38] <eph3meral> kchodoro_, essentially "nothing" i get no response from the mongo shell at all, i hit enter, it goes back to the prompt, and when I do a subsequent db.dashboards.find() the data is not there
[22:05:59] <dstorrs> modcure: dude, you're in a channel with kchodoro_ answering questions! don't ask me, ask her. :>
[22:07:44] <kchodoro_> eph3meral: can you pastebin what do you get doing db.dashboards.findOne({_id: "4fce7c96b29a7b141c000001"})?
[22:08:43] <modcure> kchodoro_, when mongodb loads. it maps the data files to virtual memory(on disk but faster than regular block on disk). then i run a query . mongodb would need to fetch the data from virtual memory into memory? this would be consider a page fault?
[22:10:14] <eph3meral> yep, i was looking in to just that issue, didn't know what the syntax was tho, so thanks
[22:11:05] <modcure> kchodoro_, since mongodb maps the files to virtual memory on disk. this would mean I would need double space for my data? one for disk and one for virtual disk?
[22:11:20] <dstorrs> kchodoro_: are all ObjectIDs with the same hash considered == ? And are they considered === ?
[22:20:47] <modcure> kchodoro_, since mongodb maps the files to virtual memory on disk. this would mean I would need double space for my data? one for disk and one for virtual disk?
[22:22:08] <kchodoro_> modcure: nope, just once... although journaling can mess with that, it can map everything a second time, i think
[22:22:57] <modcure> kchodoro_, im lost. the data in on disk... mongodb loads the data files to virtual memory(this is in disk too). wouldnt this be double the space?
[22:42:15] <timebomb> heya, say i have a company with many locations, should these be embedded via embeddeddocuments? and will i then still be able to make use of the spatial indexing?
[23:05:21] <dstorrs> kchodoro_: (back. had to go interview someone) Bug report is here: https://jira.mongodb.org/browse/SERVER-6010
[23:16:23] <dstorrs> kchodorow: fjay is doing something like this: db.profiles.find().limit(900) and getting only ~800 records back from a >2000 record sharded collection. Is there any reason you can think of for this, or is this a "file a bug" thing?