Log file Viewer
#mongodb logs for Monday the 23rd of July, 2012
[00:00:03] <Siegfried> don't know if i should post here, i use python to access mongodb, when inserting a large integer i get "MongoDB can only handle up to 8-byte ints" but works with the cli
[00:04:39] <vsmatck> It'd be less efficient to store a ASCII integer than to store a 128bit int as binary data.
[01:01:36] <krz> my data structure looks like this: https://gist.github.com/3161542 how do I return only the visits that have minute greater than 12 ?
[01:03:25] <vsmatck> http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-%3C%2C%3C%3D%2C%3E%2C%3E%3D
[02:59:25] <krz> my data structure looks like this: https://gist.github.com/3161542 how do I return only the visits that have minute greater than 12 ? is such possible without doing a map reduce?
[06:52:58] <sunoano> I didn't find any info on whether or not current pymongo (2.2.1) has already TTL support. Also, didn't find any issue regarding this in jira.mongodb.org. Any pointers?
[09:09:14] <omid8bimo> hey. can i dump only multiple collections for a databse via mongodump command?
[09:18:52] <omid8bimo> NodeX: what do you mean loop it? i just want to dump just a couple of collections in my database. mongodump -c takes only one collection name! any way around this?
[09:29:01] <omid8bimo> NodeX mmm should i put the collection variable in an array in bash or just a white space seperated line?
[10:36:21] <Mortah> hello, we've somehow got two system.replset collections in our replica set... any suggestions?
[11:49:50] <ra21vi> I have two collectios. One contains logs with a key, and other collection has users with list of their keys. I have to generate report of logs for each user. Generally, I need to search all logs of given keys for a user. How can I achieve this with map reduce? I am totally clueless on how to proceed. Do map reduce support multi collection?
[11:50:42] <ra21vi> i just have to find the logs of containing any key in users keys array, for a given date range (start date end date)
[11:55:52] <ra21vi> jY: yes. I tried to understand and corelate with my problem. But I am really clueless.
[11:56:06] <jY> http://stackoverflow.com/questions/3837394/mongodb-map-reduce-over-multiple-collections
[12:06:03] <ra21vi> jY: does that stackoerflow solutions really cover multi collections joining. I see that they are just reducing each collections that has similiar data (logs) to another and then re-reduce. In my case i have log and users. Just to make the problem clear, here is what i do to find all logs for a given user: user_collection.find({'key': {'$ne':[]}}, fields={'_id': 1}, partial=True) . this gets
[12:06:04] <ra21vi> me all users which has atleast some items in key array. I have logs with structure: { 'timestamp': ISODate("2012-04-22T20:00:00Z"), 'ops': 'GET', 'bytes': 2047, 'http_status': 200, 'reqid': 'B1CF908AE5' }.
[12:07:35] <ra21vi> Waht I am trying to find is, all GET ops logs count for all keys of a given user in given timeframe.
[12:08:16] <ra21vi> Right now, I am generating with async tasks, which runs for each users in my program.
[12:09:19] <ra21vi> jY: yes, but I cannot merge them, Since logs ccomes from different source and I get to sync users keys as well as new users from app db. I don't have control on third party system which sends the log (Amazon S3)
[12:16:41] <neil__g> in mongo m/r, how would i get the number of attributes in a document, this.length?
[12:55:05] <Mortah> Just replaced a secondary with another server (same hostname/port but different IP) and although the replica set confirms it is there we're not seeing mongos send any slave queries to it
[13:01:36] <_pid> hi, any hints..... I have a schema like this { offers: { "ID_329487123984" : { alias: "offer_1" } } } now I want to query offers.XXXX.alias: "offer_2" .... where XXXX doenT matter..... is it a shit of schema or is it possible? similar query on array aaa.$$.bbb ?
[13:05:38] <_pid> I havenT the key.... I want all documents where for ex. in offers.**.alias: "offer_1"
[13:07:03] <Mortah> I don't think you can do that with that schema... changing offers to a list and moving the ID inside would let you do .find({offers.alias: "offer_1"})
[13:09:26] <Mortah> http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-JavascriptExpressionsand%7B%7B%24where%7D%7D
[13:23:19] <matubaum> hello, I'm getting this error on ubuntu when i try to start mongo: terminate called after throwing an instance of 'std::runtime_error'. what(): locale::facet::_S_create_c_locale name not valid
[13:27:04] <adamcom> matubaum: what OS and version are you trying it on? (does ./mongod --version work?)
[13:29:25] <matubaum> adamcom: it was working but the server was shuted down unexpectedly, and it stopped working
[13:39:19] <adamcom> edit /etc/locale-gen uncomment your LOCALE set in /etc/rc.conf and run locale-gen
[13:45:34] <matubaum> adamcom: I found the same too, but is for arch, I don't have those files in my server
[13:46:45] <matubaum> adamcom: I'm going to see if i can set LOCALE some other way, I've tried with locale-gen command but it didn't work out
[13:53:55] <ron> silly question, assuming a field is indexed, what's the complexity of sorting by that field?
[13:56:41] <ron> yes. what's the computational complexity of sorting the documents if the field is indexed?
[14:03:39] <matubaum> adamcom: I setted the locale with locale-gen and I forgot to log in again. So, thank you very much
[14:05:08] <NodeX> iirc a sort is always applied to every result set as it's either $natural : 1 || $natural : -1
[14:05:40] <ron> NodeX: I see. so finding the 'new'est document requires sorting, which means the query will still take nlogn even though it can take n.
[14:08:22] <ron> adamcom: it's been too long from school, I don't remember the specifics of b-trees :-/
[14:09:59] <adamcom> heh, ron, don't worry, that's what I use wikipedia for http://en.wikipedia.org/wiki/B-tree
[14:10:15] <NodeX> ra21vi : in addition to my comment - it's not really logical to make a massive query because your data will have to be indexed for it which will make your indexes huge
[14:10:18] <ron> I'm asking hypothetically, by the way. One of my employees was concerned regarding retrieving the 'newest' document and its performance. After I realized he was talking about a capped collection of 100 entries, there was a bit of yelling to go around.
[14:12:21] <NodeX> http://docs.mongodb.org/manual/reference/javascript/?highlight=findone#db.collection.findOne <---
[14:12:49] <NodeX> "natural order: The order in which a database stores documents on disk. Typically this order is the same as the insertion order. Capped collections, among other things, guarantee that insertion order and natural order are identical."
[14:14:36] <ron> come to think of it, it's probably not the latest document we need, but rather the one that was updated most recently (i.e. using a timestamp field).
[14:14:40] <adamcom> you also want to be careful with updates on capped collections too ron - you can't have an update to grow a document
[14:14:50] <NodeX> Google seems to loe them too... I do my lazy "Mongodb findOne()" into google and it lands your links right near the top
[14:17:01] <NodeX> I'm surprised you guys didn't use elasticsearch ... it's very similar to mongo in structure (json/bson)
[14:17:02] <adamcom> yes, as long as you don't do something like cross an integer boundary or something weird
[14:18:03] <ron> adamcom: well, for a while there, the first developer here actually configured the driver to ignore errors, so we had no idea why things didn't fail but didn't work too.
[14:18:18] <adamcom> wikipedia to the rescue again: http://en.wikipedia.org/wiki/Sphinx_(documentation_generator)
[14:18:58] <NodeX> I assumed you made some trendy search thing to run through a sphinx index and spit out faceted like answers or something!!
[14:19:40] <adamcom> heh, no, generally easy to use, but to generate the PDFs you have to install LateX, which is huge, on my mac at least
[14:23:45] <ron> okay, so just to sum things up (even though it makes perfect sense), if I want to find a document with a highest value of a field, it will have to traverse the entire b-tree no matter what in order to find the relevant document?
[14:25:00] <ron> and traversal will be the same no matter which method is used to query the highest value of a field (assuming it's not only sort(...).limit(1))?
[14:26:46] <NodeX> err, you're like the biggest troll on the interwebz, it should be me not trusting what you say not the otehr way round :P
[14:30:15] <ron> ah wait, maybe it does. is there a direct query form to find the document holding the maximum value of a given field?
[14:31:20] <ron> excellent! so mongo doesn't offer a O(n) method of finding a maximum value no matter what! is that correct?
[14:31:27] <NodeX> a MAX() would be nice but there isn't one .. probably because find({foo : { $gt : 0}}).limit(1).sort({foo:-1}); does the trick
[14:32:21] <ron> it does the trick, though computationally it should be more expensive (unless I'm missing out on something).
[14:35:06] <ron> If you need to find the maximum value, you need to go through all the entries one. that is, O(n). if you sort and limit, sorting would be O(nlogn)...
[14:35:19] <Mortah> http://www.colinhowe.co.uk/2012/jul/23/ssds-on-aws--impact-on-conversocial/ <- MongoDB on SSDs on AWS if anyone cares for performance examples :)
[14:37:28] <NodeX> it's not somehting I have ever really looked into, I assume that the reason behind no MAX() is my above query works and perhaps some internals optimise for it?
[14:37:31] <ron> The real question is, though, assuming what you previously said, if the query is used once, then there's a cache of the result and it will be sorted next time? Is that correct?
[14:38:23] <ron> to be clear, there will be a cache of the sort, so it won't be sorted again. correct?
[14:39:37] <ron> but if a new document is inserted to the collection, I imagine the sort will happen all over again when the query is executed again.
[14:39:41] <NodeX> what I do know is ... if I do a query and a count with a sort in it then run the same query again, the second query is faster
[14:40:25] <NodeX> I doubt the indexer is smart enough to know the inserted document effected the query cache and add to it - it probably invalidates the cache ^^
[14:42:11] <ron> hmm, that's a bit difficult to know, but in any case, from what you're saying, there no way to execute the query differently, so there's no way to make it more efficient.
[14:55:02] <amitprakash> Hi.. How can I copy all records from collection A to collection B via pymongo?
[15:15:52] <amitprakash> NodeX, I dont want to really clone a collection.. I have collection X where I insert records 1 by 1, I have a collection Y which already has records, I want to add all records in bulk from collection X to Y
[15:42:09] <jibay> Hi all, I'm looking for a solution which provides a mail server (only to receive e-mails), with an unlimited number of e-mail accounts. One main feature should be that received e-mails can be accessed via an API.
[15:43:10] <Derick> I think you're in the wrong place then? We're a database, nothing to do with email.
[16:18:17] <hdm> random question - when passing in a json { key : value } block, what is the actual syntax for providing a variable for the key? var mkey = "bob"; { mkey : "value" } just uses the key 'mkey'.
[17:27:15] <hdm> fml, 11:27:29 [conn22] Assertion: 10334:Invalid BSONObj size: 1952214620 (0x5C6E5C74) first element: :/
[17:27:35] <hdm> 72 hour data load with --objcheck, --journal, mongo 2.0.6, still hit those on m/r jobs
[19:45:27] <k_89> hey .. i am trying to figure out a use-case for mongodb in a real world app .. i tried googling but all i can find are abstract stuff like big data .. key0value storage etc etc
[19:48:49] <crudson> k_89: http://www.mongodb.org/display/DOCS/Use+Cases http://www.mongodb.org/display/DOCS/Production+Deployments
[19:49:52] <anthroprose> k_89: I just used a collection to implement a very simply workflow management routine...
[19:51:55] <k_89> anthroprose, there is on thing which gives me a mental block when i try to think in terms of mongodb ... many-to-many relations
[19:54:28] <anthroprose> you have to make them up... you must structure you documents in a way to allow that
[19:55:03] <anthroprose> like I generate a GUID at the start of my processes... and it gets stamped on logs, workflows, metadata documents
[19:56:45] <k_89> gona give mongo a shot tonight to see what its all about .. thanks fir your replies
[19:57:12] <anthroprose> k_90: put some data into an array structure that makes sense to you... and then store that in mongo
[20:28:23] <dAnjou> hey, i thinking about a little "weekend" project. basically just a tweet-long text with tags. of course i want to search by tags and so on. would mongodb be a good choice or should i stick with sql?
[20:30:56] <anthroprose> production for 2+ years... but nothing bigdata enough to think about performance
[20:31:27] <anthroprose> "much faster than everything else" is usually my casual experience, unless you have high write load
[21:23:02] <b0fh_ua> wereHamster: that makes more sense, but how to configure amount of RAM in mongo?
[21:24:37] <b0fh_ua> it's not true, as I can see - I have like 2GB of free memory and mongo demonstrates huge IO utilization
[21:27:38] <TkTech> wereHamster: I apologies, I do not believe the stock mongo can actually limit its memory usage. Looks like something in the fork.
[21:43:44] <gustonegro> wereHamster: hmm. that's how I thought it would work...and that is how my data is structured. Not sure why it is not working on my data.
[21:48:37] <gustonegro> wereHamster: yeah, gotit. I'm using mongoosejs. Had to write it in the shell like you say using a cursor.
[21:53:41] <gustonegro> oh, I see now. I was saving my stuff from JSON data. ..so all INTs were stored as strings
[21:57:46] <gustonegro> wereHamster: cast items stored with string data, such as {"major": "6"} to int data such as {
[22:26:08] <gustonegro> how do you search for a field that is stored in an object that is itself stored in an array of objects?
[22:28:34] <gustonegro> I want to do something like this: db.datas.find( { "content.patcher.boxes[*].box.class" : "message" } );
[22:28:51] <doxavore> gustonegro: if you have { things: [ {name:'first'},{name:'second'} ] } ... I believe you can do: find({ 'things.$.name': 'second' })
[22:43:36] <gustonegro> find({ 'content.patcher.boxes.0.box.class' : 'message' }) works, but find({ 'content.patcher.boxes.$.box.class' : 'message' }) does not work
[22:46:58] <gustonegro> nevermind, I see that you don't even need a wildcard. what I want is: find({ 'content.patcher.boxes.box.class' : 'message' })
[22:47:24] <augustl> are there any web based mongodb GUIs around, phpmyadmin/couchdb futon/etc style?