Log file Viewer
#mongodb logs for Monday the 14th of October, 2013
[01:03:32] <BurtyB> Derick, ah - not as bad as the "for developer" ones then (well unless one of the languages they offer is the one you use)
[01:11:57] <BurtyB> Derick, I did signup when they first announced them but had no clue about python and making it work with apache so gave in when they expected it to be working
[01:12:24] <metasansana> BurtyB, mostly shell so far yes but I believing there are parts related to deployment like clusters and sharding etc.
[04:07:24] <bin> okay guys here is the question. I have an array of objects. How to specify dynamically (based on the value of some property) to which element of the array to update its property
[05:48:25] <Xzyx987X> there appears to be a rather gaping flaw in ruby's mongo library. say you need to have some code run while mongo is write locked. and say you have to run the code in a critical (exclusive) thread, because in ruby's thread model a global hang will occur if another thread tries to write to the database while it is locked. that would be fine, except say there is a bug in ruby's mongo library that causes queries to hang if excecuted i
[06:02:29] <Xzyx987X> also, if I excucute a lot of queries one after another, it makes my program exit without even throwing an error
[06:17:52] <Xzyx987X> ok, now I've figured out how to get the program to hang when executing a read query after write locking the database, even without excecuting it in a critical thread
[06:54:15] <Neptu> python driver, I mean i do not find any specific method so I might use a command??
[07:11:38] <Xzyx987X> ok, I guess to generalize my question here, what might cause mongodb to hang on a find query when write locked?
[07:13:07] <Xzyx987X> I've isolated all the variables, and have determined that all my issues go away if I don't attempt to write lock the database, but I still don't understand how a write lock would cause a query to hang that only reads
[07:42:49] <mboman> Hi guys. I have some issues when I want to update (upsert) a record. Python code looks like this: result = self.db.malware.update({'sha1': sha1sum}, malware, upsert=True) and the error I get is InvalidDocument: key '$oid' must not start with '$'. Suggestions?
[08:20:21] <Xzyx987X> *sigh*, I don't suppose either of you know why a write lock would cause a read query to hang do you?
[08:21:51] <liquid-silence> ok guys, I need to build a hierarchy tree, where users have a "container" (similar to your home folder in linux), where you can create folders and sub folders and upload files
[08:22:13] <joannac> Xzyx987X: there's one lock. While the write lock is taken, no reads can progress.
[08:22:23] <liquid-silence> also you need to be able to give access to someone else to access a file/folder, read, write, browse
[08:23:48] <joannac> Or, do it the brute force way, and see how bad it is, and that'll help guide you to how to make it better.
[08:24:00] <Xzyx987X> then why is it when I ran my code in the debugger or irb it would process read queries after write locking?
[08:24:49] <Xzyx987X> if a write lock isn't the anser, how can I query two collection while ensure no writes update then while running the query?
[08:26:02] <Xzyx987X> so what you're saying is, mongo has no useful built in locking capability whatsoever? wonderful...
[08:26:53] <liquid-silence> db.containers.update({ "folders._id": ObjectId("525baa0cf70dc4d6b7ceecc0")}, { $push: { "files": { _id: ObjectId......
[08:27:19] <joannac> Xzyx987X: sorry, misread. you want to read 2 collections and not allow writes while you're doing so?
[08:29:51] <liquid-silence> db.containers.update({ "folders._id": ObjectId("525baa0cf70dc4d6b7ceecc0")}, { $push: { folders.files: { _id:
[08:31:51] <liquid-silence> db.containers.update({ "folders._id": ObjectId("525baa0cf70dc4d6b7ceecc0")}, { $push: { folders.files: { _id: ObjectId(), name: "test.mp4" }}})
[08:33:04] <Xzyx987X> joannac, it sort of is, I'm not really sure how to apply the information to my code though. here are the two queries I need to make run without any write occuring between them:
[08:34:31] <joannac> Xzyx987X: similar principle. Have a collection with a "I'm reading flag". Set the flag (if not set) when you start your read. Any write should check if that flag is set; if so, wait.
[08:37:42] <joannac> liquid-silence: oh duh.you need to quote a string with a . in it". Also you may need an upsert flag?
[08:37:44] <Xzyx987X> but then if I find that it's enabled, from what I understand the only way to know when garbage collection is complete is to keep polling the flag
[08:38:19] <liquid-silence> db.containers.update({ "folders._id": ObjectId("525baa0cf70dc4d6b7ceecc0")}, { $set: { "folders.files": { _id: ObjectId(), name: "test.mp4" }}}, true.
[08:38:41] <Xzyx987X> depending on the amount of pending write queries, the traffic generated by polling could be rather large
[08:41:12] <liquid-silence> b.containers.update({ "folders._id": ObjectId("525baa0cf70dc4d6b7ceecc0")}, { $set: { "folders.$.files": { _id: ObjectId(), name: "test.mp4" }}}, true
[08:42:19] <Xzyx987X> joannac, so is there any way the run a mongo query that will block until a certain condition is satisfied to avoid the polling issue?
[08:43:38] <joannac> Xzyx987X: Erm, no. What if the query gets stuck? You want to take down the whole server?
[08:45:01] <joannac> Everything will yield: http://docs.mongodb.org/manual/faq/concurrency/#does-a-read-or-write-operation-ever-yield-the-lock
[08:47:47] <Xzyx987X> I'm probably not going to be much help... I've been working with mongo for all of three days now...
[08:59:08] <liquid-silence> joannac db.containers.update({ "folders.folders._id": ObjectId("525bb112f70dc4d6b7ceecd3")}, { $push: { "folders.$.folders": { _id: ObjectId(), name: "Sub folder" }}})
[09:01:50] <Xzyx987X> haha, wow, I just realized that if I changed the order of the queries, it actually wouldn't matter in this particular case if the data was updated between then >.<
[09:11:23] <Xzyx987X> you feel dumb? I just spent the past four hours trying to solve a problem that could have been solved by switching to positions of two lines of code
[09:13:48] <jackblackCH> hi, anyone know know how to define the encoding when using mongoexport --csv ? i have special chars which looks bad in my csv
[09:15:43] <jackblackCH> i have a collection of users with email addressees , they registered on the web. now my project manager needs a list
[09:15:48] <liquid-silence> db.containers.update({ "folders.folders._id": ObjectId("525bb56df70dc4d6b7ceecf7")}, { $push: { "folders.$.folders.folders": { _id: ObjectId(), name: "Third Level" }}})
[09:17:10] <Kim^J> jackblackCH: Then you're free to handle encoding how you like. And format and suck.
[11:34:57] <Nomikos> leifw: if you remember that search/sort found-count discrepancy I had last week, turned out the index on that field had only indexed about half the documents >.>
[11:35:31] <Nomikos> removing index, all 700 found (sort() wasn't adding docs, the search was just not finding them all)
[11:44:59] <bin> Hello guys! I have a question and it is: Can i use $elemMatch operator when i want to update particular element in an array and if yes , could you give me an example please. Cheers in advance. ( Couldn't find example in google)
[12:01:01] <Nomikos> joannac: is there some way to use the new Text Search for the $match part of the aggregation pipeline? the regular index fails because the field contents are too long sometimes..
[12:08:04] <Nomikos> I'm wanting to use the aggregation pipeline to show found-docs-per-category, but I guess I'll have to run that one without an index then.
[12:08:44] <Nomikos> the actual search results showing up in the main part of the page could then still use the new Text Search
[12:10:36] <bin> joannac: sorry to bother ya i will give you an example .. tried to align them properly. -> http://pastebin.com/95b711kR
[13:03:52] <_Heisenberg_> Hi folks. I have a problem with the readPreferred settings. Even if I use a command like db.products.find().readPref (’primaryPreferred’).count() the operation blocks while the cluster is doing a failover and answers after a new primary is elected. I was expecting that the secondaries answer the query even if the cluster is performing a failover?
[13:04:31] <_Heisenberg_> Same for my node.js application where I set the readPreference in the application of course...
[13:22:52] <kali> _Heisenberg_: during failover, everything is more or less on hold until the election whatever the read preference is
[13:27:45] <_Heisenberg_> kali: the documentations says something different: "In most situations, operations read from the primary member of the set. However, if the primary is unavailable, as is the case during failover situations, operations read from secondary members." ( http://docs.mongodb.org/manual/reference/read-preference/#primaryPreferred)
[13:29:38] <Neptu> hej quiestion about the python driver... I do nto find a method the drop the collection, should I use command or maybe Im mistakend and there is a method on the mongoClient?
[13:30:02] <Neptu> second question is proper to use mongoclient against a replica or or I need the mongoclientReplica ?
[13:31:46] <_Heisenberg_> kali: Thats my problem! As I'm using mongo in my thesis I have to make sure that this behaviour is not my fault ^^
[13:33:14] <kali> _Heisenberg_: you may want to try it with another client as the node.js one may not be the most "compliant" of them. try the java one for instance
[13:34:15] <_Heisenberg_> kali: I think that is not needed since I watch that behaviour even in the mongos shell :/
[13:36:02] <_Heisenberg_> I'm trying to set diaglog=3 in my mongos config file for a jira issue but if I put that parameter the restart of mongos fails. suggestions?
[13:46:45] <Danielss89> when i use find() on my collection i get some documents but i have to type "it" to show more.. can i get it to show all instead?
[14:32:40] <_Heisenberg_> kali: fyi, the problem with the readpreference was already reported: https://jira.mongodb.org/browse/SERVER-7246
[15:09:44] <PiyushK> hey .. any considerations / pointers to consider while choosing between mongodb and couchbase .?
[15:25:17] <PiyushK> Nodex, for personalization service and recommendation engine to store and analyze web usage data ...
[15:25:55] <Nomikos> I just found http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis and .. whoa.
[15:46:16] <theCzar> Mongodb n00b here. I'm working on setting up some scripts to configure data migration for MongoDB from one host to another. With databases like Redis, in the past I have just told one running instance to become the slave of the other, waited for them to get in sync, and then shut off the master. What is the best way to do this in MongoDB? I have only seen a way to configure master-slave replication at startup.
[16:02:57] <jiffe99> there something wrong with this statement? db.orders.remove({'coin':'INF','$lt':{'timestamp':1381766396}})
[16:03:13] <jiffe99> after running it db.orders.find({'coin':'INF'}).pretty() shows entries with timestamps less than that value still in there
[16:35:57] <clarkk> I have a collection of documents with a "category" field - thus multiple documents have the same category. I need to query the docs so that I get arrays of documents with the same category. Could someone give me a hint as to how I would go about this?
[16:42:37] <Nomikos> can aggregation be used in any way with an indexer in front of MongoDB, or do those indexers have something similar to aggregation?
[16:49:05] <tripflex> you may not even need to use aggregation but it all depends on how your schema is setup
[16:56:47] <clarkk> tripflex: Nomikos my schema is very simple (although the dataset is much larger than this, obviously) http://pastebin.ca/raw/2466634
[16:59:46] <tripflex> { category: 'News', data: [ { title: 'Google News' }, { title: 'independent' } ] }
[17:01:17] <tripflex> the results will be the document it found with category News or whatever you use find for
[17:03:02] <clarkk> groupby seems to reduce the items down to one value (eg to count all items in a category)
[17:03:54] <tripflex> http://smyl.es/how-to-count-the-number-of-values-found-for-a-field-in-mongodb-using-node-js-and-mongoose-using-aggregation/
[17:04:36] <clarkk> tripflex: I don't want to count the fields - I was giving an example of what I don't want
[17:07:00] <Nomikos> the logic is in the code, not in mongodb, so it may not be what you were looking for..
[17:07:26] <Nomikos> the mongodb part would simply be a db.coll.find().. I'm pretty new to the db side of things >.>
[17:08:45] <Nomikos> if you sorted on category, then title, it might be simpler still to use the result set. simply set a var for this_category and compare that to category on each .. whatever it is you're doing with the loop
[17:10:34] <clarkk> if (! out[doc.category]) checks whether teh category field exists. However, they will always exist in those records
[17:19:25] <tripflex> so like, go through my model, group everything by the _id that equals $category, then add category field with $category, and title field. In title field add to the array in the title of current document
[17:21:24] <clarkk> tripflex: ok, but I don't want to look at the code until I've gone through the article
[17:21:29] <tripflex> and it would group by documents that have the same cateogyr and parentcategory
[17:22:51] <tripflex> unfortunate thing is there's not much information on the internet about aggregation framework
[17:24:15] <tripflex> but if you watch those two videos from my blog, it will give you a good understanding of it
[17:24:28] <tripflex> http://smyl.es/how-to-count-the-number-of-values-found-for-a-field-in-mongodb-using-node-js-and-mongoose-using-aggregation/
[17:37:08] <liquid-silence> is there a way to pass in 525bdda1000000b975000089 and find the document with that name?\
[17:38:13] <liquid-silence> but I dont want to do the dot annotation because from the client side it will not be great
[17:44:54] <rafaelhbarros> joannac: hello, friday I had to leave asap. two nodes of aws were terminated, thats the reason why the mongodb wasn't picking up a master.
[17:50:05] <sulo> i'm kinda new to mongodb and have actually only read the o'reilly book about it... but i'm kind of wondering whether mongodb is also a good fit for small applications with only one db server.. in the book is written that you should always use at least 3 db server because of the way the primary server is selected
[17:50:41] <sulo> so i'm kind of wondering whether it is a good idea to start a example application with just one mongo server...
[17:52:44] <kali> sulo: well, it's more or less the same for any kind of DB. you need backups to recover from a total db server crash with any kind of DB, and at least one spare server if you can't afford to be down
[17:53:40] <Nomikos> well, there's the live and the dev server, and both have backups, but it's not distributed or anything yet. not needed yet.
[17:54:21] <sulo> Nomikos: so ho wdo you currently back up your data? is there something like mysqldump?
[17:57:36] <sulo> in the book they say that if a record grows it gets moved to an other place which can be slow... as i currently see it (and i'm a big noob on that topic ;) ) the whole point of the records is the ability to make them grow... like saving comments to an post in one document... (else i could also use a join in a relational db) .. is there any good practice to prevent such things or make them performant?
[17:58:32] <kali> sulo: on a post/comment model, you would need a HUGE number of comments per seconds for this to become a problem
[17:59:13] <sulo> kali: well in the book they don't give any numbers or something.. they just say that it is slow
[17:59:59] <kali> http://docs.mongodb.org/manual/reference/command/collMod/#usePowerOf2Sizes and that can help
[18:00:45] <sulo> kali: is there something that tells you what is good schema design in mongo db? maybe a good blog post/article
[18:01:44] <kali> start by having a look at this section... http://docs.mongodb.org/manual/data-modeling/
[18:02:32] <sulo> kali: thanks for your help :) .. i think i'll just to have to try it out in an actual application
[18:02:55] <kali> identify the most frequent and critical data access patterns, focus on making them fast and efficient
[18:03:06] <sulo> maybe it's a better idea to think about performance because of document movements when performance issues occur
[18:03:25] <kali> a friend of me says "one http request, one mongo query". i think this is extreme, but...
[18:04:27] <kali> sulo: so moving a few documents around, with a typical web load (99% read, 1% write) is a non issue
[18:05:19] <kali> sulo: if you start logging access on each post inside the document, then growing documents is going to be a problem earlier
[18:06:31] <sulo> ok.. in the book it reads kind of radical .. so i was a bit unsure how big of a problem that realy is
[19:40:48] <clarkk> joannac: in the diagram there are 4 tables. The third is the result of the map function. Why are the keys (holding the array of amounts) the cust_id key?
[19:45:04] <clarkk> joannac: I don't suppose you have any ideas how to achieve this result, from a collection like that stored in the d object here... http://pastebin.ca/raw/2466722
[19:48:23] <clarkk> joannac: d is a collection, as returned by mongoose. Please disregard the fact that it's an object in that code
[19:51:16] <clarkk> it's frustrating - the mongodb people mention pivoting collections, but it's very difficult to find any resources explaining how to do it
[19:52:52] <clarkk> but also need to have access to other fields in the collection - not category, but others
[20:03:51] <kali> i don't know where to start... have you read and understood that ? http://docs.mongodb.org/manual/core/map-reduce/
[20:23:52] <liquid-silence> db.assets.find({ parent: null }, $or: [{ "permissions.userid": 2 }, { "owner": 2 }], { name:1 })
[20:24:12] <liquid-silence> how can I only add the "permissions.userid" :2 if the permissions array exists in the document
[20:46:34] <leonardfactory> I have a question.. I'm using mongo2.4.6 and when I try to sort a query with $geoWithin, it's very slow.. those are the output from .explain(): http://bit.ly/16bWtQ8
[20:48:15] <leonardfactory> It seems it's not using the index at all, even in the $geoWithin query due to the "S2Cursor". However i tried both with a { radius: -1, geometry : '2dsphere' } compound index and a { geometry : '2dsphere', radius : -1 }
[21:51:39] <leonardfactory> https://gist.github.com/leonardfactory/75c677413b02ee67e399 this is the query
[21:52:09] <leonardfactory> Regarding the indexes, I'm using mongoose, building them with: schema.index({ radius : -1, geometry : '2dsphere' });
[21:52:45] <leonardfactory> but nothing happens! Probably I'm doing something really bad here, but I can't see it
[21:54:54] <leonardfactory> yep, I checked it and I have one with "name" : "radius_-1_geometry_2dsphere" currently, checked before and even "geometry_2dsphere_radius_-1" didn't work
[21:56:40] <leonardfactory> https://gist.github.com/leonardfactory/a7fe76c3df063f8bebc7 this is the sample document
[21:57:31] <leonardfactory> cheeser: how can I check the index content? Sorry, I'm pretty new to mongo
[21:59:44] <leonardfactory> https://gist.github.com/leonardfactory/af9ed8eaae36f0575b3f <- this is the real getIndexes output
[22:02:03] <leonardfactory> the index size, from db.areas.stats(), seems reasonable: "radius_-1_geometry_2dsphere" : 41068048
[22:03:14] <leonardfactory> I tried, but the cursor was every single time a `S2Cursor`, no performance boost, nothing
[22:04:23] <leonardfactory> cheeser: I didn't understand if for sorting with a 2dsphere index the sorting field must be placed in the compound index BEFORE the 2dsphere field, however I tried swapping them and it didn't use the index anyway
[22:07:14] <joannac> I tried this yesterday with a 2d index, with the 2d field first, the the non-2d field later
[22:12:51] <leonardfactory> Mhhh joannac cheeser: https://jira.mongodb.org/browse/SERVER-10801 I think this is closing the question, right?
[22:15:20] <leonardfactory> I'm a noob so, can you tell if it is ok to do what I'm trying to achieve (search 2dsphere -> sort results) even having the 2dsphere index first?
[22:38:15] <Gaddel> if i want something like coll.distinct(), but returning values from multiple keys instead of one key, what is the fastest way to do this? multiple "distinct" queries, or a regular find() with projection operator, or aggregation framework?