PMXBOT Log file Viewer

Help | Karma | Search:

#mongodb logs for Friday the 11th of October, 2013

(Back to #mongodb overview) (Back to channel listing) (Animate logs)
[00:12:44] <rbd_> crudson: ok, thanks....so if I'm doing a query (Django ORM) like MyModel.filter(type='bla', when__gte=start, when__lte=end) .... my index could then be [(type, 1), (when, 1)] and I should be fine? instead of [(type, 1), (when, 1), (when, 1)] ??
[02:38:29] <zokko> hello folks
[02:38:48] <zokko> how can I check why replicaset has a lag?
[02:38:59] <zokko> oplog first event time: Wed Sep 11 2013 10:30:02 GMT+0200 (CEST)
[02:38:59] <zokko> oplog last event time: Fri Oct 11 2013 02:04:00 GMT+0200 (CEST)
[02:39:00] <zokko> now: Fri Oct 11 2013 04:35:44 GMT+0200 (CEST)
[02:39:01] <zokko> :/
[04:14:09] <Xzyx987X> hey, I'm a little confused about atomic updates in mongodb, and I'm having trouble finding something that anwsers my exact question. if I wanted to do an update that would update a document's "test" field from the value of "true" to the value of "false", but fail if the value was already "true", how would I do it?
[04:16:18] <jonasschneider> hey Xzyx987X! doing this atomic update is actually pretty easy -- you can just do collection.update({ test: true }, { $set: { test: false } })
[04:16:31] <jonasschneider> note that this will update all documents that have test: true set
[04:16:53] <jonasschneider> to only update a single doc, you'd do collection.update({ test: true, _id: 'my_id' }, { $set: { test: false } })
[04:17:04] <Xzyx987X> ok, so if test is already true when you run that, what happens exactly?
[04:17:18] <jonasschneider> well, the query will just not match the document
[04:17:45] <Xzyx987X> ah, ok, makes sense I think
[04:18:20] <Xzyx987X> I'm thinking about doing this so I can add a collection that contains documents that act as locks
[04:18:52] <Xzyx987X> so I can do larger scale atomic operations
[04:19:49] <Xzyx987X> and I need to make sure I have exclusitivity on a lock, so I need to make sure nothing else locks it while I'm obtaining a lock
[04:19:56] <Xzyx987X> if that makes any sense
[04:20:55] <Xzyx987X> I'm not sure if this is the best way to do it, but it's the only one I could think of
[04:37:06] <jonasschneider> Xzyx987X: yea, that's totally a thing, we do it like that as well
[04:37:21] <jonasschneider> http://docs.mongodb.org/manual/reference/command/findAndModify/ might interest you
[04:37:40] <jonasschneider> the output should be a bit more helpful to tell you if your update succeeded
[04:53:26] <Xzyx987X> one problem with that method though, is that if you want to wait for a lock, you have to spam mongo with queries until you get it
[04:53:33] <Xzyx987X> do you guys have a way around that?
[04:56:49] <jonasschneider> Xzyx987X: we sleep :)
[04:57:08] <jonasschneider> (and poll)
[04:57:35] <jonasschneider> if you're expecting to be waiting for a long time, you can do shenanigans with the mongodb replication log to be sort-of notified when something happens
[04:57:40] <Xzyx987X> right, that's what I would do. but the more competition there is for a lock, the more likely you will waste time between polls
[04:58:22] <Xzyx987X> in practice, what do you guys find the best amount of time to sleep is?
[04:58:43] <Xzyx987X> I was thinking around 5ms to avoid delays as much as possible without overloading mongo
[04:59:25] <jonasschneider> we sleep really long
[04:59:32] <jonasschneider> i thiiink (0.1s) + (0.2s per failed attempt)
[05:00:47] <Xzyx987X> That could lead to long delays depening on how the locks are used, but then again I suppose the idea is to only use one lock whenever possible
[05:01:31] <jonasschneider> yeah, and also we don't have any global (as in: system global) locks, these are per-user
[05:02:02] <jonasschneider> so high lock contention will just lead to a single user timing themselves out
[05:03:05] <Xzyx987X> haven't you guys ever considered building locks into the system? I don't think it would be that much of a stretch to implement
[05:03:32] <jonasschneider> like, into mongo itself?
[05:03:39] <Xzyx987X> it seems like a common sense feature since you so often need exclusive access when updating a set of data
[05:03:55] <Xzyx987X> yea, why not?
[05:04:06] <jonasschneider> i'm not working at mongo, so i don't know (:
[05:04:16] <Xzyx987X> ah, ok then
[05:05:03] <jonasschneider> but i think you'll probably want locks spanning multiple documents at some point, to "fake" transactions
[05:05:42] <jonasschneider> and at that point, there's no real sense in tying a native "lock" object to a single document
[05:06:15] <Xzyx987X> no, but with native locks you'd have enough performance to lock global objects
[05:06:24] <Xzyx987X> and that could be useful in many situations
[05:06:28] <jonasschneider> oh, i see
[05:06:44] <jonasschneider> (use zookeeper already :P)
[05:06:56] <Xzyx987X> what's zookeeper?
[05:07:18] <jonasschneider> http://zookeeper.apache.org/ -- a huge project
[05:07:36] <jonasschneider> among a pile of other things, it can do highly-available lock management
[05:08:02] <jonasschneider> anyway, signing off now -- take care!
[05:08:11] <Xzyx987X> k, thanks for the help
[05:37:31] <ns5> what is a mongodb document? A key-value pair?
[05:38:43] <ron> a set of key/value pairs, where the value can be a primitive, an array or another document.
[05:43:06] <ns5> http://docs.mongodb.org/manual/tutorial/getting-started/ "The operation displays 20 documents."
[05:43:45] <liquid-silence> ns5 care to help me with my data modeling?
[05:44:20] <ns5> liquid-silence: sorry but I'm a newbie
[05:44:25] <liquid-silence> same here
[05:45:57] <joannac> liquid-silence: data modeling for what?
[05:46:28] <liquid-silence> joannac I have to model a directory structure, with permissions
[05:46:44] <liquid-silence> so I was looking at materialized trees
[05:48:18] <liquid-silence> so we have directories and files, users may have acess either to entire folders or to one file in a folder
[05:48:29] <liquid-silence> so I was thinking creating a container per user
[05:48:49] <liquid-silence> and when he gets access to a folder, we link the folder from another users container to that users container
[05:49:16] <liquid-silence> and if he has access to only one asset, he will have the folder linked but on one asset
[05:49:53] <liquid-silence> so when you require to get the user's visibile items, you just do db.containers.find({ "user_id", "my user id")
[05:50:07] <liquid-silence> and that will give all the assets / folders that the user can see
[05:50:10] <liquid-silence> and you can drill down
[05:50:18] <liquid-silence> to get only assets in a folder
[05:51:04] <liquid-silence> so user a creates a folder in his container
[05:51:13] <liquid-silence> he then gives user b access to that folder
[05:52:00] <joannac> Back up. What's a "container"?
[05:52:08] <liquid-silence> container is the top level
[05:52:15] <liquid-silence> each tenant gets one
[05:52:20] <liquid-silence> we have many users
[05:52:29] <liquid-silence> the containers never get linked
[05:52:40] <liquid-silence> container is like the users home dir
[05:53:15] <liquid-silence> and if he gets permissions to something that another user gives him, we "ln -s" it to his container ( home dir)
[05:57:35] <liquid-silence> joannac make sense at all?
[05:58:51] <liquid-silence> I am confused as to how, as I come from a sql background for the last 10 years
[06:50:09] <ns5> how to use batch mode with mongo client? I need to use a db and drop it in a shell script
[06:50:29] <ns5> something like: use mydb; db.dropDatabase()
[06:50:57] <liquid-silence> joannac no adivce?
[07:25:29] <liquid-silence> joannac this is what I have now
[07:25:29] <liquid-silence> http://pastebin.com/KeWQv0Be
[07:25:47] <liquid-silence> user 123 needs access to the test folder
[07:26:06] <liquid-silence> so I could sym link it
[08:46:53] <joannac> liquid-silence: so you're trying to store user permissions in your database, right?
[08:47:18] <joannac> is there anylimits to how many folders a user has?
[08:47:33] <joannac> are folders nested?
[09:28:03] <bmcgee> hey guys I'm seeing some strange behaviour with capped collections. Specifically I had a test which started a tailable cursor, then inserted some data. My cursor was never receiving anything. However if I insert a single document first, start my cursor, and then proceed to do more inserts the cursor gets the remaining inserts
[09:28:24] <bmcgee> this is with a new collection created from scratch
[09:28:41] <bmcgee> seems the tailable cursor fails to pick up inserts unless there is at least one document already in the collection…?
[09:29:51] <bmcgee> just found this blog post which describes the same problem. Seems mongo thinks the cursor is dead
[09:31:08] <bmcgee> seems the simplest solution is to create a dummy doc to be inserted when the collection is created
[09:36:23] <bmcgee> well that was a really annoying problem...
[10:47:21] <liquid-silence> joannac yes
[12:33:06] <birdy111> Hi Mongo Community.. Is there a socket timeout between mongos and mongod?
[12:35:16] <birdy111> For ex: if on mongod a command takes more that X seconds, MongoS timeout, While mongod keeps on processing that job?
[12:45:31] <birdy111> I have seen mongos timing out in 30 seconds on mongod connection
[12:48:41] <birdy111> Can I configure this timeout on MongoS?
[13:12:11] <CIDIC> how is gridfs comparable to regular file system performace wise?
[13:21:15] <Number6> CIDIC: GridFS sits on top of a normal filesystem. It's not a replacement for, say, ext4
[13:23:20] <Number6> CIDIC: http://www.markus-gattol.name/ws/mongodb.html#why_use_gridfs_over_ordinary_filesystem_storage is a good overview
[13:25:28] <CIDIC> Number6: in a php web app would it be viable to store content files this way insteal of on the servers file system with a file path stored in the db?
[13:34:05] <Number6> CIDIC: It depends - how large are the files, and how many of them are there?
[13:34:46] <CIDIC> Number6: usually just images like the content of pages
[13:36:23] <Number6> User generated (like Flickr), or site images (like a banner, icons, etc) ?
[13:36:39] <CIDIC> Number6: site images
[13:36:42] <CIDIC> like for a cms
[13:37:26] <Number6> Ordinary FS.
[13:38:27] <Number6> With the relative path stored in the DB
[15:21:08] <arthurnn> wondering what is the best approach do sum 2 values from a doc, and update itself(everything within a atomic op)
[15:22:03] <arthurnn> something like this: doc = {foo:1, bar:2, t: 'A'} , w need to push a new field call baz to this document which it should be the sum of foo + bar .
[15:34:06] <arthurnn> nvm: http://docs.mongodb.org/manual/tutorial/isolate-sequence-of-operations/#update-if-current
[15:44:31] <quattr8> sort() is case sensitive? :O
[15:45:29] <ron> makes sense
[15:46:42] <quattr8> why?
[15:47:15] <ron> because outside of Oracle, everything is case sensitive?
[15:49:40] <counterwall> I'm trying to write into MongoDB an array with the Java driver. Is this correct? http://dpaste.com/1413391/
[15:50:51] <ron> Champi
[15:50:53] <ron> er
[15:50:55] <ron> cheeser
[15:55:04] <counterwall> *list
[15:55:42] <ron> counterwall: well, when you try... what happens?
[15:55:54] <counterwall> ron, it writes a null.
[15:56:09] <counterwall> it doesn't even add that field.
[15:56:13] <ron> counterwall: then I guess the answer is - 'no'. :)
[15:56:40] <ron> what's s?
[15:56:41] <counterwall> Well, I'm looking for guidance into what I'm doing wrong.
[15:56:47] <counterwall> s is a DBOBject.
[15:57:29] <ron> okay, this I'll be all ##java and such, but dude, show a full relevant set of code, not just a tiny snippet that's unclear.
[15:59:50] <counterwall> http://dpaste.com/1413400/ ron
[16:00:26] <ron> well, obviously, that code will do absolutely nothing to mongodb.
[16:04:16] <jiffe99> so with the cxx driver and using scons, it builds how do I go about installing the driver?
[16:06:16] <ron> counterwall: do you understand why?
[16:08:10] <jiffe99> doing an `scons install` I get 'scons: `install' is up to date.' nothing gets installed though
[16:34:15] <counterwall> When I update mongodb, I provide a DBOBject for the query and the DBObject for what to change. For the DBObject that says what to change,can I specify just the fields I want to change or do I have to have all the fields that the document should have?
[17:00:37] <preyalone> Does mongo support user defined atomic operations?
[17:38:44] <kurtis> Hey guys, is it possible to ignore certain 'keys' when reducing data in M-R?
[17:39:02] <kurtis> I've tried to just not emit from my reduce() function but it still creates data with a 'null' value
[17:40:27] <kurtis> I could probably handle this in my mapper() function for some use-cases, but not all of them.
[17:44:34] <airandfingers> is there a good way to dynamically configure MongoDB, other than appending lines to /etc/mongodb.conf and restarting the service?
[17:47:07] <kamal_> I'm trying to set up a replica set, the user I'm authed as has the clusterAdmin role on the admin database but rs.add() returns { "ok" : 0, "errmsg" : "unauthorized" }
[17:47:25] <kamal_> Not sure what's going on..
[17:56:15] <astropirate> Hello friends
[17:56:28] <astropirate> Hello friends "relationship" collection?
[17:56:35] <astropirate> wow screw you xchat
[17:56:38] <astropirate> let me say that again
[17:56:55] <astropirate> is it kosher to have a "relationship" collection
[17:57:08] <astropirate> with documents that describe the relationship of N->N relationships?
[17:57:11] <Derick> for modelling n:m?
[17:57:15] <Derick> no... not really
[17:57:27] <Derick> store the relation on one of the other collections
[18:00:07] <patrickod> is there a nice way to use fail2ban with mongodb?
[18:07:13] <kurtis> http://stackoverflow.com/questions/19324555/how-do-i-not-save-data-in-my-reduce-function-in-mongodb
[18:07:31] <kurtis> I've got to go pick up my son from school. If anyone has any ideas, I'd really apprecaite it. Thanks!
[18:52:41] <jiffe99> so with the cxx driver mongodb-linux-x86_64-v2.4-latest.tgz and using scons, it builds but how do I go about installing the driver? doing an `scons install` I get 'scons: `install' is up to date.' nothing gets installed though
[19:22:49] <mrpoundsign> hello everyone. :) Having a problem finding an elegant solution for finding documents that either haven't expired, or expire in the future.
[19:23:18] <mrpoundsign> something like db.foo.find({expires: {$in: [{$gte: 1}]}})
[19:23:37] <Luksor> Hello, I need help to apply this fix in database/mongodb http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/182110
[19:23:39] <mrpoundsign> we do a sort on the query so it's pretty slow with $or
[20:53:52] <rafaelhbarros> I have a 3 node replica and I want to restart the master
[20:54:00] <rafaelhbarros> how to do that gracefully?
[20:57:26] <joannac> stepDown the primary, wait for a new node to be elected, then shut down the ex-primary
[21:07:12] <rafaelhbarros> joannac: thanks!
[21:43:31] <ench_> Using Gentoo, mongodb refuses to start. Whenever I try, it just says "* WARNING: mongodb is already starting" and never actually starts. Anyone have any suggestions?
[21:58:31] <crudson> ench_: you're not running off a slow disk (usb or network) are you?
[22:04:42] <ench_> crudson: No, running on a hard drive