PMXBOT Log file Viewer

Help | Karma | Search:

#mongodb logs for Friday the 2nd of May, 2014

(Back to #mongodb overview) (Back to channel listing) (Animate logs)
[00:25:15] <acidjazz> hi all.. is it possible to get a count of a field thats an object/array ?
[04:41:28] <sahanh> anybody around?
[05:35:14] <joannac> sahanh: sure
[05:35:37] <joannac> and if you had asked your question in the first place, you might have an answer by now :)
[05:35:59] <sahanh> what's the best (fastest query time) way to aggregate large set of data,
[05:36:18] <sahanh> 1. Pipeline, 2. Add indexes and then use usual querying ?
[05:36:48] <sahanh> I have around 1.5 mil record set now and, group by queries are running slow
[05:36:54] <sahanh> running on indexes
[05:36:54] <joannac> What do you mean by "aggregate"?
[05:37:30] <sahanh> calculations,
[05:37:36] <sahanh> mainly counting documents
[05:37:52] <joannac> why don't you just use find(...).count()?
[05:38:20] <joannac> I think that would be faster than using aggregation framework
[05:38:51] <sahanh> thought the same, wasn't sure if it's the best approach, since I have to define several indexes just for the count part
[05:39:08] <joannac> and you don't for the AF?
[05:41:04] <sahanh> do I have to ?
[05:41:15] <sahanh> I'm coming from couch, so still learning
[05:41:17] <sahanh> :)
[05:42:35] <joannac> well, no, you don't have to. But indexes will get you better performance than collection scans
[05:43:08] <sahanh> so with indexes defined, AF is the way to go?
[08:06:59] <BaNzounet> Hey guys, if I've a doc multiple keys and I run an update with only one key in the update section will it override other field or will it just update the matching field?
[08:07:41] <BaNzounet> by override I mean delete other keys (fields)
[08:39:35] <kali> BaNzounet: you neet to look up "$set"
[08:53:48] <ahawkins_> hey everyone. I have an array of strings, I need to query documents where an entry matches a regex. I used $in with $regex before but newer mongo versions are complaining about that. Any other way to construct this query?
[08:55:19] <kali> ahawkins_: can't you build the "or" in the regexp ? (regexp1|regexp2|regexp3|...)
[08:56:12] <ahawkins_> kali: there is nothing to or against. I don't know what's in the array.
[08:56:25] <rspijker> ahawkins_: what is it you are trying to do exactly?
[08:57:45] <ahawkins_> rspijker: given a document with an property "things" [foo, bar, baz], find all documents where a member of "things" matches /b/
[08:58:15] <rspijker> ahawkins_: {things:/b/}
[08:58:45] <kali> arf
[08:59:16] <ahawkins_> rspijker: seems to work :)
[08:59:28] <rspijker> yeah, $in is the reverse
[08:59:35] <kali> ahawkins_: sorry, i misunderstood the question :)
[09:00:01] <ollivera> Hi, is there any way to check the last time my collection has been updated? I tried db.getLastErrorObj()
[09:00:19] <rspijker> say "things" is a regular string, either "foor", "bar", or "baz". Then you can use $in:["foo", "bar", "baz"] to find any document that has things either one of those values
[09:00:34] <kali> ollivera: not trivially (you can look at the oplog if you have ha replica set, but it's a bit hackish)
[09:00:51] <rspijker> also, your oplog might not go back that far
[09:02:04] <rspijker> if this is something you want to do routinely, you need to keep track of it yourself ollivera
[09:02:51] <ollivera> okay, thank you
[09:02:57] <rspijker> you could potentialy check insertions, but only if you use ObjectIds
[09:03:06] <ahawkins_> thanks guys!
[09:03:09] <rspijker> and it wouldn't work for other modifications
[10:55:08] <BaNzounet> Is there anythings wrong with this ` db.bars.update({installedOnce: true}, {$set: {installedOne: false}}, false, false);`? Cause it doesn't update my doc
[10:56:59] <Nodex> db.bars.findOne({installedOnce: true});
[10:57:03] <Nodex> what does that give you
[10:57:26] <Nodex> I can see a typo
[10:57:44] <Nodex> {installedOne: false} != installedOnce (missing a "c")
[10:59:07] <BaNzounet> Nodex: thanks dude
[10:59:10] <BaNzounet> that was it
[11:00:12] <BaNzounet> btw the first false is the multi param right ?
[11:01:34] <Nodex> you can use an object for ase of use
[11:01:40] <Nodex> {multi:true, upsert:true}
[11:01:44] <Nodex> ease*
[11:34:04] <oskie> hello, it seems the minimum size of an empty collection is 8MB, is there anything you can do about that?
[11:36:17] <rspijker> oskie: noprealloc might help a bit
[11:36:27] <rspijker> but I thought the inital file was 16M even...
[11:36:52] <Derick> rspijker: db vs collection
[11:37:50] <rspijker> aaah, does smallfiles impact it then?
[11:38:17] <rspijker> all I can recall is that it limits the max size to 512, don't know if it also makes inital files smaller
[11:39:55] <oskie> rspijker: I don't think the files are the issue - I have like 100 databases, each with 16 or so collections with "size" : 0 and "storageSize" : 8192...
[11:40:51] <oskie> but I'll try noprealloc
[11:41:44] <oskie> I should also try smallfiles
[11:42:56] <rspijker> I can't find any documentation that explains what those settings do *exactly*, only somewhat vaguely
[11:43:26] <rspijker> I remember there being a video seminar that went into mongo storage mecahnisms into a little more depth, but that was a while ago
[12:06:12] <kali> smallfiles reduce the size of the preallocation chunks from 2GB to 128M iirc. it's a must for integration tests
[12:06:56] <kali> noprealloc is good when running on filesystem that do not support preallocation (can be usefull for tests, too)
[12:07:12] <kali> but i would not consider these two tweaks to be suitable for production environment
[12:18:03] <oskie> i thought i read somewhere that preallocated files are NOT sparse
[13:14:09] <Industrial> Hi. I have several collections that represent time based data. Right now they all have one property 't' which is a Date object (so down to the millisecond).
[13:14:35] <Industrial> Is it possible to merge several collections (with the same format) based on one property, also sorted by that property?
[13:19:39] <kali> Industrial: you may be able to do it with multiple map reduce
[14:18:03] <mac_cosmo> hey! i’m new to mongodb and can’t get my head around basics things … let’s assume, that i have an old projekt with a psql/mysql db, like this: http://pastie.org/9133802 what would be a good design when i have to redo this in a mongo db?
[14:22:17] <mac_cosmo> (in a real world example an user would have ~20 memberships)
[14:26:36] <q851> mac_cosmo: https://gist.github.com/qjshanley/f21854c0989e5744257f
[14:27:38] <Nodex> proably better in a "parties" collection
[14:28:33] <q851> Nodex: depends on what you are going to do with it.
[14:29:11] <Nodex> not really
[14:29:34] <Nodex> hte primary focus seems to be surrounding "parties"
[14:29:37] <Nodex> the*
[14:29:59] <q851> ah, I see. You are correct
[14:30:01] <mac_cosmo> it’s something like messages in facebook. an user can create a “message group” and invite other users.
[14:30:52] <Nodex> I would have the group and the members invited etc as a sub document
[14:30:55] <mac_cosmo> but in my case, the invited person has to accept the invitation
[14:30:56] <q851> So a user owns a party?
[14:31:40] <mac_cosmo> one user is the creator, but there can be more party owners
[14:32:31] <mac_cosmo> a user should be able to query the parties he joined, and a party should be able to query all of it’s users
[14:32:32] <q851> A relational db seems more appropriate, imho.
[14:32:35] <mac_cosmo> *an user
[14:33:37] <mac_cosmo> yeah but all the other collections/tables are better suited in mongodb :D
[14:34:03] <Nodex> you can do both queries you wish with one collection
[14:35:42] <Nodex> parties : -> ... { _id : ...., oid : ...., moderators: [], users : [{id:1,name:"bar"....}] }
[14:36:47] <mac_cosmo> oh ok that looks good
[14:37:16] <Nodex> you get a free index on _id so that takes care of looking up parties
[14:37:33] <Nodex> you can get to all the users in the users:[] array
[14:38:05] <Nodex> and drop an index on users.id to be able to query all parties a certain id belongs to
[14:38:27] <mac_cosmo> wow, thanks! i’m going to implement this right now :D
[14:38:50] <Nodex> BUT... there is a 16mb document limit so you will need to rethink this when your app gorws
[14:38:53] <Nodex> grows*
[14:39:53] <mac_cosmo> 16 mb should be enough per party ;)
[14:41:15] <artimuscf> hi, im trying to write a .js script to clone a database. it works fine, but I would like to add some logic so I know if it was successful or not. I currently use var out = db.cloneDatabase('somehost'), but out is an object that im not sure how to parse
[14:41:19] <Nodex> in the future you can always chunk it
[14:42:24] <artimuscf> so basically i just do db.dropDatabase(), then db.cloneDatabase(). i just want to add some error handling. can anyone point me in the right direction
[14:47:43] <emr> Hello how i can find object which have array list and contains item? its something like { "_id" : "37a6259cc0c1dae299a7866489dff0bd", "fruits" : [ "apple", "banana" ] }
[14:48:14] <emr> i want to find all items which apple item in fruits
[14:48:17] <rspijker> you can just act as if fruits is a simple field containing a string you want to match against emr
[14:48:34] <rspijker> so just: {"fruits":"apple"}
[14:48:56] <emr> cool thank rspijker
[14:51:26] <rspijker> yw
[15:12:57] <asturel_> if i made accidently a typo in my field name can i someshow change them to something else?
[15:21:07] <whaley> asturel_: http://docs.mongodb.org/manual/reference/operator/update/rename/
[15:21:31] <asturel_> i found it
[15:21:37] <asturel_> hmm i typed $rename to somewhere else :D
[15:31:40] <artimuscf> please, can anyone help me determine if db.cloneDatabase() was successful in javascript
[15:43:39] <ekristen> good morning
[15:43:48] <ekristen> I’m looking at one of my currentOps on a replica
[15:43:57] <ekristen> and a query has been running for 98 minutes for some reason
[15:44:14] <ekristen> I’m afraid this isn’t the first time this has happened — trying to figure out why
[15:48:15] <ekristen> is it possible for an index to be unhealthy?
[15:54:05] <jbrechtel> I've searched a bit and haven't found anything....has anyone seen anything done around rollback strategies for failed document schema migrations when using a replicaset?
[15:55:46] <jbrechtel> I'm looking for ideas around what to do (either automate or plan to do) if we do a document schema migration and decide it has failed and we want to rollback. We've considered taking a member out of the replicaset and subsequently building a new replicaset around that in the event of a failure. This requires two application configuration updates though. Looking for possibly simpler approaches.
[16:25:10] <ekristen> I’m having what appears to be issues with one of my replica’s secondaries, certain queries seem to get stuck and will run for hours
[16:25:15] <ekristen> when they should return almost immediately
[16:25:30] <ekristen> I’m having to kill off the operations so they don’t overwhelm the server
[16:25:50] <ekristen> is there anyway to troubleshoot why this is happening? it doesn’t seem to be happening on my primary and/or my other secondary
[16:32:19] <magglass2> ekristen: have you checked to make sure any indexes required by the query are in place? compare the indexes to those on your master
[16:32:39] <ekristen> just the getIndexes() yes?
[16:32:47] <magglass2> yeah
[16:33:20] <ekristen> yeah they are identical
[16:34:10] <ekristen> is it possible for indexes to get corrupt?
[16:43:49] <dfinch> I'm using a third party component which creates tons of collections implicitly, I want to ensure these are configured correctly and indexes are added when they are created. my plan is to edit the library's calls to update/etc. to insert some step to do this. this will presumably have to be done on every write operation, so what's the best way to do this so I don't seriously affect performance of the queries
[16:43:50] <dfinch> themselves, and avoid multiple round trips between client and server?
[16:46:58] <magglass2> ekristen: can you run the query with a .explain() and compare it to the master?
[16:50:35] <ekristen> magglass2: no
[16:52:51] <magglass2> ekristen: takes too long to run or what?
[16:53:09] <ekristen> really don’t want to risk killing my production servers again
[17:21:17] <unholycrab> Is it possible to create multiple groups within MongoDB MMS?
[17:21:36] <unholycrab> I want to monitor multiple clusters that cannot reach eachother, and so cannot be monitored from a single mongodb-mms-monitoring-agent
[17:39:18] <daidoji> hello, how do I know if I'm running into memory limits or whatever with MapReduce in Mongo?
[17:39:37] <daidoji> alternatively, here's some output explaining an issue I'm having that I can't seem to figure out https://gist.github.com/daidoji/1ef55617e69553afc4f8
[17:39:49] <daidoji> any help would be greatly appreciated
[17:46:07] <daidoji> also note, that second bit of output the count is correct, there are three documents with that type not two as in the first set of output
[17:58:39] <daidoji> anybody?
[18:04:48] <pure> So, considering a sharding use case whereby the users of the service can host either shards themselves or redundancy shards. Would there be a nice way to make sure the sharded databases are somehow encrypted from the host but still able to be used by the rest of the DB network?
[18:14:30] <pure> On another note: is having redundancy clusters a good idea for sharded systems?
[18:15:26] <magglass2> pure: not sure what you're asking; typically each shard would be a three-node replica set
[18:15:32] <cheeser> well, every shard should have its own replica set...
[18:16:17] <pure> I mean, each shard in the database should have its own redundancy databases thingy?
[18:28:44] <magglass2> pure: o_O
[18:29:01] <magglass2> pure: each shard should have at least one replica, but 2 is recommended
[18:29:12] <bemu> I'm trying to remove an arbitrator from a version 2.0 replica set such that:
[18:29:15] <bemu> PRIMARY> cfg.members.splice(4,1)
[18:29:16] <bemu> [ ]
[18:29:42] <bemu> I would expect to see some kind of output, perhaps the remnants of cfg, but no dice. Am I doing something wrong?
[18:30:29] <daidoji> aren't you getting output?
[18:31:04] <bemu> I'm getting: [ ] as outpu
[18:31:13] <bemu> but no remnants of my cfg, which is what I would expect.
[18:31:15] <daidoji> yeah, emtpy set
[18:31:31] <bemu> empty set is to be expected, even if cfg still has contents?
[18:31:35] <daidoji> what does your document look like
[18:32:09] <bemu> 2 shards and an arbitrator.
[18:32:19] <bemu> I'm trying to remove the arg and replace with a real shard.
[18:33:35] <bemu> Literally none of the docs I've found show what output to expect from splice.
[18:34:08] <daidoji> what does cfg.members.find({}) display?
[18:34:41] <bemu> cfg.members.find is not a function
[18:35:05] <bemu> specifically:
[18:35:05] <bemu> Fri May 2 18:33:49 TypeError: cfg.members.find is not a function (shell):1
[18:36:05] <daidoji> bemu: where does cfg come from? I've never done sharding, but empty [] is acceptable output for splice when there's nothing at that index
[18:37:08] <bemu> cfg is a full array of information, including members.
[18:37:23] <daidoji> also, in case anyone could help with my issue above, its now also a Google Groups post for mongodb-users https://groups.google.com/forum/#!topic/mongodb-user/C68gT2kn7O8
[18:37:47] <bemu> each member has an _id, host, etc.
[18:39:23] <mbeacom> Hey guys, have a quick question.. well, hopefully quick.
[18:39:39] <mbeacom> I have a collection structured like this
[18:39:54] <daidoji> bemu: isn't splice a javascript function anyways? I don't think its a Mongo specific thing
[18:39:54] <q851> daidoji: are you getting my messages?
[18:40:01] <daidoji> q851: oops sorry
[18:40:40] <bemu> daidoji: Sure, but it's all over the mongo docs.
[18:40:47] <mbeacom> {'id': 1, 'another_key': 2, 'messages': [{'text':'hey', 'timestamp': 1397671947403}, {'text':'hey', 'timestamp': 1397683296710}]}
[18:41:16] <mbeacom> I want to filter my collection where the child messages timestamp is less than a timestamp i provide
[18:42:25] <mbeacom> I have db.messages.find_one({'id': id, 'messages.timestamp': {'$lt': 1397683296710}})
[18:42:37] <mbeacom> But it always returns the full list of messages inside :/
[18:44:06] <xissburg> how about db.messages.find_one({'id': id, 'timestamp': {'$lt': 1397683296710}})
[18:44:33] <mbeacom> xissburg: timestamp is inside the 'messages' array. not outside
[18:45:00] <mbeacom> Let's call every item in db.messages collection a "thread"
[18:45:02] <xissburg> ah heh
[18:45:06] <mbeacom> haha
[18:45:11] <mbeacom> Yeah. D:
[18:45:20] <daidoji> bemu: what does cfg.members.constructor return?
[18:46:09] <bemu> daidoji: function Array() {
[18:46:10] <bemu> [native code]
[18:46:11] <bemu> }
[18:46:27] <daidoji> what does cfg.members.splice(4) return?
[18:46:45] <bemu> cfg.members.splice(4)
[18:46:45] <bemu> [ ]
[18:46:51] <daidoji> what does cfg.members.splice(0) return?
[18:47:23] <bemu> cfg.members.splice(0)
[18:47:24] <bemu> [ { "_id" : 3, "host" : "8.8.8.8:27018", "priority" : 2 } ]
[18:47:33] <bemu> where 8.8.8.8 is a legit ip
[18:47:38] <bemu> oh
[18:47:45] <bemu> sec
[18:47:50] <daidoji> so there's nothing at [4] in that array
[18:48:01] <daidoji> https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/splice here's the docs on that function though
[18:48:13] <daidoji> or a doc rather
[18:49:13] <bemu> Thanks a lot!
[18:49:56] <mbeacom> Any help with my issue is greatly appreciated :)
[18:50:20] <daidoji> bemu: np
[18:51:09] <daidoji> Mongo docs assume you're familiar with Javascript though in case you run into other things that don't seem to have documentation in Mongodocs proper
[19:42:30] <mbeacom> Can you limit subdocuments in a aggregation?
[19:43:13] <mbeacom> https://gist.github.com/mbeacom/2662bb0c520abb0d2be8
[19:43:22] <mbeacom> That is my current aggregation
[19:43:40] <mbeacom> I want to limit the "messages" subdocument to 2 at a time
[19:44:43] <q851> mbeacom: I only know of http://docs.mongodb.org/manual/reference/operator/query/elemMatch/ to limit subdocuments.
[19:51:41] <mbeacom> q851: Ah, thanks. I don't think that is what I'm looking for though :(
[19:52:49] <q851> Why do you need to limit it to two at a time?
[19:53:01] <mbeacom> q851: To limit the subdocuments it returns, would you just let it return what it is going to return and then programmatically add htem to an array
[19:53:14] <mbeacom> I have threads.. some could have 1000 messages in them
[19:53:23] <mbeacom> I want to grab 15 before a certain timestmap
[19:53:45] <mbeacom> q851: so I can make a call to the server, grab 15 before a timestamp. then if the user scrolls up more, grab another 15 before a timestamp
[19:53:50] <mbeacom> etc.
[19:54:06] <q851> ah
[22:22:44] <frodo_baggins> Hmm... I have an interesting problem to solve.
[22:22:48] <frodo_baggins> Interesting for me anyway. :D
[22:40:50] <frodo_baggins> Which brings me to my question... is there a way to have timed tasks occur within mongo, without using crontab?
[23:17:58] <frodo_baggins> Hmm... crontab scripts, or timed mongodb scripts?
[23:19:22] <frodo_baggins> Hmm, I just thought of a better idea. :D
[23:20:03] <frodo_baggins> But, that brings me to another question...
[23:20:42] <frodo_baggins> Should I create a collection called application, and then a document for settings? Or should I add a collection under system called application_settings?
[23:21:00] <frodo_baggins> Or something along those lines.
[23:21:37] <frodo_baggins> Okay, so, documentation is telling me not to create collections with the system.* prefix.
[23:22:10] <frodo_baggins> So, I guess I'll create a collection called application. :D
[23:30:46] <frodo_baggins> Another question... can $in be used with findAndModify?
[23:31:08] <frodo_baggins> I want to use findAndModify to query for the presence of an element within an Array on one document.
[23:31:18] <frodo_baggins> And if it's not there, add the element.
[23:31:28] <frodo_baggins> Or, $push, rather.
[23:32:12] <frodo_baggins> Ah, I see it is $addToSet I'm looking for.
[23:49:02] <frodo_baggins> Is there a way to tell if an update operation ends up inserting a new document?
[23:50:04] <frodo_baggins> I would like to be able to call a method, or at least specify a conditional $addToSet on the event a new document is inserted.
[23:50:56] <ranman> frodo_baggins: maybe findAndModify with upsert is what you want?
[23:51:10] <frodo_baggins> I am using a findAndModify call.
[23:51:26] <frodo_baggins> And specifying an update parameter.
[23:51:51] <ranman> frodo_baggins: so the callback has the updated document
[23:52:12] <frodo_baggins> Hmm, okay.
[23:52:24] <ranman> function(err, object)
[23:52:27] <frodo_baggins> Is there a way to tell from the result whether an insert or an update was performed?
[23:53:10] <frodo_baggins> Ah, I see it's WriteResult I'm looking for.
[23:54:10] <frodo_baggins> So, the docs say update() returns a WriteResult object.
[23:55:09] <frodo_baggins> I'm assuming findAndModify behaves the same.
[23:56:03] <frodo_baggins> And by "returns" I'm hoping by passing arguments to the callback it receives.
[23:57:18] <frodo_baggins> I'm using the node-mongodb-native driver, and I don't see any mention of the WriteResult object.
[23:57:24] <frodo_baggins> In the driver documentation.