Log file Viewer
#mongodb logs for Tuesday the 26th of November, 2013
[01:43:37] <Antiarc> How can I $project an array value to a field (or otherwise reference a specific array element in the aggregation framework)?
[01:45:11] <Antiarc> I have an array "values" which contains arrays which are pairs of [timestamp, value]. I'm unwinding values, then I want to sort the unwound set on values[0][0], but $sort: {"v.0.0": 1} doesn't seem to do anything useful
[01:45:37] <Antiarc> So I thought maybe I could $project the fields into date and value fields, but I can't project $value.0 either - it ends up projecting an empty array
[01:47:34] <Antiarc> That is, after unwinding, I end up with something like this: https://gist.github.com/cheald/8c3408b2d609424b9125 (though it's several thousand documents)
[01:48:12] <Antiarc> I want to sort on v[0][0] there, but I can neither project v[0][0] to another field, or $sort on it.
[01:51:05] <Antiarc> https://gist.github.com/cheald/8c3408b2d609424b9125 -- that's effectively what I'm seeing. The first bit there is the result of just the unwind, then the second bit is what I get once I attempt to project from the array.
[01:51:31] <Antiarc> I could switch to using subdocuments here, but since it's always a 2-member array, the array seemed like it'd be more efficient. No real need for field names here.
[01:57:10] <dangayle> Does anyone know the #irc channel for the M101P MongoDB course that starts today?
[02:11:40] <richo> I've got some collections I want to convert to capped. I don't really care about data loss. If I replicate into a slave, then try to cap it on the slave, is that going to upset the master?
[02:12:10] <Antiarc> I may be wrong but I don't think mongo's going to let a slave modify a collection
[02:21:26] <Antiarc> richo: I don't think you're going to be able to do that from a slave - mongo will just refuse to execute it on a slave. Maybe just mapreduce your origin data into your target collection?
[02:22:00] <richo> Antiarc: that sounds massively unpleasant. plus the whole issue is that the master is basically out of space
[02:22:29] <Antiarc> Ah, yeah. If the concern isn't the write lock then I'd just rs.stepDown the master and convert it on your slave
[02:22:30] <richo> which would give me the required breathing room, but capping a collection without write locking the db doens't seem like it should have that many operational constraints
[02:23:59] <Antiarc> If fragmentation is significant you could regain some breathing room on the master by stepping it down and then just stop mongo, blow away the data dir, start it back up, and let it resync from the replica set
[02:24:12] <Antiarc> Basically performing a repair except without the need to have the extra disk space.
[02:24:57] <joannac> Antiarc: that means putting replication load on your old secondary / new primary
[02:25:23] <Antiarc> It does. But if you're just flat out of disk space on your old primary, you can't repair/compact
[02:25:38] <richo> is the behaviour of trying to convert from master-slave to a replset even defined?
[02:26:34] <richo> but then I discovered that some of our genius devs put data they actually care about into mongo
[02:26:35] <dangayle> joannac: I know we (the students) created an ad-hoc channel the first time the M101 course was offered last year.
[02:27:07] <richo> I just want to do some blocking operations without actually writelocking the whole isntance
[02:27:27] <joannac> richo: http://docs.mongodb.org/manual/core/capped-collections/#convert-a-collection-to-capped
[02:27:52] <Antiarc> replsets are a superset of master-slave - it's basically the recommended way to do master-slave setups now. You aren't going to get away from the write lock easily, though - that tends to be a big problem for this kind of thing, in my experience
[02:28:18] <joannac> you could probably do that when the secondary is in standalone mode, and then when you put it back it'll keep replicating, but I'm not 100% on that
[02:29:02] <richo> I might just see if our dataset is append only on the other tables and just replicate the missing data by hand after I lock the slave for a while
[02:29:43] <richo> I saw the only config option but the docs don't really mention how/if it works with several db's or collections
[02:30:08] <richo> because that'd be easiest, I'll cap the collections I don't care about, then replicate again to pull in the new data in the other collections, then just promote and blwow away the old instance
[02:34:00] <Antiarc> You might want to spin up a miniature deployment to test behavior when you have a replication scenario with a collection capped on one node and not on the other
[02:34:34] <richo> yeah, I've been trying to avoid that if I can but it looks like that's what I'm doing
[02:35:06] <Antiarc> Something else to consider is that based on how long your secondary will be in standalone mode, make sure your primary's oplog size is large enough to hold all transactions during the time it's offline
[02:35:19] <Antiarc> else you'll have to do a full replication from master, which would undo your capping work
[02:36:46] <Antiarc> If this is just trying to get some breathing room, you could run a manual delete to drop any documents that'd be dropped by the capping anyhow. Quick fix to get you some space to work.
[02:37:37] <Antiarc> I suspect that may make the capping faster, too, but I'm not 100% on the internals of how capped conversions happen when you have more data than the collection will allow
[02:42:47] <richo> the docs suggest that to reclaim space from delorted documents you need to run a repair
[02:54:21] <Antiarc> If you're using BSON ObjectIds, they're orderable by insert time, since the most significant bytes of the ID is the date the ID was created. So you can just sort by ID desc, find the nth+1 document where n is the capped collection size, and delete where _id: {$lte: whateverID}
[02:54:38] <Antiarc> That won't free up disk space, but it'll add usable space for that collection, and will mean less data to move around
[02:57:16] <richo> nfi, it's the mount of disk that lib/mongo is consuming with it's .n files on disk
[02:57:35] <Antiarc> okay, so that's fileSize -- you may be using significantly less actual data. check db.stats() from a console
[02:57:49] <richo> I'm not well versed in mongo, we've always been pretty clear about not putting data we care about into it
[02:58:45] <richo> but when that failed to help, I just found some other crap on the same partition to nuke in the meantime. The on disk footprint is growing again, so I'd suggest it's eaten up all that reclaimed space now
[03:02:24] <Antiarc> Well, if you can get your devs on board, you could have them start writing to a different capped collection so that data moving forward is capped, then you could just drop the collections in question once they would have been rolled out of the capped collection, but that may not be viable without more disk space
[03:03:16] <richo> It's pretty trivial to switch collections, but doing it in a coordinated fashion across all the things that want to poke at them won't be that much fun
[03:04:32] <richo> I did toy with just sticking it on a 8TB EBS volume and planning to have died of something before it became a problem again though
[03:04:36] <Antiarc> Well, kinda, it's a time-constrained capped collection rather than recordcount-constrained (and doesn't have the write profile of a capped collection obviously)
[03:06:53] <Antiarc> since replication is designed to permit failover, which would necessitate that both nodes have a complete view of the data
[06:58:40] <floatingpoint> how would I go about using the findOne() method in the context of nodeJS?
[07:01:03] <floatingpoint> I don't know if that was directed at me, but I can't get findOne() to work in the context of a nodeJS route
[07:06:33] <floatingpoint> the following code should return a single document containing a user's password. however, NodeJS is crashing hard when I run the function from a route. Yes, I've cheked the DB connection. Yes, I've checked the input's validity. the findOne() function is the problem. http://pastebin.com/YbusXb4A
[07:43:54] <floatingpoint> it looks like I am getting the data, but I am having problems rendering it into a nodejs route
[08:02:29] <yati> Hi. This is about mongoengine. I have a Document called User and when I call disconnect() and then connect('some-new-db'), I expect User.objects to be empty. But that is not the case. In short, the Documents are using the old database(the first connect() call is in the settings module). Any insights?
[10:21:20] <tiller> What's the better tree structure to retrieve the full path from a child? I mean, I've a document per node, and I want to retrieve all documents which are within the path to the given child
[10:25:28] <cheeser> it'd be interesting to try writing a (dynamic) aggregation pipeline to find those without having to store the whole path on each node.
[10:26:42] <tiller> cheeser> I'll fetch the collections way more often then I'll modify it. So it's better if the search cost is minimized :)
[10:30:42] <Derick> tiller: no, store not only the parent and childer, but the whole path - it's called materialized paths or something
[10:31:08] <Derick> http://docs.mongodb.org/manual/tutorial/model-tree-structures-with-materialized-paths/
[10:31:09] <tiller> Derick> http://docs.mongodb.org/manual/tutorial/model-tree-structures-with-materialized-paths/
[10:32:13] <tiller> yes but at the end I'll have to first find the child to get the path, and then find again to get its parents (Because I don't want only the ids)
[10:57:45] <tiller> Just to know, can we do "recursive" aggregate with mongodb? I mean, if I've for example: {val: 1, childs: [{val: 2, childs: [{val: 3, childs: []}]}, {val: 4, childs: []}]}]} to find if I've "val = 4" somewhere into the "tree", without knowing the deepth
[11:33:22] <Peeter> http://stackoverflow.com/questions/20215508/normalized-vs-denormalized-data-in-mongo
[11:38:25] <Peeter> Doesn't seem to be in any way related to the question I posted, more like something else :p
[11:40:28] <Derick> because you do a lot more views than updates, it makes littles sense to split this out.
[11:50:29] <Peeter> Could you expand on the data locality point? Wouldn't it require more memory if data is duplicated?
[12:10:08] <tijmen> Hi all, if I restart a running mongo with oplogging enabled can I then still do a mongodump with --oplog? I understand it will only contain oplog data from the moment of restart. If this works, will this also allow mongorestore to read this partial oplog data?
[13:23:52] <iliyas> I'm working on a particular use-case wherein I have imported a JSON log file in MongoDB. The log file has the following format as provided in the reference below,
[13:23:53] <iliyas> Can anyone help me in understanding how I can query the inner elements such as awsRegion, eventName etc ?
[13:30:05] <iliyas> Nodex: A single log file is loaded as a single document in Mongo. When I execute the above query it prints the entire document. I'm looking for how I can fetch the respective values for a given key.
[13:34:19] <scrdhrt> But that will match all awsRegions in the log file. If you want to match a specific document in the collection you have to match on a unique value
[13:38:04] <Nodex> something like .. db.foo.find({"Regions":{$elemMatch:{awsRegion:"us-east-1"}}},{"Regions.awsRegion.$":1});
[13:45:46] <ckrause> I *think* I want to use the code data type to store functions, but I am not sure. I cannot find an example that uses it. Can somebody point me to info on how to properly use the code data type?
[13:58:17] <ckrause> Nodex: Let me try to explain. I want to store documents that are "nouns" and documents that are "verbs"
[13:58:44] <ckrause> noun documents will have properties. Imagine an RPG one noun document is a character
[14:00:30] <ckrause> I assumes I can retrieve a function from mongodb and "type cast" it as a function and use it in javascript
[14:10:15] <kali> ckrause: i'm not surprise. i can not say for sure what it was intended for, i've never heard about anybody using it at the application level
[14:11:45] <ckrause> kali: I don't want to write all kinds of code at my applications level only to find out later that I could just have used the code data type. But at this point, I have no idea what they code type can and cannot do.
[14:41:42] <Repox> Hello. I have some time data that I'd like to have converted to an ISODate object. See http://pastie.org/private/pded4zfyvwczr29a1tenw - Is it possible to get the field "updated_at" converted correctly to an ISODate object?
[14:55:30] <paulo_cv> hi, how do I use an AND condition in a remove command? for instance db.collection.remove({condition1: "xyz" **AND** condition2: "abc"}) ?
[15:19:04] <Neptu> if you mongo a secondary machine what was the command to let you utilize it and do some search querys
[15:24:37] <cheeser> Tue Nov 26 16:22:52.436 TypeError: Property 'slaveOk' of object test is not a function
[15:31:23] <Neptu> I was thinking on running mongo on some arm quads or the new intel atoms quads with 8GB ram unsure if minimalistics nodes works good with mongo
[15:33:08] <Derick> you don't really need lots of CPU if you don't do complicated calculations (through f.e. the A/F)
[15:52:23] <du2x> hello there, I would like to know if there are anybody who had problem with mongodb intermittent connection failures.
[15:54:50] <Derick> du2x: can you please tell a bit more about setup, language, driver, versions, hosting platform?
[15:57:27] <tiller> Is there a way, within an aggregate <hen grouping, to merge arrays? I thought it would work with $addToSet but it creates me a set of arrays :)
[16:03:36] <ckrause> I have been able to use the node native driver to create a document with a code data type
[16:12:30] <timhansen> morning all. i'm having an issue getting my rails app to conned to my mongo db. it was working fine a month or two ago, and nothing on the server side has change. was wondering if anyone can help me debug this issue: https://gist.github.com/willc0de4food/403d3a99f668e92c64e3
[16:16:10] <tiller> Tried something like this: db.permissions.aggregate({ "$match" : {"_id": { "$in" : [ ObjectId("...")]}}}, { "$group" : { "_id" : 1 , "roles" : { "$addToSet" : {"$each": "$roles"}}}});
[16:16:39] <Derick> tiller: can you put your example in a pastebing - complete with start document, a/f query and expected output?
[16:18:14] <ckrause> I am looking at the source code for the native driver. In particular https://github.com/mongodb/node-mongodb-native/blob/1.4/lib/mongodb/index.js
[16:27:04] <tiller> hmm, I see a way to do it, but it seems to be "the hard way". unwind, before grouping
[16:30:15] <sec^nd> Is there a place I can get an in depth guide of mongodb? From both using the database, best practices, administration, schema design, and maintenance?
[16:32:53] <sec^nd> I want to store massive amounts of data in mongodb (terabytes maybe petabytes eventually). I want to upload a bunch of images and videos to a deduped gridfs like schema, hash each file, store meta data on it and make it easily searchble via api and frontends. Can mongodb support this? Also I need to scale with replication and sharding w/ servers that have about 20G of ram and a lot of hard drive space.
[16:33:43] <Derick> sec^nd: that's quite something you want to do - my (biased) opinion is that MongoDB can do this - but it will need a lot of thinking
[16:35:10] <sec^nd> Derick: the dedup storing of the images will use a schema almost exact to gridfs except with deduplication of the blocks, I will also use a chunk size of either 256K or 512K.
[16:45:27] <sec^nd> What is the best way to take a backup / snapshot of a large shard / replication setup?
[16:45:37] <du2x> Derick, I have a lot of connections, opening, and closing. I do this when I am importing data. I see the log and there is always one or two opened connections. My question is: do I have to avoid these opening and closing connections? I refer to the intermitent connection failures problem.
[16:45:43] <ckrause> I am now able to create a document with a property of code data type. That function takes two documents and performs math on properties of those two documents. But I can still only do this in the shell. Haven't figured out how to do it with the node native driver.
[16:46:24] <Derick> sec^nd: an extra replica set member that's hidden per shard that you can do backups off, or just use as backup. Also have a look at delayed replica set members.
[16:47:06] <sec^nd> ahh Derick so I could use the replica sets and take a moment in time snapshot from them?
[16:48:56] <Derick> sec^nd: I did look for the gridfs specification, but that doesn't have useful info: http://docs.mongodb.org/manual/reference/gridfs/
[16:51:17] <Derick> http://antydba.blogspot.co.uk/2010/02/mongodb-backup-with-zfs.html has some info
[17:12:34] <du2x> hey Derick, I have some details about my problem. Maybe worth reporting, then. The made a script the treats the connection failure by sleeping one second and try again, and repeat until it gets connected and make stuff. Then I detected a pattern: a lot of connections happens, one connection failure happen, and keeps failing by 5 to 8 seconds, then I get a lot of connections happening again.
[17:21:45] <Derick> du2x: I think the python driver uses connection pooling, so it will then create a few connections if it can.
[17:22:05] <Derick> du2x: what you do need to find out is why the server breaks connections. Does something show up in the MongoDB log?
[17:28:36] <du2x> Derick, well, the log says only connections opening and connections closing. So, can I conclude that it's a python driver issue?
[17:29:36] <Derick> du2x: it sounds like that, but I have too little evidence. I do have to leave now though, but I'll be back tomorrow.
[18:40:51] <ron> still can't get over today's interview question - if you didn't use Big Data why did you use MongoDB?
[19:54:37] <bui> hi ! I have a couple of documents, but I want to check if the number of unique('id') from those represents more than X% of unique('id') from documents sharing a property (ie, 'zone') present in the whole collection (from where my documents came from), is x.find(['zone' : zone]).unique('id').count() a proper way to do so ?
[22:16:32] <DanielKarp> Having trouble getting queries that I think should be indexOnly to be indexOnly. Can anyone tell me why this simple example is not explaining to indexOnly? http://pastebin.com/6zaNs9vQ
[22:17:55] <DanielKarp> (obviously, the real-life example is more complicated, but first I want to see if I can get it working in a simple test)
[22:21:18] <astropirate> I got an array filled with ObjectIDs. I want to return all of the corresponding documents, they are all in the same collection. What is the maximum number of IDs I can' pass using the $in operator?
[22:37:04] <tehroflmaoer> hey, what would be the best way to store an enum? I feel like using a string would be pretty unnecessary
[22:39:58] <cheeser> others don't like that because you can't rename them and so prefer to ordinal value.
[22:40:32] <cheeser> i don't like *that* because now you can't reorder them (even if only by adding a new on in the middle) and that's a far more common change than renaming
[22:41:52] <tehroflmaoer> I've got a thing that I'm storing that's got at least 4 enum fields, so I think I'm kind of leaning towards storing ordinal values
[22:43:18] <cheeser> spend enough time debugging queries and their output and you'll wish you had those names, i'll bet.
[22:43:36] <cheeser> but then, again, spend enough time doing that and you'll have the ordinals memorized.
[22:49:55] <DanielKarp> joannac: hey, I was able to get that query to work indexOnly in the mongo command line, but not through the php driver.
[22:57:41] <floatingpoint> where is the function documentation for the nodejs driver? I can't find anything obvious for .update
[23:36:14] <DanielKarp> joannac: Final word: it seems as if I can't get a query to hit indexOnly through PHP when going through findOne and restricting the fields appropriately.