Log file Viewer
#mongodb logs for Wednesday the 27th of May, 2015
[00:03:36] <xaxxon> can you restrict what fields you send back when you call find? like if there's some big field that you don't care about?
[00:05:08] <xaxxon> http://docs.mongodb.org/manual/tutorial/project-fields-from-query-results/ - I think I found it
[01:02:42] <kryo_> sometimes when i call the procedue though, i'm left with less than 6 new elements
[01:08:13] <Boomtime> @devinceble: you need to use the legacy API for GridFS for now as the async specification for the API is not final
[01:25:25] <kba> if I'm using mongoose in nodejs and I have a mongoose object that I save. If I then update that object and save it again, is all the data transferred?
[02:16:09] <MacWinne_> in mongo, i see a certain document field is of type ISODate("2014-06-24T23:10:10Z") however when I query with mongoose, I seem to get back just the string '2014-06-24T23:10:10Z'
[02:17:13] <MacWinne_> is this expected behavior? I've recently started using Mongoose.. was using PHP before, and I think PHP returned the date field correctly.. though am not completely sure
[04:13:19] <fontanon> Hi everybody! Observing my database db.stats() I have one doubt ... is the free space on disk pre-allocated by mongo db equals to (fileSize - (storageSize + indexSize)) ??
[04:22:27] <Boomtime> @fontanon: i believe that will be very close, although not exact - the difference is in the amount of space remaining in the extents already allocated to the indexes
[04:25:09] <Boomtime> this means of course that you can't know precisely how much free space exists already in the free lists, but you can get a very close approximation
[04:25:10] <fontanon> Boomtime, so mongodb does not only pre-allocate space for data but for indexes too
[04:27:07] <Boomtime> from within those files, section are carved out called extents, and allocated to collections for data or indexes
[04:27:41] <Boomtime> when you work out the "free space" using the method you have, you are getting a combination of three things in the result:
[04:30:16] <Boomtime> unassigned extents exist inside pre-allocated files but are not attributed yet to any collection within that database
[04:31:34] <Boomtime> indeed, it could be as little as one byte increase to dataSize that forces a new extent to be allocated - which might be 512MB
[04:32:22] <Boomtime> if you insert a billion documents, then slowly remove all but one, you'll still have a large index
[04:33:37] <fontanon> the current behaviour I observe in my disk (by looking df -h frequently) is; new writes uses additional disk space but this space turns free again once consume a couple of GB or so.
[04:35:13] <fontanon> the point is i moved the half of the chunks from the primary shard but I expected to see the proper free space
[04:36:40] <Boomtime> it is even possible that no extents even got returned to the free list, depending on the distribution of documents that got transferred to the other shard
[04:37:06] <Boomtime> document storage will have been freed up, but whole extents might not have been released
[04:39:01] <Boomtime> mongodb tries to keep ahead of the curve, so the last file allocated is almost always empty
[04:39:23] <Boomtime> it's also the largest file at the start, until the filesize reaches it's maximum of 2GB
[04:39:49] <fontanon> "dataSize" : 23278657872, "storageSize" : 36217163664, "indexSize" : 4388868624, "fileSize" : 62180556800,
[04:40:56] <Boomtime> btw, passing "1024 * 1024" as a parameter to db.stats() will make that eaiser to read
[04:42:01] <fontanon> "dataSize" : 22202, "storageSize" : 34539, "indexSize" : 4186 "fileSize" : 59300,
[04:44:16] <Boomtime> it reflects what you're actually using on disk, which ultimately is the most important
[04:51:01] <zamnuts> db.authenticate hangs when attempting to authenticate against multiple mongos instances using node.js driver version 2.0.33: http://pastebin.com/WsxpvGxq any ideas?
[04:59:56] <fontanon> Boomtime, this is really interesting :) http://www.awesomescreenshot.com/image/266237/823d898ff3eb54b01c2cdaddd52d3909
[05:01:47] <fontanon> the dataSize was shrinking during the chunk migration, once finished the storageSize shrinked abruptly, while fileSize remains the same
[05:03:40] <fontanon> Boomtime, I had MMS installed by the time the chunk migration started. I didn't realized how useful this graphs was to understand what happened with the expected free space
[05:04:52] <fontanon> By me getting nervous, I meat me seeing how no new free space was returned to the SO so I restarted mongod (big fail)
[05:05:44] <zamnuts> DragonPunch, you don't have to worry about things like SQL injection with the nodejs mongodb driver
[05:06:42] <zamnuts> well... as long as the user input is being stored as a value, and not used as a field, config option, query struct, etc.
[05:08:54] <zamnuts> fontanon, Boomtime, chunk migrations will consolidate data files? i.e. compact?
[05:09:39] <zamnuts> how else would physical space decline? unless the chunk migrations are moving from mmapv1 to WT w/ compression?
[05:31:21] <Streemo> INot only would it be convenient, but it was be beautiful if we could perform arbitrarily nested array queries. Unfortunately, this doesn't even work for N=1 level of nesting.
[05:32:47] <Streemo> for example doc = {field:'val','arrayField':[doc1,doc2,doc3,doc4,doc5,...]} arrayField could be considered its own sub-collection, in which case we could run a query on the parent collection, with a nested query on the sub collection so that the returned document does not contain a full array in the arrayField.
[05:34:59] <Streemo> db.collection.find({_id:ObjectId},{arrayField: subQuery: {find: {name:"john"}}}) would return {_id:ObjectId, arrayField:[{name:"john"}]} where arrayField only contains the elements which were matched by the subQuery.
[05:35:39] <Streemo> db.collection.find({_id:ObjectId},{arrayField: {subQuery: {find: {name:"john"}}}}) I mean
[05:36:36] <Streemo> i was dissapointed to find out that elemMatch was like a subquery that didnt actually return the matched results
[05:38:56] <Streemo> and it could be off by default for performance, but turned on if the array has "isQuery: true" or something
[05:41:56] <Streemo> so you could return a denormalized document with a Post/comment type relationship, even though the udnerlying collections are normalized
[05:44:03] <Streemo> zamnuts: anyways, i think it would be a nice API addition to at least be able to choose whether elem match returns a modified document or the document itself. Sure, it can be done in the aggregation framework ,but to me it seems like such a common thing that i would want a general syntax for it and not have to implement it on the fly every time.
[05:53:21] <Streemo> zamnuts: i suspect my reuqest is so common because it is a very obvious thing to expect possible, that someone else must have already openend a similar ticket.
[05:54:55] <Streemo> zamnuts: honestly, i will npot open a ticket. there are several solutions around what i am talking about - specifically, normalization.
[06:03:49] <Boomtime> @Streemo: the answer will be aggregation, what it's designed to do - why is that not an option for you?
[06:40:59] <zamnuts> bump: any idea why authenticating against multiple mongos would hang the node.js driver?
[06:51:27] <Boomtime> @zamnuts: no, but are you using multiple mongos in the seed list to the same database or do you mean seperate instances?
[06:54:00] <zamnuts> it connects just fine, i can attempt queries and get auth errors... but when doing db.athenticate (node.js driver), it just sits there, forever. when i use a single mongos in the seed list, everything is snappy.
[07:06:57] <Streemo> Boomtime: it is an option, but Meteor does not support livequery for aggregation operations
[07:10:36] <Streemo> structural question here. if you keep track of users' searches in the DB, would you store searches as an array field on users, or would you normalize it and have a serches collection which stores documents like this: {userId: someUser, search:searchCriterion}
[07:20:30] <acidjazz> then a sudo service mongod start and got: /usr/bin/mongod: symbol lookup error: /usr/bin/mongod: undefined symbol: _ZN7pcrecpp2RE4InitEPKcPKNS_10RE_OptionsE
[07:22:16] <acidjazz> http://stackoverflow.com/questions/20872774/epel-mongodb-will-not-start-on-ec2-amazon-ami
[07:22:52] <acidjazz> i have no problem going direectly to mongodb for the repo and trying out 3.0/etc
[07:23:20] <preaction> doesn't seem like a mongodb problem. seems like an image problem or an os problem
[07:24:32] <acidjazz> ok so ill purge this and add monogo's epel.. just hope my old mongo dump goes into this 3.0 setup
[07:25:58] <phocks> Can anyone tell me the difference between var mydate1 = new Date() and var mydate2 = new ISODate() ??
[07:31:39] <phocks> the two commands produce the same time it looks like (a few seconds later for the second one of course)
[07:37:30] <acidjazz> 2015-05-27T07:28:43.277+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
[07:53:57] <acidjazz> good lord this t2.micro current gen ec2 is WAY faster than my older medium instance
[07:58:55] <Boomtime> micro instances are a crapshoot, they have no assurances so you run on spare cycles only - there is no point doing performance tests because it's rolling dice every time
[08:02:08] <acidjazz> because reading that i 100% imagined the skit with her saying that in response to my excitement
[08:04:12] <Boomtime> well, if you want to put a tonne of micros into a computing cluster where you don't care what each one achieves then go nuts
[08:30:47] <f31n> hi i'm pretty new to monogo actually to nosql i asked yesterday whats wrong with this table layout in the eyes of a nosql: http://pastebin.com/kJqSB0TB and worked this one out - is that better - do you have any ideas to make it even better http://pastebin.com/wrh5axJw ?
[12:20:03] <aps> So I am in a weird sticky situation here. I had 1 primary + 1 secondary + 1 arbiter. Due to some unexpected shutdown both primary and secondary are in "recovering" mode now. How do I fix this since there's no primary?
[12:25:47] <aps> I don't have any important data, so I can go back to clean slate as well. But how do I do that?
[12:26:33] <cheeser> well, if you don't need to save your data, shut them down, delete the data dir, and bring 'em back up
[12:28:04] <aps> cheeser: I had tried something like that and I was getting "assert failed : no config object retrievable from local.system.replset" when I tried to start again
[12:30:32] <aps> cheeser: is there a way to fix this without deleting any data? In case this happens i future where I have some precious data
[12:37:14] <leandroa> hi, I've a doc like: {val: {"0": 231, "1", 323, "2": 23, ...., "31": 23}.. is there a way to get a portion of val? like $gte 5 and $lte 20?
[12:50:40] <aps> joannac: What I think happened was that 1st secondary went down, and by the time it was back, even primary was down and just arbiter was running. So, both members think they are out of sync and need to recover from primary
[12:54:04] <aps> joannac: This is how r.status( ) on the Arbiter looks like - http://paste.ubuntu.com/11391107/
[14:46:03] <ToTheInternet> is it bad practice to create new collections for every session? named something like collection-132095102 where the number is a session identifier
[14:49:05] <ToTheInternet> kinda figured. i'm new to mongodb, and whenever i try to model data i end up using mongodb like sql
[15:31:15] <pcr0> I'd like some help with a seemingly common error, "Failed to load C++ bson extension, using pure JS instead"
[15:31:34] <pcr0> I've described my problem here http://stackoverflow.com/questions/30486594/failed-to-load-c-bson-extension-using-pure-js-version-error-when-running-mong
[15:34:25] <jubjub> Hi guys i need to make a notification collection with multiple types and there are 2 approaches i can think of. 1) multiple collections and merging them later on, 2) a single notifications collection with a type field. Which configuration is done more often in practice?
[15:35:45] <StephenLynx> if they are so similar they only have the "type" being different, I would use a single collection
[15:36:46] <StephenLynx> pcr0 you mentioned xcode, that is probably what is going on. from what I know, xcode is responsible for compiling stuff on OSX.
[15:41:07] <pcr0> Hmm, I'm trying to make do with what I have. Not having the giant bloated thing was my intention, I thought the CLI tools were all that other apps depended on. The problem is, reinstalling Xcode didn't fix it either.
[15:43:42] <jubjub> StephenLynx: well in the future there may be a diffrent notifictation with different people effected and may need to modify the structure
[15:48:49] <StephenLynx> in general, I have actively developed in osx, windows and gnu/linux for a job. I cannot even compare how gnu/linux is superior to both of the others, pcr0
[15:51:25] <pcr0> While I might change my mind in the future...I'm perfectly happy with OS X for all my nascent dev needs so far
[16:05:11] <jubjub> How can i parse a collection into a new one. For example {a: 10, b:20, c:30} -> {test1: 20, test:30}
[16:09:56] <StephenLynx> if will get your documents and create a new collection composed by these results
[16:10:23] <StephenLynx> but if its a one-time only thing, you can just move or copy collection and change its fields and remove others.
[16:26:32] <Foxandxss> Hello, I installed mongo yesterday on my vagrant, I created an app (without user/pass) and today it is not working
[16:47:43] <laurentide> i'm following this tutorial: http://code.tutsplus.com/tutorials/authenticating-nodejs-applications-with-passport--cms-21619. under the 'configuring Mongo' section, the author assumes i know how to set up a local Mongo instance, which I don't. is anyone able to point me to an accurate resource for this? thank you
[16:50:37] <Foxandxss> StephenLynx: I think I had messed up two installations, nevermind my question
[16:51:12] <Foxandxss> vagrant's mongo was 2.x and my client on my computer was 3.0.3 so I uninstalled the older one to put a new one from mongodb's repo and after restarting vagrant it got screwed
[16:55:32] <StephenLynx> I never heard about vagrant. any particular reason you use it instead of virtualbox?
[16:57:52] <saml> it's a wrapper around virtualbox (and others) so that mac osx frontend devs can create development environment with ease
[17:51:58] <Parabola> hey folks, I need to create a read only user for all databases (including future ones) the issue is there are no DBs, and i have no idea what DBs the admins will add (its a dev box)
[17:52:25] <Parabola> so i need a read only account for all non system DBs, for the rest of the developers who are testing. is this possible? i cannot find what im looking for in the docs or google
[17:57:24] <Parabola> nevermind, turns out that was easier than i thought it would be, theres a role for that
[18:50:00] <Vitium> "If the field holds an array, then the $in operator selects the documents whose field holds an array that contains at least one element that matches a value in the specified array (e.g. <value1>, <value2>, etc.)"
[18:50:22] <Vitium> Is there a way I can enforce that it must match all of the elements of the specified array?
[19:58:57] <kenlik> hello. Is there any tool or way to create a data model visualization like a UML diagram?
[22:16:12] <Dev0n> hey, when modling mongo collections, should I try to "forget" about the rational way of thinking (ER) and just dump what's related into a single collection?
[22:25:15] <jrcconstela> I've got a collection with bulk data. Does anyone knows a tool to process its documents, whenever theres any?
[22:38:43] <Dev0n> just read about the capped collections, would I be right in saying that these would still allow the normal find/sort/group etc. queries to be run as well?