Log file Viewer
#mongodb logs for Thursday the 26th of May, 2016
[00:01:07] <Boomtime> -> "Is there anyway to massage or suggest an _id?" - you can specify the _id field for any document during insert, is that the question?
[00:01:17] <zsoc> Do tell kind sir. The direction i'm walking in at the moment is just to set ._id as .api_id and then delete ._id and let mongo generate it.
[00:01:34] <zsoc> Boomtime, yes but apparently only if it's 12 bytes or 24 hex characters. I have.. I think this is an MD5 heh
[00:04:22] <zsoc> Wait, but if an _id can be a string, then why the heck was I experiencing the "Mod on _id" error
[00:04:33] <rkgarcia> zsoc, you need yo update all your records creating the api_id field with _id value
[00:05:36] <Boomtime> indeed - "Mod on _id" specific wording isn't any mongodb error i know of but rkgarcia is right, _id cannot be modified after the fact
[00:09:18] <zsoc> and then the entire object as the update param... ie: .update({_id: acct._id}, acct, ...
[00:09:50] <zsoc> which i thought was "update if you find an _id like this.. otherwise insert" (well, with the upsert option)
[00:13:21] <zsoc> hmm.. yeah it totally worked when i just moved _id to api_id and then deleted _id and let mongo generate one... that's odd tho isn't it?
[00:30:49] <zsoc> If i assign an _id it yells at me, if i don't assign an _id then the _id in the inserted object is null
[00:42:09] <zsoc> Right soooo... sure but. Mongoose. I can just define _id: String, in my schema? Because that didn't appear to work
[00:44:05] <AlexKvazos> Never heard that someone would want to change the _id of a document when creating it
[00:45:09] <zsoc> StephenLynx, yes you are right, I can't possible argue with that. but I use it for tidiness/reference/it keeps me organized
[00:46:53] <zsoc> well i liked the 1.) /models folder for referencing, and 2.) it affords modules that i can include into express routes which... realistically is no different then exposing the entire db but exposing an individual model feels cleaner
[00:47:19] <zsoc> the alternative... document my models on paper and.. expose the db object as middleware.. i suppose
[00:48:53] <zsoc> i mean as far as 'pick up and go' goes.. it's leaps and bounds ahead of anything php has to offer imo
[00:54:45] <zsoc> AlexKvazos, I think I'm going to set { _id: false } in the schema options and index: true on my api_id field. That should... "fix" it I guess. It's still a poor work around imo
[01:02:13] <zsoc> I'm certain I could make mongoose upsert a document with the _id field set (and yes, you're supposed to be able to define it in the schema as a string..) but i just don't want to fight with it anymore
[01:30:25] <zsoc> ok I actually /can't/ do that because mongoose only honor's _id: false in subdoc schemas.. sooo.. i guess i'll just have a compound index
[06:23:40] <Ryuken> It's a sad day when you can't ask programming questions on the internet without someone responding with a 5 hour lecture on why you're an idiot for using MongoDB
[06:26:26] <Boomtime> @Ryuken: you are in a safe place to ask whatever MongoDB related question you had
[06:36:52] <kurushiyama> Ryuken: The thing is that most people ranting about MongoDB do not know the slightest bit about it. "Looses data" is my favourite. "Can't handle relations" is close second. "Has no atomic operations" is outside competition, ofc.
[06:39:29] <kurushiyama> Ryuken: If somebody gives you a lecture about it, send them to me. I actually take it as a mental martial art to... lets say to enlighten them.
[07:26:28] <kba> I'm migrating a database and some keys contain dots, causing some errors, e.g. "Error: key (this._options['wordwrapwidth'] must not contain '.'"
[07:27:00] <kba> I'm not sure how it got in there in the first place -- I assume an older version of mongodb didn't enforce this no-dot policy or something. Regardless, it needs to be migrated as well
[07:27:40] <kba> According to http://stackoverflow.com/questions/12397118/mongodb-dot-in-key-name, there's a `check_keys` option that I should be able to set to false, and then the error would be ignored
[07:28:28] <kba> however, I seem unable to do that with the node driver. I can't find any documentation on the subject, but I've tried coll.insert(docs, { checkKeys: false }, fn), but to no avail
[07:34:01] <kba> I need to write a migration script for others to use, so it needs to be a script of some kind
[07:39:52] <Boomtime> @kba: keys with inline dots has never been valid and never intended to be allowed - there were ways to do it in the past and it caused trouble even then - you can't get around it, nor should you try - good luck with the migration, it is the right thing to do
[07:42:52] <kurushiyama> Hm, iterate over the docs and the keys, eliminate any dots you find in the key strings and us bulk operations to write back to a temporary collection. I would not dare in-place updates...
[07:43:35] <kba> I am copying to a different collection and the inserts throw errors because of the dots
[07:44:25] <kba> but eliminating dots is very likely to break everything. So I think I'd rather just report to the user that some documents were unable to be migrated
[07:46:46] <kurushiyama> kba: I'd rather identify the docs with said problem and hand them over to the user for review.
[07:47:40] <kba> Yes, that's what I'm doing -- I'm generating a list of documents that were not migrated
[07:49:39] <kurushiyama> kba: Sure. But may be the keys are substitutable? And I would not create a list of docs, I'd create an export at all costs. Probably replacing dots with double underscores or sth.
[07:52:33] <kba> The data I'm saving is jsonml representations -- the only properties I've seen so far are JavaScript code lines such as this.options or whatever. The issue also only occurs with the operation logs (i.e. operational transformations performed on the jsonml), but a snapshot of the current document will already exist
[07:52:58] <kba> what will happen is that versioning won't work for the specific documents -- or rather, old versions will be inaccessible
[07:53:30] <kba> I think it's a sufficient solution. However, it doesn't seem to matter, because nodejs is running out of memory again now during the migration
[07:54:01] <kba> I've rewritten it thrice now based on suggestions from in here, from first usiong toArray.forEach() to .each and now using .hasNext/.next
[07:55:26] <kurushiyama> kba: If that is possible in node (which I avoid like the devil holy water).
[07:55:51] <kba> I tried .each, which uses the cursor, but I was told that it would still load everything into memory
[07:59:21] <kurushiyama> kba: I think you are constrained by your tools, then. The very last thing I would even theoretically use node for is database change management.
[08:00:03] <kurushiyama> But I can not really imagine that the cursor is returned as a whole. It would be impossible to deal with large datasets.
[08:10:43] <kba> It is incredibly slow now as I'm processing just a single operation from the log at a time, but it doesn't seem to be running out of memory anymore
[08:14:58] <kurushiyama> kba: Well, I'd use bulk ops, which should speed things up considerably. but then, I do not know node.
[08:15:59] <kba> well, loading a bunch of ops into node, performing the transformations and inserting them caused node to run out if memory
[08:16:30] <kba> I could probably spend a lot of time fiddling with it to figure out how many ops it *probably* can handle at a time to speed it up
[08:16:51] <kba> but right now this seems to work -- and even if the migration takes 30 minutes, it's a one-time thing
[10:34:54] <electsleeper> hello :) Is it possible to have mongodb as the database/authentication service in active directory?
[10:48:24] <gain_> I have a documento with an array inside it and I need to update only an element of the array
[10:50:42] <gain_> I've tried with db.test.update({ _id: "1893fi1p938f4iwef" }, { $set: { myArray[2]: { field1: "aaa", field2: "bbb" } } }); but not works... that's correct?
[12:14:52] <kurushiyama> gain_: https://docs.mongodb.com/manual/reference/operator/update/positional/
[12:18:45] <diegoaguilar> gain_, you should use $matchElement operator :) just google mongo match element
[12:45:42] <zsoc> many would beg to differ lol. i have no personal experience so i can only read the angry blogs
[12:55:33] <Keksike> Is there a way to write a update-operation which uses a value from the document for the update. So heres an example to show you what I want to do: https://gist.github.com/Keksike/25fe53863acaf169240c3ab75bb93e6c
[12:56:27] <Keksike> http://stackoverflow.com/questions/3788256/mongodb-updating-documents-using-data-from-the-same-document/3792958#3792958 I think this is what I'm looking for
[13:06:43] <electsleeper> kurushiyama, yeah, I found some info but not sure if I can adapt it to what I want
[13:08:44] <kurushiyama> zsoc: Well, the angry blogs tend to be written by angry people. Angry people who refused to learn the very basics of the persistence technology they were using. As for loosing data, that would have been the write concern. To give you an impression: Not caring about the writeConcern is a bit like doing concurrent updates in any given RDBMS without transactions.
[13:12:01] <kurushiyama> zsoc: As per scalability. The biggest cluster I have worked with was a 14 shard cluster, with around 10TB of data. Try to do that with a DBMS of your choice. And after you have set it up, get cleanly rid of one or 2 machines.
[14:22:47] <Zelest> Derick, I have created a cursor and as I loop through it, I will run other queries and get the following error: "Uncaught MongoDB\Driver\Exception\RuntimeException: Another cursor derived from this client is in exhaust."
[14:23:10] <Zelest> Derick, any ideas what I'm doing wrong? do I need to loop through the cursor and store all data "locally" before looping through it again and run the other queries?
[14:28:13] <Derick> Zelest: that looks like a C driver message. Do you have a reproducible (small) script perhaps?
[14:29:28] <Zelest> Sadly not :/ But using $cursor->toArray() seems to solve it through.. Let me see if I can reproduce it in a smaller script...
[14:32:14] <saml_> when update, can i make it only update existing fields but never add a new field?
[14:39:26] <Zelest> I can't seem to duplicate it.. :( seems like some totally unrelated script caused it though.. o_O
[14:49:46] <Keksike> Is there a way to write a update-operation which uses a value from the document for the update. So heres an example to show you what I want to do: https://gist.github.com/Keksike/25fe53863acaf169240c3ab75bb93e6c
[15:00:19] <Keksike> and I wouldn't really like to do a extra 'find' for each of the documents I want to update this way.
[15:39:47] <Jidoor> is it possible to project some subfield in a dictionary with find()? i'm trying db.activities.find({ },{ "gpx": { "trk": { "name": 1}}}).pretty() but i get error
[17:09:13] <zsoc> kurushiyama, thank you for your real life experience, there's not enough of that on the internet
[17:09:47] <zsoc> I feel like this channel is pretty polar when it comes to Mongoose.. do you have an opinion there too
[17:10:46] <kurushiyama> zsoc: It basically enforces or at least encourages a wrong approach to data modelling.
[17:11:39] <kurushiyama> zsoc: The SQL way, namely: Identify your entities, their properties and relations, model them and then bang your head against the wall to get the questions you have on your data answered.
[17:12:03] <zsoc> Could you point me toward a mongo-centric write up about the /right/ approach to data modeling? Part of the issue for someone new to mongo is basically 99% of the MEAN or MEAN~ish tutorials or writeups use mongoose because of clarity of subject.
[17:12:19] <kurushiyama> zsoc: Imho, the proper way of approaching data modelling for MongoDB is quite the opposite.
[17:14:08] <kurushiyama> zsoc: I try to keep those questions in the form "For a given X, what is/are the..."
[17:15:03] <zsoc> So mongo has the extra step of "when should I embed?" which i feel like should have a book of it's own written on it lol
[17:15:53] <zsoc> Right, every angry article abount mongo scaling is because of embedding, from what I can tell
[17:16:07] <kurushiyama> zsoc: 3. Redundant data, if it saves you queries, is not only ok, but desirable.
[17:16:58] <zsoc> That's not one I would have thought of. I jump through hoops to avoid saving the same data point in different collections
[17:17:05] <kurushiyama> zsoc: http://stackoverflow.com/questions/32409635/mongodb-query-comments-along-with-user-information/32411030#32411030
[17:23:07] <kurushiyama> zsoc: Regarding Mongoose. If you do your data modelling correctly (which _is_ possible with Mongoose), all that remains is an additional, and – as far as I understood - rather useless layer. And it eats quite some performance.
[17:24:28] <zsoc> The issues I've run into with Mongoose have all been inconsistencies with the methods. Like Model.update is not /exactly the same/ as the drivers .update() and the documentation tends to be obscure
[17:35:20] <kurushiyama> zsoc: I'd suggest using a similar approach to mine, ditch mongoose and use the raw node driver.
[17:37:14] <kurushiyama> zsoc: Aye. As per redundancy and optimizing: Optimize for use cases which are most important. As detailed in the second link.
[17:37:20] <zsoc> I feel like I'm going to have to /un-learn/ a lot of stuff because there's no clear line to where mongoose is abstracting something or of it's just how mongo does it
[17:54:46] <EzeQL> hi, do you spot any typo here: db.items.update(document.news:null},{$unset: {document.news:1}}); ?
[18:09:00] <jmorales_> Hello, I'm a little bit confused about the purpose of keyfiles in a replica set cluster environment, does the usage of openssl generated keyfiles encrypth the traffic between replica nodes or it only provides authentication?, does it affect mongodb clients?
[20:20:00] <alexi5> i followed your advice and analyzed the possible questions that will be asked for the data and came up with something that is far simple than the ERD representation
[21:54:22] <uuanton> i have 'customfields.localization' array how do i find document with most of element in array ?
[22:09:31] <uuanton> i need to find document with most of elements of localization where en:["chicken"] one of the element
[22:17:23] <uuanton> db.getCollection('translation').find({'customfields.localization': {$elemMatch: {"language": "en-gb","title": "Max Adults","author" : {$eq: "xxx"}}}}).limit(2)
[22:18:37] <uuanton> now im trying to find document with most translations of max adults where author is xxx
[22:31:51] <tombin> trying to connect to mongodb cluster over ssl i see SSL: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown is this an issue with the certificate on the mongodb side ?
[22:33:08] <tombin> my mongodb connection string is mongodb://user:password@mongos:27017/dbase?ssl=true
[22:57:00] <alexi5> have you guys used mongodb to store images and other large files (say no greater than 3MB)
[23:11:05] <kurushiyama> alexi5: Personally, I'd suggest to use MongoDB as a long-term storage, and use stuff like varnish (preferred), ehcache, groupcache, memcache to cache the images. As per hard data, I have nothing to come up with, but it is not shabby.
[23:11:47] <kurushiyama> alexi5: If you use a language supporting streams, things can be _really_ fast.
[23:14:35] <alexi5> i am thinking of using a document for thumbnail and metadata and gridfs for the full image
[23:17:30] <kurushiyama> Have the original image either in GridFS or in a normal doc – I do not care, and that very much depends on the size.
[23:18:17] <kurushiyama> But as for the thumbnail, I'd generate them on first request, and then either use a CDN or a cache to store them.
[23:20:50] <kurushiyama> alexi5: Aye, pretty much that. There _may_ be use cases valid for prerendering the thumbs, but unless you have _plenty_ of visitors, that should not make a noticeable difference. And if you have that many, you can easily afford a proper CDN or dedicated large cache machines.
[23:21:49] <kurushiyama> Again, as for the modelling, such decisions depend heavily on your use cases.
[23:22:40] <kurushiyama> For example, with an image site (something like flickr), where you need pretty much a thumbnail for every picture all the time, it may make sense to prerender the images.
[23:25:02] <kurushiyama> For thumbnails of user images? Not so much. Say a thumb is 5k in size. Put a varnish in front of your deployment, give it 500MB of cache space, that'd be 12500 thumbnails. Served from RAM for the second request. Not even hitting your application any more.
[23:30:41] <kurushiyama> alexi5: So, as you can see: it depends. I do not know enough about node, but on the fly resizing should be relatively easy. Since it makes sense to use docker with node, you might want to have a look at docker-compose. You can use an existing varnish image and your application image and fire it up with a simple "docker-compose up", if you have everything in place.