Log file Viewer
#mongodb logs for Monday the 11th of April, 2016
[07:44:23] <energizer> In python I have a list of dictionaries with unique ids. What is the best way to upsert it into mongodb?
[08:28:37] <Keksike> Hey. For some reason, after updating mongo from 2.6.11 to 3.2, I get this kind of error when trying to authenticate: 2016-04-08T14:58:11.907+0300 I ACCESS [conn88] Failed to authenticate thisUser@myDB with mechanism MONGODB-CR: AuthenticationFailed: MONGODB-CR credentials missing in the user document
[08:35:06] <Derick> the format for auth changed, and if you upgraded correctly, it should have automatically fixed that
[08:35:46] <Derick> Keksike: https://docs.mongodb.org/manual/release-notes/3.0-upgrade/#upgrade-existing-mongodb-cr-users-to-use-scram-sha-1 gives a hint (and a link) too
[11:43:03] <torbjorn> I need to merge data vectors coming in from different source, and detect possible conflicts, ie if { SampleID : "1234", Age: 24 } merges wtih { SampleID: "1234", Age:25 }, then thats an error, however { SampleID: "1234", Age: NaN } // is ok to merge, and then keep Age:25 for that sample
[12:21:06] <kurushiyama> torbjorn: That is definetly an application layer thingy, with application layer having a broad meaning. You might be able to do a diff/merge on your input data, depending on how the data is generated.
[12:25:10] <torbjorn> id rather keep it as null or na, and put all collected errors in an separate list
[12:25:51] <kjekken> hello. I need to insert a day , month , year and week record into a collection. I allready have a record called creation_time that contains a ISODate. Just need to use the isodate to make the other records.
[12:26:13] <kurushiyama> torbjorn: here is what I would do: Write a merge tool, which loads the data from the spread, looks up the according doc and does the merging/diffing/whatever. And log the errors.
[12:29:58] <kurushiyama> kjekken: I am not aware of something else than iterating and updating. But you may describe what you want to _do_, so that we can find an alternative.
[12:30:55] <kjekken> db.documents.find().forEach(function(doc){ db.documents.update({_id:doc._id}, {$set:{"year":doc.creation_time.getFullYear(), "month":doc.creation_time.getMonth(), "day":doc.creation_time.getDay(), "week":{$week: doc.creation_time.getWeek(),}}}); });
[12:31:55] <kurushiyama> kjekken: WHY do you need the day/month/year as separate fields. Why _exactly_?
[12:33:49] <kurushiyama> kjekken: I got that. Again, why _exactly_? And how does this query look like? And can we change it? It really, really, really does not make sense to have single fields for those values if you already have an ISODate.
[12:36:19] <kurushiyama> StephenLynx: Basically, there seem to be legacy queries/design reqs which need to be fulfilled by querying by y or m or d. Other than that, I am still interrogating the suspect ;)
[12:38:34] <kurushiyama> Most likely, and in place aggregation would make more sense, provided we have the possibility for an early match.
[12:40:49] <kurushiyama> may look different when we talk of time series data and for example the average number of users for a website by dow should be caclulated.
[12:41:46] <kjekken> i just want day(day number in month) year(ex 2015) month(ex 4) week(ex 34) as records in my db.documents collection
[12:43:15] <kjekken> "creation_time" : ISODate("2016-01-17T18:36:55Z"), will create : day 17 year 2016 week 2 month 0
[12:45:36] <kurushiyama> Uhm, I might be wrong, but that is a question of how to display a given date, no?
[12:52:25] <kurushiyama> kjekken: The thing is that you already have a date and you are putting the cart before the horse. It is simply extremely bad design to have this redundant data in the db. Load the data as is, and either use your controller (or however it is called) to split the date in the parts you need, or do it in the templates, if your template engine provides that.
[12:52:56] <kurushiyama> kjekken: If you really have to have it split when it is returned from the database, use an aggregation which does that for you.
[12:56:16] <StephenLynx> then you query the date you have stored, which will be just a simple date object
[12:59:20] <kjekken> db.documents.find().forEach(function(doc){ db.documents.update({_id:doc._id}, {$set:{"year":doc.creation_time.getFullYear(), "month":doc.creation_time.getMonth(), "day":doc.creation_time.getDay(), "week":{$week: doc.creation_time.getWeek(),}}}); });
[13:00:24] <kurushiyama> StephenLynx: It is the update for the data which would create the redundancy.
[13:01:04] <StephenLynx> use the desired range, get the date and use the date functions to extract the fields.
[13:01:52] <ren0v0> Does mongodb not have a remote sync option? I know there is copyDatabase(), bu this requires the db/table not to be existing
[13:02:26] <kurushiyama> kjekken: If you want to find the docs between two dates, it is pretty straightforward: db.date.find({date:{$gte:ISODate("2016-02-01T00:00:00"),$lt:ISODate("2016-05-01T00:00:00Z")}})
[13:03:04] <StephenLynx> just use the date functions to extract the parts of the date you need for the front-end
[13:03:48] <StephenLynx> one linking to https://gitgud.io/LynxChan/LynxChan/blob/master/src/be/graphsOps.js#L236 and the other to http://www.w3schools.com/jsref/jsref_obj_date.asp
[13:04:01] <StephenLynx> the first one links to how you will query all documents within the specified range
[13:04:16] <StephenLynx> the second one links to how you will extract the parts you need from the obtained date.
[13:05:40] <kurushiyama> kjekken: You want to use the _existing_ date, and extract what you need either in your controller or your templates.
[13:12:16] <kurushiyama> kjekken: The point is: regardless of where you do the conversion of the date you need, you do _not_ have to store them in the database. You can either use the aggregation, as shown above. Or, when you have loaded the data, you can extract the acording parts in your controller. Or (and that would be the best thing, if technically possible, in order to decouple logic and presentation), do it in your presentation layer.
[14:07:16] <echelon> what does it mean when the fatal.log reports.. "Still in error state. iteration #######"?
[15:55:14] <ren0v0> hi, i'm trying to use copyDatabase, but i'm getting complaints about a table already being there, but i've dropped the entire database prior to copy?
[15:57:31] <ren0v0> "errmsg" : "failed to create collection \"sites_data\": collection already exists"
[15:57:38] <kurushiyama> ren0v0: Maybe you want to show the command you are running and the actual error message.
[15:59:37] <kurushiyama> ren0v0: You clone it to a different instance. Maybe you want to drop the collection there? Or is this a replset?
[16:02:57] <ren0v0> i can't find any "sync" ability, this was the best solution i could find, copying from remote using ssh tunnel
[16:03:57] <kurushiyama> ren0v0: If you do not answer my questions, how should I be able to answer yours? ;) So I implicitly assume that we are not talking of a replica set.
[16:05:11] <StephenLynx> I'd like to make a comment. If you want to sync your staging to production and don't want to just copy what you got from one to the other, you might have deeper issues.
[16:08:06] <kurushiyama> StephenLynx: Nice spot. And it might well explain why the collection was (re-)created.
[16:09:19] <ren0v0> kurushiyama, i have no idea what one is, which is why i explained the scenario i was trying to overcome
[16:10:12] <ren0v0> StephenLynx, yes i'm just looking to copy one to the other for now, isn't that what "copyDatabase()" does though?
[16:10:52] <kurushiyama> ren0v0: Ok, let us assume you have none. But StephenLynx holds a valid point here. And the question is actually wether your application is still running. Because if inserts happen, databases and collections get created on the fly, which might well happen during the copy of the other data.
[16:12:37] <kurushiyama> ren0v0: NAh, you got that wrong. You are trying to clone the data to the prod database. But, when your application is still running, it might do inserts on said collection, in which case the collection gets created automatically.
[16:13:42] <StephenLynx> are you checking what data is in the destination db when you get that error?
[16:14:56] <ren0v0> StephenLynx, mongodump/restore yea this may be best, i'll need to write some scripts for it. Its a shame mongo doesn't have builtin functions for syncing et
[16:15:59] <kurushiyama> StephenLynx: with clone you basically can, but you have to make sure that the target DB is not in use.
[16:17:15] <kurushiyama> StephenLynx: Of course, that is not possible. Multimaster replication, that would be, no?
[16:18:38] <StephenLynx> however if you want to completely overwrite the destination, you might not want to.
[16:19:13] <StephenLynx> afaik mongorestore will just delete everything on the destination before writing, if there's anything there
[16:22:07] <kurushiyama> StephenLynx: Uh, I am not too sure about that. There is a "--drop" option, after all. Iirc, as long as there is not dupe violation, restore will happily add to an existing collection.
[16:24:02] <kurushiyama> Ah, here it is. By default, dupes are simply ignored and restore carries on, unless "--stopOnError" is set. At least this is how I interpret it.
[17:09:00] <kurushiyama> ren0v0: You are using mongodb out of its intended purpose? What do you expect? Would you expect a formula one car excel at creating furrows?
[17:11:51] <kurushiyama> Well, one _could_ argue that if a software can be used aside from its intended purpose it has too many features aside from its intended purpose, I grant that.
[17:15:15] <kurushiyama> ren0v0: To be more precise. You have a certain use case, which, as StephenLynx pointed out, has quite a few risks and is most likely far away from what can be considered best practice. Then, you misuse a feature and complain that it does not work as you would like to behave it to fulfill said flawed use case.
[17:23:09] <kurushiyama> ren0v0: Please do not get it wrong. Nobody wants o bash you. But the discussion should have told you that you may have other challenges than the one at hand.
[17:24:21] <StephenLynx> and when one blames the tool before doing a little introspection, people won't think too much before telling the equivalent of "git gud"
[18:07:30] <kurushiyama> StephenLynx: I always feel like I should implement a git subcommand when I read it ;)
[18:14:42] <keldwud> I'm following the tutorial for disabling transparent huge pages (found here: https://docs.mongodb.org/manual/tutorial/transparent-huge-pages/) in centos7 and I've added the init script and added the tuned.conf then ran tuned-adm profile <profile name>. I run into an issue when verifying. enabled shows [never] but defrag still shows [always].
[18:19:32] <keldwud> is it because centos7 uses systemd? I guess the issue lies with tuned (which I have no experience with)
[18:24:08] <keldwud> what's interesting is that the first echo was successful but the second echo wasn't
[18:35:35] <kurushiyama> keldwud: With the init script you are referring to https://docs.mongodb.org/manual/tutorial/transparent-huge-pages/#init-script ?
[18:36:01] <keldwud> I'm researching tuned now to see where maybe it is getting stuck on passing 'never' to /sys/kernel/mm/transparent_hugepage/defrag
[18:36:31] <keldwud> I have tested by manually echoing 'never' to both enabled and defrag and that works fine
[18:37:10] <kurushiyama> keldwud: If you are using tuned or ktune (for example, if you are running Red Hat or CentOS 6+), you must ********additionally********** configure them so that THP is not re-enabled. See Using tuned and ktune.
[18:37:23] <keldwud> and I am using the profile found on the same page under the centos tuned portion
[18:38:03] <keldwud> so after I load the no-thp profile, enabled is set to never but defrag either is not set to never or gets changed back to always
[18:39:26] <keldwud> wonder if maybe there's a setting missing that specifies /sys/kernel/mm/transparent_hugepages/defrag separately from
[18:42:23] <keldwud> or are the changes added to tuned to make sure it doesn't change a setting back to a previous setting?
[18:44:19] <keldwud> it appears that the script is running because /sys/kernel/mm/transparent_hugepages/enabled gets set to never and stays set to never
[18:44:32] <keldwud> it's just the defrag portion that is either not getting set or is not staying set
[18:45:01] <kurushiyama> keldwud: Since tuned does not touch the defrag, I am pretty sure the init script is not run.
[18:45:44] <keldwud> that's what I was suspecting, wasn't sure if tuned accessed the script to make changes or if it did its own thing
[18:46:33] <kurushiyama> keldwud: Nope. What happens if you run the script manually? Maybe there was some problem during c&p. Just please check to make sure.
[18:49:39] <keldwud> so when I run the script, both changes are made but when doing it through tuned it only affects /enabled
[18:50:08] <keldwud> during testing, I manually echo 'always' to both files to make sure I notice when it changes to never
[18:55:14] <kurushiyama> Ok, lets try sth. You can add "transparent_hugepage=never" to the kernel command line in grub.conf
[18:55:51] <keldwud> I didn't want to set it in grub because I didn't want to have to rely on a reboot to make sure it takes place
[18:57:05] <keldwud> I want to make sure I'm understanding correctly, though, /sys/kernel/mm/transparent_hugepage/enabled *does* get set to never when I follow the mongod documentation after I add the init.d script and the tuned profile then run tuned-adm
[18:57:55] <keldwud> but /sys/kernel/mm/transparent_hugepage/defrag does *not* set to never. Should it be getting set to never? I assumed that it should be since the init.d script modifies it and the documentation states that it should show as never
[18:59:41] <keldwud> this is a fresh minimal install of centos07 and installing only mongod, if that helps any
[19:02:34] <keldwud> http://serverfault.com/questions/688392/disable-thp-and-thp-defrag-on-centos-7-ec2-instance
[19:02:43] <keldwud> "The tricky part is that the vm plugin can only configure the /sys/kernel/mm/transparent_hugepage/enabled setting. To disable the /sys/kernel/mm/transparent_hugepage/defrag setting too, I had to create a script that is called by the profile on start."
[19:03:08] <keldwud> so that's what I was suspecting the whole time. that tuned just wasn't changing the defrag setting
[19:04:17] <keldwud> I wonder what the mongod documentation did to get it to work using just the configuration mentioned
[19:05:25] <keldwud> but yeah, it feels pretty hacky to have tuned call a script just to get defrag to turn off
[19:05:35] <keldwud> I'm wondering if at this point I should just have it call the init.d script instead
[19:07:57] <keldwud> I don't understand enough about this portion of linux to know why both need to be set to never
[19:09:36] <kurushiyama> keldwud: Me neither, tbh. I do not see the point in TPH, anyway. It is a feature only there for very narrow use cases, and I can not see why it is on by default.
[19:11:52] <kurushiyama> keldwud: Do not forget to add your findings to the docs with a small explanation. On or two sentences.
[19:20:24] <kurushiyama> shlant: The better question would be "What do I do with a single member replset?"
[19:26:08] <kurushiyama> shlant: Uhm. A single node replset is behaving exactly the same as a standalone instance, except for writing the oplog...
[19:27:25] <shlant> kurushiyama: so you're saying if doing a single member replica, I might as well just do a standalone instance
[19:30:21] <kurushiyama> shlant: However, I stringly suggest using a 2+1 replset to do failover testing.
[20:43:36] <hackel> Is there a way to query multiple fields for the same value without repeating them in an $and? e.g. col.find({/a.(b|c).isActive/: true})
[20:44:52] <kurushiyama> hackel: You could use text searches... ...for text values, what a surprise ;)
[20:46:35] <hackel> kurushiyama: Heh, fascinating concept! It is just a boolean I'm after in this case, and I'm just being my anally DRY self, as usual.
[20:48:38] <kurushiyama> hackel: Although, I have to admit, "true" isn't a very distinct value, and by having all field names written out, it becomes much more clear what you want to do. Much more readable, at least.
[20:51:03] <hackel> kurushiyama: That's fair. In this case, these sub-documents were added as properties of a parent object, with the type identified by the property name, instead of adding them as an array of with the type as a separate field. The latter would have been a much better deisgn, but I'm not in a place where I can do a migration.
[20:52:09] <kurushiyama> hackel: As far as I got it, both models do not seem to be... well, likeable.
[20:54:55] <kurushiyama> hackel: Every time I hear or read array of subdocs, I have to make a concious decision so that my brain does not translate it to "overembedding".
[20:57:23] <hackel> kurushiyama: There's definitely an unnecessarily level of embedding currently. Would you say that it's better to move the child documents to a separate collection instead of an array, even if the relation is always going to be 1:1?
[21:00:27] <kurushiyama> hackel: More often than not, even without knowing the use cases, the answer is a big, fat, red, blinking *YES*!
[21:02:04] <kurushiyama> hackel: There are edge cases, however. a 1:1 makes it worth a closer look, surely
[21:02:57] <hackel> Huh, that's interesting. I was told the opposite, since Mongo's not optimized for joins, blah blah. Definitely will have to keep that in mind on the next project.
[21:05:29] <kurushiyama> hackel: http://blog.mahlberg.io/blog/2015/11/05/data-modelling-for-mongodb/ http://dba.stackexchange.com/questions/134841/how-can-we-ensure-security-and-integrity-of-data-stored-in-mongodb/134858#134858
[21:05:41] <kurushiyama> hackel: The example section of http://dba.stackexchange.com/questions/134898/lots-of-indexes-mysql-vs-mongodb-migrating/134961#134961
[21:10:34] <kurushiyama> hackel: All those translate to a few rules of thumb: 1) If you find yourself having an array of subdocs, stop and think again. 2) If you would need a JOIN in SQL to achieve what you want, you might be able to do it relatively cheap with some redundancy. 3) Document operations are atomic. If in doubt, _document_ events or points in time.
[21:17:49] <hackel> Thanks for the advice. In this case, I wouldn't need the atomicity. What would be a good use case for embedding an array of subdocuments? I'm also doing it one case where I need to store some extra data for a relation, where I might have used a pivot table before.
[21:20:47] <kurushiyama> hackel: A good use case for an array of subdocs? tbh, I have yet to find one. Since one way or the other, it comes with problems.
[21:22:03] <kurushiyama> http://blog.mahlberg.io/blog/2015/11/05/data-modelling-for-mongodb/#the-mongodb-way-with-embedding
[21:22:33] <kurushiyama> Or, more general: http://blog.mahlberg.io/blog/2015/11/05/data-modelling-for-mongodb/#conclusion