Log file Viewer
#mongodb logs for Friday the 29th of April, 2016
[00:56:06] <oky> kurushiyama: cool! i was curious what software you are using for plotting the analytics / query exploration, or what exists already
[00:59:13] <oky> kurushiyama: i was thinking in terms of stuff like kibana, caravel or other query builders / explorers for other DBs
[01:00:44] <kurushiyama> oky: Uhh, iirc, there are some. But I am just kind of sleepless, and my brain probably does not work properly, yet.
[01:01:47] <kurushiyama> oky: cloud9charts might be good if you need sth quick and dirty (ond have nothing to hide).
[01:08:26] <kurushiyama> oky: That was what was lurking in my mind. I saw a presentation of JasperSoft during MongoDays Germany 2014. Looked very promising back then.
[01:08:57] <oky> thanks for the list, i'm looking at them now - i see they are SaaS / paid products?
[01:10:21] <kurushiyama> oky: Well, but they have a much larger audience and can much easier monetize.
[07:46:02] <aps> Boomtime: Really? So If client forgets or chooses to add a high value, mongo's screwed?
[07:46:12] <Boomtime> @aps: look like somebody else wants that feature too; https://jira.mongodb.org/browse/SERVER-13775
[08:03:35] <goyalr41> I wanted to know is there a way to append to a field's string value while doing update?
[08:16:17] <Boomtime> @goyalr41: there are currently no remote string manipulation update operators - arrays, numbers, and bitwise manipulation operators are available though
[08:25:48] <Zelest> kurushiyama, so.. i got my 3.0 cluster running now (havn't moved to 3.2 yet) .. though, it runs the old storage engine (non-wiredtiger) .. would it be a good idea to upgrade/change to wiredtiger before I migrate to 3.2?
[08:32:33] <Derick> the data format for mmap and wt is totally different, and a node won't start up in wt mode if there is mmap data available
[09:10:25] <Zelest> Derick, well, the idea was to shutdown a 3.0 node, remove the data, enable wt and start it up again.. :)
[09:17:11] <Zelest> as in, i sitll have 2.6.7 nodes in the replicaset, so I can't go to 3.2 just yet.. need to migrate them/remove them first :)
[09:18:38] <Ange7> just one question : http://pastebin.com/tsmE5sKi If i want calculate average of count per uid per date i don't think my aggregate query is wrong... :/ why ?
[09:55:02] <Zelest> Euhm, how can I lower the log verbosity? Says the default is 0, but it seems to log more or less everything :(
[09:56:26] <Zelest> can't i just disable some of it? also, why is https://docs.mongodb.org/manual/reference/configuration-options/#systemLog.verbosity so confusion?
[09:57:26] <aps> I am getting some very long operations. I turned on the profiling and got this - https://www.irccloud.com/pastebin/kn0wU0KS/
[10:06:07] <kurushiyama> aps: I think it is wrong from the start. The whole concept is wrong in my book. And let's say the models Mongoose creates tend to have their own style. Mongooses signature field '__v' for optimistic locking (done wrong, if I get it right from that query) does not make it harder. ;)
[10:08:40] <kurushiyama> aps: Use the raw driver. Get your user stories right. Derive the questions you need your data to answer from it. Model accordingly, optimized for the most common use cases.
[10:09:17] <kurushiyama> aps: It is not that Mongoose is wrong at all. It just makes it way to easy to do it all wrong.
[10:09:47] <Erni_> I mean, do I need to construct the URI with the private IP´s AWS gives to the mongodb EC2 instances?
[10:10:34] <Erni_> do I need to do this with the private IPs? https://docs.mongodb.org/manual/reference/connection-string/
[10:11:32] <kurushiyama> Erni_: Aye, that should work, if you can reach each replset member via shell from your application's machine.
[10:12:43] <kurushiyama> aps: And if you do proper data modelling, there is very little advantage in using Mongoose.
[11:08:32] <Erni_> kurushiyama what if one of the nodes in the replica set goes down? it then starts with a new IP address and therefore not visible to the application?
[11:10:18] <kurushiyama> Erni_: uhm, that is definetly something you need to prevent. Iirc, elastic IPs is how it is called...
[11:18:19] <Erni_> I know what a VPC is, in fact I deploy each replica in a different subnet in the VPC
[11:18:46] <Erni_> but I don´t understand how that solves the problem of the new IP when a replica goes down
[12:17:01] <livingBEEF> Hi. Docs say that db.collection.copyTo() is deprecated but I don't see it mentioning what it is deprecated in faver of... or is it just deprecated without any replacement?
[12:57:59] <arbitez> hi all. has anyone converted mongodb databases to postgresql here? need some tips on how to mass convert dbs
[12:59:05] <arbitez> ask on #postgresql earlier, got some ideas, but a bit of work, so asking here as well
[13:15:49] <crazyphil> do I need to worry about a [UserCacheInvalidatorThread] warning? I can't seem to find much info about it
[13:18:14] <oky> arbitez: so, going from unstructured data to a table scheme? or using postgres jsonb columns?
[13:26:49] <arbitez> so i suppose a small "read-all-collections, but into json cols in respective tables" should be easy enough if no tool exist
[13:28:08] <oky> f.e. you have table "Foo" in mongo with { bar: baz, honk: dazzle}, you'll get Foo table in postgres with one column (call it Blob for now) that contains the JSON?
[13:30:47] <oky> arbitez: i guess you would also want to get rid of the mongo BSON id or whatever it is when transfering the blobs to postgres
[13:32:50] <oky> arbitez: fwiw, i stored JSON data inside postgres JSONB columns - the queries are a pain to write ;)
[13:33:15] <owg1> I made a blog post about Mongo + ZFS: http://www.clock.co.uk/blog/mongodb-performance-on-zfs-and-linux
[13:33:16] <oky> arbitez: also, check if there is indexing available for JSON blobs - might be nice to add indeces
[13:35:48] <arbitez> oky: write performance in our case, on indexed jsonb, is two orders of magnitude faster than mongodb, which explains our switch :)
[13:38:57] <oky> arbitez: using JSONB or JSON? (supposedly JSONB has some small overhead during write, so it's even more amazing if that's the one that's much faster than mongo)
[13:39:35] <arbitez> oky: jsonb. It's not that postgresql writes are so fast, it's that mongodb writes are so slow
[13:40:58] <oky> arbitez: i wonder if you'll be able to horizontal partition with your postgres (or if you are already doing that)
[13:41:33] <oky> (citusDB has been looking cool, in that respect - i'm uncertain how that works with JSON)
[13:43:00] <oky> that's so cool, tbh - the simple horizontal sharding + support for JSON + support for column based storage
[13:46:19] <arbitez> and we have this statistics system that works on relational data, and the current setup with copying mongo data off of to a rdbms is hopeless
[13:46:21] <Derick> as to your original query of converting to postgresql, they're different types of databaes, with different data models. There is no magic to convert between them.
[13:56:06] <Derick> JSONP is more of a single blob, where you can index fields from. Whereas Mongo's BSON really still consists of the individual fields allowing for much greater query flexibility
[13:56:40] <oky> Derick: do you mean JSONB? i'm familiar with JSONP in the context of browsers and cross-domain JSON
[14:01:57] <oky> Derick: i don't remember the exact semantics, i have a query builder i use, but it's something like: SELECT * FROM <db> WHERE blob::json->'b' > 5;
[16:26:48] <webm> Hey, I'm new to this web scale stuff. Can you mirror two server databases with mongodb? Aka two servers with mongodb installed sharing a database?
[16:29:49] <crazyphil> do I need to worry about a [UserCacheInvalidatorThread] warning? I can't seem to find much info about it
[16:37:23] <StephenLynx> there is no way to scale how large of a maintenance nightmare that would be
[17:36:28] <kurushiyama> webm: I can not even count on how many fronts it is wrong to assume that a data river (which basically is a real time ETL) has the slightest thing to do with MongoDBs notion of replication. StephenLynx is rather guilty of understatement than exaggeration.
[17:43:08] <kurushiyama> webm: If you really need semi-structured data and SQLish access, use Cassandra.
[17:43:34] <webm> not my choice, my boss called me and asked if mirroring to an external mongodb server was possible
[19:10:16] <webm> I'm just thinking I'm on the wrong track here, the data we have is already relational and doesn't need to be non-relational.
[19:11:03] <webm> It's just that this other company needs some of our data, and they use MongoDB. So maybe we should just force them to use a second engine or something.
[19:13:16] <kurushiyama> webm: Almost all data is relational. Take graph databases, for example. There is not much more than relations. Would you call it a relational database?
[19:16:58] <kurushiyama> webm: Writing a data access API given all the tools and frameworks should not take more than 2 or three work days.
[19:18:15] <kurushiyama> StephenLynx: Well, we all know your ... fondness... towards frameworks. Especially Mongoose, iirc? ;)
[19:48:12] <alanoe> hi. How can I get a value from DictField in mongoengine? I have tried field['a'] and get 'DictField' object has no attribute '__getitem__'
[19:50:12] <StephenLynx> a couple of people here use mongoengine, but you could have a better change at somewhere dedicated to it
[20:06:20] <kurushiyama> StephenLynx: Hm, regarding Mongoose: let's say that you can do data modelling the right way with Mongoose and that it makes it faster. Then there should be a point with the maximum gain, with the smallest overhead, no?
[20:12:04] <StephenLynx> even if you were to save something like 80 hours, what is 2 weeks in 2 years?
[20:12:23] <StephenLynx> and if you were to save 80 hours by using an ODM, this person should look for a new job.
[20:13:17] <StephenLynx> you are going to waste much more time by having to maintain code that is more complex.
[20:32:08] <starfly> although it can be the case, a bigger overall stack doesn’t always translate into slower performance and complexity, that approach can be used to create additional, targeted code sets to minimize code execution paths for high frequency operations (as an example)
[20:33:57] <StephenLynx> only if the used stack for a given operation differs from the used stack for a different operation.
[20:36:43] <starfly> yes, I wasn’t suggesting redundancy, there are just reasons (at times) for logical code organization that can often result in a bigger stack
[20:37:42] <StephenLynx> you just have a more complex system where each scenario uses a diffent stack
[20:51:46] <Ryzzan> let's say i have a String variable like var string = "123, 543" and I want to use it with $in in find, like db.collection.find({_id : $in : [ string ]})
[21:41:05] <kurushiyama> My new fav question title about MongoDB on SO: "How to make two successives synchronous queries on collections in MongoDB?"
[21:55:39] <kurushiyama> catbeard: Well, I'd probably do some runs without any indices, To have a complete list, so to say.
[23:16:46] <Lonesoldier728> Hey so I am using mongoose with Nodejs and trying to figure out how I can store DateType as 0 - I do not want it to be blank is there a way to have a dateType that is 0?
[23:39:08] <Boomtime> @Lonesoldier728: MongoDB Date type is just a 64bit integer under the bonnet; https://docs.mongodb.org/manual/reference/bson-types/#date
[23:39:36] <Boomtime> however, a value of zero maps to the Unix epoch, which is a perfectly valid date; the first second of 1970
[23:49:58] <Lonesoldier728> It is a cool friend to have, just kidding it is the one field I can just leave it blank