Log file Viewer
#mongodb logs for Wednesday the 13th of April, 2016
[00:27:13] <wgreenberg> hi, is there an easy way to add a date field to a bunch of records that's equal to "foo.id.getTimestamp()"?
[00:27:31] <wgreenberg> without calling .find() on the collection and individually updating each field
[02:26:46] <sx> Hi, I did not change any settings, but somehow my collection is allowing for duplicate "_id" values... not sure how to fix.
[02:57:32] <sx> I create a new collection, "test", and then add a blank document. An "_id" is generated as expected. Then I insert another document with the same "_id" and it works fine. Now I have a collision on what I thought was a uniquely constrained index. Is this expected behavior?
[02:59:12] <Boomtime> can you provide all the outputs, in gist/pastebin, including a find showing the duplicates?
[02:59:38] <Boomtime> can you provide all the outputs, in gist/pastebin, including a find showing the duplicates?
[03:01:07] <sx> Boomtime: I was working both in shell and in python shell and in adminMongo web gui, so I will try to reproduce in a single env
[03:52:56] <sx> Boomtime: I figured it out, the admin interface I was using was not differentiating between "ObjectId(xx)" and "xx"
[03:53:29] <sx> but in reality the objectIds were unique but I was able to override them with strings (which I think is normal)
[04:05:31] <Boomtime> @sx: yep, that sort of thing is somewhat common - always use the mongo shell to verify
[04:07:06] <Boomtime> and yes, _id is mostly a field like any other, it has the unique property of being immutable, and it may not be an array, but otherwise it behaves like a regular field
[04:07:38] <Boomtime> (actually, immutable is not strictly unique to _id, a shard key index gains this super-power automatically as well)
[06:14:37] <Spritzgebaeck> hello, it's possible to store the mongo db files in multiple files? our backup guys asks
[06:17:29] <Boomtime> MongoDB uses a path, specified by thew dbpath option, it expects to have complete ownership of the files in that path - there are multiple files for both WT and MMAP storage engines
[06:18:52] <Boomtime> the simplest method of backing up a mongodb instance is to shut it down and copy the entire dbpath content - if you have a replica-set then you can shut down a secondary which should have low/none operational impact
[06:22:52] <kurushiyama> Boomtime: btw: I am not too sure wether you can not change the value of a shard key in a doc. I do not think it is immutable. Though I haven't tested, but it should be possible. However, you can not change the designated shard key.
[06:28:46] <kurushiyama> Spritzgebaeck: Not by default. He can create a tar, though and use http://linux.die.net/man/1/split
[06:29:23] <kurushiyama> Spritzgebaeck: Of course, he should not do a tar on the datafiles of a running instance
[06:29:59] <Spritzgebaeck> kurushiyama: not by default? so i can configure the server to store 2gig files?
[06:30:28] <kurushiyama> Spritzgebaeck: Sorry for not being clear: No, there is no way to configure the server accordingly.
[06:31:47] <kurushiyama> Spritzgebaeck: I am not aware of a halfway modern filesystem, though, which can not deal with file sizes by far exceeding most storage capacities of today.
[06:35:04] <Spritzgebaeck> i think it's more to backup only one changed file and not the hole collection file, but this is not my problem :D so the backup guys have to save all files
[06:36:21] <kurushiyama> Spritzgebaeck: Uhm. Max file size for NTFS is 17TB... And no, they can not simply copy the files over.
[08:21:54] <mroman> for example you have data sets, you analyze them, produce results. but some data sets might turn out to be wrong later.
[08:23:27] <kurushiyama> mroman: So, remove them and rerun analyze? Fail to see why versioning is needed.
[08:39:43] <master_op> hello, is this repository https://github.com/mongodb-labs trusted by the community ??
[08:42:44] <kurushiyama> master_op: Trusted? Yes. Would I test the hell out of the projects from there? Bigger YES!
[08:57:28] <master_op> https://github.com/mongodb-labs/mongo-connector/wiki/Usage-with-ElasticSearch
[08:58:34] <master_op> is the data inserted in mongodb will be copied into elasticsearch or just indexed
[08:59:20] <kurushiyama> master_op: Better to ask there, then. Personally, I suggest https://github.com/richardwilly98/elasticsearch-river-mongodb
[09:03:43] <kurushiyama> master_op: Believe it or not, I _can_ read. It is just that I'd rather suggest using the river if you want to connect to elasticsearch. Mainly since I know it works.
[09:04:20] <kurushiyama> master_op: With an emphasis on "know", which I can not say for the connector ;)
[09:08:33] <kurushiyama> master_op: https://github.com/mongodb-labs/mongo-connector/wiki/Usage-with-ElasticSearch#elasticsearch-indexes-mappings-and-types As I read it, it is only an index ;)
[09:10:31] <kurushiyama> Derick: Rivers were deprecated in Elasticsearch 1.5 and removed in Elasticsearch 2.0.
[12:08:02] <Keksike> I'm updating our servers mongodb from 2.6 to 3.2. How can I make sure that the mongodrivers (our software is in Clojure) are up to date?
[12:32:28] <kurushiyama> Keksike: Maybe you should ask the to maintain a compat matrix. Unless you have verified, I would not continue with the update. Wire proto has changed, iirc.
[12:37:35] <kurushiyama> Keksike: Or maybe you can get a hold of Michael somewhere and ask him directly.
[12:53:15] <aguilbau> Hi, I want to update 2 fields in a document, either one of the other but not both. They can be recognized by their value. Is it possible in a single query ?
[12:56:47] <kurushiyama> aguilbau: I think what you are tyring to do is to combine 2 queries into one. But go ahead, do a pastebin.
[12:59:55] <markizano> I have 2 mongod instances, let's say replSet=myset01, and 2 mongod instances of another replSet=myOtherSet02
[13:00:25] <markizano> in each of the instances, they have databases of the same name, but different collections of data.
[13:01:54] <kurushiyama> aguilbau: What you want is "if value for x is foo, update y to bar, otherwise update different."
[13:02:20] <kurushiyama> markizano: Though it is not the best idea to do so, especially given the price of a cheapo VPS.
[13:03:09] <markizano> kurushiyama: If I had different database names in each of the replSets, would the 1arbiter;many-replSet's work then ?
[13:04:45] <markizano> right - we were debugging a thing last night, and this was unrelated to what we were discussing, but it was a question I was curious about... so my idea that you'd need 1 arbiter per replSet was correct.
[13:05:00] <kurushiyama> markizano: And saving on an arbiter is really, really the wrong thing to do. What are even Softlayers prices for a cheapo VPS nowadays? Like 10 bucks?
[13:05:36] <markizano> it was more along the lines of the mechanics of - does the software allow this.
[13:06:45] <kurushiyama> markizano: The reason why I would not do it is because when the dual use arbiter machine goes down, the chances for both of your replsets to become unavailable increase.
[13:08:11] <kurushiyama> markizano: But one question crosses my mind: Why do you have two replsets for the same DB, even when they don't overlap? I'd rather shard, then.
[13:10:03] <kurushiyama> markizano: Last but not least I am not too much of a fan of 2+1 replsets, anyway. You loose redundancy during maintenance.
[13:44:29] <Ange7> i'm sorry but i already had this error, but i don't understand : i try to update one row in my mongocollection : collection.update({_id: A}, {field: foo}) with option upsert=true but i have this error : duplicate key error collection: db.collectionName index: _id_ dup key : { A }
[14:53:28] <mroman> It started at 40k records per second, dropping to 30k, dropping to 20k now at 19k
[14:55:20] <mroman> maybe it inserted the first few mio into RAM not directly flushing them to disk?
[15:52:38] <hiro`> Hey all. Having trouble getting mongo to start up. I can run it okay as root, but otherwise, `mongod` will fail to properly startup, giving me errors about "wiredtiger_open: /data/db/WiredTiger.lock: Permission denied"
[17:54:05] <Gloomy> I'm trying to manipulate a mongodb object client-side, any idea why it looks like this? https://www.dropbox.com/s/qtganali7kgcbwy/Screenshot%202016-04-13%2019.45.34.png?dl=0
[17:55:41] <kurushiyama> Gloomy: Ok, my first advice then: Ditch all graphical tools. All and any. Use the mongo shell. Nothing else.
[17:56:54] <Gloomy> Hmm, I'm building a website, kind of hard without a browser :-) that's a snapshot from firebug, maybe I should have begun by stating that
[17:59:15] <StephenLynx> in the future, use the terminal client with either findOne() or .pretty() to format the output of the data you want to show and paste the text somewhere
[18:00:28] <kurushiyama> Gloomy: Do not take it wrong, but not all of use a MEAN(-ish) stack. Personally, I avoid EAN like the devil holy water.
[18:00:47] <StephenLynx> I use node and mongo a lot ,but I avoid express and angular like the plague
[18:01:16] <kurushiyama> Gloomy: So, in order to enable us to help you, we need a lingua franca of some sort – and that is shell output.
[18:02:14] <Gloomy> Well that's the thing, I don't know where exactly things get messed up. The data seems to be stored fine in the database, but the object structure is a bit weird once it gets out from ... mongo? monk? express?
[18:03:00] <kurushiyama> Gloomy: Ok, have you _every_ connected to your MongoDB instance using the shell?
[18:04:27] <StephenLynx> node's api is already a high level tool for you to work, so as the browser JS api.
[18:04:46] <StephenLynx> that makes both of them redundant, but now you are adding a whole new layer of stuff to break
[18:05:52] <StephenLynx> but you can say the same for any other web framework for scripted runtime environments aimed at web.
[18:06:21] <kurushiyama> Gloomy: Ok, to join the choir. I am kind of with StephenLynx here. People take MEAN and think it will be easy to implement something and basically it will do itself. Nothing could be farther from the truth, imho, since it decouples you from what it is going on in the background. Is it possible to achieve awesome results with MEAN? It sure is. But only if you know your tools.
[18:07:04] <StephenLynx> that too, people skip the fundamentals of the tools they are using and their software end up as black boxes to themselves.
[18:08:11] <StephenLynx> the whole idea of full stacks is retarded, too. it makes no sense to talk about the front-end and the database at the same time.
[18:08:12] <kurushiyama> Gloomy: Ok, enough bashing. First, we need to see your structure. Please pastebin a sample document. You can get one with db.yourcoll.findOne().pretty()
[18:08:17] <Gloomy> Here is the text version directly from the mongo shell. looks similar to what I had in Firebug, it's just not the Json object structure I'm used to?
[18:12:21] <Gloomy> And yes, I see what you mean (no pun ;-) ). I find it much easier to learn by breaking things though than going through the whole theory first)
[18:15:44] <kurushiyama> cheeser: Why sadly? at least 50% cut in performance sounds pretty severe to me.
[18:16:29] <Gloomy> I was using monk. There seems to be something wrong with (how I'm using) it indeed :-)
[18:17:04] <kurushiyama> Gloomy: Really, do yourself a favour: Use the plain driver to get a hang of it.
[18:17:17] <cheeser> you're extrapolating your experience to the broader world and that's just foolish.
[18:18:58] <cheeser> well, it's not just mine. thousands upon thousands of apps use them successfully.
[18:25:29] <kurushiyama> Gloomy: If by any chance you can use a Linux shell, I'd start out with M102
[18:27:57] <Gloomy> Ok, perfect. One last question, I have no knowledge whatsoever on databases. Is learning a noSQL DB as first approach detrimental in any way?
[18:28:52] <kurushiyama> Gloomy: Imho, it is better to learn a few NoSQL databases first, and than SQL, which imho is devils work, anyway (and always was).
[18:30:26] <kurushiyama> Gloomy: I tend to suggest MongoDB for doc databases (unsurprisingly), Cayley as a graph database (since it build up on your MongoDB knowledge), Redis as KV and Cassandra, which is sort of a type on its own.
[18:39:27] <uuanton> anyone knows how to reload: Job for mongod.service failed. See 'systemctl status mongod.service' and 'journalctl -xn' for details ?
[18:42:18] <kurushiyama> uuanton: Sorry, typo. "Logs or it did not happen" is a variation of "Pics or it didnt happen".
[18:42:54] <kurushiyama> uuanton: A futile try to humurously convince you to give a bit more info. ;)
[18:45:40] <kurushiyama> uuanton: Really, hopefully the old log was rotated. Have a deep look there.
[18:46:34] <kurushiyama> StephenLynx: It has its advantages. Though it is NOT relational. Well, it is as relational as MongoDB.
[18:50:55] <kurushiyama> StephenLynx: Well, not in the RDBMS sense, but you can relate data appside, right? And within limited capabilities, $lookup does help.
[18:51:14] <cheeser> the database supports references. it just doesn't enforce referential integrity.
[18:52:09] <kurushiyama> Anyway, you can have similar approaches on cassandra. And the query language is SQLish
[18:56:20] <kurushiyama> StephenLynx: See ;) But does SELECT * FROM numberOfRequests WHERE cluster IN ('cluster1', 'cluster2') read familiar?
[18:57:19] <StephenLynx> I am not interest in how I interface with the db, but on how the db operates internally.
[18:59:02] <kurushiyama> StephenLynx: That was not really clear to me. There was a good screencast on why SQL clustering is pretty hard to do... Can't remember really where I saw it. Darn
[19:01:17] <cheeser> that's one reason mongodb doesn't (really) support joins yet. $lookup the notable exception
[19:01:49] <kurushiyama> cheeser: I haven't tried $lookup yet, but I have seen horrible reports regarding performance.
[19:03:38] <kurushiyama> cheeser: I'd guess that $lookup is best to be used when you already have narrowed down to (very) few docs and can save a $in query by using $lookup.
[19:04:18] <kurushiyama> Well, as it was to be expected, people misuse it generally to emulate JOINS.
[19:04:27] <StephenLynx> >Performs a left outer join to an unsharded collection in the same database
[19:04:58] <cheeser> i was just wondering about support in sharded envs. i couldn't remember if we restricted that or not.
[19:05:28] <kurushiyama> StephenLynx: I do some SO for fun... ...and yeah, I am a bit morbid... You would not believe what people are complaining about...
[19:07:48] <kurushiyama> My current fav: "I do an aggregation on 7M docs, match 45k docs, generate a new field, sort by that field and all that takes 8 seconds!"
[19:14:21] <kurushiyama> kexmex: So unless those 45k docs happen to be in the working set, there are quite some read ops involved.
[19:21:18] <kurushiyama> Or to put it different. Selecting 45k docs out of a set of 7M, missing docs read from disk, then sorted by an unindexed field which had to be created by an operation before the sort could even start. ;)