Log file Viewer
#mongodb logs for Wednesday the 8th of June, 2016
[00:47:35] <Forbidd3n> Hey everyone. Quick question. I am loading a csv file in into multiple collections. If the first few rows of the csv file are duplicate, would it be best to create an array of all the data and then loop the array to insert data and only do one check if line exists or would it be alright to do the calls to the collection to see if the value exists, if so then grab id for next collection insert?
[00:48:49] <Forbidd3n> School,Subject,Grade,Teacher - the school, subject and grade are repeated for each different teacher name
[01:05:00] <sumobob> quick question, I have a model with an array of objects with start: Date, and end: Date, given another start and end date how can i find all the models where the query start and end are not in range
[01:09:49] <Forbidd3n> Hey everyone. Quick question. I am loading a csv file in into multiple collections. If the first few rows of the csv file are duplicate, would it be best to create an array of all the data and then loop the array to insert data and only do one check if line exists or would it be alright to do the calls to the collection to see if the value exists, if so then grab id for next collection insert?
[01:09:51] <Forbidd3n> School,Subject,Grade,Teacher - the school, subject and grade are repeated for each different teacher name
[07:22:40] <chris|> when I stop the balancer, will that interrupt a currently active balancer run or will the balancer complete the current run and then not get scheduled again?
[07:25:27] <kurushiyama> chris| The other way around. If you try to stop the balancer, this operation will block until the current balancer run is done. https://docs.mongodb.com/manual/tutorial/manage-sharded-cluster-balancer/#disable-the-balancer
[07:28:31] <chris|> kurushiyama: stopping the balancer is documented for sharded clusters: https://docs.mongodb.com/manual/tutorial/backup-sharded-cluster-with-filesystem-snapshots/#disable-the-balancer
[07:29:49] <kurushiyama> chris| Ah, FS snapshots. Well, in this case it makes sense. However, in the long run, I rather suggest to set a balancer window and take the snapshots automatically _outside_ said windows.
[07:46:53] <cppking> hello, my mongodb cluster balancer is not working , https://bpaste.net/show/258289e040b4 anybody help me ?
[07:54:12] <chris|> the docs say "operations have propagated to the majority of voting nodes" for majority write concern. That seems strange to me, should that not read "majority of replica set members" ?
[07:57:59] <cppking> kurushiyama: you mean balancer in normal state? but current time it is not working ?
[07:58:37] <kurushiyama> cppking It has errors. While this surely is a thing to monitor and debug, it does run and already has done some balancing.
[08:00:55] <kurushiyama> cppking " this surely is a thing to monitor and debug" Is there something unclear in this sentence? ;P
[08:01:23] <hentis> We're using TTL expiry on some collections, but I need to be able to restore old backups, without loosing that data. As part of our DR policy we use mongodb cloud bacup services, which restores data to /data. Is there a way to run collmod or disable the TTL expiry so we can run collmod to remove the expiry from the index until such time as we want to expire the data manually ?
[08:02:52] <kurushiyama> cppking That depends on your configuration and OS. Please see the docs for that.
[08:04:03] <hentis> Or can the background process that expires the data be disabled without having to collmod the index ?
[08:04:16] <kurushiyama> hentis Uhm... not sure, but shouldn't it be "konnichi wa, Kurushiyama-san". My Japanese is worse than my French, and that means something. ;)
[08:06:30] <kurushiyama> hentis If it is a temporary thingy, I would probably disable the TLL monitor altogether: https://docs.mongodb.com/manual/reference/parameters/#param.ttlMonitorEnabled
[08:09:46] <hentis> kurushiyama: haha my japanese is very crappy .. been way to long since I've active used any of it ... nevermind typed it. thanks for info :)
[08:43:29] <Ange7> is it possible to group by regex with mongo ? i have documents with id field which contains for example : AA12345, AA54321, BB12345, BB54321... i want count number id started by « AA » and count number id started by « BB »
[09:02:23] <eugenmayer> hello. Suddenlty, installing mongdob 3.2.7 under wheezy does not work, since it depends on libc >= 2.14 and wheezy does only have 2.13 - how can this happen since this packages are wheezy specific ?
[09:02:46] <eugenmayer> i am using deb http://repo.mongodb.org/apt/debian wheezy/mongodb-org/stable main as the source
[09:03:45] <eugenmayer> following this guide : https://docs.mongodb.com/manual/tutorial/install-mongodb-on-debian/
[09:06:20] <eugenmayer> seems like 3.2.6 is going well with wheezy, while 3.2.7 does not. Is 3.2.7 jessie only - eventhough there is no support for jessie yet ( according to the docs? )
[09:59:09] <ramnes> just interested if you guys are going to consider this as an issue or not, and if not, how you'll justify
[10:09:12] <kurushiyama> Funny post: https://engineering.meteor.com/mongodb-queries-dont-always-return-all-matching-documents-654b6594a827#.uhdnfhnzd
[10:10:44] <kurushiyama> I immediately tried to beat the shit out of my test cluster, with not a single operation failing.
[10:20:28] <kurushiyama> ramnes Just came in. What was it? I am still giving myself a hard time to reproduce the problem, with 10M docs and 10 concurrent goroutines modifying the data, and an equal amount of goroutines reading the data...
[10:21:37] <ramnes> kurushiyama: not much, I just asked if MongoDB was going to do anything and Derick said that he flagged it internally
[10:53:22] <kurushiyama> @Derick Well, I'll let you know when I got my little test project finished. It generates noise and reordering on purpose, and very much so.
[10:57:41] <Derick> https://en.wikipedia.org/wiki/Isolation_%28database_systems%29#Phantom_reads is interesting too
[11:02:12] <kurushiyama> Derick Another thing I have thought of (because the technical details were not disclosed) is that he may have run into SERVER-3645 in some way. But admittedly, I haven't had the time to think it through, yet.
[11:04:12] <Derick> kurushiyama: it'd be great if you can publically post your musings at some point :)
[11:07:45] <kurushiyama> Derick Ah! You bet. That is the reason why I started the project. Actually, it is part of a bigger project, called mongodb-bc. Go figure ;P
[11:10:18] <Zelest> kurushiyama, mongodb stores lots of stuff in RAM and doesn't fsync! so if the power cuts, I lose my data!
[11:14:31] <kurushiyama> tl;dr No, you do not, since if you need to make sure that data was persisted, you should have a proper write concern, only after which is satisfied the operation returns.
[11:51:37] <eugenmayer> https://packages.debian.org/de/wheezy/libc6 .. max 2.13 and no backports. Upgrading libc6 without backports is madness, so i do not think this is suggested
[12:07:18] <kurushiyama> Derick Hm, I was able to reproduce the problem when doing a count on the cursor.
[12:21:30] <sector_0> kurushiyama, so the shell is the accepted way to administrate a MongoDB server?
[12:23:56] <kurushiyama> Derick But as far as I can see, count actually utilizes the index. So, imho, it is not the document which is in undefined state, but the index.
[12:25:56] <kurushiyama> Derick Which should be provable by doing an iteration over the query result, no?
[12:30:26] <kurushiyama> urbanizator The locked _field_ (terminology is important, there is no such thing as a column in MongoDB) is one you define, yes.
[13:10:04] <kurushiyama> ramnes Derick This is the first version. Still has some problems, but it proves that it is a problem of .count() https://github.com/mwmahlberg/incomplete-returns/blob/master/main.go
[13:13:34] <kurushiyama> I do not trust count() too much, anyway, since as of now it is rather useless in sharded environments (see SERVER-3645) for details. As far as I can see, it is only the index which might be in an undefined state. The program freezes all modifying operations as soon as a mismatch of expected documents and the number returned by .count() is found. As of now, iterating through the result set always gives the correct number of documents.
[13:14:09] <Derick> kurushiyama: for that, you need to talk to one of the server guys really - I don't know
[13:14:56] <kurushiyama> The programs has some locking problems, which I need to debug. Once a mismatch is found, the locking causes basically every count to fail in a way. But that may be worth a look in itself.
[13:15:21] <kurushiyama> ramnes I can reproduce that .count() might return wrong information in write-heavy environments.
[13:18:12] <kurushiyama> ramnes Well, the author did not disclose how he measured the problem. My guess is that he used .count()
[13:22:02] <kurushiyama> I don't know. He might simply be mislead in a way that he assumes .count to be reliable.
[13:23:18] <kurushiyama> Derick I will spice up and document the code a bit. Is there a way to reach out for your "sever guys"?
[13:29:49] <jayjo_> I have figured out that aws cli or s3cmd cannot stream recursively (at least effectively). What would be the workaround for a problem like this? Do I need to write a script that recursively finds files in the bucket and streams to mongoimport?
[13:34:17] <kali> jayjo_: command line interface tools for s3 are often not that helpful for scripting. many scripting languages have better interface (python, node, ...). whatever you're confortable with
[13:55:36] <kurushiyama> As for the verification of MongoDB having an undefined state under certain conditions. Well, I actually found out that I did not have a locking problem at all. What happens after the first count() returns a wrong value, all concurrent writers are paused. As soon as the result set is verified, those concurrent writers are unpaused _simultaneously_ (almost), resulting in a lot of concurrent writes again, causing .count() to return a wrong
[13:55:36] <kurushiyama> value and it restarts from there. So, my conclusion holds and is even enforced: It is a problem of the way .count() behaves, not in the result sets.
[14:01:30] <kurushiyama> ramnes As soon as I have updated the code, you can download and run it against your env, if you want to.
[14:10:56] <kurushiyama> ramnes Assuming you have installed Go and a proper GOPATH set and all, it should work with "go get https://github.com/mwmahlberg/incomplete-returns", after which $GOPATH/bin/incomplete-returns would be the executable. Append "-h" for the options.
[14:12:45] <kurushiyama> Well, I might update it, so you better run "go get -u ..." once in a while.
[17:05:10] <cheeser> probably the list would be better. i don't think any of the devs are here and most of the users here use other languages
[17:33:40] <_1009> Hey all - just wanted to check: for shards, is it recommended to put config nodes on a dedicated node? I noticed that deployments in MongoDB Cloud do so by default. If that is the case, why?
[17:37:55] <GothAlice> _1009: Configuration servers are queried by mongos query routers to determine which actual data nodes contain the data being queried. It's important that these answers are fast.
[17:51:06] <_1009> OK, thanks - in my case, all data is sharded by id, but I'm always searching by geolocation - does it still hit the config router then, as it will query all nodes for (their part) of the geolocation?
[17:59:29] <kurushiyama> _1009 Shard key not in query => monogos has no clue about the key ranges of the queried field => Query is sent to all shards.
[19:23:33] <speedio> i have the following mongoose schema for my mongo db http://dpaste.com/0FEPM52 , and im trying to add data to it with the following command http://dpaste.com/2DWQVA6 , but for some reason the nested objects wont get added.. what am i doing wrong?
[19:30:17] <speedio> ok, ive setup my api to use mongoose now so that would mean alot of work todo.. will keep that in mind for my next project..
[19:34:26] <alexi5> whats up with the nodejs community been hear some bad new s about a library consisting of only a string padding function that broke alot of apps
[19:39:02] <StephenLynx> and used some "no feefees hurt" rule to forcibly transfer a module to a company
[19:44:01] <StephenLynx> https://medium.com/@mproberts/a-discussion-about-the-breaking-of-the-internet-3d4d2a83aa4d#.qm1qa0jf1
[19:44:23] <StephenLynx> >Our goal is to make publishing and installing packages as frictionless as possible. In this case, we believe that most users who would come across a kik package, would reasonably expect it to be related to kik.com. In this context, transferring ownership of these two package names achieves that goal.
[19:44:48] <StephenLynx> npm never mention anything about "hm, yeah, that could give us legal trouble"
[20:28:27] <Forbidd3n> How do I store _id in another collection->document in mongo? refId = new MongoId('the id of the reference I want to store') or do I just store this - refId = 'the id of the reference I want to store'
[20:31:51] <alexi5> have you guys heard of developers making statements to use postgresql to store document data given that it has json and jsonb field types that can be queried ?
[20:35:19] <alexi5> what I like about mongodb is the ability to map application objects to documents easily
[20:36:20] <alexi5> what I relized is that systems can use both a document database and relational based on their requirements instead of staying with just one
[20:56:56] <Forbidd3n> StephenLynx: sorry for my ignorance, but can't seem to find this in the manual - can you show a reference to it, please?
[20:57:55] <Forbidd3n> ok so when I create the index for the collection use dot notation and that will make the sub nested item unique?
[21:00:23] <Forbidd3n> If I have a list of ports for example and many ships go to the same ports, would it be best to put the ports in its own collection and reference it by id versus duplicating the port names in each document
[21:00:50] <Forbidd3n> I am guessing that is best to have separate port collection so if the port name changes I only have to update it in one place versus all documents that have that port name
[21:12:30] <tuvo> sounds bad: https://engineering.meteor.com/mongodb-queries-dont-always-return-all-matching-documents-654b6594a827#.lmjvuetgb
[21:38:10] <kurushiyama> tuvo https://github.com/mwmahlberg/incomplete-returns actually proves that this is only a misbehaviour of the count method, which was presumably used to count the number of results.
[21:38:49] <kurushiyama> tuvo when you look at the channel logs, I have made some statements regarding that, too
[21:39:02] <tuvo> kurushiyama: "I have run extensive tests" doesn't mean much. He had huge input, lots of traffic, happened rarely.
[21:40:04] <kurushiyama> Actually, this programs punches the crap out of any installation you can find.
[21:40:29] <kurushiyama> I can easily reproduce a count mismatch, but the cursors are always in the correct size.
[22:06:49] <Doyle> Hey. is the db.metadata.json needed to restore? It's not included in stdout of a dump. Thanks
[22:13:22] <Boomtime> @Doyle: https://docs.mongodb.com/manual/reference/program/mongodump/#cmdoption--out
[22:13:33] <Boomtime> "When writing standard output, mongodump does not write the metadata that writes in a <dbname>.metadata.json file when writing to files directly."
[22:45:31] <Forbidd3n> Do I create an ObjectID on the nested item and store it on the referencing collection document(s)?
[22:56:52] <Forbidd3n> If I have to have a nested items reference documents in another collection would it be best to have the nested items in its own collection or add id to the nested items and add id to other collection documents
[22:59:53] <Forbidd3n> Boomtime: for example. I have ships, lines, ports, itineraries. I was going to embed ships into lines and link each line to port and then link port and link ship and port to itinerary
[23:00:31] <Forbidd3n> The reason I have ports as another collection is because I didn't want to duplicate the port name in each line item embedded
[23:00:58] <Forbidd3n> I can embed itineraries, but need to link the embedded itinerary to the port by id
[23:02:37] <Boomtime> and is that frequency of change worth the extra effort of having to look up port names on every query?
[23:03:23] <Forbidd3n> probably not, but is it good practice to repeat data in each document embedded port object?
[23:04:01] <Boomtime> 'good practice' is another way of saying 'people always did it this way and i'm not sure why'
[23:04:42] <Boomtime> what is the advantage of avoiding putting the port details directly in the lines?
[23:04:44] <Forbidd3n> well if I put the same name on all cruise line object items and if the name changes I would have to filter through and update them all
[23:04:57] <Boomtime> you can still have the port collection if you like as 'master' reference or something
[23:05:26] <Forbidd3n> correct, I was thinking of having a collection of ports with _id and link to it
[23:05:54] <Forbidd3n> the way it is laid out is a cruise line has many ships, ships go to many different ports, itinerary is the date at a specific port
[23:06:23] <Forbidd3n> port name changes don't change, but where they go in the itinerary, the port, may change so the id of the port in the itinerary will have to be updated
[23:06:50] <Forbidd3n> are at least not often, rarely do they change, but of course anything can change
[23:07:00] <Boomtime> so it doesn't matter if the itinerary contains port names and other details, or _ids are well because those need updating anyway?
[23:07:34] <Boomtime> meanwhile, embedding only the _id of the port means you now have to go look up the details in another collection, every query
[23:08:34] <Forbidd3n> so it comes down to either update all port names in the itinerary if it changes or do the extra query to get the port based on the id
[23:08:37] <Boomtime> rather than trying to avoid having the same name more than once, despite it rarely ever changing anyway
[23:09:09] <Boomtime> "so it comes down to either update all port names in the itinerary if it changes" <- if what changes?
[23:09:46] <Forbidd3n> then I would have to do a find with the old port name and update each item that contains it
[23:10:30] <Boomtime> so, worse case condition is you have to "filter through and change the port names" once per day - oh noes!
[23:10:31] <Forbidd3n> the port name itself doesn't, but the day/itinerary may change the port it will be at
[23:11:39] <Boomtime> if the information is too big to fit, or produces an array that might grow unbounded, then embedding is bad
[23:11:46] <Forbidd3n> it is an slim api where the user can get ships based on line, itineraries based on port or line
[23:13:00] <Forbidd3n> Boomtime: so the schema should be designed based on queries versus memory used to update
[23:14:44] <Boomtime> i don't know, i'm not going to go through your whole schema - i called out one detail to hopefully get you to start thinking differently
[23:15:39] <Forbidd3n> Boomtime: just trying to get an better understand the line between when to use additional collections versus embedding
[23:15:40] <leptone> how can i .find() items in my database by a subproperty of a property of the document instance?
[23:15:50] <Boomtime> queries are almost always the vast majority of the work that a database does - it's non-sensical to optimize writes by avoiding duplication and using references etc, at the cost of query complexity
[23:15:53] <leptone> id like to do something like this: https://gist.github.com/daaceab73f70c9c70500bec80a254fc5
[23:16:44] <Forbidd3n> Boomtime: I am coming from a sql background where you don't duplicate data and use foreign keys to normalize data
[23:17:07] <leptone> but im gettin this error https://gist.github.com/leptone/39c3ecddf2e0eca3a9557684a0cb87d1
[23:17:35] <Forbidd3n> it appears, based on what you are saying, is it is ok to duplicate data in collections and documents
[23:18:42] <leptone> it appears to work even if its not vaild JSON as long as it a direct property im querying for
[23:20:00] <Forbidd3n> Boomtime: also is it efficient when the user queries the api for cruise lines, it pulls the entire data even if all tehy want is the line name itself
[23:22:17] <Forbidd3n> so in your opinion based on my conversation would you say I would create a new document for each line, embed ships and in each ship embed itineraries
[23:27:15] <leptone> Boomtime: so i tried querying for JSON.stringify(queryObj) but it doesnt seem to fix the issue