Log file Viewer
#mongodb logs for Tuesday the 5th of May, 2015
[00:00:49] <GothAlice> Apparently. :/ min/max is darned useful. (Note: I don't actually use MongoDB's geo capabilities.)
[00:09:38] <JokesOnYou77> Hi all, I have documents of the from {from: [list of string], topics: [list of string]} I'd like to get an aggregation of {topicName: {address: count}} where address is a unique "from" value count is the sum of how many times that from occurs in that topic. A document may have many topics
[00:30:10] <JokesOnYou77> Hi all, I have documents of the from {from: [list of string], topics: [list of string]} I'd like to get an aggregation of {topicName: {address: count}} where address is a unique "from" value count is the sum of how many times that from occurs in that topic. A document may have many topics. What is the best way to do this in mongo?
[00:31:37] <JokesOnYou77> joannac, Yes, I'm trying to figure out the best way to do that with the framework and I'm having trouble using a nested $group, and I'm not even sure that's how I should be doing it
[00:41:23] <JokesOnYou77> srry, there's a typo in the post, should reat "$from" instead of entities, but you get the idea i think
[00:59:29] <JokesOnYou77> joannac, That's what I'm trying to figure out. I need to get a count of EACH unique from in that topic
[05:14:27] <Streemo> is there a simple way to convert an array to a cursor so i can query on the JSON in it?
[05:15:18] <Streemo> for exmple i want to use the power behind mongodb onmy arrya on the fly instead of intitalizing a new collection
[05:25:00] <Streemo> i think creating a collectio nfor the sake of rujnning a mongo query is not good, might as well just do the math explicitly on the array
[11:17:58] <jdo_dk> How to insert a record from pymongo like the mongo shell: created: new Date() ? I have tried: "created":datetime.datetime.now(). But there is 2 hours different on the inserted time.
[15:31:32] <GothAlice> salty-horse: It can. Indexes being considered for use are weighed against how effective they would be at reducing the work later in processing—highly selective indexes are preferred. Some times an index won't be used when one might think it should because of this.
[15:32:16] <GothAlice> salty-horse: I.e. you want indexes that are highly selective: if 90% of your data has the same value for an indexed value, that index isn't very useful, and may only be used in scenarios where you're hitting that remaining 10%.
[15:32:36] <salty-horse> I'll be more specific then. (I want to hint() something, but I'm worried if it's the right approach)
[15:32:54] <GothAlice> salty-horse: When in doubt, measure. (Esp. as any form of optimization is by definition premature without measurement.)
[15:33:44] <GothAlice> Hinting can be extremely useful, as MongoDB only allocates so much time to trying to figure out the best approach. Hinting lets it skip that heuristic process.
[15:33:55] <deathanchor> of course explain() is only a snapshot in time, you should check your logs for slow queries over time to try to optimize the slow things
[15:34:23] <salty-horse> 1) I have a sharded collection with a shard index {a: "hashed"} and an index on {a:1, b:1, c:1}. this is the type of query I'm interested in: find({a: 1, b: {$in: [1, 2]}}).sort({c: 1}).limit(100)
[15:34:53] <salty-horse> 2) According to SERVER-12783, a sensible merge sort should happen for each of the '1' and '2' sorted results
[15:35:19] <salty-horse> 3) If I create this dummy collection, shard it, and explain() an empty find, it chooses the a, b, c index.
[15:35:39] <salty-horse> 4) If I try it on a collection I filled with some data, it chooses the { a: hashed} index
[15:37:48] <salty-horse> I don't know what the database actually does if I hint it. explain() seems to scan more documents than it needs to, given the limit I set.
[15:38:24] <salty-horse> GothAlice: what kind of measuring methods are you suggesting? timing it, or something more involved?
[15:38:42] <GothAlice> salty-horse: If you're ordering (not using natural sort), then it needs to process the whole candidate set as part of that sorting operation, prior to limiting the count. (The sort itself is smart enough to only hold on to as many items as the limit specifies, though.)
[15:40:45] <GothAlice> No, $natural is natural sort. If you're sorting on something in the index, direction can matter.
[15:41:07] <GothAlice> (Sorting on {foo: -1} might not be able to make use of a {foo: 1, bar: 1} index.)
[15:45:02] <salty-horse> GothAlice: oh, I'm planning not to access the actual data :) (my actual index has the fetched field as an extra last value, and I'm upgrading to 3.0 so the shards would only access the index. I hope)
[15:49:30] <salty-horse> I'm suspicious something isn't working optimally. if I'm searching for b:{$in[1]} with limit(20) and the "proper" index, mongodb says it scanned 20 records. If I'm searching instead for b:{$in:[1,2]} (and hinting the proper index) mongodb suddenly scans 45 records. No matter the data, it would only need to scan at most 20 + 20 = 40, since I set a limit.
[15:52:49] <cheeser> you *have* to scan all the results to apply the query filters. then limit() just constrains the number you get back to your application.
[15:58:03] <pamp> windows users, how to add permissions to key_file? I cant start mongod, error open file, permission denied
[16:03:45] <salty-horse> cheeser: If I understand correctly, according to this bug, queries can be limit-aware - https://jira.mongodb.org/browse/SERVER-12783
[16:12:24] <GothAlice> salty-horse: That is correct; the sorting system is smart enough now to only track N items, where N is the limit you have provided.
[16:14:06] <salty-horse> GothAlice: what does it have to evaluate, for example? all of my fields are in the index, in the sort order I request.
[16:19:26] <GothAlice> salty-horse: In that scenario it'd walk the btree and just gather the number of elements you want.
[16:21:11] <salty-horse> GothAlice: exactly, but "nscanned" is too high :) - is it safe to ignore this anomaly?
[16:35:09] <salty-horse> GothAlice -- thanks a lot for trying to help :D - http://pastebin.com/WPcjNLG0 -- two things I don't understand: Why the second query (the one with ['123', '456']) has two similar "clause" sections, and why one of them scans 45 documents, when it should at most be 40, given the limit
[16:37:25] <matthavard> Let's say I have a Mommy collection where a Mommy can have zero or many Children from the Children collection. To represent this in mongo I might have a child-ids array field in a Mommy document that is just an array of ObjectIds pointing to various Children records. But is there a way that I could tell monog that these object ids refer to documents in the Children collection so I can operate on entries in the Mommy's child
[16:37:26] <matthavard> as if they were the full Children document for each and not just the ObjectId. So I could do `db.Mommy.findOne({_id:ObjectId(...)}).children[0].name`, to get the name of the first child of the Mommy document, instead of doing `var child_id = db.Mommy.findOne({_id:ObjectId(...)}).children[0]; db.Children.findOne({_id:ObjectId(child_id)}).name`
[16:38:41] <GothAlice> salty-horse: My initial impression would be that $in requires the planner to divide its plans based on the number of elements you are matching against. (Combinatorial problem.)
[16:39:07] <GothAlice> The second likely comes down to how the btrees in MongoDB work. They're buckets, each holding references to more than one actual record.
[16:39:38] <salty-horse> GothAlice: so it scans as much as there is in the leaf, but not too much more?
[16:40:13] <salty-horse> GothAlice: ok, I'll take the plunge and load some more data. (I guess I could test it with dummy data first)
[16:40:32] <GothAlice> Try it out with different limits, i.e. +/-1 1, and see if the nscanned changes. ;)
[16:40:49] <salty-horse> GothAlice: oh, thanks for the offer, but don't bother. I'll do more tests and if things don't make sense, I'll bug the channel again :)
[16:44:01] <GothAlice> matthavard: A child list is one approach, but it's tuned for very specific use cases. And one of those use cases is actually better served by other approaches. (Preserving the order of the children.) A list of child IDs is actually not very useful as you can't efficiently get the child records back in that explicit order.
[16:45:00] <GothAlice> https://gist.github.com/amcgregor/901c6d5031ed4727dd2f#file-taxonomy-py-L23-L35 illustrates the fields needed for several other approaches.
[16:45:20] <GothAlice> (And the code needed to manage an arbitrarily deep nesting; it's more than you seem to need, though.)
[16:48:48] <matthavard> I guess I don't really understand what that code is doing. I really just want to know how to store a reference to one document (Children) in a field in another document (Mommy) so that you can do `db.Mommy.findOne().child.name` instead of storing ObjectIds and looking up the document in the referred to collection (Children) by its ObjectId
[16:53:48] <GothAlice> It's basically creating a sub-document like {id: ObjectId(…), name: "…", acl: […]} for each "parents" list entry, and the same without "acl" for the direct parent reference. Saves needing to perform additional queries to load "related" data.
[16:54:21] <GothAlice> But, what you need to have the most use out of your heirarchy (preserved order, efficient querying) is a parent reference (ObjectId), and a "sort order" integer.
[16:54:51] <GothAlice> http://docs.mongodb.org/manual/tutorial/model-tree-structures-with-parent-references/
[16:55:14] <GothAlice> My own code does the parent reference, array of ancestors, and materialized paths. (All of them. Each optimizes for a different querying approach.)
[17:17:12] <Gevox> Hello i need to acheive something almost similar to this http://stackoverflow.com/questions/13732735/query-in-a-mongodb-map-reduce-function
[17:17:54] <Gevox> But in my case i have a key "title" stored into an event document. I need to split each word stored in that title field and count how many words each one repeated
[17:19:04] <Gevox> Its 1st time ever i hear about this mapreduce, so i'm bit confused. How do i get a reference to the "title" field which is stored inside the "Event" document stored in the "Events" collection and tell the map function to split each word in each title of these and return the key value to reduce function?
[17:20:36] <GothAlice> Gevox: When I was first learning map/reduce good (simple) examples were hard to find. Have mine: https://gist.github.com/amcgregor/1623352
[17:22:12] <GothAlice> In map/reduce, you write JavaScript functions. The map function gets run once for each document from the initial query, referenced as "this". The map function emits values which are then fed into one or more runs of the "reduce" function. The output of the reduce function should match the input to the reduce function—something that tangled me up for a while—as it may be run over its own output to further reduce.
[17:23:16] <Gevox> Ok I quite understood this part by now. Get ready for some dumb questions and bear me. That's how i get to understand things
[17:24:08] <Gevox> i want to tell this map to look at the "title" of each "Event" document. How can i explain this ?
[17:24:47] <GothAlice> Gevox: Instead of str.split(), you need to reference the field: this.title.split()
[17:25:03] <GothAlice> ("this" being the "current document being processed from the query" within the map functionl.)
[17:25:27] <GothAlice> Then you'd emit(word), not emit('word') (which would emit only one word: "word". ;)
[17:25:34] <Gevox> i mean in regular i use the .get method to get things from a certain document ( i use java )
[17:25:48] <GothAlice> JavaScript is a fully dynamic language; no need for setters/getters in most situations.
[17:27:14] <Gevox> and then i make sum of these "1's" of each word to get how many times each word mentioned
[17:27:49] <GothAlice> From my example: https://gist.github.com/amcgregor/1623352 you can see that this is effectively what I'm doing, just grouped by month.
[17:28:20] <GothAlice> (line 2 gets the creation time field from the record, then I emit a set of month counts for the given year for that record. (There'll be a 1 in one of the months.)
[17:28:36] <GothAlice> Then in the reduce function, I combine the values for the given year together.
[17:29:23] <GothAlice> You generally want to start counting from zero. So I create a record that is all zeros, then populate it from the passed-in values.
[17:29:56] <Gevox> What does the reduce function actually takes as arguments? The key is what and the value is what?
[17:31:01] <Gevox> so the exact thing i emit from map is sent as argument to reduce? in my case (word, 1) is what is taken for the reduce ?
[17:31:52] <louie_louiie> hey guys. sorry to interrupt- i got a quick question: what is the best way to ensure a search index on url's in mongodb?
[17:35:02] <Gevox> I think i need to use sort of HashMap (key is the world, value is the total sum) and return that from the reduce. Correct thinking?
[17:38:21] <GothAlice> Gevox: In the reduce function, the key is preserved. All you need to do is return the sum.
[17:38:53] <GothAlice> louie_louiie: http://docs.mongodb.org/manual/tutorial/create-an-index/ and http://docs.mongodb.org/manual/tutorial/create-indexes-to-support-queries/ should be useful resources for you.
[17:40:16] <sijis> is this connectionstring correct "mongodb://mdb-11,mdb-12,mdb-13/?replicaSet=rs0-seyren" based on my configs? http://paste.fedoraproject.org/218708/14308474/
[17:41:11] <GothAlice> sijis: I typically always include the port number explicitly against the hosts in my URIs when using a replica set. (Explicit is better than implicit.)
[17:41:43] <GothAlice> sijis: "mongodb://mdb-11:27017,mdb-12:27017,mdb-13:27017/?replicaSet=rs0-seyren" — otherwise, yes, looks fine, as long as the servers can DNS resolve "mdb-11" and friends. :)
[17:44:33] <GothAlice> Gevox: No points for style, but that should work. I haven't used map/reduce in a long time, so I'd test that and see if it requires objects to be returned. (I.e. return {sum: result}, not just return result.)
[17:48:31] <louie_louiie> there was this example of indexing urls as an BasicDBObject to remove duplicates
[17:49:08] <GothAlice> I don't even know what BasicDBObject is, so this sounds like over-engineering.
[17:49:49] <GothAlice> If you need to perform exact searches, a string is sufficient. If you need to perform prefix searches (i.e. https://example.com) strings can work for that, too, and still make use of an index. Note that "www" being there or not might not matter for many URLs, but will create edge cases for prefix searches.
[17:51:28] <GothAlice> louie_louiie: If you _really_ want to store a URL in the ultimate queryable way, store an embedded document like: {scheme: 'http', user: null, password: null, host: 'example.com', port: 80, path: '/', params: {}, query: {}, fragment: null}
[17:52:15] <GothAlice> From that you can rebuild the full URL, or query components, including query string argument names and values. (Need to use an array of embedded documents if you want params and query to benefit from indexes.)
[17:53:51] <GothAlice> louie_louiie: https://github.com/marrow/util/blob/develop/marrow/util/url.py#L188-L252 is my Python URL object. :)
[17:54:09] <GothAlice> sijis: If you want to nuke it and start over, you can shut down the node and delete the contents of your data directory, typically /var/lib/mongodb/*
[17:54:46] <Gevox> GothAlice: very bad exception happens, i tried both methods http://pastebin.com/eQfZE8JF
[17:55:09] <GothAlice> Gevox: Your loss of points to style is obscuring the issue. You forgot a semicolon.
[17:56:03] <GothAlice> Oh, not just a semicolon, also a closing parenthesis. Line 11 of your last paste.
[17:57:02] <GothAlice> You're passing a function to forEach, but you don't close the argument list for forEach (missing ")").
[18:00:31] <GothAlice> Gevox: Let me link you my list of Laws. #28 to #34 apply to code readability: https://gist.github.com/amcgregor/9f5c0e7b30b7fc042d81 ;) Step 1: take those JS functions out of those blasted strings and format it properly. ;)
[18:01:25] <kakashiAL> hey guys, why does mongoDB allow with save() to overwrite an document with the same _id?
[18:01:48] <GothAlice> Gevox: https://gist.github.com/amcgregor/b395d6136db7cfcae071 < line 6, not line 8 in the actual function is where the ); should have gone.
[18:04:31] <GothAlice> kakashiAL: MongoDB performs no security checks (other than read/write access to the db/collection) when performing any of the CRUD operations, excluding some optional fancy redaction stuff on reads.
[18:04:56] <GothAlice> kakashiAL: It's up to your application to handle your data's security, really.
[18:06:34] <GothAlice> Gevox: The point is to write the code as code first, with something akin to a proper style guide; then syntax errors and whatnot can be easily caught. You can even copy/paste these functions into the MongoDB shell to try them out first. *Then* you can wrap it in strings. Wrapped in strings to begin with is maddeningly difficult to read, and encourages one to not make use of newlines.
[18:07:08] <kakashiAL> GothAlice: so it takes my object, look if there is the same ID, and overwrite it with no mercy?
[18:08:32] <GothAlice> kakashiAL: Yup. Your application told MongoDB to save the record, so that's what it does.
[18:08:45] <GothAlice> If you do not like that behaviour, avoid .save(). (I avoid it like the plague.)
[18:11:00] <sijis> GothAlice: thank you. that worked. i think the app i'm playing with doens't support it or something
[18:11:06] <kakashiAL> GothAlice: I dont know how the mongoose update function works, that only updates one value, but I guess it uses save() somehow, since mongoDB doesnt care from the beginning
[18:14:13] <Gevox> GothAlice: it runs, but it does not seem to split each word in the title. It takes the entire title . Look the output http://pastebin.com/mCzbGgAZ
[18:15:40] <GothAlice> Gevox: https://gist.github.com/amcgregor/b395d6136db7cfcae071 and http://showterm.io/8b877b27715956e03581a (my testing it out)
[18:16:47] <GothAlice> You forgot to loop over the individual words, and were instead emitting all of the words that were split as a single key.
[18:24:15] <Gevox> GothAlice: it works now! Well, appearently i got punch of DBObjects as return, if i want to show only words which got total of more than 10 for example. I shall make another regular java method that iterates over these DbObjects and prints out which has a value of greater than 10?
[18:25:01] <GothAlice> Gevox: That would work. (You can see from the end of my second link the structure being returned by the mapReduce call.)
[18:25:02] <Gevox> in paractice the reduce function already returns the value as "total" : 2.0". So i will parse this DBObject.get("value") to get only the number out of the string, and print it out based on my specified condition.
[18:26:07] <GothAlice> The _id of a record in the "results" array will be the key you emit in the map, and the "value" will be the value you emit and subsequently return from reduce.
[18:26:56] <GothAlice> In my example I emit and return basic integers. Thus result.results[0].value == 1 in my example. How you get that in Java… others may be better suited to answer.
[18:31:04] <Gevox> One last thing, what to run mapreduce on Events that happened between 5 and 6 PM for example? I store the time in a key "hour" as part of the Event document. Shall i make a function that gets all of the documents which has a "hour" of 6:00 PM for example and store it in a new temp collection, then run my mapreduce function on that collection?
[18:31:31] <hfp_work> Hi all, when you do db.collection in NodeJS, is collection a method of MongoClient?
[18:32:04] <GothAlice> Gevox: http://docs.mongodb.org/manual/reference/command/mapReduce/#calculate-order-and-total-quantity-with-average-quantity-per-item < the "query" option you pass in the options to the mapReduce call. These examples (down a bit from the anchor I linked) demonstrate this.
[18:33:24] <GothAlice> hfp_work: Ref: https://mongodb.github.io/node-mongodb-native/api-generated/collection.html
[18:34:59] <hfp_work> GothAlice: So to test inserting and removing documents with sinon in nodejs, I should `var Collection = require('mongodb').Collection; sinon.stub(Collection, 'insert');`?
[18:35:43] <GothAlice> "db" (a Database object) is a factory for them, as MongoClient is a factory for Database objects.
[18:36:29] <hfp_work> Right, but how do I get a Database object for a test? I don't want to test against an actual MongoDB, I just want to make sure my function inserts or deletes depending on what it is given
[18:37:18] <hfp_work> This is the function I want to test: https://gist.github.com/anonymous/db196cf7e1144e2a27f0. I want to make sure it inserts and deletes when it should. But I dont want to ahve to run a mongodb to run my test suite.
[18:37:47] <GothAlice> hfp_work: The typical approach is to not wrestle the driver, as monkeypatching would, nor to replace the API as mocking would, but to simply test the database state after execution of your function.
[18:38:32] <GothAlice> I.e. clear the collection, insert a value, run your function that inserts or updates, check if it added (count() > 1) or modified (old value is now new, expected value).
[18:39:27] <hfp_work> StephenLynx: It's not about that, it's because our circleci test suite doesnt have a mongo setup and I dont want to go and change all that for one test out of 1200+
[18:40:24] <GothAlice> Gevox: It never hurts to help. When I was learning map/reduce the biggest thing I learned was to avoid it wherever possible. ;)
[18:42:24] <GothAlice> hfp_work: Are you able to explicitly query the DB in your test, to see the result of any operations made in the test block?
[18:43:36] <GothAlice> … so you can run code that performs operations on the DB. But can't run code that performs operations on the DB. I'm really not grasping something, here. ;)
[18:43:36] <deathanchor> yeah map/reduce, good for never using or running on secondaries for one-offs.
[18:44:00] <GothAlice> deathanchor: Sometimes it's the only thing that'll work, but I always make those one-offs to correct the "issue". ;)
[18:44:21] <GothAlice> hfp_work: The former being the code you're testing, the latter begin the code testing your code.
[18:45:24] <sijis> are these equivalent? mongodb://mdb-11:27017,mdb-12:27017,mdb-13:27017/?replicaSet=seyren OR mongodb://mdb-11:27017,mdb-12:27017,mdb-13:27017/seyren?replicaSet=seyren
[18:45:40] <hfp_work> GothAlice: I managed to get away with stubbing all the calls to a MongoDB so I monkeypatched whatever uses mongo so it returns what I would expect from a successful mongo operation.
[18:45:51] <GothAlice> hfp_work: I'm apparently too used to testing that lets you do things like this: https://github.com/marrow/task/blob/develop/test/test_queue.py#L23-L25 and https://github.com/marrow/task/blob/develop/test/test_queue.py#L86-L99
[18:47:13] <hfp_work> GothAlice: So far, it took me 3 times longer to write the tests than the actual code. I hate it.
[18:47:16] <GothAlice> This was also discussed yesterday: http://irclogger.com/.mongodb/2015-05-04#1430765075
[18:48:11] <GothAlice> My argument from yesterday stands: you're not testing against Mongo at that point, you're testing against your mock.
[18:49:39] <GothAlice> (With a passing test meaning nothing about the ability for that code to actually run in a production environment.) This has bit me in the past, notably on the map/reduce call spec, which changed.
[18:50:36] <hfp_work> GothAlice: Thanks for the insight, I'll bang my head on the wall a little more, see if any better solution than mocking comes up
[18:51:30] <GothAlice> The chat history from yesterday should include a link to my "start a cluster on a single node" shell script, which I use heavily for making reproducible tests of sharding behaviour, not to mention a clean way to start and stop a mongod for testing purposes, separate from one running on the normal port, too.
[18:51:31] <StephenLynx> <hfp_work> GothAlice: So far, it took me 3 times longer to write the tests than the actual code. I hate it.
[18:57:52] <GothAlice> hfp_work: Testing should be fun, it should be enjoyable, and it should be easy. There's a trend in my own packages to have more tests than there are executable statements of code: https://travis-ci.org/marrow/schema/jobs/41019976#L317-L319 (578 statements, 629 tests)
[19:13:03] <deathanchor> StephenLynx: guess you don't believe in coding the tests before coding the functions?
[19:13:54] <StephenLynx> I just rather make a clear documentation that any issue is easily spotted than writing more code.
[19:14:20] <StephenLynx> well, if I don't write tests, I guess I wouldn't write them before the actual code because I am not writing them at all.
[19:14:50] <StephenLynx> I don't say they are bad or pointless. I just have experimented with both and stuck to the later.
[19:17:15] <StephenLynx> I rather spend my time manually testing than writing code that will test and have then to test it.
[19:20:40] <GothAlice> StephenLynx: Actually, I do sometimes write tests that test tests. I also write tests that test everything down to the informational logging messages.
[19:21:40] <GothAlice> One of the reasons why marrow.schema has so many tests is that many of them are simply testing input data against an expected output; bam, feed it a list of input/output pairs for good, bad, and error conditions, and you instantly have hundreds of tests… having only actually written three.
[19:23:13] <GothAlice> https://github.com/marrow/schema/blob/develop/marrow/schema/testing.py#L11-L36 < is the factory, provided by the library being tested for all users of that library. https://github.com/marrow/schema/blob/develop/test/validate/test_geo.py#L19-L32 is an example of using it.
[19:23:34] <GothAlice> Face it: if I'm too lazy to write that style of resulting test, I have no business being a library author. ;)
[19:24:48] <GothAlice> https://github.com/marrow/schema/blob/develop/test/validate/test_compound.py#L87-L99 < it also lets you factor out things common to multiple tests more easily
[19:24:52] <hfp_work> GothAlice: Haha I know I keep reading Mocha's headline that testing should be "simple an fun". I wish I was better at testing so it actually becomes easier and funnier because right now it's a drag. And I don't want to not write tests.
[19:28:32] <StephenLynx> anything that you wouldn't voluntarily do is not worth doing. a counter part of alice's automation moto.
[19:29:10] <GothAlice> StephenLynx: Garbage collection; just because you wouldn't drive a garbage truck doesn't make the task any less worthwhile. -_-;
[19:30:00] <GothAlice> fxmulder_: Did you have network connectivity issues long enough in duration to burn through the oplog duration?
[19:31:50] <GothAlice> Hmm. That particular socket error indicates that the connection disappeared unexpectedly, either due to intermittent connectivity or the remote side closing the connection without sending a FIN, with MongoDB figuring that out only after it tried sending data there.
[19:33:52] <GothAlice> MongoDB does try to be quick about those things, and usually Does the Right Thing™. It'll "start over" if it was unable to complete the first sync, or if the node is unable to catch up with the primary before exhausting the oplog.
[19:34:24] <fxmulder_> I have everything shut off so nothing is reading or writing to mongodb, the only thing going on is this sync
[19:34:35] <GothAlice> In the event where you have enough data that takes longer to transfer the initial sync than your oplog size, you have two choices: increase the oplog size, or pre-populate the secondary and pray *that* is fast enough.
[19:35:29] <fxmulder_> its not uncommon for us to have to rebuild one of these machines, I'd like to be able to get this syncing process working without having to rsync
[19:36:25] <GothAlice> Then increasing the socket timeouts and increasing the oplog size will likely be the solution for you.
[19:37:21] <GothAlice> deathanchor: That's weird. You're saying making the timeouts more strict alleviated the problem of them being hit?
[19:37:43] <deathanchor> my tcp_keepalive_intvl was 75, and tcp_keepalive_time was 7200, I set them both to 5 and goodbye socket timeouts
[19:40:56] <fxmulder_> [rsSync] clone mail.message_data 329061375 is just indicating the insert process
[19:42:15] <fxmulder_> but if something happened and it didn't notice for 7200 seconds maybe decreasing that time helps
[19:44:43] <deathanchor> GothAlice: fxmulder_: here is where I found the suggestion to use 5: https://jira.mongodb.org/browse/SERVER-9199
[19:46:54] <GothAlice> deathanchor: Indeed, that's a diagnostic strategy, not a production one. Lets just hope you aren't billed for server-to-server traffic—a value of five means send a packet every five seconds, and that can add up over time. ;)
[19:53:19] <fxmulder_> so lets say I have 1 primary replica member and I'm cloning to a new secondary, I restart that primary member, does that restart the cloning process on the new secondary?
[19:53:48] <cheeser> the secondary reads from the oplog from a particular time and will simply pick up where it left off.
[19:53:59] <Gevox> GothAlice: I would like to configure my db to allow shards and replica sets. Is there any good reference you would recommend for me to help get the db configured?
[19:54:58] <Gevox> currently i got this one http://www.severalnines.com/blog/turning-mongodb-replica-set-sharded-cluster
[19:55:43] <fxmulder_> cheeser: I just gave it a shot, it looks like the secondary did perform a dropAllDatabasesExceptLocal
[19:56:12] <GothAlice> Gevox: The construction of a sharded cluster is simply the application of several replica sets with a few management daemons (configuration services, routers) to glue them together.
[19:56:58] <GothAlice> Gevox: In my "local testing" script, you can see the process in these lines: https://gist.github.com/amcgregor/c33da0d76350f7018875#file-cluster-sh-L130-L158
[19:57:04] <drags> I'm putting together a system wide /etc/mongorc.js file to update my prompts but having some trouble. If I put the file at /etc/mongorc.js it is seemingly ignored. If I copy that file to ~/.mongorc.js then it is used
[19:57:27] <drags> interestingly: if I delete the ~/.mongorc.js file it is recreated as an empty file at the start of each shell invocation
[19:57:55] <GothAlice> Gevox: The process is: construct two, three-node replica sets, spin up some config servers (that track metadata about what data is where), optionally handle authentication by adding the first user, then stat a router and configure it with two shards.
[19:57:59] <drags> from the docs it seems like the client should be merging the effects of the global and user level mongorc files, but I'm wondering if the 0 length file is winning
[19:59:05] <GothAlice> drags: http://docs.mongodb.org/manual/reference/program/mongo/ < makes no mention of a "system wide" rc file.
[19:59:09] <deathanchor> fxmulder_: yeah the initial sync is precarious on these older mongo versions
[20:00:09] <GothAlice> Gevox: That's a shell script, so those would be things you could type into an interactive terminal.
[20:00:47] <GothAlice> Gevox: I use functions in that script to isolate the common commands, so you may need to jump up a bit to reference those functions and see what they're doing. It's pretty straight-forward.
[20:00:49] <deathanchor> fxmulder_: I banged my head on that problem for a week trying to sync to another DC.
[20:01:09] <GothAlice> Gevox: BASH shell, not mongo shell. The first line of a script typically tells you what it runs under, if it starts with a "sh-bang" (#!).
[20:01:16] <deathanchor> fxmulder_: I think after the sync is done, you can change the timeouts back
[20:01:36] <drags> GothAlice: but switching the docs to be for my version (2.4) show that the global level one didn't exist :P
[20:01:48] <deathanchor> fxmulder_: would get about 90% done, then network drop on other DC would kill it off
[20:05:23] <fxmulder_> deathanchor: yeah, it would be longer but we just got the second shard set in place, probably looking at about 15TB of data to sync
[20:09:23] <deathanchor> fxmulder_: when it happened for me, the socket timeout occurred right after building some giant indexes on a large DB when it tried to grab the next thing to sync.
[21:41:19] <tubbo> is there any guarantee that two BSON::ObjectIds representing different documents are unique?
[21:41:42] <tubbo> say i have the SHA '5547bc60a7a206c7dd000580', would it be possible for another model in the mongodb database (not the collection) to have that ID?
[21:52:49] <Gevox> tubbo: depends on the hashing algorithm you are using, some hasing algorithms has a collision chance. You need to google bit about which hashing function you shall use to avoid collision
[21:53:22] <ToeSnacks> what do I need to put in the mongos /etc/mongodb.conf file if I only want to connect to a stand alone db? I have been using a sharded system so far and I specify configdb= for that. Is it the same or is there a different thing I need to define?
[23:37:03] <GothAlice> But, one more brave soul now knows how to set up 2x3 sharded replica sets on Windows, of all things. ^_^
[23:42:15] <GothAlice> tubbo: As a late reply (sorry; t'was distracted) all hashing algorithms offer the chance of collision (same hash, different data). This is because you're trying to create a compressed summary of larger data; depending on the algorithm you may only have 16 bytes of hash (a la MD5). Larger hashes reduce the statistical likelihood of collision at a rapidly growing cost in terms of space. (SHA512… wow.)
[23:42:54] <GothAlice> As an example that isn't quite a hashing algorithm (though it can be used that way): UUIDs have enough bits of data in them that you could generate one a second until the heat death of the universe and only have a 50% chance of collision.
[23:44:54] <GothAlice> ObjectIds store 96 bits of data. (12 bytes.) Any hashing algorithm you apply would need to algorithmically "expand" the data to at least fit the hash size. This should give you the distinct impression that hashing ObjectIds is… probably not a good solution for anything.