Log file Viewer
#mongodb logs for Thursday the 19th of February, 2015
[01:45:32] <bros> http://ideone.com/onOOKt I think my MySQL ways are hurting me. Am I doing this right?
[02:08:07] <GothAlice> Looks like a user only ever has one. Instead, you might consider embedding the value.
[02:13:29] <bros> 1 account, 1-5 users, 1-5 stores, 300-500 barcodes, 1k+ orders/scans/stats/batches
[02:14:08] <GothAlice> 1-5? Embedding may be worth it. There are some limitations when embedding that become important when you have two lists of sub-documents.
[02:15:51] <GothAlice> (Well, more than one.) Specifically, you can only $elemMatch on one at a time, it complicates aggregate queries (lots of unwinding and re-grouping), and there was something else I'm forgetting at the moment.
[02:16:27] <GothAlice> Stores and users may be embeddable. It simplifies some things, as often you're checking a login and also need the account if successful anyway, right? :)
[02:18:17] <GothAlice> On item_barcodes are you storing the item_id/last_update/quantity in addition to barcode_id as a caching mechanism?
[02:19:50] <bros> GothAlice, That'd be a parent-child relationship from what I can see. What could I do? Make barcodes have an array of item_ids?
[02:20:46] <GothAlice> Without reference to a particular store? (IDs should be unique, so there would be no need?)
[02:23:41] <GothAlice> You're overly nesting and overly complicating. I'm about to massacre your last paste. ;)
[02:24:47] <GothAlice> You're trying to put all eggs in one basket. It wasn't going to be query-able at all like that.
[02:28:23] <GothAlice> Quick q, bros: within your nested model there, what did orders->status and batches->status represent? Why order_number and not order_id? Were you expecting orders embedded documents to have IDs?
[02:30:39] <bros> order_number is a string. eBay has order numbers like '100-1000', as well as Amazon
[02:34:14] <bros> After good schema and proper indices, what else is there to a properly good Mongo setup?
[02:35:25] <bros> I think I might be in the clear here. I'm dealing with very few strings, and a lot of straight forward IDs.
[02:36:16] <GothAlice> It's still important to plan for those queries. How one models in MongoDB is heavily effected by those questions.
[02:38:46] <GothAlice> Does Mongoose not have a way of saying "this is a dynamic embedded document"?
[02:39:43] <bros> item_barcodes represents the barcodes for the item. quantity is how many times they are required
[02:42:09] <GothAlice> Right. So we're back to the questions thing. What do you need this data to answer?
[02:46:12] <GothAlice> Do you ever need the answer to: "What are all of the items for all of the stores for account X?"
[02:53:18] <GothAlice> Hmm. Barcodes shared between items? (For some reason I'm not seeing an "item"-specific schema here.)
[02:57:16] <bros> account_schema.index({ 'users.email': 1, type: -1 }) could you explain that that does?
[03:08:25] <bros> accounts have stores and batches. batches have orders. stores have barcodes, orders.
[03:16:53] <GothAlice> I use singular to describe the schema, plural to name the collection. (A "collection" has many things, a schema describes a single thing.)
[03:18:53] <GothAlice> When appending a new value to an embedded list like that, use mongoose.Types.ObjectId() to generate an ID for the "embedded record".
[03:18:53] <GothAlice> (Having IDs like this isn't required per se, but it can be invaluable if you need to update nested documents. I do this on my forums to embed replies to a thread within a thread, while still allowing liking/editing/removal.)
[03:19:55] <GothAlice> It should be kept small, it shouldn't fluctuate in size too much (because that requires moving data around), and you should almost always only embed documents that only make sense in the context of the parent. I.e. if you delete the parent, you want the nested values deleted too. When querying for a nested value, you always _also_ want the parent record. Etc.
[03:20:23] <GothAlice> See also: http://www.javaworld.com/article/2088406/enterprise-java/how-to-screw-up-your-mongodb-schema-design.html <- an excellent short article which covers the subject from the perspective of migrating from SQL.
[03:20:47] <GothAlice> bros: Because it will require extra queries client-side, and you can introduce race conditions. MongoDB doesn't have joins.
[03:21:39] <GothAlice> http://mongoosejs.com/docs/guide.html was the only bit of Mongoose documentation I have ever read in my life, BTW. XP
[03:24:22] <GothAlice> It wouldn't be query-able within MongoDB. This means you would pretty much need to load the entire record every time, to do anything. It would also require you to effectively re-save the entire record on every update. If it's possible to have two updates come in at the same time, in that scenario, good luck knowing how your data will end up.
[03:25:50] <GothAlice> I.e. you can't do a query inside one list of embedded documents and _another_ list at the same time.
[03:26:28] <bros> i don't really want to load all of the users for an account every time i need to load a user...
[03:29:40] <GothAlice> Include any other fields you want, including those you may need from the rest of the account information. Note, this $ operator is the thing you can only use once in a query, and why you can't just lump everything together.
[03:30:47] <GothAlice> The returned document from that would look like: {_id: ObjectId(…), users: [{email: "bobdole@whitehouse.gov", …}], …} — note, only one "user".
[03:34:18] <GothAlice> db.account.find({'user.email': /@gmail.com$/i}, {'stores.store_id': 1, 'stores.credentials': 1}) — give me every store ID and credential for each account with a user registered with a gmail e-mail address.
[03:37:14] <GothAlice> You pointed out one thing that you were doing already that is perfect for MongoDB.
[03:37:36] <GothAlice> In MongoDB, you don't need to do that. You just embed whatever structure you want, as needed.
[03:38:18] <GothAlice> You already had a type to differentiate, so you can know what fields to expect at any given time, if you want/need.
[04:13:39] <MacWinner> can you have a subdocument with an objectid? eg, I want to have a main level document like this domain = { domainname: 'example.com', websites: [{name: 'site1', url: 'site1.com'}, [{name: 'site2', url: 'site2.com'}] }
[04:14:05] <MacWinner> if I want to reference the 'websites' attribute in another subdocument, what would be the best way to do this?
[04:17:50] <Boomtime> MacWinner: do you mean you want to store an ObjectID as the value of some other field? if so, yes, this is quite common
[04:19:23] <MacWinner> Boomtime, cool.. just wanted to make sure I wasn't doing some weird design practices.. like i have ads and campaigns that are part of a domain.. i want the campaign to reference ads
[07:28:54] <lessthanzero> are there any known pitfalls when dealing with nested arrays in mapReduce (I'm having problems emitting deep into arrays)
[10:35:35] <gma> Are there any mongoid users here who know how to use `.and()` ? I'm struggling to make it produce the right query…
[10:42:16] <kali> andefined: if you can't restart mongod, you probaly can't see your database anywgat. you should get an error message in the log when you try to start it
[11:29:45] <ladessa-db> hi, pls...help me with this question https://stackoverflow.com/questions/28594558/mongodb-find-where-count-5/28595270?noredirect=1#comment45498328_28595270
[11:39:26] <StephenLynx> hey, is there a download available for documentation? I got a pdf with a manual, but I'd like something for technical reference.
[14:46:31] <evangeline_> I would like to get all _id unique IDs from the collection from pymongo; any ideas on how to do that?
[14:49:13] <evangeline_> jaitaiwan, that works in the mongo shell; I guess I should ask how to do it with pymongo in #python
[14:50:44] <jaitaiwan> Depending on your implementation you can either iterate or theres probably a helper function to pull out all the results at once, although that could lead to some fun memory issuesxs.
[14:54:13] <evangeline_> jaitaiwan, thank you; one more question; how could I easily add new attributes to objects - where the old values should be overwritten if already exist?
[14:59:07] <jaitaiwan> bin: alternative depending on your data structure is to use a full text index and search $text: '@gmail.com'
[15:00:40] <jaitaiwan> What you had was fine I think, just replace in between the quotations with /@gmail.com$/i
[15:07:24] <GothAlice> Using full-text indexing that way is a bit abusive of the system. FTI does a lot more than one needs to answer the proposed question.
[15:08:29] <jaitaiwan> GothAlice: definitely. In my benchmarks for production, we foudn that text was many times faster than regex. So that's what we resorted to.
[15:08:57] <GothAlice> jaitaiwan: Indeed, there is that. Excluding regexen in the form /^…/ – prefix searches should still be relatively speedy.
[15:12:04] <GothAlice> Of course, if that type of querying is required it might even be worthwhile to adapt the data. Split e-mail addresses into two fields, recipient and server. Index server and your queries become fast indexed hash comparisons.
[15:13:27] <GothAlice> With an appropriate ORM/ODM/DAL it would be seamless for the rest of the application.
[15:13:49] <GothAlice> (In Python I'd have an "email" @property which re-combines the two dependant fields, the app would just use that where needed.)
[15:17:39] <jaitaiwan> I've only recently been getting back into python after a 5 year sabatical. Hardest thing is transitioning from PHP cause I know it so well. But that's probably #offtopic for this room.
[15:23:44] <GothAlice> It's a reasonable template language. It's not a real general purpose language, however, and its fundamental design encourages anti-patterns. Also the fractal.
[15:24:08] <GothAlice> parse_str = register globals that never went away, as an example of my favourite function in PHP ever.
[15:24:50] <GothAlice> cheeser: Have you seen the "do not use" MySQL post, similar to the PHP fractal?
[15:26:48] <jaitaiwan> Sounds cool. I think PHP is making improvements, all thanks to the guys at facebook with their hack lang and HHVM. In some benchmarks PhalconPHP for instance beats a lot of python mini web framworks
[15:27:20] <GothAlice> Python (specifically the RPython runtime written by the Pypy folks) runs PHP faster than HHVM.
[15:28:55] <GothAlice> https://github.com/marrow/tags/blob/develop/examples/widgets.py < indentation in this file is effectively 100% arbitrary.
[15:30:32] <GothAlice> There's also a better-than-Ruby Ruby interpreter called Topaz: http://www.topazruby.com
[15:31:22] <GothAlice> StephenLynx: Then you'll hate my own language, Clueless. Clueless only recognizes tabs for scope nesting.
[15:33:01] <GothAlice> Heh, cult. Casual dismissal of others opinions. Not the basis of a logical argument or constructive discussion, sadly. :(
[15:33:44] <GothAlice> jaitaiwan: Clueless (https://gist.github.com/amcgregor/016098f96a687a6738a8 and https://gist.github.com/amcgregor/a816599dc9df860f75bd) may be amusing to you. :)
[15:34:04] <StephenLynx> but check this out about python: "Van Rossum is Python's principal author, and his continuing central role in deciding the direction of Python is reflected in the title given to him by the Python community, benevolent dictator for life"
[15:34:47] <StephenLynx> all of them have bad syntax, are slow as hell and really have no purpose to be used when there are better tools.
[15:35:00] <GothAlice> StephenLynx: BDFL is a typical title in long-running open source projects. Note the Python Enhancement Proposal process for a structured method to make changes to the language. Vs, I don't know, throwing things at the wall to see what sticks a la PHP and JS. (The full-page numbered list for evaluating automatic typecasting in JS is kinda nuts.)
[15:35:52] <GothAlice> StephenLynx: My Python code is faster than equivalent C code. My HTTP/1.1 server supports 10,000 concurrent requests per second and compiles to 171 Python opcodes. Attempting to use a C extension for header parsing actually slowed it down…
[15:36:14] <GothAlice> StephenLynx: So it seems your preconceptions about the language do not match reality.
[15:36:54] <GothAlice> I'm guessing you haven't really investigated the Pypy (Python-on-Python) JIT compiler.
[15:37:17] <StephenLynx> I got my benchmarks from here http://benchmarksgame.alioth.debian.org/u64/compare.php?lang=gcc&lang2=v8
[15:38:05] <StephenLynx> I assume the cause of some python code being faster than some C code is very badly written C code rather than some superb python VM or JIT compiler.
[15:38:53] <StephenLynx> and speaking about web-servers, python does not support non-blocking IO, if I'm not mistaken.
[15:38:57] <GothAlice> http://morepypy.blogspot.ca/2011/02/pypy-faster-than-c-on-carefully-crafted.html and http://morepypy.blogspot.ca/2011/08/pypy-is-faster-than-c-again-string.html pardon the potentially borked images on those pages.
[15:40:12] <jaitaiwan> One of the key things to these sorts of arguments is that we developers are passionately romantic about our languages in some ways. Very hard to be objective, especially when benchmarks vary so widely haha.
[15:41:10] <StephenLynx> "Hence, PyPy 50% faster than C on this carefully crafted example. The reason is obvious - static compiler can't inline across file boundaries." Interpretation strength vs compilation weakness, nothing exclusive to python on the first example
[15:41:27] <StephenLynx> " This is clearly win for dynamic compilation over static - the sprintf function lives in libc and so cannot be specializing over the constant string, which has to be parsed every time it's executed." same case.
[15:41:52] <StephenLynx> so these two links mean nothing in favor of python, just in favor of interpreted tools.
[15:42:33] <GothAlice> Specifically, the Python implementation written in Python and its ability to deeply understand and optimize your code, including JIT compilation to machine code.
[15:44:07] <StephenLynx> http://benchmarksgame.alioth.debian.org/u64/compare.php?lang=python3&lang2=v8 these benchmarks show the distance between V8 and python is greater than the distance between C and V8.
[15:44:28] <jaitaiwan> GothAlice: got through some of your links. You have WAAAY too much time on your hands
[15:46:13] <GothAlice> https://gist.github.com/amcgregor/405354 compares operations between Python 2.7 and 3.2 (in most cases, a version two major versions behind).
[15:46:37] <GothAlice> There are two separate things at play here: the Python syntax, and implementation details for a particular runtime.
[15:46:58] <GothAlice> Python 2 is also supported. Both are offered to let you pick between advanced features or greater raw performance.
[15:47:40] <jaitaiwan> I have to say StephenLynx. I do like node but gee the callback hell gets me. Promises are good but their a bit of a mind-screw sometimes
[15:49:50] <jaitaiwan> Mmmk... If I have to open 3 files, to do one operation, I'm seeing callback hell as gonna happen (Without promises obviously).
[15:50:09] <GothAlice> Oh, Pypy also recently added automatic STM (software transactional memory) features. The biggest unique feature of Pypy is that the compiler acts as a filtering pipeline. Enabling STM simply adds a filter layer to the compiler. You have pluggable garbage collection schemes, also a filter, and the intermediate representation is a flow graph of your entire application with multiple different back-end compilers. I.e. compile to C, or
[15:50:58] <StephenLynx> building to other languages only adds to the argument the language itself is useless.
[15:52:51] <GothAlice> http://www.rfk.id.au/blog/entry/pypy-js-first-steps/ — see also: http://www.pypyjs.org/demo/ (have some Python in your web browser)
[15:53:23] <StephenLynx> jaitaiwan this is my biggest project so far https://gitlab.com/mrseth/bck_lynxhub
[15:53:25] <GothAlice> The importance of this work isn't that hey, you can now run Python in your browser, but rather the improvements this project brought to asmjs and Emscripten.
[15:55:19] <GothAlice> (https://www.rfk.id.au/blog/entry/pypy-js-faster-than-cpython/ is an amusing instance of a benchmark running faster under Pypy.js than CPython. Also links to some of those improvements to other packages I mentioned.)
[15:56:46] <jaitaiwan> Its natural progression of the language which has required an elegant solution
[15:57:41] <StephenLynx> not that I defend javascript as a good language. it is hilariously inconsistent and you need to use strict mode and a lint to keep on track.
[15:58:08] <StephenLynx> but nothing you find in it is bad than the abomination of using whitespaces as part of syntax.
[15:58:25] <GothAlice> typeof '' === 'string' — typeof new String() === 'object' — also the existence of ===
[16:02:09] <jaitaiwan> The prototype object system takes a fair bit to get used to when you come from a c background. Not that its technically a con for the language itself.
[16:02:27] <GothAlice> The "what not to do in JavaScript" conversation, however, never really ends. It's a minefield that requires a rather substantial head space to navigate. Core language features are inherently broken, requiring non-obvious workarounds and heavily defensive code. Or, even better, automated conversion tools (CoffeeScript) to gloss over the multitude of issues for you. (Which is not really any better than going from Python->JS.)
[16:03:43] <StephenLynx> specially when you get back the best performance with interpreted languages.
[16:05:31] <GothAlice> I run many servers with it. I linked to my HTTP/1.1 server already. I also run XMPP, a secure pub/sub proxy through to MongoDB's capped collections, and MUSHes (telnet-things), amongst others.
[16:06:06] <GothAlice> Python (Pypy runtime again) is used for all of the management scripts, and even the entire package management system of the distro I use, Gentoo. This automation (also using MongoDB) runs a nearly 2000-node cluster.
[16:07:17] <StephenLynx> the effort put into adding bloat instead of learning how to code is baffling.
[16:08:52] <GothAlice> StephenLynx: http://blog.kgriffs.com/2012/11/13/python-vs-node-vs-pypy-benchmarks.html compares them, sorta. It's benchmarking gibberish in most of the graphs. (wsgiref = unoptimized HTTP server for debugging) The Gevent vs. Node.js req/sec tests indicate that for 64 KiB responses Njode.js and Gevent are at par, with Node.js having higher standard deviation.
[16:08:57] <StephenLynx> yeah, most language communities are eco chambers because most programmers don't step outside their zone of confort.
[16:09:16] <GothAlice> (It's also a really old article, not taking into account current Pypy optimizations.)
[16:09:23] <StephenLynx> I could easily use java for fucking everything. instead I went out of my way to learn from javascript to C++
[16:12:14] <GothAlice> Ignoring the performance bits on http://www.cdotson.com/2014/08/nodejs-vs-python-vs-pypy-a-simple-performance-comparison/ (since his method was flawed) the memory comparison remains valid, however. Node uses multiples of the amount of RAM that Python does on the same problem, until the problem scope grows to the maximum.
[16:12:53] <GothAlice> Pypy (due to needing to graph the whole problem at the start) starts off with ludicrous memory usage, but it flips at the mid-point of problem difficulty to being less memory-hungry than the others.
[16:14:21] <StephenLynx> but, it is the most we have. you could come up with your own benchmark to try and show python is not completely obsolete and redundant as technoloty when it comes down to web.
[16:16:29] <GothAlice> jaitaiwan: Of course. Hopefully each also factors in finding the right solution for a problem, too. (I've written stream filters in Brainfuck before… right tool for the right job. ;)
[16:17:13] <jaitaiwan> Yeh exactly. Anyway great discussion StephenLynx and GothAlice. Its about 2am in my timezone so its time for some much needed rest haha
[17:28:32] <wayne> each player has an array of scores. i want all scores over 100 to be reset to 0. what's an update query that i can use to do this?
[17:33:00] <StephenLynx> the problem is, I don't know if you will be able to update several scores at once.
[17:35:05] <wayne> StephenLynx: yeah that's what i have now, but i'm having huge performance issues due to application-level joins
[17:37:50] <wayne> well, i'm just sticking the many from a many-to-one relationship as an array in the one
[17:43:10] <Penagwin> Hey guys, I need to search through objects and find those that don't have a specific string in a field
[17:47:23] <Penagwin> Testing it on mongolab atm, just get an "unexpected error" with most of my tries.
[17:49:30] <giowong> currently its returning as a array, and i was pulling data directly from the json file itself
[18:28:59] <MacWinner> if I have 3 files with different filenames, but the same md5 hash of the data (ie the same data), is there some best practice on storing them without duplication?
[18:29:33] <MacWinner> i feel like the chunks reference back to the files.. rather than files referencing the chunks
[18:30:06] <GothAlice> MacWinner: That's due to the way they're queried. More often you're looking for all of the chunks for a file and you already know the file ID.
[18:31:06] <GothAlice> MacWinner: I abstract GridFS with a metadata collection storing the actual "file metadata" and GridFS storing BLOB data without metadata (i.e. no file names). The metadata collection then references the ID of the GridFS BLOB to use, which means I can de-duplicate without issue.
[18:31:40] <MacWinner> GothAlice, cool.. sounds good. I was planning on something like that. just wanted to make sure I wasn't missing some built-in feature
[18:39:27] <MacWinner> GothAlice, do you typicall split out your gridfs database from the rest of your app data onto a different mongo cluster?
[18:40:27] <StephenLynx> I would save the file regularly then have a document to track it's aliases and md5.
[18:40:29] <GothAlice> Depends on the application. With my 26 TiB Exocortex project, yeah, the BLOB data is separate. It'd be flatly unqueryable if I didn't do that. For smaller projects where the dataset (even with BLOBs) still fits in RAM, nah.
[18:41:18] <GothAlice> StephenLynx: Most filesystems behave badly with large numbers of files in one inode (directory), which then adds the complication of hierarchical organization, usually by substring prefixes. This way lies madness.
[18:41:41] <GothAlice> MacWinner: Yeah, it's a transparent proxy that records every digital bit of information I touch, and has been doing so since 2001.
[18:42:40] <GothAlice> StephenLynx: GNU/Linux isn't a filesystem. ext* filesystems behave _terribly_ (and you can run out of inodes! `free -i` to check), reiserfs is better (no limit), but directory listings can consume huge amounts of RAM when you have millions of files. Others get worse.
[18:45:44] <GothAlice> Exocortex serves several purposes: it's a great dataset for natural language processing and personal AI (computer vision, fact extraction, etc.) research. It's a development playground to try out new technologies, i.e. I added full-text indexing and compression to MongoDB 6 years ago. :)
[18:47:00] <GothAlice> It also serves an interesting purpose in predictive linking, i.e. I Google something and ignore the result page. Within 30 seconds I get two links pushed to my browser that are the "best" result, taking into consideration my active project, previous related searches, etc. (And since they were already fetched, no need to fetch them again.) Exocortex visited each result on the first page and performed its own ranking on them.
[18:49:20] <GothAlice> That includes a complete copy of Wikipedia, minus history deltas and discussion pages.
[18:50:08] <GothAlice> StephenLynx: Three Drobo 8-something-i RAID arrays and three Dell 1U rack servers.
[18:50:43] <GothAlice> (I get unlimited electricity at my apartment complex, and have a spare room with racks.)
[18:54:06] <GothAlice> It's this dataset that would cost me $500,000/mo. or so to host on compose.io. Silly cloud pricing.
[19:01:44] <StephenLynx> going to look into that. storing 26tb for just 5 bucks per month got my attention
[19:01:58] <GothAlice> T'was a bit of a frankenstein monster to get MongoDB on-disk stripes to back up that way, but it works. (A point-in-time freezing filesystem like ZFS makes this much easier.)
[19:03:16] <GothAlice> It's also deep-storage, so unlike Dropbox you don't have "live" access to your files. You have to queue up restores.
[19:04:08] <GothAlice> (Also fully encrypted if you so choose. That's the big reason I went with 'em.)
[19:12:19] <MacWinner> GothAlice, i figure I can do that later once hosted mongo costs become an issue.. right now I depend on reliability and I don't quite trust myself on it
[19:12:38] <GothAlice> StephenLynx: https://github.com/amcgregor/ (personal) https://github.com/marrow/ (open source collective) https://github.com/illico/ (work)
[19:13:48] <GothAlice> StephenLynx: Clueless is an unpublished WIP toy lang of mine. Most of its bits are in gists. https://gist.github.com/amcgregor/016098f96a687a6738a8 (docs) https://gist.github.com/amcgregor/a816599dc9df860f75bd (some sample code)
[19:14:42] <wayne> StephenLynx: another riddle for you! this is a player: {games: [{scores: [8,9,13213213213], cheat: true}, {scores: [1,2,3], cheat: false}]}
[19:15:16] <giowong> im getting a http://localhost:3000/[object%20Object] error but my json returns correctly, should i just ignore this?
[19:16:06] <GothAlice> StephenLynx: Not yet. I ran into a hiccup with the dynamic EBNF parser and flow grapher and Real Life™ side-tracked me.
[19:19:47] <GothAlice> wayne: Store a "top score" field on the player and have your updates (which add scores or games) only update that value if a) it's smaller than the maximum for that round/game and b) that round/game wasn't cheat.
[19:21:44] <GothAlice> It's technically pre-aggregation, which deserves a slightly different philosophy than "duplication". ;)
[19:24:44] <GothAlice> http://soundcloud.com/gothalice — I'm also on iTunes, etc., etc. and bitterness my own albums.
[19:25:56] <GothAlice> Many video game remixes, much random other stuff. My schtick is hybrid electronica with a small handful of natural instruments.
[19:26:38] <GothAlice> wayne: I'm an unusual human being. If it exists, at some point I've been paid to do it. ;P
[19:28:50] <windsurf_> which operator do I use to find all event objects whose categories array (array of _id) contains at least one matching _id in an array I have to compare it against?
[19:29:07] <GothAlice> wayne: I've been dox'd by a gaming group in the past. Always fun to have "that talk" with employers.
[19:29:21] <windsurf_> use case: user passes array of category ids he wishes to search within, need to find all events that are tagged with any of those categories
[19:31:26] <wayne> db.events.find({tags: {$elemMatch: {$in: ['trees', 'cars', 'chairs']}}}) // maybe?
[19:48:55] <whaley> GothAlice: is it ok if I PM you, regarding your music (I don't want to be offtopic)
[19:56:37] <giowong> so i have a array of objects, each object has a unique name, but have similar categories attribute
[20:15:02] <fewknow> giowong: not sure I follow..you have an array of sub-documents? each has a count and you want to sum them up ?
[20:17:48] <jake__> Hi. I'm trying to optimize a mapreduce job. Looking at db.currentOp() I see that "planSummary" : "COLLSCAN". Does this mean it isn't using an index for the query?
[20:25:05] <giowong> each object has a defaultCategory, which cna be the same across sub-documents, and each one has a unique count
[20:28:11] <MacWinner> fewknow, i have one database that will have a bunch of configuration information that will rarely change.. but another set of data that is activity data related to the configuration information
[20:28:32] <MacWinner> the activity and configuration info is heavy read... teh activity is also heavy write
[20:29:42] <fewknow> so you could test...but unless you need more than 2k writes a second you shouldn't need to worry
[20:33:05] <fewknow> but a read won't fail on a lock....and unless and unless you are hitting 2500 writes a second the read will be sub second most of the time
[20:33:45] <fewknow> but I never worry about reads due to a lock...espically since the working set of data is in memory
[20:53:16] <cook01> When you’re dealing with large collections (each document is fairly small) and trying to efficiently page over the entire collection, then is the recommended way to page with $sort and $limit? http://docs.mongodb.org/manual/reference/operator/aggregation/sort/
[20:55:27] <GothAlice> cook01: I'm not sure of the technical details in MongoDB, however as kali touches on, skipping will still require ordering the result set and massaging the records you're skipping. (Even the most efficient method of doing this, sorted on _id, would be a "log n" situation as it walks the index b-tree.)
[20:56:57] <kali> yeah, log(number of doc in the database) but at least it means you 1000e page will have the same performance as the first
[21:00:58] <cook01> It sounds like skipping on an indexed field is one of the better ways to page over a large collection
[21:02:13] <GothAlice> I.e. get the first page, remember the last record's sorted value, re-query asking for $gt that value.
[21:05:08] <cook01> fewknow: just trying to calculate a few stats that start with the objects in this collection
[21:07:02] <cook01> think of business stats/kpis that are derrived by using the documents in that collection
[21:11:00] <cook01> fewknow: Yep, I completely agree. This is just a quick solution (v1) that’s subject to a lot of change down the road
[21:12:31] <gansbrest> hi. I need to move one database from one replica set (beta) to prod replica set ( new db needs to be created there )
[22:11:44] <gansbrest> one other question, what's the best strategy to update beta replica set with production data? We have similar needs for solr and we ended up sending multiple writes to both though proxy library. Maybe there is something simpler for mongo?
[23:37:51] <GothAlice> dacuca: As a side note, I hope you are aware of http://docs.mongodb.org/manual/reference/operator/meta/comment/ which allows your application to send extra information to the back-end you seem to be writing.