Log file Viewer
#mongodb logs for Thursday the 20th of November, 2014
[00:02:53] <xissburg> THis is pymongo specific.. it seems to me it is not properly handling date timezones
[00:03:22] <xissburg> I set the tzinfo in the date attribute but when I retrieve it is has no timezone
[00:04:20] <GothAlice> xissburg: There are all sorts of caveats to using Python datetime objects with timezones.
[00:04:51] <GothAlice> xissburg: Well, it sucks less than most who run into issues think. Install pytz, use it, be happy. :)
[00:05:23] <xissburg> GothAlice: I am using twisted.amp and it has a amp.utc thingy that I can set to the tzinfo and I am doing it
[00:08:10] <xissburg> datetime.datetime.now(utc) where utc is defined here http://twistedmatrix.com/trac/browser/trunk/twisted/protocols/amp.py line 2584, instance of _FixedOffsetTZInfo
[00:10:01] <GothAlice> xissburg: Read through the 'pytz' manual — it covers why naive timezone casting will be wrong.
[00:10:26] <GothAlice> (And why it provides it's own Timezone.in_timezone(datetime) casting function.)
[00:11:34] <GothAlice> Regardless, PyMongo by default stores datetimes in UTC. (So a naive/tz-less datetime produced by utcnow() would work fine without any extra machinery.) Are you attempting to *store* dates in the DB in different zones (generally a Very Bad Idea™) or is this a display issue?
[00:13:45] <GothAlice> Basically .replace() avoids triggering timezone conversion machinery (it just drops the replacement zone in and says "really, this is a datetime in X timezone!").
[00:14:28] <GothAlice> xissburg: http://leach.it/python/2013/12/21/understanding-pymongo-and-datetimes.html may be a useful article.
[00:19:47] <xissburg> This is the date I have in mongo "date" : ISODate("2014-11-20T00:14:57.131Z"), this is what I get in python 'date': datetime.datetime(2014, 11, 20, 0, 14, 57, 131000)
[00:24:48] <xissburg> Yea, that date.tzinfo is None, even though it was set to something before saving
[01:59:28] <shoerain> why would mongostat --username <username> --password <password> (with the correct versions of both) give me assertion 18: auth fails? db.auth(<username>, <password>) work just fine
[02:07:05] <GothAlice> shoerain: Users are tied to databases in MongoDB, so if you added your user to the 'admin' database, you'll need to tell the tools to authenticate against that database specifically, regardless of which other database the user will use.
[02:53:34] <kevc> is it possible to shard over multiple replica sets? so that one collection has a master in NY and another collection has a master in London?
[02:59:51] <shoerain> So... it looks like I'm making some 50-ish queries on a few database collections in my teensy node.js app every time I serve "GET /". Do I need to worry about batching them myself to reduce the number of queries to the database?
[03:01:31] <GothAlice> shoerain: Have I linked you that Java article about schema design yet? (I swear that's my #1 most used link. ;)
[03:01:57] <GothAlice> (http://www.javaworld.com/article/2088406/enterprise-java/how-to-screw-up-your-mongodb-schema-design.html)
[03:02:00] <shoerain> I have 50-ish tasks, I look up a collection "Permission" for each task, then I look up a collection "Organization" for each "Permission". so I guess it could be 3x50==150... hmm
[03:03:00] <shoerain> heh, what a huge difference testing locally and against a faraway machine makes
[03:04:23] <kevc> GothAlice: So it seems you can create multiple shards, each with their own replica set. Think where I get stuck is how to say which shards a collection is using. Seems you can only tell a collection to use all shards.
[03:05:41] <GothAlice> I.e. you'd have a primary of each shard in the 'local' DC, secondaries in other DCs. (This gives 'local' reads for those other DCs and offsite backup.)
[03:06:48] <GothAlice> You could also have a field, i.e. "dc", on your documents which you shard upon, with the sharding index selecting which DC to send the data to. (But this means that larger queries across DCs get kinda crazy.)
[03:07:09] <kevc> GothAlice: So I get shard A (rs0, primary in NY) and shard B (rs1, primary in London)
[03:08:40] <GothAlice> kevc: http://docs.mongodb.org/manual/core/sharding-shard-key/#sharding-shard-key-query-isolation
[03:08:44] <shoerain> How should I batch a bunch of queries together? this aggregate framework sounds kind of like what I would want...
[03:09:46] <ObjectiveCopley> Not trolling: What advantage does a document based DBMS have over relational? When would I use document based over relational?
[03:11:11] <GothAlice> ObjectiveCopley: I issue a single query, with no joins, returning a single record representing the entire contents of a forum thread, with the sub-documents (replies) sliced to the current page. Query time: unmeasurable. Document systems let you manage, model, and explore your data with fewer (*cough*) constraints.
[03:12:45] <ObjectiveCopley> My coworker and I both have used RDBMS's and Mongo and couldn't think of a use case for Mongo that couldn't be easily solved in a RDBMS but I guess a forum thread would be a good example
[03:13:22] <GothAlice> ObjectiveCopley: Additionally, MongoDB provides three different (but re-using of their semantics) query systems: normal find (akin to SELECT), map/reduce (map/reduce ;), and aggregate pipelines. Aggregate pipelines (and to a lesser degree map/reduce) can be optimized to run in parallel for some truly extreme performance.
[03:14:00] <GothAlice> http://docs.mongodb.org/manual/core/aggregation-introduction/ gives an overview; it's a very different way of thinking about how you ask questions of and process your data.
[03:14:39] <GothAlice> Nor does it have referential integrity, nor any way to perform automatic joins of returned data.
[03:14:47] <ObjectiveCopley> So if you absolutely have to have constraints, MongoDB is not a good solution
[03:15:20] <GothAlice> ObjectiveCopley: http://www.javaworld.com/article/2088406/enterprise-java/how-to-screw-up-your-mongodb-schema-design.html is an excellent (and quite short) overview of how MongoDB thinks about data vs. relational databases when comparing standard data modelling practices.
[03:16:41] <GothAlice> ObjectiveCopley: You *can* implement two-phase commits in MongoDB if you wish, but yeah, pragmatism usually wins. Use the right tool for the job: if you need transactional compliance or enforced referential integrity (instead of referential durability which is the approach in MongoDB), a relational database may be a better solution. It's the same with modelling graphs—or even worse, depending on how naively one tries to implement that in
[03:16:55] <ObjectiveCopley> And I guess if you had a (large) piece of data that could be used by multiple different objects
[03:17:37] <GothAlice> ObjectiveCopley: Depends quite heavily. A question often raised in this channel revolves around apparent "duplication of data" when the issue is actually one of optimizing query performance.
[03:17:45] <ObjectiveCopley> we found Postgresql and mysql to be absurdly slow for our needs, but mongo was exceptionally fast
[03:17:53] <kevc> We've got a system where data is coming in from multiple sources and new features are being added all the time. We want developers to be able to develop tools querying those datasets as they become available. Doing all of this with say Postgres would be a lot of work.
[03:18:24] <ObjectiveCopley> but that was the only solution we could come up with where mongo made sense
[03:18:59] <ObjectiveCopley> Because you could have a server, which had reports, and each reports had metadata, and if reports ran every 5 minutes, and each report had 3000 pieces of metadata, it became a massive tree of queries
[03:19:14] <ObjectiveCopley> Mongo made it exceptionally fast because we could keep it all in the same object
[03:19:53] <GothAlice> ObjectiveCopley: I have a personal dataset exceeding 25TiB in size, and at work I perform large-scale service monitoring, aggregated logging, and click stat analytics, use a cMS (component Management System, not Content ;), and even wrote our own Sphinx-like Okapi BM-25 full text indexer and parallel ranker for it. All on MongoDB.
[03:20:31] <GothAlice> (The search ranker can rank 10 years of indexed PDFs in less than two seconds. It's so fast we don't even bother to paginate results in the cMS.)
[03:21:59] <GothAlice> We even use it to replace Celery and other distributed task queue systems, a la the presentation I gave on that system: https://gist.github.com/amcgregor/4207375 (1.9 million dRPC requests per second per host at last stress test.)
[03:25:20] <GothAlice> Oh… and the authentication service and forums I wrote to power a community site for the second largest alliance in EVE Online. (12K members, also part of a coalition of alliances who also use the services totalling maybe… 16K?)
[03:28:53] <GothAlice> ObjectiveCopley: Apologies for the… tirade… but I wanted to very clearly illustrate that MongoDB can solve many, many different problems in very acceptable ways.
[03:29:09] <scrandaddy> Hey there. I am designing an application that is heavy on http request processing in real time. I would like to use mongo but am concerned about doing a lot of single writes as each request comes in. A solution I thought of is to use redis and batch write to mongo every so often. Is this a better idea? I'm looking at a cloud solution like mongolabs, maybe
[03:30:05] <GothAlice> scrandaddy: In theory the majority of execution time will be spent either fetching data over the wire or processing that data, not actually spent inside MongoDB.
[03:32:13] <GothAlice> Well, first, AFIK MongoDB has no such concept as a "batched write" (i.e. no multiple insert)… but I'll double check that.
[03:33:45] <GothAlice> scrandaddy: https://gist.github.com/amcgregor/7be2ec27adc80c9fafa1#file-sync-cvm-py-L21-L54 is an example of our HTTP scraping data processing pipeline from work. (In this case a simplified example that parses RSS instead of having to literally scrape pages, but it still does HTTP for this.) The pipeline is designed to stream—as each job is scraped it's passed up the chain for insertion. (Source gets a minimal set, Parser would then fill
[03:34:12] <Boomtime> however, builk ops are wire-protocol only, the operations themselves still take place individually at the server
[03:34:35] <Boomtime> http://docs.mongodb.org/manual/reference/method/db.collection.initializeUnorderedBulkOp/
[03:34:37] <GothAlice> (In this way we can separately scale each step of the processing… the top level JobSync doesn't care about the order of the scraping, so I can just run one which consumes jobs produced by all parsers, etc., and this would simplify batching of data quite substantially if I went that direction.)
[03:39:10] <GothAlice> And the important caveat that if there is an error in the data part way through, it'll stop processing that batch.
[03:39:50] <scrandaddy> I'm just worried that if I have too many requests coming in doing single writes to mongo I will big the system down
[03:42:54] <Boomtime> scrandaddy: you want to use redis as a "cache" except that writes have to be written sooner or later to MongoDB.. so the same number of writes still exist, and the cache is invalidated when you do this.. so how does redis help at all?
[03:44:44] <GothAlice> scrandaddy: If you want deferred processing (i.e. one process that streams HTTP data into MongoDB, another that processes that data into real documents) you can insert the raw data into a capped collection that is being watched by a tailing cursor in the process that parses the data, but that will potentially make the "bogging down" problem worse, as the capped collection is by definition limited in size. This can give you some buffer time to
[03:46:28] <Boomtime> you seem to be inventing reasons to use redis, but can't quantify those reasons
[03:46:29] <GothAlice> scrandaddy: As a note, redis scaling badly as a way of tracking distributed task execution (similar to using it to buffer data processing), oddly enough, is what killed my last project.
[03:47:11] <Boomtime> if you don't want persistent storage then you might be better off with redis only
[03:48:43] <scrandaddy> Yeah, it will eventually live in s3 but I would like to make some of it available over an api before hand and my understanding is that redis might only live for minutes at a time
[03:48:44] <Boomtime> adding redis to your setup (in conjunction with mongodb) will add a new point of failure, add latency to all persistent storage (your data is more vulnerable) and add complexity to your deployment
[03:50:51] <scrandaddy> GothAlice: what if instead of watching with a tail cursor I just processed the raw data every minute?
[03:51:41] <GothAlice> scrandaddy: https://gist.github.com/amcgregor/4207375 may be helpful. In that situation (which you certainly could do!) you'd have to make very sure that your capped collection is large enough to handle the expected time period… with sufficient head-room to give you time to react to scaling issues.
[03:52:16] <GothAlice> (At work, using a capped collection as per that presentation, we estimated 8GB capped collection to maintain 1M simultaneous users' games.)
[03:52:50] <GothAlice> The "head-room" on that was that we didn't have a million users… not even close. ;)
[08:02:22] <GothAlice> One of my work datasets is ~100MB, my home dataset (as mentioned in the log) is rather terrifyingly large.
[08:04:10] <GothAlice> 100MB isn't very much at all. (Maybe 20,000 records total in that set spread across several collections; most of that data was slurped via web scraping.)
[08:05:31] <GothAlice> Duikb00t: I do have to apologize, though, for having to run. It's 3am here and I need some offline maintenance so I can be functional tomorrow. ;) I mainly wanted to point you at that log as your question isn't uncommon.
[09:09:06] <Sunuba> Greetings all. I am new and I do not know how I should ask for help here or privately. Can someone please guide me?
[10:18:36] <kevc> but when I look at db.locks.all() there's an entry there for "upgrading config database to new format v5"
[17:23:15] <AlexZanf> Derick, so lets say, i have articles, and articles has comments, and comments has tags, I would like to get articles, with the comments and with the comment tags
[17:23:54] <AlexZanf> not using mongoose, thought it was a standard term, if not, how do you refer to that?
[17:29:23] <brianseeders> you would typically manually query for the related documents in your application code
[17:29:33] <brianseeders> i'm not super familiar with mongoose, but I believe it has a helper for it called populate
[17:30:31] <AlexZanf> brianseeders, oh im not using mongoose, but is there an example of manually querying? wouldnt that be slow? if you get 1000 results, you have to manually query all 1000, and say they each have 100 associasions, then you need to query those, and so on
[17:33:20] <brianseeders> although you should be able collect up all the IDs you need to pull on the second step and do them in bulk with something like $in
[17:35:08] <brianseeders> mongo shines when the data is de-normalized and you don't need arbitrary joins like that
[17:41:31] <sekyms> I'm not sure why this isn't working https://gist.github.com/smykes/6f9b86d5c9c1bb26354f
[17:49:06] <kephu> in my aggregation query I wanted to pick the largest of the two: $size: {$setDifference: [array1, array2]} and $size: {$setDifference: [array2, array1]}
[17:50:53] <kephu> GothAlice: yeah, I figured that had something to do with that, but I'm apparently messing it up a bit, I think
[17:51:40] <GothAlice> http://docs.mongodb.org/v2.6/reference/operator/aggregation/max/#grp._S_max — this would require unwinding on the array of the two sizes first
[17:54:25] <GothAlice> OTOH… I'm wondering if you really need to do that $setDifference twice… if you're only counting the length of the intersection (or length of the difference… since a~b is congruent to b~a when comparing common membership…)
[17:58:20] <GothAlice> rendar: Mispronunciation of "milk" — my friends and I have some atypical ways of saying things, so don't mind my butchering of other language snippets.
[18:02:20] <GothAlice> kephu: When working with sets… it's pretty important to have a solid grasp of the basic set operations. Combining them in creative ways is the path to greatness lined with much mathematical theory.
[18:05:01] <GothAlice> Get the length of the union of the two sets subtracted from the maximal of the length of those two lists. You could potentially use an $if (to pick the larger of the two) with a $let (to define the two lengths for the conditional).
[18:06:31] <GothAlice> (You could also implement a non-$group'd $max that way, too, for the first case. I'd pre-calculate and store the lengths (or single maximal length) if this will be a common query. I.e. every time you $push, also $inc, each time you $pull also $inc: -1)
[18:07:07] <GothAlice> Pre-calculation of that can become a PITA if you don't carefully handle maintaining those lists as actual sets.
[18:25:58] <jshultzy> hey guys, anyone kind enough to help me out with a problem im having in my mongo master/slave config?
[18:29:32] <jshultzy> i've searched all over the net including mongo forums but couldnt find a solution. basically whats happening is that im recieving the following error in mongo logs on the slave: [replslave] all sources dead: data too stale halted replication, sleeping for 5 seconds i tried shutting down the mongo daemon, clearing out the data directory on the slave, and doing a full re-sync from master but what happens is that
[18:29:32] <jshultzy> Thu Nov 20 18:17:51.890 [replslave] repl: end sync_pullOpLog syncedTo: Nov 19 19:17:11 546cecb7:d
[18:29:33] <jshultzy> Thu Nov 20 18:17:52.899 [replslave] repl: old cursor isDead, will initiate a new one
[18:29:33] <jshultzy> Thu Nov 20 18:17:52.971 [replslave] repl: nextOpTime Nov 20 13:10:08 546de830:8 > syncedTo Nov 19 19:17:11 546cecb7:d
[18:29:36] <jshultzy> [replslave] all sources dead: data too stale halted replication, sleeping for 5 seconds
[18:30:32] <jshultzy> sorry but, new to IRC. I just set up account to see if I can get any help as a last resolt
[18:35:48] <jshultzy> thanks kali for responding. hmm, even after clearing out the old database data directory from the slave and re-syncing, what do you recommend to do
[18:37:17] <jshultzy> oh gosh probably like 14 or so hours of running the resync to pull all the 850GB of database over to the slave
[18:39:18] <kali> jshultzy: yeah, ok. when you say "master/slave"... are you actually using master/slave or is it a replica set ?
[18:43:28] <kali> ok, not so bad. well, i have no idea how to operate and maintain actual master/slave setup
[18:44:16] <jshultzy> think its just time for me to upgrade and use replset instead of master/slave. Thanks anyways kali
[18:45:14] <kali> one more thing: your actual problem is that the oplog is too small for the amount of time the slave needs to pull your data from the master
[18:46:02] <GothAlice> kephu: You have two possible approaches: $if with $let to calculate the lengths for use in the condition and for use in the true and false result output (simulate "max" on two elements) or use $group and $max in the $group projection. (This will require a $unwind before-hand.)
[18:49:10] <kali> jshultzy: it is possible that just stopping the master, changing the command line argument (or conf) and starting it again does the trick, but I have no first (or even second) hand experience in the matter
[18:51:50] <kali> jshultzy: the oplog size. but i would feel better if you find out somebody who has done this before on a master/slave setup :)
[18:53:42] <kali> jshultzy: for reference, this is the procedure for a replica set: http://docs.mongodb.org/manual/tutorial/change-oplog-size/
[18:53:57] <jshultzy> Kali, I will look into this but you gave me something to look into now. Before, I literally was out of ideas. Thank you soo much!
[18:54:37] <kali> jshultzy: it involves keeping a "seed", but i think this is only relevant on a secondary
[19:22:14] <shoerain> hmm is there an equivalent of mongoose's populate on the node-mongodb-native driver?
[20:10:27] <shoerain> er, yeah I can't seem to find something akin to http://mongoosejs.com/docs/populate.html in node-mongodb-native. DBRef/manual references seem like what I would want
[20:14:22] <mtrivedi> Hello IRC users, we want to achieve write scaling in production mongodb cluster. for my tests, I have added hashed sharded key to achieve even writes to all shards. but overall throughput remains same before using hashed key
[21:36:53] <qutwala> hi, there, collection.findAndModify(criteria[, sort[, update[, options]]], callback) wont work on 2.6.3 version :( Any ideas for callback?
[21:38:49] <reelman14> Hi, I created even after creating an index on all the fields queried on, mongodb is scanning the entire collection, I have the index creation and query both here: http://pastie.org/9733221 Can you please tell what is wrong with the index or what is the right index to create?
[21:49:02] <ianp> is there a convenient way to do an update to copy a property tree to another place in mongo? like I have document.props which is nested json
[21:50:12] <GothAlice> Well, by "write it back out" a better approach would be to $set the new value and $unset the old one, rather than pushing the whole document back out.
[21:51:26] <GothAlice> var props = db.example.findOne({_id: ObjectId('…')}, {props: 1}).props; db.example.update({_id: ObjectId('…')}, {$set: {'otherprops.props': props}})
[21:57:08] <GothAlice> (Without unsetting) each record will grow, and potentially grow substantially if that's a nested document being moved, so each record will have to be moved in order to accomplish the update.
[22:00:42] <GothAlice> (Asking the last due to the duplication issue; if those nested values are written to, you'll have to worry about moving the data back to the top-level field.
[22:06:02] <hfp_work> Hey all, I have a running `mongod` on my machine. I want to run some queries against it so I do `mongo`. And then nothing happens. mongo just hangs there saying nothing whatsoever. If I enter any command, nothing happens. What am I missing?
[22:06:50] <GothAlice> hfp_work: If you run the following, do you see the running mongod process? ps aux | grep mongod
[22:07:24] <GothAlice> (Also, what command-line arguments did you pass to that mongod process? If you specified a configuration file, could you pastebin it?)
[22:09:51] <GothAlice> hfp_work: So far so good. What's the output (should be one line) of: sudo lsof | grep mongod | grep TCP
[22:19:17] <sandis> Im new to mongodb from postgresql. Is there a good "migration guide" to get the thinking?