Log file Viewer
#mongodb logs for Friday the 15th of June, 2012
[00:11:50] <multi_io> is there a way to atomically create a document ONLY IF it doesn't exist yet (as determined by a query for it returning nothing)?
[00:15:24] <richardraseley> I have a question related to sharding in MongoDB. So, I understand that sharding is done at the collection level, does that mean that any fields within any documents within that collection could be on any shard in the MongoDB infrastructure?
[00:16:19] <richardraseley> I didn't know if the document was the most granular unit that could be sharded or if fields within the same document could live in different shards?
[00:17:56] <richardraseley> So, the name of the document is the primary attribute that determines the placement within the parition?
[00:18:51] <richardraseley> Or is there a specific field within the document that is keyed off from?
[00:21:07] <richardraseley> multi_io: So, to clarify - a document can never be spread across multiple chunks
[00:31:26] <richardraseley> so the mongos process acts as a transaction coordinator / proxy for user requests - does that service have to query the config service for each query to determine where the chunks exist in the shards, or do they maintain knowledge of the chunk / shard configuration?
[00:38:31] <therealkasey> anyone have a pointer to startupStatus code meanings? I'm trying to debug a replica set where the nodes don't seem to see each other.
[01:24:56] <therealkasey> hmm, i figured out my replication issue, but i'm not quite sure what to do about it. i have a replicaset config that has gone out to all nodes with an incorrect hostname (the host that originally was the primary). the primary demoted itself to secondary, so i can't reconfigure the set from there. so i've got a set with no primary and a bad config on all nodes.
[01:51:11] <kotedo> To all the MongoDB guru's out there: Can I have something like a many to many relationship in MongoDB?
[02:05:23] <webjoe> if you want to get complicated (and cute), what happens a user is deleted? what happens if a group is deleted? etc
[02:06:55] <kotedo> so, maybe I should have a second database for millions of user and groups in a SQL fashion and the rest lives in MongoDB?
[02:07:15] <therealkasey> i would have an indexed field for groups in each user, where groups is a list of simple strings, then an indexed name attribute in the group
[02:09:05] <webjoe> kotedo - just go with whatever feels right, remember it's mongo, you're not committed to any structure
[02:09:25] <kotedo> maybe really going SQL on groups and users and all the other needed data is in mongo?
[02:11:02] <kotedo> yeah, I know, and I am sorry, I am just trying to find the right tool for the job
[02:30:41] <dstorrs> will there always be exactly 3, or is that a rolling log / constantly growing log ?
[02:31:29] <dstorrs> otherwise, yes, your schema looks good, but your indexes don't. you need multi-key indices in order to support those queries.
[02:34:29] <dstorrs> as to the indices, try something like this: { node : -1, client : 1 } => "has [node] logged [client]?"
[02:35:03] <dstorrs> you want to put the higher-selectivity ones first, so node before timestamp before client
[02:36:19] <Kane`> sorry, does that mean i should run my queries like: db.log.find({node: n, client: c}) as opposed to db.log.find({client: c, node: n}) ?
[02:36:52] <Kane`> dstorrs, "10 or more" is more of an average. some clients may only be logged be each node once. in that case, there'll only be one timestamp
[02:37:01] <dstorrs> the order you put them in the 'find' doesn't matter, but the order in the index matters a lot.
[02:37:29] <dstorrs> You want to build your index with the most selective item (i.e., the one with the fewest elements) on the left
[02:38:13] <dstorrs> check out this article on indexes: http://kylebanker.com/blog/2010/09/21/the-joy-of-mongodb-indexes/
[02:51:39] <dstorrs> what exactly is the difference between $pull and $pullAll in update()? they seem to do the same thing.
[02:53:41] <dstorrs> oh wait, I see. $pullAll takes an array and removes everything that matches any element of it.
[06:28:55] <sagarchalise> Hi this may be wrong question to ask but are there any resource on mongodb database samples for people who have only worked on RDBMS
[06:41:13] <sagarchalise> ukd1: I am having hard time designing mongo schema. I am thinking in mysql point of way of tables and relations.
[06:41:40] <sagarchalise> ukd1: thinking about a mapping sample of mysql database to mongo db design just for reference
[06:45:16] <ukd1> - this might be handy : http://www.mongodb.org/display/DOCS/SQL+to+Mongo+Mapping+Chart
[06:46:31] <sagarchalise> ukd1: what I am having problem with is say I have 10 tables on sql then would I go on creating 10 collections on mongo
[06:47:07] <ukd1> you could basically copy the layout from MySQL - but it's usually not the best way of doing it in Mongo, but will work.
[06:59:22] <heoa> Is there some command to quickly see examples/manual about things such as db.test.find(...)?
[07:51:43] <skot> maik_: ec2 is still bringing things up so we need to wait for EBS volumes to checkout
[08:48:58] <spillere> i'm doing z = db.users.find(), I want to get all data from the db, when I print z, it gives <pymongo.cursor.Cursor object at 0xb0d120ec>
[09:17:38] <multi_io> can a "find or initialize" sort of functionality be implemented atomically without locks?
[09:18:07] <multi_io> i.e., is there a way to atomically create a document ONLY IF it doesn't exist yet (as determined by a query for it returning nothing)?
[09:25:50] <multi_io> NodeX: problem with "craft a sneaky upsert that would not update anything" is that, if the document already exists, I don't know its contents. So I can't craft that upsert...
[09:26:22] <multi_io> (without first fetching the document, which would mean two operations, so it would no longer be atomic...)
[09:52:20] <millun> i am rewriting a part of code to work with BasicDBObjects as advised. it is going ok but i don't know how to issue "ensureIndexes" command
[13:29:00] <rhqq> which field will ensure me that i can switch master w/ replica and there will be no problems with it
[16:29:51] <rhqq> how big time-gap can be between nodes to make sure that replica will catch up to master?
[18:00:46] <kchodorow> DPP: http://www.mongodb.org/display/DOCS/Reconfiguring+a+replica+set+when+members+are+down
[18:04:13] <mitsuhiko> does the startup order for mongos and mongo controllers matter? is mongodb supposed to start up properly if the services go up in random order?
[18:04:26] <mitsuhiko> i noticed mongos being stuck if we reboot the whole cluster at once for testing purposes
[18:28:47] <mikerobi> is it possible to get a javascript stack trace for exceptions in db.eval calls?
[18:48:35] <adamt> If anybody sees the auther of the Haskell-bindings, do let him know that there's a problem with the dependencies in the cabal-files for both version 1.2 and 1.3 of the bindings.
[20:19:03] <zonetti> Derander_, I'm trying PHP requesting a nodejs API through CURL... I already tried htmlentities, utf8_encode...
[20:19:10] <zonetti> Derick, , I'm trying PHP requesting a nodejs API through CURL... I already tried htmlentities, utf8_encode...
[20:24:41] <dstorrs> I'm trying to do a match that returns only one embedded document and having trouble. Example is here: http://pastie.org/4094338
[20:25:16] <dstorrs> I thought I could make this work with the positional document, but no joy (maybe just PEBKAC error, though)
[20:29:56] <dstorrs> I tried this > db.temp.find({ 'pages.owner' : 'worker_9'}, { 'pages.$':1}) but it returns { "_id" : "carol", "pages" : [ { }, { } ] }, when what I actually want is { num:1, owner:'worker_8'}
[20:34:35] <zonetti> some post in stackoverflow gives the link to mongodb repository when they say they will implement find() in embedded docs...
[20:34:36] <dstorrs> zonetti: ah, ok. So, I have to return the main document, although I can restrict to the particular top-level fields I care about?
[20:35:12] <zonetti> dstorrs, in the app I'm developing, when I need to do that, I find like you did and then iterate over the results =/
[20:39:38] <multi_io> mitsuhiko: allocate using a separate "counter" or maybe using an optimistic strategy
[20:40:03] <multi_io> the mongodb docs at one point say that you should use natural id if applicable, iirc.
[20:41:20] <multi_io> dstorrs: the point is, we need that unique integer anyway (it's required by the use case), so it's a "natural" key, not an artificial one
[20:42:00] <multi_io> I was thinking of making the counter values globally unique using only information that's available locally on the shard
[20:46:06] <multi_io> the customer requires that the numbers be roughly incrementing values, starting from 1. They are supposed to be exposed to end users :P
[20:47:53] <mitsuhiko> multi_io: well supported format in general, entirely random if you use uuid4, perfect for sharding
[20:48:52] <mitsuhiko> multi_io: i don't like objectid's because a) specific to mongodb, b) 32bit timestamp, c) not entirely random
[20:49:30] <mitsuhiko> the only reason to use mongodb is if you want sharding. otherwise it's just a pain
[20:53:58] <multi_io> mitsuhiko: you would prefer Postgres over MongoDB for any use case as long as it doesn't involve sharding?
[20:54:24] <mitsuhiko> tilleps: if they don't find the mongo config servers they become headless zombies
[20:54:37] <mitsuhiko> tilleps: they keep retrying but never manage to find them, if you restart them they can
[20:55:31] <mitsuhiko> multi_io: mongodb is great for my particular usecase, don't get me wrong, but it's very unreliable still and it needs someone to sit next to keeping it running
[20:55:57] <mitsuhiko> and if you run it with less than three controller servers i just don't trust it yet
[20:56:15] <multi_io> that's kind of a minority opinion I guess :P I can't imagine that it would be so hard to implement app-specific sharding yourself in Postgres
[20:59:42] <richardraseley> Just trying to clarify some points with regard to MongoDB sharding a replication. So the smallest component that will be sharded is the document. It is sharded according to the shard key which is selected and must be unique to all documents. multiple documents with contiguous shard keys will exist in a chunk and chunks are divided across shards as needed? Is that correct?
[21:00:12] <multi_io> mitsuhiko: so you wouldn't use mongodb because it's too unreliable, not because it doesn't provide any more features than, say, Postgres.
[21:08:39] <richardraseley> multi_io: Hmm, I thought that was the case - but assuming your right, can you speak to the rest of my assumptions?
[21:10:11] <richardraseley> multi_io: Also, can you comment on whether or not the mongo routing service has to talk to the mongo config service for every query, or if it maintains a copy of the shard configuration locally as well?
[21:13:01] <richardraseley> multi_io: Just confirming that you are correct with regard to shard key uniqueness (per http://docs.mongodb.org/manual/core/sharding/#sharding-shard-key)
[21:13:13] <jstout24> i'm trying to think of the best schema design for an events passing an object like, `db.events.insert({ name: 'impression', parameters: { page: 'some_page_id', layout: 'some_layout_id', ……., visitor: 'some_visitor_id' });
[21:13:26] <jstout24> i was thinking about turning the parameters into a key / value pair and do a multi-index on that
[21:14:47] <multi_io> richardraseley: I guess it keeps it in memory, but I don't know, sorry. I haven't done sharding in production :)
[21:16:38] <richardraseley> Can anyone else confirm my assumptions in the comment above? Documents being the most granular unit that can be sharded, being gathered in ranges in "chunks" and spread across shards as needed?
[21:33:01] <jstout24> if i have a document with field "data" which has key / value pairs… how can i search all documents where `(data.k = 'foo_k' & data.v = 'foo_v') AND (data.k = 'bar_d' & data.v = 'bar_v')`
[21:59:48] <mitsuhiko> multi_io: mongodb without replication is unreliable. mongodb with replication is a very complex setup compared to postgres
[22:00:15] <mitsuhiko> on top of that you don't have transaction safety or the expressiveness of sql
[22:02:23] <mitsuhiko> richardraseley: "Also, can you comment on whether or not the mongo routing service has to talk to the mongo config service for every query" <- talks to the config service yes
[22:03:07] <richardraseley> Client Request -> Router -> Config to determine placement -> to appropriate shard -> back to client?
[22:03:40] <richardraseley> Also, with regard to scalability write performance - it is safe to say that you are limited in write performance on a single document to that maximum performance of the master node in the shard that owns the chunk in which it lives?
[22:04:25] <mitsuhiko> richardraseley: actually, it does not need to talk to the config all the time
[22:05:18] <mitsuhiko> see also: http://docs.mongodb.org/manual/faq/sharding/#when-do-the-mongos-servers-detect-config-server-changes
[23:54:39] <dstorrs> ok, this has to be an ID10T error on my part but I don't see it. count_pages = function(v) { print(v.pages.length) }; count_pages.apply({pages : [] }) => 'TypeError: v has no properties (shell):1'