Log file Viewer
#mongodb logs for Monday the 8th of October, 2012
[06:32:56] <tpae> is there a comparable speed difference with bulk insert vs. iterative approach insert?
[06:51:45] <LouisT> i don't think it can upsert multiple, but i don't think you need upsert as you stated you wanted to use multiple collections..?
[06:55:32] <LouisT> i have/needed to download huge text files and convert each line to an object to be inserted into a db
[08:24:44] <k610> i was planning to use sqlite but since my table schemas are going to change often I was considering mongodb
[08:44:33] <k610> ok, from what i read redis is going to perform faster but mongodb syntax is going to be more like sql
[08:47:50] <kali> k610: mmm not realy. the syntax is quite different, but it is query-based, whereas redis interface is closer to "hashmap and list on server" with similar interface to what standard api provides
[09:10:29] <christkv1> kali: I consider redis a swiss army knife of data structures and more often than not people use both mongodb and redis
[09:11:35] <christkv1> mongodb is more flexible due to it being a document database and having a query language and aggregation framework
[09:12:58] <kali> christkv1: actually, i've stopped using redis for queuing and shove everything into mongodb
[09:13:29] <kali> christkv1: redis was not ready for HA operation at that time, i don't know where it stands now
[09:15:25] <tpae> quick question.. i've been reading the docs, but can't seem to find good practice for adding indices.. when should i add an index, and when should i re-index?
[09:18:42] <kali> tpae: realy depends... you can do it in your application deployment process, or in your application initialization phase
[09:25:35] <kali> ppetermann: sure. from a development pov, it's a nice theory. but from the deployment and operations side, it's nice to have flexible tools and limit the number of infrastructure services
[09:26:08] <NodeX> "increments field by the number value if field is present in the object, otherwise sets field to the number value. This can also be used to decrement by using a negative value."
[09:27:16] <NodeX> and you should add indexes at the start really as they're blocking operations that stop reads/writes to your data while they're building
[09:29:33] <coalado> I have a large collection, and I need some kind of distinct feature on one field. what's the prefered way to do this?
[09:41:30] <mrpro> with c# driver, when you do getCollection() while there are connectivity issues, will it throw?
[10:20:06] <loganBS> Hi all. I'm puzzled about compound shard key. Is there a limit on number of fields which form a shard key? For example, can I use a compound shard key with 6 fields?
[10:33:59] <loganBS> christkv1: So can be a good idea to use a shard key as is: {coarsely ascending key (time); username; app;}?. My target is to store for every app of a user the hourly number of access.
[10:37:26] <loganBS> user document is like: {_id, user, app, year, month, day, {hour_1, hour_2,...hour_24}}
[10:40:47] <christkv1> to pick the shard key you should figure out how you are planning to access the data ?
[10:42:52] <christkv1> yes BUT no guessing is a good replacement for testing the shard key with real data :)
[10:43:21] <loganBS> but what happened when a collection is like this: {_id, user, app, PAGE, year, month, day, {hour_1, hour_2,...hour_24}}
[10:44:19] <loganBS> where number of PAGE is different for each user (i.e. a user can have only 2 pages, but another user can have 1 million of pages)?
[10:48:59] <loganBS> my concern is that a single shard become dedicated to an only user, without chance to split
[10:50:13] <loganBS> that's why is start thinking to use {coarsely ascending key (time); username; app;} shard key
[10:57:37] <loganBS> but using only user as shard key isn't there the risk to unbalance the cluster (a user with more page have a greater number of writes)?
[12:02:08] <coalado> I have a huge list of documents. each doc has a non-unique id. I'd like to count the amount of unique ids. any ideas?
[12:11:45] <NodeX> coalado : the aggregation framework will give you the list also if you want it to
[12:57:36] <loganBS> Another question about sharding: multi-range sharding (i.e. a shard node is responsible for different ranges, for example {-Inf - 100} and {2000-2100}) is what happen by default?
[13:05:28] <loganBS> NodeX: the question is another: a shard node can be responsible for more than one range? For example, when using a monotonically increasing shard key, all the writes are directed to the current shard; but when the last shard is reached, continue from the first shard?
[13:07:35] <tpae> according to this article: http://viktorpetersson.com/2012/02/13/comparing-mongodb-write-performance-on-centos-freebsd-and-ubuntu/
[13:33:26] <remonvv> tpae, that's an incredibly questionable benchmark. The main performance differences come from chosen filesystems (the defaults may be different for those OSs), memory manager, etc.
[13:35:36] <christkv1> tpae: we know there might be some penalty on freebsd due to super pages being turned on by default
[13:37:09] <remonvv> I wonder if there any metrics available concerning FreeBSD usage for medium to large MongoDB clusters.
[13:38:12] <christkv1> we would love to hear if anyone investigates and finds an optimal freebsd setup
[13:40:01] <remonvv> There are many presentations and articles available for optimizing Linux based setups.
[13:59:07] <ppetermann> if i have { _id: randomid, name: "sag", something: "foobar"} and { _id: randomotherid, name: "sag", something: "barfoo"}
[14:22:32] <sgtpepper> hello to all! Consider this situation: in a sharded environment one node reached its memory limit. What could happen at this point: 1) Add a more capable disk. 2) Add a new shard node. In the case 2 the full node is rebalancing?
[14:25:24] <kali> sgtpepper: a more capable disk will be what... at most 20% better than whatever you have now, whereas memory is orders of magnitude faster than disk.
[14:26:03] <kali> so if you add a node, the whole cluster will be rebalancing, stealing about 1/n of the content from the existing nodes
[14:29:24] <remonvv> sgtpepper, I guess the assumption that the shard key is actually correct doesn't fly or it wouldn't be just one node with memory issues.
[14:30:14] <sgtpepper> Ok, and if one has chosen a wrong shard key, is there a way to hot change it?
[14:30:27] <remonvv> sag, your question is a little vague but i'm pretty sure you are indeed looking for $in rather than $or. In terms of performance $in is faster if your use case allows for $in.
[14:31:46] <sgtpepper> remonvv: it's what i've suspected...so it's very crucial to chose the correct shard key
[14:35:10] <remonvv> Depends a bit on your data whether or not you should. Smaller shard keys tend to be "better"
[14:37:33] <sgtpepper> Because my assumption is to use: "month - username - category - type - page" where page is contained in a type, which is contained in a category
[14:38:08] <sgtpepper> or can be better something like this: "month - page", bypassing username, category and type?
[14:44:07] <remonvv> month shouldn't be in there, that's an aweful first key. it means all your data for a certain month is likely to go to the same (couple of) shards.
[14:46:48] <remonvv> sgtpepper, the goal is to make sure your shard key uses high cardinality fields that ideally are not based on time (at least not as the first field in the key). Things like UUIDs, usernames or other highly distributed IDs are good.
[14:47:20] <remonvv> sgtpepper, that's the point. You want it to be as large as possible. It means there are more unique values for that field.
[14:48:02] <remonvv> sgtpepper, imagine you have {month:1} as your shardkey and you have 100 shards and you'll see the problem.
[14:49:43] <remonvv> eka, --quiet helps a little and you can write the log output to the void. Why do you want it disabled?
[14:50:15] <sgtpepper> remonvv: ok, but using {month:1, username:1} the cluster is not balanced based on username?
[14:52:26] <sgtpepper> remonvv: I've read Scaling MongoDB of Kristina Chodorow...what I understand is that a shard key of the type {Coarsely ascending key + search key} could be good. Isn't true?
[14:53:20] <remonvv> sgtpepper, let's just say that will probably be worse than just {username:1}. Especially if "month" is something like the current month which means all new data will have the same value for month and will lean towards the same shard initially. MongoDB will split based on data hotspots so that'll eventually sort itself out but it just never actually helps the situation. Then if the next month arrives you'll be hitting the worst case scenario
[14:54:26] <remonvv> sgtpepper, kchodorow works for 10gen so I assume she knows what she's on about. I'm assuming the coarsely ascending key wasn't time based though. In my experience that's a particularly bad idea.
[14:55:53] <remonvv> sgtpepper, http://docs.mongodb.org/manual/core/sharding-internals/#sharding-internals-shard-keys
[14:57:49] <remonvv> sgtpepper, from my perspective the preference is high cardinality non time based fields with even value distribution.
[14:58:23] <remonvv> username is a very good value unless for example 30% of the database has "Guest" as their username.
[14:58:41] <sgtpepper> ...but when a certain username receive much more write/update operation compared to other? (for example has a greater number of pages, for each one an hourly counter should be stored)?
[14:59:07] <remonvv> sgtpepper, that's not relevant. If a single user receives a lot of writes those writes will always have to happen on the same shard anyway.
[14:59:33] <remonvv> And if that single user has a lot of associated data it will only arrive on the same shard if you use the same field(s) as shard key for that data.
[15:00:18] <sgtpepper> Isn't there a risk to have unbalanced shard (for example the one with this username has more data)?
[15:00:22] <remonvv> UUID is better than username due to length variability and usernames generally being larger.
[15:00:43] <remonvv> sgtpepper, well, unless you have incredible powerusers that can fill a single shard, no.
[15:01:23] <remonvv> sgtpepper, note that mongodb splits chunks. If a single shard is dealing with a lot of data from a few users it'll automatically split related chunks and rebalance them across the shards.
[15:01:46] <remonvv> sgtpepper, and it's not very realistic. You won't have a system where 0,1% of your users generate 99% of your data usually.
[15:05:33] <sgtpepper> but at the same time i would have balanced writes, i.e. not all writes should be directed to only one shard
[15:07:38] <NodeX> that has a 48gb MAX size across any of your shards at mongo's current doc size cap
[15:07:39] <remonvv> sgtpepper, you can add more fields to the shard key if you're worried some users might produce the bulk of your data. E.g. {username:1, productId:1}
[15:08:13] <remonvv> username sounds good but if you have other fields that have higher cardinality then by all means, use that.
[15:08:56] <remonvv> if I have 100k users on my system they tend to be roughly evenly distributed amongst the shards
[15:10:39] <sgtpepper> but in my case some user can generate more traffic than others, and what i don't what is that all the writes inherents to this users are directed to the same shard....that's the problem
[15:20:58] <remonvv> sgtpepper, and each user generates their own statistics? If so the scenario of having power users becomes a bit more realistic which makes username a bit less ideal.
[15:21:11] <kali> month-first sharding key... i think there is some value here if your app hammers recent activity... chunks with old data will slip out of the cache
[15:21:20] <remonvv> If you have a big customer that pumps GBs of data per day to you splitting on username wont help you much.
[15:22:06] <remonvv> and if you have month as your first shard key all data will be written to a single or a specific few shards with the rest idle.
[15:22:46] <remonvv> now if it's {second:1} it's somewhat different but then you might as well use _id
[15:23:01] <kali> remonvv: i don't know how the chunk storage is segregated... data from one chunk can be interleaved with another ?
[15:23:33] <kali> remonvv: as for the rest, i aggree, at the beginning of the month, all writes will got to one single shard until the second key starts to play
[15:24:12] <remonvv> kali, no, but caching happens based on memory pages. What chunks those pages belong to isn't that relevant. You are right when it comes to indexes though since that might help right-balancing the index. But for shard distribution it's not a good idea.
[15:24:21] <sgtpepper> the more complex collection is: username - application - category - type - page - year - month - day
[15:25:59] <sgtpepper> category and type cardinality is low, page (a visited page) could be high for certain users
[15:33:32] <remonvv> {username:1, day :1, page:1} would probably be reasonable then. Username ensures chunk distribution, once there the day+page part will help making sure often accessed data (assuming it goes per page) will end up in cache as kali suggests.
[15:36:43] <sgtpepper> Sure, realistic test will be run...but I wanted to have a clearer idea before :)
[15:42:09] <kali> sgtpepper: similar to month: at the beginning of each day, one single chunk will handle the write
[15:49:47] <gheegh> Hi all, I've outgrown my single Mongo server, and I'm about to upgrade. I've got 2 servers now to migrate to.. I'm wondering if I should shard them or run them master/slave? Everything we do is very bursty..and we have periods of super sustained writes. Once those writes are over though, the "write" load gets very low. What else should I be looking at to figure out how to most effectively use 2 servers?
[15:50:18] <kali> sgtpepper: when the chunk has outgrown the wanted chunk size... and at that point you get an expensive split and rebalance
[15:50:50] <Guest70315> can you update an index to be background: true? or do you have to destroy and recreate it?
[15:55:05] <ianblenke> if you're doing a find on all three at the same time, that makes sense as a clustered index.
[15:55:44] <ianblenke> if you're doing a find on them separately as well, you probably want an index on them separately as well.
[15:56:17] <gdoteof> i am trying to massage a database. i haven't interfaced with mogno very much at all.
[15:56:51] <gdoteof> if i do db.gaming_sessions.find({end:null}); i get a list of 7 'gaming sessions'
[16:03:23] <ianblenke> like Date()? something like this perhaps: db.test.update({end:null}, {$set: {end:Date()}}, false, true);
[16:07:09] <playcraft> What would be the most efficient, multi-user and shard friendly way to calculate the difference between two values and then calculate the average of the differences?
[16:09:19] <playcraft> MapReduce can get the job done but, from what I read it is not good to do it this way because it locks Javascript Context.
[16:10:51] <SergeyUkolov> Hi all! is there an analog in mongo for USE INDEX (index_list) (from MySQL)?
[16:17:03] <DrShoggoth> i find a lot of examples on doing "near" queries, but I only want to find shapes my point falls within
[18:41:21] <krispyjala> kali: I guess how does that work? Since initially standalone is just one "shard" right?
[18:42:54] <kali> krispyjala: http://docs.mongodb.org/manual/tutorial/convert-replica-set-to-replicated-shard-cluster/
[18:46:27] <kali> krispyjala: nope, it's actually the right one. you just need to make your app talking to the mongos instead of the mongod before enabling sharding
[18:48:02] <krispyjala> there's just one huge collection that we're trying to shard cuz it's getting too big
[18:48:49] <kali> yes. the only downtime is when you stop your app, change its config to make it speak to mongos instead of mongod
[18:50:41] <krispyjala> kali: one other question is, we're not even on replica set. We're still on master-slave, so would we need to convert to replica set first? or can just do all of it in one shot w/ the new sharding?
[19:05:53] <jgornick> Hey guys, are there any plans to have Mongo remove the limit of 16MB documents and use something similar to how GridFS partitions documents?
[19:06:16] <jrdn> so using the aggregation framework, i want to count all the items that exist in a document?.. just a simple count
[19:07:15] <jrdn> i.e.) document1 = { clicked_at: Date(), created_at: Date() } / document2 = { created_at: Date() }
[19:47:32] <mrpro> Unhandled Application Error – Key duplication when adding: authorization||System.ArgumentException||ArgumentException||Void PutImpl(System.Object, System.Object, Boolean)|| at System.Collections.Hashtable.PutImpl (System.Object key, System.Object value, Boolean overwrite) [0x00137] in /home/abuild/rpmbuild/BUILD/mono-2.10.5/mcs/class/corlib/System.Collections/Hashtable.cs:779
[19:47:32] <mrpro> at System.Collections.Hashtable.Add (System.Object key, System.Object value) [0x00000] in /home/abuild/rpmbuild/BUILD/mono-2.10.5/mcs/class/corlib/System.Collections/Hashtable.cs:442
[19:50:36] <_m> So, if you have a document with fields: foo, bar, baz And a unique constraint on foo
[19:50:59] <_m> You cannot insert a duplicate value for foo, which is what that error is telling you.
[19:53:42] <_m> Assume some records exist like with this data: {username: "foo", email: "foo@foo.com", remember_me: 'asdasdfasdf' } and { username: 'bar', email: 'bar@bar.com', remember_me: 'asdasdfasdf' }
[19:56:20] <_m> And given the message, I'm assuming you're inserting into an "authorization" collection
[20:00:40] <crudson> is it not that you are trying to overwrite an existing key in a C# hashtable where "overwrite" is false
[20:00:58] <crudson> from the C# docs: ArgumentException An element with the same key already exists in the Hashtable.
[20:02:28] <crudson> check that your client code is not adding the same thing twice to a mongo query perhaps?
[20:23:29] <kanzie> I have a collection of movies and I want to save which users like which movies by adding the userId in an array stored in the movies-collection. Every time the user press like on a movie I want to add their id to the array and increment the "likes"-field by one if the user has not liked the movie before, if already liked then skip. I can do this brute force with php but think there are nicer ways of doing this in the insert-statement for mo
[20:24:40] <kanzie> so my questions is: should the "likes"-property be a js-function to count items in the likedBy-array or should I have a integer value that I increment
[20:27:37] <kanzie> and should I do something like if ( db.movies.find({'_id': $myid, 'likedBy': $userId}) )
[20:30:01] <crudson> 1) the latter, 2) yes, as part of an update in which you can $inc the count and $push the userId
[20:30:50] <jrdn> depending on the traffic / amount of likes you get, it may be bad to push the userid
[20:31:23] <kanzie> jrdn crudson: thanks guys… since I was planning to build a scoring-function too that takes a bunch of properties from the movie item and does some magic on it I planned on doing this as a Score-field with a js-function that would calculate the score, but you don't recommend storing js-functions?
[20:32:08] <kanzie> jrdn: why is that? I was thinking perhaps just store the userid as strings instead of actual userIds, but I have no idea which is best from a mongo point of view
[20:32:52] <jrdn> i'm sure you can get a lot of user ids in there before hitting the 16MB document limit :P
[20:34:15] <jrdn> also, documents move on disk and you'll see high CPU / IO if they continuously grow in size.. i've noticed this with our current schema.. but i'm not a complete genius :X
[20:37:45] <crudson> kanzie: http://www.mongodb.org/display/DOCS/Updating#Updating-CheckingtheOutcomeofanUpdateRequest
[20:38:18] <kanzie> thanks, Im reading up on addToSet and push now but it seems a little wonky given what I need because I only want to increment if I successfully add
[20:39:35] <kanzie> maybe I should count the items in the array after addToSet and update the count thereby
[20:39:59] <kanzie> nah, better then to check if the id is in the set already, if not then perform update and increment, else skip
[20:40:52] <crudson> kanzie: that's why the query part of the update should have {users:{$ne:'123'}} so you are matching a movie that a user hasn't liked already
[20:42:03] <crudson> kanzie: "find movie x that user y hasn't liked, and increment the count and add the user to the list" can happen in one go
[20:42:12] <kanzie> but I already know the movieId and the userId, however if I do an update I would need to consider the return value and there doesn't seem to be any simple and fool proof way to do that
[20:42:46] <kanzie> crudson: well, if so I really need some help constructing that query cause I can't get my thick fingers around that
[20:46:10] <crudson> db.movies.update({_id:'titanic', userlikes:{$ne:'user123'}}, {$inc:{likes:1}, $push:{userlikes:'user123'}, false, false) could do it