Log file Viewer
#mongodb logs for Tuesday the 18th of June, 2013
[01:49:09] <jeffreytgilbert> hi! new to mongo. Trying to set a unique to all document properties of name XYZ, but it looks like based on the docs you have to do this when you insert the document, not by what would by the column name (as you would in mysql)
[01:49:24] <jeffreytgilbert> is that the right understanding or am i missing something? http://docs.mongodb.org/manual/tutorial/create-a-unique-index/
[07:41:50] <ncls> anyone knows if the perl driver error "can't get db response" is about to be fixed ?
[09:21:51] <hieutran> Should I use MongoDB for website email marketing? http://stackoverflow.com/questions/17144655/should-i-use-mongodb-for-website-email-marketing/
[09:37:29] <Hounddog> uhm i am getting duplicates when trying to implement a pagination using $lt... checking in the app and on mongo directly i see the data beeing different... let me paste something...
[09:39:39] <Hounddog> and i am checking in the database directly by just using the first find and using it then without the limit
[09:45:21] <Hounddog> http://pastebin.com/DWkas4XA the data is diverging at line 62 from the client against line 211 from mongo output
[09:49:51] <Hounddog> and there are no duplicates in the database... they are beeing produced during the find query...
[11:48:11] <manjush> I am confused with disks and memory. What does disk store in shared Disk architecture?
[11:57:29] <joannac> manjush: I don't know what you're asking. There's many nodes and one disk. THe disk stores all the data.
[12:04:43] <manjush> what kinds of data does the disk store is my question. I find shared disk and shared memory. What is the difference in type of data storage in memory and disk?
[12:11:28] <kali> manjush: http://docs.mongodb.org/manual/faq/storage/ make sure you understand every single work of that
[12:18:24] <Derick> joannac: your nick is really close to a friend which has only an "h" added after the "o" in your nick
[13:32:08] <landstalker> simple question, I want to turn my single server into a master/slave sync setup. Is the following logical:
[13:36:53] <landstalker> set the existing server as master without a slave present, then allow the backup to backup the oplog, use that backup to restore a slave server using mongo offline and dbpath option (while things still happen to the master) and then bring up the slave. It will then sync the remaining data from the master since the oplog changed? I only need to delete the local.* files on a slave if I'm going to turn it into a master right?
[13:48:35] <oceabreeze> Hello guys, where should i start learning MongoDB? Can you please recommend good books or other sources? Maybe some real life applications with open source code?
[13:51:28] <kali> dp: try that, maybe http://blog.mongodb.org/post/475279604/on-distributed-consistency-part-1
[13:51:57] <starfly> landstalker: suggest that you read up more on MongoDB replica sets (lots of doc available). Other than running a command to step down a master (a.k.a. primary) and kind of stacking the deck as to which is primary with config parameters, MongoDB generally manages which is primary and which is slave (a.k.a. secondary). It is suggested that you have at least 3 members of a replica set as well, since a majority of members is needed to elect a new
[13:52:51] <kali> dp: but mainly, it revolves around 1/ write concern 2/ replica set "read preference" settings
[13:54:55] <dp> kali: I'm fairly certain that we have no replicas (though it's possible I'm wrong about that). What do you mean by "write concern"?
[13:55:25] <dp> kali: I also noticed that those pages didn't mention the "consistency level can be picked query per query". that's specifically what I'm looking for
[13:55:42] <dp> if there's a way to force a higher consistency level for this particular query, it would be best
[13:57:59] <starfly> dp: p.s., if you really aren't using a replica set, then you can't get any more consistent, you are reading and writing to one location
[13:58:29] <dp> starfly: understood. but it seems in some cases, duplicate data is being written, and I'm not completely sure how
[14:06:59] <kali> dp: if you have no RS, then look for write concern. depending on the driver version, it's "safe" (which is required if you want strong consistency) or "fire-and-forget" (which is not even eventually consistent)
[14:12:43] <starfly> dp: so, what kali is saying is that to ensure your writes make it into the database, you can increase your write concern at many levels (e.g. you could just configure for the most rigorous write concern on critical write operations), but as far as whether another thread is reading consistent data, that can't be enhanced more than configuring your read preference to use the primary
[14:16:04] <dp> ok; looks like I was assigned a ticket that's already finished; just hasn't made it to production
[14:17:43] <dp> the code was changed from using an upsert, to a code-side if block, and then calling insert/update
[14:27:04] <starfly> dp: it isn't recommended, but depending upon how vital the data is and whether you need to manually review which of the dups need to stay or go, you could use the "nuclear" option to remove dups :)
[14:38:14] <freddj> We found a geometry (Polygon) refused by the 2dsphere index, if that interests someone.
[14:46:29] <freddj> yep, with kali, we tried to guess what could go wrong since a small perturbation makes the polygon valid again. It's not obvious to guess a pattern.
[14:52:53] <Derick> freddj: http://www.gcmap.com/mapui?P=116.0W+87.0181S+-+64E+87.0181S%2C+64E+87.0181S+-+64.0E+71.0181S%2C+64.0E+71.0181S+-+116.0W+71.0181S%2C116.0W+71.0181S+-+116.0W+87.0181S&MS=wls&DU=mi
[14:59:08] <landstalker> thanks starfly but we don't want to make any coding changes at the moment, which is required to use replicasets
[14:59:47] <Derick> landstalker: why do you need to make code changes? And yes, please avoid master-slave setups
[14:59:48] <landstalker> the reason we're doing this is the disk on master needs fscking and we'd rather switch to another server while it runs then have a lengthy outage
[15:01:49] <landstalker> we're a 24/7 operation, downtime isn't feasible for any length of time. The on disk size of a restored MongoDB is just over 205GB
[15:02:43] <landstalker> Derick: I thought I read somewhere using replica sets requires changes to how your application speaks to mongodb?
[15:09:17] <freddj> @Derick: ok, my fault. All our example were like this. Our geometry are only rectangles and we used Polygon to represent them, which is a mistake
[16:12:29] <nail_> Hello, with the safe mode 'on', is it guaranteed that if an update is issued by process A, then if a process B reads the same document after the update in A returns, it will see the update?
[16:26:29] <nail_> starfly: ok thanks! I was having problem with a producer notifying a message queue about a change, and the consumer not seeing the update, but that was actually a caching problem on the consumer side
[16:26:42] <nail_> and we can make the same assumption with the 'acknowledged' write concern, right?
[16:46:09] <starfly> nail_: yes, but if I were you, I'd double-check with your MongoDB's version of the write concern doc
[17:00:53] <Tommis_work> Hi there. I'm running a sharded cluster with 2 repl sets. For my collections im using _id as the shard key with _id being a md5 hash I generate from various fields of that record. Is this a good shard key? I'm finding 85% of my data ending up on one repl set.
[17:03:04] <starfly> Tommis_work: do you mean 85% of your data resides on one shard that is replicated in a replica set?
[17:03:51] <starfly> Tommis_work: shard key decisions are highly-application and data specific, what version of MongoDB?
[17:05:19] <Tommis_work> I'm assuming a "good" shard key is one that would split the data (as close to) evenly throughout the repl sets?
[17:06:17] <starfly> Tommis_work: you're actually talking about two different things, yes, you want to distribute data evenly across your shards, but then each shard may be part of a replica set.
[17:31:50] <diegows> for example, I have documents that I disabled and they are more than a half of my collection
[17:32:16] <diegows> and I wonder if sharing could work like an HSM, moving older docs (disabled) to a secondary mongo
[17:39:42] <diegows> right now I'm reading the documentation to find how to shard by my true/false field, I want to be sure that documents with enabled:true exists on a specific server
[17:47:15] <starfly> diegows: just use shard key of enabled and define two shards; you aren't going to get some of the traditional benefits of having many more shards to spread I/O around, but you can separate the enabled:true from the enabled:false documents, so your queries that go for enabled:true will result in a smaller working set and better performance. You will have document/chunk migration between the shards when enabled flips in document records, so hopefull
[17:48:26] <diegows> ok, but reading documentation I can select the shared key but I couldn't find and example about how to determine lo location of each document
[17:48:41] <diegows> I want the docs with enabled:true to be in a specific server (which has more resources
[17:51:34] <starfly> diegows: test it out, but generally you can do some preloading to see which end up where
[18:27:10] <starfly> diegows: you should be able to force enabled:true into the shard on the more capable server if you can take an outage and load that shard with enabled:true first, then the rest into the 2nd shard on the less capable server. I'd encourage you to do some trial runs outside of prod to see how it'll play out, but that's my best guess.
[18:35:25] <cmceldowney> mapReduce question-- having an issue with the collection that is output from this. It's like i'm getting records in the output collection that include un-reduced records as well as those that contain reduced records.. is there some flag I need to set to prevent outputting un-reduced content?
[18:49:29] <crudson1> cmceldowney: I'd wager that you are missing some assumptions about the reducer: 1) that it will not necessarily be called if only a single document matches a key, 2) the output of reduce can be passed into reduce again for the same key as a value
[18:51:27] <crudson1> cmceldowney: reduce: many_docs->one_doc for a given key, that's it. Seems like you want to push some logic to finalize
[18:52:10] <crudson1> http://docs.mongodb.org/manual/reference/command/mapReduce/#dbcmd.mapReduce Read carefully the "Requirements for the map|reduce|finalize Function"
[18:54:05] <cmceldowney> crudson1: thanks. yeah, it just seems like I have to move the data around either before I pass it into the reduce function, or in finalize to handle the cases where I only have a single value for a given key
[18:57:24] <crudson1> cmceldowney: finalize shouldn't care (or know) if there were 1 or 10000 documents that resulted in a given value. So thinking about it "handling the case of a single value" should be understood. The map function should do the work in creating documents that are appropriate for the calculation you are doing, and emitting those.
[19:38:32] <shayco> Hi all, does MongoDB have any advantages over Riak / Cassandra when implementing a grpah-like DB (for account / monitoring of ~500000 nodes * ~100 counters per node)
[19:39:41] <shayco> We're trying to grow our current environment (5000 nodes -> 20000 nodes) and when the interval hits msync, the IO subsystem just doesn't cut-it.
[21:17:29] <astro73|roam> How can I tell exactly how far behind a replica slave is behind the master?