Log file Viewer
#mongodb logs for Saturday the 9th of June, 2012
[00:24:56] <dstorrs> in the Perl driver, is 'ensure_index' a no-op if the index already exists? how would I find out?
[01:58:35] <dstorrs> from the Perl driver, what's the correct syntax for the 'scope' param on a mapreduce?
[02:19:34] <Max-P> Hi, just a quick advice request: I have to store daily data for a user, which one is better: having a collection for each actions independant or to store all of that in a big "day" collection?
[05:50:21] <Max-P> sirpengi: I went with one collection, sorry I went back to work ;) "each actions independant" meant that a user can add multiple activities to their calendar, but they can edit them individually, move them from a day to another, reorder them. I still considered it as a whole day, as managing them seperately would have been too much trouble. Thanks anyway, it confirms that I went the right way ^^
[05:58:03] <jgornick> hey guys, would it be possible to find records where the date lands on a tuesday?
[06:01:17] <henrykim> I would like to ask the situation I got now. I am under testing write operation of blog articles. we have now about 1 billion blog articles now.
[06:01:59] <henrykim> yes, I started to insert(write) articles one by one. shard is url which has about 70 bytes length.
[06:03:00] <henrykim> at the beginning, the performance is over 3000 TPS. but, 1 or 2 hours later, the performance is decreased the half of the first of it.
[06:05:52] <henrykim> Internally, It's because of paging. to write an article 'a' url, mongo need to load 1 chunk. to write an article 'b' url, mongo need to load 10 chunks .. and then. and then...
[06:23:14] <Max-P> henrykim: I'm new here and don't know much about Mongo, but you might need to change the way your stuff is linked so I don't need to do that much read/write...
[06:24:48] <Max-P> henrykim: sorry it's 2:30 AM here, I will try again: why do you need to load that much documents when inserting your article? (if I understood your problem)
[06:27:39] <henrykim> Max-P: different time zone ;) here I am living is quite shiny day. anyways, I am testing mongdb to use for our business.
[06:30:16] <henrykim> problem I already described is write operations are happened randomly. random.random.
[06:32:13] <Max-P> If you properly indexed all of this, then I'm sorry I can't help. I thought you had some linking between your blog thing that slowed down the process, so I thought you might have had to change the way you index and traverse the collection, but I'm no use there, sorry
[08:13:55] <stefancrs> I basically want to store "heartbeats" from different sources in a collection. My first naive idea was to just store one document per source name and update it's timestamp, but that wouldn't give me any logging
[08:14:30] <stefancrs> so my next idea was to just insert new document for each heartbeat received (name + timestamp per document, that's all I need to be able to retrieve)
[08:15:23] <stefancrs> at the same time, I want to keep it "name" agnostic. so how would I with a simple query retrieve the latest heartbeats from every source?
[08:16:17] <stefancrs> (if there are sources "s1", "s2" and "s3" for example. the send heartbeats whenever they "feel" like it, but I want to get all the latest ones...)
[08:18:51] <stefancrs> dstorrs: and they'll all tell the main backend every now and then that "hey, I'm alive"
[08:20:29] <dstorrs> well, first off, this might be useful : http://api.mongodb.org/wiki/current/Optimizing%20Object%20IDs.html#OptimizingObjectIDs-Extractinsertiontimesfromidratherthanhavingaseparatetimestampfield.
[08:21:18] <dstorrs> second, you could use a capped collection. that guarantees that insertion order is preserved, so you can trivially retrieve most recent using findOne()
[08:22:16] <dstorrs> so, but the two together and you've got a capped collection of docs like this: { _id : ObjectID(...), name : 's1' }
[08:27:00] <stefancrs> kollapse: type of what kind of object? like if it's a string, int, array or object?
[08:27:03] <kollapse> For example something like db.customers.find( { name : 'some_name' } ).type() ??
[08:29:22] <dstorrs> use the appropriate driver command to say "run map-reduce / run function / etc"
[08:30:31] <kollapse> Basically I have a collection of objects that have a field called 'field1'. 'field1' can be of different types. I want to retrieve the most predominant type.
[08:31:08] <kollapse> For example if 10 objects have `field1` as strings and only 5 as integer, I want to know that string is the most predominant type.
[08:32:02] <stefancrs> if it's never a lot of data, and you want to be able to retrieve it whenever, I'd go for the latter
[08:34:09] <stefancrs> I'd think so yes, especially if you want to be able to deal with a lot of data
[08:34:55] <stefancrs> dstorrs: I think going with two collections make the most sense for my problem :)
[08:36:19] <stefancrs> dstorrs: the "common" scenario is that I want to retrieve the "latest timestamps document"
[08:39:07] <stefancrs> dstorrs: this is for a server that literally will be used for about five days
[08:42:40] <stefancrs> dstorrs: good to be able to see if there were any issues while we were asleep, for instance
[08:44:13] <stefancrs> the client will be calmer when I tell them "and we log the up time of all the nodes in the system, as long as the main node is alive. if it isn't, we'll know that too."
[08:48:16] <dstorrs> hot chocolate made by melting chocolate bricks and adding cream, chocolate pizza, chocolate fondue, etc
[08:49:08] <dstorrs> regular pizza dough with chocolate, then marshmallow or etc on top. sounds weird. utterly amazing
[12:33:04] <natim> And I have a question for you : http://stackoverflow.com/questions/10960926/distinct-group-by-using-mongodb-with-pymongo
[12:33:41] <natim> I would like to do a similar request on mongodb to this mysql query SELECT DISTINCT pin, value FROM mesh_captors WHERE arduino = 203 GROUP_BY pin ORDER BY date DESC
[12:34:42] <natim> For now I have this : db.mesh_captors.group(key=['pin', 'value'], condition={'arduino': int(arduino_id)})
[12:40:18] <natim> What about something like this : https://www.friendpaste.com/2r2D79DYoaV3CuSOUmGdLr
[12:51:36] <omid8bimo> hey i have a question. i have a replicaset, im trying to dump my db in secondary server but i get this error: assertion: 13106 nextSafe(): { $err: "not master and slaveok=false", code: 13435 }
[13:07:02] <trax> I inserted "corrupted" bson in my mongodb base and now I am getting this http://pastebin.com/k6HJiAkK
[13:16:30] <omid8bimo> hey i have a question. i have a replicaset, im trying to dump my db in secondary server but i get this error: assertion: 13106 nextSafe(): { $err: "not master and slaveok=false", code: 13435 }
[13:54:35] <omid8bimo> natim: im trying to dump a database off my secondary server in a replicaset and i get this error
[13:57:51] <trax> natim: ok, so if someone is on the same network as my db, it can just crash it \o/ . Yes there is authentication (on a none secure stream):
[14:23:33] <Goopyo> Q: If you have hundreds of users making trades and each trade has a P/L and you want to query by three things: user, the item traded, and the period of the trade how would you structure the data best?
[15:00:14] <ub|k> i found a query expression which behaves differently depending on the order of the elements inside a JS object
[15:18:04] <pranavk> installing mongodb and its is shooting so many selinux warnings man, what is this ?
[15:19:51] <dstorrs> pranavk: do you have a question we can help with, or do you just need to vent? if the former, I can try to help.
[15:20:55] <pranavk> dstorrs: yes i want to know if it is the usual problem with mongodb or its just my system that it weird ?
[15:23:58] <dstorrs> pranavk: I haven't used selinux myself, so I don't know. a quick Google turns up only a few number of hits for this search: +mongo +selinux
[17:27:31] <Electron> at Jun 9 13:24:48 Error: couldn't connect to server 127.0.0.1 shell/mongo.js:84
[17:28:45] <Electron> Sat Jun 9 13:26:07 [initandlisten] MongoDB starting : pid=1015 port=27017 dbpath=/data/db/ 64-bit host=Zakerias-MacBook-Pro-2.local
[17:28:45] <Electron> Sat Jun 9 13:26:07 [initandlisten] build info: Darwin gamma.local 11.3.0 Darwin Kernel Version 11.3.0: Thu Jan 12 18:48:32 PST 2012; root:xnu-1699.24.23~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_49
[17:28:45] <Electron> Sat Jun 9 13:26:07 [initandlisten] exception in initAndListen: 10296 dbpath (/data/db/) does not exist, terminating
[17:28:46] <Electron> Sat Jun 9 13:26:07 [initandlisten] shutdown: going to close listening sockets...
[17:30:59] <dstorrs> Electron: look at the output you pasted. What is the big glaring problem right in the middle of it?
[17:32:25] <dstorrs> look at the spew you dumped in channel. smack in the middle of it is your problem.
[17:35:22] <multi_io> can I execute a sequence of JS statements (reads and writes) with mongodb's normal rw lock enabled?
[17:36:41] <dstorrs> Worry about changing config options for hygenie and platform-appropriateness is relevant for a production system, not testing
[17:37:45] <dstorrs> multi_io: I don't understand the question. you basically want to have only one process that's able to execute writes?
[17:38:19] <dstorrs> Chillance: there is always an _id, but it can be things other than an ObjectId. e.g., an object
[17:39:38] <Chillance> dstorrs, hmm, well, the thing is, that the key Im will be using is larger than that (I think), so if _id is always there, it may not be unique eventually
[17:40:55] <Chillance> dstorrs, say I want to use a 512 bit hash as the key, isnt that larger than the _id?
[17:43:17] <Chillance> I liked a different name, to describe what it is, but I guess this can work too
[17:44:02] <Chillance> what will happen if I don't use _id, but my name and create an index for it... _id will be kinda redundant?
[17:44:54] <Chillance> redundant, but always there? also, what happens then if there are two with the same _id? doesnt matter I suppose
[17:46:10] <multi_io> dstorrs: well... what I really want is versioned documents. I'll think about it some more.
[17:46:16] <Chillance> Im also interested in the more indepth info on how the indexing works for strings... use first characters for the Btree?
[17:47:13] <Chillance> dstorrs, yea, that may not be enough in the end, but thanks for the info! I guess I will set the hash to the _id instead
[17:50:26] <Chillance> dstorrs, I though I could use some other name as a unique key, and dont use _id, but now I get it as the one to use as the unique key, where I set the key. cool!
[17:51:18] <dstorrs> Chillance: you /can/ have other keys be unique. they will enforce uniqueness on those constraints...but you can't get away from the _id
[17:52:14] <at133> Hi, I have a document with two counters. I would like to increment them independently. when I use {$inc {x:1, y:0}} and {$inc {x:0,y:1}} it creates new documents. Is there a way to increment them independently in the same document?
[17:54:37] <dstorrs> Electron: when you write to Mongo, it stages the writes in RAM for speed. periodically it flushes them to the actual data files
[17:55:37] <dstorrs> this is bad, but you can't just say "write everything immediatelY" -- that kills performance, as inserting into a BTree can be slow
[17:56:17] <dstorrs> so, changes get appended to the journal files almost immediately about being issued. then, when the next write trans happens, they are written to the real data files.
[17:56:47] <dstorrs> since this is your first startup, you have no journal files yet. for some reason, it thinks that things are not valid so it's trying to do a recovery
[17:57:04] <dstorrs> that means "hit the journals and replay them into data files to catch up"...but there are no journals
[17:57:39] <dstorrs> I don't know what your specific issue cause is; I'm doing some googling. but that's the general problem
[17:58:27] <dstorrs> rm -rf /data/db, then follow these dirs: http://shiftcommathree.com/articles/how-to-install-mongodb-on-os-x
[17:59:02] <dstorrs> btw, be certain you are ALWAYS ALWAYS ALWAYS running the 64-bit version of Mongo.
[18:03:38] <dstorrs> at133: what exactly are you trying to do? show me a start doc and an end doc that you want to achieve
[18:07:57] <dstorrs> at133: ok...that looks like you just updated one field. Is that what you mean by "updating fields indepedently"? if so, look to the '$inc' operator on update()
[18:09:21] <at133> dstorrs: Yes, but when I do that with {$inc {view_count:0, click_count:1}} it creates a new document for every increment. So incrementing one works fine, but if I try to increment the other I get two documents.
[18:14:28] <at133> dstorrs: I deleted my collection and restarted mongo. Now it works. Thanks for the help.
[18:16:58] <at133> collection.update({page: 0 }, {"$inc": {view_count:1, click_count:0}},{upsert:true})
[18:17:48] <at133> collection.update({page: 0 }, {"$inc": {view_count:0, click_count:1}},{upsert:true})
[18:18:02] <dstorrs> good to know. I haven't had occasion to use 'inc' and the docs are totally clear
[20:44:30] <spaam> the repo on http://downloads-distro.mongodb.org/repo/debian-sysvinit is kinda broken.
[20:44:50] <spaam> W: Failed to fetch gzip:/var/lib/apt/lists/partial/downloads-distro.mongodb.org_repo_debian-sysvinit_dists_dist_10gen_binary-amd64_Packages: Hash Sum mismatch