Log file Viewer
#mongodb logs for Thursday the 6th of September, 2012
[00:07:21] <_m> EvanCarroll: One you're logged-in, there should be a "create issue" button next to the search bar. Or use the keyboard shortcut 'c'
[00:43:56] <bizzle> In salat, how does one write a query that only returns a subset of the fields in a document
[00:43:59] <goleldar> is it common practice when refrencing another collection by including the collectionId to also include into the document any information you would like available from that referenced collection to avoid making another query
[00:45:03] <bizzle> like a UserBasic(email: String, username: String), UserDetailed(email.. bio... interests...)
[00:48:46] <goleldar> i started to design my app and ran into a wall with modeling data that would cause me to use EAV in mysql
[01:03:17] <retran> goleldar, no, it's a development system i'm 'deploying' but we hope to be live with it under 1month
[01:04:30] <retran> it's a system for creating 'profiles' for actors showing demo reels, press photos, etc
[01:05:24] <goleldar> can you take a look at my collection schema and tell me if it looks good? i am just getting started
[01:08:35] <retran> i can understand why you'd want to return JSON document that in say, a web service request
[01:19:25] <fractalid> hey - I have a hanging connection problem w/mongodb 2.2.0. on the server I've got about 3000 connections, 99% of which show as "established" in netstat
[01:20:19] <fractalid> on top of that, according to mongodb.log, it's refusing connections because it's hit its max of 6553
[01:22:33] <fractalid> I'll be afk for a bit, sorry - time to go home, I just wanted to get this out there. should be back in half an hour or so
[02:57:35] <crudson> Bilge: what other than storage size were you hoping for? Determining the size of an object depends on what language you are using.
[03:38:23] <Init--WithStyle-> Does anyone know how running 6000 consecutive DB inserts into a mongoDB via a javascript driver could be causing the process to consume massive amounts of RAM??
[03:49:51] <Bilge> And I dunno, I just thought there would be some kind of stats thing, but I really just want to know how big documents are
[03:55:58] <Gavilan2> Is there any good "something" on throw and recover arbitrary objects to mongo from javascript (i want to associate the objects with their class once they get back)?
[04:09:29] <crudson> Bilge: you can get a fair amount of stats for server and dbs, but not really at the document level, other than average (b.stats()) and individual size (Object.bsonsize(o))
[04:11:43] <Bilge> In particular I really want to know if storing something as binary is smaller than storing it as a hex string (it should be)
[04:13:15] <crudson> Bilge: as I said, use Object.bsonsize(o) from the shell. Otherwise google "mongodb bson size <some language>" for other languages, as there are different methods.
[04:33:04] <Ventrius> Is it at all efficient to have a clone of a db sitting on your local machine, and have it copy data from some far off machine
[04:33:34] <Ventrius> I ask because I need to minimize latency between queries of a single entry collection
[06:37:22] <null_route> Hey all - Where can I get "internals" information on how the Config servers work?
[06:48:39] <null_route> For example, if I understand correctly, the Mongos processes keeps the conf servers updated
[06:48:55] <null_route> where does that happen in the code, or is there a WIKI page which describes teh process?
[07:25:39] <null_route> When performing a moveChunk, it seems as though " the destination shard connects to the config database and updates the chunk location in the cluster metadata" . How does the destination shard know where the config server is?
[07:59:58] <cmex> Nodex: tell me is it any visual tool to see what happening between replica set servers?
[08:12:25] <Lujeni> cmex, http://blog.mongodb.org/post/28053108398/edda-a-log-visualizer-for-mongodb
[08:15:53] <cmex> Gargooyle: good morning :) we are using here a database master 5 . theres option to add replica set
[08:18:34] <dstorrs> Hey folks. Anyone seen this before: Thu Sep 6 08:17:20 uncaught exception: error { "$err" : "socket exception", "code" : 11002 } ?
[08:20:26] <dstorrs> I'm Googling around and seeing something about "entire shard down" / "replica set primary unavailable" / "auth information unavailable" but none of those seem to apply.
[08:30:23] <dstorrs> yes. This shard was originally added last week and was not having data sent across. we added / removed it a couple of times before we got it working today
[08:31:35] <null_route> My guess: It's related to any annoying property of MongoDB - REMOVING a member causes the PRIMARY to step down and hold elections. When it does this, it closes all TCP connections. This causes the mongos', obviously, to lose connectivity.
[10:12:26] <Azoth> 32-bit builds are limited to around 2GB of data. See here for more information on the 32-bit limitation. from the mongo dowload page
[10:12:32] <algernon> but the reason is probably similar, with the addition that a higher cap would increasingly waste more space.
[10:51:05] <golddog> I can't seem to find any examples for how to best merge duplicate documents in a mongodb collection. I thought i sounded like a job for map reduce but i can't find any examples of map reduce that are not basically just collating statistics
[10:54:29] <golddog> if anyone could point me to any examples of how to do this I'd be eternally grateful
[11:12:03] <kali> golddog: it's not a easy problem, but if you can come out with a signature of your document (some id, or some generated id) you can group your collection counting the occurence of this signature, then look at the counters > 1
[11:19:06] <golddog> kali: every document has an id and I only want to merge the ones with equal id's so identifying the documents are no problem. Basically I have the output of an sql join in a collection and i want to convert from 5 documents with mostly identical properties except for one property to one document with that property as an array as it should be
[11:20:53] <golddog> but I think I'll just remove everything and fix it before i insert it instead for now, just seemed like something one might like to do rather often
[11:23:42] <golddog> kali: the problem is basically this: I have {id: 1, name: 'foo', tag: 'dog'}, {id: 1, name: 'foo', tag: 'cat'} (in a large scale) and I'd like to merge every document with an identical id into {id: 1, name: 'foo', tag: ['dog', 'cat'] }
[11:24:18] <golddog> I'm sure it's either impossible or incredibly easy but I just can't find any examples
[11:25:36] <kali> you could also imagine to do it in mongodb-side javascript, but mongo side javascript is a Bad Idea for production
[11:26:05] <golddog> wasn't intending to do it in production, just trying to do a one-time fix of a broken data-set
[11:28:05] <kali> Goopyo: as a matter of fact, I'm realizing the merging or rewritting you describe falls just into the scope of the aggregation framework
[11:38:15] <Vile> I have documents like {_id: {a:1,t:"2012-01-01T00:00:01Z"}, value:123}, {_id: {a:1,t:"2012-01-01T00:00:51Z"}, value:100}, {_id: {a:1,t:"2012-01-01T00:01:41Z"}, value:0}. I want to build minute averages on it using the fact that value is considered constant until changed
[11:40:57] <Vile> Currently i'm using map/reduce for that, but it is very slow, due to db.coll.findFirst( { t: {$lt:"2012-01-01T00:01:00Z"} )
[12:43:31] <cmex> someone knows this error? "Unable to connect to a member of the replica set matching the read preference Primary"
[12:52:14] <unsleep> it sound weird to need to configure a firewall to avoid it ... why i cannot disconnect it?
[12:54:29] <algernon> unsleep: you can set nohttpinterface = true in /etc/mongodb.conf to disable it completely.
[12:59:33] <unsleep> it would be better to ask for a password. i think it will improve a lot in tehe future
[13:37:58] <algernon> ravana: kilobytes, by default (http://docs.mongodb.org/manual/reference/database-statistics/)
[13:40:06] <thecodecowboy> Hi. I am considering using MongoDB for this data requirement: http://programmers.stackexchange.com/questions/161602/would-this-data-requirement-suit-a-document-oriented-database. Anyone have any advice on how to structure the data? Would it make sense to allow users to create collections ??
[13:46:00] <thecodecowboy> ppetermann, baby steps in nosql, interested to learn and wondering if my requirement is a good fit. any thoughts?
[13:46:15] <ppetermann> thecodecowboy: http://www.mongodb.org/display/DOCS/Use+Cases you might want to read this one
[13:49:34] <thecodecowboy> ppetermann, yeah, i am looking for the advice of someone with experience of mongodb rather than documentation which will not mention my use case
[14:02:03] <Vile> NodeX: Read teh aggregation docs. Does not looks like something thats going to help me :(
[14:02:25] <algernon> ravana: the string is 3 bytes + null + 4 byte length; then there's the field name, 5+1 bytes + length + type; then there's the _id, 3+1 bytes for the name, 1 for the type, 12 for the data, and wrap it in a document (4+1+1 bytes)
[14:05:09] <null_route> Take a look at http://www.slideshare.net/mdirolf/inside-mongodb-the-internals-of-an-opensource-database
[14:06:37] <algernon> ravana: I suppose the bsonsize there is not the size of the document, but the db object instead
[14:06:49] <algernon> at least http://www.mongodb.org/display/DOCS/dbshell+Reference seems to suggest that.
[14:10:24] <ravana> "pasting a document into Object.bsonsize() returns the correct result" i didn't get this
[14:36:21] <joshua> what exactly goes on when you initiate a replica set? The local database is 6GB now
[14:38:56] <joshua> Is it possible for one server to be arbiter for 2 sets, or will I have to run on two ports? (Not my design)
[14:39:51] <joshua> Thanks. Thats what I thought. I guess they wanted to use more shards so its shared
[14:46:31] <joshua> Vile I guess functially it doesn't hurt. I might create a second init script with conf so they can be started with a service instead of hacking something into rc.local.
[14:50:52] <timeturne> what is the best way to check the size a particular object will have when added to mongo as a document?
[14:53:22] <timeturne> basically I want to embed the maximum latest blog posts in a "section" document and then push the older ones to another collection
[14:53:45] <algernon> timeturne: http://mongodb.github.com/node-mongodb-native/api-bson-generated/bson.html#bson-calculateobjectsize
[15:17:48] <timeturne> would the index be as efficientif I added objectids from subdocuments in it? also, how would I goabout actually adding the objectid to the default _id index?
[15:27:29] <timeturne> I think I'll implement a character limit on the blog post and just add a fixed number of blog posts rather than checking the size for now
[15:37:39] <fg3> data = { name: 'foo', items: [{ name: 'joe' }] // how can I add a key/value to the hash under items?
[15:38:28] <goleldar> i am new to mongodb can someone look at my document schema and tell me if it looks good
[17:52:18] <FreeFries> hello...I'm running into a problem with mongo shards+replica sets and I'm wondering if there's a way to turn in more debugging information...
[17:52:43] <FreeFries> I have two shards which I want to replicate to three servers, replica set repl_mongod02 works fine
[17:53:09] <FreeFries> replica set repl_mongod01 doesn't work at all failing with a "couldn't connect to new shard socket exception [CONNECT_ERROR] for repl_mongod01/..." error
[17:53:22] <R-66Y> is there a $size for associative arrays that just returns how many keys are in the array?
[17:53:32] <FreeFries> but when I change the "repl_mongod01" to "repl_mongod02" it says that it's not part of the set
[17:54:06] <FreeFries> when I run mongo to the host and do a rs.status() I get "set" : "repl_mongod01",
[17:54:53] <FreeFries> any way to get more information out of mongos, or anyone happen to know how to fix the problem?
[18:02:36] <elux> im finding very simple queries are taking 120ms+ ... they really shouldn't.. and they should be using indexes
[18:23:36] <peter_c> what was lacking in the industry with all the relational databases around, that spawned mongodb
[18:23:44] <kali> peter_c: not a subject for irc, i think... you can find dozen of well argumented blog posts for one side or the argument or the other
[18:24:56] <Gargoyle> peter_c: As kali said, there's plenty of resource on the web dig through if you want to bring yourself upto speed.
[18:25:17] <peter_c> im just a project manager and one developer suggested to switch to mongodb for our product
[18:26:14] <Gargoyle> peter_c: Seems the first task is to fully research if a document based db fits your product requirements.
[18:28:00] <peter_c> so whats the most cookie cutter example that would draw the scenario where mongodb (document based dbs) would be hands down the better choice versus a rows/tabular
[18:29:20] <FreeFries> (by that I mean, how do I clear out all replica config information and start over?)
[18:30:18] <Gargoyle> FreeFries: That normally works. shutdown server, delete (or move) data dir, and start fresh.
[18:30:52] <Gargoyle> But if you have your replica set info in the config / startup param, it will reconnect and resync itself automatically
[18:34:01] <Gargoyle> FreeFries: Sounds like you are bringing the server back up still telling it to be part of a rs. Do you have it in the config file / startup options?
[18:34:53] <FreeFries> yeah, I want it to still be part of repl_mongod01, but I want to kill the entire thing and start over as if repl_mongod01 never existed
[18:35:46] <Gargoyle> FreeFries: You will have to take all the servers in the set down together - then I think, initially boot them up without any reference to the RS
[18:40:09] <Gargoyle> FreeFries: It lives in a database - if you have removed the data dir, it should be gone.
[18:41:41] <FreeFries> server 3 couldn't connect to server 1, server 2 saw itself only, server 1 could connect to all three
[18:42:28] <bizzle> Is there a way to drop all the documents in a collection without dropping the indexes?
[18:42:30] <FreeFries> so I figured the easiest approach would be to destroy the rs configuration and start over
[18:45:39] <FreeFries> when I was initially setting it up I issued a rs.initiate() and rs.config() on servers 1 and 2
[18:50:08] <muffs> is there a best practice for modifying / removing indexes with minimal effect to prod?
[18:53:20] <FreeFries> so...I dunno, I can't start a new one, but I apparently have no rs configuration
[19:41:55] <timkuijsten> someone any experience with using Timestamp objects and mongo's node driver?
[19:42:34] <timkuijsten> trying to search the oplog with it.. but somehow it looks like my timestamp objects or not used correctly in my finds..
[20:14:13] <FreeFries> I've cleared out the data directory in all of my mongods, along with all of my mongo configs
[20:15:28] <FreeFries> or would anyone know of some linux tool that would let me find where it's getting the info from? (similar to filemon on windows?)
[20:30:30] <FreeFries> I also can't manually setup the rs config and run rs.reconfig(cfg, force : true)
[20:39:20] <FreeFries> maybe I can approach this from another side...how can I see what data is in db.local.system?
[20:50:57] <Gavilan2> So! Is there any "transparent" library or thing to ORM into MongoDB from JavaScript. I'm looking for something that is, taking any app that runs in memory, replace Array or a Collection with CustomCollectionThatGoesIntoMongoDb... And then the app continues working with persistance... Is there anything like that? or do I need to make it?
[20:54:27] <gibletsngravy> Hi, I'm learning how to use the Mongo's profiling feature. When I query the system.profiles collection, why isn't the inserted data shown for "insert" operations (as updateobj is shown for update ops)?
[21:41:19] <Neptu> I just wnat to confirm if i pass a list of servers he will detect and connect to the master automaticaly
[21:41:43] <Neptu> and if I pass only one and its active should find the master aswel even if its not in the list
[21:43:26] <Neptu> The nodes passed to Connection() are called the seeds. As long as at least one of the seeds is online, the driver will be able to “discover” all of the nodes in the set and make a connection to the current primary
[21:46:10] <FreeFries> which seems to have worked to remove whatever it was that mongo had left around remembering the repl set
[21:53:47] <jefferai> Hello...I'm trying to figure out how to put an TTL index on a value in an array
[21:54:17] <jefferai> I have records like { "registrations": [ { ......, "lastupdated": (date)}, {......}] }
[21:54:47] <jefferai> is that possible, or do I need to split each item up into a separate document?
[21:56:24] <fg3> data = { person: { name: 'joe', cars: [{name: 'ford'}] } // how to add property to car?
[23:24:07] <zanefactory> so much docs about how to turn it on, can't find anything about how to turn it off