Log file Viewer
#mongodb logs for Monday the 27th of August, 2012
[00:09:31] <ryanair> for example, all of the documents that I see are formatted like this {"_id": 1, "values":{...}}
[00:10:40] <ryanair> why can't I have {"_id":1, "value1": ..., "value2": ..., "value3": ..., ...} in the output from a mapreduce job?
[00:21:58] <Aleross> not really sure the validity of this because I haven't used Mongo's map reduce myself, but have you looked into the finalize function?
[00:22:32] <Aleross> but finalize is called on the last reduce's output before it's saved to a temporary collection
[00:40:26] <Aleross> jtomasrl: not sure if you can do it on the MongoDB level… doubt it, kind of goes against the whole idea of it, but you could (and should) do it at an application-level if necessary
[00:46:36] <Aleross> I think that's a pretty common requirement though, somebody else may want to chime in about how they do it
[00:59:04] <ryanair> Aleross, I'm talking about *after* reduce's output is saved. I'm seeing documents formatted like this {"_id": 1, "values":{...}} when what I want is this: {"_id":1, "value1": ..., "value2": ..., "value3": ..., ...}. It doesn't seem like finalize would help me do this.
[01:07:15] <Aleross> ryanair: my understanding is that you can do basically anything with finalize, so what I would do is *before* saving the temporary collection just run finalize and break value1, value2, and value3 out of "values", creating a new collection that will ultimately be saved in the format you want
[01:07:39] <Aleross> ryanair: there's no point in saving it as _id:, values: if that's not what you want… might as well manipulate it before saving
[02:34:57] <emocakes> ok, so im dropping one step back in my yearning for new things, still gonna use mongo, but coupling it with something I know, node
[08:14:37] <firejocker> i'm starting using django-mongo, but following the run of a User.objects.get(username=username), i saw that all database values are in QuerySet of query.py
[08:16:03] <firejocker> am i right to think there is a problem there, i mean, to see all the users in memory is bad for performancties
[08:26:31] <firejocker> IAD nop, i mean, to select one user, i can see all in self.QuerySelf of django/model/sql/query
[08:37:17] <mdupont> hi there, i have some app using the mongohq server, is there an api compatible server i can install? the rest api?
[09:25:44] <topriddy> trying to find previous version of mongo on my cent os server, delete it and do a fresh reinstall from some other repos=> 10gen
[09:57:47] <zeus_> hey, i've build cluster with 9 machines. At first i got 12000 ins/sec now its down to 400 ins/sec - what happend ? Whole indexes fit into RAM, its sharded properly.
[10:16:58] <Bartzy> Besides cursor timeout - there are no other timeouts for a connection in itself ? For example, I have a long running PHP worker. It creates a MongoCollection object when it starts, and when it gets a job - it calls update() on that MongoCollection object.
[10:17:09] <Bartzy> Can that MongoCollection object time out ? meaning the connection itself will timeout ? And if it can - the drivers handle it internally, or I need to do something ?
[10:43:41] <Bartzy> NodeX: Instead of each worker disconnecting and connecting on every job, I want to save the connection
[10:46:48] <Bartzy> NodeX: So on each job the workers get - they should do a $mongo = new Mongo... ?
[10:47:15] <emocakes> which brand of computer is best for mongodb? I like HP but people have told me that Gateway works better
[10:47:46] <NodeX> https://groups.google.com/forum/?fromgroups=#!topic/mongodb-announce/bcehXiVfQpE <----
[10:48:40] <Bartzy> NodeX: What does that mean? How can there be only 1 connect per that combination ? What about multiple processes ?
[10:50:21] <Bartzy> NodeX: But for the current one (not the new 1.3 version) - If I do $mongo = new Mongo(..) on each job the workers get - that would not create a connection for each job ?
[10:55:30] <NodeX> http://us2.php.net/manual/en/mongo.connecting.php <--- read down on connection pooling
[10:56:06] <NodeX> "This section is not relevant for 1.2.0+. In 1.2.0+, connections are always persistent and managed automatically by the driver. See MongoPool for more information on pooling. "
[11:03:34] <kali> emocakes: i actually use scala and unfiltered as a http framework, but just for webservice-like endpoints
[11:04:13] <kali> emocakes: i use it to bypass a rails stack that is too slow and resource greedy for the most critical endpoints of our app
[11:09:22] <emocakes> well the company im currently working for is rather stock standard (read boring stack)
[11:10:22] <kali> emocakes: i think it's not a good idea to use node.js as your main framework, but rather as a way to handle endpoints with special features, like comet like features, or file upload, etc
[11:13:26] <kali> same way we use scala and unfiltered here, actualy. if rails + cache is enough, then fine. if the endpoint is not cacheable and performance critical -> scala.
[11:14:36] <NodeX> there is nothing wrong with normal php, just avoid bloated frameworks - they are what drag the performance to a crawl
[11:17:24] <kali> emocakes: but it's not a showstopper as far as I'm concerned. just something to think about
[11:23:49] <NodeX> not sure about the benchmarks... I know for a fact lighty is not faster than nginx
[11:29:49] <emocakes> yah I got that feeling as well algernon after a night of reading the forums etc
[11:30:54] <algernon> emocakes: it's not open source. that wouldn't be such a big problem if the thing actually delivered, and its author wasn't a nut
[11:37:41] <NodeX> saying that single server performance is better than horizintal scaling .. yeh dude what happens when your single server dies IDIOT!
[11:40:39] <NodeX> give me 90 euro 16gb RAM + 2tb servers (plus replica sets) any day over a single server wiht no fault tollerance
[11:55:38] <zeus_> how to debug why writes to mongo cluster are so slow ? is there a better tool than mongostat ?
[13:43:14] <solars> hey, quick question - how do user restrictions work? I've added an admin user and user on a different db, but can still access the db without password from a rails app - how exactly does this work?
[14:02:13] <jiffe98> can I start with a single shard consisting of one replica set and add more shards later?
[14:11:38] <bjori> solars: if you authenticate against the admin db you can browse all other dbs. otherwise you have to authenticate against the individual dbs
[14:31:48] <SpanishMagician> create a ref for each client document and insert it in users or the other way around?
[15:08:52] <SpanishMagician> I have 2 collections in my database, users and clients, how do I create references?
[15:08:59] <SpanishMagician> create a ref for each client document and insert it in users or the other way around?
[15:09:31] <Derick> It's impossible to answer this, as it all depends on how you want to do queries, and how updates work in your app
[15:10:28] <SpanishMagician> however I want a security mechanism so that a user cannot modify someone else's clients
[15:11:13] <Derick> which, in this case, probably means storing the user ID in the clients' collection's document
[15:14:17] <SpanishMagician> so an update would be sth like db.clients.update({{'_id': id},{'user_id': userid}}, blablabla)
[15:14:51] <Derick> that will replace the document with "id" with just { _id: id, 'user_id' : userid }
[15:17:30] <SpanishMagician> i want to replace the document with _id id and user_id userid with whatever
[15:18:48] <SpanishMagician> update the email for example would be update({{_id: id}, {user_id: userid}}, {$set:{email: 'blabla@blabla.com'}})
[15:23:41] <geoffeg> hmm, creating a capped collection blocks all other operations while the collection is being created?
[15:51:12] <jgornick> Hey guys, I created a collection that contains documents with one field that mapped to another document using a DBRef. If the driver support dbrefs, I should be able to make queries on that collection that allow me to sort or filter the resultset? Something like db.tempCollection.find({ ref.someField: 'test' }).sort({ ref.anotherField: -1 });
[15:52:53] <NodeX> you may embedd / nest documents but you cannot reference or join anything inside one query - it MUST be 2 or more queries
[15:56:15] <jgornick> This is a temporary collection where I've aggregated data using map/reduce. Instead of populating the temporary collection with cloned documents, I wanted to populate it with dbrefs to those documents.
[15:56:59] <NodeX> as Derick said it does not auto follow them so it's prolly not useful for you in your case
[15:58:48] <jgornick> Derick: Good talks at lone star php this year :) I never did get the chance to meet you. I enjoyed the openstreetmap talk!
[16:13:17] <Bartzy> I have a long running worker that is using mongoDB to update documents as it gets job
[16:16:05] <Bartzy> On a CLI php - in the process memory somewhere - and with PHP-FPM - it is shared with all fpm processes ?
[16:18:12] <Bartzy> Ah - so that was my next question - that (the link to the Mongo object) is that reason that the connection doesn't drop?
[16:18:24] <Bartzy> Because otherwise, if I don't "save" the Mongo object, it will go out of scope ?
[16:18:48] <Derick> even if it goes out of scope, the connection doesn't drop as we reference it in the extension itself too
[16:21:28] <Bartzy> That is correct I guess? And so it doesn't return to the pool because the MongoCollection object references the Mongo object ?
[16:21:59] <Derick> if it goes out of scope, it goes back to the pool - the pool of established connections
[16:22:21] <Bartzy> So there is absolutely no way of a Mongo connection timing out , like a MySQL connection for example ?
[16:22:48] <Derick> if you sever the connection from the server (or network side) the connection will still break
[16:22:51] <Bartzy> For example, my worker doesn't get jobs for a day. What happens if it gets a job suddenly? The Mongo object is live and well?
[16:28:13] <Derick> but still, the server *will* break the connection if it steps down from primary to secondary f.e.
[17:32:17] <ramsey> I'm a little worried about the "couldn't send query" messages I get intermittently with the PHP driver (1.2.12). They seem to happen a lot on my development VM, but they don't happen at all for another developer. Should I be concerned about the stability of the driver?
[17:43:03] <ramsey> TheSimkin: the forums are probably the best place to get answers, but I like IRC. :-) https://groups.google.com/forum/?fromgroups#!forum/mongodb-user
[17:43:17] <TheSimkin> i got couchdb too, but couchdb is producing a piles of errors even with their own test cases on a base install
[19:17:51] <jgornick> hey guys, are there any statistics related to performance with large collections?
[19:20:57] <jgornick> I guess I'm looking for some sort of a guideline that says, "You should try avoiding n number of documents in a collection as this is where performance degrades."
[19:31:53] <BurtyB> if I do a db.collection.count() on a sharded db/collection should it add the counts on all shards shouldn't it?
[19:59:47] <BurtyB> Ahlee, I've stopped the import now all mongos say 163480979 mongod direct gives 16464257 / 57956572 / 89060150 now
[20:04:03] <BurtyB> Ahlee, (I don't have replica sets atm) I was failing at noticing the length of the numbers so what's wrong is my eyesite :/ I figured it was me doing something wrong somewhere
[20:16:54] <addisonj> so, I am moving our production mongo to AWS and am using RAID on EBS, currently, we just use mongodump for backups, is the biggest benefit to using EBS snapshots just speed and offloading the work?
[20:30:42] <addisonj> right, I just already have code for using mongodump, so was wondering if it was worth the effort
[22:30:35] <Yiq> So I thought nosql and sql databases were basically for different use but a lot of people are saying nosql(couch, mongo) is the way to go for webapps, dynamic applications. Why is it better(or is it? obv there must be stuff where each is better)?
[23:51:41] <bluethundr> hello.. I've installed mongodb on centos 5.8 .. I've added the line extension=mongo.so into my php.ini file but for some reason a phpinfo(); page does not show that it is enabled
[23:54:30] <_johnny> if php does not list mongo, then make sure you edit the correct php.ini, or that your php process/daemon uses the ini you expect