Log file Viewer
#mongodb logs for Tuesday the 11th of August, 2015
[00:16:49] <grandy> hello, i'm using mongoose and one of my models has this schema def: {type: mongoose.Schema.Types.ObjectId, ref: "User" },
[00:17:09] <grandy> wondering if there is a way to populate based on that, or if it would have been necessary to define the ref in the schema
[01:14:23] <ThePrimeMedian> i have a large dataset, and I need to regex search on the name field. However, I need to sort on the status, and then name field. If I do not sort, the query is really quick. But if I put the sort in, it take > 1m -- any help would be appreciated.
[01:30:27] <joshua> did you run the query with .explain() to get some more info as to whats going on?
[01:32:38] <joshua> It has been a while since I did that kind of thing so might not be able to explain (har har) what all the results mean, but I think its a good idea in general to see what is going on.
[01:35:52] <ThePrimeMedian> the first cursor of the explain says it used the "sort" index -- to be honest, i dont know how to read it.
[01:51:32] <acidjazz> so say ['joannac','cheeser','acidjazz'] instread is joannac: {stuff}, cheeser: {stuff}, acidjazz: {stuff}
[06:18:33] <sigfoo> Excuse me if this is a little off-topic. I have a question about working with Mongo with the node.js native driver
[06:18:36] <sigfoo> 02:12 < sigfoo> Hello all. I'm working on a web app in node and I'm a little new to the paradigm. My app has a MongoDB backend and I'm working with the native Mongo node driver. Should I be instantiating a single connection to the DB and serving all requests from that connection? Or should I be calling MongoClient.connect once for every HTTP request?
[06:40:38] <Boomtime> note that if you use the nodejs clustering module, you will end up with lots of these anyway because the cluster module instantiates multiple copies of the runtime on your behalf
[06:42:11] <mskalick> Hello everybody, is there any option how to run rs.remove() on the server-site before the server is terminated? Thank you very much, Marek
[06:44:10] <mskalick> joannac: But could it be done automatically? (without connecting with mongo shell)
[06:45:12] <Boomtime> mskalick: do you mean you want to run a custom command whenever the server is shutdown?
[06:46:35] <Boomtime> it sounds like you need a stop script (not mongodb), but perhaps if you explain why you need this we'll understand and be able to consider alternatives
[06:49:09] <mskalick> jonnac Boomtime: I want to have working replication in Docker container - in this container is only one process (mongod). So have running for example 5 containers~mongod. And I would like to know if it is possible to remove one mongod server when the container is going to be killed. Without any other control script (which must run in new separate container)
[06:50:08] <mskalick> So it would be nice to be able to run rs.remove() when the mongod is going to be killed...
[06:56:46] <Boomtime> if a node needed to pre-announce it's own failure then that isn't a very redundant system
[06:57:46] <mskalick> Boomtime: why?... not the failture! It is the part of scaling ~ to be able to "automatically" add or remove shards...
[06:57:48] <Boomtime> if you have a correctly configured replica-set, you can happily lose a member through any cause, and the rest will plow on without that one
[07:00:22] <Boomtime> ok, you also used the word "scaling".. replica-sets are intended to provide high-availability, sharding is the preferred method to scale up
[07:00:45] <Boomtime> regardless, the need for any given member to pre-announce it's own demise is fundamentally the wrong way around
[07:02:26] <mskalick> Boomtime: "you can happily lose a member through any cause" -> And it is not problem when the loose is intended? (I know that the mongod will never come online again) ~ so after some time of running there could be a lot of never active members...
[07:04:06] <Boomtime> in any case, you clearly have some control system that is creating/destroying these members - why can't that process perform the maintenance?
[07:04:27] <mskalick> Boomtime: Adding is no the problem, bacause in the container you run mongod in the background, connect with mongo shell + add to the replica-set. Then mongod is stoped and there is "exec mongod" (to properly receive signals,...). So with execed mongod it not possible to run rs.remove()
[07:05:25] <Boomtime> : "bacause in the container you run mongod in the background" <- what is doing this?
[07:09:12] <Boomtime> mskalick: mongodb is a database, not a linux process control system - if you need to do tasks related to the starting and stopping of containers, you need a process control system
[07:11:59] <mskalick> Boomtime: I know. I only wanted to know if there could be some hooks to run on system terminating...
[07:18:17] <joannac> mskalick: and the something special is "connect to the current primary and remove the member that just got "shutdown""?
[07:20:12] <joannac> if you are so certain that you are shutting down a server and never bringing it back up, you can engineer your own way to call rs.remove()
[07:20:17] <mskalick> yes. Thought to be able to register function which will be executed on the mongod shutdown. And in this function connect to the master and remove the mongod
[07:20:36] <joannac> the server cannot tell the difference between "I'm shutting down for some maintenance" and "I'm shutting down and never coming back"
[09:42:42] <benjick> Hi. I'm trying to update an object nested in an array like this: https://gist.github.com/benjick/14ce6f8ea68a648ee649
[09:43:54] <benjick> I hate when you paste something, look at it again and find out the problem. So sorry
[11:09:07] <_MMM_> I have a collection of documents with a "data" field and I want a query result to only return the contents of the documents' data fields. Is that possible?
[11:09:48] <_MMM_> I tried the fields parameter of find() but that include the "data" fieldname itself for every document
[11:12:04] <dddh> _MMM: var c = db.table.find( { } ).sort( { data : 1 } ); c.forEach( function( doc ) { print( doc.data ) } );
[11:12:57] <bartzy> I have a collection of posts by users. The user id field is called uid. I want to get a list of all the post count per user, like this: Count of all users that posted 1 time or more, 2 times or more, 3 times or more.. etc …
[11:13:13] <bartzy> db.posts.aggregate([ {$match: { created_at : { $gte: ISODate("2015-07-11T00:00:00.000Z") } } }, {$group: { _id:"$uid", count: { $sum: 1 } } }, {$group: {_id:"$count", count: {$sum : 1} } }, {$sort: {_id: 1} }]);
[11:13:55] <bartzy> This gives me only exact counts - meaning the _id would be the number of posts, and the count field would be the number of users that posted this number of posts. But I don’t want exact equality, I want to get greater or equal.
[11:14:45] <_MMM_> dddh: well, yes but I was wondering if mongodb doesnt have some option to retrieve only the contents of subdocuments in the first place since that would probably be more elegant and better performing
[11:19:09] <_MMM_> ring3: but if you want to have an order on your documents you can use a key to order by and put an index on that key
[11:20:41] <ring3> i need to do catalog.find({customer:'acme'}, function(cus) { cus.contact.append ...
[11:26:03] <ring3> ah ok: findAndModify({query: {customer:'acme6'}, update: { $push: {contacts: 'c1'} }});
[11:33:35] <_MMM_> bartzy: wouldn't it be more efficient to do the >= grouping after retrieving the query since many documents would end up in a lot of groups of the posters usually have more than 1-2 posts only?
[11:36:55] <_MMM_> not sure how to do it exactly but I bet you need to make use of the pipelining here
[11:46:12] <aps> Hi all. Is it true that if my data and journal files are on the same volume, I don't need to fsyncLock( ) the secondary for take snapshot backup on Amazon EBS?
[11:51:46] <deathanchor> I think otherwise you might have to do a repairDatabase if it is in a bad state.
[11:57:44] <aps> deathanchor: thanks, do you do this manually or using scripts? I'm asking 'cause I'm not sure if I should rely on scripts
[11:58:45] <aps> deathanchor: one more thing, do you just db.fsyncLock( ) or do you stop the mongo service itself?
[12:23:08] <_MMM_> var cursor = OHLC.aggregate([{ $sort: { 'data.timestamp': 1 } }, { $project: { timestamp: '$data.timestamp', open: '$data.open', high: '$data.high', low: '$data.low', close: '$data.close', volume: '$data.volume', _id: 0 } }]);
[12:45:47] <_MMM_> StephenLynx: does the callback give the same result as cursor.toArray's callback?
[13:07:54] <deathanchor> how do I filter for a sub doc? { some : [ { want : "this", dont : "want" } ] } so that I only see the want field of the first array subdoc?
[13:10:36] <sterichards> I’m trying to add a field ‘code’ with the value of ‘NA’ where ‘code’ doesn’t exist. I’m using the following, but it just returns 3 dots (…) via CLI - db.object.category.update({"code" : { "$exists" : false }, {$set: {"code":"NA"}}, false, true)
[13:20:12] <jhertz> Hi, I am really new to MongoDB and working in a project were it is suggested that we use it as a (query) cache for an oracle database. Is this a good use of MongoDB, does someone have a link to an article about this approach?
[13:21:36] <StephenLynx> if you just want a cache on top of the main database, there are tools meant specifically to be used as a cache, such as
[13:22:27] <StephenLynx> word of advice with redis: I heard its performance degrades drastically when your dataset is larger than your RAM
[13:25:12] <jhertz> to me it sounds like we are trying to use the wrong tool for getting the job done, my hunch, which might be completely wrong is that MongoDB is not at all designed for this task, and therefore it probably does not makes sense to use it to solve this problem.
[13:30:12] <jhertz> then I think you can argue that even if it is not the best thing, it might be good enough and since there is knowledge on how to use it inside the team, it makes sense to do so. Anyway thanks for the incite, it helps me understanding this a bit better and now I just have to do my homework :)
[13:40:12] <deathanchor> StephenLynx: yep did that in house here. RDB was so slow for some operations we started cloning the data over to mongodb for speedy lookups
[13:47:10] <kaseano> hi, I was wondering if it was possible to update all the documents using a different value/column from each document
[13:50:52] <kaseano> StephenLynx: even with bulk write it looks like you have to use the same/static value for every document, instead of using the values on the document being updated
[14:18:29] <frankydp> Can anyone point me in the right direction to fix an index that needs to be unique, but that is not? Or is find update my best bet?
[14:42:19] <StephenLynx> run an aggregate and get the duplicates, then run a bulkwrite and make them unique
[16:27:45] <dijack> I have a question, I want to use mongo --eval from the command line to setup a cluster
[16:33:52] <dijack> function mongocolocation { echo "Executing mongo cluster setup" ~/applications/mongo/bin/mongo commandCenter --eval "cfg=rs.conf() && cfg.members[$1].priority=0 && rs.reconfig(cfg)" }
[16:55:46] <dijack> is there anyway I can assign mongo eval javascript value to a command line variable?
[17:01:12] <dijack> I want to get cfg.members.length and do a comparison between my bash command line argument and that value
[17:24:48] <StephenLynx> and then parse it using whatever tool you have to transform the string into an object.
[17:32:51] <StephenLynx> bash is funny. some stuff that you deem simple are actually complicated and vice-versa in bash.
[17:33:51] <happiness_color> @dijack - are you sure whatever you are doing in bash, can't be done in a javascript and run against mongo?...
[19:25:22] <liviud> quick question, can you have a collection thats not sharded and one that is sharded in the same database ?
[19:25:56] <liviud> if so what happens when your app talks to a router and asks for a specific collection which is not sharded
[19:42:53] <windsurf_> Does anyone know a good GUI for copying my local database or selected collections to a staging server? drag and drop perhaps?
[20:00:02] <windsurf_> What's the easiest way to copy collections from my local server to a remote server? i'm open to CLI
[20:00:38] <StephenLynx> I don't think mongo will be able to open a connection to a second database.
[20:27:55] <atomicb0mb> StephenLynx thank you so much!!! Do u know the mongodb university? I'm trying to learn from there
[21:31:03] <Astral303> hi, does anyone have any insight as to how stable or safe 3.0.6-rc0 is vs 3.0.5?
[21:31:13] <blizzow> We're inserting ~100mil records a day via 3 servers connecting to 3 mongos instances in front of a sharded cluster with 3 replica sets. Our mongos instances are melting their own faces off. Assuming an average 100kb document size with usually 20 fields, does anyone have recommendations on the number of connections and batch size for the inserts into our routers??
[21:58:21] <joannac> Astral303: it's a release candidate. check the changelog and see if there's anything in there that you need?
[21:59:24] <Astral303> I am faced with upgrading from 3.0.2 right now, and i am debating whether to do 3.0.4 or 3.0.6-rc0
[21:59:49] <Astral303> I'm typically hesitant to go with rc0 releases, as sometimes there are banal regressions in WT that don't get caught until rc0 has been around in a few hands
[22:07:02] <Astral303> blizzow, what does "melting their own faces off" mean for mongos instances? are you maxing out CPU or network?
[22:07:26] <blizzow> joannac: I'm currently deploying some mongos processes on a few servers to try and reduce the load but the mongos only started face-melting after upgrading to 3.0.x. Astral303: Load and CPU.
[22:09:51] <Astral303> blizzow, if you are pegging CPU after upgrading to 3.0 on an otherwise identical setup with 2.6 (load-wise and cluster-topology-wise), then you're likely hitting an issue of some kind or a performance regression. i would imagine the storage backend would have no difference on mongos performance. it might be worth doing some stack traces or whatever the techniques are for starting performance diagnostics
[22:10:53] <Astral303> it's also possible that if you changed your storage engine to WT, then you might have considerably faster insert rates that are stressing mongos simply due to increased throughput
[22:28:05] <blizzow> Astral303: We didn't move to WT. I'm wondering if the older java and spark drivers we use to communicate to mongo are bashing our mongos instances.
[22:53:17] <Astral303> blizzow, did you notice any decrease in your insertion throughput or latency? are the mongos processes pegged or the java drivers talking to mongo?
[23:31:48] <EllisTAA> hello. im trying to insert a record into my mongodb, but when i added this line: https://github.com/ellismarte/ellis.io/blob/master/server.js#L71-L75, i got this error message: https://gist.github.com/ellismarte/5a1be11878d3db879a39. it says db is not defined but according to this documentation that’s how you write it: http://docs.mongodb.org/manual/tutorial/insert-documents/
[23:41:21] <Boomtime> EllisTAA: the link you quote from is for the mongo shell, but you are clearly using nodejs
[23:46:17] <Boomtime> nevermind, you should just refer to the mongoose docs for the methods you need