Log file Viewer
#mongodb logs for Monday the 22nd of April, 2013
[01:44:20] <j83> After I added in the $match object, I'm now getting an Unexpected Identifier error. I cannot seem to correct it. Also is this proper usage of $project? For some reason I cannot get the $unwind and $group to work with "$alerts.name" unless I alias it through $project.... http://pastebin.com/Z6z40ay1
[02:27:01] <someprimetime_> i'm doing a reset pw functionality and curious if i should create two collections for it (one for storing the entry of the pw reset action) and then setting a boolean within the User schema to `reset: true..current algorithm: http://pastebin.com/21he4E2j
[05:39:37] <pringlescan> I'm using MongoDB 2.4.1. I'm inserting 624k documents, with safe and w:1 using the Node.JS driver. That works, the callbacks are fired as the inserts occur. Immediately after that from the same app, I insert another 250,000 documents into a different collection (same database). This time, NONE of the callbacks fire.
[05:39:53] <pringlescan> The documents are being inserted, for the first 43k [same number every time], then it stops for a minute or so, then it starts inserting at 11,000-13k a second according to MongoDB's stats, and then all the callbacks fire, including the 43k that hadn't fired before.
[09:03:19] <alcuadradoatwork> I'm lost with something about map-reduce.. I must be misunderstanding something
[09:03:31] <alcuadradoatwork> I made the most trivial example on earth, and don't understand the results
[09:04:08] <alcuadradoatwork> how can I get an ISODate in a value if my reduce function always return 1?
[11:39:01] <sinclair-linux> however, i am wanting to start mongo remotely, (on a remote linux server)
[11:43:37] <alcuadradoatwork> does a simple {$project: {name: "$_id"}} in an aggregation takes O(n) time?
[11:50:58] <sinclair-linux> Nodex: looking in the mongodb.conf, it says it is bound to 127.0.0.1 on port 27017
[11:58:41] <alcuadradoatwork> it would show the parameters of the mongoclient and a bunch of stuff more
[11:59:41] <alcuadradoatwork> if not, how can I count the number of documents that are identical except for their _id?
[11:59:49] <sinclair-linux> Nodex: note tho, im not sure i know why i would need to tinker with php mongo client/driver, its connected successfully to other stuff
[12:00:22] <sinclair-linux> including a mongo instance on this very machine, when i run the command...
[12:03:59] <sinclair-linux> alcuadradoatwork: sorry, im not here to be cursed at, ill figure it out on my own
[12:05:46] <Nodex> you started to question why I was asking you to test things which I don't have time for and am not interested in discussing. You can either accept help or not - choice is yours
[12:08:43] <Zagdul> hi1:PRIMARY> db.testreplication.insert({_id:new Date()});printjson(db.runCommand({getLastError:1, w:4, wtimeout:18000}));
[12:15:00] <Zagdul> somehow it seems like the application nodes are connecting to the node that is just replicating and in state "STARTUP2"
[12:17:19] <Zagdul> the second bug did occur only with 2.4.2 - as it worked fine with 2.2.4 and 2.4.1
[12:17:53] <alcuadradoatwork> in the aggregation fw, who does $group work when _id is an object? the doc says nothing about it
[12:23:01] <Zagdul> is it possible that the timout bug is caused by having one node "hidden" and "priority=0"?
[12:27:23] <alcuadradoatwork> I see, but now I read that $group works in memory, so that won't work for me. thanks Nodex
[13:03:20] <woozly> new BasicDBObject("$set", new BasicDBObject("message_id", Long.valueOf(packet.getMessageId())))
[13:08:33] <Kosch> Creates "Oplog" a set of files similar of Oracles Redolog, which can be used as kind of incremental backup?
[13:11:01] <Zagdul> Kosch, I don't think so. Technically you're probably right, but I don't think there's a way to play back such a stored oplog
[13:11:39] <Zagdul> woozily - this is strange, we have a similar collection here, updating (from Java!) works about 1000/sec
[13:14:17] <Kosch> Zagdul: how is usually the backup problem solved when having large amount of data? always backup all data sounds not really practical to me.
[13:17:39] <Zagdul> Kosh: we're exporting everything (right now about 250GB only - so that works for now)
[13:24:50] <Kosch> Zagdul: Unfortunality the situation is that the data is located on hardware sitting in customers office. so I've only usb or network as transfer medium... indeed, maybe FS snapshots using LVM could be an option.
[13:52:42] <Kosch> Zagdul: or other idea: lets say I'd configure an 2nd mongod as secondary. when backup is started, the secondary node is started and data will be replicated. the secondary stores their data on the backup device. when finished, secondery mongod will be stopped. Is in these cases also everything replicated, or just the stuff between the last time secondary is started?
[13:53:35] <Zagdul> kosch: this is actually valid scenario. The replication only replicates the things, that did change since last replication
[13:54:12] <Zagdul> kosch: creating a node in a replica set as Hidden and with primary 0 is a backup node
[13:55:01] <Zagdul> let me know, if you got any problems with that - we have a setup similar here (4 nodes Replicaset, 1 Hidden Backup node)...
[14:34:06] <Snebjorn> I'm trying to decide whether to embed a bunch of documents or keep them in a separate collection. How do you usually decide?
[14:37:06] <Zagdul> as more people are here: anyone having problems with syncing a node in a replica set with 2.4.2?
[14:46:27] <sinclair-work> saml: oh, i was looking at the Linq stuff for the Mongo driver, and although i can query fine, it would be nice to use the Linq to filter a subset of documents to delete
[14:47:13] <sinclair-work> saml: in the C# driver, Linq queries return type MongoDB.Driver.Linq.MongoQueryable<T>
[14:48:48] <sinclair-work> basically, i need to turn a MongoDB.Driver.Linq.MongoQueryable<T> into a MongoDB.Driver.IMongoQuery
[14:52:02] <saml> http://docs.mongodb.org/ecosystem/tutorial/getting-started-with-csharp-driver/#remove-an-existing-document
[14:56:03] <Snebjorn> Zagdul: The embedded documents are small in size but there are a lot of them. Lets say a document look like this { _id: ObjectId, owner: userRefId, ... } then I'd like to query on all owners. When I put them in a separate collection even if I index on owner I can only get about 30 queries/sec (benchRun) which I find odd. However if I embed them and do a find({"subs.owner:xyz"}) then I get about 30k quries/sec however I get a lot of
[14:59:54] <saml> if app connects to mongo01.mongodb.com , it doesn't have to know if the end point is replica set or not
[14:59:55] <sinclair-work> well, if you are going to use some software in a commercial product, better be familiar with the license, otherwise....you'll get burned one day
[15:00:57] <scoates> I stumped Derick with this one yesterday. Maybe someone else might know why my query (second stanza) isn't using the "right" index (see the next stanza for the hint), here? http://paste.roguecoders.com/p/4face5649612e840da04c5fea0491c9b.txt
[15:01:33] <Zagdul> Snebjorn, this is weird. R U sure the index was hit? usually the separate collection should be faster
[15:02:00] <sinclair-work> ninkotech: just things, i don't really have much time to go into detail as i have a meeting to be in in 5 mins
[15:03:36] <meekohi> .aggregate({ $group: {_id: "$identifier"}}).count() doesn't work how I'd expect.
[15:03:51] <Zagdul> Snebjorn, and your query looks like? when you only have owner in the query, all should be fine. if your query is more complicated, the index needs to be as well
[15:29:19] <scoates> this is linked from that page: http://docs.mongodb.org/manual/reference/glossary/#term-write-concern
[15:42:36] <Snebjorn> Any mongo gurus that wanna have a look at this http://stackoverflow.com/questions/16150942/to-embed-or-not-embed
[16:00:10] <Snebjorn> How do I cover an entire query with index if I only query on one element? Example: find({a:1}).explain() says indexOnly is false even though ensureIndex({a:1}) is called on the collection
[16:04:21] <saml> "errmsg" : "db exception in producer: 1000 replSet source for syncing doesn't seem to be await capable -- is it an older version of mongodb?"
[16:42:10] <dgarstang1> For some reason, this isn't working... find({'fun': 'pkg.list_pkgs'}, {'jid': 1}).sort([('jid', -1)]).limit(1) .... if i remove the limit(), the sort is fine. However, if I put the limit(1) back, the document I get is not the most recent...
[16:44:41] <dgarstang1> Nodex: I did. "owever, if I put the limit(1) back, the document I get is not the most recent..."
[16:46:35] <Nodex> just an fyi, if you're using ObjectId's as your _id you get a free sort by using _id:-1
[16:47:08] <dgarstang1> Nodex: Without the limit(), I get a list of documents in descening order. I see the most recent first. However, after adding limit(1), I don't see that most recent document as I would expect, but apparently an arbitrary one.
[16:50:23] <dgarstang1> Nodex: Yes. Most recent, sorted by jid. Index? Don't know... i'll check.. just as soon as I google how to check...
[16:53:10] <dgarstang1> Nodex: ~ 2,500... but normally it will be a lot less. Hang on... I need to go back and have a chat with some people. I don't like how the document is structured anyway.
[17:30:59] <jol> Just upgraded a replicaset to 2.4.2 running with "noauth=true" and pymongo now spits "OperationFailure: database error: not authorized".. Tried upgrading to pymongo 2.5 without luck. Any idea of what could be wrong?
[17:43:35] <Kzim> hi, i have this issue on mongodb 2.0.6 : right object doesn't have full shard key and shard version not ok in Client::Context: client in sharded mode, but doesn't have version set for this collection
[17:46:51] <kurtis> Hey, we have a fairly large project which dumps all of our analytics into MongoDB. I'm working on a web-service which includes Users (and possibly billing Info in the future). I'm worried about write guarantees. Would it be recommended to take a hybrid approach and use a traditional RDBMs for the user management aspect?
[17:47:23] <hallas> Hmm, anyway I can improve the performance of an _id: { $in: […] } query, it's running incredibly slow..
[17:49:41] <dgarstang1> I'd like to do an update(query), and do an insert if the document isn't there
[17:53:29] <zagdul> I already asked that a couple of hours ago, maybe there are some new people around that might have experienced the same problem: when setting the W to the actual number of nodes, it ends up in a timeout in our productive environment, no matter what timeout is defined
[17:55:22] <zagdul> one other bug we have is the initial sync of a member in a replica set of 4 nodes: it causes the other nodes to "delay" answers or something, that caused a downtime of our application and cost us a couple of users
[17:55:53] <zagdul> I don't want to change the DB, but people in our company are discussing the migration back to MySQL again!
[17:57:22] <kurtis> saml, I'm not a mongo expert by any means but maybe this helps? http://docs.mongodb.org/manual/reference/commands/#touch
[17:58:35] <scoates> saml: just `mongo --host hostname.of.primary` (and --port if it's non-standard)
[17:59:49] <scoates> the shell is meant to connect to a specific node, not to be a general-purpose driver
[18:01:49] <dgarstang1> because..... this.... db.doc.update({'_id': 'hot.foo.com'}, {'fun': 'test.ping', 'jid': '20130422175559545439', 'result': 'True'}, {'upsert': 1}) keeps inserting a new document with a new _id, rather than inserting one where _id = 'host.foo.com'
[18:03:29] <scoates> db.doc.update({'_id': 'hot.foo.com'}, {{_id: 'hot.foo.com', 'fun': 'test.ping', 'jid': '20130422175559545439', 'result': 'True'}, {'upsert': 1})
[18:13:10] <saml> https://gist.github.com/saml/a9d1dfd14d8d199f1795 why can't mongoimport connect to my mongo replica set?
[18:20:52] <Kzim> hello i have a "right object doesn't have full shard key" when i update. is there a way to rebuild the chunk or something ?
[19:08:10] <jol> hallas, check out 3rd section of http://snmaynard.com/2012/10/17/things-i-wish-i-knew-about-mongodb-a-year-ago/, maybe related to your issue?
[19:13:34] <scoates> there are probably tools for it. We manage ours, ourselves, and I don't find it painful.
[19:13:48] <scoates> (I do wish it could do auto-discovery like ElasticSearch does, sometimes, though)
[19:17:31] <saml> ouch... when i change replica set configuration, clients need to reconnect using different mongodb:// uri
[19:18:04] <saml> oh man................ so i can't really have stable endpoint for clients to connect if i'm using replica set
[19:19:26] <scoates> the clients will ignore nodes that are not currently in the replicaset (after initial connection)
[19:20:13] <saml> when clients connect to mongodb://m1,m2,m3,m4 when only m1 and m2 are live, clients (casbah, scala) keeps polling m3 and m4
[19:21:58] <saml> m1 should work as a loadbalancer distributing reads and writes transparently to m2,m3,m4,m5, ...
[19:22:23] <scoates> because it's not fine for everything to go down in the problem case that relicasets were designed to solve
[19:23:10] <scoates> any of my database nodes can go down without affecting clients. that's the point.
[19:24:12] <scoates> you might be able to coax mongos to do what you want, but I don't know. Sounds like you're trying to make MongoDB do something that it's not designed to do, to me.
[19:24:13] <saml> m1 isn't really database. it can be. but it's transparent. it can be proxy, lb, or whatever
[20:18:14] <scoates> hallas: a large $in isn't going to perform well. you probably need to denormalize.
[20:18:53] <hallas> scoates yeah actually it was because I was using Mongoose.js, which adds alot of ORM magic, which takes up a lot time and probably memory too
[21:08:51] <JoeyJoeJo> I have a collection of parent documents and a separate collection of child documents. I need to know which children belong to which parents, and each child will only belong to one parent. My plan is to include the ObjectID from the parent as a field in each child. Is that a good idea?
[21:09:46] <JoeyJoeJo> I know that this is trying to replicate SQL joins, but I need mongo for its lack of schema
[21:10:15] <scoates> that will work, JoeyJoeJo. Depending on your actual needs, it might be best to just store everything in one collection.
[21:29:11] <jstallar1> JoeyJoeJo: I've never used it, but this exists: http://docs.mongodb.org/ecosystem/drivers/php-libraries/#drupal
[21:33:38] <bartzy> What would be the best way to insert a document with a unique index, but if it exists - return the _id of the document ?
[21:46:20] <scoates> bartzy: if it doesn't need to be atomic, I'd do a find and then an insert if the find fails
[21:46:59] <bartzy> insert, and if failed with code 11000 (duplicate key), get the document with the unique indexed field.