Log file Viewer
#mongodb logs for Thursday the 26th of June, 2014
[00:25:05] <cheeser> nothing comes to mind. except something like truecrypt but that still leaves things visible running on the machine.
[03:39:56] <MacWinner> f###, i just upgraded from 2.6.2 to 2.6.3.. a simple 2 node rpelica set..both nodes are stuck as secondary
[03:48:05] <gogolook> how could I delete massive data in mongodb without hurting server performance?
[05:23:27] <zeus13i> I'm still interested in knowing how to achieve at-rest encryption without application-layer complications, in case anyone can help. Thanks
[06:35:14] <Viesti> I'm using mongodb basically as kv-storage with a 3 node replica set, writing to master and reading from secondaries
[06:36:11] <Viesti> daily I'm updating the dataset by writing to the master node (currently ~75GB worth of json documents)
[06:37:29] <Viesti> was able to cut that into 6 hours in a development cluster which doesn't have the read activity that the production cluster has
[06:49:11] <Repox> Hello. I have this query: db.members.find({"list_id": 57, "created_at": { "$gt": ISODate("2014-05-24T00:00:00.000Z") }, "$where": "created_at == updated_at", } ).count(); This gives me this error: 16722 ReferenceError: created_at is not defined near 't == updated_at'
[08:18:45] <danijoo> for example if i have a name and a country field and want to reject all documents where theres already an entry with same name+country
[10:02:01] <remonvv> Meaning, what is your question? Can MongoDB be used as a caching solution? Does MongoDB cache query results?
[10:03:23] <tilerendering> remonvv: ok my question was if there existed a mongodb cache. and yes the 2nd question would have been: does mongodb cache queries. and then: can it be made a distributed cache
[10:05:37] <remonvv> tilerendering: It doesn't cache query results but due do its storage engine being mmap based it will usually have "hot" parts of collections in memory (provided there is enough of it). That would result in pretty good query performance although it's not strictly caching. If that is good enough for you then sharding your MongoDB cluster would make it effectively a distributed "cache".
[10:16:22] <Nopik> _boot: yes, http://docs.mongodb.org/manual/reference/method/db.setProfilingLevel/
[10:40:08] <Nomikos> In the docs I'm reading "mongoimport and mongoexport do not reliably preserve all rich BSON data types because JSON can only represent a subset of the types supported by BSON"
[10:40:20] <Nomikos> but also "JSON can only represent a subset of the types supported by BSON. To preserve type information, mongoimport accepts strict mode representation for certain ptypes."
[10:40:53] <Nomikos> and the same for mongoexport - does that mean it's safe to use these, as long as they only use eachother's data? together
[10:42:15] <Nomikos> reason I was wondering (and not using mongodump/restore) was if I could use mongoexport/import to copy a collection to another db using pipe '|' on the commandline
[11:14:21] <Viesti> remonvv: I'm also seeing a slowdown when writing into a replicate set master compared to writing into single node
[11:55:56] <Viesti> rspijker: hmm, seems I'm using the default that pymongo gives, which is lookep from the collection...
[12:04:23] <rspijker> ok, I don’t know what the writeConcern of your collections is… That’s a driver specific thing, I suppose...
[12:04:51] <rspijker> it has to do more work, it has to replicate to another machine before it tells you it’s done
[12:05:15] <rspijker> if you use writeconcern: 1, it should theoretically do the same amount of work before telling you it;s completed
[12:05:28] <rspijker> that is, acknowledge the write on the primary (or on the single node, if not a RS)
[12:27:13] <Viesti> and actually there's one other process doing updates continuously at ~30 updates/s into the same database but different collection
[12:39:34] <Viesti> and write concern gets me somewhat similar control as say in riak probably (to choose to how many nodes to write)
[12:41:53] <Viesti> well anyway, the other problem is that my testing replica set data set and read/write activity don't quite match with production and testing in production isn't too neat... :P
[13:21:41] <AlexejK> Hi.. I've got a requirement recently to ensure that we only keep x-month worth of data (meaning we want to backup and remove the data that is older than this threshhold). TTL Collections don't work for this, and It'r really some maintenance task that we could schedule via cron to e.g backup data, and then remove.. But is it a good idea to just do a delete of data that is older than specific period? This DB we are operating on will get regular
[13:21:41] <AlexejK> writes all the time and i'm assuming they will be "waiting for lock" during the delete operation..
[13:23:32] <kali> AlexejK: unless the combination of the deletion and the regular insertion overwhelm your system, it should work
[13:24:50] <adamcom> the most efficient thing to do is segregate date into one DB per timeframe (e.g one per month), then when you want to delete, just drop the database - granted it only works if you can deal with changing DB names over time, but it's by far the most efficient way to do it (from a locking perspective, and a disk space use perspective)
[13:25:51] <adamcom> also makes restoration from backups pretty easy, assuming you take file based backups
[13:26:07] <AlexejK> thanks.. so the splitting data per DB would be good.. but i think right now we are unable to achieve this, even though this probably is the smartes way indeed.. this could be solution v2 :-)
[13:26:36] <cheeser> one recommendation from mongoworld yesterday was similar but to use separate collections instead of DBS
[13:26:56] <AlexejK> so as our writes to this specific DB are Unacknowledged, It should be relatively safe for our application when mongo has a long lock
[13:28:24] <adamcom> AlexejK: would also recommend adding a compact/repair cycle to your cron (run after delete) - compact will help with fragmentation and space reuse, repair will reclaim space
[13:29:05] <adamcom> usual caveats apply - best to run on a secondary when out of the set, then return secondary to set then repeat for each member of set until all are done
[13:29:28] <adamcom> you can also just wipe a secondary, resync from scratch and repeat rather than repari
[13:30:20] <AlexejK> Got it.. Will see if we can get that working (repair is a must i think) and then also see if we can do the DB split of some kind in near future
[13:31:01] <adamcom> note - for repair you need a lot of free disk space - it rewrites the files in a temp folder while the old ones are there
[13:31:19] <adamcom> if you resync a secondary you are down a member, but no such free space requirement
[13:31:29] <AlexejK> we didnt start sharding yet (i kow, we shouldve done it from day 1)... so considering different things to shard on.. would time be smart considering this case?
[13:32:05] <adamcom> if you shard using time, then you will generally get hot chunks and get no scaling benefits from a write perspective
[13:32:36] <adamcom> basically Google for monotonically increasing shard key or similar - plenty of detailed explanations available on that one
[13:32:47] <AlexejK> A lot of free disk space = correct to assume tht as long as we have at least the size of current data free it should be safe?
[13:34:48] <rspijker> sharding on time might be good for your use case.. It depends really. Like adamcom said, you won;t get write scaling. But in return you’ll get good query isolation, which might be important for you, I don;t know…
[13:36:34] <AlexejK> Got it.. really didnt get a full grip on sharding yet, but was interesting if this may be smart to do based on time for this specific case.. Will read up more and see how we can solve this short-termna dn then long-term
[13:36:34] <adamcom> agree, can be suitable for particular use cases as long as you are aware of the down sides - can also be important for certain types of sharding strategies - I've seen people have a short term storage (recent data) set of shards and a long term storage (old data) set of shards. To achieve that with shard tags you really need time as a piece of the shard key
[13:38:11] <rspijker> basically you have to update your maxRange to rotate the oldest stuff out and route the new stuff to that server again
[13:40:11] <AlexejK> I see.. thanks :-) Sharding is on my next to-do list so will have to really consider this things and see now if i can solve this short-term and apply a smarter & better way the same time we do sharding :-)
[14:10:48] <richthegeek> hi - I'm using the aggregation framework to get the number of interactions of each type over a timeframe (like so, plus some date filtering: http://pastie.org/9326749)
[14:11:20] <richthegeek> is there any way to at the same time get the total number of interactions (ie, in this case 948) included in the results?
[14:18:58] <richthegeek> perhaps there's a way with a group (_id: null), project, and then second group? probably not very efficient though
[14:21:33] <rspijker> well.. once you do the initial group, your pipeline will only have 1 document in the stream (the grouped result) and you won’t be able to ‘dis-aggregate’ it anymore to get the groups you really want
[14:22:10] <rspijker> afaik, there’s no way to get a ‘side-result’ or add a document in the pipeline
[16:19:06] <og01> if i have a replica set with two mongod instances, a is primary, b is secondary, and the network connection between the two drops, b elect itself as a primary? and a will stay as a primary, what happens when the connection is re-established?
[16:20:42] <nukulb> how can both a and b have data? do you mean the oplog wasn't applied to b before the network connection dropped
[16:21:58] <nukulb> b will only elect itself as primary I think till it knows that its the only primary
[16:24:04] <nukulb> q85: so if the hearbeat between the two is lost, then both a and b can think they are primary?
[16:25:18] <og01> no i think he's saying they will stay the same, because there is no way to win a voting process
[16:25:56] <nukulb> also in line with this - http://docs.mongodb.org/manual/core/replica-set-elections/
[16:26:00] <og01> what happens when b (which is secondary) receives writes when there is no connection to the primary
[16:27:03] <og01> again what happens if a secondary with no access to a primary receives a write operation?
[16:33:54] <daidoji> hello, I have a field stored as an ISODate() but I'm having trouble searching it
[17:31:29] <cozby> hi, I'm running into a weird problem. I launched a mongodb instance I had with data, I wiped out all the data by removing the files in /data /journal /log
[17:45:49] <mattt_> Is it possilbe to create a compound index on a set of sub-document keys and properties of the sub-documents? For example, given the document { ‘stuff’: {‘foo’: {‘id’: 123}, ‘bar’: {‘id’:456}, ‘baz’: {‘id’:789} } } is it possible to create a compound index on the keys of ‘stuff’ and the key ‘id’ of each sub-doc? What I’m struggling with is how to refer to foo, bar, baz.
[17:53:13] <q85> og01: you cannot perform write operations on a secondary. you'd likely receive an error of some kind.
[18:43:16] <alexgorbatchev> curious if anyone had issues recently getting mongodb to work in vagrant?
[22:59:58] <schimmy> I am making a fairly large query (~4.3 MB) to mongo 2.6.1, and am receiving the error:
[23:00:05] <schimmy> "BSONObj size: 17630996 (0x10D0714) is invalid. Size must be between 0 and 16793600(16MB)"
[23:00:09] <schimmy> This indicates that either the query is over the 16MB limit, or the response is
[23:00:20] <schimmy> I have tried changing the ids to not match anything in the database, so that no data is returned, still get the error
[23:00:37] <schimmy> db.cars.find({'owner': {'$in': ['aaaaaaaaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbbbbbbb' ... x 200k
[23:00:46] <schimmy> It seems as though my 4MB request somehow gets blown up into 17MB, but that sounds ridiculous.
[23:01:13] <schimmy> What has changed between 2.4.10 and 2.6.1 that would throw this error, and how is this error possible?
[23:05:23] <schimmy> oh - one correction, the find also has a limit(1) for the purposes of debugging, still get the error
[23:08:25] <wfaulk> today I just happened to notice looking at logs that one of our config servers was corrupt and thus the config set was out of sync
[23:26:15] <wfaulk> "local.oplog.$main Assertion failure !loc.isNull() src/mongo/db/pdfile.cpp 1816"
[23:27:05] <wfaulk> "update config.lockpings query ... exception: assertion src/mongo/db/pdfile.cpp:1816 locks(micros) w:6203 3ms"
[23:37:51] <wfaulk> only thing that happened within an hour of it starting was an 8s jump in time as NTP was fixed
[23:54:47] <wfaulk> looks like the system was not shut down properly. fsck not forced, but possibly should have been