Log file Viewer
#mongodb logs for Thursday the 29th of September, 2016
[00:24:15] <AlmightyOatmeal> https://jira.mongodb.org/browse/SERVER-17688 <-- open for almost a year and a half but yet the default storage engine doesn't support this feature and https://docs.mongodb.com/manual/reference/command/parallelCollectionScan/ makes no mention of wiredTiger not supporting it
[00:39:25] <AlmightyOatmeal> GitGud: you would be better off dumping the data to a backup. if you just copy the directory then you risk inconsistent data amongst a number of other potential problems
[00:40:47] <AlmightyOatmeal> GitGud: actually i am wrong. copying the data directory isn't as fragile as it is on a larger RDBMS
[00:41:03] <AlmightyOatmeal> GitGud: https://docs.mongodb.com/manual/core/backups/ and more of what i was referring to: https://docs.mongodb.com/manual/tutorial/backup-and-restore-tools/
[00:41:21] <GitGud> well AlmightyOatmeal what im doing is just more a rough copy not for a production solution
[00:43:34] <AlmightyOatmeal> GitGud: makes sense. looks like mongo gives you a plethora of options then :)
[00:44:37] <GitGud> AlmightyOatmeal, basically have 1 db. 1 repl set. and 1 backup file system copy i do every week
[00:45:07] <AlmightyOatmeal> GitGud: if you use ZFS then you can send filesystem snapshots to a remote host ;)
[06:57:30] <crazyadm> "errmsg" : "could not find host matching read preference { mode: \"primary\" } for set shard1",
[06:59:18] <crazyadm> bin/mongod --shardsvr --dbpath /data0/mongodb/shard2db --logpath /data1/logs/shard2db.log --logappend --fork
[07:01:44] <crazyadm> bin/mongod --shardsvr --replSet set1 --dbpath /data0/mongodb/repl11db --logpath /data1/logs/repl11db.log --logappend --fork -port 28018
[07:04:14] <crazyadm> "errmsg" : "could not find host matching read preference { mode: \"primary\" } for set set1",
[07:10:58] <crazyadm> bin/mongod --shardsvr --dbpath /data0/mongodb/shard1db --logpath /data1/logs/shard1db.log --logappend --fork
[07:11:45] <joannac> you're going to find it much more difficult to convert it to a replica set after adding as a standalone
[07:57:00] <crazyadm> replSetReconfig should only be run on PRIMARY, but my state is SECONDARY; use the \"force\" argument to override
[12:37:39] <jrmg> Hello everyone. I've got a question on what's the best way to rebuild a collection that acts as a cache layer. The collection contains aggregations results from another collection (>1M documents). What would it be the best way to rebuild the entire cache collection preventing the process from taking ages? via shell? Multithread client?
[13:19:27] <mumla> hi everyone! does somebody know how to execute db.stats() in java with die mongodb-driver? I just downloaded the newest version 3.4.0beta and the command "getStats" doesn exists as expected according to docs.
[14:51:44] <AlmightyOatmeal> is there a secret to using a compound index? i'm trying to query a single field that is part of a compound index but the query planner is telling me that it's going to scan the entire collection instead of using the compound index :(
[15:14:44] <AlmightyOatmeal> StephenLynx: what do you mean by "how is the compound index built"? it's a simple find() with a field that is in the compound index and a corresponding value
[15:15:52] <AlmightyOatmeal> oh. the field i'm querying is one of the last fields in the compound index
[15:33:33] <AlmightyOatmeal> mongo decided to stop returning results around 14M documents which is under half of the documents in that collection. so now i need to re-run a find() and upsert all of the 14M documents that mongo has been spending the past 12 hours spitting out at me.
[15:36:24] <AlmightyOatmeal> GothAlice: if i'm doing a find() on the entire collection, wouldn't the data structure be irrelivant?
[15:37:18] <AlmightyOatmeal> GothAlice: it's doing a collection scan so the index would be irrelivant and i'm not trying to filter on any criteria
[15:38:14] <GothAlice> Does the data fit entirely in RAM? If not, you're limited by repeated page faulting.
[15:39:03] <AlmightyOatmeal> GothAlice: the entire collection won't fit in RAM but that's a good suggestion, let me check my vm stats
[15:40:24] <GothAlice> However, more generally, a process which requires examining every single record in a collection of millions in a single pass is sub-optimal. At a minimum you're going to need to track progress, handle cursor timeouts, and retry. If bulk write and bulk read is purely the goal, you might also investigate capped collections, which are somewhat more efficient for FIFO use.
[15:44:44] <GothAlice> (Capped collections also provide a means to stream process data instead of buffering it all then processing it all in one go.)
[15:45:37] <AlmightyOatmeal> GothAlice: mongo is grabbing ~1k documents at a time and even if the entire collection doesn't fit into memory, a majority share can fit in memory but that still wouldn't explain the slowness.
[15:48:30] <GothAlice> http://s.webcore.io/1x0q2c002y0E/ludicrous-kill.gif — that's a lot of faults!
[15:50:41] <GothAlice> AlmightyOatmeal: Faults represent access to memory that was not loaded into the process. It might already be in cache, so the fault would "link" the missing memory into the MongoDB process. This wouldn't show up as disk IO directly.
[15:51:59] <GothAlice> However faulting freezes the thread that faults until the linking is complete, so, it's terrible for performance.
[17:32:06] <FrancescoV> Hi all, I try to use mongodb in a docker container but I got this error: 'KeyError: 'DB_PORT_27017_TCP_ADDR'' using this code: client = MongoClient(os.environ['DB_PORT_27017_TCP_ADDR'],27017)
[18:10:02] <StephenLynx> if he were to not use docker and it would work, I am quite sure he shouldn't say is not helpful.
[18:11:23] <FrancescoV> I have no experience with mongodb, just starting a basic prototype project using tensorflow & docker
[20:27:26] <offlim> is there a way to upgrade mongodb from the existing community version to the exterprise version
[21:00:40] <edrocks> do you have to do anything special to upgrade a replica set's `protocolVersion` to 1? I just upgraded everything to mongodb 3.2.9
[22:22:19] <DarkCthulhu> I have a question about mongodb replicasets. If I'm running a replicaset in production, I need to always send writes to the master right? How do I discover the master?
[22:23:51] <joannac> connect using a replica set connection, the driver will determine which one is the primary
[22:30:08] <joannac> https://docs.mongodb.com/manual/reference/connection-string/#replica-set-option
[22:31:02] <DarkCthulhu> joannac, Ah! and typically, is that what a connector library for mongo also would do?