Log file Viewer

[04:02:39] <Bach> hi everyone, moving mongodb to new server using db.copyDatabase. Any way to show progress or any verbose output?

[04:03:41] <Bach> no output at the moment, but I can see collections being copied by querying stats on the db

[04:15:01] <cheeser> i don't think so

[09:05:03] <durre> I'm trying out the new scala driver (https://github.com/mongodb/mongo-scala-driver). it isn't clear how to use it together with case classes. ofc I can write my own logic for serializing/deserializing the classes, but I dont want to rewrite it if it already exists

[09:12:15] <Kwoth> Hello, noob question: I have a post and a post has many comments. Do i keep comments as an array inside the post in db, or in a separate collection (that is what i am doing now).

[11:55:17] <fish_> hi

[12:09:58] <fish_> can I somehow replicate a whole, sharded mongodb cluster to another cluster? or does replication only work within a shard so I would need to add replication to each shard in the old cluster?

[12:16:14] <livcd> fish_: what are you trying to achieve ?

[12:26:31] <fish_> livcd: on a high level, migrate data from one cluster to a new one

[12:26:57] <fish_> both clusters are on AWS fwiw

[12:27:34] <fish_> and preferably I'd also like to migrate from 2.6 -> 3.2

[12:27:56] <fish_> the new cluster is up and running, has the same shards and numbers of replicas as the old one

[12:28:54] <fish_> I found https://docs.mongodb.org/manual/tutorial/migrate-sharded-cluster-to-new-hardware/ but this replaces basically one instance at a time

[12:30:02] <fish_> that works, but it would be IMO much nicer to stream replicate everything from to old cluster to the new one so they are in sync. after that, I could simply stop writes to the old cluster and start accepting writes on the new one

[12:30:19] <fish_> but I might might a reason why this can't work etc. not very familiar with mongodb yet

[12:56:09] <fish_> I think the biggest challenge will be migrating the config server state from the old 2.6 to the new 3.2 cluster

[12:58:35] <fish_> but if I could somehow stream all date from the old cluster via mongos on the new cluster, it should (re)create the necessary state in the config servers anyway I assume..?

[13:29:16] <fish_> so I think I have a plan now. given that the cluster isn't particular large and we should be okay with a small downtime (or read-only time), I would dump everything with mongodump against mongos on the old cluster. after that, I should be able to restore it running monogrestore against mongos on the new cluster (which already has replication and sharding setup)

[13:29:24] <fish_> does that sound reasonable?

[13:56:41] <amitprakash> How do I use bulk operations with mongocxx driver?

[14:49:10] <bros> Can you use a unique index on a field that is an array? (subdocument)

[15:01:59] <cheeser> i probably wouldn't

[15:02:22] <cheeser> the uniqueness would be applied to the array values and not the whole array itself, i believe

[15:07:42] <fish_> if I would add a a mongos instance of cluster B as a replica for each shard of cluster A, the world would break appart I assume? or is mongodb so smart that it can actually replicate data to a mongos which will take care of routing the data to the correct shard?

[15:11:01] <cheeser> a mongos is not a data bearing node. it's a router, essentially.

[15:12:35] <fish_> cheeser: I know, in that scenario I would hope that mongos routes the data to the actual shard replica

[15:12:56] <bros> cheeser: So I have to pull the entire array then do client side comparing to see if an element exists?

[15:13:28] <cheeser> bros: you can query against array elements directly

[15:13:35] <fish_> cheeser: basically I want to replicate my whole cluster A to cluster B. the clusters are sharded and each shard has a primary and two read replica

[15:13:40] <cheeser> https://docs.mongodb.org/v3.0/reference/operator/query-array/

[15:14:06] <cheeser> what's the distinction between A and B fish_ ?

[15:14:16] <fish_> the docs on migrating a cluster suggest to replace instances one by one: remove a replica from the old cluster and add the new node from the new cluster to it

[15:15:16] <fish_> cheeser: the reason behind this is that the old mongodb cluster was setup pretty much manually on AWS where the new cluster is properly deployed via cloudformation templates and packer built AMIs

[15:15:30] <fish_> but technically it's like moving to new hardware

[15:15:53] <cheeser> got it.

[15:16:32] <cheeser> for the replica sets, you could just add the new "machines" to the existing replica sets and let sync happen then decommission the old nodes once the sync is up to date on the new ones.

[15:19:02] <fish_> cheeser: yes, that's an option but a) then I don't have a easy way to fall back to the old cluster if things go wrong, b) I need to make all newe nodes reachable which isn't the case currently and c) (I didn't talk about that yet) I still need to somehow migrate the config servers to use replication sets

[15:19:17] <fish_> but yes, that's pretty much what the docs say as well and this should work

[15:20:14] <fish_> but it would be much less work and imo more robust if I could just replicate to mongos, similarly like what mongodump + mongorestore does. that way the configserver's state gets recreated automatically

[15:20:50] <cheeser> ah. the config servers will require downtime to migrate.

[15:21:27] <fish_> right

[15:22:18] <fish_> anyway, if it would be possible to add the mongos of the new cluster as "replica" for each shard of the old cluster that would be pretty great. technically it should be possible but probably isn't supported

[15:23:05] <cheeser> i don't know off hand. i haven't dealt with sharded clusters directly enough to feel good about trying to answer.

[15:23:19] <cheeser> if I get it wrong and you lose data, well, i'd feel really bad about that. :)

[15:23:33] <fish_> ...or some tool which connects to the old mongos instance and to the new one and just does all the replication things :)

[16:38:05] <Bodenhaltung> I build the phongo 1.1.1 driver but php-fpm crashed with segfault

[16:38:22] <Bodenhaltung> The cli has no problems.

[16:38:50] <Bodenhaltung> Dec 28 17:13:56 dev kernel: php-fpm[10716]: segfault at e87b5c0 ip 0000000000758252 sp 00007ffcd0061d40 error 4 in php-fpm[400000+968000]

[18:32:55] <bros> How do I select only matching elements from a subdocument?

[18:33:36] <cheeser> you know how to use projections?

[18:35:47] <bros> cheeser: I have to use an aggregation?

[18:51:45] <cheeser> you don't

[23:52:51] <jiffe> any idea what might cause [repl prefetch worker 7] Assertion: 13440:bad offset:0 accessing file: /var/data/mongodb/mongod/mail.0. See http://dochub.mongodb.org/core/data-recovery ?

[23:53:07] <jiffe> was humming along and then that happened, I don't see any fs or disk related issues