pmxbot IRC Log Viewer

[08:22:18] <thapakazi> Any mongorocks users inhouse online ?

[09:06:40] <krion> hi

[09:06:52] <krion> i got huge directory moveChunk on a primary node of a replicaset

[09:07:01] <krion> the thing is i've almost no more space

[09:07:18] <krion> I since switch on the secondary node

[09:07:27] <kurushiyama> thapakazi Not me. The implications of using a storage engine of which the implications for using it with MongoDB are not well understood (and hardly battle tested) and with a project of rather basic (and severe) bugs open are too much for me.

[09:07:47] <krion> in order to do the tricks rm -rf data to freeing space. But i wondering about the moveChunk

[09:08:20] <krion> https://docs.mongodb.com/manual/faq/sharding/#is-it-safe-to-remove-old-files-in-the-movechunk-directory

[09:09:21] <kurushiyama> krion Now what is the question? ;)

[09:10:22] <krion> To determine if all migrations are complete, run sh.isBalancerRunning() while connected to a mongos instance.

[09:10:35] <krion> Not sure to get that sentence ;)

[09:10:50] <krion> isBalancerRunning would be true no matter what.

[09:11:00] <kurushiyama> krion No

[09:11:04] <krion> kurushiyama: I suppose it's safe.

[09:11:35] <kurushiyama> krion Not necessarily. In general: yes. But there seem to be a few factors we should check first

[09:12:03] <kurushiyama> krion You are only talking of a replica set, but moveChunk actually comes from sharding.

[09:12:17] <krion> I don't understand. If balancer is not running, chunk are not moved.

[09:12:29] <krion> The phrase "To determine if all migrations are complete, run sh.isBalancerRunning() while connected to a mongos instance.

[09:12:30] <kurushiyama> krion running != enabled

[09:13:03] <krion> In short: if sh.isBalancerRunning return true, migrations are complete ?

[09:13:10] <kurushiyama> krion Wait

[09:13:21] <kurushiyama> krion Calm down and _think_ first.

[09:13:51] <kurushiyama> krion Are wa talking of a sharded cluster?

[09:13:56] <krion> I'm almost calm ;)

[09:14:04] <krion> Yes we are. Two replSet

[09:14:40] <kurushiyama> Ok. So data manipulation should truly and only be done through a mongos.

[09:14:42] <krion> each with the mandatory primary, secondary, and arbiter. My problem is that the primary of one of the replset keep growing.

[09:14:56] <kurushiyama> krion We will see into that later.

[09:14:59] <krion> ok

[09:15:33] <kurushiyama> krion Now, you have the balancer enabled, and it might be running (or not).

[09:16:22] <kurushiyama> krion In order to be absolutely safe for the deletion of the moveChunks directory, we should ensure that it is not running and will not run while we do maintenance.

[09:17:08] <krion> Hum, ok.

[09:17:21] <kurushiyama> krion So before we do anything, we should first check wether the balancer is currently running, wait until it is finished and then disable the balancer.

[09:17:29] <krion> (FYI, i was ready to do that procedure https://docs.mongodb.com/v2.6/faq/storage/#resync-the-member-of-the-replica-set )

[09:17:52] <krion> Oh, never did it before, except for upgrading.

[09:18:04] <kurushiyama> krion https://docs.mongodb.com/manual/tutorial/manage-sharded-cluster-balancer/#disable-the-balancer

[09:18:51] <kurushiyama> krion No big deal, but you do not want to cause harm out of negligence, do you? ;)

[09:19:19] <krion> Absolutely not, especially on friday ;)

[09:19:27] <kurushiyama> krion You bet!

[09:19:48] <krion> ;)

[09:20:00] <krion> I already stop mongodb on the primary that cause me trouble.

[09:20:28] <krion> (old primary, since i also already have forced secondary to become primary and vice versa)

[09:21:13] <kurushiyama> well, that should be obvious. As a rule of thumb: I always disable the balancer before doing maintenance work on a sharded cluster.

[09:21:34] <krion> Is it bad to stop it afterwards ?

[09:22:00] <kurushiyama> krion Well, after maintenance, I tend to reenable it ;P

[09:22:29] <krion> Of course. I mean since i've already stop one of the mongodb.

[09:22:57] <kurushiyama> krion Well, it is not ideal, at least in my book.

[09:23:05] <krion> Should i restart secondary, the stop balancer ?

[09:23:11] <krion> I can do that.

[09:23:11] <kurushiyama> Wait

[09:23:13] <kurushiyama> no

[09:23:21] <kurushiyama> stop the balancer now.

[09:23:22] <krion> ;-)

[09:23:36] <kurushiyama> And if just for excercise ;)

[09:23:51] <kurushiyama> do this on mongos, ofc.

[09:25:28] <krion> Waited for active ping to change for host entity-03:27019, a migration may be in progress or the host may be down.

[09:25:34] <krion> hum...

[09:26:18] <krion> This take some time.

[09:26:52] <krion> Waiting for the balancer lock...

[09:29:10] <krion> (but if I understand this correctly, chunk are on the secondary node that I've stopped)

[09:36:07] <krion> kurushiyama: https://gpldr.in/v/GnfuUmm7t5/AJ2Pu3N0fJfSfXbH

[09:36:38] <krion> may be this help to see the whole picture

[09:36:58] <krion> Still 'Waiting for the balancer lock'

[09:39:36] <kurushiyama> krion Ok. While we are waiting. Do you have an idea why only one shard runs full and there are supposedly a lot of migrations going on?

[09:40:25] <krion> runs full, you mean full of disk space ?

[09:40:44] <krion> and we recently migrate from 2.4 to 2.6 and have deleted collection

[09:41:15] <krion> on mongodb-02, 113G /data/db/mongodb/moveChunk

[09:41:21] <kurushiyama> krion Well, 2.6 is close to EOL, as a side note.

[09:41:29] <krion> on mongodb-12, 215M /data/db/mongodb/moveChunk

[09:41:42] <kurushiyama> krion I am talking of shards.

[09:41:54] <kurushiyama> krion not replset members.

[09:42:21] <kurushiyama> krion You do seem to have the same problem on the other shard, right?

[09:43:41] <krion> I'm not sure, sorry. My customer handle the shards.

[09:43:46] <krion> (FYI, msg:Waited too long for lock balancer to unlock )

[09:46:17] <krion> https://gpldr.in/v/h3OQHdbEjW/e12jnXf2zZ1agvrC

[09:46:19] <krion> state right now

[09:51:08] <krion> I'm sure you didn't like my answer about shards, kurushiyama ;)

[09:54:28] <kurushiyama> krion Your _customers_?

[09:54:42] <kurushiyama> krion As in Managed-DB customers?

[09:54:51] <krion> Yes. I'm managing the mongodb for my customer.

[09:55:04] <krion> Hum, I don't think so ?

[09:55:34] <krion> I do apt-get install mongodb, then my customer push data etc

[09:55:53] <krion> (I try to do a bit more, it's just for summarize)

[10:00:40] <krion> kurushiyama: you see ?

[10:01:31] <krion> For sure not ;)

[10:01:50] <krion> I'm glad you try to help me btw

[10:04:14] <krion> I will fell more safe if I restart the stopped mongodb, and do a sh.startBalancer().

[10:05:40] <kurushiyama> krion First, you might want to remove the data you want to remove.

[10:06:23] <krion> Even with the msg that said "Waited too long for lock balancer to unlock"

[10:07:13] <krion> Can you explain me what you mean by "only one shard runs full" ?

[10:08:52] <kurushiyama> krion Well, you seem to have a lot of chunk migrations going on. Might well be that you have a bad shard key.

[10:10:25] <krion> I'll ask.

[10:11:26] <krion> He is aggreeing with you ;)

[10:11:50] <krion> He setup a date of the day in the shard key.

[10:12:13] <kurushiyama> Bad does not even start to describe it.

[10:13:12] <krion> I guess he will be happy to know.

[10:13:16] <krion> ;-)

[10:16:10] <krion> Now i'm wondering about deleting only the moveChunk directory or do a complete resync as mentionned here https://docs.mongodb.com/v2.6/faq/storage/#resync-the-member-of-the-replica-set

[10:22:42] <krion> I'll do a full resync.

[10:27:30] <krion> Resync in progress. I will wait the end before reenabling balancer.

[10:41:11] <mroman> Is there a way to query for the first and last element of an array?

[10:41:45] <mroman> i.e. {data : [1,7,8,9]} and I want all rows where the first element is 1 and the last element is 9.

[10:42:27] <mroman> maybe through group and exploding the array then using $first,$last?

[11:12:41] <mroman> yeah unwind, and $first,$last works

[12:34:38] <mroman> Can I do if's in group?

[12:35:57] <mroman> like uhm with documents {id : 0, data : 1} {id : 0, data : 2} and I want to count the one's and two's

[12:41:36] <mroman> I'd need to do a group inside a group

[14:16:37] <kees_> is it possible to use mongos in a non-sharded cluster?

[14:16:53] <UberDuper> kees_: Yes

[14:17:10] <kees_> ok, nice :)

[17:39:26] <deathanchor> is there a way to show shell history from the mongo shell?

[18:47:14] <Ben_1> hi

[18:49:26] <Ben_1> is there a way to configure mongodb to write all log output to an file?

[18:52:49] <UberDuper> As opposed to writing to syslog?

[18:53:03] <UberDuper> There's a config option to specify a filename.

[18:54:19] <Ben_1> My problem is that I get bulk write errors with the async java driver and I want detailed information. I could start mongod in the terminal but this is inconvenient. that's why it would be better for me to log it.

[18:54:47] <cheeser> turn up the logging level prior to the writes

[18:57:20] <Ben_1> cheeser: searched for a "logging level" in my config file. Is it the verbosity?

[19:00:12] <Ben_1> set the verbosity to 5 but after deleting all content of my log file and processed the write operation again it is still empty

[19:01:22] <Ben_1> and it is not configured quiet

[22:30:26] <nofxx> Hey, we need a new mongoid release wth activemodel >3 <6 , rails 5 out

Log file Viewer

Help | Karma | Search:

#mongodb logs for Friday the 1st of July, 2016