pmxbot IRC Log Viewer

[00:51:03] <pksadiq> When I create a new schema, what is the first field of the key valu pair called in mongodb's terms: say like what to call owner, title, and date (is it key, or field, or something else?) in 'new Schema({ owner : ObjectId , title : String , date : Date });' (the code is from mongoose)

[02:43:21] <k_sze> Is there a strict order in which I should start the configdb, the replica set, and mongos? I can't remember. It's been too long.

[06:05:15] <DarkCthulhu> So, if I am trying to configure a replicaset using just the configuration files, is this possible?

[06:05:46] <DarkCthulhu> Or must I always go via the mongo client and rs.initiate() and so on?

[06:23:10] <Boomtime> @DarkCthulhu: both, the config file is local information only (thus, the name of the replica-set this host expects to particpate in), rs.initiate/add/remove are group commands, these involve all members of the set

[08:54:13] <virtualsex> { "ok" : 0, "errmsg" : "no such cmd: replSetInitiate", "code" : 59 }

[08:58:01] <virtualsex> please give a page for command addshard

[08:58:41] <Derick> "mongodb addshard command"

[09:02:05] <Zelest> Derick, how hard is it to get patches applied to mongodb?

[09:02:36] <virtualsex> https://docs.mongodb.com/manual/reference/command/addShard/#dbcmd.addShard

[09:02:46] <virtualsex> how do i addshard

[09:02:53] <virtualsex> i have two shard

[09:03:00] <virtualsex> no rep

[09:03:07] <virtualsex> no replica

[09:03:39] <Derick> Zelest: depends on how good they are...? :)

[09:03:56] <Derick> Zelest: I think we have a person to look at PRs specifically

[09:04:14] <virtualsex> { addShard: "<hostname><:port>", maxSize: <size>, name: "<shard_name>" }

[09:04:15] <virtualsex> i use this

[09:04:46] <virtualsex> 2016-08-10T03:06:25.279-0400 E QUERY [thread1] SyntaxError: missing ; before statement @(shell):1:38

[09:07:34] <virtualsex> "errmsg" : "addShard may only be run against the admin database.",

[09:08:01] <virtualsex> forget it , i got it

[09:09:30] <Derick> OK :)

[09:09:34] <Derick> Zelest: what are you thinking of?

[09:10:04] <Zelest> oh yeah, sorry, forgot this chat.. haha

[09:10:53] <Zelest> Derick, OpenBSD is having W^X mandatory in upcoming 6.1.. in 6.0, they just log W^X violations in order to log and fix things.. It would be really nice to see those fixes applied upstream later on.

[09:11:05] <Derick> OK

[09:11:15] <Derick> are there many of these in MongoDB ?

[09:12:19] <Zelest> no idea yet

[09:12:26] <Zelest> had a few in mongo (the client)

[09:12:38] <Zelest> and read on the lists about some people having issues with it

[09:12:49] <Zelest> (and that applies to a LOT of ports, so nothing magical about mongodb)

[09:13:13] <Derick> I don't think we'll do lots of things to OpenBSD ourselves, but can't see, why a reasonable patch shouldn't be applied.

[09:13:31] <Zelest> Good to hear :)

[09:13:48] <Zelest> 6.0 isn't even out yet so, it will be a future project I guess :)

[09:14:12] <Derick> hah, ok

[09:19:14] <Zelest> but yeah, 6.0 is released on the 1st of september..

[09:19:27] <Zelest> and the 6.1 work has already begun so :)

[09:23:33] <virtualsex> how do i create a database?

[09:26:40] <newoob> I need a little help with MongoDB query

[09:44:43] <virtualsex> is there easier way to type those command

[09:44:49] <virtualsex> commands are too long

[09:47:44] <Derick> virtualsex: you create a database by inserting data into it

[10:01:55] <virtualsex> Derick, i create it and enable shard

[10:02:16] <virtualsex> but i don't know how to create user and use that newdb only

[10:02:30] <virtualsex> i'll import mysql db into it

[10:03:27] <Derick> I don't know that either, sorry.

[10:03:48] <virtualsex> is using shard and replica recommended?

[10:04:15] <Zelest> if you plan on using shards, you *should* run a replicaset

[10:04:23] <Derick> depends on what you are trying to do, how much traffic you hvae, and what your data load is

[10:04:45] <Derick> oh, and how much data you want to import

[10:05:07] <virtualsex> when i have a db with distributed on two shards, when one is down, what happen

[12:14:07] <alexi5_1> Hello

[12:14:49] <Zelest> hi

[12:19:40] <dump> hey

[12:47:41] <malorie> hi all! I'm trying to use mongodb for a small project, and was wondering, wether there's something like a convenient web-ui for data-entry? or should I write my own forms, etc.?

[12:59:37] <StephenLynx> using mongo's terminal client is pretty good.

[12:59:49] <StephenLynx> I wouldn't bother with a GUI for inserting development data.

[12:59:55] <StephenLynx> nor querying.

[13:00:15] <StephenLynx> actually, is not about the client, but the lack of requirements for inserting data.

[13:00:30] <StephenLynx> you don't have to create a table, you don't have to specify types, respect schemas.

[13:00:46] <StephenLynx> so its way more practical to use without a GUI than a RDB.

[13:04:50] <malorie> but don't schemas and types help with input? otherwise I'd have to specify the full object, each time I enter something, no?

[13:05:48] <StephenLynx> you can do that.

[13:05:50] <StephenLynx> but you are not forced.

[13:06:12] <malorie> and data entry on the terminal is via json, then?

[13:06:21] <StephenLynx> yes.

[13:06:34] <StephenLynx> db.yourcollection.insert({field:value})

[13:07:04] <malorie> and when you were to insert a bunch of data, you'd just automate?

[13:07:20] <StephenLynx> yeah, i'd just write a mock script for doing so.

[13:07:28] <malorie> ok, I see

[13:33:13] <deathanchor> malorie: are you generating random data or specific data? or are you migrating some data from another source?

[13:34:05] <malorie> I'm creating data, think dictionary entries. but I'm beginning to question whether nosql is the right choice for me

[13:34:38] <deathanchor> I forget does mongo3.2 have join functionality now? or is that still on the roadmap?

[13:35:28] <deathanchor> malorie: I always choose nosql, but that's because I always hated RDBs. My thought processes always worked with nosql architecture.

[13:35:59] <malorie> but what about many-many relations?

[13:36:03] <deathanchor> and I grew up learning DBase IV

[13:36:16] <malorie> afaiu nosql is suited for tree-like structures, right?

[13:36:45] <deathanchor> malorie: with nosql there are no relations, it's all about how your application would query what is needed, join/parse/process after it go the data.

[13:37:42] <deathanchor> many-many relationships are pain regardless of RDB or nosql.

[13:38:07] <StephenLynx> I usually lean towards mongodb. not that I hate rdbs, I just don't find all their features and constraints required.

[13:38:14] <StephenLynx> most of the time

[13:38:33] <deathanchor> StephenLynx: yeah coming from perl to nosql was a breeze.

[13:38:49] <deathanchor> StephenLynx: does mongo3.2 support some kind of join function yet?

[13:39:59] <StephenLynx> I think so

[13:40:10] <StephenLynx> but I heard its slow.

[13:40:22] <deathanchor> better to just do it application side?

[13:40:41] <malorie> well, I intend to serve only static content

[13:41:42] <StephenLynx> I don't really know.

[13:41:46] <StephenLynx> I haven't looked it up.

[13:42:16] <malorie> deathanchor: you're saying that there are no relations in nosql, then how do you relate things? you have to have some kind of keys present at least, I pressume

[13:42:45] <bethge> Hi, does anyone have advice on automatically restarting a crashed mongodb on an aws linux?

[13:43:19] <deathanchor> malorie: you make the relations however you want, you can use the _id (autogenrated always unique identifier) or you can do whatever you want

[13:43:19] <StephenLynx> malorie, you just create fields that allow you to relate stuff.

[13:43:28] <StephenLynx> but these relations are not validated on the database.

[13:43:38] <StephenLynx> as far as the server knows, they are just yet another field.

[13:43:49] <StephenLynx> unless, of course, you use the new schema validation feature.

[13:44:07] <StephenLynx> no, wait

[13:44:10] <StephenLynx> that's another thing.

[13:44:11] <deathanchor> StephenLynx: that's only validated on insert/update right?

[13:44:12] <StephenLynx> nvm

[13:44:48] <malorie> hm. and these _ids are autogenerated how?

[13:45:25] <deathanchor> malorie: by the DB, see _id in the docs

[13:45:41] <deathanchor> https://docs.mongodb.com/manual/reference/method/ObjectId/

[13:46:44] <deathanchor> malorie: I would highly suggest taking one of the free courses on mongodb university. like M102.

[13:48:38] <malorie> woah, those are thorough

[13:49:15] <Derick> deathanchor: the _ids are generated by the drivers though

[14:31:31] <deathanchor> really?

[14:31:35] <deathanchor> didn't know that

[14:31:56] <deathanchor> so in mongoshell that generates it, not the mongod on the server?

[14:32:00] <Derick> deathanchor: yup

[14:32:13] <Derick> to be fair, if the driver would not have generated it, mongod will

[17:00:23] <sinni800> is it normal that an idling mongodb server (even with 0 connections) would use 3% cpu all the time?

[17:01:23] <cheeser> i wouldn't be alarmed by that, no.

[17:01:31] <sinni800> well, no other daemon does that

[17:04:08] <sinni800> mysql has actual activity and only goes up to 0.7% on some activity

[17:04:13] <sinni800> and idles at 0

[17:20:55] <StephenLynx> I wouldn't be surprised if it has some background task.

[17:21:18] <StephenLynx> like the one that deletes documents automatically.

[17:22:22] <cheeser> and replica set/sharding maintenance threads

[17:23:52] <StephenLynx> mongo being a much more modern software than mysql, its only expected it is designed around more powerful hardware.

[17:24:06] <StephenLynx> mysql stays under 1% with today's servers.

[17:24:26] <StephenLynx> but I am sure it wouldn't stay like that with 90's hardware.

[17:25:52] <StephenLynx> besides, I have a hunch rdbs have to do much less CPU bound work due to the static format of data.

[17:34:51] <RoyK> StephenLynx: what do you use to monitor the cpu?

[17:35:03] <RoyK> sinni800: that was for you, sorry StephenLynx

[17:36:50] <sinni800> well that was just a htop

[17:37:23] <sinni800> StephenLynx: somehow that seems like excuses, I can see if I am using replicas and sharding that they will try to sync and maintain

[17:37:37] <sinni800> but software even today can be made efficient and not wasteful of resources

[17:38:36] <StephenLynx> one thing is being wasteful

[17:38:39] <StephenLynx> another is being cheap.

[17:38:56] <StephenLynx> one thing is burning a 100 bill and another is not jumping in traffic for a dime

[17:39:25] <sinni800> i'm just saying if the server is doing nothing, it should probably... do nothing

[17:39:35] <StephenLynx> and I am saying that it might be doing something.

[17:39:43] <sinni800> i would be asking what

[17:39:46] <StephenLynx> that results in a better result if it weren't.

[17:40:13] <StephenLynx> I gave you an example

[17:40:18] <StephenLynx> the task that deleted expired documents.

[17:40:26] <StephenLynx> and might be other stuff, I don't know mongo that well.

[17:41:22] <sinni800> my databases have 26mb dumped but 750 mb on disk... i can see that with the way mongodb saves data, no prob

[17:42:24] <sinni800> but what kind of maintenance does data that is not even touched for 98% of time need, and at that, 24h

[17:42:59] <StephenLynx> what

[17:43:14] <sinni800> i can disconnect every client, leave mongodb running for 24h and it still uses 3% of cpu

[17:43:24] <StephenLynx> so.

[17:43:40] <sinni800> it would probably be the same even if i removed all the data, but i haven't tested

[17:44:14] <StephenLynx> so

[18:54:45] <tinco> hi guys, one of our hdd's failed, and the resyncing process slows our database down so much our customers experience slowness

[18:54:56] <tinco> so we cancelled the sync, and now we're looking for alternatives

[18:55:16] <tinco> one is taking the database down, and performing it offline, which would be a disruption but a lot shorter than resynching

[18:55:18] <StephenLynx> weren't they supposed to be synchronized already?

[18:56:00] <daumie> Use SSD's tinco

[18:56:04] <tinco> well actually the database corrupted because a bug in 2.3.1

[18:56:22] <StephenLynx> isn't that a development version?

[18:56:29] <daumie> Had a back up set up?

[18:56:48] <tinco> anyway, so I had an idea that might do it without downtime, if we'd add a new shard, and disable the desynched shard

[18:56:58] <tinco> and then just wait for the desynched shards data to go stale

[18:57:17] <tinco> could that work somehow?

[18:58:27] <tinco> SSD's would've been nice, unfortunately this was provisioned when multi-terabyte SSD's were a little too expensive for us

[18:59:14] <tinco> can I stop data from going to a particular shard?

[19:02:39] <tinco> eh 3.2.1 I guess haha

[19:02:45] <tinco> I don't think it was a development version

[19:02:56] <tinco> yeah was 3.2.1

[19:03:10] <n1colas> Hello

[19:03:12] <StephenLynx> nvm then

[19:05:36] <tinco> so you don't know if this is possible? or doesn't it work at all?

[19:09:05] <tombin_> hello, i'm going through my first mongodb maintenance window and we need to change the hardware for one of the config server in our sharded cluster. do i need to do anything special to the cluster, or can i just take down the config server, replace the hardware and turn it back up and it will sync with the rest of the config cluster ?

[19:18:20] <StephenLynx> no, I don't have much experience managing deploys, just developing using mongo.

[19:18:33] <StephenLynx> I just thought it was really weird the version you said you were using.

[19:31:16] <AndrewYoung> tinco: Can you describe your environment a bit more?

[19:31:36] <AndrewYoung> You have a sharded cluster, with each shard being a replica set, and one server in one shard's replica set died?

[19:33:04] <AndrewYoung> tombin_: What version of MongoDB are you running?

[19:46:18] <tinco> AndrewYoung: yes exactly, so I'm thinking I'll add 2 new machines, we have them ready to go into the set anyway, to take on the write load, and then disable writes to the defunct shard

[19:46:54] <tinco> then we could simply wait until all the chunks on the defunct shard are empty

[19:46:58] <tinco> our database is append only

[19:47:41] <tinco> any data that's there will be stale and removed after about 4 weeks

[19:48:53] <tinco> I've been looking through the docs and couldn't find it exactly, there's the removeShard command, but it will attempt to move chunks, which I feel might bog the system down as well, not sure

[19:49:01] <tinco> it's a bit awkward that this data is on HDD's

[19:54:49] <AndrewYoung> You can also move data by hand to another shard.

[19:55:52] <AndrewYoung> But only one of the replica set members in that shard is defunct, right?

[19:56:54] <AndrewYoung> I don't think you need to remove the shard at all, you just need to bring up a new member of the replica set, then set that member to replicate from one of your secondaries. That way it won't negatively affect the performance of the primary that is servicing your customers.

[19:58:22] <AndrewYoung> Unless you're reading from secondaries. (Which would make sense if your data is write-only.

[20:00:11] <tinco> we have replicasets of only 2, so the only node that's active is a primary

[20:03:34] <AndrewYoung> Ah, interesting.

[20:03:44] <AndrewYoung> Yeah, that makes this more complicated.

[20:04:11] <AndrewYoung> If you're not writing anyway, you can fsynclock the primary and then copy the data directory.

[20:04:56] <AndrewYoung> https://docs.mongodb.com/manual/reference/method/db.fsyncLock/

[20:05:08] <AndrewYoung> While its locked it can still read but it won't update the files on disk.

[20:05:32] <AndrewYoung> Then you can copy the data directory over to the new box while mongod is shut down on the new box.

[20:05:47] <AndrewYoung> And once it's done you unlock the primary and fire up the new secondary.

[20:08:52] <AndrewYoung> I realize there's extra cost involved, but running with 3 replica set members helps alleviate this issue. For one, it allows for an election if one of the replica set members fails, and secondly it allows the replacement member to sync from the secondary.

[20:09:16] <AndrewYoung> Do you have an arbiter in your configuration?

[20:10:07] <tinco> yes

[20:10:39] <tinco> yeah we made the decision to go for 2-member replicas, knowing it might bite us, but thought it wouldn't be too bad

[20:11:18] <tinco> the write-only mode is interesting, that at least would allow us to keep our frontend up while the sync is happening

[20:11:57] <tinco> but we do have arbiters so the election problem is solved

[20:15:37] <AndrewYoung> That's good. (about the arbiters)

[20:16:08] <AndrewYoung> Yeah, my suggestion would be to freeze writes on the master and then make a copy of the data directory.

[20:16:18] <AndrewYoung> You only need the master to be frozen long enough to back up the data directory.

[20:16:30] <tinco> so there's no magic balancer sharding invocation that would prevents chunks from being allocated on a specific shard?

[20:16:43] <tinco> like, what if we could make it act like it's full? :P

[20:16:49] <AndrewYoung> The only thing I know of is shard tagging.

[20:17:19] <AndrewYoung> https://docs.mongodb.com/manual/tutorial/administer-shard-tags/

[20:17:33] <tinco> oh hmm

[20:17:55] <AndrewYoung> You could tag all the other shards with some tag like "ACTIVE" and you could make a shard key range that includes all your shard keys and tag that range as "ACTIVE".

[20:18:15] <AndrewYoung> Then new chunks would only go to the shards tagged "ACTIVE".

[20:19:07] <AndrewYoung> But, I think that will actively move data off the shard.

[20:19:21] <AndrewYoung> In which case you might as well just remove the shard.

[20:22:43] <tinco> oh but our balancer is disabled

[20:22:58] <tinco> so it wouldn't actually go move those chunks

[20:23:02] <tinco> I think this might be the solution

[20:26:25] <tinco> thanks AndrewYoung I'm going to work this plan out :D

[21:15:00] <tombin_> tinco: sorry for the late reply, i'm running 3.2

[21:26:48] <docmur> Is it possible to change commands? so something like db.col1.col2.drop()

[21:28:01] <docmur> *chain

[22:32:11] <cheeser> docmur: no

[23:20:29] <silasbarta_> So ... I'm on Mongodb 3.0. When I do db.currentOp() on admin, I get negative opIds like -1697108067. I understand that it may be an issue of it converting unsigned 32 bits to signed 32 bits ... but I'm still rapidly reaching 2^32 total operations. Should I be worried?

[23:20:54] <cheeser> probably not. unless they're all running at once.

[23:21:16] <cheeser> then i'd have some serious discussions with your devs. :)

[23:22:32] <silasbarta_> @cheeser Yes but if it exceeds 2^32 total ops, it will overflow, right? And then the server will thing that a later operation came before an earlier one?

[23:23:17] <cheeser> i don't think it uses those for anything but discrete IDs

[23:24:56] <silasbarta_> Hm, okay. But doesn't it have to keep a strict ordering for replica sets?

[23:25:40] <cheeser> no, that'd all get recorded in the oplog. i don't think the opId gets logged.

[23:27:10] <AvianFlu> I don't think replicas apply things serially in the first place

[23:27:14] <silasbarta_> Hm, okay, thanks. Had me worried. Doubly so that no one else seems to be posting questions about that :-/

[23:27:16] <AvianFlu> I think it gets applied in batches in parallel

[23:27:43] <cheeser> well, they get applied in the order things happened on the primary, sure.

[23:27:50] <cheeser> they'd have to be.

[23:29:59] <silasbarta_> @cheeser Well, thanks for the help. Or at least get ready for an I-told-you-so when the primary randomly crashes j/k

[23:30:48] <cheeser> heh

Log file Viewer

Help | Karma | Search:

#mongodb logs for Wednesday the 10th of August, 2016