pmxbot IRC Log Viewer

[00:28:18] <starseed> Is there a way to determine if a specific cursor id is still valid?

[00:29:17] <starseed> pymongo or mongo shell would both work, but I can't find a method that allows me to ask a MongoD if cursor_id xxxxxxxxxx is still alive

[00:31:49] <starseed> unfortunately I have notimeout cursors to deal with and am not in a position to change the application code which generates them, so I've written some code to clean them up - but it would be more efficient if I could ask mongo if each cursor I'm vetting still exists before parsing logs and issuing a pymongo.MongoClient.close_cursor(id)

[00:34:03] <cheeser> i don't think there is.

[00:34:14] <cheeser> db.currentOp() maybe?

[00:38:45] <starseed> currentOp doesn't seem to use cursor IDs

[00:38:55] <starseed> so I'm not sure I could cross reference

[00:39:34] <cheeser> yeah. i was looking over it, too. i'm not sure those values are exposed.

[00:41:31] <starseed> it isn't the end of the world I guess. The use case is I get a list of cursor IDs from log output which are blocking chunk migrations. I then search logs for each ID and determine when that cursor ID first appeared in logs (determine its age). If it has existed longer than x amount of time (our longest running jobs are like 20 hours)...then I kill it

[00:41:52] <starseed> but it would have been nice to find out if a cursor was still active before searching two weeks worth of compressed log output

[00:42:46] <starseed> but I think I can blindly issue a close_cursor(id) against each ID that is >24 hours old, worst that can happen is that it can't find the cursor to kill because it no longer exists.

[08:57:51] <dnmngs> Hi folks. Is there separate downloads for MongoDB community and enterprise edition? Or is it possible to just update the community edition to enterprise by using a license?

[08:58:33] <dnmngs> ah I see, it seems to be a different build

[10:35:29] <basil_kurian_> I 'm using an invalid ssl cert on my replica set members. I have allowInvalidCertificates: true in my mongo config file. It is working fine on mongo 3.0.12 . But when I upgrade to mongo 3.2.6 , it is giving me error saying that invalid ssl cert

[10:36:17] <basil_kurian_> when I try rs.status() , i can see a message ""lastHeartbeatMessage" : "could not find member to sync from""

[10:36:38] <basil_kurian_> any idea what is the issue here ?

[10:37:29] <tinco> can you read from the loaded config somewhere if it picked the value up?

[10:39:03] <basil_kurian_> @tinco using db.serverStatus() ?

[10:39:56] <basil_kurian_> how can I see the loaded config ?

[10:40:52] <basil_kurian_> @tinco tried db.runCommand({getCmdLineOpts:1})

[10:40:57] <basil_kurian_> i can see the options

[10:41:09] <basil_kurian_> i can see the correct values

[10:42:02] <tinco> weird

[10:42:37] <tinco> I don't know, I've never used mongo with ssl

[10:43:26] <tinco> if it doesn't work in 3.2.8, perhaps file a bug

[10:43:41] <basil_kurian_> @tinco ok

[10:48:16] <ultrav1olet> Hi. Perhaps I'm asking a stupid questions but we'd like one server to host two shards. Do I have to have two mongodb instances on this server with different configuration files and TCP ports, or two shards can be be created within the same mongodb process?

[10:51:18] <Derick> you need two mongod with confif files and TCP ports

[10:51:29] <ultrav1olet> Derick, thanks!

[10:51:33] <Derick> but you also really don't want to have them on one machine - they will fight for memory

[10:51:59] <ultrav1olet> We've got 32GB or RAM - I guess that'll be enough for us

[10:52:13] <basil_kurian_> @tinco if I downgrade it is 3.2.4, there are no ssl errors

[10:52:22] <Derick> ultrav1olet: what is the reason why you want to do this?

[10:52:35] <basil_kurian_> But I 'm seeing another error when I try rs.status() " "lastHeartbeatMessage" : "could not find member to sync from","

[10:52:36] <ultrav1olet> to save money obviously

[10:52:56] <Derick> ? why do you want two "shards"?

[10:53:42] <ultrav1olet> We want to have two shards and two replicas on two servers, and an extra server for mongos, config and arbiter - a very tough configuration, but it will allows us to grow horizontally in the future

[10:54:15] <Derick> you still didn't answer why you want two shards

[10:56:14] <ultrav1olet> At the moment we have a replica set across three servers - all based on plain old HDDs - however they are not capable of withstanding our load (we store roughly 100MB of data every few seconds and release roughly 20MB - all randomly)

[10:56:42] <ultrav1olet> Our IO load hovers around 100%

[10:57:26] <ultrav1olet> In fact we omit 95% of data because if we dump everything, then our servers stall completely

[10:58:18] <ultrav1olet> So we want to migrate to SSD based servers and implement sharding so we'll be able to grow even beyond SSD disk bandwidth

[10:58:24] <Derick> having two shards on one box isn't going to fix that IO load

[10:58:36] <Derick> especially when they use the same drive...

[10:59:52] <ultrav1olet> SSD disks will momentarily alleviate our problems, and then, when we add new shards, we'll be able to withstand even a higher load

[11:01:14] <Derick> I will stay with this advice:

[11:01:22] <Derick> 11:42 <@Derick> but you also really don't want to have them on one machine - they will fight for memory

[11:03:21] <ultrav1olet> Derick, is there a way to specify the maximum amount of RAM mongo can occupy?

[11:04:12] <Derick> no

[11:04:47] <Derick> although using the WiredTiger storage engine should make that a bit better

[11:08:14] <ultrav1olet> Actually I'd love to talk about one problem we faced with mongo 3.2.5 (all 3.x.x releases are affected): after we switched to the WiredTiger engine with compression (and without compression as well) indexing of one of our fields started to take over 2,5 hours instead of 2 minutes using mmapv1

[11:09:13] <ultrav1olet> at the same time all reads and writes were severely affected - more likely dead as each query never completed during indexing

[11:09:22] <ultrav1olet> Is this a known bug?

[11:11:11] <ultrav1olet> I don't know how to file this bug because our data and workload are confidential

[11:15:00] <ultrav1olet> Derick: ping

[11:30:43] <jokke> hello

[11:30:57] <jokke> i'm having trouble with a java app that's using mongodb

[11:31:22] <jokke> i pass it a mongo uri and it uses MongoClientURL

[11:31:38] <jokke> the mongo uri contains a username and password

[11:31:47] <jokke> which should be correct

[11:32:02] <jokke> but the connection fails with "auth failed"

[13:29:57] <jokke> anyone?

[16:44:59] <teprrr> hi, I'm doing a find with $lt & time to select the elements not updated in, e.g., the past hour, and if not, updating and setting that field. my question is, will $lt match for non-existing field?

[16:46:42] <teprrr> solution is apparently using $or and $exist :)

[16:47:57] <StephenLynx> yes.

[17:08:03] <kenalex> from the mongoworld presentations mongodb 3.4 looks it will have some great new features

[17:08:31] <Derick> what's your favourite?

[17:16:23] <tom> In my Python app (using pymongo), I'd like to send multiple find_and_modify commands at once and then receive all the responses without wasting round trips in between every find_and_modify command. Is this possible? Would I need multiple sockets or could I write all requests at once into the socket? How would I implement it?

[18:37:16] <teprrr> tom: does find_and_modify do something else than just doing an update?

[18:37:22] <teprrr> because you can filter your updates..

[19:09:34] <shortdudey123> anyone know if the background option can be added to an index w/o reindexing? https://docs.mongodb.com/manual/tutorial/manage-indexes/ only mentioned TTL

[19:12:53] <Derick> shortdudey123: background indexing is only for the initial index creation

[19:13:03] <Derick> so it makes no sense to add it when an index is already existing

[19:13:40] <shortdudey123> hmm did not realize that

[19:14:26] <shortdudey123> new docs are always index in the background or the foreground after the initial index creation?

[19:15:03] <shortdudey123> new docs always*

[19:42:58] <brunoais> Hi. I found in 2-year old and older answers that MongoDB does not support asking for the attributes of a document. Does that still hold true?

[19:43:40] <brunoais> If it does, what prevents such feature from being made? (I'm good with technical stuff so please do tell or point/link me to it)

[19:44:55] <brunoais> I've checked on a schema that seems to be perfect to what I want but, out of the ~15 operations I want to do on those data, 2 seem not possible to do

[19:45:51] <StephenLynx> you can ask for it.

[19:45:54] <StephenLynx> $exists

[19:46:26] <brunoais> I can't guess them

[19:46:58] <StephenLynx> hm?

[19:47:07] <brunoais> They relate to a data organization that uses a triple leveled "Map" relationship

[19:47:16] <StephenLynx> ok

[19:47:34] <StephenLynx> what exactly is the operation you have to perform?

[19:47:35] <brunoais> a.set1.[something].set2.[something2].set3 <-

[19:47:42] <brunoais> E.g.

[19:47:52] <StephenLynx> dot notation.

[19:47:57] <brunoais> I have: a.set1.[something].set2

[19:48:03] <StephenLynx> "subdocument.field"

[19:48:06] <brunoais> Now I want to know all [something2] that exist

[19:48:15] <StephenLynx> $exists

[19:48:23] <StephenLynx> with dot notation

[19:48:25] <brunoais> checking again

[19:48:41] <StephenLynx> aside from that

[19:48:50] <StephenLynx> nesting subdocuments that need is really bad.

[19:48:55] <StephenLynx> and limits what you can do.

[19:49:02] <StephenLynx> that deep*

[19:49:16] <brunoais> I've assessed a lot of it

[19:49:25] <brunoais> not indexable <- Expected and not needed

[19:49:42] <StephenLynx> what

[19:49:58] <brunoais> I mean, I checked a lot of the limitations MongoDB has when using such structure

[19:50:07] <brunoais> including that those fields cannot be indexed

[19:50:10] <StephenLynx> they can.

[19:50:21] <StephenLynx> you can put index on fields of sub-documents.

[19:50:32] <brunoais> Only if I know their names ;)

[19:50:41] <brunoais> * their full names

[19:50:49] <StephenLynx> eh

[19:50:53] <brunoais> In this organization I can't

[19:50:59] <StephenLynx> do you have a dynamic model?

[19:51:00] <brunoais> wait

[19:51:07] <StephenLynx> that you simply cannot document?

[19:51:11] <brunoais> yes

[19:51:15] <StephenLynx> start over.

[19:51:22] <brunoais> I have 2 schemas

[19:51:35] <StephenLynx> a dynamic schema is not a schema at all.

[19:51:38] <StephenLynx> is a bunch of random data.

[19:52:08] <StephenLynx> unless some other poor sod is going to maintain that

[19:52:20] <StephenLynx> I strongly suggest you don't go with a dynamic schema.

[19:52:29] <brunoais> Hum...

[19:52:35] <brunoais> The alternative ain't good too

[19:52:46] <StephenLynx> multiple collections.

[19:52:52] <brunoais> I need the concept of "set" and $addToSet doesn't do the trick :S

[19:53:11] <brunoais> The concept of "set" must be enforced no matter what

[19:53:13] <StephenLynx> then make lists of ids to relate the documents.

[19:53:19] <StephenLynx> no problem

[19:53:24] <StephenLynx> just use it on an array of ids.

[19:53:34] <brunoais> I'm ordered to use a single collection of the data <_>

[19:53:43] <brunoais> *for the data

[19:53:43] <StephenLynx> retarded manager?

[19:53:49] <brunoais> no

[19:53:54] <brunoais> Atomicity is required

[19:54:08] <StephenLynx> a single collection doesn't assure atomicity.

[19:54:20] <brunoais> It does, according to the manual

[19:54:40] <StephenLynx> not if you have to perform multiple operations on documents.

[19:54:43] <brunoais> In MongoDB all write operations are atomic.

[19:54:53] <brunoais> I won't

[19:55:05] <StephenLynx> you are still better implementing transactions on your own.

[19:55:15] <brunoais> Each intention to write will only consume 1 write operation on both schemas

[19:55:22] <StephenLynx> dynamic schemas are literally the worst thing you can do.

[19:55:36] <StephenLynx> if transactions are so important, you might as well use a RDB

[19:55:36] <brunoais> Hum...

[19:55:55] <StephenLynx> ANYTHING is better than a dynamic schema

[19:56:07] <Derick> "if you can't document your keys, you're doing it wrong"

[19:56:27] <brunoais> I can document them

[19:56:30] <brunoais> I know what they mean

[19:56:36] <Derick> you just said they were dynamic?

[19:56:38] <brunoais> they correlate to the hierarchy

[19:56:44] <brunoais> As in a Map

[19:56:45] <Derick> 20:38 <brunoais> a.set1.[something].set2.[something2].set3 <-

[19:56:54] <Derick> you said you can't know what something or something2 is

[19:56:58] <Derick> these are keys

[19:57:10] <brunoais> I know their meaning but I don't know their string content

[19:57:18] <brunoais> so I can document them

[19:57:21] <Derick> example please

[19:57:25] <brunoais> OK

[19:57:25] <Derick> (of a document)

[19:58:02] <StephenLynx> also, the indexing is a more serious issue than you think.

[19:58:23] <StephenLynx> if you can't index, your db is scheduled to become unusable due to performance.

[19:58:33] <StephenLynx> unless you throw away data constantly.

[20:03:27] <brunoais> Derick, http://pastebin.com/cBp0DyLg

[20:03:59] <brunoais> StephenLynx, By the design of the program, there's only the need to index the "keys" of the outermost document

[20:04:25] <StephenLynx> inb4 featured on TDWFT

[20:06:55] <brunoais> Derick, I need to have dinner now, can you wait for my answer then?

[20:08:22] <brunoais> brb

[20:46:49] <brunoais> Derick?

[20:47:13] <StephenLynx> kek

[20:51:53] <brunoais> StephenLynx, http://pastebin.com/cBp0DyLg <- 2 scheming for the collection

[20:52:41] <brunoais> StephenLynx, How do I ensure that elements of an array are considered sets according to a certain parameter of the document in it?

[20:52:42] <StephenLynx> that tells me nothing.

[20:52:51] <StephenLynx> I don't now if your fields are actually fields

[20:52:55] <StephenLynx> or if they are dynamic.

[20:53:38] <brunoais> Only the ones that are immediate values of the params with "setsOfTypeX" (where X is a digit) are dynamic. All the rest is stable and static

[20:54:06] <StephenLynx> both are dynamic.

[20:54:13] <StephenLynx> I really can't see any difference.

[20:54:20] <StephenLynx> you are screwed either way

[20:54:53] <brunoais> The schema 2 has all documents well defined

[20:55:05] <brunoais> The variability is only in the number of elements in the array

[20:55:50] <StephenLynx> then why is the name setOf

[20:56:40] <brunoais> That's how I decided to name it in the example. Also because it must to be worked in a "set" fashion

[20:56:51] <StephenLynx> then your first information was wrong

[20:56:58] <StephenLynx> >Only the ones that are immediate values of the params with "setsOfTypeX" (where X is a digit) are dynamic.

[20:57:02] <brunoais> no two objects in an array can have, in this case, the same name

[20:57:10] <brunoais> That is for schema 1

[20:57:13] <brunoais> (sorry about that)

[20:57:22] <StephenLynx> that's doable.

[20:57:24] <brunoais> both schemas have the same info

[20:57:44] <StephenLynx> but i'd still separate into multiple collections instead of having that much nesting.

[20:57:54] <brunoais> That was my idea before too

[20:57:58] <brunoais> but I was told "no"

[20:58:02] <StephenLynx> dude

[20:58:07] <StephenLynx> you need more time in the industry.

[20:58:12] <StephenLynx> you don't tell how you are doing it.

[20:58:19] <StephenLynx> you ask what has to be done

[20:58:21] <StephenLynx> you deliver.

[20:58:25] <StephenLynx> and that's it.

[20:58:35] <brunoais> The one that rejected is the senior programmer -_-

[20:58:41] <StephenLynx> as I was saying

[20:58:41] <brunoais> He's also writing code for it

[20:59:03] <StephenLynx> no matter what you come up some dumb fuck will cut it down for one reason or another

[20:59:11] <StephenLynx> other programmers, managers, executes.

[20:59:14] <StephenLynx> executives*

[20:59:39] <StephenLynx> now you can't do that because he already knows you wanted to dothat.

[20:59:59] <brunoais> I had proposed 1 collection for each one

[21:00:03] <brunoais> it was rejected

[21:00:03] <StephenLynx> you don't propose.

[21:00:07] <StephenLynx> you just do it.

[21:00:14] <brunoais> I can't

[21:00:18] <StephenLynx> and if someone else doesn't like it he can change it himself.

[21:00:20] <brunoais> I was ordered to propose only

[21:00:23] <StephenLynx> welp

[21:00:24] <StephenLynx> RIP

[21:00:33] <StephenLynx> well

[21:00:37] <StephenLynx> yeah, NVM

[21:00:45] <brunoais> T.T

[21:01:00] <brunoais> double unwinds doesn't seem healthy too...

[21:01:11] <brunoais> specially for a very recurring query

[21:01:38] <brunoais> *quite recurring

[21:02:20] <StephenLynx> yeah

[21:02:34] <StephenLynx> just so let you know, that project is heading an iceberg

[21:02:46] <StephenLynx> because of dogmatic stupidity

[21:03:02] <StephenLynx> the least you can do is brace yourself.

[21:03:08] <brunoais> :S

[21:20:38] <tom> teprrr: not sure what you mean by "filter your updates". my findAndModify does an update and returns the updated document, which is important since I need the exact document back for audit purposes.

[21:33:59] <teprrr> tom: well, it's been a day or two since you started asking these questions, and you still haven't defined really what you want.. makes it really hard to answer without knowing that..

[21:34:22] <teprrr> tom: but okay. can't you just update them & use find afterwards? or should it be atomical?

[21:34:41] <teprrr> s/atomical/atomic/

[21:36:25] <teprrr> probably yes, if it's for auditing.. oh well, can't really help, but maybe someone else can now as your reqs are known :)

[22:53:48] <tom> teprrr: sorry, I just haven't properly set up notifications on my IRC client. it needs to be atomic for auditing purposes

[22:54:34] <tom> maybe to rephrase, could I issue multiple findAndModify requests over the wire at once, and then wait for all the responses to come back, or does it have to be one by one?

[22:58:06] <teprrr> tom: I'm just a beginner working on his first mongodb project, but I'd assume that if there's no info available in docs that's not so easily doable. the problem being that for atomicity all the nodes in cluster needs to be synced in time of the query => needs support from the backend

[22:58:17] <teprrr> so doing it on the wire won't help. but that's just complete guesswork...

[22:58:30] <teprrr> I hope someone knowledgeable would chime in, I'd be interested also in the answer

[22:58:56] <AvianFlu> tom, you can send all the requests you want to, but mongo has no concept of a transaction

[22:59:02] <AvianFlu> so there's no multi-doc atomicity

[22:59:04] <AvianFlu> only single-doc

[23:09:04] <tom> AvianFlu: thanks, I'm aware of that. just wondering if it's technically possible to bundle a few queries and save on round trips

[23:11:01] <tom> now the question is what's the easiest way to do this with the python driver?

[23:13:33] <AvianFlu> oh, I see what you're asking

[23:13:40] <AvianFlu> I'm not sure, I'd assume it makes separate requests

[23:14:02] <AvianFlu> some logging or something like wireshark could tell you for sure, I'd imagine :)

Log file Viewer

Help | Karma | Search:

#mongodb logs for Monday the 1st of August, 2016