Log file Viewer
#mongodb logs for Wednesday the 27th of January, 2016
[00:30:25] <Ricoshady> im trying to make a replica set, do I need to start the mongo process as a master in order to create a user? then convert to replica set?
[00:31:03] <Ricoshady> i had it set to master, but then I couldnt add the replica set, then changed it to a replica, then I couldnt create a user
[00:45:02] <GothAlice> Ricoshady: Have you read through https://docs.mongodb.org/v2.6/tutorial/deploy-replica-set-with-auth/ ?
[00:45:35] <GothAlice> Authentication needs to be configured first, with a shared key generated to let the secondaries communicate securely.
[00:47:03] <Ricoshady> can I assume if you don't use the --config option, that the only option mongo uses are from the command line?
[01:54:53] <rpdprd> according to https://github.com/mongodb/mongo/blob/master/src/mongo/bson/oid.h and my experimentation, the first 4 bytes with the timestamp is big endian as opposed to the little endian used everywhere else in bson. The source claims that the decision is because mongodb uses "memcmp to order OIDs". But what does that actually mean?
[01:55:29] <rpdprd> What if I (for shits and giggles) construct my own objectIds and put the timestamp in little endian? Would there be a performance penalty or something?
[01:56:13] <rpdprd> It says "we want to ensure an increasing order" but is that really important? What if I wasn't using ObjectId for _id but were using a string or something like that instead?
[01:58:17] <GothAlice> rpdprd: Because the memories are compared as strings, the order needs to be "natural" and not machine order. Like storing a filename prefixed with a date as 2016-01-20 instead of 01-20-2016 and expecting it to lexicographically sort.
[02:46:14] <Waheedi> I'm using mongo cxx driver 2.4 in my c++ app, I have configured query.readPref(mongo::ReadPreference_Nearest, BSONArray()); to read from the nearest node I believe its not picking up the nearest node the nearest node ping is time=0.210 ms but the node its picking ping time is time=1.32 ms
[02:47:44] <joannac> Waheedi: https://docs.mongodb.org/manual/core/read-preference-mechanics/#replica-set-read-preference-behavior-nearest
[04:14:22] <Waheedi> http://api.mongodb.org/csharp/current/html/P_MongoDB_Driver_MongoClientSettings_LocalThreshold.htm
[04:57:21] <Waheedi> candidate = _selectNode(nodes, tags->getCurrentTag(), false, localThresholdMillis,
[05:07:11] <Waheedi> I believe i need to use this function setLocalThresholdMillis with scoped_lock mutex
[07:08:36] <purezen> I am using mongo as my persistence.. and the object on which I need to call delete on will have 1-many relat. with another object which will again have 1-to-many with another object
[07:09:27] <m3t4lukas> Waheedi: it's generally your choice what you hardcode :P The localThresholdMillis tells mongodb up to what ping length a node is considered local. The drivers do not necessarily need to be equal in functionality.
[07:09:45] <purezen> How do I efficiently perform a delete on the top-most object such that, all objects belonging to it will also be deleted?
[07:10:41] <Waheedi> :) m3t4lukas I ended up telling the client his connecting to localhost:27017 and using haproxy to pass back to mongod
[07:11:05] <m3t4lukas> purezen: traverse the tree and start deleting at the leaves or change your model so you embed the documents the are dependent on what you want to delete.
[07:17:27] <purezen> @joannac: yeah.. that is what I thought initially.. but that would result in a lot of api calls.. I need a more efficient solution.. since for a situation where 1-to-many is like A->B.. to delete an A object.. I would first need to get all Bs under A.. then call cascading delete on Bs and then delete the A.. all using Api calls.. which would crunch my server!
[07:19:19] <purezen> @m3t4lukas: hmm.. the problem with the efficiency that i just wrote would still persist.. also changing the schema is not an option :P
[07:21:48] <m3t4lukas> purezen: you have those two options. You need to get away from thinking that calls are expensive. They are by far not as expensive as with SQL. Many API calls is quite normal for mongo to a certain degree.
[07:22:49] <m3t4lukas> purezen: API calls don't need to get interpreted first. They are pretty straightforward operations.
[07:24:05] <m3t4lukas> Derick, cheeser: are there any plans for the forseeable future for mongodb to be able to aggregate using OpenCL or CUDA?
[07:25:59] <m3t4lukas> Derick, cheeser: or at least parts of it. It may be needed to be able to save callable aggregations on the server first because of compute shader compilation efficiency and stuff.
[07:37:33] <Waheedi> m3t4lukas so what i was trying to do is that i have two clients on two different physical locations, talking to two(third is Arb) nodes in a replicaset that are also in two different physical locations.
[07:38:51] <Waheedi> right now I'm using nearest read preference and I'm seeing that its going to node1 70% of the time
[07:39:11] <m3t4lukas> Waheedi: yeah, that much I figured out. Just don't use localhost but the real DNS name of that machine or at least the machine name :P
[07:39:59] <m3t4lukas> Waheedi: nearest does not mean it connects to the nearest node in terms of distance but in terms of oing time
[07:40:33] <Waheedi> yeah ping time for the same physical machine is definitely less than the one somewhere else
[07:41:35] <m3t4lukas> Waheedi: on spikes you may have less ping on another machine or when your application introduces some heavy load on the system
[07:41:58] <m3t4lukas> Waheedi: it does not use the real ping. It uses server response time. Sorry for confusing those two
[07:42:42] <m3t4lukas> Waheedi: if mongo uses the other server for 30% you can assume that it is best for you that it does that
[07:43:41] <m3t4lukas> Waheedi: good ;) If you find out it's doing cr*p you should file a bug report on jira
[08:36:37] <Buzer> Hey. We are storing large numbers (e.g. timestamps in milliseconds) as integers in MongoDB. I noticed that when using mongoexport, those numbers are reprsented differently (e.g. 1446552221635 -> 1.446552221635e+12). Is this expected outcome?
[08:40:21] <joannac> Buzer: it should come out as extended json https://docs.mongodb.org/manual/reference/program/mongoexport/#considerations
[08:43:58] <Buzer> joannac: Thanks, that pointed to right direction. So plain numbers were actually treated as floats
[08:45:52] <m3t4lukas> when I use the java driver to store for example a double value of 2.0 there can be values like 1.999999999999994345 in the database :P Just noticed it. Out of curiosity, does anyone know how mongo stores Double?
[09:21:31] <m3t4lukas> and I was wrong. Depending on how you do it you'd not need to store aggregations of the server. It would be best to implement the stages themselves in CUDA/OpenCL and just pass config and data into it and let it do its thing.
[09:22:28] <m3t4lukas> The method I was thinking of before was to compile the aggregations directly into a shader (would be more runtime efficient if they are stored)
[10:35:54] <BlackPanx> does mongodb have any option for georeplicated setup? i'd like to have one mongo replicaset in US and one in EU. is this even possible with mongodb?
[10:40:53] <BlackPanx> the main problem is master and writes. i'd love to have each replicaset in each region write to it's own database which would be then replicated to anothe region or something
[10:43:18] <m3t4lukas> BlackPanx: there are some caveats and some things to be aware of when doing this
[10:45:19] <BlackPanx> if you look at this picture... there's a problem. only 1 master in DC1 which can do writes. I'd prefer having master in each DC so that all instances in that DC can write to it locally and then gets distributed to other DC's
[10:47:25] <m3t4lukas> BlackPanx: for every record add the country where you want it to be stored as string or coded integer or any way you like and use that as (part of) the sharding key
[10:51:13] <BlackPanx> https://docs.mongodb.org/v2.4/_images/sharded-cluster-scatter-gather-query.png something like this
[10:52:46] <BlackPanx> in this case shard A would be US datacenter replicaset of 3 instances, shard B would be EU replicaset of 3 instances and shard C would be Asia/Pacific replicaset of 3 instances. Then Driver would use tag-aware sharding which would basically make it use specific "shard" which is closest to it.
[10:53:16] <nixstt> BlackPanx: https://s3-us-west-1.amazonaws.com/morpheus-staging/system/spud_media/309/original/tagawaresharding2.jpg?1421507273
[10:54:14] <m3t4lukas> BlackPanx: yeeeaaahnaah, it doesn't really choose the nearest but the one you define
[11:11:45] <t0th_-_> hi, i have this problem when try connect in php: Remote server has closed the connection
[11:19:50] <julio> Hi! Which is the best way to install mongo-hacker at Debian 8 Jessie?. I'm trying "npm -g install mongo-hacker' but I'm receiving an permission error (I'm using sudo)
[12:27:02] <Freman> so, mongo, I want to find anything > $time and increase it's sequence_id by 100000
[12:29:33] <nixstt> Freman: https://docs.mongodb.org/manual/reference/method/db.collection.findAndModify/
[12:30:26] <Freman> db.collection.update({date: {$gt: ISODate("2016-01-27T05:00:00Z")}}, {$inc : {sequence_id: 100000}})?
[12:33:12] <Freman> db.collection.update({date: {$gt: ISODate("2016-01-27T05:00:00Z")}}, {$inc : {sequence_id: 100000}}, {multi: true})
[12:35:05] <StephenLynx> some drivers, at least the node one, have a separate function that includes the multi:true behind the scenes.
[12:35:27] <StephenLynx> in case you are also writing software that will require an update on multiple documents.
[12:38:13] <Freman> no not normally, just something went wrong, probably due to the huge backlog it didn't get around to updating the saved sequence_id so a restart meant it was writing from an old one
[12:39:54] <Freman> I'll make the app read the old sequence_id (from the one document collection it's in), find the greatest sequence_id after that, then add 10k
[12:49:35] <StephenLynx> yeah, mongo is pretty good, but when bad devs fuck up, they fuck up real good.
[12:55:03] <Freman> yeh and I'm so busy re-writing all the things (this used to be a nodejs process until it pegged the cpu at 4 mbit - for example)
[13:00:18] <m3t4lukas> julio: did you try to leave out the -g to see if it works when you don't install system wide?
[13:04:55] <julio> m3t4lukas, no. If I try without -g option, where I need to try. In my user folder?
[13:07:20] <m3t4lukas> doesn't matter where I think. As far as I know npm has a .npm directory in the user home
[13:19:02] <m3t4lukas> julio: how did npm do it? Does it create a folder in CWD or does it use a directory in your home?
[13:39:51] <hernanhht> hey guys, I want to change how the query planner decides over indexes. I have 2 indexes and sometimes it chooses wrong (we have different workloads on the collection). Besides using a $hint or index filters, is there some other type of fine tuning ?
[13:47:37] <cheeser> i think what's probably tripping you up is that I believe the query planner caches that plan so it doesn't have to reevaluate all the time. so if you hit it with two different needs before that cached plan is evicted you get the old one.
[13:49:51] <hernanhht> yes, that's what is happening. But I would like to fine tune this kind of behaviour
[13:50:57] <hernanhht> it happens usually with compound indexes with 3 fields for instance {name:1, lastname:1, age:1} and {name:1, lastname:1, address:1}
[13:51:27] <hernanhht> sometimes I make a query that matches exactly one of the indexes and the query planner chooses the other one
[13:52:03] <hernanhht> I know I can hint the query planner, but I don't like this solutions, it doesn't scale
[13:52:51] <hernanhht> I would like a way to tell the query planner, if there is an index that matches the query, uses it right away
[13:54:31] <hernanhht> yes, but it's scanning a lot of objects because it's choosing the wrong index
[13:58:28] <cheeser> i don't know of any beside $hint. you might ask the mongodb-users google group. more of the kernel devs monitor that.
[16:25:08] <alexi5> how practical is it to use mongodb to develop a catalog for various types of equipment with command and diffren attributes. store reports on the maintenance of these pieces of equipment. and have the database and application to be easilty modified when new equipment types are available
[16:26:29] <alexi5> most of the use cases I have come across involve the performance improvements of using mongodb
[16:27:22] <mkjgore> hey folks, so I'm trying to run an update from CLI… mongo —eval "use 'config'; db.shards.find().pretty()" is a no go because "use" is a shell command… any other way to do this? Internet is failing me atm
[16:32:47] <mkjgore> Derick: I'm not seeing the right results… I can definitely see the proper output in the shell (proof that it's there) but not on the CLI as described ¯\_(ツ)_/¯
[16:33:37] <Derick> that's something else I'm sure, but "mongo config" starts the mongoshell with the config database selected
[16:38:29] <alexi5> do you pefform any refrences between documents or is everything self contained ?
[16:41:03] <alexi5> I am developing an application that will manage network equipment where the diffrent types of equipment of the same type can have common and varying attributes
[16:42:12] <alexi5> also want the technician to fillout their maitnenance report and the content of the maintenance report may varying depending on the equipment type and their attributes
[16:44:54] <alexi5> my intial of the schema resulted in 6 six tables and I could see it growing depending on the type of attributes
[16:52:39] <mkjgore> alexi, I know time is tough and learning new tech equally so but if you're talking about some major infrastructure you should definitely look into salt and it's contemporaries
[17:01:48] <alexi5> The equipment that I would be managing is Cell site (LTE,GSM) equipment . so far salt stack looks like it manages data center equipment and it operations
[17:03:42] <GothAlice> alexi5: I use an Object Document Mapper (ODM) that supports subclassing. If, for example, SomeTypeOfEquipment has certain properties, but AnotherTypeOfEquipment has others, YetAnotherType might be "a type of" SomeTypeOfEquipment, inheriting its attributes and adding more or mutating the existing ones.
[17:04:46] <GothAlice> (The net result is there is a field on most of my documents to identify what type of document it is.)
[17:05:58] <GothAlice> Pretty much, except in MongoDB there is no common schema requirement between documents in the same collection, making the whole process infinitely easier.
[17:06:09] <GothAlice> Esp. if you want to search attributes common to several types at the same time.
[17:07:16] <GothAlice> https://gist.github.com/amcgregor/c6ae92325a941ba70568?ts=4 is an XML dump example of this, with Page being a type of Asset, and all of the blocks being a type of basic Block.
[17:10:28] <GothAlice> There can be _some_, but it's usually a fire and forget update in a mongo shell. ;)
[17:11:42] <GothAlice> And most ODMs let you divorce native attribute name from in-db field name, so $renames can be avoided in most circumstances, but there are others, like pivoting a badly constructed sub-document of metadata attributes to a list of sub-document-per-metadata-attribute style, that takes a touch more work.
[17:13:40] <alexi5> the application will need to be able to make queries e.g what are all pieces of equipment that have attribute A with value 112
[17:14:54] <GothAlice> Or if A is a string on some of them, that's okay, too. (They won't match, but it's not explosion-worthy.)
[17:16:24] <GothAlice> db.fobs.insert({A: 112}); db.fobs.insert({B: "artoo"}); db.fobs.insert({A: 42, B: 27}}
[17:16:53] <GothAlice> With that you can query all sorts of ways. There are three "types" of things, the latter combining the attributes of the other two, but with a differing type for B.
[17:17:12] <GothAlice> db.fobs.find({A: {$exists: 1}) – find everything with a value for A, any value for A.
[17:17:34] <GothAlice> db.fobs.find({B: {$type: "string"}}) – find records whose B value is a string (only the second record inserted in this example)
[17:20:50] <GothAlice> It's mostly simple at a glance. It's insanely powerful, too. For example, using the capped collection feature I use MongoDB as a distributed message queue and RPC system. Also as a logging target. Also as an atomically updated monitoring system. Also … ;)
[18:21:40] <euxneks> howdy, I'm interested with regards to this: https://docs.mongodb.org/manual/mongo/#exit-the-shell why the decision was ctrl+c instead of ctrl+d ... (I mean, I'm not complaining, I'm just curious - perhaps it will help me understand mongo better)
[18:32:22] <GothAlice> euxneks: Control+C sends a SIGINT (interrupt) signal to stop whatever the foreground process is doing, generally by quitting. (A la running "tail -f".) Control+D signals "end of file" (literally by writing an EOF character) which to an interactive shell means exit, the (interactively entered) script is over. (cat > /dev/null, ^D to end transcription for a non-mongo example).
[18:33:15] <GothAlice> There is no particular meaning meant by mongo beyond "follow standard POSIX practice".
[18:35:36] <GothAlice> (Interactive REPL shells for programming languages tend to capture SIGINT in some way, a la Python's shell transforming it into a KeyboardInterrupt exception.
[18:41:13] <mkjgore> Derick: FYI, the issue wasn't that it wasnt returning the data, I just didn't realize it was returning a cursor so I had to surround the "db.shards…()" with "printjson(…)"
[19:13:20] <GothAlice> Hmm. Selecting multiple embedded documents in an array with a filter. My eternal problem.
[19:16:32] <pancakes9> Hi, do you offer help with third party packages like https://www.npmjs.com/package/mongo-express?
[19:28:37] <pancakes9> I guess this is more of a sysadmin question but whenever I start it keeps throwing the error "Error: listen EACCES 127.0.0.1:80"
[22:09:56] <GothAlice> Freman: Might be a good idea to check iotop to see if your disks/filesystem is capable of keeping up.
[22:10:50] <GothAlice> (Would exhibit as an absurdly high load average when it gets particularly bad.)
[22:12:59] <GothAlice> Hmm; I might be wrong about the LA thing. Linux kernel devs might have fixed the "io backlog = infinite LA" issue.
[22:14:12] <Freman> eventually whatever is causing the backlog goes away and our backlog empties into mongo at quite the rate (our bulks are limited to 1 (up to 4 concurrent) x up to 1000 documents every 1 second (which ever comes first, 1000 documents, or 1 second)
[22:16:05] <BigOrangeSU> Does anyone have a strategy for running an update with a forEach but only for 50,000k records?
[22:16:52] <Freman> disk itself is plenty fast, cirtainly faster than the 200kb/s the input steram maxes out at
[22:17:35] <GothAlice> Freman: That's a pretty good batching strategy, though one second seems a bit tight to me. ;)
[22:18:29] <BigOrangeSU> cheeser: How can i write a query for hte specific 50k ids? Do I use an $in : [] with 50k records
[22:18:30] <Freman> tho today I'm going to propose they subscribe to the log processor directly to get the logs vs pull them from mongo post insert
[22:19:09] <GothAlice> Freman: Heh. Our atomically updated analytics are "live" (no count-based or time-based batching), with hits over a capped collection for realtime tools to get more detailed information. (Tailing capped collections as a data processing pipeline we find to be quite effective.)
[22:19:35] <BigOrangeSU> cheeser: there is no way within mongo to determine those 50k without iterating over evry record which would be inefficent
[22:22:15] <GothAlice> BigOrangeSU: That's a strategy we use at work for bulk processing updates against things like our invoices. (Though we spawn a distributed task per grouping to let us scale the parallelism of that separately.)
[22:27:36] <GothAlice> BigOrangeSU: It sounds, though, that whatever you're using as selection criteria for those 50K IDs should instead be merged into the dataset you are actually querying; then you wouldn't need to $in anything.
[22:28:49] <GothAlice> (MongoDB encourages one to model your data the way it's used, rather than in some pristine "perfect at-rest structure".)
[22:29:11] <echelon> exception:BSON representation of supplied JSON array is too large: code FailedToParse: FailedToParse: Bad characters in value: offset:361
[23:27:11] <gardenmwm> Hi, I'm *really* hoping this is a stupid question, but I'm running into a problem where I can see several hundred files via the mongo api's (pymongo and some java mongo) but when I connect with the mongo client, I only see a few files
[23:28:35] <gardenmwm> I'm using gridfs, using the gridfs api drivers I can see everything fine. However I need to delete a few files, so I'm attempting to do it via the mongo shell
[23:29:07] <gardenmwm> none of the file id's I am trying to delete are in the fs.files collection, only about 20 or so files exist.
[23:33:18] <zeioth> Hi, I have a model like (_id, date) where index is (_id + date). If I filter by date, i need to create another index for the date field, or the (_id + date) index will be used?
[23:35:56] <gardenmwm> @StephenLynx ok, so I'm an idiot. Didn't realize mongo paginates output. Thanks for helping