Log file Viewer
#mongodb logs for Wednesday the 22nd of June, 2016
[01:44:24] <madprogrammer> Hey guys. Given a document structure something like: {x: [{a: 0}, {a: 1}]}, how could I select documents that had a {a: 0} in the array?
[01:47:33] <GothAlice> When x is an array, queries against x compare against the values in the array (i.e. a query of {"x": 27} against a document like {"x": [1,2,27]} will match) and attribute notation will match against any embedded document in the array that has the requested field.
[01:49:15] <GothAlice> There's a caveat: if you want to search for two values, {"x.a": 0, "x.b": 0} means something very different than most expect: that'll find documents with any x element whose a == 0 and contain any x element whose b == 0, not necessarily the same element. For that, use $elemMatch to wrap the embedded document portion of the query.
[03:10:18] <Lonesoldier728> So I have a document with the field releaseDate originally stored as a Date type in mongodb - how do I now change it to a String --- I tried by just using $set on the field with the new string but nothing happens
[03:23:58] <Mattx> Hey there! Quick question. How stupid it is to have collection like this? { id: 1, name: "mycol1", children: [{foo: 1, bar: "yada 1"}, {foo: 2, bar: "yada 2"}] }
[03:25:16] <Mattx> or should I use different collections, one for the main object, and another one for child objects?
[03:26:54] <Boomtime> @Lonesoldier728: what you tried will work fine if done properly - can you gist/pastebin what you tried?
[03:27:03] <Lonesoldier728> Mattx if you can give me the real example I can understand what you are looking to achieve it all depends
[03:27:13] <Lonesoldier728> @Boomtime http://stackoverflow.com/questions/37957860/how-to-change-data-type-of-fields-that-exist-in-mongodb
[03:28:43] <Lonesoldier728> Mattx but if you are trying to index a nested field in an array it is better to create a separate collection
[03:30:01] <Mattx> here is an example. collection "nasdaq" containing this [{ id: 1, name: "AAPL", orders: [{id: 1, price: 1.00, type: "buy"}, {id: 2, price: 5.00, type: "sell"}] }
[03:31:29] <Mattx> I guess the other option is collection "nasdaq" containing this [{ id: 1, name: "AAPL" }], collection "orders" containing [orders: [{id: 1, exchange: "nasdaq", price: 1.00, type: "buy"}, {id: 2, exchange: "nasdaq", price: 5.00, type: "sell"}]
[03:31:42] <Boomtime> @Lonesoldier728: i will assume you're using the Node driver - but what you've posted proves nothing; 'stringDate' could easily be a date object
[03:32:23] <Mattx> Lonesoldier728, the ids are going to be unique inside nasdaq, but they can repeat on other exchanges
[03:32:39] <Boomtime> before the Project.update line add this: console.log( 'stringDate type: ' + typeof stringDate )
[03:34:59] <Mattx> Lonesoldier728, what do you think? What would be a good data model for this data?
[03:35:57] <Mattx> if I use the original design (all in one collection) I'm going to have not just orders but many other arrays within the parent object
[03:37:15] <Lonesoldier728> I guess it is hard for me to understand what collections you want or how they pertain to one another but I would avoid arrays of other objects if possible
[03:39:19] <Lonesoldier728> then turn the array into a collection of its own and connect it with the other collection either through means of a name type etc
[03:41:53] <Mattx> collection stocks: [{exchange: "nasdaq", symbol: "appl"}, {exchange: "nyse", symbol: "gogl"}]
[03:41:54] <Mattx> collection orders: [{exchange: "nasdaq", stock: "appl", price: 1}, {exchange: "nasdaq", stock: "appl", price: 5}]
[03:45:33] <Mattx> the only problem is that data from different exchanges have different format/schema
[03:46:46] <Mattx> for instance in the case of the orders, if the exchange is nasdaq, there will be bid, quantity, etc. if it's nyse, there will be qty instead of quantity, etc
[03:48:18] <Mattx> that's why I thought it would be a good idea to have one collection per exchange. so all data inside that collection has the same "format"
[03:49:55] <Boomtime> @Lonesoldier728: you also aren't checking the return value - how do you know the match works?
[03:50:02] <Boomtime> http://mongodb.github.io/node-mongodb-native/2.0/api/Collection.html#~updateWriteOpResult
[03:52:56] <Boomtime> the result of the update - the code you've posted checks for an error only - ain't no error if the update statement doesn't match anything
[03:53:50] <Boomtime> also, right now you're using a framework (mongoose?) can you try the same thing in the node driver because right now you don't know where the problem is
[06:37:42] <Boomtime> @CtrlC: you know that nosql is just a term which means a database other than SQL right?
[06:39:13] <aslmx> good morning - or evening - i'm trying to run mongodb server, but at the moment i struggle to understand why i have a mongod.conf and a mongodb.conf. The mongod.conf uses the notation with ":" to separate keys from values, the mongodb.conf uses "="... i purged any mongodb related package and reinstalled... however i'm unsure if that is correct
[06:39:17] <Boomtime> yeah, but you've got the same problem - you're trying to join a group like "not alcholics anonymous" for people who aren't alcholics
[06:40:55] <Boomtime> @aslmx: https://docs.mongodb.com/manual/reference/configuration-options/#file-format
[06:41:47] <aslmx> Boomtime: is it correct to have both the files? i saw from the init script that only one seems to be used... becaues i also seem to have a mongod.service and a mongodb.service... i'll cehck you link
[06:44:16] <Boomtime> the one using YAML format, probably the one with key:value is the one you want to keep
[06:45:50] <aslmx> so.. i started with sudo mongod -f /etc/mongodb.conf --smallfiles and now finally the logfile says its waiting for connections :-)
[06:46:02] <aslmx> so i need to find out why it doesnt really want to work smoothly with the init scripts
[08:40:45] <kurushiyama> Zelest Well... Data modelling for a new project of mine. Exhausting, because the PO puts in new user stories every day. While I technically could (and maybe should) ignore them, often enough there are game changers ;)
[08:42:02] <Zelest> Coding a new project and the users are invited today basically.. where we most likely will have to redo tons of stuff for their needs (as they barely know what they want/need yet)
[08:42:52] <Zelest> I've heard that name so many times and so different definition of it, so I barely know how to reply to that :P
[08:43:16] <Zelest> But I expect anything I write to change the day after and try to keep it as "open" as possible to changes
[08:53:22] <Zelest> Derick, How am I supposed to push an item to an array using PHP? I do this client-side and wish to array_push($foo, $bar), but seeing $foo is a object (MongoDB\Model\BSONArray) it fails.. :o
[12:10:26] <chujozaur> Hello, how can I tell that replication is not going to finish because of oplog being too short
[13:18:58] <boutell> Hi! What really happens if I make a bunch of simultaneous async queries via the nodejs mongodb native driver?
[13:19:38] <boutell> Do they queue up over the same connection and run sequentially, but without the latency of another round trip? Do they run in some kind of parallel but not in an unbounded, could-fail kind of way? Or do they potentially hit some limit and fail?
[13:47:38] <GothAlice> boutell: Most drivers implement connection pooling, so you get "true parallelism" up to N connections, after that, operations block on a connection being freed.
[13:53:53] <GothAlice> boutell: Yup, sorry for the delay, it is roughly structured and implemented the same as the Python MongoDB driver, with a maxPoolSize option specifiable in the connection URI (?maxPoolSize=200) or as part of the optional options object you can pass in.
[13:54:45] <boutell> GothAlice: huh. So it’ll actually open 100 sockets when I make a boring connection with a mongo URL like mongodb://localhost:27017/foo for example?
[13:55:15] <GothAlice> So initially it'll populate 5, then when your operation load begins to exhaust those, it'll make more.
[13:55:31] <cheeser> the java driver actually keeps a core pool (lazily connected) and then will expand beyond that pool to a point.
[13:55:47] <cheeser> once the load dies down, the extra connections over that core size will be closed/collected.
[13:55:53] <GothAlice> Some drivers have a 'minPoolSize' option, too, but I have no idea if the node native driver supports that.
[13:56:34] <boutell> I see. So let’s say I usually have four or five things going on (each of which involves many potentially parallelizable queries), but occasionally I have a thousand things going on. It sounds like it should be safe to fire off queries in parallel just to speed up those four or five queries, because in the “thousand things” case, they will just queue up and wait. Yes?
[13:58:54] <GothAlice> (FYI, https://github.com/christkv/mongodb-core/blob/1.3/lib/connection/pool.js#L51 is where the min size of 5 is defined, and it strongly appears that that represents the maximum size.)
[14:00:46] <GothAlice> MongoDB operations are linearized server-side, so in reality there is never a possibility of two operations really happening "simultaneously" or "in parallel". (Mostly; some things like background index creation do exist, of course.)
[14:01:18] <GothAlice> The trick is that in streaming the results back, much time is spent waiting on I/O, so the results can be parallelized.
[14:03:38] <boutell> what about multicore setups on a single-node server? Is there still no simultaneous querying going on there?
[14:04:15] <GothAlice> That's the thing, the operation log is a linear list of operations, applied sequentially. If you have two updates issued within the same millisecond, one will "win" and go first.
[14:05:14] <GothAlice> Semi-true for queries, with the aforementioned "waiting on IO" bits to allow for parallelism, there. I haven't checked in a while, but it was my understanding that read operations are replicated to secondaries the same way writes are.
[14:09:17] <GothAlice> An update and a read at the same "instant", how deterministic is the result vs. two updates at the same "instant"?
[14:10:08] <cheeser> but that gets trickier in a replicated environment. how "written" is it really?
[14:10:26] <GothAlice> Perfect, that answers my question. :D boutell: Ignore what I said, reads are fully parallelizable.
[14:10:33] <cheeser> if it's committed to the primary but hasn't replicated, should it be read? 3.4 introduces semantics to configure that.
[14:11:42] <cheeser> read concern i think we're calling it. not to be confused with read preference.
[14:14:40] <boutell> Cool. So I should ease up on my current very conservative “async.series, async.eachSeries” approach to firing off mongo queries, because I should trust the driver to manage parallelism at a sustainable level and queue things when necessary. I am not making things worse in the high-traffic case by doing so.
[14:17:58] <GothAlice> https://docs.mongodb.com/manual/core/bulk-write-operations/ — as applies to writes, of course.
[14:19:44] <boutell> (Oh, and I should probably set that size parameter higher than 5, since it doesn’t grow on its own)
[14:25:07] <boutell> GothAlice: still trying to figure out the relationship between the maxPoolSize URL parser stuff and what’s really going to show up in pool.js by default, but it’s probably simplest just to instrument it and see (:
[14:25:30] <GothAlice> Yeah, I don't JS enough to be able to grok the code to see where the value there is flowing.
[14:27:02] <GothAlice> Heh. I'm one of those heathens that uses transpilers to run JS within Python (such as autoprefixer), and Python as JS client-side. (So… Python all the way down for me. ;)
[14:29:13] <GothAlice> My gods, I just realized what that means… I can write MongoDB-side JS functions and map/reduce… as Python code.
[14:31:36] <boutell> It looks like the self parameters might make methods async-safe (as opposed to relying on “this”), which is sweet
[14:31:53] <GothAlice> boutell: We use a fork of that one at work. We'll eventually be writing our own based on that (already forked, named "transplit") and https://github.com/azazel75/metapensiero.pj for cleaner processing, with the goal of ES6 generation to be extremely specific. (With Babel for ES6→ES5 conversion already built-in to the JS execution-in-Python lib we use, https://github.com/amol-/dukpy
[14:33:04] <agh> When i connect to the ELB with the mongo shell, everything is working fine, Mongo reconnect automaticaly to the PRIMARY node
[14:33:36] <agh> Now, when I use the native node.js driver it does not work when the ELB forward me to a SECONDARY node
[15:45:43] <dino82> So I think I discovered why my syncs are failing... 9001 socket exception SEND_ERRORs
[15:49:04] <chris|> I have an issue in on of our config clusters where one replica set member has a different replicaSetId that the rest of the members. I am curious as to how this can come about
[16:12:08] <n3ssi3> Hey, I am trying to import a db useing mongorestore... but It always stops with an error.... I found this Issue https://jira.mongodb.org/browse/TOOLS-939 and tried to --batchSize as well.. but even with --batchSize=1000000000000000000 it still stops with the error "insertion error: EOF"
[16:13:09] <n3ssi3> Yes my dbdump is huge... but isnt there a way to still use it? I only use this 1 db, so if I have to reinstall or whatever that is all possible
[16:37:14] <sha1sum> Hello. I'm looking for the proper operator/query to "reverse" what a normal find does, i.e. if I pass in `{foo: "bar", bam: "bat"}` and there's a document with `{foo: "bar"}` then that document is returned
[16:40:21] <sha1sum> The real-world scenario here is that, after creation of a document, it searches a collection for a "criteria" field where all of the embedded criteria fields are in the passed document that was created
[17:27:29] <kurushiyama> sha1sum Well, I guess it works as expected... ...and I should go to bed. http://hastebin.com/ufagecivon.coffee
[21:01:40] <Sagar> i am trying to create mongodb time using new MongoDB\BSON\UTCDateTime(time() * 1000); since i am on php 7, but the time is not comming in right, i want the time in ISODATE format so that i can set index, when i use the above fuction, it gives me date of 22-06-2016, my timezone is IST, what can i do?
[21:09:27] <ss23> Generally speaking, are inserts likely to be faster on WT than MMAP? That is, say just 15 threads doing db.colection.inserts() in loops
[21:10:02] <ss23> The docs talk about "document level locking", but I don't know enough internals to know if on MMAP an insert alone aquires a collection wide lock, and on WT it doesn't, etc