Log file Viewer
#mongodb logs for Wednesday the 8th of July, 2015
[00:09:11] <jecran> Just want to make sure, i tried to set 'upsert' to true, and tried a few variations of this db.collection(collection).insertOne(doc, {upsert:true}, function(err) and cannot get an 'upsert' but duplicate errors. What am i doing wrong? (using node)
[01:10:22] <symbol> Alright...I'm still hung on on the proper use case for Mongodb. Research and stack overflow is telling me that the moment I start realizing I need joins then I need to go back to a RDBMS.
[01:11:03] <symbol> I've been watching a lot of the webinars on mongodb.com and they suggest to embed by default and then consider referencing...then they make referencing seem like it's no big deal.
[01:11:46] <symbol> I should embed by default...but if the data changes a lot they recommend referencing it. A lot of different opinions from the same source...or I'm just really confused :D
[01:13:25] <bros> Can somebody please help me with an aggregation over a collection with subdocuments?
[01:15:28] <bros> I have a collection (orders). The collection has documents. Each document has a subdocument called "log".
[01:16:10] <bros> I'm trying to aggregate over all of the orders, but am having trouble aggregating. I'm aware on unwind, which helps me get one of the values out of log, but I do not know how to get the second.
[01:16:39] <bros> Once I unwind, the array is broken down in the pipeline. I've tried a combination of $project, $math, and $group pipelines to rebuild it with the array broken down into fields, but have had no luck.
[01:27:03] <bros> symbol: if (log[i]['action'] === 'opened') { started = log[i]['time']; } else if (log[i]['action'] === 'shipped') { ended = log[i]['time']; }
[01:28:30] <symbol> If you'll be doing the query a lot - I suppose a compound index would help...that won't fix your aggregation problem though.
[01:35:40] <symbol> http://docs.mongodb.org/manual/reference/operator/aggregation/subtract/#exp._S_subtract
[01:37:37] <symbol> Oh...why don't you just do it like this example? http://docs.mongodb.org/manual/reference/operator/aggregation/subtract/#subtract-two-dates
[01:39:53] <symbol> You might have to do something fancy with map/reduce unless someone else here knows the aggregation framework better (which is very, very likely)
[01:41:21] <symbol> http://stackoverflow.com/questions/13708857/mongodb-aggregation-framework-nested-arrays-subtract-expression
[01:42:39] <symbol> Oh, I'm not suggesting muliple $unwind...I was mainly referring to the use of $project with $unwind and $subtract
[01:46:10] <bros> I guess I could get really nasty. $match, $project log as log *and* some dummy field log2
[01:47:00] <bros> Are there extreme advantages for using aggregations over regular finds + client logic?
[01:47:52] <symbol> I'm not really that sure...everything I read claims that the aggregation framework is the fast way to go.
[01:48:54] <symbol> preaction: Have you been following? Really interested if this is something that can be done in the pipeline.
[01:48:57] <bros> Are subdocuments commonly used? All of my headaches Mongo come with my newbie decision to use them.
[01:50:46] <Boomtime> bros: (sorry for arriving late) the SO question you linked is answered.. is there an updated question?
[01:52:21] <bros> cheeser: Do you have an answer to my aggregation subdocument problem or did I just construct bad schema?
[01:53:24] <symbol> Wouldn't map/reduce be a viable solution? More verbose but it'd be easier since you can work with the document and emit the proper values.
[01:57:15] <Boomtime> bros: if the SO question is a relevant summary of your current question, i will read it in a little while (am busy right now, but i will get to it)
[01:57:44] <bros> I need to get fields out of a subdocument in an aggregation pipeline, but it's not possible.
[01:57:49] <symbol> Oh, check out the answer on this http://stackoverflow.com/questions/27473315/subtract-group-aggregate-values-in-mongodb
[02:00:15] <Boomtime> bros: your summary in words is trivial to achieve, clearly there is something particular about your data.. can you gist your troublesome document and what you've tried?
[02:00:34] <bros> Boomtime: https://gist.github.com/brandonros/d00d73f53589416b04e8 https://gist.github.com/brandonros/3b65d4d6ba5765e26c80
[02:04:49] <Boomtime> this still looks trivial, so clearly i'm not understanding.. you want to match on the first time entry in log array (log.0.time) but then retrieve the time field of the last entry in the log array for the matching document?
[02:07:15] <Boomtime> my summary of the problem can be achieved in a single targeted query, aggregation is overkill: http://docs.mongodb.org/manual/reference/operator/projection/slice/
[02:11:56] <Boomtime> um.. i'm confused, are you attempting to use aggregation for something just for kicks? what is the reason for using aggregation if it's not necessary?
[02:14:42] <Boomtime> you can use aggregation for this if you like, but it'll be less efficient since there isn't anything to aggregate.. the solution i see, if you must is $match, $unwind, $sort on time (for assurance), $group with a $last, + $project somewhere if formatting is needed
[02:24:54] <symbol> Boomtime: Just because I'm interested in this...would it be more efficient to simply $slice than to do the subtraction between the two times with aggregation?
[02:30:46] <Boomtime> as a general rule, any format or calculation that doesn't affect the result-set outcome is better off being done on the client - why add load to the server for something that could be distributed?
[02:31:04] <Boomtime> i.e if the amount of data being retrieved is unaffected, then it doesn't matter which end of the pipe you do it
[02:31:54] <Boomtime> you can do at the server where your server takes the capacity (that's great if it's 1975 and all your clients are thin terminals), or you can do the math at the client which is almost certainly free
[02:32:54] <symbol> So why are a lot of the webinars trying to swing the "real time analytics" with the aggregation framework as a selling point?
[02:34:03] <Boomtime> analytics specifically avoids the data being returned - the server _aggregates_ the data set to a manageable report
[02:34:29] <symbol> And it's faster to aggregate with mongo than send the million records over the wire?
[02:35:12] <Boomtime> definitely, sending the million records over the wire in that case has probably made the problem worse - the client still has to process it all, only now it isn't indexed
[02:36:15] <symbol> That makes sense...granted this is nothing new to the world of databases it seems. Aggregations have been around but I will say the aggregations in Mongo are easier to write.
[02:36:50] <Boomtime> they are definitely easier to debug.. since the pipeline can generally be viewed at any point to see if it contains what you expect
[02:45:06] <Boomtime> yes, many find it daunting / seemingly verbose to begin with - once you learn to manipulate them in the shell for dev work, eg. by using a var for each stage, it becomes ever more powerful
[02:49:17] <symbol> Something I'm still hung up on is determing the proper use case...a lot of the documentaion advises to embed by default then reference when things either grow or are large but rarely accessed. At the same time, I've been advised that if relationships start to emerge that Mongo isn't a good fit.
[02:52:35] <Boomtime> one common mistake that people make is thinking that embedding versus linking is an exclusive choice
[02:53:24] <Boomtime> why not link the user-id but also embed the username of the comment into the comment document (as it was when the user made the comment)?
[02:54:16] <Boomtime> this will avoid having to look up the username via the reference just to get the name of the user who made the comment.. meanwhile, it liikely provides all the information that was actually needed by having the reference
[02:55:09] <Boomtime> and if you use wiredtiger with mongodb it might compress it out of existence anyway
[02:55:44] <Boomtime> maintenance is a funny thing.. when you sit down and look at how often metadata changes, a rounding error usually makes it zero
[02:56:16] <Boomtime> if you are super-keen to make sure the username stays up-to-date at all times then yes, you have some overhead to permitting users to change their name
[02:56:56] <Boomtime> here's a interesting thing about comments on blogposts etc.. how many times have you read past the first 5 or 10 comments?
[02:57:03] <symbol> Or more real-world, a product with reviews...we wouldn't want to pull 1000 reviews everytime we look a product.
[02:58:13] <Boomtime> understanding your data well often solves the problem in ways you only have available when embedding is an option
[02:58:41] <Boomtime> and yes, you can stuff the overflow into another collection that you call on for those 1% of sticklers that are gonna read everything
[03:08:18] <markizano> like in mysql you can use "!cmd" to exec system commands outside the shell as the db user.
[04:05:14] <kba> When doing: `db.getCollection('houses').find({ rent : { $gt: 0 } });`, I only get one house, though, even though a lot match the criteria.
[04:23:34] <RegexNinja47> (Taconut) I'm having an emergency here: some of my users were saying they couldn't get into my program. I checked the database and half the users' database entries are just "_atomic". What do i do?
[04:26:15] <Boomtime> your last two statements are contradictory - how can you fix something if you didn't know what was wrong?
[04:29:15] <Boomtime> .. and of course, you were only deploying this untested migration script on a test QA copy of your live data, no harm done anyway, right?
[06:08:24] <dimon222> update and insert methods dont retrieve object, they only retrieve result of process
[06:09:19] <dimon222> i'm not even sure that any kind of database retrieves affected objects on update methods oO
[06:11:10] <dimon222> so i guess if u want to get result after update you need to do find and pick it :C
[08:07:04] <androidbruce> well in the agents tab under administration, the "state" is active or standby
[08:07:30] <androidbruce> logs on standby nodes says " Nothing to do. Either the server detected the possibility of another monitoring agent running, or no Hosts are configured on the MMS group."
[08:43:05] <androidbruce> bogn: answered, https://docs.cloud.mongodb.com/faq/#do-i-need-a-monitoring-agent-for-every-mongodb-instance
[08:44:53] <bogn> yup, if you would've written "only one of the agents in the group are "active"" I could have answered right away
[08:45:16] <bogn> but with the words node and replica set I thought you were talking about actual mongod instances
[09:02:51] <puppeh> I'm having problems with the mongo ruby driver version 2.0.6: http://pastie.org/private/20xubcr0oaacw3vclgkba
[09:36:43] <puppeh> I'm doing a client.use(:foo) and then client.database still returns the previous database
[10:43:12] <puppeh> in mongo driver v2 (ruby) this command no longer exists: https://www.omniref.com/ruby/gems/mongo/1.8.5/symbols/Mongo::MongoClient/ping
[11:01:13] <bogn> puppeh, regarding the logs, provide the constructor with {logger: logger_instance}
[11:08:58] <bogn> and by the way: http://api.mongodb.org/ruby/1.12.2/Mongo/MongoClient.html#ping-instance_method
[11:24:15] <bogn> but on the other hand upsert should handle that, I'd say it is not being reflected
[11:28:15] <bogn> hm, can't see anything else but the different collection handling you have there now
[11:43:42] <puppeh> hm, I think that the logic of this query is wrong. It says: "if you don't find a document with this _id and these conditions, then create it". It doesn't find it and then it tries to create it, with the same _id
[11:47:16] <bogn> of course, but then upsert kicks in takes all query conditions and update operations and inserts that
[11:47:57] <bogn> http://docs.mongodb.org/manual/reference/method/db.collection.update/#upsert-parameter
[11:50:09] <bogn> If all update() operations complete the query portion before any client successfully inserts data, and there is no unique index on the name field, then each update operation may result in an insert.
[12:36:50] <deathanchor> Is there any performance gain for bulk inserts aside from network performance?
[12:54:35] <cheeser> there are async variants with the java driver that'd probably provide a nonblocking bulk write
[13:27:58] <leporello> I have an error: exception: the group aggregate field \'reference\' must be defined as an expression inside an object
[13:51:57] <StephenLynx> unless you want extreme precision for financial operations, thats just fine.
[13:52:16] <cheeser> barring the usual caveats of storing currency values in double precision data types
[13:53:27] <symbol> I'm not sure I'm familiar with those caveats but I suppose that's something I can google for.
[13:57:40] <gabrielsch> hi, I need to make a little migration on my collection, my document property usar a "field: object", now I need to update all to field: [ { object } ], how can I do that?
[14:01:07] <gabrielsch> cheeser: sorry, I'm kinda lost to do this, should I loop with "db.Foo.find().forEach" ?
[14:20:06] <thikonom> Hi i will attempt to create a replica set. My set is as follows: I have an existing node with 50gb of data. Does anybody know if the database will be accessible when the data will be replicated to the newly added node ?
[14:25:21] <thikonom> cheeser, I just found that: http://docs.mongodb.org/manual/tutorial/deploy-replica-set/#add-the-remaining-members-to-the-replica-set. At then of the paragraph it says "When complete, you have a fully functional replica set. The new replica set will elect a primary.".What is the newly added node is elected as primary ? Then its data wont be in a consistent state which makes the db inaccessible ?
[14:30:39] <fxmulder> is there a way to speed up rsync if the two replicas are mostly in sync? rsync seems to be writing over existing files
[14:31:46] <thikonom> cheeser [15:18] <@cheeser> the primary will, yes. (Was that going to me or to someone else ?).
[14:42:40] <cheeser> http://docs.mongodb.org/manual/tutorial/resync-replica-set-member/#replica-set-resync-by-copying
[14:46:21] <fxmulder> if the timestamps are off but the file content is exactly the same, then I can save a network copy and a disk write
[14:46:39] <fxmulder> but if the file content isn't exactly the same then I am just adding a second read operation for each file
[14:49:07] <thikonom> cheeser. Thanks, regarding the link I sent before, I am just trying to clarify what "When complete, you have a fully functional replica set. The new replica set will elect a primary." means. I am wondering what happens when the newly added node is elected as primary. If the data has not been replicated to it from the original node then the database wont be accessible ?
[14:50:27] <cheeser> during the election process, you can't write, no. if you have secondary reads enabled you can read, though.
[14:55:15] <puppeh> I can connect to a single instance using the ruby mongo client but when I enter 2 servers I get a connection refused (2)
[15:01:46] <kas84> when doing update with upsert =true the node.js mongo driver is not returning the _id of the inserted documents
[15:02:06] <cheeser> you can run as many instances of the server as you want on a machine provided they use different ports and data dirs
[15:02:08] <puppeh> actually I'm using the very same example from http://docs.mongodb.org/ecosystem/tutorial/ruby-driver-tutorial/#creating-a-client
[15:12:03] <thikonom> cheeser: Thanks, its a bit complicated cause then i have to switch off the secondary reads. Is there a way to explicitly specify the primary ?
[15:13:43] <deathanchor> is there another command besides .stats() for getting info on a collections' settings?
[15:17:34] <cheeser> thikonom: the primary is selected from amongst the repl set members. you can't really force it other than setting priorities on the secondaries and that has its own set of issues
[15:35:51] <puppeh> cheeser: this is what I'm doing exactly http://pastie.org/private/xiijnlkfyucqoumuzdzxq
[15:39:44] <puppeh> I'm just following the tutorial: http://docs.mongodb.org/ecosystem/tutorial/ruby-driver-tutorial/#inserting
[15:49:08] <RegexNinja47> So i managed to find a json string of my mongodb datavase! How can i use that to recover everything?
[17:43:15] <adsf> cheeser: may have answered it myself :p Was just wondering on a sensible pattern for the java driver
[17:43:54] <adsf> should i close connection after each use? atm i instantiate a class that connects and never really shut it down :p
[17:45:30] <cheeser> MongoClient maintains a connection pool so, no, you won't see those connections go away.
[17:45:32] <adsf> i find if i instantiate the class from other classes, if i close in one it seems to mess with the others
[17:47:37] <cheeser> unless that client is used elsewhere you should manage resources appropriately.
[17:48:48] <adsf> reason i ask is i had odd issues where it would complain if i instantiated it in a method and closed it, but had it still open in another method where it was instansiated
[18:11:48] <t-ready> trying to do an ubuntu (trusty) install. apt update fails - can't find multiverse/source/Sources. my sources.list.d file: deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.0 multiverse
[18:14:39] <t-ready> following instructions from here: http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/
[20:43:20] <GothAlice> Ough. http://www.bbc.com/news/technology-33442419 < 17 year old, 50K hacks, MongoHQ included. (They use ColdFusion stuff? Huh.)
[20:44:59] <StephenLynx> "It also confiscated his PC and ordered him to handover €6,588 (£4,725) worth of property obtained through his crimes." wow, public enemy #1 :^)
[20:45:59] <mocx> Zelest: no kidding, I don't hang out in the DO channel anymore so that's probably why
[20:47:28] <Zelest> official or not, I love the community.. we even have a tiny core channel now where people hang out :)
[22:00:18] <chrisinajar> Hi! I have a for loop and within it I call collection.update with some data. Only 1 update occurs, however if I run the exact same updates 1 at a time then all of them update. Update isn't returning errors, it just silently fails. What am i doing wrong?
[23:03:15] <chrisinajar> Hi! I have a for loop and within it I call collection.update with some data. Only 1 update occurs, however if I run the exact same updates 1 at a time then all of them update. Update isn't returning errors, it just silently fails. What am i doing wrong? I tried passing in { w: 1 }, suggested from a stack overflow
[23:11:34] <chrisinajar> i'm looping through all entires in a table, making an RPC to gather data, and storing that data in the model
[23:12:20] <chrisinajar> if i copy the parameters logged out from the loop and paste that into mongo cli then it works
[23:13:45] <dimon222> i was about to assume that update query once being called only updates one record - default for update parameter "multi" is false, which means only one document that fits requirements will be updates
[23:14:16] <dimon222> but if you're calling it in for loop, it means its 100% supposed to update more than one document, weirdo
[23:14:48] <chrisinajar> looks like the first one always work and the rest silently fail and return success witn n: 0
[23:15:09] <chrisinajar> yeah, i'm shrinking the code, i have a lot of debugging crap in there right now
[23:16:15] <chrisinajar> the first one returns { ok: 1, nModified: 0, n: 1 } and the ones after all return { ok: 1, nModified: 0, n: 0 }
[23:16:46] <dimon222> well, it means it passed successfully, but there's no element that follows requirement so nothing was updated after that
[23:17:00] <chrisinajar> maybe findOneAndUpdate will work, or whatever that method is called... i was trying to use mongoose with .save() but it had the same behaviour (only the first one works)
[23:18:07] <EXetoC> the same part of the message header is used for the batch size and limit apparently. I dunno what's going on there
[23:18:48] <chrisinajar> not according to console.log, as i said i debug the query and paste it into mongo cli
[23:18:55] <dimon222> try to print your document id before you execute update in your code inside of loop
[23:19:46] <chrisinajar> i'm prooooobably going to switch to RethinkDB instead, just got the approval, but thanks for the debugging help!
[23:19:59] <dimon222> try to print id before executing update query inside of loop, so make sure you're processing different documents
[23:27:41] <chrisinajar> dimon222: if your curious here's the code simplified a bunch, switching DB's though so I don't actually need any more help. Thanks!! http://pastebin.com/mFmSmLMc