PMXBOT Log file Viewer

Help | Karma | Search:

#mongodb logs for Saturday the 20th of December, 2014

(Back to #mongodb overview) (Back to channel listing) (Animate logs)
[08:45:40] <Synt4x`> how do I append to a document? for instance if have a collection, and I want to find a certain document within that and add something to it
[10:22:17] <_roland> Hi all. I'm new to mongodb. I'm creating a REST service in Node using the Node.js MongoDB driver and I'm experiencing performance problems. I could use some guidance to improve performance.
[10:22:50] <_roland> The service supports a call to "profile" the data in a mongodb collection - basically to make a summary of all docs in a collection and present an inventory of the different types of docs found in the collection.
[10:23:47] <_roland> I tried 2 different approaches: 1 using find and then useing a cursor to update the profile; 2 using a map reduce job (using the mappper to update the profile and the reduce only to return the profile
[10:24:20] <_roland> I noticed that just the work of scanning the collection is much faster with map reduce (no surprise since the data doesnt get pulled to the client)
[10:25:02] <_roland> however, when the mapr job actually has to do something, it is in my case much slower (3x) than using the client side cursor.
[10:25:23] <_roland> I was wondering if someone could give me some advice on how to best troubleshoot performance.
[10:26:45] <_roland> Oh, final bit of information - when using the scan approach but not updating the profile I find that the work of profiling adds about 25% of extra work as opposed to simply scanning the collection with a cursor.
[10:27:07] <_roland> I'm at a loss as to why that same work executed in a mapr context can slow down the job so much.
[10:28:14] <_roland> my code for the profiler as map/reduce is here http://pastebin.com/x9R83hbJ
[12:18:52] <alexi5> hello
[14:04:44] <royalharsh95> hi
[14:04:53] <royalharsh95> I'm new to mongoDB
[14:14:14] <royalharsh95> anyone here ? :o
[15:02:39] <davidcsi> Hello all, I'm doing upserts to $push data into an array call packet. Is it possible to also add a top-level field? like "add this to the array field and add the top-level field?"
[15:06:31] <davidcsi> looks like i can't mix field updates/add with $push, i'm doing "u: { "deleted": "0", $push: { “field”: “whatever” } }" but this is not working...
[15:08:22] <kali> davidcsi: look for $set
[15:09:15] <kali> this has come a lot over the last few days... has the update doc page been "improved" ?
[15:09:28] <davidcsi> BOOM! I was JUST looking into that! thanks!!
[15:09:58] <davidcsi> just for the record, it's: "u: { $set{ "deleted": "0"}, $push: { “field”: “whatever” } }"
[15:10:01] <davidcsi> nice, thanks!
[15:10:45] <davidcsi> is it possible to use issue " $set{ "deleted": "0"}" ONLY is it is going to insert? Or do i need to do that manually?
[15:11:14] <kali> sorry, i don't parse your question
[15:12:39] <davidcsi> I only want to add that "deleted" field when upsert is inserting... when it's updating i don't want it to do it
[15:13:40] <kali> davidcsi: http://docs.mongodb.org/manual/reference/operator/update/setOnInsert/
[15:14:47] <davidcsi> oh wow... amazing! Thanks kali!
[15:48:47] <Hw2k> hi, is there some good way that I can query mongo direcly from the terminal to see if a user exists for example?
[15:49:52] <Hw2k> I've tried with --eval, but it seems to return nothing(?), and I tried a syntax like 'echo "db.systems.users.find()" | mongo admin
[15:50:41] <Hw2k> the later seems to work, but I don't really understand why :)
[16:23:10] <clearcut> i have like 7000 quotes - i want to create fast search so that when user types in keyword he gets near-live search and filtering - what is the bes option for this fast searching and getting the results, mongodb/map-reduce / elastic search?
[16:23:47] <kali> definitely not map reduce
[16:24:15] <kali> you may want to check out mongodb internal full text engine
[16:24:36] <kali> if it is good enough for your use case, it's certainly the easier option
[16:25:43] <kali> 7000 small documents is so small anyway, even map/reduce may work (but i don't recommend going this way)
[17:32:18] <Mia> kali, it's just _id and name atm
[17:32:35] <Mia> I was thinking, maybe it's hidden in _id or something
[17:33:31] <kali> well, the _id contains a timestamp
[17:33:55] <Mia> oh I didn2t know that
[17:34:50] <kali> i mean. _id contains whatever you put in it, or an ObjectId if you leave mongo pick whatever it wants. and ObjectId contains the timestamp
[17:35:11] <kali> http://docs.mongodb.org/manual/reference/object-id/
[17:48:56] <chetandhembre> is there any way in which i can find records for given array of ids present in mongodb without fetching them ?
[17:51:43] <ehershey> find them without fetching them??
[17:54:28] <chetandhembre> yeah
[17:54:39] <chetandhembre> i just want to see if they are exists ?
[18:15:06] <HoloIRCUser> So I'm I getting it wrong with Mongo or what
[18:17:43] <ij888> Say something
[19:18:11] <_roland1> Hi! I'm new to mongo. I'm trying to get a "data profile" of the documents stored in a single collection - a single object that summarizes what structures and datatypes are in there.
[19:18:51] <_roland1> I tried this in 2 different approaches - 1) map reduce 2) client cursor - the actual profiling code is more or less the same in both cases.
[19:19:36] <_roland1> currently both approaches are too slow - at least it feels it should be possible to speed it up; I'm looking for someone that could guide me a bit.
[19:19:40] <kali> _roland1: https://github.com/variety/variety
[19:20:00] <kali> ho, it will be slow. you need to scan the full collection
[19:20:05] <_roland1> kali: that looks useful, thanks
[19:20:14] <_roland1> kali: ok. but what kind of throughput should I expect?
[19:20:38] <_roland1> kali: with my client cursor I'm doing about 2k docs / sec for a particular dataset.
[19:20:55] <_roland1> kali: with map/reduce it's considerably slower, ~700 docs /sec.
[19:21:14] <_roland1> kali: I'm confused as to how that can be, I epxected m/r to be faster since I don't have to pull the docs to the client.
[19:22:28] <_roland1> kali: I'm also looking for an efficient way to sample, as I do not want to be limited by the size of the dataset. iow I can afford to have it wrong sometimes as long as I can make some estimate as to how reliable my profile is.
[19:22:28] <kali> _roland1: well, first of all, mongodb focus is on making selective queries fast, so this code-path (full scan and map/reduce) are probably not as optimum as they could be
[19:22:53] <kali> wow, sampling is hard
[19:23:02] <_roland1> kali: the thing is, if i keep the m/r job as is, but cut out the actual work, m/r scan is really fast.
[19:23:09] <_roland1> kali: faster than a client cursor.
[19:23:22] <_roland1> kali: so I'm wondering how the server side code execution can be so slow.
[19:23:32] <_roland1> at least compared to client side code execution.
[19:23:43] <kali> what language are you using client side ?
[19:23:51] <_roland1> kali: javascript. nodejs.
[19:24:09] <_roland1> kali: using "the" nodejs driver for mongodb.
[19:24:40] <kali> well, i don't know
[19:24:50] <kali> how big is your m/r job ?
[19:24:53] <_roland1> it's literally the same code in both cases, only diff is that for m/r I had to wrap it in a scope.
[19:24:54] <kali> in LOC ?
[19:25:09] <_roland1> kali: uhm...forgive my ignorance. I don't understand that question.
[19:25:21] <_roland1> kali: how do I measure "loc"?
[19:25:40] <kali> how many Lines Of Code ? can I audit in a few minutes ?
[19:25:45] <_roland1> a.
[19:26:21] <_roland1> kali: 190.
[19:26:36] <kali> ok, paste it somewhere
[19:26:42] <kali> i'll check if i see something abvious
[19:27:02] <_roland1> kali: http://pastebin.com/x9R83hbJ
[19:27:11] <_roland1> kali: TIA :)
[19:28:11] <kali> wow, that's very nodish. how do you run it in map/reduce ?
[19:28:53] <_roland1> kali: well, lines 165 ..175 is that.
[19:29:11] <_roland1> kali: the actual code is in the scope var, "s".
[19:29:32] <_roland1> kali: the main method is updateProfile, which runs through the keys of the doc.
[19:29:57] <kali> why do you even bother using map/Reduce ?
[19:30:28] <kali> because you're basically just scanning the document there
[19:30:43] <_roland1> kali: I'm new to node. I don't know any better. I started with the client cursor, was unsatisfied with perf. I figured I could gain by not pulling docs to the client, enter m/r.
[19:31:16] <_roland1> kali: not just one document, a bunch. Currently the entire collection, in the future, hopefully only a sample.
[19:31:54] <kali> yeah, i meant the collection.
[19:31:55] <_roland1> kali: my action is conceptually a reduction as I'm compressing info from all docs into one.
[19:32:21] <kali> you can try db.eval to run some javascript code in the server
[19:32:35] <kali> it could also be relevant to try it in the mongodb shell
[19:32:59] <_roland1> kali: that sounds like good advice, I shoul probably try that to see what I can expect in a best case.
[19:33:25] <_roland1> kali: I read about db.eval, haven't found the proper way to do that with the nodejs driver (yet.)
[19:33:38] <_roland1> All very new environments to me, still have to learn a lot.
[19:35:46] <kali> even if there is no helper for eval in the nodejs driver, you can always call it as a command
[19:36:34] <kali> but let me just make this is very clear: mongodb is not the good at doing that kind of stuff. it's not designed for that
[19:38:13] <_roland1> kali: thanks! Good to know. kali yeah I read about how you can alwyas fire a query to run a command. Might need to give that a try.
[19:54:19] <_roland1> kali: wow. That certainly makes a big difference. in the shell it's 800ms. Via node it's almost 5s.
[19:55:21] <_roland1> _roland1: oh wait.
[19:58:06] <_roland1> nope. ghost. actually about as fast in both cases.
[21:11:17] <_roland> kali: it's interesting. documents with long arrays seem to suffer substantially when I do them trhough map/reduce as compared to the cursor approach. can't really explain it, perhaps the BSON to JSON conversion is - for some reason - not very efficient for that particular case.
[21:12:06] <_roland> kali: anyway - thanks for the help. I think I can move on for now.
[22:14:19] <qswz> Hi, is it possible to search with a missing value, for example instead of find({'hello': 1}) I want to find({'hello': ?}) all items with a hello key, or find({?: 1}) all items with value 1
[22:15:03] <qswz> for find({'hello': ?}) I guess $exists can help
[23:12:58] <ij888> !help
[23:12:58] <pmxbot> !8ball (8) !acronym (ac) !anchorman !annoy (a, bother) !bender (bend) !bitchingisuseless (qbiu) !blame !bless !boo !bottom10 (bottom) !calc !chain !cheer (c) !c
[23:12:58] <pmxbot> ompliment (surreal) !config !ctlaltdel (controlaltdelete, controlaltdelete, cad, restart, quit) !curse !dance (d) !danke (dankeschoen, ds) !deal !define (def) !
[23:12:59] <pmxbot> demotivate (dm) !disembowel (dis, eviscerate) !duck (ducky) !embowel (reembowel) !emergencycompliment (ec, emercomp) !esperantomotivate (em) !excuse (e) !featur
[23:12:59] <pmxbot> ecreep (fc) !fight !flip !fm (frenchmotivate) !fml !gettowork (gtw) !gm (germanmotivate) !golfclap (clap) !google (g) !grail !haiku !hal (2001) !hangover !help
[23:12:59] <pmxbot> (h) !hire !imotivate (im, ironicmotivate) !insult !jm (japanesemotivate) !job (card) !karma (k) !keelhaul (kh) !klingon (klingonism) !lm (latinmotivate) !log !l
[23:12:59] <pmxbot> ogo !logs !lunch (lunchpick, lunchpicker) !meaculpa (apology, apologize) !motivate (m, appreciate, thanks, thank, gracias) !murphy (law) !nailedit (nail, n) !na
[23:13:00] <pmxbot> stygram (nerf, passive, bcc) !norris !notify !oregontrail (otrail) !panic (pc) !password (pw, passwd) !paste !pick (p, p:, pick:) !pm (piratemotivate) !progress
[23:13:00] <pmxbot> !quote (q) !r (r) !resolv !roll !rubberstamp (approve) !saysomething !schneier !send_text !simpsons (simp) !stab (shank, shiv) !storytime (story) !strategy !st
[23:13:00] <pmxbot> rike !tgif !therethere (poor, comfort) !ticker (t) !tinytear (tt, tear, cry) !top10 (top) !translate (trans, googletrans, googletranslate) !troutslap (slap, ts)
[23:13:01] <pmxbot> !urbandict (urb, ud, urbandictionary, urbandefine, urbandef, urbdef) !version (ver) !where (last, seen, lastseen) !wolframalpha (wa) !zinger (zing) !zoidberg (
[23:13:01] <pmxbot> zoid)
[23:56:34] <emgram> hey I'm having a problem with inserting into a database. I'm getting _filename.empty()
[23:57:23] <joannac> emgram: erm, what. how are you trying to insert?
[23:57:54] <emgram> > db.user_dbs.insert({"test":"test"})
[23:57:54] <emgram> _filename.empty()
[23:58:31] <emgram> to be clear the second line is the response