Log file Viewer
#mongodb logs for Friday the 26th of June, 2015
[00:15:08] <astromaddie> I'm not even sure if what I'm trying to do is possible, but I'm trying to sum up a few million arrays (to get an average). the arrays all have the same dimensions.
[00:17:06] <astromaddie> I'm also using mongo by proxy of pymongo. I tried doing nested python loops first but after 13 hours, it wasn't finished -- that's a no-go
[00:37:32] <astromaddie> @joannac I've been trying to use the aggregation framework... it doesn't seem to add multiple arrays together
[00:38:32] <astromaddie> eg, if I have array A=[1,1,1] and B=[2,0,1], I'm trying to do an aggregate sum so that my resultant output is [3,1,2]
[00:47:14] <astromaddie> daily.aggregate(pipeline=[{'$group' : {'_id' : {'appid' : '$appid', 'time' : '$time'}, 'traffic' : {'$sum' : '$traffic'}, 'cnt' : {'$sum' : 1}}}, {'$group' : {'_id' : 'null', 'traffic' : {'$sum' : '$traffic'}, 'cnt' : {'$sum' : '$cnt'}}}], allowDiskUse=True)
[00:48:29] <astromaddie> joannac: doesn't $unwind just operate on all the individual values, without preserving the array structure?
[08:00:32] <m3t4lukas> https://github.com/mongodb/mongo-c-driver/pull/222/files?diff=split#diff-a8dc54c42bee460afe8f133af3220069L771 for anyone who also whishes to do that
[10:26:05] <PasWhoop> Hello there! anyone knows what the output on the mongod terminal window means?
[10:52:01] <mark_at_minds> hello! how well does mongodb support frequents deletes/updates, partically on indexes columns?
[10:54:04] <gemtastic> Hmm anyone knows why I get "command not found" in the console when I'm running mongod in one terminal and in the bin directory try to run mongo in OSX?
[10:58:16] <gemtastic> mongod is running and I tried to access mongo. For some reson it wouldn't work
[10:58:43] <gemtastic> But it seems to be now for some reason. Looks like third time new terminal was the charm O.o
[12:19:37] <tibyke> any idea on getting the unique _keys_ of a nested 'array' in a collection? can it be a oneliner or is it possible only with a loop?
[12:21:33] <tibyke> i have something like { whatever: { foo: 1, bar: 2}}, then { whatever: { foo1: 3, bar1: 53}}, and {whatever: {foo2: 234, bar2: 555}}, and i need foo, foo1, foo2, bar, bar1, bar2
[12:23:31] <tibyke> thats a good question, i'll convert it to that schema, but im curious if I can get it out from this schema.
[12:25:51] <tibyke> no, they are in several documents, all nested in 'whatever', and i need the distinct collection-wide keys of 'whatever' in the above example: foo, foo1, foo2, bar, bar1, bar2
[12:30:02] <tibyke> you dont get it. or me :) i need every single key in 'whatever' in every document
[12:31:26] <gemtastic> Have I understood it right; A key feature in mongo is that if something doesn't exist, it will create it?
[12:32:07] <tibyke> StephenLynx, sure i will remove, but im courious if it can be achieved this way.
[12:34:50] <gemtastic> So, in other words; let's say I have a node.js application interacting with my mongodb and it wants to put stuff into a collection that doesn't exist, will it create it or will it throw an error?
[12:40:12] <gemtastic> Do you still need to patch Mongodb for windows? If you don't, what will happen?
[12:40:53] <gemtastic> When I downloaded mongodb for windows it said I had to download a patch for win7
[12:41:43] <gemtastic> If you are running any edition of Windows Server 2008 R2 or Windows 7, please install a hotfix to resolve an issue with memory mapped files on Windows.
[13:47:28] <tibyke> StephenLynx, and how would you get every single field/key from every document without the nesting? so eg. {foo: 1, bar: 2}, {foo1: 0, bar2: 434}.
[13:48:14] <gemtastic> The idea is to put my mongo in a cloud, but right now I'm just trying to learn how to get it set up at all :P
[13:48:42] <StephenLynx> that creates a new document with the fields directly in the document, no nesting.
[13:51:24] <gemtastic> StephenLynx: I really don't think it matters when you're jsut n00bing around.
[13:52:01] <danijoo> I want to move a huge database (400gb) to another server while in production (cant take it down). what do you think is the best way to do so. db.copyDatabase() ?
[13:52:43] <StephenLynx> I have never did anything like that, but putting the new server as a replica and waiting for it sync is a valid option?
[13:52:56] <vagelis> Hello, I have to make a query in MongoDB and then return the results in an aggregation style :| I am not good with Mongo and I dont know if its possible to combine aggregation with query. Does $match has to do with that? :S
[13:53:34] <danijoo> data loss is not a problem. its more of a giant cache so if entries get lost thats ok
[13:53:58] <danijoo> also both servers are in the same data center so data transfer rate is also high
[13:55:01] <tibyke> StephenLynx, its something like a counter for every field, and i need to loop thru every "field" (counter) to get the top5 of them.
[13:55:39] <vagelis> StephenLynx I want it to be like this: { 'Separated_by_a_Field': '$Field', 'My_Results': [... , ... , ...] }
[13:55:42] <tibyke> eg {apples: 5, tables: 8}, {whatever: 15, baskets: 885}. and i need every TOP5 of every article.
[13:56:40] <vagelis> Actually let me go back to the code to examine something and ill be back StephenLynx
[13:57:26] <tibyke> StephenLynx, thats what im trying, but still pretty confused :) is it push? or is it addToSet?
[13:57:56] <gemtastic> Would you say that ElasticSearch has a similar way of storing the data as MongoDB? Feels like everything's called pretty much the same
[13:58:42] <whaley> gemtastic: no, because elasticsearch stores data in lucene index files... I'm reasonably certain mongos underlying storage engine is not lucene
[14:01:08] <gemtastic> whaley: well, it may not code-wise work in the same way, but it looks like they're trying to follow the same logical pattern on the interface end
[14:01:46] <gemtastic> I mean; ElasticSearch IS a java application after all, I'm guessing Mongo is far from it?
[14:03:54] <whaley> gemtastic: what logical pattern are you referring to btw? (I think this is largely irrelevant, but I'm in the mood for conversing)
[14:07:18] <gemtastic> I've been working kinda much in ElasticSearch (still a n00b there too though, but I've worked a bit with it and gotten the hang of it) and I can't help but to notice that they seem to try to follow the same noSQL pattern, but maybe that just IS the noSQL pattern..
[14:08:14] <gemtastic> That's what I thought, but ES seems to be following the same logical pattern for the user and references as Mongo
[14:08:32] <whaley> both mongo and elastic store documents often represented to clients as some json-like structure, but that's where I think the similarities end
[14:08:33] <gemtastic> And I don't know much about NoSQL apart from that it's anything that isn't SQL :P
[14:10:13] <gemtastic> Yeah, the internals are nothing alike, but there's benefits to making the UX (can you call it that when we're talking about developers?) consistent is beneficial since it decreases the learning curve
[14:10:24] <vagelis> StephenLynx if I use $match only, returns the results that I want but not grouped. When i insert $group.. it returns only the groups. So I guess I have to make a new field and put inside the results but how to define the results $what?
[14:52:25] <vagelis> Hello, I am trying to groupby some query results using $match but I cant succeed. I use it like this:
[14:53:41] <Derick> yes, your group stages only includes _id, all other fields you need to add yourself too
[14:54:15] <vagelis> I want: [{$match: QUERY}, {$group: {_id: "$FIELD", "RESULTS_from_match": ??? }}]
[14:55:45] <vagelis> I dont want to make avg or something I just want to show the results from the Match grouped by 1 field thats all
[14:56:12] <Derick> but how would you fix the case where you have multiple documents for each group'ed field?
[14:57:00] <Derick> I mean, if you have three docs: { a: 4, b: 5 }, { a: 4, b : 7 }, and { a: 8, b: 7 } - you do [{$group: { _id: '$a' }} ]
[15:02:21] <Derick> okay, make a pastebin, and in there: show those 10 documents, and what you'd like as result
[15:04:37] <leev> Derick: don't suppose you know how to fix "Primary for rs1/mongodb02:10001,mongodb03:10002 was down before, bypassing setShardVersion. The local replica set view and targeting may be stale."?
[15:14:42] <Derick> db.ex.aggregate( { $group: { _id: '$city', 'Results' : { '$push' : { city: '$city', population: '$population', country : '$country' } } } } )
[15:16:16] <Derick> db.ex.aggregate( { $group: { _id: '$city', 'Results' : { '$push' : '$$CURRENT' } } } );
[15:16:40] <Derick> db.ex.aggregate( { $match: { 'city' : 'Paris' } }, { $group: { _id: '$city', 'Results' : { '$push' : '$$CURRENT' } } } );
[15:17:11] <Derick> docs say "CURRENT is a system variable that defaults to the root of the current object"
[15:18:17] <vagelis> Should I use the way u just showed me OR get the docs and then loop them and hardcode the way i want them to be?
[17:08:46] <und1sk0> i'm a community user and i've had mongodb segfault, should i open a JIRA with mongodb?
[17:10:28] <und1sk0> i think we tried yesterday but weren't able to, but we crashed the db last month (using an older rev of mongodb) under similar circumstances...background reindexing
[18:06:36] <cheeser> greyTEO: i think you send an option with the right to determine whether to fail or not
[18:09:26] <greyTEO> I found this deep in the docs: http://docs.mongodb.org/manual/reference/method/db.collection.initializeUnorderedBulkOp/#error-handling
[18:25:53] <chairmanmow> I am trying to perform an update operation on a record where the values of the subdocument array will be replaced with the array contents of the incoming query. Not getting an error, figured I'd paste a link to a bit of code here in case anyone might see where I might be missing something http://pastebin.com/r0Bdyaev
[18:30:56] <StephenLynx> almost everyone that comes here with issues using node/io its because of mongo.
[18:31:49] <chairmanmow> interesting, I'm junior so I don't make these kinda calls, any reading material you might know of on alternatives/comparisons/etc to mongoose?
[18:33:34] <akoustik> chairmanmow: mongoskin is a possible alternative, if you don't actually need the DBMS sorta stuff.
[18:35:02] <StephenLynx> besides, you don't have to use mongoose, if there is enough people on the project.
[18:36:40] <chairmanmow> Yeah, I'll have to look into the criticisms and alternatives. I'll research it a bit and see if I can bring anything to my boss's attention when he's back in the office
[18:42:12] <akoustik> anyone here get mongo 3.0 running on centos 6.6? the repo on the mongo site has a libstdc++ dependency that apparently is not provided in c6.6
[18:43:05] <chairmanmow> Ahh, that looks like it might be useful, thanks StephenLynx for the tip. Now I need to see what I can do to keep myself busy today if it is indeed a mongoose issue
[18:44:00] <akoustik> StephenLynx: sorry - do you mean that's what you've run it on, or that's all it *can* run on?
[18:45:06] <akoustik> gotcha, thanks. now why we're running c6 is only known by a guy who's on vacation. haha
[18:46:51] <akoustik> yeah, i can't think of a reason not to... we shall see. so, i'm right in thinking that SSL is not supported pre-3.0, yeah?
[18:47:47] <StephenLynx> it just had an issue with cross-compiling for a while, if I am not mistaken
[18:49:15] <akoustik> ah, maybe that's the way to go. now... i don't wanna push it, but... is there any chance i can expect to get a replica set running with members running different versions of mongo?
[19:22:26] <jfhbrook> What kind of things can cause mongo to peg the cpu? Mine's at 600% and getting <10 writes per second
[19:31:47] <jfhbrook> deathanchor: any reason why my cpu is pegged and I'm getting <10 writes a second
[19:32:53] <jfhbrook> to clarify, this is a single CI instance being used by multiple APIs so the amount of connections is sane
[19:33:16] <jfhbrook> but when I started writing to this collection I was getting 100+ inserts a second
[19:34:44] <akoustik> not sure, but maybe you're using a bad index? or no index? (i don't know exactly how much that impacts write performance)
[19:34:59] <deathanchor> what about your mongotop? the only thing on your mongostat is line 6 had a bit of a lock.
[19:38:58] <jfhbrook> not sure what I'm looking for either here https://gist.github.com/jfhbrook/e37d636e0558c9929028
[19:42:14] <jfhbrook> serverStatus looks like this https://gist.github.com/jfhbrook/d6a1886c119ee3845a94
[19:47:51] <deathanchor> jfhbrook: if you used mms, you'd have nice graphs of all the data in that db.serverStatus()
[19:55:41] <jfhbrook> which makes sense given that my writes are limited by how many I can push through---I'm hitting an API and have like 8 in-flight requests at any given time, so low write perf means low throughput
[19:57:50] <jfhbrook> I had a few page fault spikes, biggest one at 0.23 (units?) but those are spikes and don't particularly correlate with overall trends
[20:23:15] <jfhbrook> okay, so what I know is that my cpu is pegged, my write throughput is outrageously low, mms doesn't think anything is alarming and the disk looks healthy
[20:51:56] <akoustik> jfhbrook: maybe you already mentioned - are you seeing similar behavior with other collections? did you say that writes used to be fast with this collection, then you started seeing this?
[20:53:46] <jfhbrook> no, no similar behavior with other collections afaik but this is also the only one I'm writing to
[20:54:27] <jfhbrook> periods of 100-200 writes per second, then periods of high cpu usage and <10 writes per second
[20:54:41] <jfhbrook> and the latter regime became more and more common until it became entirely dominant
[20:55:21] <akoustik> i'm sure you've seen this, but just in case: http://docs.mongodb.org/manual/core/write-performance/
[20:57:18] <akoustik> you might not be able to, but i'd be curious to see what happens with other interactions. say you remove some or all of the docs in that collection, how does that perform? does it affect performance of subsequent writes?
[21:01:37] <jfhbrook> I have some indices but not an outrageous amount---I believe I index id/revision and maybe mtime
[21:06:45] <akoustik> i wish i had the experience to give better suggestions, but honestly it sounds like you've gone through a pretty good number of tests. at this point, i would just be banging on it until i notice a change of behavior. drop the whole collection, insert a whole bunch of records again, update them, whatever. maybe clone the DB onto a VM or some other host, try it there.
[21:54:39] <greyTEO> I think the oplog capped colelction started to pop operations off before mong-connector could process them.
[22:11:04] <lxsameer> guys, which one has better performance generally, embeds documents vs references
[22:36:00] <greyTEO> lxsameer, it depends on what type of performance you are looking for. Embeds usually tend to be more write heavy as it will produce duplicate data. References are read heavy requiring more lookups to produce the end result
[22:36:44] <greyTEO> referneces are necessary supported by mongo, that all depends on , im guessing odm, for that performance
[22:40:11] <greyTEO> it is a bit harder to maintain and needs proper planning but shoehorning a reference feels wrong
[22:43:59] <StephenLynx> it can be done properly, but depends on how you query, can become a huge clusterfuck and kills your performance.
[22:54:26] <StephenLynx> from what I understand, its better to make your software read heavy instead of write heavy with mongo.
[22:55:21] <Boomtime> "StephenLynx: if you have to join data on the application, its awful" <- this seems to be greyTEO is referring to
[22:57:46] <Boomtime> right, doing more work to structure your documents in a way that makes reading eacy is a good idea because you almost always read more often than you write
[23:45:29] <jfhbrook> was due to some idiosyncrasies in how we do search indexing, each write requires at least one get
[23:45:52] <jfhbrook> and due to me not thinking, when I wanted to reset the database so I could try to run my migration script from scratch, I did db.dropDatabase()