Log file Viewer
#mongodb logs for Friday the 24th of April, 2015
[00:54:16] <Boomtime> ToeSnacks: hmm.. 2.4.12 is a bit tricky to inspect, can you gist/pastebin sh.status(), sh.getBalancerState() and sh.isBalancerRunning()
[00:54:55] <Boomtime> balancer state is an aspect that was specifically addressed in 2.6+ due to it being a bit difficult to get a good view of exactly what was happening
[03:43:21] <arussel> what does that mean: "exception: The argument to $size must be an Array, but was of type: EOO"
[05:07:02] <Gevox> I'm trying to reterive a field into a document using one of the keys inside a document, i.e get email of user who has id of "x". The output is given in a json format, how can i retreive the value only and store it into a string? this is the output { "email" : "medo@live.com"}
[05:11:30] <joannac> Gevox: for each element in the cursor, pull out the value of the element "email"
[05:12:03] <joannac> in pesudo javacode: while cursor.hasNext() { mydoc = cursor.next(); myemail = mydoc.email }
[05:12:16] <Gevox> jonnac: I'm stuck really, this is what i want. Can you take a peek on my method? http://pastebin.com/bs3wWguv -- exactly line 19 is what i want to create the object.
[05:14:09] <Gevox> jonnac: I'm getting the right field man, but the format isn't correct. I'm getting a cursor.hasNextt() return which is like this { "email" : "medo@live.com"}
[05:18:33] <Gevox> at this line System.out.println("OUTPUT: " + emailCurosr.next().get("email").toString());
[05:19:21] <Gevox> and in my failed attmpt to pirnt the email value i do one more cursor movement, so i point to a null value
[05:19:47] <Gevox> I should have do all in the same line like this System.out.println("OUTPUT: " + emailCurosr.next().get("email").toString());
[05:21:40] <Boomtime> Gevox: why do you need to do everything in a single line? are you not allowed to use carriage-returns?
[05:22:30] <Boomtime> if you do one method call at a time you will be able to tell which one is failing
[05:27:15] <Boomtime> btw, if you print out each step via toString(), you might be able to figure out what is happening at each step
[05:27:41] <Boomtime> this should give you some idea of what your working with at each point in your code
[08:00:06] <Gevox> Hi, i'm making a method which retreives all of the documents found into a mongoDB and stores them into an ArrayList, finally return that list. However, it seems that the method only reads the 1st document in the collection only and it does not even store it into the specified list. Line 27 is never printed. Can someone figure out the issue? http://pastebin.com/cJd0gePa
[08:26:46] <MongoUser001> I have this appearnign in my log and it's hanging my filesystem for some reason ... http://pastebin.com/Rie7pKav
[09:07:49] <Askilada> Hi, I'm working on a project using mongodb-ruby driver. I'm have googled how to use ISODate in a Aggregation, but I can't figure how in rails, I can do it in my mongoDBQuery
[09:19:24] <luffy_> (im using django) so in a function i run a map_reduce , and depending on the function variables i want it to select between a set of reduce_functions (1 map, several reduce)
[10:28:23] <rearth> Hi, how do i do a find with regexes on one field with AND, see http://pastebin.ca/2980173
[10:37:04] <StrangeBrew> hi, i'm new to mongo, and I was wondering..why would someone use a service such as mongolab/mongohq instead of running it on his server?
[10:37:53] <Derick> for the same reason you would use AWS instead of your own hardware: maintenance time
[10:39:00] <Derick> it's likely also easier to scale up, as you wouldn't have to wait for new hardware to show up
[10:39:10] <kas84> Hi, I’ve just installed mongodb 3.0.2 on my server, how can I tell which version is running right now?
[10:39:58] <Derick> StrangeBrew: maintenance isn't that much, but you need expertise, and it's not 0 time
[10:47:32] <smogg> I’m working with nested array and I’d like to add-or-update values, can anyone help me how to do that with this kind of collection: https://gist.github.com/smogg/abcf5aff7d02eac265a0 ? I basically want to update user’s value IF he exists in the “players” set, or add new user with value if he does not.
[10:51:01] <Derick> kas84: in that case, mongodump with 3.0.2 without WT - stop mongod - start mongod with WT engine - mongoimport
[10:53:08] <sudomarize> Derick: i'm trying to implement a category search feature using mongo as the DB. So far my Category model looks like {name: "Category", parent: "9kh2kjahdfew"}, where Categories.findOne({name: "Category"}).tree() returns the parents and children of that particular category. I've wondering how i can implement an autosearch where it would say "Walls & Fences in Repair work" or "Pies and Pastries in Food", etc
[10:53:51] <sudomarize> currently when users search for a category, it returns only the category name, e.g. "Walls & Fences" (without the context of "repair work").
[10:55:11] <Derick> {name: "Walls & Fences", parent: "Repair Work" } << wouldn't that be your model? Seems trivial that way
[11:37:10] <mick27> anyone around using mongodb and elastic search ? I wonder what are people going to use to move away from rivers starting 1.5
[11:41:21] <BurtyB> Is there a way to search documentation for old versions ? when I set the the version to 2.6 (via options) and search it reverts to 3.0
[11:57:43] <sudomarize> Derick: would it be better to use the parent._id for the parent key in my document, or does it not matter?
[12:10:04] <_rgn> I'm inserting a bunch of objects in parallel and getting duplicate key errors for _id, is the right way to always insert sequentally in the same collection or?
[12:19:03] <sudomarize> Even when I try to update a document, and it returns 1, the parent key is still undefined??!?!??!
[13:04:43] <razieliyo> I mean, db.data.find({}, {somefield:1, _id:0}), this has a projection so that only 'somefield' is selected
[13:05:00] <razieliyo> the question is, how to get the distinct values of 'somefield' from that set?
[13:05:50] <razieliyo> yep, I tried to take a look at distinct function but I think it has other usage
[13:06:30] <StephenLynx> http://docs.mongodb.org/manual/reference/operator/aggregation/addToSet/#grp._S_addToSet
[13:09:06] <razieliyo> errr, StephenLynx would this work if the find query is not {} but {otherfield:'value'}?
[13:10:03] <StephenLynx> a find is an operation where you query and project. an aggregate is a flexible operation where you can do both of these plus a lot more.
[13:10:14] <watmm> One of those "how long is a piece of string" questions, but how long would you say is too long, for a flush time?
[13:11:00] <StephenLynx> watmm as much as it impacts the application under its regular requirements.
[13:13:09] <StephenLynx> how much concurrent operations are you expecting? what operations you need to realize.
[13:26:10] <Jester01> I wanted to do mongo foo.js --eval 'use stuff from foo.js' but apparently mongo does the eval first ... any workaround ideas?
[13:59:13] <sudomarize> what is the best method of creating a category system re: http://docs.mongodb.org/manual/tutorial/model-tree-structures/? can i get some advice on this?
[15:48:48] <pjammer> if you have 1 primary and 2 secondary boxes, all using the same AWS image for their /data directory; does that quicken up rs.add() for the two secondary boxes? It's been 12 hours at least, i must be doing something wrong.
[15:53:05] <pjammer> thanks @cheeser i figured but this is the second time i tried as i was too 'anxious' last time and followed the log messages i was getting on the secondaries, turning them into primaries...etc
[15:55:03] <pjammer> each box i want as a secondary should also be started, and using the replset= in mongo.conf right? docs don't say explicitly that but hint at it.
[15:58:46] <cheeser> using mtools on my local box, i can bring up 3 mongoses, 3 shards with 3 member replSets, and 1 configserv in < 15s
[16:04:05] <mtree> hey, i need to update object selected not by id but by date AND string field, how can i achieve this?
[16:04:32] <mtree> i know i can make an query before and get id of that field, but is there any way to avoid this?
[16:12:21] <deathanchor> anyone use tokumx on centos 7? I'm having issues starting it up using systemd. :(
[16:44:42] <EdwardL> hi everyone, I'm seeing high CPU usage across my mongos processes -- about 25% CPU load when I'm running stress tests. The mongos processes are running on our application servers and only handle simple queries (find by id -- yes, it's indexed as well). Is this normal?
[17:14:17] <JokesOnYou77> Hi all. I'm using pymongo (3.0.1) and I'm trying to loop over a cursor and modify each document in it but I keep getting strange errors and nothing I've found on line is helpful. I can't even tell if using $set in the loop is modifying document order in the cursor
[17:19:03] <juliofreitas> OMG, I've a simple question. I'd like to select all values of my field cities. I saw the documentation and there is query just for more complex query such as or, and, gte, lt... For example: select cities from collection
[17:24:34] <JokesOnYou77> GothAlice, Do you have any advice for looping over a cursor and modifying each element with pymongo? A simple for doc in cursor: do stuff... db.collection.update_one(blah blah) has been causing problems for me
[17:25:19] <juliofreitas> cheeser, db.myDB.find({cities: "city1"}), I want to select all cities, not just city 1...
[17:38:23] <cheeser> juliofreitas: http://docs.mongodb.org/manual/reference/method/db.collection.find/#projections
[17:39:16] <EdwardL> hey guys, not sure if you had a chance to look at the question i posted about an hour ago (probably got pushed out of everyone's screens)
[17:40:45] <EdwardL> I'm seeing high CPU usage across my mongos processes -- about 25% CPU load when I'm running stress tests. The mongos processes are running on our application servers and only handle simple queries (find by id -- yes, it's indexed as well). Is this normal?
[18:18:45] <EdwardL> of course, CPU usage is dependent on hardware but it's been consistently around 20-25% across EC2 and bare metal machines
[18:21:38] <StephenLynx> I don't think that using 25% of CPU is bad then. are you having issues with performance? have you read from a source that it should be using less?
[18:25:02] <EdwardL> we're not having issues with performance just yet -- we're planning to migrate databases and we noticed mongos eats up quite a bit of CPU... our application servers already use 70% CPU during peak times and so adding another 25% on top of that would not leave a lot of overhead
[18:25:55] <EdwardL> no, i haven't read from another source that it should be any lower, i wanted to see if there was any thing we could do to possibly lower that figure
[18:27:35] <sudomarize> StephenLynx: the thing is i'm not really sure what the 'basics' are when it comes to a category system in mongo
[18:28:09] <StephenLynx> it isn't advised to keep the db on the same server of the application on heavy applications.
[18:28:30] <StephenLynx> if you had that db by itself on his own machine, then 25% wouldn't be a problem at all.
[18:29:20] <EdwardL> the DB cluster (shards and replicas) sit on their own machines -- only mongos is sitting on the application machines
[18:31:23] <sudomarize> StephenLynx: how would you recommend storing my categories? in the actual product Document that has a category you mean?
[18:31:30] <StephenLynx> "To resolve DBRefs, your application must perform additional queries to return the referenced documents. "
[18:33:54] <sudomarize> i don't understand why using those models isn't recommended tbh. Some of them seem like they'd do a good job: http://docs.mongodb.org/manual/tutorial/model-tree-structures/
[18:36:22] <StephenLynx> I am just saying there isn't any actual implementation in mongo to make sure you are referencing correctly or that will resolve these references in the db layer.
[18:36:46] <StephenLynx> that example is not an example of mongo doing things, is just a thing you can do with mongo.
[19:28:11] <fewknow> sudomarize: without knowing the query pattern or how you wish to update the data it is hard to design a model
[19:28:57] <fewknow> they differ in a lot of ways for mongo...depends on what needs to perform well and what doesnt'...depends on if you need to relate to other objects....
[19:32:58] <sudomarize> fewknow: right fair point. the context is i'm working on an app for doing tasks locally, and i want a user to be able to search intelligently for categories. An example category tier would be Home -> Repair -> Flooring, so when a user types say "repair" into the input, it will return a list of categories that are children of the Repair category
[19:35:48] <fewknow> then you would have chidren of children...with out an actual relation between the documents
[19:36:21] <sudomarize> fewknow: yeah tried to set that up, was having a bit of trouble so just defaulted to the mongo solution for now.
[19:36:58] <sudomarize> fewknow: like this? http://docs.mongodb.org/manual/tutorial/model-tree-structures/#model-tree-structures-with-an-array-of-ancestors
[19:39:32] <StephenLynx> do your objects can have multiple categories or have either a parent or children category in the same field?
[19:40:15] <StephenLynx> but I would also have a separate collection for these categories so you could be able to know what you can search.
[19:41:16] <sudomarize> StephenLynx: ah ok so in my Task document i'd have a parents and children array, and then i'd also have a Categories collection?
[19:42:32] <StephenLynx> the object that belongs to a category may have in its category field flooring (subcategory) or it may have repair too?
[19:44:02] <fewknow> more complicated then the application has to keep everything in sync with parent and child
[19:47:23] <fewknow> you need to have that control....another reason why developers shouldn't be in the data layer
[19:47:52] <StephenLynx> what I would do in your case: have several fields, each one for a category level. so a document would have supercategory: repair, subcategory: flooring
[19:49:05] <fewknow> or you could use elastic and just insert the document(using log-stash) ....then it automatically is ready to search for cateogories....
[19:50:13] <StephenLynx> I would store each document as the highest-level category and have it containing an array of sub-categories which could also have each its array of subcategories.
[19:51:23] <sudomarize> I definitely think ES is the way to go, i suppose i just need to figure out the model for my Task collection in that case. so StephenLynx, you suggest each Task document should have {name: "midcategory", parent: "parentcategory", child: "childcategory"}
[19:52:34] <fewknow> you just need to insert the data into ES the right way.....but that is a different storry
[19:55:11] <sudomarize> StephenLynx: is that significantly inefficent enough to choose another method? Sorry for all these questions, but i'm basically new to mongo and to elasticsearch, so trying to implement the best solution is proving quite difficult
[19:56:44] <StephenLynx> and considering you would have to perform several queries to find out what is the super category, I believe that duplication is the best approach.
[19:58:53] <sudomarize> fewknow: i get that it's REST, just wondering how the data is stored (or perhaps 'in what format' is a better way to phrase that).
[20:00:29] <sudomarize> my understanding was that i would need to set up a stream from the meteor mongodb, and then you can search ES independtly of mongo (Taking some load off the application)
[20:01:24] <StephenLynx> even though it is very relational, it can be solved with a little duplication and pre-aggregation.
[20:03:36] <StephenLynx> if one day the shithead decides he want's recursive categories, it won't do.
[20:05:34] <StephenLynx> the object that belongs to a category has just one category that may be n levels deep in category hierarchy.
[20:06:41] <StephenLynx> no, but is pretty much unmaintainable and would require several queries to resolve depending on your requirements.
[20:09:31] <StephenLynx> aside from "javascript" it has all the magic words that would make me kill it in a fire.
[20:11:25] <StephenLynx> it is just bundling a bunch of unrelated stuff, again it may add bloat and removes flexibility.
[20:11:52] <StephenLynx> then framework: it is just doing every day things that already provides a high-level interface for development, so its just bloat.
[20:12:45] <StephenLynx> I have been using nothing but io.js and the node.js driver, you really don't need anything else.
[20:13:48] <sudomarize> honestly ideally i would be using those, but the development speed that meteor provides is (thus far) well worth the bloat
[20:15:02] <StephenLynx> addings things that don't do anything that couldn't be already done only increases the learning curve and add sources of problems.
[20:15:59] <StephenLynx> after these pieces of reusable code are written, you just have to bother with business logic.
[20:17:44] <StephenLynx> unless you are a highly experienced developer in these specific tools and is thinking in the short term.
[20:19:26] <StephenLynx> as I said, if a framework only does stuff that could be already done, then its useless.
[20:20:01] <StephenLynx> extending a system is just as easy because you have already written the reusable parts.
[20:20:14] <sudomarize> how do you do routes for instance, using only node? (without writing a large amount of code)
[20:23:38] <StephenLynx> if people would bother to actually try and work without these frameworks, they would see how much they don't need them.
[20:26:15] <StephenLynx> 5 if you take in account that one line is used for some unrelated business.
[20:28:21] <sudomarize> StephenLynx: im certainly not as skilled a programmer as you are, i imagine doing all this myself would take me forever
[22:04:04] <daidoji> Mongo seems to eat up all my disk space beyond the data sizes of the underlying databases
[22:12:19] <daidoji> oh I guess I should have been doing this? http://blog.mongolab.com/2014/01/managing-disk-space-in-mongodb/
[22:29:52] <abishek> right now I am doing a group by date and I am generating the missing timelines on the application server.
[22:57:21] <daidoji> abishek: keep everything in epoch time in Mongo and treat it like an integer. The tooling doesn't really play well with Date() objects.
[22:57:48] <daidoji> abishek: as for generating a time series, shouldn't you just do that in Javascript or whatever language you're using to access Mongo?
[22:58:23] <abishek> yes, that is what I am doing, but it is a bit slow when the datetime range is large
[23:01:05] <abishek> that problem that I need to solve is, if I am generating the time series by hour, and if a particular hour is missing on the grouped data, then I need to set that hour as 0. am not quite sure on how to do that when using mongo
[23:03:35] <daidoji> oh, generate some collection like "y: <year>, m: <month>, d: <day>, hour: <hour>"
[23:04:20] <daidoji> alternatively, your chosen programming language should let you define some such sequence similarly and just aggregate in your application
[23:05:25] <abishek> on a map reduce can I retrieve data from another collection? could you give me an example?
[23:08:51] <abishek> from a simple googling, i understand that this might not work in a sharaded env
[23:09:18] <abishek> i meant, how to access another collection from within a reduce function. is this possible?
[23:11:43] <abishek> what is the last step that you mentioned `reduce(res)`. is this on mongo or on my app server ?
[23:11:56] <daidoji> abishek: well here. http://tebros.com/2011/07/using-mongodb-mapreduce-to-join-2-collections/
[23:12:13] <daidoji> if you want to have a mongo only solution you can "join" two collections using some form of this pattern
[23:12:50] <daidoji> imo, its usually much easier to do all the aggregations/calculations in your app than muck around with Mongo Agg framework or map/reduce
[23:12:52] <abishek> yes, its pretty easy except for the performance issue that I am having and I am trying to see if this can be solved from within the DB
[23:13:20] <daidoji> abishek: well then best bet will be to denormalize or make your collection data the right shape for whatever calculations you're trying to perform
[23:13:44] <daidoji> imo map/reduce or agg framework aren't very powerful tools for doing stuff like that