Log file Viewer
#mongodb logs for Tuesday the 16th of October, 2012
[01:39:19] <veesahni> Using ruby/mongoid, anybody know how to get bson serialized representation? I'm trying to stash an object to disk if i have mongo connectivity failure (to unserialize and save! in the future)
[01:55:54] <ralphholzmann> If I want to upgrade my database from 1.8, can I simply mongodump, install the new version, and restore?
[03:18:15] <michaeltwofish> Pleasure. I'm happy and shocked that I'm able to answer a question here :)
[03:20:14] <michaeltwofish> I have a question that isn't mongo specific ... I'm trying to raise ulimit -n on a CentOS box, as mongo is refusing connections, but it's not sticking.
[03:22:13] <svm_invictvs> This may seem like a dumb quesitons, but what functions are availble when I'm running a mapreduce job?
[03:53:23] <cmatta> Does anyone know if there's a way to just get the sec value of a PHP MongoDate from inside an aggregate function? So far all I've been able to get it to output is MongoDate objects
[06:46:22] <saby> db.MyCollection.aggregate({ "$project" : { "_id" : 0 , "studyId" : 1 , "status" : 1 , "lag" : 1 , "lastChanceTime" : 1 , "startTime" : 1 , "priority" : 1 , "score" : 1 , "query" : 1 , "creationDt" : 1 , "Mode" : 1 , "updateDt" : 1 , "calScore" : { "$multiply" : [ { "$multiply" : [ { "$add" : [ { "$multiply" : [ { "$divide" : [ { "$subtract" : [ 1350366954224 , "$lastChanceTime"]} , 1350366954224]} , 0.2]} , { "$multiply" : [ { "$divide" : [ { "$subtract" :
[06:46:22] <saby> [ 1350366954224 , "$startTime"]} , 1350366954224]} , 0.1]} , { "$multiply" : [ { "$divide" : [ "$lag" , 100000]} , 0.7]}]} , { "$divide" : [ { "$subtract" : [ 10 , "$errorCount"]} , 10]}]} , db.eval(function() {if(this.priority = 6){return 6} else {return 1} })]}}},{ "$sort" : { "calScore" : -1}},{ "$limit" : 1});
[06:47:33] <saby> This query also contains a db.eval function(), is there a way to get the result of that function without using the db.eval method ?
[07:13:29] <saby> wereHamster this query executes fine when run in the shell, but when i create the same query using the java driver, the db.eval part gets enclosed in double quotes, hence making it a string and instead of executing it
[08:43:34] <pgcd> failing that, I have a (possibly quick) question: if I have an embedded document, is there any way of retrieving the container document without a find()?
[08:46:05] <NodeX> or do you mean you have stored the ID of another document in the document and want to do a join?
[08:52:10] <pgcd> NodeX: perhaps it's a django-mongodb quirk, but I can "act" on a specific reply to a post
[08:53:44] <pgcd> to be honest, I cannot be sure - some basic introspection didn't show an evident way of accessing the document
[08:58:40] <NodeX> ok and you want to get that whole document from querying db.foo.find({"replies.user.username":"Terry Carr"}); for example?
[08:59:26] <pgcd> another, which I'm having more difficulty with, is to be able to rate a single reply AND the topic it belongs to
[09:00:00] <NodeX> well by default it -should- return everything, do you have a link to the django mongo docs and I'll take a peek
[09:01:24] <NodeX> http://www.mongodb.org/display/DOCS/Updating#Updating-The%24positionaloperator <----
[09:02:43] <NodeX> http://django-mongodb.org/topics/embedded-models.html <--- I assume you did something like this and made a model for your comments?
[09:03:48] <NodeX> it's very confusing for users and tries to be to much like RDBMS / MVC when an open schema should not be so restrictive
[09:03:50] <pgcd> then I realized that if I had two collections the data got saved twice, and I didn't understand how/if it became consistent
[09:04:13] <NodeX> the short answer is I dont know enough about python / django to advise on how to reverse it
[09:05:32] <NodeX> pgcd: I imagine it's something you would have to re-create as in a new model for this specific type of query
[09:05:35] <saby> i'm making a aggregation framework query, which takes values from each row and performs calculation on them and projects that value of each row
[09:06:02] <saby> is it possible, that i can have a check condition, so that if the value of a row is this, it multiplies the value with some x
[09:07:00] <pgcd> NodeX: thanks, I will hang around - i'm sure there's lots to be learned =) (and I'll have to really understand how/when to use map/reduce)
[09:07:35] <NodeX> pgcd : avoid map/reduce if you can, I recommend aggregation framework if you can / have 2.2
[09:08:10] <NodeX> some things do need M/R but quite alot can be done with aggregation framework and it's alot faster
[09:09:20] <saby> So value of each row is finally multiplied with either 6 or 1 depending a field's value
[09:10:16] <NodeX> $match can take a value or the value of another pipeline i/e the value of some key
[09:17:01] <saby> the first $multiply is multiplying the result of all the calculations being performed with the return of the if condition
[09:18:18] <ZacS1234> anyone know what you need to do to get the mongodb node.js driver/mongoose to auto reconnect?
[09:20:20] <saby> NodeX so basically in short i have a variable called $score, if the $priority=6 then multiply $score with 6 else multiply $score with 1
[09:22:26] <NodeX> "Projections can also add computed fields to the document stream passing through the pipeline."
[09:28:22] <saby> NodeX yes we can pass them to the pipeline, but the problem comes when i need to perform a check condition
[09:33:40] <ron> NodeX: how big is your dataset? I know it's a complex answer, but I'd appreciate a general idea. I'm trying to grasp the performance mongodb gives with it as its map/reduce is less than optimal.
[09:37:03] <ron> sorry for all the questions, just trying to get a better grasp out of other people's experience.
[09:37:40] <ron> my boss wants to compare that solution and amazon's m/r but we're short on time and I think going with mongo for the initial stage should be more than enough.
[09:40:22] <NodeX> basically in that 3 or so seconds I get everything I need to know about my app - i/e every single thing that's been clicked or visited and also counts (totals) for them
[09:41:14] <NodeX> you basicaly tell it to $match - similar to find(); .. then you tell it what to do with those found docs
[09:41:29] <ron> I guess the real question is how it performs, but I imagine that on a single server it doesn't matter.
[09:42:20] <NodeX> well in terms of performance I dont worry because I aggregate every ten mins or so and only aggregate today's data
[09:42:59] <NodeX> when I finish a day's aggregation I export $gt : 3 days data to json and store it in Amazon arctic
[09:43:12] <jordiOS> hello! I have document where I store the content (per language and key as) $model->content[$language]['title']. I am looking for using ensureIndex on title for each language but I can seem to find how. Any ideas or hints?? Thanks!
[09:44:29] <NodeX> if you had it in this format .. foo : {languages : [{en:....]} ... you couldve applied it to foo.languages
[09:49:39] <ron> NodeX: I see, so the aggregations are ongoing, and the reports are generated on the previously aggregated results?
[09:50:26] <NodeX> I aggregate into a doc that's perhaps today's data then my app reads that data and generates a report
[09:51:01] <NodeX> say take the last 7 days of stats and aggregate them from these docs (not from the history)
[09:51:59] <NodeX> if I then need to get results over time I can aggregate (or loop) the daily counters
[09:52:06] <ron> and I imagine that using Hadoop it would be the same.. aggregate on going data and not by demand.
[09:52:59] <ron> you just need to make sure that the data resolution is fine enough. I imagine that yours is daily since you don't need 'by hour' aggregations.
[09:53:29] <ron> do you need to aggregate several times for different views? for example, per day, per user, per url?
[09:54:33] <kali> we generated daily and hourly aggregates and push them into a mysql. saiku slurps data from there
[09:57:38] <ron> well, right now I'm looking for a very minimal solution. I have a fixed set of reports and views I need to generate.
[09:58:14] <NodeX> if I need to go to older data I have to get it from Arctic which is very expensive
[09:59:24] <ron> I have a problem with the term 'size' though, since I have no real grasp of what's big what's small.
[10:01:58] <ron> of course. I'm not even looking for a long run solution. if I go with your method, I think it could hold us for at least a few months efficiently.
[10:02:05] <martinrue> if I store a `new Date()` in mongodb, is it automatically converted to UTC timezone?
[10:03:13] <kali> martinrue: this is a client issue actualy, what goes over the wire has to be an utc timestamp
[10:03:25] <martinrue> NodeX: when I do `new Date()` in JS, that's local time… but in MongoDB is shows as UTC
[10:03:44] <saby> NodeX in the previous scenario, where I was trying to use If conditions, instead of using that, I would be inserting another field like priorityMultiplier and the value would be pointing to an id number and the value of that id number would be present in another collection
[10:03:46] <martinrue> kali: does this mean the mongodb-native driver in JS is likely adjusting my local time dates to UTC before storing them?
[10:04:01] <saby> this leaves me room for having the value of that multiplier configurable for later purposes
[10:04:55] <NodeX> the computed variable you're looking for must come out of either a document or be projected to a variable
[10:06:17] <saby> i would be storing the id number of the other collections document in $priorityMultiplier
[11:16:25] <fotoflo> hello all. I have a collection playlists with an property videos. some videos have a property comments -- how do i select those comments?
[11:18:51] <NodeX> db.foo.find( { some:"query" }, { comments : 1 } ); <--- that's just to get back the comments
[11:22:08] <fotoflo> db.foo.find({videos: {$exists : true} }, {comments: {$exists : true} } ) not working
[11:29:14] <fotoflo> i can't do that, but i can tell you i would want to access playlist.videos[1].comments
[11:46:11] <remonvv> Why on earth can't you paste an example document? The odds of someone running away with a schema that currently doesn't even work are slim at best.
[11:49:25] <NodeX> http://www.dailymail.co.uk/news/article-2218387/EXCLUSIVE-Gary-McKinnon-saved-extradition-Theresa-May.html
[11:55:19] <remonvv> If you mean what I think you mean, then no. MongoDB is not more efficient with reusing diskspace that is no longer used since 2.0+ that I'm aware of.
[11:59:08] <remonvv> At least take the time to come up with a similar document if it's full of passwords or something
[12:00:14] <NodeX> the ones I like the most is "Mysql can do it so why can't mongo .... or APache can do it so why not nginx" ... my answer is "Go use Mysql / apache then" LOL
[12:16:07] <remonvv> http://stackoverflow.com/questions/7501100/mongodb-not-that-faster-than-mysql/7501739#7501739
[12:58:40] <omie__> suppose I have some data with keys/columns name,surname,city. If I try a query like: {'$or': [ {'name': {'$ne': 'xyz'}}, {'city': {'$ne': 'newyork'}} ]}
[12:59:19] <omie__> it doesnt work right. I mean what I expect it to do is exclude all the records if either name is xyz or city is newyork
[13:04:38] <omie__> it works like as if it was OR, takes out all the records where either of conditions is true
[13:05:50] <omie__> so far I know only this. you want me to investigate more ? how should I do that ? (I am new to mongodb, also non relational databases)
[13:41:35] <omie__> suppose I have 4 records in a collection. (john, newyork), (kelly, newyork), (john, chicago), (kelly, chicago)
[13:46:36] <omie__> actually I am working with django-nonrel. Initially I thought problem is with django's filter() and exclude(). but later I checked the queries created by django and tried them on mongo shell
[13:46:51] <NodeX> no, it's doing this... (SQL) SELECT * FROM foo WHERE name !='john' OR city !='newyork'
[13:47:30] <NodeX> from your documents which are you trying to select and which do you want left out...
[13:50:11] <NodeX> The $or operator lets you use a boolean or expression to do queries. You give $or a list of expressions, any of which can satisfy the query.
[13:51:37] <omie__> can you take a look at this python script I used for testing: http://pastebin.com/TyTuJ29i
[13:52:56] <omie__> in image, on left its normal django using sqlite3 and on right its using django-nonrel (mongodb)
[13:54:39] <NodeX> Warning : You cannot use $where or geospatial operations in $match queries as part of the aggregation pipeline.
[13:57:52] <omie__> NodeX: for ANDing part exclude, its [{'sql': "<MongoQuery: {'city': {'$ne': 'Ratnagiri'}, 'name': {'$ne': 'Omkar'}} ORDER []>"}]
[13:58:11] <omie__> for OR'ing part exclude, its: [{'sql': "<MongoQuery: {'$or': [{'name': 'Omkar'}, {'city': 'Ratnagiri'}]} ORDER []>"}]
[14:00:54] <omie__> I expect the output same as on left. What has happened is o/p of AND-Exclude should be o/p of OR-Exclude
[14:11:00] <omie__> its [{'sql': "<MongoQuery: {'$or': [{'name': {'$ne': 'Omkar'}}, {'city': {'$ne': 'Ratnagiri'}}]} ORDER []>"}]
[14:12:54] <omie__> yes, there are multiple 'Omkar' and multiple 'Ratnagiri' in data. is that what you asked ?
[14:14:50] <omie__> see, the current data I am showing you. I made it for simplicity. I am making a simple log-viewer. and there are really very simple filters I wish to apply
[14:15:33] <NodeX> that's great but you have not told me still what data you want back from the queries and why what it returns currently is wrong
[14:16:11] <omie__> for example: ( DEBUG, "some message", "actions.py", "line-no=28"). If there is any such thing in logs, I want to exclude it
[14:16:22] <NodeX> "The $or operator lets you use a boolean or expression to do queries. You give $or a list of expressions, any of which can satisfy the query."
[14:17:53] <omie__> if I stick to MongoDB's logic of boolean operation, it won't work if I switch my backend
[14:18:10] <NodeX> {'$or': [{'name': {'$ne': 'Omkar'}}, {'city': {'$ne': 'Ratnagiri'}}]} -> translates roughly to .... SELECT * FROM foo WHERE name!='Omkar' OR city!='Ratnagirl'
[14:19:11] <NodeX> if you want SELECT * FROM foo WHERE name!='Omkar' AND city!='Ratnagirl' .... then that's a different query
[14:20:17] <omie__> I totally get your this point. but the problem is, its working opposite if compared to other backends.
[14:21:55] <omie__> my code should be able to do that, right ? I mean I don't get why boolean logic should change if I switch backend
[14:23:15] <NodeX> about what - reversing your code to make mongo statements work in another backend?
[14:25:48] <omie__> to be honest I don't mind staying with MongoDB right now. I'm much satisfied with it
[14:26:16] <NodeX> it's a nominal idea to be able to switch backends quickly but it bloats your code and is less efficient
[14:27:21] <omie__> thats what django framework is supposed to do ! I only have to change backend string and no changes needed in ORM/my code
[14:30:19] <omie__> hmm.. will look into that later sometime. for now I need to get this thing to work !
[14:51:24] <ThomasJ__> If I have a shard that's extremely low on disk space, what can I do to make sure it doesn't completely run out of space?
[14:55:16] <ThomasJ__> It's only this one shard that's full. Most of my other shards have lots of space
[14:59:03] <bgilbert> Hey guys, I'm seeing a strange issue using the java driver......I keep running into the following exception: java.lang.IllegalArgumentException: response too long: 1634610484, when the driver tries to create a response from mongo
[15:00:01] <bgilbert> this seems to be happening at random, and happens in cases where mongo shouldn't be returning any results from the query......and the size reported is inconsistent and normally way larger than it should or could be
[15:00:45] <bgilbert> I'm guessing some form of corruption with the response.....has anyone seen an issue like this or have any suggestions on where I should look next to further debug this issue?
[15:00:52] <ThomasJ__> I tried draining that overloaded shard but I get "Can't have more than one draining shard at a time" (I am draining another one as well). Is this a hard limit on mongo?
[15:49:04] <Industrial> At what scope should I be opening connections to MongoDB though node-mongodb-native? application scope (1 connection per app) or request scope?
[15:58:23] <Mmike> I'm upgrading mongo in my cluster - i upgraded one secondary and I see 'sycing to' in it's web interface
[15:59:00] <Industrial> what im really trying to avoid is wrapping my whole application in a db.open(cb). Is there a way to do this once? (Iḿ cleaning up with db.close on process.exit/SIGINT)
[16:04:57] <Mmike> Also, on the server i just upgraded I see it's syncing to another secondary. But on that (another) secondary, and on primary, I don't see 'syncing to' status on any box
[16:26:29] <Bartzy> Is it possible (through the drivers or otherwise) to try to insert a document, but if it exists (i.e. a unique index with that value already exists in the collection), just return the _id ?
[16:28:01] <algernon> Bartzy: but you can try an insert with safe mode, and if that returns an error, do a query on the unique index and return whatever you want.
[16:28:38] <rybnik> Hi there fellows, the mongoosejs channel is rather idle, so perhaps someone here can give me some advice on a "mongoose schema design", I've created a gist describing the problem, anyone care to give it a try ? https://gist.github.com/3900332
[16:28:46] <Bartzy> algernon: Why not try to query that unique index , and if nothing returns - inserT ?
[17:10:52] <thewanderer1> hi. let's say I have a movie database, each movie has an "actors" array. now, I want to obtain a list of actors and movies each has played in. how would I do that?
[17:11:25] <thewanderer1> it's easy to do if I want the movies in which "Tom Thanks" has played, but what if I need this for all actors? looping through them all and issuing queries like mad doesn't sound wise.
[17:12:19] <kali> thewanderer1: you can look at the aggregation framework if it is an occasional query
[17:13:56] <kali> thewanderer1: mmm... well, the implementation is actualy very different. m/r relies on a JS interpreter, whereas the AF is a native implementation
[17:14:20] <thewanderer1> the most straightforward way is to create <actor,movie> pairs and then reduce that to actor documents, I imagine... would the AF do that?
[17:14:44] <thewanderer1> (and it doesn't really sound very efficient, maybe I'm not using the right tool for the job... SQL comes to mind as "righter")
[17:15:44] <twrivera> kali: i'm getting the date values in json format for ex: /Date(1250121600000)/ from an API and I want to insert that value into mongo docs
[17:18:03] <kali> thewanderer1: not really, you're really hitting an algorithmic problem. in bost cases, you're down to a sort of one of the collection, so you're O(n*log(n))
[17:19:28] <thewanderer1> well, yes, the datasets (actors, movies) would need to be sorted in both SQL and Mongo
[17:20:11] <kali> twrivera: i can't help with python, all i remember about manipulating dates in python is a lot of pain
[17:21:22] <thewanderer1> actually, it's not that bad... I only wonder about the space needed (disk, RAM). would Mongo or SQL have any advantage in this regard?
[17:22:11] <twrivera> kali: I'm learning that the hard way but how would you store the dates in the db? as string or timestamp object? I figured out how to convert the epoch to a string and I can convert from there
[17:23:25] <kali> thewanderer1: to be honest, the thing that strikes me is... mongo and *sql are more designed for "small" queries that impact only a small part of your dataset
[17:26:55] <thewanderer1> kali, I'm asking all this because I'm writing a task assignment system (human resources, all that stuff), and need to show who's doing what, etc... and aggregation plays a key role here
[17:27:31] <thewanderer1> and Mongo seems really great because of document storage (rich information), only aggregation so far is troublesome
[17:28:01] <kali> thewanderer1: sure. you need to have a look at the aggregation framework. it's much better than map reduce, but will explode if the dataset is too big
[17:29:43] <kali> thewanderer1: "If any single aggregation operation consumes more than 10 percent of system RAM the operation will produce an error."
[17:29:58] <kali> thewanderer1: http://docs.mongodb.org/manual/applications/aggregation/#limitations
[17:31:43] <kali> thewanderer1: AF limitations are even worse. no concurrency, and it's slow as hell
[17:35:05] <thewanderer1> so it does seem that AF is the go-to technology in this case... now I only need to get Mongo >=2.1 (Debian stable is still at 1.4) and I'm all set. thanks!
[17:36:03] <kali> thewanderer1: yes. AF is designed to avoid 99% of m/r use cases, and for good reasons
[17:50:23] <kali> ralphholzmann: you want a third one, or an arbiter. two machines is the worst setup ever
[17:51:15] <_m> Mmike: To get a quick idea, try this command from the mongo shell: db.printSlaveReplicationInfo()
[17:52:57] <kali> ralphholzmann: the problem is that when one of the server can't talk to the other, it has no way to make the difference between a network split or a host failure
[17:53:15] <kali> ralphholzmann: so in order to avoid the dreadfull split-brain, it has to go down too
[17:53:16] <Mmike> kali, where do I get those? I'm checking the _replSet web page on each server, but they doesn't seem to give me the same data. On srvA i see that srvB is syncing to srvC, but I don't see that on srvB and srvC
[17:53:42] <Mmike> _m, i get something like this: source: ded778:27017\n syncedTo: Tue Oct 16 2012 13:44:23 GMT-0400 (EDT)\n = 340 secs ago (0.09hrs)
[17:55:00] <ralphholzmann> so would you then also suggest that I keep my servers in separate data centers?
[17:57:01] <kali> ralphholzmann: i'm sure you need an arbiter or a third server. as for datacenter, it comes with its bundle of costs and issues. i won't answer that question for you
[18:00:45] <kali> Mmike: in the /_replSet, at the top of the page, there is a table with the state of your servers
[18:03:32] <Mmike> that is on ded778, but on other boxes, under ded788, I don't see that 'syncing to'
[18:09:14] <ralphholzmann> kali: http://docs.mongodb.org/manual/administration/replication-architectures/#geographically-distributed-sets
[18:11:02] <kali> ralphholzmann: yep. start by adding an arbiter, your two-node cluster is making me nervous
[18:17:35] <kali> you haven't make the screenshot in the same millisec, anyway, so i guess they are stuck
[18:23:53] <kali> you were at 1450s. 22s is not that bad, but you'll loose the updates made in the lag
[18:25:21] <mheld> is there any way to manually open and close mongo connections in java (as opposed to using the connection pool)?
[18:37:43] <Mmike> kali, well, someone thought it would be neat for mysql and mongodb to share the hardware
[18:38:21] <jmar777> any chance that someone has some input on http://stackoverflow.com/questions/12917943/casbah-java-mongodb-driver-java-lang-illegalargumentexception-response-too-l ?
[19:03:15] <uroborus_labs> Probably a dumb question, but what would be the most appropriate way to store appointment book data in mongo?
[19:13:41] <_m> uroborus_labs: Have you perused the docs on storing documents in mongo? You'll probably want a field to represent a 'user id' and fields for whatever appointment data you care to store.
[19:14:32] <uroborus_labs> I think my main question whould be, how much data to actually store in the document itself or how much to infer from the application itself
[19:16:44] <uroborus_labs> Or would you store availability and what appointments have been made and then show what slots are open from that data on the application side
[19:47:30] <pgcd> if anybody's using django-mongodb: is there any (API-based) way of finding an object in a list of embedded objects? I mean, if I have the usual Post with comments, can I find a specific comment by ID?
[20:24:50] <ashley_w> i have some strangeness in my program since i added some code yesterday: http://pastie.org/5069647
[20:39:57] <jmar777> i know there used to be some issues with using indexes on a count() operation. is that still the case?
[21:50:51] <mgriffin> by do not stop, i mean also perhaps disable cron/daemons that can do that (monit, etc)
[21:53:00] <rybnik> Bilge: just a quick question, I'm rather interested in this…. if you perform a db.coll.find() will this have any negative impact on the cache ?
[21:57:45] <mgriffin> probably also i would want to do something like mongodump or something that does a logical export of the data (i dont know how to back up mongo well, someone else comment)
[22:01:08] <mgriffin> yeah, it seems that mongodump is a really good idea probably, since it seems to: connect over socket, request data from running instance
[22:01:38] <Bilge> Well I managed to copy my database back from the file descriptors like in the article
[22:03:03] <Bilge> As it happens this is just dev data anyway so it's all expendible, but it would still be a pain to lose it. I'm really impressed by your knowledge mgriffin :)
[22:05:34] <Bilge> Yeah I'm actually glad I deleted the wrong directory because I learned something invaluable in the process
[23:06:22] <cjhanks> So the C++ doxygen is entirely out of date an incomplete, is there any better source?
[23:44:27] <kenyabob> new guy here. I've used php to create a list of json objects I want to import into mongodb, not sure what the simplest way to to just feed in this comma delineated list of objects.