Log file Viewer
#mongodb logs for Tuesday the 28th of January, 2014
[00:26:19] <NaN> can I do a push to a value that doesn't exist jet? I mean, it is like upsert or something?
[00:55:05] <george2> I currently have these functions http://paste.chakra-project.org/6322/ saved to my db.system.js. Can anyone help me figure out how I should run them over my dataset http://pastebin.kde.org/pc163fbcb/08004631/raw ?
[00:55:05] <george2> I want to collect all the posts matching userID "X" and put them through gyration() to get a radius of gyration measurement for each individual user. (pseudocode -> http://pastebin.kde.org/pa6a1e057 )
[00:55:05] <george2> The "frequency" mentioned in the JS is the number of data points for the user at location (lat, long)
[00:55:05] <george2> I'm unsure whether I need to do something with map/reduce (which I haven't used before), or if I can just use an aggregation pileline, or if I need to do something entirely different.
[00:55:05] <george2> If it matters, I will be running this over a few hundred million data points, with probably a hew hundred thousand unique userIDs.
[00:55:06] <george2> I don't want to spend hours on this only to realize I'm going about it completely wrong, so if somebody could point me in the right direction, I'd appreciate it. :)
[02:04:50] <steve1> Hyperking: try this: https://stackoverflow.com/questions/4146452/mongodb-what-is-the-fastest-way-to-update-all-records-in-a-collection
[02:08:42] <Hyperking> looking at the mongo docs. http://docs.mongodb.org/manual/reference/method/db.collection.update/
[02:13:33] <Hyperking> that obviously doesn't work. but i have no clue on how the syntax should be written
[02:15:28] <george2> > db.data_test.aggregate({$group: {_id: "$userID"}}).result.forEach(function(i){db.data_test.find({userID: i._id});})
[02:21:34] <jamiel> Hi all, db.serverStatus() is reporting 65 open connections. Could anyone recommend a way to debug where these are coming from? Currently there is little to no utilisation on my application, although I do have replication enabled to a single secondary node.
[02:21:40] <george2> Logicgate: http://stackoverflow.com/questions/2824157/random-record-from-mongodb
[02:33:27] <george2> > db.data_test.aggregate({$group: {_id: "$userID"}}).result.forEach(function(i){ a.push(i._id) })
[02:33:27] <george2> a has the correct data. but that's going to be a lot of information to store in a temp variable
[02:42:46] <sheki> i am trying to insert data into a mongodb and Update the object using another connection in quick succession,
[02:57:30] <Hyperking> I have a key with a ISODate() value and need to update my collection to show a different key. How would you call a method new Date() within the update method?
[04:52:11] <Hyperking> Any way to grab data from one collection to create another? I want to generate a menu that takes posts with post_type: "page" and add them into a insert for a new collection.
[04:52:37] <cheeser> it helps to think of what you'd *normally* want to fetch when getting, e.g., that user
[04:53:14] <cheeser> if you normally wouldn't want all those assets every time you loaded a user, it's usually best to store them separately and reference the user from each asset
[09:39:31] <ollivera> ncls, I had problems with pymongo but I managed to insert a file with mongofiles
[09:58:05] <NyB> How would I go about testing my Java BSONDecoder implementation? Are there any MongoDB database dumps somewhere that would exercise all corner cases?
[09:59:23] <ollivera> ncls, would it be possible to define paths? it seems that I always insert in the same location
[10:04:07] <ncls> you might put a file on the "name" field like { "name": "/my/file/path/name.ext" }
[10:08:18] <ollivera> ncls, ok .. so I can use that "path" in the URL? Do you server the files with Apache or NGinx?
[10:18:35] <ncls> ollivera: yes, if you want, but remember that this is not an OS file system : there are no "read / write" permissions on it, you can only emulate the tree and the permissions with your application. You might want to store additional informations (owner, permissions, filetype, size, etc) in a "file_infos" Collection for example
[10:19:58] <ncls> No I didn't use Apache nor NGinx, but it doesn't matter : Apache will serve script files (php or python) that can handle the GridFS and render the file with the right "content type" I think
[10:26:44] <ollivera> ncls, what happens when we remove a file? Is it like GlusterFs? will it trigger a synchronization?
[10:27:40] <ncls> ollivera: I don't know because I never used MongoDB with synchronised databases on different servers, but I think that yes, it has the same behaviour as "normal" collections
[11:12:44] <arount> I have little trouble with mongoexport command, maybe someone have the explanation to my problem
[11:14:15] <arount> I'm using mongoexport command, I have a pretty big collection (92263.063MB) indexed by _id and a field report_id
[11:15:29] <arount> I whould like to export all the collection sorted by report_id, so I use a command like: mongoexport -h 127.0.0.1:27017 -d database -c collection -q '{$query: {$orderby: {report_id:-1}}' -o dumps/2014-01-28/collection.json
[11:16:04] <arount> Ooops, bad copy / paste, the real command is: mongoexport -h 127.0.0.1:27017 -d database -c collection -q '{{$orderby: {report_id:-1}}' -o dumps/2014-01-28/collection.json
[11:18:50] <arount> I tried to add $query: {} to my parameter -q, or $query: {report_id:{$ne:null}} but nothing .. if I delete -q parameter all my data are exported .. I don't really understand why
[11:19:48] <arount> Derick: Fuck ... I'm stupid ! haaa, I got: { "$err" : "too much data for sort() with no index. add an index or specify a smaller limit", "code" : 10128 }
[11:36:08] <arount> Derick: I added --forceTableScan (to avoid $snapshot error) and it's works ! perfect :)
[11:44:09] <arount> Derick: maybe i do something wrong with my db, i check (you seem to say that forcetablescan is not necessary with indexes)
[11:44:38] <DevRosemberg> http://pastie.org/8673538 in there, Achievements, Permissions and Purchased Items dont work
[11:48:19] <arount> Derick: U'r right, just an error on my shell script who exec the mongoexport command, perfect, really thx
[12:07:46] <johnbenz> Hi, I need to move one of my mongo db to a new server. So I've decided to first configure the standalone mongo instance I have to a master instance. I went in the /etc/mongod.conf and uncommented master=true
[12:08:49] <johnbenz> but I couldn't connect to the mongod instance anymore doing "mongo" from the command line
[12:18:49] <johnbenz> I have only a standalone server and I want to move it to a new server with the less downtime as possible
[12:19:38] <johnbenz> my idea is to first do master/slave and when everything is synchronized change the server ip in my code and set the slave =true as comment and restart
[12:19:59] <byalaga> So, did you start the existing mongodb as a replicaset member? or a standalone one?
[12:20:41] <byalaga> afaik, you can't add a slave to it.. it has to be started as single replicaset memeber.
[12:21:49] <johnbenz> I've read this, but because I'm starting the server with "sudo service mongod start"
[12:22:19] <byalaga> you should stop the current node and start it back as a single replicaset member.
[12:23:28] <byalaga> take a hot backup of the existing one and setup a new node(this should be started as single replica member)
[12:23:28] <johnbenz> not sure I've understand. You confirm me that replicaset and master are two different things, right??
[12:25:42] <johnbenz> change the parameter in /etc/mongod.conf => http://docs.mongodb.org/manual/reference/configuration-options/#master
[12:27:51] <byalaga> johnbenz: http://docs.mongodb.org/manual/tutorial/convert-standalone-to-replica-set/ follow this
[12:28:35] <johnbenz> byalaga: is this mandatory? mongod --port 27017 --dbpath /srv/mongodb/db0 --replSet rs0
[12:29:37] <byalaga> then, iirc there's something you have to mention for replicaset in your config file.
[12:30:25] <johnbenz> because the daemon is runned by /sbin/service so I don't want to begin to tweak how mongo is runned
[12:33:06] <bzikarsky> Hm. Why is this collection not balances across both shards? http://pastie.org/private/rqcenolh5xlt2ztjzmekzw
[12:33:43] <johnbenz> ok and when I'll add a new server to the replica? Is there any risk that the master (formerly standalone) becomes slave?
[12:42:42] <RaviTezu> and it depends on many factors.. to decide which should be a primary(master) and which is secondary(slave)
[12:43:16] <johnbenz> RaviTezu: from what I'm reading here "master/slave" is different from "replicaset" http://docs.mongodb.org/manual/core/master-slave/?pageVersion=%5B%2721%27%5D
[12:43:59] <johnbenz> this is this kind of "master/slave" I want, I don't want to begin with replicaset as I just need to copy the database before moving it
[12:44:38] <johnbenz> Derick: It's just for some hours, or maybe less, I don't want failover as it's just copying from one db to another with less downtime as possible
[12:45:17] <johnbenz> Derick: if I just set in /etc/mongod.conf => master = true it should do the trick for the master right?
[12:48:43] <johnbenz> Maybe but I don't want to take the risk, it's a website with a very high traffic
[12:51:33] <johnbenz> this should work without downtime? db.copyDatabase( "test", "records", "db0.example.net" )
[13:00:48] <DevRosemberg> http://pastie.org/8673538 in there, Achievements, Permissions and Purchased Items dont work
[13:22:30] <Mmike> Hello. I have 3 boxes in replica set, and I'd like to run repair against my datadirectory (as I need to reclaim disk space). Is this ok path to take: 1) shut down one secondary. 2) run mongod --repair, 3) fire it up. When it's in sync, do next secondary. 4) When that secondary is done too, step down primary, and do him.
[13:24:38] <DevRosemberg> http://pastie.org/8673538 in there, Achievements, Permissions and Purchased Items dont work
[13:48:06] <DevRosemberg> Punishments, Achievements and Purchased Items are the only thing that dosent work
[13:58:55] <Nodex> [13:44:48] <DevRosemberg> Punishments dont, Achievements dont, Purchased Items dont
[14:04:33] <remonvv> That's actually quite unlikely. The DBCollection.save() method certainly isn't broken.
[14:05:01] <remonvv> And I'm assuming that getPlayerCollection() method for example returns a DBCollection and not some sort of abstraction.
[14:08:27] <remonvv> Because that try/catch around just the mongoClient initialization is doing my head in
[14:10:19] <remonvv> DevRosemberg: There doesn't seem anything wrong with the code you're showing. Also, "unvanish" -> "appear"?
[14:18:16] <remonvv> No idea. It shouldn't be broken but there's a lot of magic going on with "TheCosmosCore"..try it and see.
[14:18:31] <remonvv> The proper way is to reduce the problem to the smallest amount of code that still has the issue.
[14:19:23] <Nodex> DevRosemberg : reduce your code until the problem goes away, then add it back till it breaks and there is your problem
[15:23:31] <mehola> hi guys is there a way to do case insensitive sorting in mongodb or is it still missing?
[15:28:22] <mehola> sigh...that's quite disappointing. I used mongo 2+ years ago, I was hoping it would have been implemented by now...
[15:37:40] <mehola> cheeser: you mean on the case insensitive sorting? It was a known issue then, so no I didn't file a ticket on it
[15:40:59] <cheeser> yeah. when i've needed case-insensitive searches i just double store with one normalized to either upper or lower
[15:42:05] <mehola> cheeser: that's fine but when you think about the fact that you to store a lower cased version of any string field you may want to sort by....just seems dirty to me
[15:58:56] <mehola> Kaim: you may have looked at this already but I think the journaling part may be of interest to you http://docs.mongodb.org/manual/core/write-performance/
[16:06:24] <joannac> Please don't disable journalling. It's a false economy. You'll get fast writes until something goes wrong, and then you'll have corrupt data
[17:00:32] <DevRosemberg> Does anyone know what cn be causing parts of my BasicDb Object to not save
[17:10:40] <cheeser> i'm all but certain it's a bug in your code. but without a test case to isolate, i really can't tell either way.
[17:27:42] <yawniek> hi! when i do a aggregation with $addToSet, is there an easy way to not get the whole set but just the length of that set in the result?
[17:33:31] <NyB> hmmm, how would I go about creating a code_w_s (JS code with scope) BSON element in mongodb? Code without scope are simple functions, but what is the scoped equivalent?
[17:33:59] <NyB> I code_w_s are the last thing that I have to test in my BSON decode implementation :-)
[17:35:27] <Nodex> [17:08:20] <cheeser> i'm all but certain it's a bug in your code. but without a test case to isolate, i really can't tell either way.
[17:58:51] <NyB> Where are code elements with scope used in MongoDB? I am trying to figure out how to have one returned within a BSON document so that I can test my decoder implementation, but I have yet to find how to create one via the mongodb shell...
[18:14:21] <Darni> hi guys... is this the right place where to ask a pymongo (mongodb python API) question?
[18:15:42] <Darni> I'm writing a Flask based server which accesses a mongo db... from what I see at http://api.mongodb.org/python/current/api/pymongo/mongo_client.html and http://api.mongodb.org/python/current/faq.html#how-does-connection-pooling-work-in-pymongo there's some connection pooling built-in
[18:16:44] <Darni> and I understnad that if I make a global "client = MongoClient(...)", I can use it in a request and I'm not risking any race condition...
[18:17:34] <Darni> if I make a global "collection = MongoClient()['some_collection_name']", will I still get connection pooling correctly?
[18:18:27] <Darni> (my app uses a single collection, and integrates a lot of legacy code which uses that global as an API, that's the source of my question)
[19:08:03] <NaN> for an action-log collection on users (db.history) do you guys recommend me to use 1 document per user and append the actions to a log: key
[19:41:38] <cheeser> otherwise your documents would grow too large eventually. and with separate docs you could purge old ones via TTL indexing or move them off into an archival collecition
[20:20:49] <swak> So say if I'm doing a db where it contains profile info on a user that they can keep adding to. Should I make a new page for each addition?
[20:22:50] <swak> one is use an array in the page for each addition or to create a new page for each addition using ids from a central location.
[20:37:11] <swak> So say if I'm doing a user profile in which the user can keep adding info. I'm planning on separating different categories into different collections. However, should I store everything for a category for a user into one page using an array or should I store an id in an array pointing to each addition?
[20:41:57] <NaN> tried to import with mongoimport but it tells me "exception:BSON representation of supplied JSON is too large"
[20:44:13] <NyB> NaN: it could also be that you are trying to upload a document larger than the maximum size allowed by MongoDB. Did the document come from MongoDB or from some other source?
[21:10:25] <kali> slyth: cursor are lazy. so it's not the conversion but the actual database access that you pay when you convert to a list
[21:10:37] <cheeser> i'm not a c# guy (though I can mostly read it) but it's virtually impossible to diagnose speed issues without any details. show your code?
[22:00:19] <h4k1m> isnt it redondant to store users & posts in the same collections, with user credentials stored in each post
[22:12:58] <bloudermilk> Just wrote a mapReduce to calculate popularity scores for one of my collections (groups) based on another collection (messages). The reduce is keyed on the group's _id and writes to a "ranked_groups" collection. What is the idiomatic way to query "groups" based on the score field in "ranked_groups"?
[23:28:30] <brendan6> Does anyone know if there is a way to create an index so that that the query db.collection.find({'foo.0.bar': 1}) uses an index for a document {foo: [{bar: 1},{bar: 2}]}?
[23:29:09] <brendan6> db.collection.ensureIndex({'foo.bar': 1}) only uses the index with a find like db.collection.find({'foo.bar': 1})
[23:34:03] <thesheff17> can you do an update on a collection with $gte and $lte for a range of integers?
[23:37:22] <pgora2013> thesheff17: not sure if this is what you are looking for : {multi: true} as the third parameter
[23:39:33] <thesheff17> pgora2013: this is basically what I have in python http://pastebin.com/wax2VJJy