Log file Viewer
#mongodb logs for Friday the 22nd of August, 2014
[02:41:55] <Guest90> hello, I have a question about the usage of libmongc driver,when I update an none existence item with a "$inc" operator,the mongod return ok and the "nModified" is zero, while the driver responses me with success,is it correct?
[03:20:58] <xdotcommer> trying to do something naughty... I want to use "new Date()" subtract 24 hours in seconds ... and use that as a limit of an aggregation search
[03:23:52] <Boomtime> xdotcommer: why is that naughty? by limit, do you mean via a "$match greater-than yesterday"?
[03:38:30] <talntid> could not obtain connection within 5.0 seconds. The max pool size is currently 1; consider increasing the pool size or timeout.
[04:24:06] <Boomtime> "{ "$subtract": {new Date(), 86400000 } }" <- this is a constant, why not just give the value you mean rather than make the database work it out?
[06:13:48] <cheneydeng> hello, I have a question about the usage of libmongc driver,when I update an none existence item with a "$inc" operator,the mongod return ok and the "nModified" is zero, while the driver responses me with success,is it correct?
[06:19:14] <Boomtime> "nModified" tells you how many documents were affected, your command ran fine (it successfully modified all matching documents, whcih was zero)
[06:20:40] <cheneydeng> so zero means update successful? I'm wondering how can i get an failure if the updated record number is zero?
[06:24:01] <cheneydeng> @Boomtime yeah,I know that it seems the response is expected as normal.while I need a way to tell me no record is updated through the libmongc driver
[06:24:51] <cheneydeng> Boomtime: is there a way to achieve that or should i do a pull request to fix it if it is normal?
[06:26:29] <Boomtime> cheneydeng: you can tell the status of the command from the "ok" field, and you can tell how many documents were affected with the nModified field.. what don't you understand?
[06:27:43] <cheneydeng> Boomtime: I'm query the mongd through libmongc driver,this driver is a third-party code for me,you get it?The driver can know the status and nModified,while i'm not
[06:33:10] <cheneydeng> Boomtime: yes,and this API just return 0 in this situation,and i can't tell if it updated some records or nothing did.
[06:33:40] <Boomtime> if it returns false then something went wrong and command may not (or may have, who knows) made it to the server
[06:34:44] <cheneydeng> Boomtime: Oh,yes,it returns true.so you understand it?As i don't know if someone is the author of that lib,so i asked the question here.
[06:36:24] <Boomtime> right, so to find out what the command actually resulted in at the server (more than just "true" meaning whether it was executed correctly) then you need to call mongoc_collection_get_last_error
[06:36:57] <Boomtime> that will give you the last "write concern document" that was returned by the server
[06:37:51] <Boomtime> that will give you access to the nModified field which will tell you how many documents were affected
[06:41:05] <Boomtime> yeah, it's not real nice, mongodb is document-oriented which translates to objects in code, but objects in C are kind of clunky
[07:20:13] <fatih> i have a collection with approx 30,000 documents. I need a way to iterate over them and do certain task.
[07:21:00] <fatih> when I put the logic to iterate over those documents in the worker, then each one will try to access it and iterate
[07:21:35] <fatih> that means duplicated work because once the iteration started there is a lot of side works to, like creating files, etc..
[07:21:55] <fatih> I was thinking of creating a lock that only one work could access at any given time
[07:22:27] <fatih> but the problem here is, the collection size is increasing and I have 10 workers which are capable of doing it
[07:24:04] <kali> fatih: or at least 2 :) 1/ you can partition the collection beforehand and associate each part to one worker
[07:25:31] <kali> 2/ you can use a document field to act as a document lock: in its main loop, a worker will findAndModify a document that does not have yet the "magic" field and set it to something like "running"
[07:26:26] <fatih> kali I was thinking about the second approach, you mean all 10 should start to iterate over all collections and pick one that is free ?
[07:27:21] <fatih> what if one worker picks up 20,000 and the rest is distrubuted amongs the other 9
[07:27:55] <kali> well, each worker will pick one and start working on it. during this time the other will pick some
[07:28:46] <kali> you may want to setup an index on the "magic" field to avoid repetitive scan of the 30k docs
[07:32:02] <kali> 30k docs, and it sounds like your jobs are on the heavy side. mongodb will be fine
[07:32:59] <fatih> I'm just thinking of evenly distributing the task, workers are going to access randomly
[07:36:35] <fatih> kali: let me say it again, each worker will work at the same time, but they will not work on the *same* documents
[07:37:38] <fatih> say I have one document with the id 123, this was picked by worker 1 and the work on that is finished, the field is cleaned
[07:37:54] <fatih> now imagine that the worker 2 get's that immediately after worker 1 is done with it
[07:40:25] <kali> then yes, use two fields: one is "locked_by" (or assignee) and the other will be "done_at"
[07:41:17] <kali> in the findAndModify, you look for unassigned jobs (locked_by:null) and done_at:{$lt:now()-5min}}
[07:41:20] <fatih> "Eventually I hit a point where Mongo did become a bottleneck (the consumer contention got too much) and I moved to a combination of Beanstalkd and MongoDB. MongoDB now holds the data and Beanstalkd holds the IDs in queue."
[07:41:55] <kali> fatih: i have a run a million user site with mongodb as a queue system. don't bloat the infrastructure for nothing. 30k docs is nothing.
[07:43:02] <fatih> because an additional stack just causes problems too, that's why I asked here, to ask how to solve it
[07:45:34] <kali> if that's of interest, one of my implementations lives here: https://github.com/kali/extropy/blob/master/core/src/main/scala/MongoLock.scala
[07:47:04] <fatih> assignee.name is the current holder, and assignee.time is the time the worker started on that
[07:48:21] <kali> because if done_at is old, it hints at a worker crash, without requiring the assignedAt
[07:49:22] <fatih> therefore, I need two fields, one is called "locked_at" and one is called "done_at"
[10:04:00] <T3h_N1k0_> Hi, I've got a pb with a mongo cluster, the Django application throw this error "OperationFailure: database error: socket exception [SEND_ERROR] for 10.5.8.83:27017"
[10:07:24] <jordana> T3h_N1k0_, yeah it could be a network issue but it could be a driver issue or issue with the host running the django instance
[10:07:41] <T3h_N1k0_> rspijker: we thought it could be a network issue too, but the network seems all good
[10:16:01] <rspijker> it might be your open files T3h_N1k0_ http://docs.mongodb.org/manual/reference/ulimit/
[10:16:25] <rspijker> open files is misleading, since it’s actually file descriptors which, I think, also govern things like sockets...
[11:52:22] <Hyperking> would mongodb be better than writting a REST api? both use json to structure data...not sure, but I want a slim stack
[11:53:50] <Hyperking> I developing a news website. Data will be dynamically feed into the site by either a json rest output or connect to a mongodb server.
[11:58:28] <Hyperking> jordana: Im not completely sure. is MongoDB serving data to a rest service and then to the client? Looking for something as slim as angularjs
[12:01:25] <jordana> Your REST API can be written in NodeJS, if you're thinking about keeping your stack simply all you need there is JS
[12:03:33] <Hyperking> jordana: Ok i see now, before I was using a flat json file which is different from a server rendered json output
[12:07:25] <Hyperking> jordana: REST endpoints would be urls that output data? if so would this be the same as writting a php page that outputs formatted json like a html page
[12:43:14] <jordana> Hyperking: Yes. Your PHP endpoint would query your MongoDB and output the documents as JSON (probably with a little processing PHP side to convert things like dates etc)
[12:46:40] <Hyperking> thanks jordana, Is the REST endpoint needed? MongoDB -> client (php or angularjs)
[12:47:51] <jordana> Hyperking, your angularJS would not directly query the database, it would request data from your API
[13:22:32] <mjburgess> hi. i have two collections a,b in are documents which have a field refering to a field in b, eg. in a {"x": "SOME-WORDS", "y": "DATA"} and in b {"x":"SOME-WORDS", "q": 1, "p": 2} i would like a result which replaces SOME-WORDS in a with the full document in b, ie. {"x": { document from b... }, "y"...}
[13:23:52] <mjburgess> hi. i have two collections A,B . In A documents have a field refering to a field in B, eg. in A {"x": "SOME-WORDS", "y": "DATA"} and in B {"x":"SOME-WORDS", "q": 1, "p": 2} i would like a result which replaces SOME-WORDS in A with the full document in B, ie. {"x": { document from B... }, "y"...}
[13:55:38] <Frozenfire_> Hello all. I was wondering about $limit for the aggregation pipeline. Is there a way to specify no limit give me all documents? http://docs.mongodb.org/manual/reference/operator/aggregation/limit/
[15:45:36] <__nesd> i wanted to try today the compact command and i was quite surprised when i saw that it actually made my collection take more place (see http://pastebin.com/rchjrv7W) did you ever heard of such a case or have a idea of why such a thing could happend?
[15:57:57] <asd3syd2> i need to store about 100k vocabularies - is mongodb up for the task and does it offer an equivalent to SQL LIKE queries?
[17:26:30] <oblio> what's the best way to allay two operations being performed on a document at the same time
[17:26:56] <oblio> e.g. im playing with an API that uses mongo and 2 updates come in for the same document, and the document is locked by one update so the other fails
[17:28:55] <oblio> i'm guessing this is specifically why people will hybridize with mongo and a sql database, but i guess i'm just wondering if i'm missing something with terminology, if there is a way to have mongo wait for one of the operations to complete and queue the other
[17:30:22] <oblio> not very long, just i cant guarantee that the two ops wont happen at the same time
[17:33:48] <jordana> I don't really understand why it would fail, they should just wait for the other to complete. Is it a timeout of some form?
[17:40:38] <oblio> jordana: well, it goes a little deeper. i guess i didn't have a full scope of it.
[17:43:48] <jordana> oblio: I can't really tell from what you've said but if you need to perform this as a transaction from the sounds of it an application level message queue might be better suited
[19:11:29] <wc-> hi all, im using the native mongodb driver for node, i cant seem to get an aggregate query to respect my cursor batchSize
[19:12:10] <wc-> im creating the cursor with the aggregate command, then calling cursor.get(err, results) then iterating on each result
[19:34:05] <wc-> anyone know if this is an issue worth sending an email to the nodejs mongo driver mailing list about?
[19:45:12] <derek-g> so if I need to search for some documents in mongodb by a field - what exactly is happening behind the doors ? Does mongo scans entire collection?
[19:54:42] <derek-g> whaley so if I have ~40000 json document i want to store and search by headword field - is search gonna be fast?
[19:59:26] <derek-g> whaley, honestly it's just dictionary data that will be used by a handful of people editing entries via PHP frontend.
[20:04:25] <whaley> derek-g: btw, I wouldn't worry much about 40k documents unless it's growing rapidly... my production system has a collection whose .count() function just returned 843,800,975 for me :)
[20:09:38] <wc-> https://github.com/mongodb/node-mongodb-native/blob/master/lib/mongodb/command_cursor.js#L20
[20:10:11] <wc-> if i do an aggregate query with a cursor: {batchSize: 1000} its going to be overridden by this
[20:14:22] <wc-> does it send a command cursor, then that returns a regular cursor or something like that
[20:37:34] <benjwadams> I am having trouble updating an array of subdocuments. I am very confused by the documentation.
[20:38:21] <benjwadams> I want to update a single field within a an array of subdocuments. here is what i have attempted, to no avail
[20:38:50] <benjwadams> db.datasets.update({"services.asset_type": "RGRID"}, {$set: {"services.$.asset_type": "Grid"}}, false, true)
[20:55:54] <benjwadams> wow, so this isn't even possible: https://jira.mongodb.org/browse/SERVER-831
[20:56:38] <benjwadams> It's mind boggling that this functionality doesn't exist. This would be trivial in SQL
[21:13:20] <andrewrk> the array update operator $, acts as a placeholder to update the first element that matches the query document
[23:17:02] <MacWinner> if a node in a replset is down, will the working node slow down on responding to requests?