Log file Viewer
#mongodb logs for Friday the 27th of February, 2015
[01:26:18] <garthkerr> echo "deb http://repo.mongodb.org/apt/ubuntu "$(lsb_release -sc)"/mongodb-org/3.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list
[01:26:34] <garthkerr> W: Failed to fetch http://repo.mongodb.org/apt/ubuntu/dists/trusty/mongodb-org/3.0/multiverse/binary-amd64/Packages 404 Not Found
[01:29:12] <garthkerr> echo "deb http://repo.mongodb.org/apt/ubuntu "$(lsb_release -sc)"/mongodb-org/testing multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list
[02:35:07] <MLM> I want to implement an ordered list. I have a model with an integer `order` field. What would be the best way to update each successive row/document when an item is moved around?
[02:37:27] <GothAlice> MLM: https://gist.github.com/amcgregor/4361bbd8f16d80a44387 < a full taxonomy model mix-in (Python, but demonstrative)
[02:38:41] <GothAlice> Most append/prepend after/before calls come down to a call to insert, here. There's maths. And likely race conditions. And it's not optimized for the case of moving something within a set of siblings.
[02:39:12] <GothAlice> That's a method provided by MongoEngine. It returns a fresh instance of self.
[02:41:18] <GothAlice> You'll notice bits of this code drop down to raw PyMongo, too. (Basically anywhere there's curly braces…)
[02:42:24] <MLM> I am not keen either way. I am not sure what Mongoose offers over the vanilla driver: https://www.npmjs.com/package/mongodb
[02:42:53] <GothAlice> In the last week I've assisted six individuals with Mongoose-related injuries. :|
[02:44:45] <MLM> Looks like some people did make wrappers: https://www.npmjs.com/package/mongodb-promises, https://www.npmjs.com/package/mongodb-promise
[02:47:51] <MLM> Ye, I don't see any difference in API to Mongoose so why not. Plus promises are awessome and this package seems pretty up to date: https://www.npmjs.com/package/mongodb-promises
[02:52:37] <GothAlice> That could help, as long as you have a sane retry and failure policy in the event of a long-term conflict.
[02:56:30] <GothAlice> Well… it comes down to how long are you willing to let a client request wait.
[02:56:46] <GothAlice> And checking to see if a lock isn't freed in a reasonable amount of time, and forcing it open if need be.
[02:58:49] <GothAlice> Often things like this may require background processing, often on a schedule. https://gist.github.com/amcgregor/4207375 is a model for this, including sample code, sadly in Python. :)
[02:59:17] <MLM> Ye I have read a bit about the job queue problem with databases. This is just a ordered list that I want to make
[02:59:57] <MLM> No processing but I could see a conflict if multiple requests are made to reorder the same list. Order indexes could get jumbled
[03:01:46] <GothAlice> You effectively need to two-phase a $inc for a range of documents (siblings after the insertion point) and a $set of the document into its position.
[03:03:20] <GothAlice> With a unique key on (parent, sequence_no) or such. In the event the $set fails, roll back the "transaction" ($inc: -1 things back). Mid-"transaction" here the sort order will be otherwise unaffected.
[03:04:54] <MLM> Would it be wise to use the latest request to re-order? And of course debounce to every second or so if a constant stream of re-ordering requesting are made?
[03:07:37] <GothAlice> There are also subtleties to how you count records when inserting with offsets from the "bottom" of the list instead of "top".
[03:09:22] <MLM> I think I will just allow a user to set a new order index of an item by id. Then just update/shove the other items of the list into place.
[03:11:23] <GothAlice> I have pretty drag-and-drop re-ordering in my MongoDB CMS that uses this type of arrangement, with an API of alterChildIndex(oldIndex, newIndex).
[03:12:22] <GothAlice> I'll be moving to the taxonomy API I linked above for a more nuanced approach, once I armour it against and verify race conditions, in the above way. ;)
[03:19:12] <GothAlice> It's more of a defensive coding practice. I'll see if I can dig up some links.
[03:21:05] <MLM> In Mongoose you define Model Schemas. Since I believe Mongodb is loose with its fields, there is no enforcement there right?
[03:21:32] <MLM> (no enforcement with the raw driver) to get back a certain object shape with properties and defaults if non-existent
[03:57:03] <GothAlice> During certain operations, one could in theory provide default values if extant values do not exist, such as during an aggregate query $project operation.
[03:58:42] <Freman> the one that fails is the first one for T8WA3j9EE2ESzN53PzH7, the second one succeeds
[04:00:02] <Freman> update(find, update, options) was just logged as console.dir([find, update, options])
[04:09:31] <Freman> I know that the second one succeeded because of where they are and what the error message is set to on callback
[04:14:11] <Freman> psudo code for the task looks like function startlog() {mongo.update(..., function(err) {'Problem saving log'})}; function appendlog() {mongo.update(..., function(err) {'problem appending log'})};
[04:14:35] <Freman> they're both being called about 100 ms apart but the second call is the one that succeeds becuase it's the first one that complains
[04:25:10] <Freman> nothing that spits Problem saving log MongoError: E11000 duplicate key error index: logsearch.log_outputs.$_id_ dup key: { : "T8WB63bTLGwQwdAnk68f" } out shows up in the log
[04:31:03] <Freman> similar event logs http://pastebin.com/xgNnTBDr (sorry, couldn't find the exact event)
[04:34:53] <Freman> some times (more rarely) the conflict seems to happen the other way - appending loses out to saving
[04:50:18] <joannac> Freman: http://docs.mongodb.org/manual/reference/method/db.collection.update/#use-unique-indexes
[09:49:11] <morenoh149> How much data would it cost to record all irc communications on the freenode network?
[09:55:28] <aslomowski> I have a problem, with quering and deleting, how can i truncat array in subdocument witch is also in array ?:>
[10:03:05] <morenoh149> aslomowski: if `schema` on 7 were the schema of timeTableSubSchema then it should have the hour, minutes and type fields as well no?
[10:05:50] <aslomowski> morenoh149: I dont understand. Hmm.. timeTableSubSchema i just subdocument of schema.stops.timetable
[10:07:04] <morenoh149> aslomowski: you should split if nesting will make the document grow larger than 16mb
[10:11:46] <morenoh149> aslomowski: in what library? that's not a mongoose thing nor a javascript thing
[10:12:32] <morenoh149> arrays have .length() which returns the length of the array. There is no setter .length(n) in js nor mongoose.
[10:45:21] <brano543> Hello, i would like to use an external index library with mongo. Can you tell me how am i supposed to do it? Just store the id in both places,in the library and in mongo?
[10:47:03] <brano543> The library i want to use is libspatialindex, right now i am able to insert into the index, but i would like to use mongo as the database backend for this index.
[10:48:25] <rymate1234> noob question - I have a mongodb collection with a date field - example field is "date" : ISODate("2015-02-27T10:40:14.962Z"). How would I query the database to get items from the last hour?
[10:50:09] <brano543> rymate1234: as far as i know, mongo is object database, you can´t just use functions like in relational databases. i guess you will need to "explode" that date into separate fields.
[10:50:43] <obeardly> brano543: your question makes no sense, MongoDB if fairly optimized for what it does, why screw with it? My only suggestion if you are looking for key/value store, and nothing else, you could look at AeroSpike, but AFAIK as a DB, you can't beat defacto MongoDB
[10:51:56] <brano543> obeardly: maybe you misunderstood me. i don´t want to replace mongoDB. i just need to use the r-tree spatial index from libspatialindex.
[10:53:50] <brano543> obereadly: You said MongoDB is optimized,i hope so, because i want to create edge-exapanded graph representations and use it for routing
[10:54:40] <brano543> obereadly: i mean each document will have list of neighbours with their cost values.
[10:55:17] <obeardly> brano543: My best guess, if if your question exceeds the basis of MongoDB capability, it's beyond my ability; I do know that MongoDB has the ability to use C libraries, but it's beyond my expertise, apologies for speaking out of line
[10:56:40] <Tobatsu> Hi. One question.. I've been told that MongoDB is somehow based on Hadoop. Is that correct information?
[10:56:55] <brano543> obereadly: i just asked for curiosity, it would be great to serve spatial queries. For my purpose right now,i will just do with using the R* tree library temporary to preprocess the data and insert into mongoDB.
[10:57:33] <obeardly> http://stackoverflow.com/questions/18233945/query-to-get-last-x-minutes-data-with-mongodb
[10:59:16] <brano543> is here someone who can tell me how to use external index library libspatialindex with mongoDB? :)
[10:59:34] <obeardly> Tobatsu: think of Hadoop as a Hummer, it can carry a lot of data, but it's kind of slow, where passenger cars carry a smaller payload, but are much faster
[11:01:46] <obeardly> Be easy my friends, true happiness lies within ones self, and not their existentialism
[11:07:19] <brano543> If anyone knows someone who might be able to help me with using external spatial library libspatialindex with MongoDB or clarify me the concept how it may work,please let me know, i will stay on the channel :)
[11:20:51] <jsfull> hi guys, everyday my crawler collects a list of top 20 urls from google for particular keyword, the results change often and what i want to find out is how volatile the result is for today, so my question is, how do i calculate the volatility/change %, wondering what math equation i should use- input on this would be really helpful
[11:47:34] <zivix> jsfull: I think that largely depends on your definition of "volatility". But as a starting point I'd use the rank delta as a measure, and you can create a scale from less to more volatile based on how often the rank changes or how big that change is. Overall volatility would be a function of those two datapoints over time, imo.
[11:48:31] <brano543> If anyone knows someone who might be able to help me with using external spatial library libspatialindex with MongoDB or clarify me the concept how it may work,please let me know, i will stay on the channel :)
[12:25:50] <brano543> If anyone knows someone who might be able to help me with using external spatial library libspatialindex with MongoDB or clarify me the concept how it may work,please let me know, i will stay on the channel :)
[13:01:15] <brano543> If anyone knows someone who might be able to help me with using external spatial library libspatialindex with MongoDB or clarify me the concept how it may work,please let me know, i will stay on the channel :)
[13:36:14] <brano543> aliasc: official documentation says 16 MB,you should read it, if you want bigger you need something called GridFS
[13:36:21] <brano543> If anyone knows someone who might be able to help me with using external spatial library libspatialindex with MongoDB or clarify me the concept how it may work,please let me know, i will stay on the channel :)
[13:37:34] <StephenLynx> probably it throws an error if you try to add stuff that exceeds this limit.
[13:38:04] <brano543> StephenLynx: Hello, can you help me with my issue of using libspatialindex as secondary index to MongoDB. what are the practices?
[13:39:12] <brano543> StephenLynx: it is a tree where you can store geographical coordinates and they are grouped according to the Minimum-bounding-rectangle approximation of each area.
[13:39:27] <aliasc> currently i structured my data like in RDMBs model with ids linking to documents but thats not what mongodb is right
[13:39:30] <d0x> Is there a way to call db.xxx.find(...) inside a map function of an Map Reduce task?
[13:40:54] <aliasc> a collection can have thousands of documents but a document in that collection can have hundreds of thousands of arrays
[13:41:21] <aliasc> what i did is sepparate data in its own collection for arrays that might get long
[13:42:14] <brano543> StephenLynx: Nevermind,you may be able to answer me second question. I want to store adjacency list in MongoDB where each record contains neighbor ids and travel costs. Is it a good idea to let mongodb actually query the DB like this and find a route by known algorithms or is it slow?
[13:42:22] <StephenLynx> for example. I make a separate collection for threads in a forum and posts in a thread. but I keep embedded the list of mods for the forum.
[13:42:28] <aliasc> exactly. and the thing i love is i program classes the way i want my collection to be and just insert the object of the class
[13:44:04] <brano543> StephenLynx: I looked into that,but i need my own route algorithm,i am doign a specific task :)
[13:44:32] <StephenLynx> yeah, I have zero experience with that. what I know is that mongo does not have any tool to relate objects and collections.
[13:44:54] <StephenLynx> so if your work relies intensively into these relations, you should probably not use mongo.
[13:46:13] <brano543> StephenLynx: I asked about the MongoDB performance. i can make those relations,thats not problem for me,adjacency list should do it. So you suggest using mongo as temporal storage,then load the graph into main memory server and run the routing algorithms there?
[13:47:24] <StephenLynx> mongo performance is good for reading and writing documents, that I know, though.
[13:48:09] <StephenLynx> but the thing is, these routing operations would have to be performed by your application
[13:48:55] <StephenLynx> if you use a graph database you will have these operations done in a tool specialized for that and at the same time your application will not have to perform any actions in this regard.
[13:49:07] <brano543> StephenLynx: i asked this school,if they could send me a paper what were they actually doign last year,but they didnt reply me http://cs.brown.edu/research/pubs/theses/capstones/2014/chen.andersen.pdf
[13:49:48] <StephenLynx> so even if mongo's raw performance is higher than this database, your operation total performance might be lower because you will be doing two jobs for a single operation if you use mongo for it.
[13:49:58] <brano543> StephenLynx: well, as fas as i know,there isn´t a graph database where i can specify my own traversal,at least its very hard, i have to nderstand Java which i don´t know.
[13:53:16] <brano543> StephenLynx: I understand, but i do it as my bachelor thesis and i even got asked by some bus travel company that they are interested in my product :)
[13:56:11] <brano543> StephenLynx: In those graph databases there are already implemented functions for shortest path which i don´t need. I can create as simple thing as graph representation myself. The point thing of database is that it keeps persistance of data and can handle many clients. I also looked at project Openstretmap Routing Machine and i am not sure how tehy server is implemented,the code is really huge
[13:57:28] <StephenLynx> so the problem is that you need to implement a custom logic for the paths and you believe these graph databases don't allow you to?
[13:58:30] <brano543> StephenLynx: My problem is i am not sure how to deal with creating multithreaded server to use my custom program to handle queries.
[14:00:58] <StephenLynx> io.js deals with concurrency by having a thread per CPU core, so if you perform cpu intensive operations, it will not be really good.
[14:02:00] <StephenLynx> python still is about 20x slower than V8 and uses much more RAM because it uses threads to deal with concurrency.
[14:03:37] <brano543> StephenLynx: okay guys, listen, lets not waste time, i will try to use mongodb for routing and if that doesn´t work out i will create in-memory representation of graph,lets say use STXXL:map for storing it, then find the path using custom traversal and get additional information about nodes from MongoDB when the route is finished. That is also a way.
[14:06:06] <brano543> StephenLynx: i will let you know in few days how i proceed, right now i am solving approximation of points using Boost:envelope minimum bounding rectangle :)
[14:06:06] <GothAlice> See also mongodump/mongorestore. These read and write MongoDB BSON files, mongoimport/export handle other, simpler formats like CSV.
[14:06:33] <Cygn> I already wrote it in mongo language so mongorestore could be the right way, thanks guys !
[14:07:38] <GothAlice> StephenLynx: Even our data disagree. https://gist.github.com/amcgregor/707936#file-terminal-3-top — in the middle of processing 10,000 concurrent requests memory usage is ~10MB. (This is without threading. Threading adds ~2-4MB.)
[14:10:37] <Cygn> Another Question guys, i had to use the Date() Function of mongo, thats why i wrote my whole import as mongodb request (db.collection.insert …) can i also import this statement as a file?
[14:11:32] <GothAlice> Cygn: Yes. I don't recall MongoDB's preferred date format for import, but you could find out by exporting some data involving dates first, then having a gander. :)
[14:12:09] <Cygn> GothAlice: Yeah, my problem is i did that and it exports .bson which is not readable :/
[14:12:36] <GothAlice> That's a dump, not an export. Try the "mongoexport" tool instead, and make sure to select a format like JSON, or CSV (if appropriate).
[14:17:06] <Cygn> GothAlice: It exports like { "$date" : "2014-01-04T00:00:00.000+0000" } … but that is my problem, my importdata has a datestring "2014-01-04" which i want to have imported using the Date() Funciton (which i could do in a normal mongo statement.
[14:17:55] <GothAlice> Yeah, that won't work. Data != code. You'll need to reformat those dates in the proper ISO format, wrapped in {$date: } to mark it as such.
[14:19:06] <Cygn> GothAlice: i think i will just insert it as a string, and afterwards update entity of the collection …
[14:38:35] <StephenLynx> Considering how fast a hash function can be, I don't think it would have a severe impact.
[14:43:31] <StephenLynx> GothAlice that benchmark means little. First, I don't know what code is running behind the test. Second it does not compares to anything. Third I don't know the hardware. Fourth, I don't know how much your server software deviates from the standard python webserver.
[14:49:15] <nir> I have the following collection http://pastecode.org/index.php/view/1ce4d941, and I don't know how to write my query to find all products that have a specific set of "filters"
[14:51:33] <nir> i also try to find products that have a specific filter but that is not working : db.products.find( {"filters": {$elemMatch: {"identifier": "HOM"} }})
[14:52:44] <Cygn> I want to use the Value of a field inside a foreach, i know that i can pass it f.e. via ….forEach(function(e) { e.value } - but what to do if my column name is "1" ?
[14:54:12] <StephenLynx> does your filter's sub-documents must have a field? it looks redundant having the "identifier" field
[14:54:30] <Cygn> If i try f.e. e.6 i get "unexpected number" if i try e."6" i get "unexpected String" :(
[14:54:32] <StephenLynx> instead of assigning the value of identifier directly to "sex" or category"
[14:55:49] <nir> StephenLynx I can't redesign this collection, it has been created like this by my client :(
[14:55:59] <sweb> how can i group by and count this collection http://paste.ubuntu.com/10449679/ via `tag special 1` ?
[14:58:09] <pamp> StephenLynx: Hashed indexes doesn t work with array, which is my case.. Have much impact create an text index?
[15:01:00] <sweb> StephenLynx: could you please show me how ... i can find out strucure of aggregation group :(
[15:24:17] <StephenLynx> you must first query to filter the results with a $match block and then use an acumulator operator, like push
[15:28:36] <StephenLynx> the group block works similarly to the projection block, you give it a field and something to put in the field. except it can take accumulator operators , it required an _id on documents, and can't take booleans to just project or not the field.
[15:35:38] <Cygn> I use a collection called dataChunk, which has a column with the identifier "5", i need it to be 5, i wanted to do it that way: db.dataChunk.update({}, { $rename: { '"5"' : '5' } }), which does not adress the column "5" apparently, any suggestions?
[15:43:58] <GothAlice> http://bsonspec.org/spec.html < technically all keys, including array indexes, are string mappings.
[15:44:27] <Cygn> StephenLynx, actually i didn't wanted to but the json format does not allow a simple number.
[15:45:48] <Cygn> My Problem is i have a whole bunch of date values, that i need to push through the mongo Date() Function in the end. I don't care if i import them using csv, json etc. but after the import i need to address them in any way inside a foreach loop to push them through that function.
[15:48:38] <Cygn> GothAlice: That is the point "24" is interpreted as "24" as column name, but in the end i need 24 (without quotes)
[15:48:45] <GothAlice> Cygn: Sweet mother of the gods. Importing CSV data, or, what could possibly be the reason to use numeric keys like that?
[15:49:48] <FunnyLookinHat> Is there a way to take my administration user and grant it access to every db in the system ? Or do I have to manually create it for each database?
[15:50:55] <GothAlice> FunnyLookinHat: There are some roles that give server-wide permissions. I.e. readWriteAnyDatabase, userAdminAnyDatabase, dbAdminAnyDatabase.
[15:52:45] <GothAlice> StephenLynx: Obviousness, less likely to run into accidental addition (instead of unary plus casting), etc. Basically, using the functional instead of fancy approach is less likely to bite in exotic ways.
[15:53:49] <FunnyLookinHat> GothAlice, when I try to update with a new set of permissions, it seems to automatically apply them to only a single database though? http://hastebin.com/ujaqufipaf.profile
[15:54:29] <GothAlice> FunnyLookinHat: User maintenance happens in the "active" database. Is your "admin" database active, or one of your target databases?
[15:59:13] <Cygn> StephenLynx, GothAlice: i really try to get on track here, and solve the problem i have in any possible way, but … right now it would most likely fix everything if i could just at least adress the value inside my forEach statement, which seems impossible since i can neither write function(e) { e."4" } / function(e) {e.4} to adress it
[15:59:54] <GothAlice> And in JS, object "attributes" or "properties" can also be accessed using array notation. :)
[16:00:43] <StephenLynx> use console.log(JSON.stringify(object)) with the object you got back from the query
[16:04:38] <GothAlice> Veejay: See: http://docs.mongodb.org/manual/tutorial/modify-documents/#update-multiple-documents
[16:07:21] <Cygn> StephenLynx: The stringify result: http://pastebin.com/bYb309nj - to your other feedback, does that mean i can't use the foreach to the find function?
[16:08:14] <Cygn> StephenLynx: Was actually based on a very high ranked StackOverflow Example… http://stackoverflow.com/questions/3788256/mongodb-updating-documents-using-data-from-the-same-document/3792958#3792958
[16:08:38] <GothAlice> … this data is flat, uses pseudo-numeric field names, and ignores native datatypes for integer and date values. Cygn: Your data fits a relational model far better than MongoDB.
[16:09:49] <StephenLynx> can't see why you are so concerned then, what you do on CLI is barely valid on any language.
[16:10:43] <Cygn> StephenLynx: Concerned? I just need to transform the date fields, and i can't since i get a syntax error all the time.
[16:10:59] <GothAlice> Hmm, that SO post makes me want to try a $rename that moves a field into a non-extant new sub-document to see what would happen.
[16:11:47] <StephenLynx> and you said you wanted to rename from a string to a number, but you can't do that.
[16:12:12] <GothAlice> Ignoring the fact that console doesn't exist in the MongoDB shell, http://cl.ly/image/2c0h1B1w0N0z < totally works
[16:13:23] <StephenLynx> HAHA indeed, if the field name can be parsed to a number, you won't be able to reference it in dot notation
[16:13:53] <GothAlice> StephenLynx: The trick seems to be the first character being numeric. If it's alpha instead, the extra digits don't matter.
[16:18:00] <GothAlice> Well, it's just the identifier naming convention. [a-zA-Z_][a-zA-Z0-9_]* — that's a pretty common pattern, same as Python's actually.
[16:22:50] <Cygn> Thank you both very much for now… especially for the answers to the confused questions.
[16:23:48] <GothAlice> But yeah. That data's not a very good fit for MongoDB in its current form. ¬_¬
[16:25:58] <GothAlice> … this data is flat, uses pseudo-numeric field names, and ignores native datatypes for integer and date values. Cygn: Your data fits a relational model far better than MongoDB.
[16:26:53] <GothAlice> mysql_fetch_array — get back the record as an array, with actual numeric separation, in PHP-land, at least.
[16:27:50] <Cygn> GothAlice: I need to have the functionality to add columns without having a fixed data structure, the data will in addition never be fetched, it will just be counted using the columns as criteria. I could only use postgres and the json functionality.
[16:28:51] <GothAlice> Cygn: MongoDB already lets you add arbitrary fields, and have those fields differ between documents in the same collection (though do this with an appropriate dose of caution). Nothing you've described suggests to me a numeric field name approach.
[16:29:04] <StephenLynx> if its a list of aditional anonymous data, you can just have a subarray and throw them there.
[16:30:45] <Cygn> StephenLynx: actually this is just an idenitifier used by the infracstructure of the whole company workflow. Each Sale has some attributes, all atrributes have an id which identifies them. In the whole system 1 means the market for example.
[16:31:23] <StephenLynx> yeah, you can just have a subarray in the document that represents the sale and throw this data there.
[16:31:58] <Cygn> Stephenlynx: you mean each entry should be a subarray? (i really try to get what you want to tell me ;) )
[16:32:22] <GothAlice> Instead of: {"1":"Belgium", "2":"WBA1N11050J155953", "3":"Test", "4":"", "5":"AppsOffer000", "6":"07-MAY-14", "7":"07-MAY-14", "8":"lifetime", "9":"", "10":"265"}
[16:32:56] <GothAlice> Have: {"data": ["Belgium", "WBA1N11050J155953", "Test", "", "AppsOffer000", "07-MAY-14", "07-MAY-14", "lifetime", "", "265"]}
[16:34:34] <StephenLynx> if you know which index represents which data, can't you use words instead of the position?
[16:34:53] <Cygn> StephenLynx: (using this value as id would be wrong since this is not the id)… but yes i could just make up names by myself for each column and associate the input data
[16:39:02] <agenteo> hi there, I remember querying a string field for its size but I can’t find how anymore… does anybody know how?
[16:43:19] <Cygn> StephenLynx: I mean i could just use only data: subarray as value, but where would be the improvement?
[16:46:39] <pamp> hi guys, its possible match a document when the difference for the others is that this one have an array field?
[16:57:57] <GothAlice> agenteo: Yes. $type, see: http://docs.mongodb.org/manual/reference/operator/query/type/
[16:58:23] <GothAlice> agenteo: Be aware that it interacts with arrays in somewhat odd ways. (I.e. if the field being checked for ObjectId-ness is an array containing at least one ObjectId, it'll match, too.)
[17:49:35] <GothAlice> voidDotClass: http://docs.mongodb.org/manual/reference/configuration-options/#storage.dbPath
[17:51:14] <GothAlice> voidDotClass: Whichever configuration file you tell mongod to use on its command-line.
[18:09:43] <Siamaster> if I haven't stored a date for my object and want to use the date which is stored in the objectid, how can I do it?
[18:10:49] <GothAlice> Siamaster: db.users.find({_id: {$gt: ObjectId.from_datetime(datetime.utcnow()-timedelta(minutes=5))})
[18:11:12] <GothAlice> Other drivers should have similar "from_datetime" methods on ObjectId to let you construct time-specific IDs for use in range queries.
[18:14:05] <GothAlice> Like saying age: {$gt: 27} — 27 is "application code" in the same way that ObjectId is.
[18:16:24] <GothAlice> Waheedi: Worst-case scenario in the event that your secondary safely promoted itself to a new primary: nuke the old (dead) primary and re-add it to the replica set as a new node. It'll re-replicate, join the set, then you can force an election if you care which node is primary.
[18:17:21] <GothAlice> (For reliability one should not require a certain node be primary for anything other than logistic reasons, i.e. to keep it in the same datacenter.)
[18:18:02] <Waheedi> GothAlice: the secondary got elected for primary, i don't really care which one is primary. The thing is the fatal state node is almost one month ahead of the current secondary (elected to primary node)
[18:19:34] <cheeser> so the secondary that just got elected primary is a month behind your old primary?
[18:20:00] <GothAlice> Hmm. Well, on the currently dead former primary you're going to want 2.1x the free space available on-disk than your data currently consumes, at a minimum, then try running mongod in --repair mode.
[18:21:38] <GothAlice> Waheedi: Yeah. "Do you have backups" means "we can't fix this". It may be time to start digging out the tape archives. ;) --repair is worth a shot (given the free space available to do it) but no guarantee in the situation you're in, with things in such a wonky state. Make a note, BTW, to add better monitoring of your system to catch things like growing replication lag in the future.
[18:25:59] <GothAlice> While technically it shouldn't really hurt to nuke that from disk, you may have hanging associations within the MongoDB configuration or remaining dataset, so it's best to avoid tampering.
[18:28:57] <Waheedi> What about the oplog.rs if i deleted it would it sync again with the replica set? if i re add this dead node?
[18:29:53] <GothAlice> Same comment from before: you may lose some important association which may cause the --repair to bail or other problems. In theory it's disposable.
[18:30:35] <GothAlice> You've run into a split-brain problem, though. You have a dead primary newer than the current primary… is new data still being added?
[18:49:41] <Waheedi> Ok can I copy the /data/my_db.* files from the dead node to the secondary? the new primary
[19:19:30] <voidDotClass> in order to find stuff within a collection where foo column's value is bar, should i run the query {foo:bar} ?
[19:33:16] <voidDotClass> when i run that, i'm gettin "fields" : null , "options" : 0 , "readPreference" : { "mode" : "primary"} , "numSeen" : 0 , "numGetMores" : 0}}
[20:23:16] <voidDotClass> Am I doing something wrong here with how i'm using the java driver: https://gist.github.com/anonymous/ec0a3e84cedc68ccfa9d
[20:23:57] <voidDotClass> everything works upto the last one, where i do cursor.size(), it dies with the exception: com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=127.0.0.1:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused}}]
[20:29:05] <voidDotClass> StephenLynx: how do i bind it to a different ip? i'm passing in my actual hostname when i create the client
[20:30:52] <StephenLynx> yeah, yo connect to it you will have to remove the binding to the local ip and add authentication.
[20:31:14] <StephenLynx> I mean, you could just remove the binding, but then anyone in the same network would be able to access it.
[20:31:15] <voidDotClass> i don't need auth as its on aws and i have the port open to my ip only, so no worries
[20:33:33] <voidDotClass> i wish i didnt have to either, only reason i'm using it is to move the data off to cassandra :)
[20:33:35] <StephenLynx> by remotely you don't mean you log in with ssh and then use the cli client, right?
[21:14:50] <voidDotClass> would've been nice if the client complained on having a null endpoint rather than silently converting it to 127.0.0.1
[21:20:53] <voidDotClass> basically i was doing private static MyClass instance = new MyClass() as the first line
[21:21:04] <voidDotClass> and a couple of lines below i was doing private static String ENDPOINT = ...
[21:36:50] <cheeser> show me what you've got. i don't have enough of a clear picture to recreate that.
[21:51:20] <voidDotClass> just saying, would've been cool if the java driver of mongo didn't act like the php / javascript driver and threw up an exception on receiving null, instead of silently converting it to 127.0.0.1
[21:52:45] <voidDotClass> nah, i'm just using mongo now to move my data to cassandra, other than that i shouldnt have to use mongo again
[23:07:35] <Waheedi> it seems the secondary that was elected to primary was the damaged one and not the primary that was giving the fatal status
[23:59:32] <joannac> a few minutes? your disk dies at 3am and within 30 mins someone will go and replace a disk?