Log file Viewer
#mongodb logs for Friday the 18th of April, 2014
[00:07:52] <simpleAJ> hi.. I was looking at db.createCollection API. It has size filed. Suppose my collection is not a capped collection and I didn't specify size field..how much does mongo db allocated a space for that collection?
[01:21:41] <Jadenn> does this look okay: db.friends.find( { $or : [ { $and : [ { friendUid : <id> }, { confirmed : 1 } ] } , { uid : <id> } ] } ,{uid:1,username:1,friendUid:1,friendUsername:1,confirmed:1,date:1} )
[01:54:43] <jadeLA222> Question: doing a nested lookup for the first time in RoR/Mongoid, and not sure what I’m doing wrong. Anyone spot something wrong w/ this syntax?
[01:54:49] <jadeLA222> Order.find( {'line_items.sku' : {'$in' : ["855906004184", "855906004245"]} } )
[02:05:57] <sx> would there be much performance difference for storing 1000 properties in one document vs 1000 documents w 1 property each
[02:08:43] <Frozenlock> Any suggestion on how to get $near functionality for timestamps? My crude approach would probably be using $lte and $gte, sort them and take the closest to my timestamp.
[02:12:34] <joannac> sx: if the 1000 properties are logically connected, what would be the point of separating them?
[03:15:11] <in_deep_thought> can someone explain the objectID to me? I have objects in my database that are autogenerated like this: db.images.insert({ "source" : "www.easyvn.net--45-best-real-world-full-hd-wallpapers--007.jpg", "_id" : ObjectId("534de25fce66b4cc1ab8fbe5"), "score" : 0, "__v" : 0 } the ObjectID filed throws everything off because I can't save add it to other dbs because its not valid json. then when I manually strip off the object id I get error
[03:17:06] <cheeser> the shell has an enhanced syntax that supports creating ObjectIDs, dates, etc. like that.
[03:17:33] <cheeser> you can convert it to a document like { $oid : "534de25fce66b4cc1ab8fbe5" } though
[03:20:29] <in_deep_thought> cheeser, so in the database itself, should the documents look like {"blah":5,"_id":ObjectID("jl3l4js234234jlk")} or should they look like {"blah":5,"_id":"jl3l4js234234jlk")}
[03:21:10] <in_deep_thought> so how am I supposed to call item._ID ? its not valid JSON so it would throw an error, right?
[03:37:35] <in_deep_thought> why is it that when I type typeof ObjectId("507c7f79bcf86cd7994f6c0e").valueOf() into the try.mongodb.org it tells me its an object? I though the whole point was that valueOf() returns a string
[04:09:25] <in_deep_thought> cheeser, Sorry my IRC quit. In any case, would a query of item._id return ObjectID("jl3l4js234234jlk") or "jl3l4js234234jlk" ?
[04:12:59] <cheeser> but assuming the document uses an ObjectID for it's _id then you'd get an ObjectID
[04:13:32] <in_deep_thought> so with { "source" : "www.easyvn.net--45-best-real-world-full-hd-wallpapers--007.jpg", "_id" : ObjectId("534de25fce66b4cc1ab8fbe5"), "score" : 0, "__v" : 0 }, item._id would return ObjectId("534....")
[04:16:22] <in_deep_thought> I have read that one of the whole points of MongoDB is that is stores everything in JSON format so that it is easily accesible. Is this not true when it comes to the autogenerated _id values
[05:02:33] <Alvein> some question %) how to safety remove files from dbpath dir with dot? (dbname.7, dbname.8)
[05:03:03] <joannac> narutimateum: http://docs.mongodb.org/manual/tutorial/install-mongodb-on-windows/#run-mongodb
[05:14:36] <Alvein> joannac: https://www.dropbox.com/s/msclzekeg87wz5f/Screenshot%202014-04-18%2011.14.02.png
[05:21:25] <Alvein> joannac, in this database I'm very often replace data.. may be about this start fragmintation? Thanks for explanation
[05:21:51] <Alvein> he-he https://www.dropbox.com/s/yzryy2tee2hztma/Screenshot%202014-04-18%2011.21.17.png
[05:24:09] <Alvein> and I think.. why? :) may be in some time I think that is good idea for our site structure
[07:21:19] <sweb> i want to know optimization about mongo... somthing like book mysql performance ebook
[07:21:39] <sweb> about query optimization... indexing and etc ... from who migrate from MySQL to MongoDB
[07:57:53] <in_deep_thought> "source": "www.easyvn.net--45-best-real-world-full-hd-wallpapers--018.jpg",
[08:00:21] <in_deep_thought> http://bpaste.net/show/N0XZSHsBq3j4WOTCZ1Ke/ this is my object. I want to return that long string. what is the way to do this? it looks like to me it might be Item._id.$oid but I am not sure if that is right? and then would I have to do .valueOf() on the result of that to get a string?
[08:02:07] <kali> in_deep_thought: what language are you using ? { $oid: } is just a json conversion trick, the _id is actually an ObjectId, so you should look for the interface of the ObjectId class in the language you're using
[08:03:36] <in_deep_thought> I am using javascript. It says that the convention is ObjectID.str to get the string out
[08:05:07] <kali> yes. it's a way of encode bson document in json. json has less types than bson, so...
[08:07:39] <in_deep_thought> do I have to call .str on the other objects I want to get? like if I want to get the url after source in that paste, would I need item.source.str?
[08:17:49] <bushart> Hi everyone. I have 60 millions documents. And i try group their by index field.... But it is a veeeeery long time...
[08:53:45] <rbnb> Ah, found it. Yes, secondaries do maintain an oplog: http://docs.mongodb.org/manual/core/replica-set-oplog/
[09:02:30] <kali> bushart: i don't know, i haven't spend enough time on 2.6 to get a good feeling on what the optimizer knows to do yet. but explain() should help
[09:03:28] <kali> bushart: then depending on what you're doing in the match and group, a composite index might be the solution, but you'll have to show me what you're doing
[09:56:13] <Guest53043> exception:BSON representation of supplied JSON array is too large: code FailedToParse: FailedToParse: Date expecting integer milliseconds:
[10:48:06] <tinix> i have a question about unicode querying.... is there some way to coerce strings to a simpler character set when querying?
[10:55:27] <tinix> hmm... i'll probably just dump that data into another DB system them, and store the object IDs
[11:05:57] <tinix> anyone know if redis does that type of coercion? i'm already using it for something else... if i can go w/o having to setup solr that'd be nice.
[11:22:21] <rbnb> I guess what I want to know is: does this image (reasonably) accurately portray MongoDB replication? http://daprlabs.com/blog/wp-content/uploads/2014/04/mongodb_repl.png << only one secondary is shown in that picture for simplicity
[11:42:26] <rbnb> Thanks, guys. Assuming that I want a writeconcern which gives a high consistency guarantee, when would it send? Would it be after journaling or application? I would assume after journaling is sufficient
[11:45:44] <rbnb> Derick, can I confirm that the only change in the flow for 2.6 is that getLastError is effectively implied by the write operation itself?
[11:47:30] <kali> rbnb: there is no general answer to that. it's a matter or infrastructure reliability and risk assessment. ack with w=majority has some benefits for instance.
[11:56:17] <kali> rbnb: (fyi, i'm not a mongodb developper) my understanding is in "ack" mode, the write goes to the journal, but you don't wait for the journal to be commited to disk
[11:56:47] <kali> rbnb: and if i understand correctly, that's what http://docs.mongodb.org/manual/core/write-concern/ says
[11:57:38] <rbnb> w=majority -> "enough systems are aware of this write and therefore you may as well consider it successful, even though it hasn't actually been made durable"
[12:00:32] <rbnb> This is how the GFC happened... a bunch of random events were modeled independently, but there was little thought put to the correlation between them... like if the DC has a power shortage just after acking :P
[12:01:02] <rbnb> strong tongue-in-cheek, but there's some truth to it... the option to have a more durable write concern seems useful
[12:02:11] <kali> rbnb: some apps can tolerate higher latency, some other can tolerate some occasional miswrites
[12:02:48] <kali> rbnb: the good thing with mongodb is, you don't even have to choose at the application scope. you can pick request per request your durability policy
[12:03:27] <rbnb> That's true, but for the apps which can tolerate higher latency, there is no "bet the business" option
[12:04:11] <rbnb> For the occasional "this write has to happen or we're f****d" operation, there doesn't seem to be a "truly durable" write concern level
[12:05:04] <rbnb> also, this image scares the shit out of me: http://docs.mongodb.org/manual/_images/crud-write-concern-journal.png
[12:05:59] <kali> rbnb: i thnk in that case, the write to journal should be understood as "commit journal to disk"
[12:09:53] <kali> rbnb: seriously, mongodb does work. we - long time users and supporters - and the developpers are not that stupid :)
[12:12:25] <rbnb> and the replication mechanism itself, all writes going through the oplog, having secondaries tail that log
[12:14:38] <rbnb> The whole system would be a heck of a lot faster & also more reliable if it was based on a distributed consensus algorithm (paxos, raft) rather than log shipping
[12:17:09] <h0rnet> if the oplog is mem only, hows it going to know where replication left off when you boounce the box
[12:17:34] <kali> so. you don't want to shard, you don't want to go ssd, you don't want to relax the write concern
[12:21:08] <rbnb> but it still seems that ever write to a primary causes 4 things to be written on that primary rather than 2 (journal oplog, write oplog data, journal target, write target data)
[12:27:29] <rbnb> But they do all need to hit the disk for data to be durable (and they will all be in separate files, iirc, so four write ops) - or am I mistaken there?
[13:51:02] <bushart> I try to make the grouping index, but it is very long. And explain show only BasicCursor :(
[14:01:55] <boutell> anyone know how to get the version number of the server via the client? I’m use the node client, if it matters.
[14:20:05] <bushart> I'm trying to reproduce this example: http://docs.mongodb.org/manual/core/aggregation-pipeline/#pipeline-operators-and-indexes but explains doesn't show that index is used for $group stage. Does it mean that index isn't used, or maybe explain just doesn't show it to me?
[14:22:11] <Derick> bushart: group can't use an index, it makes no sense as the whole collection (or pipeline) is being read anyway
[14:44:01] <Jadenn> joannac: i didn't realize it yesterday, but yes the $and was supposed to be there, i was trying to replicate WHERE (friendUid = <id> AND confirmed = 1) OR uid = <id>
[14:47:02] <Jadenn> yes megido, mongod should be able to boot from it, providing all the files are intact
[14:52:23] <Jadenn> how much of a performance burden would i be looking at if i just used normal indexes
[14:52:42] <Jadenn> there are thousands of queries, i don't think i'm quite as ambitious as to compound all of the fields
[14:53:49] <Nodex> I am yet to test their efficiency in the query planenr so I don't know how well they perform
[14:56:41] <Nodex> if you're haing to write complex queries wiht complex indexes then you might want to think about changing your data structure
[14:57:44] <Jadenn> well i had changed the data structure before, but i was running into even more problems
[14:58:51] <anildigital> guys.. I want to map a tree kind of data structure in mongo... basically for multi level marketing scenario
[15:00:14] <anildigital> Nodex: yep.. I am reading that link.. but is mongo suitable for such kind of data?
[15:00:55] <anildigital> I need to show the count of all the n nodes including nested .. for a parent.
[15:04:18] <Jadenn> i have verified the tables do exist, just can't get it to select AND inside of the OR query
[15:07:56] <Jadenn> {$or:[{uid:ObjectId("532dabede1ac8a14108b4568")},{friendUid:ObjectId("532dabede1ac8a14108b4568"), confirmed:1}]}
[15:09:25] <Nodex> what does this give you... db.foo.count({uid:ObjectId("532dabede1ac8a14108b4568")});
[15:09:47] <Nodex> and then what does this give you db.foo.count({friendUid:ObjectId("532dabede1ac8a14108b4568"), confirmed:1});
[15:11:54] <Jadenn> well when i imported the data from mysql i casted everything *but* the confirmed field
[15:12:37] <Jadenn> luckily mongo takes mere seconds to change data so ill just go in and foreach everything int
[15:14:22] <Joeskyyy> Just playin' around with stuff here to preface, I know elasticsearch does more what I'm looking for, BUT. Anyone know if you can do a partial text search using the new fancy $text op?
[15:15:31] <Joeskyyy> Only thing it's lacking, or at least obviously from the docs outright, is partial searching.
[15:20:50] <Jadenn> at this point i can't really tell what or what isn't supported with the php driver
[15:21:40] <Jadenn> this should be the equivalent to that query [ '$or' => [ [ 'friendUid' => new mongoId($id) ], [ 'uid' => new mongoId($id) ] ] ]
[15:27:48] <jet> Is there a way to not create automatically a database? I would to be able to create a database only with the create command
[15:28:11] <boutell> Jadenn: thanks, that is helpful. Need to figure out how to invoke that command from the node client now.
[15:30:38] <Jadenn> jet: how do you mean automatically? as far as i know, it can only be created by command
[15:34:14] <Jadenn> the data online about that is a bit sparse, but i'm sure someone else here knows
[15:38:31] <Nodex> $query=array('$or'=>array(array('friendUid'=>new MongoId($id),'confirmed'=>(int)1),array('uid'=>new MongoId($id))))
[15:42:09] <Jadenn> before i just glomped everyones friends into their own user object, and dealt with sorting and paging in php
[15:43:05] <dragoonis> I need to emulate this MySQL clause for a mongodb collection.. WHERE answer_id IN(2845, 2846)
[15:48:09] <dragoonis> Nodex, take a look at this: https://gist.github.com/dragoonis/071680922eac5ac4f724
[15:48:21] <Nodex> soething has changed either in the driver or the PHP core in the last year because typing was always required
[15:48:41] <dragoonis> response_id: 40516 .. has entries for 2845 and 2844, but the $all doesn't find this entrty
[15:51:47] <Nodex> db.answer_to_response_map.find({answer_id: { $in: ["2845", "2846"] } }).count(); <--- that will get you "WHERE answer_id = 2845 OR answer_id=2846"
[15:53:20] <Nodex> if your answers and responses are always integers then you should cast them as int's - you will save some space and sorting will work better
[17:42:19] <boutell> in thanks for an answer earlier, here is node code to verify the server is at least mongo 2.6:
[19:48:41] <Dimrok> I'm having trouble with a request, so I would like to know if you'd mind giving me a hand? I want to atomically find or modify a document.
[19:49:54] <Dimrok> My case is: I want get a field from a document (User that I get from users.find_one({'email': email}). The problem, it's a new field I want to add if not present in the document.
[19:57:40] <Dimrok> @saml yeah but how do I distinguish users that already have the field (to use this one) ? Is your answer to do something like: users.update({"email": email, field: Null}, {"$set": {field: The_New_Value}}) every time?
[20:03:27] <Dimrok> I'm scared of: -a race condition / -a cpuvore function. I explain my self: I want to compute a special hash per user, based on there email and some other data if they don't have one. The problem if I use a python driver that will compute the hash before knowing if the {"$exists: False} is positively evaluated by mongo.
[20:07:35] <tscanausa> Dimrok, why dont you start off with what problem are you trying to solve and then we might be able to help?
[20:09:18] <Dimrok> Sorry: > Imagine that database: [ {"email": "foo@bar.io"}, {"email": "bar@foo.io", "hash": "4242} ]. I want a function that return the hash field of the user (compute it if not present). so : method("bar@foo.io"} will return directly "4242, method("foo@bar.io") will compute the hash, store it in the document and return it.
[20:11:49] <Dimrok> So what I really want is an atomic function that find_OR_modify my document (and lazily evaluate the update argument).
[20:12:42] <kali> Dimrok: well, you don't have much choice but writting exactly what your describe. your computation has to be application side, as mongodb has no provisions for doing that kind of stuff, so you'll need two request anyway
[20:13:44] <kali> Dimrok: as long as your hash is a function of other fields of the document, your compute-and-store branch will be idempotent, so the worst that can happen is having two client compute the value at the same time, and getting the same result
[20:15:13] <Dimrok> In fact, as long as my parameters are the same, the hash function should be idempotent so it should work.
[20:29:48] <aaronds> Hi I'm trying to get the id of a newly inserted object using the Java API. It looks like I should be able to use getUpsertedId(); on the returned WriteResult object but it seems to always return null. Running server 2.6 so this should work. Anyone got any ideas?
[20:32:14] <kali> aaronds: the usual approach is to set the _id yourself. just call new ObjectId(), that's exactly what the client does anyway
[20:36:42] <aaronds> kali: just out of interest how does creating a new ObjectId ensure it's unique?
[20:37:30] <kali> aaronds: seriously: http://stackoverflow.com/questions/4677237/possibility-of-duplicate-mongo-objectids-being-generated-in-two-different-colle
[21:30:30] <saml> I have two databases: articles and images (not in the same database). can I do a query like: find all articles whose image is larger than certain size? article example: {img: 'foo/bar.jpg', _id: 'a/b.html'} image example: {_id: 'foo/bar.jpg', width:100, height:200}
[21:31:29] <saml> currently, do, articles.articles.find({img:{$exists:1}}) and for each doc, query images.images.find({width:{$gt:100}})
[21:45:16] <saml> so when they update image metadata, need to find all articles that use the image and update metadata on the article, too
[21:59:23] <tscanausa> so it your image collection has a field called article then it is really easy to do other wise you need to loop through all of your article collect your image ids and then go fetch the info from the images collection
[23:23:52] <LetterRip> I'm doing a $project and the field isn't appearing, here is my code and an example data and my output