Log file Viewer
#mongodb logs for Friday the 13th of June, 2014
[02:01:45] <tonyacunar> Can someone point me to documentation which explains how to do a full text search for a field in a collection which is a DBRef?
[02:32:53] <tonyacunar> for example, assume I have an document that has a DBRef, is it possible to create a textIndex on that?
[02:33:54] <tonyacunar> yes, on the field with the DBRef, so say the object with the dbRef is called User and user has attribute name
[02:35:37] <joannac> and you want to index in the original collection on the field name in the other collection?
[02:36:51] <tonyacunar> Thanks for confirming. So what i’m doing is I’m creating a wrapper around the original collection which contains the fields I want to index on. So I am “joining” for lack of a better word at the application layer
[02:37:25] <tonyacunar> I wanted everything to magically work. Sad it doesnt but the work around is not that bad.
[02:37:27] <joannac> if you're going to store the wrapper document, why have 2 collections in the first place?
[02:38:24] <duozhang> Hi, I met a exception when I tried to insert docs, it confused me: <type 'str'>: (<type 'exceptions.UnicodeEncodeError'>, UnicodeEncodeError('ascii', u'insertDocument :: caused by :: 11000 E11000 duplicate key error index: myproj.mycol.$_id_ dup key: { : { binary: "S\x99\udf03\udf67\x12=\x14\x9f\u0380", generation_time: 1402599184 } }', 119, 122, 'ordinal not in range(128)'))
[02:39:26] <tonyacunar> Well, take this case. Assume I have a user collection and a orderHistory collection. If the user changes an attribute in their user document I don’t want to have to go back and update his 10,000 old orders. I need orderHistory to always have current info of the user but the indexable searchable version can be stale
[02:49:19] <duozhang> Hi Joannac, if the key was auto-generated by pymongo, how could it be duplicated? and I've droped that collection and re-insert again, it still raised this exception~
[04:00:02] <troy__> I'm trying to insert a google map encoded polygon to mongo and the "\" characters are being removed automatically. Do I have to escape them? I thought all UTF8 was safe?
[04:00:21] <troy__> "zone_poly_encoded" : "q{hxGdpmjK{zAre@sAaKeOyM}DuEqBcHeL{w@c@yCdd@eMrDjSzRgFlK{Db\hFhEwAtGwBfNzfA"
[04:02:16] <troy__> I've tried adding another "\" to escape it but the result is two "\" get entered into the collection.
[04:04:06] <troy__> is this a known issue or am I missing something? It's really late here so thats totally possible..
[07:01:49] <gancl> Hi! How to populate other collection's subdocument in mongoose? http://stackoverflow.com/questions/24199444/how-to-populate-other-collections-subdocument-in-mongoose
[07:26:05] <Viesti> while importing data to a 3 node replicate set, 700 file, 20 000 documents each, so currently 14 million documents, but growing
[07:26:51] <Viesti> doing upserts, should probably look how much of the run actually is updates vs new data...
[07:30:40] <Viesti> I tried out 2.6 on a development cluster, and replaced mongoimport with a Python script using the new Bulk write API with pymongo
[07:31:38] <Viesti> locally on my mac (with SSD) the Bulk write api really is fast, but seems that I run into IO speed on the dev cluster, which prevents me from moving into production :/
[08:37:39] <frankblizzard> hello - i have a problem: due to a bug in my application (tip game for world cup) it was possible that User's could create a tip twice or more times (using the browsers back button). I have a Tip model and a User model. A Tip references a Match. The tips get embedded into the User model (Array). Now I need to find all Users that have tipped the same match multiple times (so in the user's tip Array there is more tha
[08:37:39] <frankblizzard> n one object with the same matchId). I can't figure out how to do that query in Mongo. Any help would be highly appreciated.
[08:42:59] <gancl> Hi! How to populate other collection's subdocument in mongoose? http://stackoverflow.com/questions/24199444/how-to-populate-other-collections-subdocument-in-mongoose
[08:46:58] <rspijker> frankblizzard: unwind the array, then group on UserId, matchId and add a new filed, say s, which is $sum:1. Then match on s:{ $gt: 1}
[08:59:35] <frankblizzard> rspijker: here is an example user doc. it has only one tip in array but you get the idea. http://pastebin.com/FE0EpJkn
[09:02:10] <rspijker> frankblizzard: db.Users.aggregate({$unwind:”$tips”},{$group:{“_id”:{“u”:“$_id”,”m”:”$match”}, “s”:{$sum:1}}},{$match:{“s”:{$gt:1}}})
[09:03:02] <rspijker> it can have the same userId twice, so you might want to do some further processing, second $group or something
[09:05:59] <rspijker> it will give you a list of every user,match combination that occurs more than once
[09:06:31] <rspijker> so, if user x has tipped matches 1,1,2,3,4,4,5. It will give you results x,1 and x,4
[09:13:05] <Derick> { $group: { "_id" : { "user" : "$_id", "m": "$match", "g": "$group" }, "sum": { $sum:1 } } },
[09:13:15] <Derick> { $group: { "_id" : { "user" : "$_id", "m": "$tips.match", "g": "$tips.group" }, "sum": { $sum:1 } } },
[09:13:53] <Derick> { $group: { "_id" : { "user" : "$tips._id", "m": "$tips.match", "g": "$tips.group" }, "sum": { $sum:1 } } },
[09:48:33] <djlee> Hi all, not sure if this is a PHP driver feature, but it looks like i can specify multiple mongo hosts in the connection, and it will attempt to connect to each one in turn, and return a connection to the first one tha succeeds. Is that right, or is the multiple hosts purely for replica sets?
[10:15:23] <djlee> rspijker: yeah i think i understand now. Unfortunately i've moved on to another problem, if i specify un/pwd in the host string, it works, but if i specify it as a config attribute, it doesn't. annoying as hell
[10:16:51] <djlee> rspijker: in php mongoclient you can specify the un/pwd as a config attribute rather than in the string (assuming it then merges it into the string).
[11:36:04] <crib_> the case is, i have an aggregation that i would like to stop when it counts up to 1000
[12:06:32] <crib_> rspijker: what about mapreduces, can i somehow terminate the reduce when I reach my goal?
[12:37:08] <goncalop> hi there - I'm running a CentOS instance which has NUMA , noticed the default configuration uses fork=true - if I can the service with the NUMA interleave=all it will not be applied to the forked process ?
[12:57:44] <thomastuts> does anyone know of a good way to migrate a mysql database to mongodb including relations etc?
[12:57:55] <thomastuts> i tried mongify, a ruby solution, but found that it was inadequate - it did not correctly save references for relationships
[13:05:10] <MrMambo007> I want to to persist just one array of similar js objects. The array may reach a length of 10 K to 100 K. I don´t need to read or write to it often, but when I do write to it, the items will be in a different order. Any best practice examples?
[13:09:59] <MrMambo007> the array is intended to be sorted any given time so the highscore entry is at index=0
[13:10:58] <rspijker> so, just make a collection Leaderboard, with documents that contain username, score, timestamp
[13:11:04] <MrMambo007> well I don´t know if it has to be an array, but I´d prefer the collection was sorted
[13:12:04] <MrMambo007> ok sounds like you have the right approach. How do I add index on the score parameter?
[13:12:04] <rspijker> you still need to include a .sort in your queries, but it will be really fast, because it’s indexed
[13:15:35] <MrMambo007> can I den query based on index? something like: db.Leaderboard.find( { score: 0 } ) ??
[13:16:45] <rspijker> if you want to find the highest score, you need to do db.Leaderboard.find().sort({“score”:-1}).limit(1)
[13:17:19] <rspijker> that finds all documents, sorts the by score descending (what the -1 means) and then limits it to 1 document, so only finds the one with the highest score
[13:18:07] <MrMambo007> Exellent answer : sorted collection of the top-ten on the highscore will be: db.Leaderboard.find().sort({“score”:-1}).limit(10)
[13:18:51] <MrMambo007> Entire higscore list sorted will be jus: db.Leaderboard.find().sort({“score”:-1}) ??
[13:20:36] <rspijker> if you want 10 to 20, you can do db.Leaderboards.find().sort({“score”:-1}).skip(10).limit(10)
[13:27:06] <MrMambo007> Hehe:) I have this very absud query for you... we had this comedy show with a weeklu monty-python like performance using a an old dutch song
[13:28:15] <Derick> he's saying that because he is living in the netherlands, he should sing in dutch; just like in a restaurant the menu should always be in french.
[13:30:14] <Derick> my favourite is still https://www.youtube.com/watch?v=s-mOy8VUEBk&list=RD-mwxE7Ky5hs&index=5
[13:31:36] <MrMambo007> Research actually shows that Danish children learn their mother tounge a bit later than other kids because of the lack of consonants.
[13:32:09] <MrMambo007> Words in Danish are much less distinct from eachother than in most languages
[13:45:59] <karan> Hi guys, some times I can see mongodb consumes high memory like 4GB, after restart mongodb service it comes to normal. I hope that because of indexing. How can I handle this?
[13:47:44] <rspijker> karan: also, you might be looking at the figure for virtual memory. Which is not necesarilly the actual (resident) memory consumption
[13:49:03] <kali> if it is the resident (RSZ) that shows this behaviour, you may have an issue. connection leak is a good candidate
[13:53:35] <MrMambo007> so rspijker: if I do a db.Leaderboard.insert( newScoreSubmit, { ordered : true} ); (assuming it is helpful to tell mongodb to keep the list sorted) - is there a way to obtain the index of the inserted object?
[13:55:30] <MrMambo007> ... so given a field that has been indexed in a collection of object (like the score field) how do I get the index of of a particular object
[13:58:02] <rspijker> there is no easy way to know the index inside of the index (ugh, one word, two meanings) upon insertion, or at all, for that matter
[13:58:34] <rspijker> what you can do is do a query where you count all documents with a larger value, relative to your indexed field
[13:59:18] <rspijker> consider the collection an unsorted bin of documents. The index is a list that has all of the documents inside of the bin in a certain order. Depending on the field(s) the index is on
[13:59:38] <rspijker> So you can have multiple indexes on the same collectino, basically meaning you have different sort orders on the same set of documents
[13:59:50] <MrMambo007> ok... so probably have to mainatin an in memory array in the node.js app and try and keep it in sync occationally with the mongodb object collection that handles the persistance of the leaderboard
[14:00:18] <rspijker> because your collection is indexed on that field, the count query will be pretty quick...
[14:00:54] <rspijker> say you insert a new document with score x, then you can just go db.Leaderboard.count({“score”:{$gt:x}})
[14:06:43] <remonvv> Indeed. One of our defenders didn't properly broke Iniesta in two with a kick to the chest. He's going to try and finish the job tonight.
[14:07:54] <MrMambo007> Ahh.. one more thing... i want equal scores to be sorted based on time.. I have a timestamp in the object. so basically I should index that field as well?
[14:09:28] <rspijker> remonvv: it was not a defender, unfortunately this disqualifies you from the better Dutchman competition. Thank you for playing and better luck next year
[14:11:54] <remonvv> Fine fine, I was crying shortly after that moment so maybe my memory is blurry. And defensive midfielder is pretty much a defender ;)
[14:13:05] <MrMambo007> You dutchmen have saved me a lot of time in researching and implementing a mongodb driven leaderboard! Lots of good karma emmenating form Holland today.
[14:13:52] <rspijker> MrMambo007: check out meteor, I believe it has an example of a mongo backed leaderboard on their website. Also, meteor is pretty awesome :)
[14:14:08] <remonvv> MrMambo007: How many score records do you think you'll have? Leaderboards are spectacularly hard to do at scale.
[14:14:16] <Viesti> I have a dutch dataset that gives me a headache when importing it to our mongo cluster :)
[14:17:11] <MrMambo007> well with the pagination suggestion from rspijker I could do pagination as well.. but the spec only requires me to show top 10 + the users rank in
[14:18:24] <Derick> if you're sorting on score descendingly, then I told you the index the wrong way around
[14:20:41] <remonvv> Derick, your last entry in that page which you only know if youre paging sequentially
[14:21:11] <MrMambo007> so for obtainint a compound indexed sort should I ask for a compound sort in my find query?
[14:21:17] <remonvv> If that's the only way to get the specific pages in your UI it's fine of course. Things like showing the page the player is on is more tricky though.
[14:21:34] <Derick> MrMambo007: no, indexes are handled for you. You don't need to do anything about it yourself.
[14:22:45] <Derick> the index is in the order you specify, but mongodb can loop the *whole* index in reverse
[14:23:13] <remonvv> Right but it's a b-tree so it changes the index data but has no effect in performance iirc.
[14:24:09] <MrMambo007> ok so mongo keeps a b-tree datastructure for the items... and anytime i run dhte ensureIndex() function It will resort the b-tree?
[14:25:23] <MrMambo007> so you can have many indexes on the same collection or will it replace the previous index?
[14:25:59] <remonvv> But adding indexes increases data size/memory usage and degrades write performance
[14:30:39] <remonvv> On a slightly different note; we're seeing issues where a removeShard returns done while there are still chunks in the db.chunks collection. It's a bit of a edge case but perhaps someone knows what's what. It's on 2.6 / replicated shard.
[14:32:21] <MrMambo007> db.Leaderboard.count({“score”:{$gte:currentscore}}) whee also the timestamp is less than the current timestamp.
[14:35:39] <rspijker> MrMambo007: simply db.Leaderboard.count({“score”:{$gte:currentscore}, “timestamp”:{$lt:currentTimestamp}})
[14:36:30] <MrMambo007> I wast thinking what you wrote rspijker..this is starting to make sense :) Thanks!
[14:37:41] <MrMambo007> rspijker: you are maintaining your number one position as my favourite dutchman
[14:41:08] <MrMambo007> Derick: You are pretty close as well. And your nick makes me happily think about this childhood hero: http://oldschoolgameblog.files.wordpress.com/2013/06/derrick-4f4e603ddfa5d.png
[14:46:54] <halfamind> Hey folks. I've been using the chef cookbooks to set up mongodb servers and started seeing that mongoimport/export were missing. Is that a change w/2.6.1?
[14:49:25] <BurtyB> halfamind, depending on your distro it looks to be in a different package now (mongodb-org-tools on centos)
[15:03:14] <halfamind> BurtyB: Thanks. Any idea how to install that via recipes? From what I can tell, it's attempting to install the mongodb-org "meta" package that includes tools.
[15:03:48] <jiffe98> nm first time I enabled it on one of the replicas I only got 7 queries but its running now
[15:49:28] <BlakeRG> so i need a text index on a field if i want to be able to search it with something analogous to LIKE %foo% am i correct?
[17:12:14] <mylord> how to update score of users "53949a18dee87d16a92d6d47" in this array? https://gist.github.com/anonymous/a5c9acdd7d88c46fbea2
[17:50:49] <voxadam> Has anyone managed to get mongodb installed from the repo on Debian sid any time recently? Every time I attempt to install the package(s) I get an error when dpkg tries to start the daemon. I also get the same error when I try to start the service using the service command. http://pastebin.com/J1jGNE6C
[17:52:27] <Hexxeh> If I have a ListField and an array, and want to find all documents where all values in the listfield are also found in the array, how do I specify that filter?
[19:01:44] <co10bok> hi... I'm spammed by MMS notification service. I have been granted to access to one of client's account for consulting purpose. It pass a lot of time up to now. I have no any credentials to access to my account and disable notification.So... Is there any way to switch off that account or disable email notification?
[19:39:11] <Chaz6> Hi there, I'm trying to use pymongo on cygwin; i've tried easy_install, and the source from git, but when i do `import pymongo` the interpreter just quits
[19:43:06] <Chaz6> 258 13778904 [main] python2.7 4772 seterrno_from_win_error: /home/corinna/src/cygwin/cygwin-1.7.30/cygwin-1.7.30-1.x86_64/src/src/winsup/cygwin/dlfcn.cc:216 windows error 127
[20:07:38] <fschuindt> Guys I am a newbie, what is model structure behind a comment model with responses. Like this: You can comment a post, but you can comment a comment. So a comment embeds itself? :x
[22:57:41] <hpekdemir> I will feetch information from several devices. around 40 different network components. let's say switches, information about ports. each switch has 48 ports and some aggregated interfaces. so about 50-55 in total.
[22:58:19] <hpekdemir> each switch will have it's own document. and I will append more and more by the time. information fetching is will be done every minute.
[22:58:50] <hpekdemir> I did some calculation. this would result in one collection or document with several million entries.
[23:03:45] <hpekdemir> ok just found one company blogging about 2 billions entries *per day*. I'm chilled now. :>
[23:05:22] <panurge> "collection dropped between getMore calls code:17356" - What that mean? Seems like is MongoCursor is getting down over the way
[23:40:45] <panurge> ok thank you very much for the support. I should evaluate if the branch install is needed