Log file Viewer
#mongodb logs for Thursday the 10th of January, 2013
[00:09:23] <sahat> Hi, we have a Mongoose collection that has a field of type array which references another collection. If we are going to remove an item from that array, will it also delete a reference from another collection like SQL and ForeignKey ?
[00:09:34] <sahat> E.G. Game collection has comments array field which references Comment schema. If I delete an item from comments array, will it also delete that object from Comments collection?
[00:41:21] <pottersky> just watched a video on CouchDB App (CouchApp)... looks a bit of an overstretching to me
[01:15:43] <thelinuxkid> Shameless HN plug: I created a Python project called Gumjabi that ties the Ping API with Kajabi and it's mentioned in the article. Please upvote. http://news.ycombinator.com/item?id=5033820. Thanks!
[03:35:37] <gjaldon> hi there! i'm following the guide on "getting started with mongodb javascript shell" here:http://docs.mongodb.org/manual/tutorial/getting-started/#create-a-collection-and-insert-documents
[03:36:24] <gjaldon> but I'm getting an error when trying to do the 'Insert multiple docs using a for loop'
[03:59:42] <Kane`> having some troubles with compound indexes. http://codepad.org/yJFtIPJV - the output of .explain() shows they're not being used in the query :/
[04:08:07] <Kane`> IAD, {mac, site, metadata.date}? not {metadata.mac, metadata.site, metadata.date}?
[04:17:31] <Kane`> it seems if i run the same query within mongo's shell, indexes are used perfectly http://codepad.org/8ScHBsHk
[04:36:04] <Guest_1448> so if I do a lot of .inserts() (with write concern 0) then .count() afterwards, it seems my count() waits until all the inserts are done
[04:41:51] <limpc> im getting an "unauthorized" error when I try to execute db.printShardingStatus() -- what am I missing?
[05:24:02] <limpc> Wed Jan 9 23:22:52 uncaught exception: error: { "$err" : "socket exception", "code" : 11002 }
[05:54:08] <nemothekid> I was wondering are there any performance issues to worry about when dealing with large documents, with many indexes, or would it be wiser to simply split them into several collections
[06:07:01] <crudson> nemothekid: that's hard to answer without knowing specifics, but normally I'd say unless index sizes are becoming a real problem keep the data logically connected
[06:10:48] <josh-k> hi guys, i need some help fixing an issue with mongodb on Travis CI (i am from the Travis team)
[06:11:21] <josh-k> we are changing to a new VM setup, and since moving to it mongodb won't start and we get this error in the logs https://gist.github.com/0c5491a1f24f83316db6
[06:12:39] <nemothekid> crudson: Thanks, I thought so. the data I have is often queried together and is hardly ever updated, however I'm expecting the collection size to eventually grow to the 10s of millions, and we have other collections like this but they are relatively small. All in all we just don't want to effect query time
[06:14:06] <crudson> josh-k: does /var/lib/mongodb exist with correct permissions? That's ignoring the glaring warnings above regarding 32bit and using OpenVZ.
[06:14:56] <josh-k> /var/lib/mongodb exists, with one file in it: -rwxr-xr-x 1 mongodb nogroup 6 Jan 5 03:37 mongod.lock*
[06:16:06] <josh-k> it seems like this lock file is old and has been around since the template was created
[06:22:04] <josh-k> crudson: so i am not sure why either the lock file persists or the service is not shutdown gracefully
[06:29:48] <josh-k> crudson: does this chef recipe look correct to you? https://github.com/travis-ci/travis-cookbooks/blob/master/ci_environment/mongodb/recipes/server.rb#L37
[06:30:25] <josh-k> crudson: we just confirmed that stopping the mongo service does not remove the lockfile
[06:59:38] <jrule> loading simple 1 column file with a header line using mongoimport tsv format into a new collection. every row except the first throws an error complaining aobut a duplicate _id_. insert local.huh keyUpdates:0 exception: E11000 duplicate key error index: local.huh.$_id_ dup key: { : null } code:11000 locks(micros) w:68 0ms How do I get mongo to generate a new ID for each row loaded.
[07:00:46] <jrule> the file looks like my-id\n1\n2\n3\n etc... the \n are real line feeds just used for examplehere.
[07:03:37] <jrule> it was a new empty collection. I get the same results for -drop and for -upsert options
[07:06:02] <jrule> the result is the first record inserts all the rest of considered dups b/c the ID column is being loaded as null
[07:08:42] <jrule> there is only one column in the table and that is unique there are 10 rows and the values are 1.10
[07:10:18] <jrule> It looks like mongoimport is generating a null internal id for each row from the log .. The column with the null value is not my column name I am giving
[07:16:11] <crudson> can you create a paste of the first 5 lines or so - I am running a db update so my server is locked now but can try it when this completes (whenever that will be!)
[07:37:19] <jrule> you only see the error above when running the server with the --verbose option turned on w/o it the loads say the all succeeded but only one record is actually loaded.
[08:10:54] <jrule> crudson... I just fired up an older version of mongo 2.0.6 on amazon and tried the example, works fine
[08:11:31] <jrule> I think upgraded that test instance to the latest 2.2.2 does not work. I think I need to open a bug with the for this... Do you know how to do that. Never done a bug.
[08:47:04] <chrisq> NodeX: remember my tracks/albums in mongodb? now i need to store playlists as well, any ideas?
[08:54:37] <chrisq> NodeX: that collection would hold documents with id's from the albums/tracks collection though?
[08:55:23] <chrisq> as rights and such keeps getting updated, i' prefer not copying the tracks into the playlist documents
[09:25:58] <NodeX> therefore there would be no renaming updates which makes referenced id's pointless
[09:52:28] <jwilliams_> i execute map reduce js with the command `mongo mydb my.js` where contains statement db.collection_name.mapReduce(m,r,{out:"tmp_output", sharded: true}); but mongo throws exception saying "unknown m/r field for sharding: sharded"
[09:53:29] <jwilliams_> checking doc (http://docs.mongodb.org/manual/reference/method/db.collection.mapReduce/#db.collection.mapReduce) sharded:true should be the right syntax.
[09:59:50] <chrisq> NodeX: tracks and albums dont change, but our rights to play them do. So i need to do a lookup somewhere to find if we're currently allowed to play each track in a playlist
[10:29:09] <Gargoyle> With the aggregation framework, is it possible to get the "top x" in a group? For example, lets say scores groupped by category and there are 20+ scores in each category, can you get it to limit to the top 5 in each category grouping?
[10:35:50] <kali> Gargoyle: well, thinking of it, i'm not 100% sure it will work. try it, and tell me :)
[10:36:20] <Gargoyle> kali: I thought that might limit the whole result to 5. Just coding it up now! :)
[12:28:28] <rawler> could someone please explain why the upper bound of my search criteria isn't used? http://pastebin.com/z8EAwucG
[12:30:19] <kali> NodeX: that won't change it... it picks the right index, but choose not to set an upper bound
[12:31:08] <kali> rawler: it's certainly because part of the index is multiple, i had a similar case but the value that was unbounded was the source of the multiplicity, so it's not exactly the same
[12:33:47] <kali> rawler: so i have a ugly patch to force the optimizer to setup the up bound on this specific index
[12:34:31] <rawler> oh, that's just not gonna fly here.. :S I highly doubt operations will accept that kind of monkey-patching on the DB-level..
[12:35:21] <kali> rawler: yeah, i am "operations" here too, so i at least i can do whatever the fuck i need :)
[12:35:37] <rawler> just for my understanding, is the problem that there are multiple _keys_ in the index, or that 'pricings.parent_id' is kindof an array?
[12:36:35] <kali> rawler: yes, in that case, the index is tagged as "multikeys", and the optimizer have to ignore some boundaries not to break some requests
[12:37:19] <kali> rawler: it would be better if the metadata info was in the form of "pricings.parent_id is multi" instead of a single boolean for the whole index
[12:38:39] <kali> rawler: feel free to open a new jira, link mine for references and give me the id so that i can vote on it ;)
[12:39:48] <kali> rawler: i hate to say this, but i am not happy with the way 10gen has deal with this issue
[12:40:16] <rawler> why not just reopen the one that is there? it's clearly a serious issue, and it's really unresolved?
[12:42:27] <rawler> the ability for embedded documents and indexes of them were the whole reason for us to evaluate mongodb for this application.. if the functionality is broken in this way, I guess I'll have to keep fighting with the current MySQL setup.. :S
[12:44:51] <kali> i can't blame you if you don't but i think it could help me a lot if you wrote that to a jire
[12:54:15] <HDroid> Will a DBCursor load each document entirely or is it lazily fetched? Trying to iterate over all keys, but at that stage I don't need the documents' contents yet.
[12:58:00] <Derick> HDroid: from the PHP side, we do this: http://derickrethans.nl/cursors-in-mongodb.html
[12:58:51] <Zelest> Derick, might be a bit random thing to ask, but when is 2.4 estimated to be released? :o
[13:01:23] <HDroid> Ok, so the cursor itself is lazy, but is there a way to get the keys without loading entire documents?
[13:02:15] <Derick> one with just a projection to get just the fields you want, and then another one for the full docs
[13:04:01] <Zelest> Derick, If I setup a single MongoDB server and do nightly dumps of it.. is that enough for production use?
[13:08:15] <rawler> NodeX: don't worry.. we're not axe-wielding invaders in longboats anymore.. *REALLY*
[13:08:48] <geekie> Not for my dayjob tho would never work but I can see the advantage for other projects I'm working on! :)
[13:10:50] <geekie> Plus Derick, they will most likely include MongoDB in a unicourse, I might take it for fun and let you know how they screw up ;)
[13:15:00] <NodeX> var fb='someFunctionName'; fb.call(); <--- works .... var fb='some.dotted.function.name'; fb.call(); <---- fail
[13:15:08] <rawler> oh? about time.. last year I spent there were mostly a waste of time, and from what I heard at least the next few years wasn't much better..
[13:15:57] <NodeX> the trouble with learning to programme at uni is you get some other person's idea on how programming should be done
[13:16:33] <geekie> NodeX: Yeah, in this case, I teached my teacher some told him "Ok this is how I wanna do it" and he said "Fine, do it."
[13:17:39] <rawler> NodeX: well, the main problem at BTH at the time was that they spent an ABSOLUTE minimum on paying teachers.. so they only got people that really would not qualify for a real programming job.. *tm*
[13:18:29] <geekie> rawler: Now the ONLY thing that makes the course is the teacher Mikael Ros, he's dedicated in what he does and he loves to discuss solutions on irc and a great ballplank for my ideas.
[13:19:14] <rawler> oh, I think he was there while I were there too.. I remember his name, but not much more..
[13:51:04] <Gues_____> hey guys, im using mongodb in a production enviornment and my collection is about to hit it's 16MB limit. What should I do? Im using the PHP driver
[13:52:13] <Zelest> Will there any a per-collection write-lock? Or is it still on a per database level?
[14:34:53] <oskie> according to mongodb, there's a lot of read on a secondary (>=100MB/s), but no client connection. what does that read come from?
[15:16:36] <emperorcezar> Recommendations on the best way to serve up files from gridfs out to the internet?
[15:20:10] <emperorcezar> kali: Yea. I was looking at that. Seems weird to have to recompile nginx. I try to avoid that. I'd rather have a package.
[15:27:33] <emperorcezar> Yea, I don't mind compiling, but then it's really harder to keep up with minor releases, security fixes especially.
[15:31:01] <marqu3z> Hi guys, do you know some benchmark page about intersection query with mongodb? I'm dealing with that and i'm not sure about performance with big collections.
[15:32:54] <JoeyJoeJo> I'm using pymongo and I have a list of strings to search for in one field. I use MyQuery['field'] = {"$in":MyList} which only works for exact matches. So I changed my list to be python re objects instead of strings, but it returns nothing. Any suggestions on what I can try?
[15:43:53] <marqu3z> it's not about a specific query. I have all the users logged with a social network (fb for example) so i store their fb id on mongo. Then another user login, and i take his social graph from Facebook, so i have an array with a list of friend ids, and i need to query against that array and find out if the there are users on my database that are friends with the new one. So i need to make this query very often and don't know how well performs mongo with this
[15:48:27] <marqu3z> i suspected about that, right know i can't switch to another db, if you know some benchmark about that will be very helpful! Thank you for the help!
[16:57:10] <nonent> question about shard-autobalancer. i've got 64 chunks, and after initial data import, 32 shards with shardKey 0-5 are on shard1, while 32 shards with shardkey 6-10 are on shard2. issue is, higher values are more likely to be accessed. will it rebalance these automatically as use continues? a strategy with 0-2, 6-8 being on shard1 and 3-5,9-10 being on shard2 would work much better.
[16:58:35] <nonent> i assumed chunks would be added to shards in an alternating fashion. but somehow they've ended up this way where the first half are all on shard1 and second half on shard2
[17:01:04] <kali> that's why it is recommended to use shard keys that gives a good, nice spread, but of course, it's not always possible
[17:01:45] <kali> it might be possible to manually rebalance some chunks to get a few chunks with higher value on the other shard
[17:06:27] <nonent> kali: yeah, i've read those notes, but since most of my queries are based on id for this collection, and inserts are few while updates are many, it seemed like the appropriate choice anyways.... well, except for the tendency for newer ids to be accessed more commonly.
[17:08:45] <nonent> i'll look into possibility of manually rebalancing. i don't know why i assumed distribution of chunks would alternate between shards instead of choosing a pivot point and splitting chunks between that.
[17:12:07] <kali> nonent: well, it make sense: first chunk grows, grows, grows, is splitted, one of the half is migrated, and then both side grow at more or less the same rate, are splitted but as the shards are even balanced, there's no reason to waste energy migrating data between shards...
[17:14:19] <nonent> i'll see what the access distribution looks like after this is all done before deciding what to do about it. i guess i also have to eval whether the savings i achieve from not needing to query both shards for _id is worth the possible uneven access.
[17:29:35] <owen1> I start my mongo with /home/t/bin/mongod --fork --logpath /usr/local/nextgen/push/shared/logs/mongodb.log --logappend --config /etc/mongodb.conf my mongodb.conf is: port = 27017 bind_ip = 10.1.72.132 dbpath = /data/db fork = true replSet = push I can't connect to the console to initiate my replica set - "exception: connect failed". looking the log i see: "couldn't connect to localhost:27017:
[17:29:37] <owen1> couldn't connect to server localhost:27017" and "replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)"
[17:30:49] <owen1> any clues? (btw, it's the primary instance. i plan to have replica set of 2 hosts, for dev only)
[17:36:13] <kali> owen1: it restricts the interface (as in network interface) on which you can connect to your mongo
[17:36:36] <kali> owen1: if you don't mention it, you'll be able to use any interface (localhost or your 10. ip)
[17:36:59] <kali> owen1: but if you set it to 10. you can no longer use localhost, as localhost is 127.0.0.1
[17:38:11] <kali> owen1: practical aspect is: on a dev environment, if you don't understand what bind_ip does, you don't need it.
[17:39:01] <owen1> bean: does it mean i will not be able to access my mongo from outside machines with mongo --host <ip> command?
[17:41:54] <bean> lets say you have 2 IPs, one on an external interface, and one on an internal, and you only want your mongo to listen internally
[17:43:59] <bean> cuz i can't really explain if you don't understand the difference between internal networks and external networks
[17:47:37] <owen1> i am running rs.initiate() and i never get the cursur back. it's still holding or something. in the log i see: "[rsStart] DBClientCursor::init call() failed"
[17:50:38] <owen1> kali: this is the log for starting mongo - http://pastebin.ca/2301085 [rsStart] couldn't connect to localhost:27017: couldn't connect to server localhost:27017
[17:50:47] <owen1> and couldn't unlink socket file /tmp/mongodb-27017.sockerrno:1 Operation not permitted skipping
[17:53:40] <kali> is there a chance there is already a mongo running on this machine (i think the message would be "socket already in use" in that case, but...)
[18:41:20] <jaimef> anything special need to be done before cloning a mongodb to another server to bring up replicas?
[18:46:30] <Auz> we just brought up a new mongod instance with the replicaset name, then from the primary did rs.add('server...')
[18:48:41] <jaimef> so always better to just let mongodb replicate all the data from scratch instead of trying to clone it with rsync(1) etc.
[19:27:09] <owen1> "killall mongod" on the primary host. and the secondary still shows 'secondary'. isn't it suppose to became primary?
[19:27:45] <kali> owen1: you can't have failover with two servers. you need a third one, at least in the form of an arbiter
[19:48:25] <Virunga> I have a collection of docs like this { _id:'..', day:'..', log: [{time:'..', messages:[..]}]}, and with these params {_id: channel, 'log.day': day}, {'log.$.messages': 1, '_id': 0} the find method returns this { log: [...] }. How can i get only the messages field?
[19:52:11] <kali> Virunga: you can get them closer to what you want with the aggregation framework, but you'll never reduce them to the the messages strings
[19:52:45] <kali> Virunga: this kind of processing is ok on the client side, which is easily scalable
[21:11:58] <tqrst> I'm going through the tutorial in the browser shell on mongodb.org, and so far I'm having problems getting the find command to work: db.scores.save({a:99}); db.scores.find() -> db.scores.find() returns [] instead of what I suppose should have been [{a:99}]? Did I misread something?
[21:24:59] <owen1> kali: oh. i didn't realize that. i guess i'll add a 3rd host on my dev environment, to test it.
[21:26:52] <JoeyJoeJo> How would I remove documents that have a specific field, regardless of what that field contains?
[21:45:32] <kali> owen1: because when a member of a two-node replica set can't see its sibling, it can not know for sure if it is not still primary, but hidden behind a network split
[22:22:01] <changerO1Sea> hey all! I got a $slice problem, maybe i'm reading the docs wrong, but I'm trying to just get one arg from the first obj in an array to show up and it's showing up everything the document. can anyone help?
[23:01:35] <changerO1Sea> perhaps let me rephrase that question, I only want the first element in an array, instead I am getting the whole doc and the first element, can anyone help?
[23:08:01] <kchodorow_> changerO1Sea: you can say {x:0, y:0,...} for all the other fields in the second arg to find, or use the agg framework
[23:47:01] <jaimef> can you merely copy an existing rs to a new host and start it up as a different replicaset name?
[23:48:38] <jaimef> I copied an existing replica set to a new server, and want to start it up as a diffeent replica name.