Log file Viewer
#mongodb logs for Tuesday the 15th of January, 2013
[00:36:13] <mattgordon> I'm on Mongo 1.8 and I'm trying to get all documents for which a field exists and is not null. What is the correct way to do this? I'm having trouble finding old docs and the information for the current version seems to be incorrect for my version
[00:51:20] <mattgordon> db.conversions.findOne({finished_at': {"$exists": true}}, {'finished_at': 1})
[00:53:11] <Auz> db.conversions.findOne({'finished_at': {"$exists": true, '$ne': null}}, {'finished_at': 1})
[00:55:32] <mattgordon> yeah that should do it. thank you! i'm seeing very different behavior between $exists: true, $nin: [null], and $not: {$in: [null}} so I was getting pretty frustrated
[00:56:00] <mattgordon> I assume 10gen cleaned it up for the new releases but that makes it difficult to find docs that don't secretly break for edge cases ;)
[01:08:39] <jaimef> so trying to split a large RS by adding 4 more systems to the rs, then setting fsynclock on, shutting down half of the servers, and then change the replicaset name of those left running. shutdown and startup other group. is there an easier way to do this?
[02:50:04] <w3pm> if i have a process I want to read/write to a mongodb database, what's the "proper" way of setting this up securely so that no other processes on the network can access the db?
[02:51:47] <w3pm> well lets say that process goes down and i have automated failover for another machine to take over as the read/write guy
[02:52:19] <w3pm> modifying the firewall each time that happens seems strange, maybe i make all the failover nodes trusted
[02:52:53] <jesta> If it's just internal networking (and not public facing) but you still want security inside, there's a user/pass auth I believe too
[02:55:15] <phira> to prevent compromise on one application from necessarily making it easy to get to data from other
[02:58:05] <w3pm> i guess i should read more indepth but i hope theres a better solution than just a plaintext password being stored somewhere
[02:59:38] <phira> it's just a defensive measure, it's not security. The protection from the general internet is handled by firewalls.
[03:00:23] <phira> the objective of the password is to prevent them from expanding out beyond the compormised app, if possible.
[03:02:44] <w3pm> yeah maybe im overthinking it, at the end of the day if the private network is compromised really bad stuff will happen, pw or not
[07:57:52] <strata> using the same version of pymongo (2.4.1) with mongo 2.2.2 on 3 different distros (ubuntu, fedora, arch), i create a record that looks like this: [{stuff: 'hi', somestuff: [{morestuff: 'hi', evenmorestuff: 'hi'}]}]
[07:59:22] <strata> when i pull that record out on arch (stuff = db.whatever.find()), stuff = that record exactly. on all other distros, it gets encapped like this {0: {the original record}}
[08:01:17] <strata> my only guess is the arch devs enabled some kind of quirk at compile time because it seems the other distros exhibit the later behavior (which to me is faulty anyway but that's my opinion)
[08:04:00] <strata> yes. actually this is a mongo thing. db.something.save([{some stuff}]) saves as { "_id" : ObjectId("..."), "0" : {somestuff}}
[08:14:42] <oskie_> on the primary node (the oldest node) in the replica set the disk usage is 350G. On another it is 280G. What's all that wasted space? Is there a way to reclaim it?
[08:17:33] <oskie> I forgot - I made a script to calculate difference between storageSize and dataSize some time ago... and that explains it
[08:18:42] <solars> hey, are v2.2.2 and v2.3.2 binary compatible? meaning, can I substitute the old version to test by replacing the binary?
[08:19:58] <Zelest> no idea solars, though, 2.3.2 isn't prod-friendly.. which I assume you're aware of. :)
[08:23:06] <solars> yep I am :) I think madveru told me that they are compatible but I'm not sure if Iremember correct
[08:41:39] <tomlikestorock> is there a way to elemmatch on items in a document and items in a list in that document at the same time?
[08:45:18] <tomlikestorock> that is, elemMatch on a field in a document, and one of the sub attributes of that field is a list, and specify conditions to elemmatch on items in that list as well
[09:02:31] <NodeX> Zelest : http://blog.mongodb.org/post/40513621310/mongodb-text-search-experimental-feature-in-mongodb
[09:14:27] <chovy> ok, to use mongohub...i have to configure my mongodb server to allow remote connections.
[09:19:24] <NodeX> [09:13:24] <chovy> ok, to use mongohub...i have to configure my mongodb server to allow remote connections.
[09:21:59] <chovy> i didn't create a user/pass or anything I figured it would only accept local connections. But the past 2 months my db has been accessible to anyone.
[09:23:22] <solars> by not ready for production use you mean it's not guaranteed to be stable, right - or are there any performance issues because of debug outputs etc?
[09:24:40] <kali> solars: i think it's the same compilation options as the stable builds, so there should not be huge performance issues
[09:26:50] <solars> but I think there are other tickets for count as well, which might be not there yet
[09:27:41] <kali> solars: i'm afraid the optims on count requires a change of format on the index, so you may need to reindex your data
[09:31:38] <solars> kali, hmm but are there any other changes that make it impossible to switch back?
[09:33:33] <solars> the question is if it is wise to install a db without knowing how it is configured :)
[09:36:54] <NodeX> on the plu side, you can always not use Mongo if you feel it's not right for you - free choice and all that
[09:37:53] <NodeX> the reason it's great and fast (not the main reason but partly) is because it takes all these things that a DB shouldn't really handle out of the equation
[09:38:14] <kali> chovy: i don't know... i have this bad habit of spending hours reading on a techno before deciding to download it, so sorry, but i can't relate there
[09:38:44] <NodeX> personaly I read the config file and locked it down to localhost before starting it ever
[09:40:40] <kali> chovy: there is also the problem of keeping things simple for people who like to play with the new toy
[09:41:59] <kali> chovy: i loke to be able to run a mongod --dbpath . & in a shell and just use it as a sandbox to demo, trials and tests... that is not possible to do when configuring gets in the way
[09:42:01] <NodeX> https://gist.github.com/0699bbecc2ee864ce439 <--- easy firewall script ..... put it in a .sh, execute it, once happy use iptables-save
[09:42:51] <Gargoyle> chovy: From a sys admin point of view, you should assume "This shit is open to the world". If you give a server an internet facing IP address without a filewall, it's like leaving your front door wide open while you pop to the shop for milk.
[09:44:00] <chovy> Thankfully, i would only have to fire myself. as this is a side project, and no harm done.
[09:45:00] <Gargoyle> chovy: You might wanna go through this:- http://docs.mongodb.org/manual/administration/security/
[09:45:07] <chovy> this is why i do this shit in my spare time instead of claim to know what i'm doing at work.
[09:45:31] <Gargoyle> Personally, for the last 5 or 6 years, my webb apps connect to mysql using root account and no password.
[09:47:06] <chovy> Gwayne: yeah, that's what i've always done, but mysql by default doesn't allow remote connections. I assumed this worked the same way.
[09:47:44] <oskie> is there a way to speed up oplog recovery? It seems the initial data replication to a new node is quite fast, but when it comes to oplog replay it takes 30s to apply 1min worth of changes
[09:48:56] <NodeX> Gargoyle : root sql account (unless you disable it) has exec privs by default on mysql which means a simple SQL injection wouldv'e rooted your entire server
[09:50:01] <NodeX> new Injection methods arrive all the time, one simply cannot keep up with unreleased ones
[09:57:47] <chovy> for ssh tunneling, do i need to setup a tunnel, or does mongohub do it for me. it is unclear.
[09:59:32] <Gargoyle> looking at the screenshot I would guess the big ass "Use SSH Tunnel" tick box is a hint
[10:01:50] <Gargoyle> In the main Host/port field, I would put localhost:27017, then Bind Address = 127.0.0.1 27017 (Assuming you dont have a local mongo instance)
[10:11:03] <Gargoyle> I combination of rockmongo and the built in shell have served me well over the last year
[10:13:35] <Derick> chovy: http://derickrethans.nl/debugging-with-xdebug-and-firewalls.html has instructions for setting up ssh tunnels
[10:16:04] <Gargoyle> Nice writeup Derick. I think for outbound tunnels, -L is needed instead of -R ?
[10:18:46] <chovy> that port is already used because i have mongo on my mac. But i have no idea how to stop it.
[10:21:48] <chovy> this is throwing an error: db.foo.items.find({ObjectId: 50708c8895f1c16257000002})
[10:25:37] <NodeX> quick question which is confusing me... say I have an images collection and normaly I lookup on gid (gallery_id) to get a list of all images in a gallery ... now I want to adapt my query to sort by _id and put a fed _id at the top but still return a list of all images in the gallery but with my "fed" id being the first in the list .. can this be done (not somehting I have ever tried but one
[10:27:52] <chovy> i'm constantly running into this problem in production only, where i have objects created before I added attributes to the schema.
[10:28:19] <chovy> but now all those objects throw errors, because they have no understanding of a sub-doc, like comments[] for example
[10:29:03] <chovy> so i either need to migrate the data, or put a bunch of logic in my views to accommodate every possible object.
[10:29:53] <NodeX> this is where the line gets blurred from relational mappers - they dont really have a place in Mongo imho
[10:29:54] <Gargoyle> chovy: Mongo is that way by design, so yes - in code you need logic to check for the existence of a specific sub doc before you use it.
[10:29:56] <chovy> which means, if it doesn't have a comments subdoc, then nobody can ever leave a comment.
[10:35:49] <chovy> i think 'jake' will work, it's like rake for node. but i'm not sure if i can run a task on the server
[10:36:09] <Gargoyle> chovy: Hmmm. Personally, I'd go for learning the "native" stuff before you start wrapping and hiding behind an ODM/ORM/Whatever
[10:44:37] <solars> kali, any idea if something regarding auth has changed? I've replaced the binary with 2.3.2 and get: Tue Jan 15 11:42:27.503 [conn24] assertion 16550 not authorized for query on history_production.a
[10:48:55] <kali> one of the dramatic impact of the ORMs is how SQL-illiterate most people yougers developper are
[10:50:08] <Gargoyle> kali: Indeed. In fact, I think that same mentality extends beyond development.
[10:51:08] <Gargoyle> kali: Some people see "apt-get install" or "brew install" and use the commands but never bother reading any further, and wonder why their systems end up in such a mess.
[10:52:26] <Gargoyle> It took me 5 mins to install mogno on OSX - It took another 2 hours reading some basic info about Launch Control.
[10:52:37] <solars> but this discussion is endless, you could also argue that because of this, everyone would have to write assembler :)
[10:52:42] <NodeX> On the flip side of that one can also get caught up in non understanding of things and get to weighed down in "other people's" versions of "the done thing"
[10:53:33] <kali> solars: it's good to have a good understanding of at least one level behind the one you're using daily
[10:53:43] <NodeX> One doesn't have to be a genious to write fast / efficient web apps - one just has to understand signal flow and have a basic knowledge of syntax
[10:53:49] <oskie> you don't need db.fsyncLock if you are backing up using LVM snapshots and the journal is on the same volume as the data, right?
[10:54:44] <kali> solars: and it's vital to have somebody understanding DBs in a team of application developpers, as it is important to have somebody around who can do a bit of assembly, or understand a tcp dump
[10:55:01] <NodeX> ORM's (from the little I know of them) just add another layer of complexity to an App
[10:57:07] <kali> NodeX: some are realy good... salat in the scala world is small, efficient, focus... and because scala is so statically typed, using mongo directlry is really a pain
[10:57:11] <Gargoyle> I see no positive outcome from using a ORM/ODM that requires ether code annotations or public accessors/methods for everything.
[10:57:46] <NodeX> I dont use (currently) a typed language, I'm teaching myself C so perhaps one day!
[10:59:04] <kali> NodeX: well, as all you get from mongodb is basically Map[String,Object] (in scala notation) every time you read a field you need to cast it to the right type
[11:01:05] <Gargoyle> NodeX: Same rule applies when if you replace "working in PHP" with "working on the web"
[11:01:20] <kali> NodeX: the good thing about salat is... it's small. it just maps hash to object. it does not try to write queries for you
[11:56:50] <limpc> hi. does mongo still lock the entire instance on a write? or is it down to the db or collection level now?
[12:06:20] <limpc> where is it in the roadmap? I've tried to find it but it's alot of work clicking all those links
[12:08:01] <noobie25> i'm having trouble testing out geospatial queries. when i runCommmand: geoNear ... i get results, be clearly there are better results available in proximity. I was storing geo x, y values as Float Double ... is that the underlying problem?
[12:08:18] <kali> limpc: this thing is... most of us (in the community, and i guess in the dev team too) think it is useless
[12:08:23] <Derick> with the lock yielding, collection level locking is likely not be going to give a lot of extra performance
[12:09:17] <limpc> well it should help reduce the number of replica sets needed for the same volume of queries, yes?
[12:10:35] <vikash_> I am trying to insert data in mongo but after the first insertion I get "TypeError: object is not a function" [Source code -> https://gist.github.com/4538178 ] Please help
[12:10:45] <limpc> the main reason for replica sets is to mitigate backlogs from large numbers of inserts (e.g. high user concurrency), correct?
[12:13:33] <limpc> so i would think that by adding collection level locking, you'd be able to get more concurrency per shard, and reduce the number of shards necessary to support certain volumes
[12:15:42] <limpc> hm isnt that like saying you're going to saturate a computer's northbridge before you max out all 6 cores of a 6 core cpu?
[12:17:01] <webber_> [13:07] <kali> limpc: this thing is... most of us (in the community, and i guess in the dev team too) think it is useless + [13:13] <@Derick> limpc: "think" is not a good benchmarking tool :-)
[12:20:43] <noobie25> Derick: hi derick. i verified that values are being stored in the following order : lng, lat
[12:20:59] <limpc> ok well i worked at zynga. one of the reasons we decided to forgo mongo last year was because of the locking. it caused concern where high loads were involved, and we werent convinced its speed justified the cost of the # of servers we'd probably need to replace mysql/memcached. Now I'm at another very high volume business and there's some similar concern.
[12:23:01] <noobie25> Derick: however, when querying for San Francisco region: (122.41, 37.77) i get results from korea (127.23, 38.31) so i think it has something to do with these lng lat pairs.
[12:24:36] <vikash__> I am trying to insert data in mongo but after the first insertion I get "TypeError: object is not a function" [Source code -> https://gist.github.com/4538178 ] Please help
[12:25:01] <NodeX> vikash__ : it's in your driver , please see if you can paste the raw query sent to mongo
[12:25:42] <noobie25> i'd rather store my lng, lat at strings ... do you know if mongo supports that datatype as well?
[12:28:46] <vikash__> ok, but In the first time the dataToPush is getting inserted into the collection. Problem is, at the second time its giving me the error
[12:36:24] <NodeX> can you goto the shell and pastebin the output of db.<your_channel_collection>.getIndexes()
[12:41:24] <vikash__> what do you mean by raw query? I have an object dataToPush. and when a user sends a message, it updates the fields in dataToPush and I am trying to insert this in mongodb
[12:42:01] <NodeX> the raw query sent to mongo .... I/e what the string looks like when it leaves the driver
[12:43:06] <vikash__> By any chance did you mean this -> result ->[{"name":"Vikash","msg":"Hi","channel":"null","is_spam":false,"is_delete":false,"timestamp":"2013-01-15T12:31:38.054Z","_id":"50f54c44206f74e145000001"}]
[12:43:15] <vikash__> Data -> {"name":"Vikash","msg":"Hi","channel":"null","is_spam":false,"is_delete":false,"timestamp":"2013-01-15T12:31:38.054Z","_id":"50f54c44206f74e145000001"}
[12:43:16] <vikash__> debug - websocket writing 5:::{"name":"add_message","args":["Vikash","I am Vikash"]}
[12:43:26] <vikash__> Data -> {"name":"Vikash","msg":"Hi","channel":"null","is_spam":false,"is_delete":false,"timestamp":"2013-01-15T12:31:38.054Z","_id":"50f54c44206f74e145000001"}
[12:45:06] <vikash__> debug - websocket writing 5:::{"name":"update_users","args":[{"0":"Vikash"},{"0":"Agrawal"}]} ?
[12:45:59] <vikash__> NodeX, just one thing, with app.'s you can see https://gist.github.com/4536605 and run it, In worst case :)
[12:54:30] <kali> webber_: my "thinking" relies on four years of mongodb prod experience, and actual study of the code, not a general principal...
[13:00:34] <vikash__> NodeX, debug - websocket writing 5:::{"name":"add_message","args":["asda","asd"]}
[13:12:30] <Guest_1448> I've tried .find({$where: 'this.foo != this.subdoc.foo'}) but that fails with subdoc has no properties errors
[13:23:18] <vikash__> I edited my JSON instead of dataToPush and made it individual in db.createCollection() and it works now
[13:32:51] <woozly> guys, how to join queries? I have two collections: db.queue( { "myid": 12345, "parent_id": 9126} ); and db.parents( { "myid": 9126, "count": 0 } ). How I can update db.parents, buy finding db.queue by id? I mean: queue.find({ "myid": 12345}) <--- got from this result 'parent_id' and use it, for increasing parents() record :/
[13:34:31] <woozly> In SQL: UPDATE parents SET parents.count = 999 LEFT JOIN queue q ON q.myid = 12345 WHERE parents.myid = q.parent_id; (something like this, if I don't do it wrong.. :)
[14:37:32] <solars> I've tried to substitute 2.2.2 with 2.3.2 to check if the performance of my count queries improves (it's a huge bottleneck)
[14:38:19] <solars> the queue had 20k messages, and I was able to insert those 20k in around 1 minute
[14:38:38] <solars> with 2.3.2 the queue was filled up faster than the messages were inserted into the db
[15:10:33] <amimknabben> http://www.allbuttonspressed.com/blog/django/2010/09/JOINs-via-denormalization-for-NoSQL-coders-Part-1-Intro
[15:20:38] <doxavore> What kind of connection pool sizes are people using in multithreaded servers? I've tried everything from 10 to 512 in JRuby and I still run out.
[15:22:40] <mansoor-s> say I have a value: "MyAwesomeValue" I want all documents that have the term Awesome in it
[15:29:08] <MatheusOl> doxavore: the pool size will depend on your box capacity and the max number of connection
[15:30:05] <MatheusOl> mansoor-s: I think only map-reduce will solve that, at least if you don't know what are the keys
[15:40:34] <Derick> gist is not a type of index, but just a data structure (says wikipedia: http://en.wikipedia.org/wiki/GiST) - if you mean their tsearch2 index, then that is just FTS - which needs split up words (http://www.sai.msu.su/~megera/postgres/gist/tsearch/V2/docs/tsearch-V2-intro.html) http://www.postgresql.org/docs/8.3/static/textsearch-indexes.html also talks for gist about "each word" and for gin about "GIN indexes store only the words"
[15:41:40] <mansoor-s> is it acceptable to do map reduce-es as a normal function of the application?
[15:41:44] <webber_> LIKE can be answered from a plain RDBMS index if LIKE uses an index prefix, e.g. LIKE "A%".
[15:43:49] <Derick> mansoor-s: and, I would not use M/R unless you *really* need it - and you probably won't as we now have the aggregation framework
[15:45:31] <MatheusOl> webber_, Derick: What I'm saying is that PostgreSQL can, in some cases, use index with LIKE, even when it comes with suffix: LIKE '%bla%'
[15:45:57] <MatheusOl> And GIN or GiST indexes (yes they are data structure, and also can be used as indexes)
[15:47:36] <MatheusOl> Derick: I agree, I just mentioned it wouldn't use an index, but it's not always a bad thing. Also, generally, we expect some slowness searching with regular expression (compared to equally searchs)
[15:47:50] <mansoor-s> Derick, We are creating a search functionality for our application. We want to be able to search by value contents.
[15:49:42] <Derick> mansoor-s: can you provide a proper (slightly) complex document as well as what you're trying to look for?
[15:51:26] <mansoor-s> Book titles. I have a book (document) with Title(key) with "Harry Potter" (Value). When I search Harry, i want this document to be returned
[15:52:08] <mansoor-s> and its not necissarily by individual words either, so it could be Harrypotter and I still want it returned
[16:00:30] <MatheusOl> This may help: http://docs.mongodb.org/manual/tutorial/model-data-for-keyword-search/
[16:05:26] <NodeX> [15:51:04] <mansoor-s> and its not necissarily by individual words either, so it could be Harrypotter and I still want it returned
[16:07:43] <Gargoyle> NodeX: You couls still use regex /^harry/ will match and use an index, but the app will have to convert all keywords to lowercase
[16:09:18] <Derick> mansoor-s: it will not be *as fast* as with an index - but I'd argue it's still going to be as fast as any other rdbms would do it
[16:09:22] <MatheusOl> NodeX: I meant, if performance is really an issue, one can change requirements. But I do agree, doesn't do what he really want
[16:14:13] <NodeX> 2.4 will address some issues but it was never meant to be a replacement for an RDBMS with FTM
[16:53:49] <coredump> So, I need to remove a replica member but it is the primary atm, use that procedure to force the primary to move to the other server and then stop/remove the primary?
[17:59:25] <konr_tra`> Is there a tool to effortless import gigabytes of kinky csv files (valid, but with entries containing newlines, escaped quotes and all sorts of nastiness) to mongo?
[18:36:44] <limpc> Our mongo struct is fairly complicated, we were wanting to make each user their own collection for speed. but --nssize is limited to 2047, which is just under 3.26 million collections
[19:49:35] <themoebius_> is there a way to reclaim disk space in place yet? i.e. without exporting the whole DB and reinitializing like in a repair?
[19:57:34] <UForgotten> themoebius_: there is a compact function that I read about, but use it at your own risk
[19:58:14] <themoebius_> UForgotten: yeah but that doesn't reclaim disk space, it just allows mongo to reuse space it's already claimed
[19:58:42] <themoebius_> I mean my /db partition is like 95% full but the actual data size is far less
[20:16:50] <akeem> hi, I am trying to track down a slow process. its ok when I do an explain on the query but when I profile it takes a good amount of time
[20:35:04] <owen1> i have replica set of 3. i killed the primary and new primary was elected. i killed the new one but the last survivor, the 3rd, is still secondary. is it normal?
[20:36:32] <Gargoyle> Your live and healthy nodes need to have a majority of your original configuration.
[20:48:25] <Gargoyle> owen1: The final server will not self promote as it does not know if the other servers are dead, or it IT has been segmented on the network.
[20:51:13] <Gargoyle> owen1: I can't remember exactly, but this is the general area of the docs. http://docs.mongodb.org/manual/tutorial/reconfigure-replica-set-with-unavailable-members/
[20:56:04] <owen1> Gargoyle: so stick to 3 or 5. i want to configure my hosts with a bash script. i don't want to run mongo console and type manual commands like rs.initiate, rs.add etc. is it possible and is there an example for doing that?
[21:05:57] <owen1> Gargoyle: oh. so u can dump data into the mongodb collections that will setup the replica set etc?
[21:07:14] <Gargoyle> if you redirect (or pipe) to mongo shell, its the same as if you were running those commands by typing them.
[21:10:03] <Gargoyle> owen1: Or you could write a script in whatever language you want, and pipe correct output to mongo
[21:14:09] <owen1> btw, i noticed that eventough i have replSet = abc in /etc/mongodb.conf it's being ignored, so i had to add it to the mongod commmand.
[21:58:26] <Hoverbear> Hi all, I'm working with Mongoose and have a query result (from like Foo.findById(myId)) and have assigned it some something… But now when I try to use the .id function on a sub doc I'm getting errors that it does not have such a method… Any ideas?
[23:15:34] <Derick> What's with all the new extra whitespace on the osm.org pages?! Lots of scrolling needed now :-/
[23:48:57] <owen1> what triggers an election for a new primary? only loss of connection to a member or are there more conditions like high cpu/memory/diskspace?