Log file Viewer
#mongodb logs for Monday the 22nd of July, 2013
[10:54:08] <pl2> Is there a quick one line command available to allow me to import an entire /dump folder?
[11:01:14] <kali> mongorestore with no argument will look for a "dump" folder in the current path and load everything
[11:54:04] <singh_abhinav> I am planning to create auto increment field by using counters collection ... what can be the potential disadvantage of this approach when we will go for sharding
[11:55:43] <kali> i guess you will not shard the counters collection, so... none. i would expect the overhead to be constant
[12:00:29] <singh_abhinav> kali: in some of the cases mongodb ids seems very big ..like userid or orderid ...in that case what should we do? mongodb ids seems too big :)
[12:03:02] <kali> they are 12 bytes... unless you've adopted very short field names, chances are this is negligible
[12:05:41] <rspijker> I'm having some 'issues' on a db setup which I don't quite understand. I'd like to run it by the channel to see if my understanding of things is correct and see if somebody might know what is going on.
[12:06:22] <rspijker> The issue is regarding storage space. In the system I have running there are frequent removals of large amounts of data from collections followed by large insertions and again large removals, etc.
[12:07:25] <rspijker> My understanding of mongodb storage is that it keeps a list of deleted records, in size buckets (256-512, 512-1024, …) and when a new insertion takes place, it will check this list and grab something suitable from a bucket, or grab something from the next bucket up.
[12:08:55] <rspijker> That does not seem to be happening for me… The storage on the system just keeps on growing (150GB for a <2GB data set). So I'm just wondering what I'm missing exactly.
[12:37:56] <puppeh> hey, what would you recommend me if I want to add the sum of all fieds of multiple documents, mapreduce or aggregation framework?
[12:40:34] <Nodex> you do realise that mongodb doesn't do anything with these things internaly for example joining at query time don't you?
[12:41:40] <CIDIC> because the _id is an object not an int I wasn't sure if I needed to make it an object to match?
[12:42:34] <CIDIC> although it says ObjectId("xxx") in quotes so maybe it is a string? I am just asking what I should do to match it? int? string? obj?
[12:46:58] <CIDIC> Nodex: if I have something like Categories: name, description, content, and categories have articles. and I am making something where I want to goto /category/foo/ and it fetches the category name, description, content ect, and all articles in that category how would you recommend designing the schema?
[12:47:59] <CIDIC> bc I want to be able to aggregate articles based on things like date or author not just by category
[12:48:32] <CIDIC> Nodex: they could the name/description is essentially page content in this scenario
[12:50:34] <CIDIC> I am basically asking what do you do when you have multiple aggregates of generally equal importance?
[12:51:04] <Nodex> I don't have a clue what your schema looks like, a typical document or what you're trying to achieve so I cannot comment on "aggregating by author"
[12:52:17] <CIDIC> Nodex: I am familiar with sql, trying to understand how you would accomplish equivalent tasks in mongodb.
[12:53:14] <CIDIC> if I had articles: id, name, content, publish_date, author_id, category_id; authors: id, name; categories: id, name, content;
[12:55:00] <CIDIC> Nodex: so the author and category data would be copied for each articles. that is fine. what if the author and the category data are somewhat large? what if a category has like 10 fields?
[12:58:27] <remonvv> CIDIC, the downsides are that your application has to enforce consistency, (so update dduplicated data if the original data changes) and average document size. You should not embed anything that's unbounded in size.
[12:59:23] <remonvv> The former only applies if (parts of) the embedded data is copied from other documents rather than being the only place where that data exists.
[13:02:44] <rspijker> The issue is regarding storage space. In the system I have running there are frequent removals of large amounts of data from collections followed by large insertions and again large removals, etc.
[13:02:47] <rspijker> My understanding of mongodb storage is that it keeps a list of deleted records, in size buckets (256-512, 512-1024, …) and when a new insertion takes place, it will check this list and grab something suitable from a bucket, or grab something from the next bucket up.
[13:02:54] <rspijker> That does not seem to be happening for me… The storage on the system just keeps on growing (150GB for a <2GB data set). So I'm just wondering what I'm missing exactly.
[13:02:54] <kali> rspijker: have you tried to compact and/or repair to validate you're actualy suffering from fragmentation ?
[13:03:30] <remonvv> rspijker, ah, familiar problem. It's documented as you describe IIRC but in my experience storage size only guess down or is reused after compacting.
[13:04:17] <rspijker> kali: I checked db.collection.storageSize() - dataSize though and the difference was 127GB
[13:28:18] <remonvv> rspijker, okay. For me it does reuse disk space after removes but it's somehwat erratic. For example, if you generate 10k documents {_id:.., a:1}, call stats, remove them, add them, and call stats again it will have reused the disk space.
[13:28:44] <remonvv> rspijker, however, in more "production-ish" environments we do so ever growing disk usage even though the dataset size remains roughly equal.
[13:46:53] <benza89> Hi all , Can someone explain me what is the meaninig of Roolback of a menber of a replicaset ?
[13:48:22] <kali> "rollback": when a switch over happens, your replica set may have two nodes believing they are master for a few seconds
[13:48:47] <kali> so the write ops accepted by the one that will not stay master have to be reverted
[13:58:48] <benza89> kali, I have in my replicaset two nodes and they are both beliving that they are master ,I want to reconfogure my node to act like secondary node
[14:05:20] <benza89> @remonvv , But I already set a repilcaset of only one node to perform my tests of sharding , so if I use a replica set of 2 menbers it can not work ?
[14:08:45] <remonvv> repsets do not have value until they have 3+ (and uneven amount of) members. You can use arbiters instead of data members but that's basically what it is.
[15:28:34] <drag> Hi. When I do a query on mongo, is there a way to specify it to ignore case? For example, I have an object with name Ed and one with name ED. Can I have 1 query that returns both?
[15:29:47] <remonvv> You can use $where or store an additional lowercased version of values in your document and query for that
[15:30:54] <drag> remonvv, is this the where you are talking about: http://docs.mongodb.org/manual/reference/operator/where/
[15:31:31] <remonvv> drag: Right, but it's powered by the JavaScript engine embedded and thus slow. I'd avoid it for any project that isn't homework or a hobby project.
[15:32:55] <remonvv> k610: You should secure mongo by not letting anything connect to it that shouldn't. Binding IPs works although I don't see the value in binding it to 127.0.0.1. Local client?
[15:33:06] <Number6> k610: There are many ways to secure MongoDB. Binding it to localhost is one, setting up IP tables rules to block inbound accesses to port 27017 (mongoDB default) and a username & password
[15:35:27] <remonvv> k610: All works, as long as the end result is that nothing can connect to your mongod that isn't allowed to. Up to you. Not a mongo specific thing.
[15:35:58] <remonvv> k610: Only thing is that if you're using mongo auth I feel you're probably doing something wrong on the security end of things.
[15:36:32] <k610> remonvv, i see some app could get corrupt and scan ports and access to a passwordless mongo
[15:37:09] <benza89> I try the db.admincommand ({removeShard:"<shardname>"}) and then the move to primary but it did not work
[15:37:41] <remonvv> k610, there shouldn't be anything that can touch it ;) Anything/anyone that can get to your private network probably has enough access to make auth not matter.
[15:38:11] <remonvv> benza89, you mean it went down and still has chunks is primary for one or more dbs?
[15:38:13] <k610> remonvv, that's what i was thinking password are always going to end up in a config file anyway
[15:41:35] <mjburgess> what's happened ot $oid ? how can i convert a string to ObjectId in php now? (MongoClient)
[15:53:42] <benza89> How I can remove a shard which is no more available and still primary for a database ?
[15:59:12] <remonvv> benza89, there is no (easy) way to fix that without the shard being up since you have no replicas of that shard apparently.
[16:04:35] <rspijker> remonvv: On my storage issue, ran repairDatabase(). My previously ~140GB storage requiring set now uses 4GB :P
[16:08:01] <rspijker> yeah, I have a theory on what's causing it. All of my records are *almost* the same size. What happens is they all go into the same freelist bucket. But the freelist algorithm only checks 30 before moving on to the next bucket (which will be pretty much empty). Furthermore, they are sorted on size, so if a new record is larger than the 30 smallest ones, it doesn't pick anything from a freelist bucket.
[16:15:55] <kali> rspijker: so you'll know pretty quickly if powerOfTwo will work or not. in case it does not i suggest you open a jira ticket, because discussions here are more revolving around mongodb usage than internals... you may get more help in the jira db
[16:16:24] <fredjiles> Does anyone know or can direct me on how to do either an aggregation or map reduce where I need to group by one field and get a count value based on whether another field $exists
[16:16:36] <remonvv> I think this came up way back when. At the time there was some resistence to have a period pass in the server that would clean that sort of thing up (find adjacent empty entries and join them into a single one, etc.)
[16:16:52] <rspijker> kali: sure, just wanted to run my understanding of all this by the channel, see if I missed something obvious :)
[16:17:31] <remonvv> fredjiles: What are you looking for? Info on how AF works or for someone to provide an example for that specific question?
[16:17:33] <rspijker> remonvv: I agree, I found a couple jira issues with suggestions for solutions to similar issues
[16:18:20] <fredjiles> I looked at the AF docs, but I don't understand how to put in the $exists check
[16:18:25] <kali> fredjiles: AF should work. you need a $project with a condition to count your record as open on closed at some point
[16:19:45] <benza89> It is the same question : how can I remove one of my shard who went down and was prilay for some database
[16:19:52] <remonvv> fredjiles: $match and normal queries have the same query capabilities. You can use $exists in $match
[16:20:23] <remonvv> benza89: If you ask the same question you'll get the same answer: You have to recover the shard and bring it back up. It has data that currently exists nowhere else.
[16:20:39] <remonvv> benza89: Alternatively you dump what's left and start over with the data you can save.
[16:23:31] <benza89> @remonvv , the problem that I have no way to do a back up , so if some data is lost it's not problematic for now I just want to get back the contol of my shard cluster
[16:28:46] <remonvv> benza89: The problem is that for those databases that had the lost shard as primary you will have lost ALL non-sharded data.
[16:30:53] <remonvv> benza89: If you can live with that and potentially cluster meta data corruption then you can do this : Go to db.databases and update the "primary" flag of all databases currently primaried on the lost shard. Then invoke a router flush on every mongos.
[16:31:51] <_pash> can i have an incomplete search for a mongo object? for example if the name is Andrew, how can i find it with just and?
[16:31:53] <benza89> @remonv In my mongos I cant show dbs for example and I want to change this shard node by another one to execute some tests
[16:31:53] <remonvv> But I would consider your cluster in "borked" state after manually screwing with the config database
[16:32:36] <remonvv> benza89: I'm hearing some mutually exclusive things so it's getting a little confusing
[16:33:23] <remonvv> _pash: You can try $regex but MongoDB isn't very good at text search yet. 2.4+ has text search but I have little experience with it : http://docs.mongodb.org/manual/core/text-search/
[16:34:40] <z2z> n00b question about db-layout: many folders that conain many files (no folder nesting); both have metadata; possible query: all files x='z' within folders y='w'. Is there a smart way without denormalizing all the data into the file collection? is denormalization really not that bad? (classic sql and join does not work, because there is no fixed set of metadata keys) thanks a lot!
[16:35:24] <benza89> remonvv, the problem is that I can not execute any command (even db.printShardingStatus())
[16:35:24] <_pash> remonvv: is there any workaround? because i am trying to make a search bar, kind of like the autocomplete on facebook
[16:37:35] <remonvv> benza89: Are you sure it's just a down shard? A down shard doesn't suddenly disable the ability to work with your database.
[16:39:02] <remonvv> _pash: Note that it's not indexable (with the exception of beginning anchored, case sensitive regex) so it's not ideal
[16:41:15] <remonvv> _pash: Not sure if it's even a database you need. We usually store the auto-complete lookups for the first X characters in memory on the app servers.
[16:41:31] <remonvv> You don't really want to send a query to a database everytime your user types a character.
[16:42:33] <_pash> remonvv: the problem is that under one db i could have approx 500+ users i would need to search for, and there could be several DB's being searched at the same time, kinda like FB
[16:45:03] <Nodex> you also need to do your javascript right.... you start a timer when you press a key and kill it when you press another - starting one again and timeout the send.... you dont send for every char - just send the buffer every X ms
[16:45:36] <fredjiles> remonvv, I have the first part working (http://pastebin.com/c7Zsj4t5) how do I do the second part to get the case where closed_date: {$exists: true} and store them in closed property
[16:47:14] <_pash> remonvv: Nodex: yes, but i wont get any suggestions. if my ajax sends the data of 'an' then i query the db for 'an' it wont give me a result, which i could suggest to the user
[16:48:11] <remonvv> fredjiles: You need to do a $match before the $group for your closed cases, then combine both $groups into a single result
[16:49:18] <remonvv> _pash: Seriously, nobody that has more than 2 users does that against the db ;) Populate your username autocomplete cache during startup (sync or async) for the first X characters (if your max username length is higher than, say, 16) and autocomplete in memory
[16:50:06] <Nodex> I never auto complete less than 3 or more than 10 chars and anything that's large is thrown into redis in chunks
[16:51:14] <Nodex> I can auto complete on 27M UK postcodes in about 24ms directly into mongo - that includes all rendering too
[16:51:17] <_pash> Nodex: i am very new to this, on the contrary to you, could you then suggest on how i should do it?
[16:51:19] <remonvv> NodeX: Redis is "in memory" I would argue ;) But yeah. Depends on usecase if you should have a max for auto complete though.
[16:52:24] <Nodex> _pash : firstly I would ngram my words / phrases and regex on that (in lower case)
[16:52:28] <_pash> Nodex: so lets say, i have a bunch of objects with names in a collection in mongo. and i send an ajax get request to the server, what do i then do with that data?
[16:53:09] <Nodex> for example .... andrew gets ['and', 'andr', 'andre', 'andrew'] and added to a cache (to avoid dupes)
[16:53:36] <remonvv> NodeX: Everybody can, but you normally shouldn't for systems that serve thousands of queries per second. It's just not something that should typically be thrown at the database. That's what Redis or other in-memory solutions are for. We often have ~100,000+ people in our systems at any one time and I wouldn't want them all throwing auto-complete queries into the database.
[17:12:11] <z2z> n00b question about db-layout: many folders that conain many files (no folder nesting); both have metadata; possible query: all files x='z' within folders y='w'. Is there a smart way without denormalizing all the data into the file collection? is denormalization really not that bad? (classic sql and join does not work, because there is no fixed set of metadata keys) thanks a lot! (second attempt; somthing wrong with style of question? shou
[17:14:02] <spicewiesel> I created an admin user as described (http://docs.mongodb.org/manual/tutorial/add-user-administrator/), then I looged in to mongodb using "mongo admin -u admin -p" to check my shard status, but I am not allowed to do so. Could anybody help me with that?
[17:16:24] <spicewiesel> the message is: "not authorized for query on config.version", my user is "admin" and has the role "userAdminAnyDatabase"
[17:24:59] <pngl> I am trying to understand how the update $ works. Doc says: "...acts as a placeholder for the first element that matches the query document", but how is that match chosen? For instance, with collection [({a:[{b:0,c:false},{b:1,c:false}]}], update({'a.b':0,'a.b':1},{$set:{'a.$.c':true}) updates the element of the a array with b:1 regardless of the ordering in the array but dependent on the ordering in the query.
[17:49:42] <pmxbot> http://docs.mongodb.org/manual/reference/method/db.collection.ensureIndex - db.collection.ensureIndex() - MongoDB
[17:54:47] <jayd16> Hi, I'm seeing a crash bug when on the mongo server. I can't really find a ticket in the mongo jira that's close, so my coworker made one. https://jira.mongodb.org/browse/SERVER-10285 Has anyone seen then? What else can we do to work around this?
[17:54:48] <_pash> Nodex: I will do when I get the time, the docs are massive, I learn best through practice.
[18:02:06] <Nodex> ok, if you don't want to read the docs then I wish you good luck because I wont help soeone who cannot be bothered to help themselves :)
[18:32:00] <greenmang0> I removed huge data from our production mongo replset, but I noticed that there's no reduction in space, so after asking around and googling I realized that I have three options 1) repair db 2) compact 3) resync
[18:34:14] <greenmang0> what i was expecting is, if i add one more replset member as a secondary, it would sync with primary and take less space, and then i would promote it to primary
[19:34:23] <zachrab> how can i construct a query for returning a document in collection which has a subfield value that is not null
[23:13:17] <dougb> Is there a better way of seeing what's happening with mongoimport? I'm trying to import a 2 GB collection, but it's stuck at filesize: 0