Log file Viewer
#mongodb logs for Thursday the 14th of August, 2014
[00:20:56] <ToeSnacks> I have hit a namespace limit on my 2.2 environment and have added -nssize 256 to the end of the init script. What should I now do to allow the affected member to rejoin the replica set?
[02:44:12] <delinquentme> Is there a way to verify that the STRUCTURE of a DB entry in mongo is sound?
[02:44:40] <delinquentme> we've got a working login system but anything which is hitting a particular collection is making the app take a pew
[05:24:03] <pratikbothra> How do you handle updating populated objects in mongodb? Take the use case of likes in post (likes: [{type: Schema.ObjectId,ref: 'User'}])
[05:24:10] <pratikbothra> While retrieving, I use populate and get the user name, link along with their id's. But on updates back to mongodb, I'm facing a lot of trouble as mongodb is expecting an array of id's, while I'm sending it populated objects back. What is the best way to handle this?
[05:25:52] <pratikbothra> On post update, I do something like this var post = req.post; which is followed with _.extend(post, req.body); ....Now imagine handling post.likes (both likes being added and people removing their likes), its so dirty
[05:28:23] <sflint> no sure i understand the issue. What doesn't you document look like? you just want to keep could of likes on a post?
[05:29:02] <sflint> to update a document you can just use 'db.collection.update({<query>},{<update>})'
[05:30:44] <sflint> can you just tell me on the most basic level what you are trying to track? what data needs to be stored?
[05:36:22] <pratikbothra> sflint: But at the moment we had a consolidated update function for posts.
[05:37:20] <pratikbothra> So in posts document, the title, body, any of its attributes could be updated. And the line _.extend(post, req.body); worked well.
[05:37:59] <pratikbothra> In a typical post document likes could be ['53cec89600c915dfcd0231d0', 53cec89600c915dfcd023123', '53cec89600c915dfcd023122']
[05:39:23] <pratikbothra> The problem being that when the data is retrieved we do a .populate('likes', 'name username picture')
[05:41:20] <pratikbothra> Do I have a separate route for like and unlike , something like postId/like/userId and for unlike postId/unlike/userId ?
[05:41:53] <pratikbothra> for me to tell mongo to automatically convert all these objects to an array of ids before saving it?
[05:44:50] <pratikbothra> tyteen4a03: well, sorta. It also gives you the timestamp. What are you exactly looking for?
[05:45:26] <davasaurous> Is there a go-to way of configuring the collections of a new MongoDB DB post-deploy? I'm looking for a script or the like for setting up uniqueness constraints and indices and the like
[05:45:39] <sflint> pratikbothra: $in allow you to say "update all document that match" $in : [ <id>,<id>]
[05:46:16] <sflint> davasaurous: you can script it...or just do it in your code....it is all dynamic
[05:48:17] <davasaurous> We're not using node and I'd rather not install it just for migrations, thanks for the tip though pratikothra we use node for other projects
[05:49:04] <davasaurous> sflint I know, I'm more looking for a nice way to have an automated series of those run when we initialize a machine or run our server
[05:49:08] <pratikbothra> +1, just the shell....And db.collection.ensureindex ....Mongodb must be v2.6 though
[05:50:26] <sflint> davasaurous: you would have to edit anything that you used. Just as easy to have a indexes.js, indexes-qa.js
[05:51:52] <sflint> davasaurous: i don't know if there are new collections, but you don't want to create indexes on the fly usually...there is a cost of locking associated. You can use the {backgroud:1 }
[05:52:32] <sflint> pratikbothra: still not sure what you are trying to do, but likes and unlikes should be easy to handle in mongo.....you could use a serperate route/resource for each, but mongo doesn't care
[05:55:04] <sflint> 'db.collection.update({_id : $in : [<id>,<id>,<id>]},{$inc : { like : 1 , unlike : 1})
[07:13:02] <BurtyB> hmm these collapsing headings on the website are driving me nutts which is a bad thing this early in the morning :/
[07:15:16] <rh1n0> Are there scripts that can be run against mongodb to report on missing indexes (similar to scripts for mysql or postgresql for example that do the same thing?) - been looking for them :)
[07:21:02] <sflint> rh1n0: what does you schema look like? and query pattern? i can tell you what index to use
[07:24:04] <sflint> rh1n0: a good rule is if you do db.collection.find().explain() and you see "Basic Cursor" you need an index. But to build to scale you want to use "indexOnly" queries
[07:24:22] <rh1n0> im very new to this stuff. I have a query that looks like this paste http://pastebin.com/4dbpFtJt
[07:27:17] <sflint> rh1n0: why are you using aggregation framework? Is there some count you are looking for?
[07:30:43] <sflint> rh1n0: but it is more flexible...you can pipe a set to another action in aggregation
[07:31:18] <sflint> with a find() you return a cursor to the applicationa and would need to do the aggregation in your app
[07:33:28] <rh1n0> im usually doing ruby and cloud automation stuff - today i got de-railed into mongodb - fine by me. Always wanted to work with it more.
[07:36:41] <rh1n0> i tried to remove the aggregate but attempting to run the query gives me the error: "$err" : "Can't canonicalize query: BadValue unknown operator: $match"
[07:38:36] <rh1n0> should i be using this format instead? db.collection.find( { field: /acme.*corp/i } );
[07:39:38] <rh1n0> as i look thru the mongo docs i dont understand why this query was written this way :)
[07:43:06] <sflint> you should validate that server side...and make the client only about to choose "Texas"
[08:10:09] <rspijker> rh1n0: (I only read the first prt of your query about looking for index suggestions) have you looked at Dex?
[09:00:31] <inad922> Is there a way to make a query against a field in such a way that I want to check if the string I'm querying the field with is a substring of the value of the field?
[09:01:42] <inad922> nfroidure_: Could you give me a link to the appropriate mongodb documentation page?
[09:17:05] <Aartsie_> i will make a mongodump but get this strange error: db already exists with different case already have: [DBNAME] trying to create [DBNAME]", code: 13297
[10:28:09] <tmh1999> Hi guys, I am new to mongodb and I am using java driver. I have this kind of error and I don't really know why.
[10:29:46] <tmh1999> when I insert a BasicDBObject with { "PATH" : "/" }. then I insert the same BasicDBObject with different key-val I get this exception com.mongodb.MongoException$DuplicateKey
[10:30:45] <rspijker> tmh1999: if you don’t explicitly specify an _id element, the driver generates one automatically
[10:32:06] <tmh1999> so I should do like this : BasicDBObject myoj = new BasicDBObject("_id", "mykey") to solve ?
[10:34:04] <Derick> sorry, remove(): http://docs.oracle.com/javase/1.5.0/docs/api/java/util/HashMap.html?is-external=true#remove%28java.lang.Object%29
[10:34:09] <rspijker> tmh1999: just to be clear, setting it to mykey won’t work. You have to set it to a new value every iteration, that’s the whole point
[10:34:54] <rspijker> also, creating a new object every iteration shouldn’t really be such a huge deal…
[10:36:39] <rspijker> col.insert(new BasicDBObject().append("Value", (int)(Math.random()*79)).append("ID", UUID.randomUUID().toString()));
[10:52:03] <sfix> hi guys, when creating a new user with the "readAnyDatabase" role, do I leave the database param blank or set it to "admin" ?
[11:05:02] <uhrt> so i’m using mongo’s text search in a python web app i’ve been building. Thing is, all the data’s in greek, but text search has got no support for greek. This is a bit of an issue 'cos people will often omit diacritics, and mongo won't find a match for, say, 'ενα' (should be 'ένα'). Right, so what i've done is duplicate fields i wanna make searchable, strip all diacritics, and make them lowercase. When a user performs a search, the search text
[11:05:03] <uhrt> has its diacritics stripped and made lowercase before we query the db. What i'm wondering is if there's something else i could or should be doing that won't mean cluttering the db with special search fields (that is, apart from resorting to using lucene/solr/fml)
[12:04:31] <Siyfion> Given an example collection of products: https://gist.github.com/Siyfion/d10d2a2894534a0cf7ed
[12:04:52] <Siyfion> Is there any easy way I can run an update command to remove all "link" values that match a given ID?
[12:07:23] <Siyfion> eg. If I wanted to null the values that currently reference "539ed92ea53e1402005f888f", I'd like the 'fields.image.link' on object 1, the 'fields.other.link' on object 2 and 'fields.image.link' on object 3 to be nulled.
[12:40:46] <remonvv> Siyfion: If you mean remove them from both the "image" and the "title" embedded documents then the answer is no. If you only need to do so for "image" then it's possible through update({'fields.image.link': <YOURID>}, {$unset:{'fields.image.link':1}}, false, true)
[12:41:06] <remonvv> Pretty sure that what you actually want is the former, and that's not possible as there are no wildcard operators.
[12:42:16] <remonvv> Then yes, a different schema. It needs to be an array of elements where one of the element fields is "type". That way you can select all elements that match one or more types and specific link values in one go.
[12:44:31] <remonvv> It's simply better. You rarely want to store a collection as something other than an array. Even maps are generally best stored as an array with elements that contain both key and value(s).
[12:51:41] <tuxtoti> Hi. I'm trying to figure out why a particular query is taking a long time to execute..and in the process learning .explain's output.
[12:52:47] <tuxtoti> i just can't understand why my query is taking a long time even though the querying field is indexed (and being used according to explain)
[13:27:05] <joannac> that means the index entries that have c4_in:null contains documents where either c4_in is the value null, or c4_in doesn't exist
[13:53:11] <cocotton> hey guys, when trying to login mongo (runing mongo command), I get the errno:111 connection refused. From what I can see, mongos is not running
[13:56:19] <cocotton> Yet on other machines (other environment) I also see a mongos process listening on another port
[13:56:51] <cocotton> on other machines, I got mongod listenening on 27019 and mongos listening on 27017
[14:00:51] <BurtyB> does anyone know if failIndexKeyTooLong is likely to stay around or is it meant just for transitioning to 2.6?
[14:02:11] <cocotton> @rspijker From what I can see, this is a "newer" version of mongo, and the package mongo-10gen has been replaced
[14:04:34] <rspijker> seems weird that you only have mongod and mongo, that would mean someone installed those separately
[14:04:49] <rspijker> or they haven’t been replaced or something. Hard to tell without knowing your setup
[14:05:46] <tuxtoti> joannac: I'm actually interested in documents where c4_in doesn't exist. Yes, I could have used $exists but that didn't help either.
[14:08:26] <tuxtoti> joannac: I don't understand ..when you say it has page all 12M+ docs. I 'm having a covered query right? (the projection has c4_in:1 and _id:0 ) ?
[14:24:32] <tuxtoti> rspijker: ..yeah I read that somewhere... a lot of confusion over whether it uses the index or not.
[14:25:32] <tuxtoti> rspijker: thats precisely why i use c4_in: null which should return documents which don't have c4_in and which has c4_in 's value set as null
[14:26:47] <rspijker> if the index were to store it as null regardless of whether the field exists or not, the query should be able to use the index
[14:29:38] <rspijker> tuxtoti: I hadn;t actually looked at your output… the query returns all fields
[14:30:31] <rspijker> well, I don;t know if it hits everything, that’s an assumption I made due to the 12M that was referenced before
[15:13:26] <mp_> i'm trying to summarize a table that tracks user activity grouped by week + user. anyone have a moment to help with adding in blank data for weeks where a user did not have activity?
[15:22:11] <mp_> sflint : i'm displaying graphing counts by week and would like to prepare the data correctly at the source
[15:22:33] <mp_> i guess i could hack it in there in my UI/etc, but would be more correct to do it at the source
[15:25:35] <sflint> but if you want you can just do db.collection.update({<key>:null},{$set : {<key>:0}})
[15:26:10] <Siyfion> So the docs say that "$pull: { results: { score: 8 , item: "B" } }" will pull all elements that contain a score field of 8 AND an item field of "B"... How would I do it if I wanted to pull all elements that have a score field of 8 OR a score field of 4...?
[15:27:49] <mp_> week 3 and week 6 aren't in the DB, but i want to have records in my results with {week3, count: 0}, more or less
[15:32:24] <mp_> sflint : that's the basic setup with some data/etc redacted, but should get the point across
[15:38:03] <mp_> i.e. if there is no date, i can't group by the event that doesn't exist in the first place
[15:38:19] <mp_> so i need to do some kind of preliminary grouping by week to indicate the last 12 weeks
[15:38:59] <mp_> whereas right now, i start at event and filter down the date, i need to reverse it, i think
[15:40:31] <sflint> mp_: yeah i don't us aggregation really at all. I do that in my schema design and the just $inc the indicator
[15:40:56] <rspijker> mp_: that only works if you can guarantee there is an event in every week though
[17:10:21] <dorongutman> a document that is “deleted” has a field named “deletedAt” with a date, and non-deleted documents don’t have it
[17:27:09] <DubLo7> Like get all users from user collection that are less than 1 year old, and all orders from order collection that are made by those users?
[18:46:43] <scorphus> I’m trying to create a unique compund index with db.collection.ensureIndex( { a: 1, b: 1 }, { unique: true } )
[18:48:15] <scorphus> right after any insertion? just after the first one? or is there some way to create or setup an index?
[18:49:52] <kali> scorphus: of course, it can fail if you do it when some documents are already there if there is a dup
[18:51:05] <scorphus> kali: is it ok to call that everytime I insert? I don’t have any “setup” or “initialize” feature
[18:53:07] <kali> scorphus: well, you'll pay a network roundtrip for nothing, and you may need to catch or ignore an error, but apart from that, it's fine
[19:39:43] <davejfranco> Does anyone knows how to connect mms agent if a database is on a private subnet? and your aplication server is behind a proxy
[19:42:05] <kali> i mean: the agent will connect the MMS platform, not the opposite. basically... it should work out of the box
[19:42:07] <davejfranco> I'm hosting on amazon ec2, and the databse server can connect via a nat instance but in this case I think it needs a reverse nat
[19:46:19] <kali> the connection will go from the agent in your private subnet, through your router, to the MMS platform. just NAT.
[19:47:09] <kesroesweyth> I have a collection of documents that all contain a property which contains an array. Is there a simple way to get a single list of distinct keys from all of these arrays?
[19:47:31] <kesroesweyth> Meaning, just one list no matter how many documents the collection contains.
[19:48:51] <kesroesweyth> kali: I guess I assumed it wouldn't just magically work since I didn't see a hint about it in the documentation. Let me try.
[19:54:44] <davejfranco> I tryed on a mongo in a public subnet and it works perfectly but in a production environment I want that DB on a private subnet
[19:56:33] <kali> the agent talks to your database (so it needs to be on the private subnet) and talks though the NAT to the MMS cloud
[20:03:34] <Viesti> just did that but apparently the oplog isn't big enough on the primary so it got rolled over
[20:06:50] <kali> Viesti: look at the replica set on the primary, it will state how much time the oplog covers. if this time is not bigger than the time you need for a full resync, it will likely fail again
[20:08:16] <Viesti> yep, just increased oplog size on the secondary, but I guess the problem is the too small size on primary
[20:11:07] <Viesti> just wondering that I have data now in the secondary, but I guess it's then useless because it's stale...
[20:11:54] <kali> yeah, if you dont have all the oplog from that secondary optime to now, the data is useless
[20:27:23] <Max-P> Hi, does any one know which is better/faster between multiple sparse indexes or just one big shared index?
[20:28:22] <Max-P> I have a big collection that holds types of objects, some of which have external references to other collections, but not always on the same field. Trying to see if I should index them all or have just one big "externalReference" field and index that.
[20:28:29] <kali> Max-P: it depends on the query. if you only have one kind of query, it's better to have an index taylored for it
[20:30:55] <Max-P> kali, Basically, I can have something like {type: "something", somethingId: ObjectId('...')} and {type: "otherthing", otherthingId: ObjectId('...')}, and I will always query with the `type` field set
[20:32:16] <Max-P> It's really either one or the other, so I'm wondering if it costs more to maintain two indexes and if I should just use a common field "otherId" for every entry
[20:33:21] <kali> i would maintain a unified index, i guess. but if it's only two index, it's still manageable.
[20:37:58] <Max-P> Actually that's going to be more like 5-6 of them, but all of them will be sparse as only their appropriate type will have the matching indexed field
[20:39:13] <Max-P> So to simplify the question, does Mongo even bother updating a sparse index when the corresponding field does not exist?
[20:55:57] <Max-P> Results are it's about 10% faster to use one big index on average, so I'll go with the unified index.
[21:00:34] <Max-P> Yeah the more indexes you add the more time it takes even if they are unused. 10% hit with 10 indexes, 20% hit with 20 indexes
[21:16:32] <ranman> Max-P: I have some questions about that, do you have a simplified test case for that?
[21:18:50] <ranman> Max-P: if I read and understand your usecase correctly you could make use of compound indexes and prefixes: http://docs.mongodb.org/manual/core/index-compound/#compound-index-prefix
[21:28:04] <Max-P> ranman, No, I'm not using any compound index. None of the index have any relation between them
[21:30:50] <Max-P> Pretty ugly, but did the job for what I needed to know. At any point only one of the said fields will be present at a time.