Log file Viewer
#mongodb logs for Thursday the 27th of August, 2015
[00:24:21] <jmo> Hey there. I am confused about the proper use case for mongo. If I have a data model that seems to fit just fine in a SQL database, is it a good idea to think about using mongo?
[00:26:18] <Boomtime> jmo: are you prepared to learn something new or would you rather only ever use what you are familiar with?
[00:27:17] <Boomtime> MongoDB is a pretty general purpose database - but like _all_ databases, unless you know how to use it, you're probably not going to have a good experience
[00:28:39] <jmo> Boomtime: very willing to learn something new. the flexible schema part for development is mostly what draws me. i'm worried that it's meant more for unstructured / inconsistently structured data
[00:31:29] <Boomtime> just because mongodb is schema-less, does not mean you get to ignore the schema
[00:32:12] <Boomtime> mongodb does not enforce a schema on you, but _you_ better think about your schema or you won't be able to manipulate your data
[00:32:39] <jmo> i know this channel will tend to be biased, but is there a case when you definitely *shouldn't* use mongo?
[00:33:30] <Boomtime> when you don't know what you're doing - otherwise nearly anything can work - just like an sql
[04:48:00] <alaee_> Hi. Can anyone help me with mongodb optimization please? I have this very simple 100 documents long collection and when I'm using it in a Flask app with uwsgi and nginx, the bottleneck is my mongodb and I can't go beyond 200 reqs/sec.
[06:37:42] <alaee_> Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passa
[06:37:42] <alaee_> ges, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum. 1394/6/5
[06:38:40] <macwinner> and haven't used nginx really.. so i'm not sure if they support some sort of connection pooling
[06:53:28] <alaee> <alaee_> Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem I
[06:53:28] <alaee> <alaee_> ges, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum. 1394/6/5
[06:53:34] <alaee> <alaee_> I send requests using ab -n 10000 -c 100 and I get the results in 50 seconds. Meaning 200 reqs/sec
[07:30:54] <alaee> Hi. Can anyone help me with mongodb optimization please? I have this very simple 100 documents long collection and when I'm using it in a Flask app with uwsgi and nginx, the bottleneck is my mongodb and I can't go beyond 200 reqs/sec.
[07:54:47] <mtree> its better to do couple of aggregate qureries, or make one search and reduce results to the required format?
[07:56:29] <kali> mtree: i try to deport as much work as reasonable to the application server, as this part is easier to scale than a database
[07:56:52] <kali> mtree: not sure if that answer your question, but without cardinalities anyway, there's not much that can be said
[07:58:05] <mtree> kali: i wrote code that gets LOTS of records from the collection and I filter/reduce these results in code to get what I need (sum of some conditional values)
[07:59:47] <kali> well, you at least ought to try what he's suggesting and see how it behaves compared to your solution
[08:00:19] <kali> but we're slowing moving to a non-technical field here, not sure i'm the best person to give advice :)
[09:38:33] <sedavand> what users and roles do I need to create inorder to use local user/simple password auth?
[09:56:50] <Derick> mtree: it's probably easier if you show me your input documents, and describe (with example) what you'd like as output (in a pastebin)
[10:17:34] <mtree> and it works fine, but it requires me to execute another query to group by hotelNames
[10:17:54] <mtree> its sad but still better than getting whole filtered collection and reducing it in code
[10:24:05] <charnel> How can I optimize a query which is looking for ip range. I am using $gt and $le which is slow. But I converted the Ip addresses to integer which gained me some time. Basically now I am only looking if the given integer is between min and max values. Can I improve performance more ?
[12:35:02] <ssutch> we are trying to add a member to our replicaset but it stays in startup2 state forever and continually crashes with this message: 2015-08-27T12:31:10.370+0000 E REPL [rsBackgroundSync] sync producer problem: 16235 going to start syncing, but buffer is not empty
[12:35:45] <ssutch> it's a 2-shard cluster with 3 configs and we are trying to startup a node on rs0
[12:49:01] <ssutch> the other that commonly occurs directly after a replicaset failure is [rsSync] 10276 DBClientBase::findN: transport error (PRIMARYNAME)
[12:49:56] <ssutch> if someone AT mongodb were available for immediate support im sure my boss would be happy to pay for someone to help us out. today is our launch day hah
[12:51:36] <deathanchor> ssutch: can you gist/pastebin the logs with the errors? (you can sanitize if you want)
[12:56:10] <ssutch> the only potentially weird thing is we are using an ELB to route traffic (so we can set up a cname in route53). it's worked for us for weeks though
[13:12:31] <mbwe> hi everybody i have a collection devices and it has a name field, how could i do an insert only when the name field does not exist already
[13:13:37] <mbwe> thus {name: "#010101"} and if there is already a record with name: "#010101" i want to cancel that insert
[14:07:17] <Grixa> I have a problem with slowly finding data in collection (~2500000 records). My script executes search query about 500000 times and it takes about 10 minutes. Collection contains records with windows paths (for example: C:\Users\admin\Documents\1.pdf).
[14:08:24] <Grixa> "cursor" : "BtreeCursor path_1","isMultiKey" : false,"n" : 1,"nscannedObjects" : 1,"nscanned" : 1,"nscannedObjectsAllPlans" : 1,"nscannedAllPlans" : 1,"scanAndOrder" : false,"indexOnly" : false,"nYields" : 0,"nChunkSkips" : 0,"millis" : 45
[14:11:05] <Grixa> It is too slow, while i am querying search for 500000 times in my python script. My idea is to store some hash (CRC32 for example) and use it for indexed search. Or it will be meaningless?
[14:12:53] <deathanchor> indexOnly" : false <- if you want faster try to make this true by making a compound index of what you want and what you are looking for
[14:14:44] <deathanchor> indexOnly" : false because you want the doc which includes the _id which isn't in that index
[14:17:28] <Grixa> I used this query: db.aps2_paths.find({"path": "..." }, {"path": 1, "_id": 0}).explain()
[14:18:40] <deathanchor> yeah gotta remember that mongo will return the full doc by def which requires a read from disk, but if the index is in memory and all you need is in the index use projection to avoid disk reads
[14:22:35] <deathanchor> Grixa: just keep your index size in mind when spec'ing out your machine memory, if it gets too large then you'll need more memory to avoid paging.
[14:30:08] <Grixa> Thanks! Aaand another stupid question: if i want only recieve _id from my collection, i use this query: db.ups_paths.find({"path" : "..." }, {"path": 0}). And this query again not used index ( "indexOnly" : false )
[14:33:13] <deathanchor> but you need to decide if you want another index built and taking up sweet precious memorry
[14:42:53] <Grixa> db.ups_paths.find({"path" : "..." }, {"path": 0}).hint({"path": 1, "_id": 1}).explain() -> "cursor" : "BtreeCursor path_1__id_1","isMultiKey" : false,"nscannedObjects" : 1,"nscanned" : 1,"indexOnly" : false,"millis" : 0
[14:50:20] <jpbjpb> I’m trying to add a key:value pair where the value is a pointer to another collection. How can I do this?
[14:51:43] <saml> http://docs.mongodb.org/master/reference/database-references/#dbrefs this is convention
[14:52:40] <saml> db.docs.find(yolo).forEach(function(x) { db.someOtherCollection.find({_id: x.idOfSomeOtherCollection}) ....
[14:52:46] <jpbjpb> I used mongoimport to pull in some JSON, but I could’nt figure out how to import the references to another collection, so I’m trying to update by hand
[14:53:45] <saml> maybe give us example json you're importing. probably better to use mongodump and restore
[14:55:21] <jpbjpb> I’m trying this: `db.plans_copy.update({"slug":"200w41st"}, {$set: {"_building":ObjectId("55dcd6aa88a093e0032e8a97"),}, false, true}`
[14:55:42] <cheeser> for the most part it works like a champ. just some corner cases where people want to reuse instances that weren't designed for it.
[15:17:56] <Grixa> Why db.ups_paths.find({"path" : "..."}, { "path": 0 }).explain() return "indexOnly" : false, but it uses "cursor" : "BtreeCursor path_1"?
[15:19:46] <cheeser> because it's using the index path_1 to resolve the query but you're suppressing the path field so it has to use more than the index to return documents
[15:22:03] <cheeser> right. so it has to load the full document from disk in order to get those _id values.
[15:31:02] <Grixa> I created new index (path: 1, _id: 1) and, again: "cursor" : "BtreeCursor path_1__id_1", "indexOnly" : false
[15:43:52] <christo_m> im trying to figure out how to create shows without overwriting the season and episode properties.. i want it to append
[16:06:25] <christo_m> StephenLynx: i started the project with this: https://github.com/DaftMonk/generator-angular-fullstack
[16:06:44] <christo_m> i dont want to start deviating from the norm unless im actually doing it for everything.
[16:08:06] <christo_m> StephenLynx: anyway, you see what im trying to do, i just dont know how to shape my queries to do it.
[16:08:52] <christo_m> it doesnt matter where i go on IRC, im using the wrong framework or tool or technique
[16:09:21] <cheeser> yeah. enough with the "mongoose is shit" screeds. it doesn't help answer questions.
[16:09:44] <christo_m> StephenLynx: im sorry i didnt do things your way the first time i started with node/mongo
[16:34:27] <deathanchor> when I have nothing good to say, I don't say anything at all. Keeps me out of trouble.
[16:47:06] <deathanchor> please critic when I ask, "What do you think about?" but unwanted criticism is just that... unwanted.
[16:52:46] <StephenLynx> because when you just cover your ears and scream "LALALALA I CAN'T HEAR YOU", it signs you are on your way to completely screw up
[16:55:20] <deathanchor> StephenLynx: so I'm thinking of using a framework for my app to work with mongo. My main codebase is python. What do you suggest?
[16:58:20] <StephenLynx> GothAlice is experienced with mongoengine and might be able to tell you the benefits of using.
[16:58:48] <GothAlice> deathanchor: pymongo is the official MongoDB driver. Pretty much everything else gets built on top of it.
[16:59:17] <GothAlice> At the same time, I use the MongoEngine ODM, as it provides many features the raw driver does not.
[17:00:04] <GothAlice> I.e. it provides a proper "model" layer to your app, with schema, data validation and transformation, event callbacks / triggers, and can even handle pseudo-relational stuff for you, including reverse delete rules.
[17:00:24] <deathanchor> yeah, I prefer frameworks that specialize in one thing. I was using django for playing around, but it is too over reaching with somethings.
[17:00:59] <StephenLynx> imo, these ODMs provide stuff that you shouldn't use in the first place, according to mongo's design.
[17:01:21] <deathanchor> yeah that's why I'm just going to use mongoengine and find a web framework that just handles requests and another to handle serving content.
[17:02:01] <GothAlice> deathanchor: Mild bit of self-promotion, here, but my web framework (WebCore) is about as light-weight as you can get. (The web.core module is < 300 lines, and everything is optional.)
[17:02:51] <GothAlice> pymongo isn't asynchronous, thus you'll need a different base driver to use Tornado properly.
[17:03:44] <GothAlice> deathanchor: https://github.com/marrow/WebCore/blob/rewrite/example/monster.py is the "one example to rule them all" for WebCore 2.
[18:14:24] <christo_m> its literally a red herring in the problem i was presenting.. i was trying to show how i wanted my data structured
[18:15:00] <christo_m> its like "yo my timing belt is broken on my car" and you go "well why did you buy a Ford"
[18:17:14] <christo_m> GothAlice: im trying to setup a model like this: http://pastebin.com/QMJhq3DH
[18:17:30] <christo_m> how would i go about inserting seasons and episodes based on the title of the show?
[18:21:01] <deathanchor> christo_m: not sure what you mean by inserting seasons/eps based on title of show.
[18:21:33] <christo_m> deathanchor: i mean i look up a show by its title, and insert a season and episode...
[18:21:54] <christo_m> the season will have a name , maybe Season 3, maybe just 3, and the episode will have a title
[18:22:57] <deathanchor> can we clarify first title is "showtitle" and second title is "Episode title" ?
[18:24:16] <christo_m> i want to post data i find.. and have my insert or upsert or whatever on the server side push the information appropriately
[18:24:43] <christo_m> im sorry, i dont know what to use.. i pasted pastie last time and people didnt like that
[18:26:32] <christo_m> im scraping some links with casperjs and trying to hit the api to insert them as i find them.
[18:28:38] <deathanchor> my best guess without trying: https://gist.github.com/deathanchor/441f0dc2e80254a46fbb
[18:32:11] <christo_m> deathanchor: like potentially ill be inserting a new sesaon, or appending an episode to an existing season by season name
[18:32:55] <deathanchor> yeah well this is where a model helps and should know how to handle this, I haven't done much updates in to nested docs like that
[18:34:17] <christo_m> are you saying i should define that in my schema as a nested collection instead of doing it how im doing it?
[18:45:19] <Owner> well im not sure where ill find the logs from last night,...its logging so much that i would be digging for a while
[18:58:18] <MacWinne_> does compacting your mongo collection only make sense in situations where you are adding and removing documents? if our collection is basically only adding activity documents, would there be any reason to do a compact?
[19:47:32] <charnel> can I create a compound index from 2 fields in an existing collection with more than 20K documents. Both fields are integer
[19:51:40] <christo_m> saml: basically im trying to upsert show episodes as they come in from the post
[19:51:56] <christo_m> but they come with a season too, which may exist in the database, in which case i want to just append to that season.
[20:04:49] <saml> give me example document. your Schema says seasons is an array but your update assumes seasons is object
[20:10:14] <christo_m> im getting episodes one by one that may belong to a season that exists or not
[20:10:27] <christo_m> if it does exist i need to update the existing entry (the title of the seasons will match)
[20:11:49] <saml> var show = db.tvshows.find({_id: TVSHOWID}); var updatedShow = insertNewEpisode(httpRequest, show); db.shows.update({_id: TVSHOWID}, updatedShow);
[20:13:27] <christo_m> in my head a show is a collection of seasons, and seasons are a collection of episodes.
[20:14:11] <saml> insert into episodes(showId, seasonId, epNumber, epTitle) VALUES ('Seinfeld', 2, 14, 'Somthing something bad episode');
[20:15:33] <christo_m> the reason i want to do it this way is so that i can build the accordians on the front end very easily..
[20:17:09] <saml> you could get (season,episode) pair as follows: (1,1), (1,2), (3,1), (2,1), (3,2), (2,2) ...
[20:19:51] <saml> i mean, i'm suggseting you do two db operations. first to GET show data. manipulate seasons array. then SET it back
[20:23:05] <saml> in normalized world, you can keep appending to file, or create new files.. and you can join things to construct final data during read. but mongodb doesn't have such capability
[20:23:29] <saml> instead, mongo is heavily read optimized. so during write, you do some calculation
[20:30:20] <gcfhvjbkn> now i sharded my data so that each one out of 5 servers has a portion that it has to write
[20:31:14] <cheeser> well, you'll still write to one host (a mongos) which will then fan out writes according to any sharding you have set up.
[20:32:05] <gcfhvjbkn> hmm, that's a pity; if i recall correctly write performance was one of the three reasons mongo docs have for using sharding
[23:54:49] <Doyle> cheeser, the instances will still remain resolvable by the names entered in the config
[23:55:47] <Doyle> a dhcp optinoset is incorrect. It's appending a misc domain on the end of the private DNS names so the instances report their hostnames as ip-x-x-x-x.ec2.internalmiscdomain.com