Log file Viewer
#mongodb logs for Monday the 15th of February, 2016
[00:00:26] <kurushiyama> Freman: if it is for development... ...well, sure. Why not? If it helps, you can have a second person type in the query after reviewing it. Will it help with existing indices which lack a field for sorting, for example?
[00:02:42] <Freman> tbh I'm making it up as I go along, they break it I find out how and fix it. it really just checks to see if the first field in the query is a first field in any index. doesn't stop them from putting in 1970-01-01 as a date but it is stoping me from getting woken up at 2 am because they crashed/cancelled out on a bung query on the wrong collection (one that doesn't even have the field their looking for)
[00:04:29] <kurushiyama> Freman: provide them with dev instances. And have profiling enabled with 75ms slow queries. Have them read them manually using the shell. ;)
[00:05:19] <Freman> wish I could, but they're querying production log data (which is a stupid amount of logging - why log the error for a request and a reference when when we can log the entire 8k request and html error page)
[00:06:47] <Freman> yeh and their bung queries stall mongo and cause my logging engine (which I had to write because they swamped whatever was there before) to backlog (it was designed to do that)
[00:07:16] <Freman> I'm trying to negotiate the moving of audit type logs to a dedicated collection and everything else into a capped collection with disk copy (and logrotate)
[00:09:00] <Freman> all of this is to facilitate a real time log ui (a webpage with a socket that I could also just as easily pipe to from the re-written logging engine)
[00:10:23] <Freman> so yes, rsyslog + logrotate + capped collection should do everything they want from the real time logging view and audit collection for things that probably shouldn't just be dumped into the same place as "your request failed, here's 8k of rubbish no-one wille ver read"
[00:14:47] <Freman> From our logging wiki page (which I wrote when I had to re-write the logging engine to deal with the load) "We log everything, sometimes even logging that we've logged logs, and because of the quantity of logs, sometimes we even email that we've logged it."
[00:15:24] <kurushiyama> Freman: if you send every incident, maybe the devs notice that they overdo it... ;)
[00:15:28] <Freman> Yeh apparently, I'm stuck with more the making things build getting agents working (holy cow they're expensive - I come from jenkins where they're all 'free')
[00:17:52] <Freman> I don't know about integration I'm still doing all the same things I was doing under jenkins to get the builds working... but instead of having containers with an agent for each type of job that can be spun up and down as load dictates I now have containers per agent... that I've yet to figure out. but that's a bit off topic for #mongodb
[00:49:18] <jiffe> well this secondary thinks its last optime is 56c0be5f:5 and minvalid is 56c0be65:15, can I somehow change last optime to 56c0be65:15 on the secondary?
[00:51:46] <jiffe> if it saves me a week of resyncing then its worth it to just skip a few ops to me
[00:54:52] <jiffe> the actual message is replSet need to rollback, but in inconsistent state; minvalid: 56c0be65:15 our last optime: 56c0be5f:5
[00:58:32] <jiffe> well right now 2 of the 3 nodes were running when this killed the other secondary
[04:16:38] <DocMerlin> I have a rather large (2TB) sharded collection and I need to find out what the shard key is to speed up some queries.
[04:30:40] <Freman> don't suppose anyone knows how mongo does their videos, that thing with the hand writing on a whiteboard, that's actually pretty cool
[07:49:37] <Zelest> Is it possible to use $addToSet and only check if some of the element exists? E.g, if I have an array with objects.. [{a: 1, b: 'foo'}, {a: 2, b: 'bar'}] .. I wish to update and add a object with a: 1, but only if it doesn't exists and it shouldn't add another one even if b: is something different than 'foo' .. Is this possible
[08:46:35] <sabrehagen> how do i restructure this logic using a query to reduce my number of database accesses? i have a query that returns documents in a certain time range. each document has a device id. for each of those documents i want to see if there is a document matching
[08:46:44] <sabrehagen> that device id in the 10 seconds prior. the initial number of documents can be 40,000+. my logic then sends 40,000 database queries to check the second piece of criteria.
[08:46:49] <sabrehagen> without the aggregation framework, how would i minimise the number of queries?
[09:32:08] <kurushiyama> sabrehagen: Sounds like there is something fundamentally wrong with your data model.
[09:36:18] <Lachezar> Hey all. What's the "MongoDB Way" of doing searches of documents by properties in related documents? Let's say an db.Order(id) has reference to db.Customer(id, name) and to a db.Employee(id, name). How should I do the model? I want to be able to search orders by text (quick-search) using the name of the customer OR employee?
[09:39:17] <kurushiyama> Lachezar: IN general, that is a relation with attributes: The Order relates the Customer to the Employee, but is itself qualified by attributes.
[09:40:15] <kurushiyama> Lachezar: I would have an additional field in the order with the customers and employees name,
[09:42:41] <Lachezar> kurushiyama: Hm. Well... Yes. That might work with the simple pointer relations. But I understand where this i sgoing.
[09:44:04] <kurushiyama> Lachezar: Data modelling with mongodb involves redundancy. The earlier you get used to it, the better you models will become.
[09:46:30] <Lachezar> kurushiyama: redundancy… That word is badly used here. Redundancy (in IT) is typically meant to provide a measure of stability, failure tolerance and coping with critical events while possibly reducing performance. What you describe seems uite the opposite.
[09:47:46] <Lachezar> kurushiyama: Thank you for the time. I'm going to say that searches in MongoDB come at a price.
[09:48:13] <kurushiyama> Lachezar: I am in the industry for like 20 and a bit and have done huge projects with MongoDB. MongoDB is not just the cooler MySQL. You _really_ need to rethink the way you model data.
[09:48:36] <kurushiyama> Lachezar: http://blog.mahlberg.io/blog/2015/11/05/data-modelling-for-mongodb/
[09:52:09] <kurushiyama> Lachezar: To put it a bit simplified: You model data according to your questions, not according to the entities.
[10:19:42] <kurushiyama> sabrehagen: Well. It is a side project (sadly), and I first have to feed the family. If you have things that especially interest you, please add it in the comments.
[14:28:54] <kurushiyama> Ben_1: bulkWrite is more flexible, however, since you can mix inserts, updates and alike.
[14:30:01] <kurushiyama> Ben_1: Yes. Be aware though that you may cause duplicate key errors. I tend to use bulkWrite with upserts.
[14:43:25] <kurushiyama> Zelest: Yup. Got warned today for telling someone (in much nicer words) that he can *piep* himself and do his *piep* work himself. Wanted a complete business logic for a booking website.
[14:47:18] <Zelest> e.g, if I have a document {_id: ObjectID(...), a: "foo", b: [{foo: 'a', bar: 'b'}, {foo: 'b', bar: 'a'}]}
[14:48:51] <kurushiyama> Zelest: first, you can use $elemMatch in projection to get the macthing array member
[14:51:52] <Zelest> I basically want to use a $addToSet, but have it check only one sub-element.. if that makes sense
[14:52:18] <kurushiyama> Zelest: using $elemMatch is cheaper, iirc. Well, in that use case, I'd fetch the whole array and push it. But I think it is rather a modelling problem.
[14:52:45] <kurushiyama> If arrays need to be updated fairly often, from my experience, you might be better of with a collection.
[14:54:22] <Zelest> it's tricky though, seeing i need the parent-document/collection updated as well then
[14:56:49] <kurushiyama> If you have a site collection and a pages collection, you can simply insert a new doc into pages.
[14:58:28] <kurushiyama> if db.pages.find({site:"foo.com"}).sort({date:-1}).limit(1) newerThan(-10s) { skip }
[14:59:08] <kurushiyama> if db.pages.find({site:"foo.com"},{date:1}).sort({date:-1}).limit(1) newerThan(-10s) { skip }
[14:59:27] <kurushiyama> (limiting the bytes to return, and with an index, you should have a covered query)
[15:01:03] <kurushiyama> Zelest: http://blog.mahlberg.io/blog/2015/11/05/data-modelling-for-mongodb/
[16:00:54] <kurushiyama> Ben_1: Aside of having concurrent bulk operations implemented by your application: nope.
[16:01:41] <Ben_1> kurushiyama: thx, I'm using the async binding so I think it's enough to start multiple bulk inserts one after another
[16:30:05] <Ange7> is it possible to modify default timeout for aggregate() in mongo configuration file ? cause timeout() function in php driver doesn't work for me ?
[16:31:43] <Derick> server side? client side? cursor? What's your (simplified) code? What's the full error you get with the timeouts? And what does "timeout() doesn't work" mean?
[17:49:53] <choro> how do i allow for remote connections? i want to use robomongo on my personal machine to access the mongo database on my server
[18:33:21] <SAiF> can anyone give me a working link for mongochef 3.4.0 windows 64 for non commercial use..
[18:33:52] <SAiF> I cant download it from official site. It just wont download. I am not sure if its my end causing problems
[18:43:22] <Ben_1> MongoCollection<Document> collection = database.getCollection(collectionKey); <<< in java this line should be enough to create a collection, but robomongo says there is no collection. could something go wrong by creating a collection? and what happens if I create a collection but do not insert something?
[18:51:18] <Ben_1> ok it seems my singleResultCallBack is not called so I think something wents wrong by using insertMany
[19:20:14] <cheeser> not in the db anyway. you'll get a MongoCollection reference but that doesn't mean it's backed by anything yet.
[19:22:23] <Ben_1> and that's my problem, the line after that getCollection I have a insertMany and that instruction won't work
[19:25:32] <Ben_1> cheeser: well, the callback is not called, so internally something goes wrong I think
[19:32:52] <Ben_1> but anyway I'm using Robomongo GUI to look inside my database and it shows the database but no collections.
[19:33:47] <cheeser> strange. i haven't used the async driver so i'm not sure what else might be off.
[19:36:54] <Ben_1> cheeser: mh it seems in the log file no command is executed. Creating the database programmatically will work but the insertMany not
[20:47:21] <kurushiyama> cheeser: ah, ok. Still do not see the need. Whenever I see numbers in collection names, I actually read "modelling problems"
[20:49:52] <cheeser> jurassic park. "they spent so much time to wondering if they *could* do something they never stopped to wonder if they *should*"
[20:53:27] <kurushiyama> cheeser: That pretty much subsumizes every SO question about MongoDB I have seen in the last half year (which wasnt a "do my (home)work" or "I dont want to read the docs, I rather ask here").
[21:11:24] <cheeser> if a use likes an article, add her ID to the list of likes on the article document
[21:13:42] <cheeser> yeah, there are few shortcomings but it's the simplest solution that'll work for most sites.
[21:14:42] <kurushiyama> Since the questions usually are simple: How many likes does a _given_ post have: db.likes.find({'postid': someiD}).count()
[21:15:46] <cheeser> kurushiyama: that'd be ok-ish if you also tracked like count on the article itself.
[21:17:24] <cheeser> but having worked on a CMS at scale, i'm aware of the various limitations in that approach. they're just not likely to apply to someone asking in iRC at this point ;)
[21:18:30] <kurushiyama> cheeser: Well, and now the app becomes the next best thing and the OP sends his gorillas to kick you in the but for suggesting a data model which does not scale well ;)
[21:19:23] <cheeser> kurushiyama: when that happens, i'll happily take the heat. but it's not gonna. :)
[21:19:40] <cheeser> and for someone trying to learn mongodb, those are complications they needn't worry about just yet.
[21:21:31] <kurushiyama> cheeser: I see this quite different. The sooner people start to get used to redundancy and to model their data according to their questions as opposed to modelling according to their entities, the better for all of us.
[21:27:19] <cheeser> well, part of my job is to help people get started. in this case, it feels more like an exploratory "what is mongo?" project. had it been, "we're migrating our customer service systems to mongo" it'd warrant a different response.
[21:27:50] <cheeser> when someone asks me how to get started building a website, i don't start by discussing scaling issues faced by facebook and twitter.
[21:45:29] <cheeser> ah. yeah. well your suggestion and how i'd actually do it only differ in that i'd keep a count on the article documents, too.
[21:47:11] <kurushiyama> cheeser: I'd rather use an aggregation for it than to store it manually. But granted, that may be just me. And granted, our solutions differ more on a philosophical level than a technical level.
[21:51:10] <kurushiyama> cheeser: An aggregation can have a match phase, and doing an aggregation is more reliable than doing a .count(), but still less harder to keep in check than updating a post document every time a like is added. There is an interesting problem related to doing counts in a sharded env: https://jira.mongodb.org/browse/SERVER-3645
[21:56:24] <kurushiyama> Well, probably I'd do an $inc on a counter, if the application really takes off. With document level locking, it does not hurt so much any more.
[22:44:03] <synapse> I am trying to build a function in nodejs like this searchUsers(query) which passes query to this db.collection.find(query) but it's proving difficult. escaping characters seems like a bad thing to do to get a literal query so what's "the right way"