Log file Viewer
#mongodb logs for Thursday the 18th of August, 2016
[06:11:54] <therealssj> @cheeser i defined the index for the collection as self.db.create_index('expire', expireAfterSeconds=3600), self.db has the collection I am storing the documents in
[06:18:08] <joannac> can you pastebin 1. list of indexes for that collection 2. a document you think should have expired, but didn't
[06:18:47] <joannac> preferably output from a mongo shell, but output from pymongo is fine too at a pinch
[06:25:00] <cheeser> in the shell do a findOne() on a document that should be gone and pastebin that.
[06:37:28] <therealssj> joannac: how would I set a expire for a specific docuement for a specific time let say 5 minutes
[06:37:56] <joannac> so if you want it to expire in 5 minutes, set the date to be 55 mins in the past
[06:38:06] <joannac> so if you want it to expire in 6 minutes, set the date to be 54 mins in the past
[06:39:16] <joannac> https://docs.mongodb.com/manual/tutorial/expire-data/#expire-documents-at-a-specific-clock-time
[08:12:11] <Kamihan> Hello. I am running npm parse server, nodejs, and mongodb on my local machine. I have an app that uses Parse to register uses to the Mongodb, but when I do "show dbs" in mongo shell, all of them are empty. Where is Parse saving users in mongodb?
[09:59:56] <wim_PA> Not sure if this is the right place to ask but I'm experiencing some weird behavior benchmarking the PHP mongo and mongodb extension. Running the following code with the mongo extension http://pastebin.com/rxZsRz9J gives me an instant duplicate key exception with the same ObjectId(). However, if I place the $result assignment inside the for loop it works fine. Is this intended behavior or a bug?
[10:03:16] <GothAlice> wim_PA: Not a bug, your example code is bad. When inserting a record, the driver assigns an ObjectId to the _id key of your assoc. array (if an _id key is not already present). Thus, the first iteration will run fine, add the _id, then each subsequent iteration will attempt to re-insert the record with the same _id, giving you the duplicate key error.
[10:04:01] <Derick> wim_PA: that behaviour is no longer there in the new mongodb driver though - you should think about upgrading.
[10:06:38] <GothAlice> Always best to be current with driver updates; sometimes the communications protocol changes (for example, adding an updated authentication mechanism), so if your host updates mongod and you don't have an up-to-date driver, it can cause problems.
[10:07:08] <GothAlice> (Recently had a bit of a panic about pretty much just that at work while I was on vacation. ;)
[10:14:25] <wim_PA> Absolutely agree, the example code was just meant to benchmark the old and the new extension. Because if I want to use the new extension I have to do some refactoring of my codebase.
[10:17:21] <wim_PA> Preliminary results show that the new extension is actually a bit faster which I didn't anticipate considering a lot of the 'query code' has been moved to userland. I'm probably missing some 'first' call overhead or something. Not sure yet
[10:20:32] <GothAlice> wim_PA: Also notably, for many inserts a better pattern is to use bulk inserts. Tight loops of bare inserts aren't overly efficient. Ref: http://mongodb.github.io/mongo-php-library/api/class-MongoDB.Operation.BulkWrite.html
[10:21:29] <GothAlice> (Tight loops of bare inserts will introduce more variability from network effects, for example, as each insert is independently confirmed with a full network roundtrip.)
[10:25:56] <Derick> wim_PA: our benchmarks showed it too - are you running PHP with opcache enabled?
[10:34:14] <wim_PA> The insert appears to be a lot slower with the new driver (insertOne() vs insert())
[10:35:58] <wim_PA> Derick: Are there any benchmarks online to verify my findings? Couldn't find them on Google which is why I decided to run my own benchmarks and get a better understanding on the performance impact
[13:06:17] <wim_PA> Derick: I expected the 'first-time' overhead from the new PHP MongoDB extension to come from the userland library but that doesn't seem to the case. The first time MongoDB\Driver\Server->executeQuery() is called It takes 20-28ms for a simple findOne() query on a very small dataset. Any query after appears to be blazing fast. Where does the overhead come from as the old extension has much less.
[13:08:35] <GothAlice> I.e. the old driver would eagerly connect its pool, the new driver likely waits for the first actual use.
[13:08:52] <GothAlice> Do you get the same delay on first insert if you query (even a non-existent collection) first?
[13:11:01] <GothAlice> Right; insert a long delay between setting up the connection and your first query; check netstat or similar to see if there is a real connection, then have a delay after the first query, check netstat again; that'll see explicitly if lazy connecting is the cause of that delay.
[13:28:16] <wim_PA> Better this way indeed. However I can't seem to understand how this can cause a 66% req/s drop .. 450 ->150. More testing required! Thanks :)
[13:34:41] <wim_PA> But it seems that the old extension does not suffer from such a delay making the connection
[13:51:55] <wim_PA> Derick: Yeah, that's what I had to double check, if I included the instantiation of MongoClient within the benchmark and it seems I did. I made a smaller benchmark only for the connection part: http://pastebin.com/m5X02v9P Not sure if that's correct but it shows from connection to first query the old extension seems to be twice as fast. hmm
[15:24:07] <Jimu> Any performance experts here? My database has 49 million documents, and simple increment ($inc) commands are taking 30-400 seconds
[17:51:11] <GothAlice> Jimu: Step one is usually: get an explain() of the query and make sure it's using an index. No index = crawls your records = you're gonna have a bad time.
[19:01:29] <deathanchor> don't forget to check it's using the right index, or else you're gonna have a bad time.
[19:11:48] <idd2d_> I have a collection with about 200 million records. Each doc has about 40 fields. I need to be able to query on any number & any combination of these fields. So far, this has been a performance issue because I can't index every single combination of fields. I'm not a mongo-pro, wondering if there's a good solution for this.
[19:12:42] <idd2d_> That's my take as well. I was hoping there was some feature I'm not familiar with.
[19:13:16] <idd2d_> right. That makes sense. My fear is that our indexes will grow so large that the app can't scale.
[19:13:20] <cheeser> if other fields are indexed, the query plan will use that index. but then to match the remaining fields, it'll have to scan the matched set.
[19:13:51] <cheeser> that might suggest a data model issue. hard to say for sure without seeing that model but that's something to consider.
[19:13:55] <idd2d_> Is it true that indexing each *individual* field is insufficient? I would need to index for combinations of field corresponding to common queries, as well?
[19:15:48] <idd2d_> That makes sense. This confirms my fears, but we'll figure it out. Thanks for the tips!
[20:25:40] <idd2d_> Is there a standout search engine solution for Mongo? I've heard of people using Elasticsearch, Solr, etc. but I'm not sure which one might be best. Does it depend on the particulars of the data to be searched?
[20:35:43] <master_op> INFOS: Exception in monitor thread while connecting to server localhost:27017 com.mongodb.MongoSocketOpenException: Exception opening socket
[20:37:42] <master_op> cheeser: it occurs on connection, my program print this in the console and after the timeout expire it handle the exception
[20:39:28] <master_op> yes, i know, i wrote this .mongoClient.getDatabase(mongoDBName); to force connection
[21:45:58] <angular_mike> I'm working on a python project that uses pymongo and I want to somehow enforce that it only supports mongodb versions starting with 3.2
[21:47:28] <angular_mike> cheeser: it would be better to catch this during the application installation
[21:48:28] <GothAlice> angular_mike: You can pin versions in your project or library's setup.py file's install_requires section, for example, like this: https://github.com/marrow/contentment/blob/develop/setup.py#L105
[21:48:48] <GothAlice> That'll ensure the dependency is resolved at install or "setup.py develop" time.
[21:50:38] <GothAlice> You can also do things like ranges. I.e. pymongo>=3.2,<3.4 (i.e. to avoid large breaking changes prior to getting a chance to vet them)
[21:50:44] <angular_mike> GothAlice: oh, there's a separate pymongo's dependency that's closer to mongodb versioning scheme?
[21:54:36] <GothAlice> https://github.com/marrow/mongo/blob/develop/web/db/mongo/__init__.py?ts=4#L62-L67 < if you have connection code like this, add a call to client.server_info() to look up the version.
[21:56:18] <angular_mike> GothAlice: that's not as good as failing during install, but I gues s better than nothing
[21:56:46] <GothAlice> Difficult to have an assumption about target MongoDB server at install-time, when that's usually an application configuration option requiring the app to already be installed.
[21:59:07] <angular_mike> GothAlice: ye, I was hoping I could extrapolate available mongodb version from pymongo version
[22:00:12] <GothAlice> https://github.com/marrow/mongo/commit/94395ba2c7df99377a62ed797640056e6f6a2440 < thanks for the idea to add that as a feature to my connection helper, BTW. :)
[22:02:47] <GothAlice> https://twitter.com/taotetek/status/766288663504388096 directly relates to your wish.
[22:03:50] <angular_mike> no argument there, though that doesn't mean that as a developer you care about it