Log file Viewer
#mongodb logs for Thursday the 1st of January, 2015
[00:04:43] <pbryan> On 2.6.6, I seem to be successfully using an index with multiple keys, two of which contain arrays of strings. This seems to contradict the documentation. Query plan explains I am using the index. How is this working?
[00:06:50] <morenoh149> mordof: the docs are pretty good. I think you store your geo data as normal and then use a 2d/sphere index to make looking them up faster
[00:09:42] <Boomtime> it is possible to use each field as an array individually, just not both in the same document
[00:11:03] <Boomtime> also, can you provide the query, and output showing the document which violates this constraint
[00:13:44] <mordof> morenoh149: hmm. i just need to learn the proper way to store geo data then, heh
[00:17:50] <pbryan> { "_id" : 2, "text" : "This is a test of the emergency broadcast system. #fail #zoolander #yahoo", "keywords" : [ "test", "emergency", "broadcast", "system", "fail" ], "hashtags" : [ "fail", "zoolander", "yahoo" ], "language" : "en", "geojson" : { "type" : "Point", "coordinates" : [ -123.14489, 49.30452 ] }, "place" : "Stanley Park", "time" : ISODate("2014-12-30T12:00:00Z") }
[00:19:08] <mordof> i'm seeing a lot of lat/lon used with the more recent mongodb indexes - i don't want earth lat/lon coordinates, is it still possible for me to use newer indexing?
[00:21:48] <pbryan> According to the docs, with an index on two fields, you can't even insert if both fields have arrays.
[00:22:04] <pbryan> If you attempt to insert such a document, MongoDB will reject the insertion, and produce an error that says cannot index parallel arrays.
[00:25:38] <Boomtime> holy cow.. the 2dsphere seems to cause it to go blind to stopping parallel array indexing
[00:26:33] <Boomtime> if you index without the 2dsphere the index is rejected if the document exists already, and the insertion is rejected if the collection was empty first
[00:33:25] <mordof> trying to find out if i can use the newer mongo 2dsphere / geojson indexes with coordinates that aren't lat/lon, and shouldn't be calculated based on it being on earth... or .. whatever it does
[00:34:02] <mordof> the "legacy" 2d indexes would seem to be what i actually want, but some articles have suggested those might be getting removed since they're being labelled as legacy
[00:34:03] <Boomtime> pbryan: will you raise a server ticket? alternatively, do you mind if i use your data with minor changes to raise one?
[00:39:58] <mordof> seems like i'm just gonna have to deal with my coordinates like regular integers instead
[00:58:52] <Boomtime> also, the query must contain the geo part of the index won't be used, no matter how otherwise applicable it might be
[01:03:36] <pbryan> In other words, if I used the geo without the other fields, it would prefer the index.
[03:15:46] <jlolofie> I'm importing a large dataset; mongoimport doesnt report any errors and keeps priting the progress as if things are going fine
[12:06:17] <gundas> Hi all, I have one node js application which is generating random data and putting it into a capped collection and a uncapped collection. I also have another node js application which is reading from the same database and tailing the capped collection and then sending data through sockets to the ui. I;m using mongoose but confused about recreating the schema any ideas on the best way to set this up?
[12:08:13] <Tailor> Social-networking: Hadoop, HBase, Spark over MongoDB or Postgres? - http://stackoverflow.com/q/27730628
[19:25:15] <Industrial> Hi. Using the NodeJS driver for mongodb, can I get an update event from when something from selected collections update?
[19:25:42] <Industrial> I would like to create a mechanism for a browser to subscribe to a collection or model for updates, and gets this streamed through a websocket.
[19:59:14] <capleton> I'd like to create new subdocuments using mongo, but when I try to use insert() with $set, i can only manage to overwrite the existing subdocument. Can someone point me in the right direction?
[20:21:51] <polhemic> I have a random and possibly noobish questions about journal files. Why do mine persist for long (hours) periods of time?#
[20:22:23] <polhemic> I thought they only needed to exist until the change were applied to the shared view and remapped every minute or so.
[20:23:38] <polhemic> But for me it keeps growing - currently the journals are twice the size of the sum of the *.[012] files
[20:47:30] <polhemic> but I was expecting them to hit a timeout or threshold and then drop to zero again when the entries were commited to the database files
[20:48:25] <edrocks> not exactly an expert on this but i will see if i can find the right docs about the files doubling i think they should have more info
[21:05:30] <polhemic> There's nothing very helpful on that page. Following through to the journaling page, there's the sentence "Once MongoDB applies all the write operations in a particular journal file to the database data files, it deletes the file, as it is no longer needed for recovery purposes."
[21:11:14] <polhemic> So, my concern is that something I'm doing client side is stopping the journal getting processed and removed after the write operation is complete
[21:18:05] <Boomtime> polhemic: what is the problem you are observing? (i did not see the start of the conversation)
[21:19:43] <polhemic> no problem as such, just concerned about the size of my journal directory continuously climbing.
[21:20:32] <polhemic> I'm running a write heavy app, and the journal just keeps growing - it's at almost 1GB after 3 hours of runtime
[21:21:23] <polhemic> Once the writes have gone through to the database files, I was expecting it to drop in size - containing only the very latest changes that haven't gone into the shared model yet
[21:26:20] <Boomtime> polhemic: afaik journal files are constantly recycled while mongodb is running
[21:27:40] <polhemic> journal just keeps going up and, because I've got an update heavy dataflow, the db size will asymtote to a final value, but the journal will just keep growing and growing
[21:28:30] <Boomtime> as in.. if you pause writes for a minute or two, then start writing again, it allocates more journal?
[21:30:32] <polhemic> Is there something I should be doing client side to periodically close and reopen or flush my connections?
[21:31:29] <polhemic> As it stands, I'm creating a persistent pymongo.MongoClient(), then I just keep calling update() on a collection (with upsert true)
[21:33:46] <polhemic> I've just tried closing my client and the journals haven't been removed. I'm just going to restart mongod to check they get removed on shutdown.
[21:37:35] <polhemic> I've restarted the app and the journal is off and running again - increasing by a few hundred kBytes a second
[21:39:09] <opmast> how to remove 10 specific _id's from database ? i have seen only remove with specific criteria or all documents ..
[21:40:36] <Boomtime> huh, 1GB is the limit to a journal file: http://docs.mongodb.org/manual/core/journaling/
[21:42:11] <polhemic> The docs talkabout running a write flush every 60 seconds, so you have to be shifting some serious data to hit 1GB with 60s of writes. Unless you're me.
[21:43:17] <Boomtime> opmast: $in specifies an array of items to match, you have 10 items to match, thus $in is perfect
[21:45:45] <Boomtime> polhemic: the journal is not read except at start-up (crash recovery), so it always writes ahead - wait for a minute or two after a new file is allocated and see if the old one disappears then
[21:46:23] <Boomtime> note it may take more than a minute after the new file is created, due to overlap - wait at least two minutes (or a bit longer) to be sure
[21:48:15] <Boomtime> yeah, that amount is normal - i have just learned that the limit is 1GB - i thought it was less than that
[21:48:51] <polhemic> But at 1GB it just creates a new journal file. The only time it ever removed the journals is when the server is rebooted.
[21:49:43] <polhemic> This means that either my server has to have a periodic restart to clean out used journals, or I've got to turn journaling off (which I really don't want to do)
[21:53:56] <opmast> Boomtime how to delete _id when i make _id : 54a57bd848177e240d8b4567 it not work
[22:19:06] <polhemic> Boomtime: I've done a bit more digging, and this seems to be a common issue - journals filling discs.
[22:19:33] <polhemic> The workaround is to db.fsyncLock() and db.fsyncUnlock() the database, which cleans up the journal.
[22:20:40] <polhemic> http://superuser.com/questions/657197/why-journaling-use-so-much-space-in-mongodb
[22:22:13] <Boomtime> please do not use db.fsyncLock that will just cause you grief in other fun ways
[22:22:59] <Boomtime> polhemic: it may end up using the same amount if that is what is required, but it will also cause the files to be available for recycling sooner
[22:23:46] <Boomtime> what is happening is that the previous journal file is not being made available for recycling until the current one is full
[22:24:17] <Boomtime> thus it sits around taking up space until a new one is needed, then it is reset and recycled for use as the new journal file
[22:24:48] <Boomtime> thus, 2GB is not a hard limit - but it's unlikely you'll be able to push it above that
[22:44:30] <Nitax> Hmm. I would assume that Mongoose wouldn't change that but it would be nice to be 100% sure before I assume I don't need to keep track of it myself