Log file Viewer
#mongodb logs for Thursday the 9th of April, 2015
[06:16:41] <spiderbyte> in Mongo, on the CLI, I'd enter something like db.deliverables.find({"_id": ObjectId('actualObjectIdReplacedToPasteInThisIRCChannel')}), and it would run
[07:24:23] <spiderbyte> The answer was here: http://www.dartdocs.org/documentation/mongo_dart/0.1.39/index.html#mongo_dart/mongo_dart.ObjectId
[07:24:41] <spiderbyte> example: Cursor cursor = _requestedCollection.find({"_id": new ObjectId.fromHexString(objectId)});
[07:33:34] <smoke_> hi im trying to make a timestamp style schema for a website recording data every 30seconds to a mongodb, i was thinking like this db -> collection -> document with { name : 'name', 00101001: { x: value, y: value }, 09129100: { x: value, y: value }} and update the document with a new 901291: { x: value, y: value } 901291 being the timestamp value would this be best scenario for me for performance?
[09:18:55] <d0x> Hi, can someone tell me with which pig version the mongo-hadoop connector is compatiable or got tested?
[09:38:47] <d0x> Okay, it should be pig 13. https://github.com/mongodb/mongo-hadoop/blob/r1.3.2/build.gradle
[09:41:45] <jdo_dk> If i have a replicaset, and wants a single / standalone server, to be a "backup" of a single db, is this possible by config in mongo or should i code this my self?
[10:44:34] <steinkaus> 2015-04-09T11:16:48.803+0200 [conn323675] command accepted.$cmd command: count { count: "tickets", query: { bet.ext.boId: { $in: [ 10 ] }, state.result: { $in: [ "*" ] } }, maxTimeMS: 30000 } planSummary: IXSCAN { bet.ext.boId: 1, bet.ts: -1 } ntoreturn:1 keyUpdates:0 numYields:1127 locks(micros) r:82935994 reslen:92 108910ms
[11:26:02] <Derick> steinkaus: I am not sure whether everything respects maxTimeoutMS - which server version is this?
[11:28:05] <Derick> there is a stackoverflow answer at http://dba.stackexchange.com/questions/73939/mongodb-maxtimems-ignored too
[14:26:32] <revoohc> Question on upgrading from 2.2 to 2.6. The docs say we need to apply 2.4 first. But what does this mean? Do I need to just start the db in 2.4 mode before I upgrade to 2.6, or is there more to it?
[14:56:08] <jigax> Hi guys I'm working on a inventory system that I'm thinking of having two tables one for transactions and one to keep running totals. Does this make sense or should I look into other design patterns?
[14:59:44] <jigax> A collection of parts which keeps tracks of how many parts I currently have available
[15:02:47] <StephenLynx> I would have a collection of products and have a field indicating the amount available of said products.
[15:10:26] <StephenLynx> instead of each document being about a single product, it would hold information about several products traded in that transaction as a whole.
[15:13:33] <pjammer> i've got an index job right now that is slow and there are a tonne of records. anyway to speed it up whilst it running? what does kililng it do?
[15:13:35] <StephenLynx> and keep in mind that using mongo for what you need will impose some limitations.
[15:14:43] <StephenLynx> if you wish to display the name of the object of that transaction, you would have to either A) query for the name of the product based on the id of the product's transction (really bad))
[15:15:39] <StephenLynx> or C) just display whatever unique value you used to associate the product with the transaction (not optimal)
[15:16:18] <StephenLynx> because you are already doing a relational thing by keeping it in two separate collections
[15:17:43] <StephenLynx> you could also keep the transactions of the product as a sub array in the product document.
[15:18:18] <StephenLynx> but that would impose even more limitation on querying and a 16mb limit when you expect to don't have a limit on how many transactions you would have on a product.
[15:20:46] <StephenLynx> I also had to use separate collections for things that could (and were) sub arrays.
[16:03:04] <revoohc> Question on upgrading from 2.2 to 2.6. The docs say we need to apply 2.4 first. But what does this mean? Do I need to just start the db in 2.4 mode before I upgrade to 2.6, or is there more to it?
[18:01:41] <ikarous> Bit of a long shot here, but has anyone ever found themselves unable to query any sharded collections after fully resyncing a secondary and electing it as the new primary on one of the shards? We get error messages like the following: https://gist.github.com/kyle-lee/be822a0fe2e9c899361a
[18:12:42] <mbrewer> @ikarous: I'm not running anything sharded, sorry... I've got a question too though (well... two, but I'll start with this one), On mongo 2.6.8, client 1.4.19 we whitnessed 28k open connections to a single client process (via netstat) to a resyncing member of a replica set, but no other time. I'm pretty confident that we aren't leaking those connections. It looks suspiciously like a bug, maybe in the clie
[18:12:48] <mbrewer> nt, but I can't find anything quite like this one in jira (best I found was https://jira.mongodb.org/browse/NODE-395?jql=project%20%3D%20NODE%20AND%20text%20~%20%22socket%20close%22)
[18:14:35] <mbrewer> I'm wondering if upgrading the client might help resolve it... resync fails for us in the end actually (it used to work using mongo 2.4.10 on the same replicaset, same scale), it seems to just run off the oplog after 3 days.
[18:18:42] <mbrewer> We *ALSO* are having some weird performance issues with 2.4.10, where lock percentage is low, but calling "db.stat()" on one large database locks up the cluster completely for 7 minutes or more (mms stops even getting data from it), but it's not CPU, disk, memory, or network starved (all those dip during that time as well, then spike presumably as it recovers and runs through a work queue), yet I've never
[18:18:48] <mbrewer> whitnessed a lock percentage above 35% or so, via mms or mongostat (maybe it's too frozen up to actually get the stat?). These issues were why we started mucking with 2.6 in the first place :).
[18:23:50] <learns> mbrewer: You know what I can do about this? - FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
[18:24:14] <learns> I got 280k docs in the db. I'm using this function: Matches.find({}).toArray (err, docs) ->
[18:26:07] <mbrewer> are you running 64 bit, are you on linux, if so do you have overallocation turned on and have you adjusted your ulimits?
[18:26:47] <wawrek> Hello. I have documents with lists of nested docs. I would like to build a query that is run on the list field and returns only the sub-documents that match the query string. Is this possile with queries or should I use the aggregate framework?
[18:30:18] <mbrewer> ah... maybe you should add a "limit" to your query and get just a couple thousand at a time
[18:33:48] <mbrewer> you might also look in to using the cursor, rather than turning it in to an array immediately
[18:44:28] <mbrewer> I've mostly being admining mongo, not writing code for it... but I believe the cursor is kindof like an iterator, so you're not pulling all the data out from mongo at once. If you turn it in to an array it has to take all the data and put it in memory in your node process. db.collection.find() returns a cursor see: http://docs.mongodb.org/manual/core/cursors/
[18:46:00] <mbrewer> basically, you can pull in the documents in chhunks as you use them, instead of all at once... so you don't run out of ram
[20:49:26] <akp> i have the following regex to try to match toll free numbers, /^(1?[888|800|877|855|866]{3}\d{7})/, i'm a little confused on how to use the mongo regex. when i run it i seem to not get my pattern match, i've tried $in and $regex... and ideas?
[20:51:48] <GothAlice> Indeed, [] doesn't mean what you thought it meant in the first reggae example. ;)
[20:53:45] <GothAlice> The NANP (North America Numbering Plan) standard specifies the following: ^(?:(?:\+?1\s*(?:[.-]\s*)?)?(?:\(\s*([2-9]1[02-9]|[2-9][02-8]1|[2-9][02-8][02-9])\s*\)|([2-9]1[02-9]|[2-9][02-8]1|[2-9][02-8][02-9]))\s*(?:[.-]\s*)?)?([2-9]1[02-9]|[2-9][02-9]1|[2-9][02-9]{2})\s*(?:[.-]\s*)?([0-9]{4})(?:\s*(?:#|x\.?|ext\.?|extension)\s*(\d+))?$ — doesn't handle limiting to only "toll-free", though.
[20:56:44] <akp> the pattern i've supplied matches what i need it to. i just need to know how to execute it directly with mongo find
[20:56:51] <GothAlice> akp: Like trying to parse HTML using regexen (http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454), telephone numbers are inconsistent enough to make basically unmaintainable regexes like the above a thing. ;) If all you care about searching for is "toll free", why not store the area code separately from the string representing the number?
[20:58:28] <akp> so my question is how to actually run it in with mongo .find(). i have the pattern built, but when i run it it returns every did
[20:58:41] <GothAlice> akp: However, I find the need to use regular expressions to be an indicator that it may be a good idea to pivot the data a bit to _avoid_ said regular expression. If you're sanitizing, just store the number[1:4] slice separately, then you can quickly search for db.stuff.find({area_code: {$in: [888, 800, 877, 855, 866]}})
[20:59:14] <akp> the data is already sanitized. its some stuff i inherited, so i can't go modifying the records.
[20:59:45] <akp> the numbers are stored in the collection already scrubbed. the only thing is some of them might not have the 1 on it
[21:00:33] <GothAlice> Here is me trying it out and seeing that it does, in fact, work: http://showterm.io/c627f605a691c4ba53e01
[21:03:08] <fewknow> mapping device_id to cookie_id.....every week it is a full replace of the mappings.
[21:04:00] <akp> GothAlice: when i tried it this db.number.find( { number: { $regex :[/^(1?[888|800|877|855|866]{3}\d{7})/] }}) it returned everything
[21:04:07] <GothAlice> fewknow: Drop and import? Upserts? Import into a temporary collection, drop the old one, rename the new one over top? Several possible approaches.
[21:04:55] <fewknow> GothAlice: yeah....i was going to create new collection everyweek, but it will fragment the disk backing the db. When you drop a collection you don't reclaim space for the OS.
[21:07:50] <GothAlice> fewknow: Alas, with what you're doing, it's hard to avoid. If it actually becomes a problem, you can use http://docs.mongodb.org/manual/reference/command/compact/ on a periodic maintenance schedule to defragment the data.
[21:07:52] <akp> db.number.find( { did: { $regex :/^(1?[888|800|877|855|866]{3}\d{7})/}}) also returns everything in the collection.
[21:10:57] <GothAlice> akp: You should read up on regular expressions and tone down your love of square brackets. They _do not_ mean what you think they mean. http://showterm.io/1cc56721a3aa61a37769a < me debugging it
[21:12:07] <GothAlice> http://rgruet.free.fr/PQR27/PQR2.7.html#reModule is a reasonable summary, even if it's Python-flavoured, not Perl-flavoured regular expressions. The basics are there.
[21:13:34] <GothAlice> akp: No worries; the frustration on my end is with myself for explaining poorly. :)
[22:34:00] <luffy_> Is it common for map/reduce functions to return failed: exception: too much data for in memory map/reduce?