Log file Viewer
#mongodb logs for Tuesday the 19th of May, 2015
[01:58:57] <sabrehagen> would really appreciate it if somebody could provide some input on my question here: http://stackoverflow.com/questions/30229872/node-js-mongodb-connection-in-master-or-forked-thread
[05:31:23] <svm_invictvs> So I made a boneheaded move and created a collection with a - in the name.
[05:43:29] <Boomtime> svm_invictvs: I just tried creating a collection named - (single hyphen) and it works fine
[05:47:55] <Boomtime> yep, shell is a javascript console, first and foremost it will be interpreted as javascript
[05:49:51] <Boomtime> you can use . in a collection name if you like, but you'll see a similar problem in the shell most probably
[05:51:31] <Boomtime> or not.. ok, the shell seems happy with . in collection names using the shorthand format, so go for gold i guess
[08:50:41] <KekSi> i have to update some legacy java code from 2.11 to 3.0.1 to use ssl - it uses WriteResult.getError() .. what do i do with it? the function is gone
[11:21:49] <CustosLimen> how exactly do I call fsync: http://docs.mongodb.org/v2.2/reference/command/fsync/
[13:08:21] <paradoxquine> Hi folks! Quick q: i have an admin interface for events, which each have a start Date. I want to do a query that gets the next one after today. is the best way to make an index on start time and ensure it uses that (how would I ensure that?) in a findOne query with $gt: <right now's date>, or is there a better way? Thanks in advance!
[13:09:21] <StephenLynx> paradoxquine if the events have its date recorded a something different than when they were inserted on the database, you just need to store this date as a regular date object.
[13:10:17] <paradoxquine> StephenLynx: yes, the start dates aren't correlated with the creation date of the entity in the db
[13:11:08] <paradoxquine> i have the storage working fine, but retrieving the upcoming one i'm not sure about, since it says findOne uses natural ordering by default, and I'd prefer not to fetch all future events just to get the next one
[13:11:09] <StephenLynx> yeah, so you need to store a date object (optimal) or any other method you wish.
[13:12:08] <StephenLynx> so there might be more than one result for that find one because there might be more than one event that fits into the date?
[13:12:51] <paradoxquine> i could do that, yea. i was hoping, since i already have an index on start date, that I could utilize that to just find the next one after today's date
[15:45:31] <Derick> V10l4t3d: make a log: see http://derickrethans.nl/mongodb-debugging.html#mongolog
[15:49:24] <saml> if my program does things based on oplog. and it went down. and oplog scrolled away, waht do I do when the program comes back up?
[15:52:37] <saml> but if db and app are separate processes, app could go down while db is running.. and app loses those oplog
[15:53:18] <cheeser> yes. that's a danger. replica sets have the same danger. it's only a problem for prolonged downtimes.
[15:53:21] <saml> consistency across differnent dbs. i have a db and another that's denormalized version of db
[17:39:39] <grazfather> is it possible to insert a new document, but fake the DT part of the ObjectID. We'd like to avoid needed to keep dt is a separate field, but then we'd need to fake first four bytes of the objectid
[17:40:56] <grazfather> sounds like i will need to, but we basically wanted to simplify querifes, filtering on the foreign key being newer than a certain date
[17:41:32] <grazfather> but migrating to this new schema would have all object ids look like they were created within a few minutes of each other
[17:42:16] <grazfather> I know I can specify the object id but ideally I'd just specify the timestamp part of the field and have mongo db create the rest of it/guarantee uniqueness
[17:44:14] <cheeser> depending on your driver, you can specify the timestamp part of the ObjectId, iirc
[17:58:46] <deathanchor> we have some ID value, and the devs made it a field name instead of a value. makes querying on that field a pain.
[18:01:13] <deathanchor> yeah, cause I have to query that data now and it's going to do a full scan.
[18:51:36] <gswallow> Howdy y'all. Will this ever end? Tue May 19 18:49:38.659 [rsMgr] replSet I don't see a primary and I can't elect myself
[18:53:11] <GothAlice> Did you snapshot after running rs.initiate() or before? I.e. where in this process (http://docs.mongodb.org/manual/tutorial/deploy-replica-set/) did you snapshot?
[18:54:31] <GothAlice> Then those snapshots can't really work for you. You need to snapshot before. (rs.initiate() populates an initial replica set containing only one node, the one you are running rs.initiate() on, the rest need to be added to this cluster by running rs.add() on the node you ran rs.initiate() on.)
[18:54:55] <GothAlice> (And because that node configuration includes things like host names… yeah… snapshot before.)
[18:56:14] <GothAlice> … are all of the nodes clones copies of a single original configured with rs.initiate() and rs.add() with all nodes? If that's the configuration, then the error boils down to the nodes just can't talk to each-other.
[18:57:02] <gswallow> all of the nodes create EBS volumes from snapshot, where the snapshot is taken (obviously) from the one EC2 instance I have.
[18:57:34] <gswallow> So when I spin up the new instances, I tried to just drop a new rs configuraiton in place with the new hostnames.
[18:57:58] <gswallow> I *thought* about setting up a private hosted zone in route53 and setting up tokumx*.prod.indigobio.com as CNAMEs.
[18:58:48] <GothAlice> Typically if one has a reason to use TokuMX, one has a reason to get a commercial support agreement from them. ;) What filesystem are you running on those volumes? You need to use one capable of snapshotting. Beyond Amazon EBS's natural snapshot capability, which will potentially hit partial writes.
[18:59:19] <GothAlice> That's not so good, then. The EBS snapshots won't actually be filesystem snapshots.
[19:00:04] <gswallow> according to the tokumx documentation, so long as I fsfreeze it it should be good. I was — when we ran mongo — experimenting with xfs, and running a db.fsynclock or whatever.
[19:00:48] <gswallow> the replication engine is atomic so if I stop the database's ability to write to the fs (using fsfreeze) it should be ok
[19:01:10] <GothAlice> … yeah, but I'd imagine that introduces a raft of new problems, depending on the duration of the snapshot process.
[19:01:53] <gswallow> actually, I think that mongodb's idempotent replication engine is better. atomic operations back up very easily
[19:02:33] <gswallow> but we needed document level write locks and 85% storage savings isn't bad, for free. I'd probably give 3.0/wired tiger a closer look
[19:03:44] <GothAlice> So, it still boils down to the nodes can't talk, if you're seeing what appears to be a valid rs.conf() on each of them.
[19:03:54] <MrAmmon> for the benefit of us noobs, can you elaborate slightly on the 'not ready' part?
[19:04:27] <GothAlice> MrAmmon: There are a large number of outstanding critical data loss, reliability, and performance issues.
[19:04:53] <GothAlice> I.e. unless you can throw an infinite amount of RAM at it, it _will_ eventually crash after it finishes consuming all of it.
[19:05:23] <gswallow> the nodes can talk. When I shut one down because I'm impatient, I get notifications that the node has gone down / up / back into startup2
[19:05:28] <GothAlice> (Gratuitously large quantities of RAM in a machine can "practically" avoid the memory issues by simply deferring them well into the future.)
[19:06:34] <leandroa> hi, I'm preparing the scheme for time series data, but I'm confuse on how to store/pre-allocate some dynamic keys. What do you think of this schema? how can I make the second case work? https://gist.github.com/lardissone/96b71cf666f5c5d71e31
[19:06:54] <GothAlice> leandroa: Avoid dynamic keys like the plague. They are a trap for unwary travellers.
[19:07:33] <gswallow> this is purely a toku thing, but I also see a lot of disk read activity. I spun this up with a cloudformation template. I'm going to let this one sit for a while now that I'm not running on r3.4xlarges, and see what happens after….say…two or three hours.
[19:07:54] <GothAlice> Also, that compound _id is… gratuitous. MongoDB already supports multi-value _ids using a nested document. (Order of the keys matters, obv.) I.e. {_id: {uid: userID, pid: productId, ts: timestamp_hour}, …}
[19:07:56] <leandroa> GothAlice, I should yes.. but I'm not sure how can I make it work well, even with queries using these dynamic keys
[19:08:56] <leandroa> and how can I preallocate this? or using them as values doesn't requires disk pre allocation?
[19:09:41] <GothAlice> No need to pre-allocate, unless you're expecting the document to substantially increase in size across several operations.
[19:10:28] <bogomips> hello to everyone, I was needed, by php, to update an object adding a sub-object like this {'a:'1',arr:[]} the problem was that the php code array('a'=>'1',arr:array()) makes {'a:'1',arr:{}} so I had to use a workaround declaring in php array('a'=>'1',arr:array(array())) and later get empty the array "arr". I bet exists a clear way!Thanks
[19:11:04] <GothAlice> leandroa: http://www.devsmash.com/blog/mongodb-ad-hoc-analytics-aggregation-framework
[19:12:08] <GothAlice> leandroa: See also: https://gist.github.com/amcgregor/1ca13e5a74b2ac318017 (some extracts from my work, where we do click tracking)
[19:13:01] <GothAlice> leandroa: In my case, I don't worry about preallocation at all. The number of unique combinations of browser and OS are limited.
[19:13:36] <GothAlice> (And since not every browser / OS / etc. will be represented in each per-hour slice of statistics, pre-allocation would be excessively wasteful.)
[19:14:28] <GothAlice> (My structures here go against the advice I gave earlier on storing "dynamic attributes" as a list of named sub-documents… *because* this is a fixed set of attributes, not truly dynamic.)
[19:14:30] <leandroa> I should preallocate these keywords, that aren't happening on every time series
[19:15:05] <GothAlice> If you scroll down to the sample record on my gist, you'll see that I explicitly don't pre-allocate. I don't list a count of zero against every possible version of Windows, Linux, Mac, Android, etc. just to have zeros there.
[19:15:19] <GothAlice> When querying, summing a zero is the same as skipping the record because the field is missing. ;)
[19:16:40] <GothAlice> Hour, though that's entirely arbitrary, and it's common to have multiple time scales.
[19:16:49] <GothAlice> (I.e. replicate the statistics on a per-minute, per-hour, and per-day basis.)
[19:17:28] <GothAlice> This helps reduce the individual number of records that need to be processed to handle different scales of query. I.e. asking for aggregate stats over the course of a week? Use the daily pre-aggregated data and only need to sum 7 records! :D
[19:23:45] <GothAlice> leandroa: http://s.webcore.io/image/142o1W3U2y0x < our dashboard, running across that pre-aggregated click data
[19:25:01] <GothAlice> Oh, and the "Live Activity Feed" at the bottom is a live stream out of a capped collection. :)
[19:26:56] <GothAlice> The aggregate query in the gist I gave generates the top left click-through comparison chart, BTW.
[19:29:05] <gswallow> I thought I was crazy when I tested this earlier with a 40GB data set and it worked. I have like 1.3TB in production.
[19:30:27] <GothAlice> ^_^ I rolled my own xz-based GridFS compression before TokuMX was around, alas, and there's no way on Earth I'll migrate that 28 TiB to a different cluster at this point. It's just too big to move. XD
[19:31:25] <gswallow> where we are contractually obligated to host our app, we're on 7200 RPM spinning rust o_O
[19:33:25] <gswallow> Ironically I worked at the ISP I'm hosted at when they picked their pricing model and I thought they were on crack back then.
[19:33:33] <GothAlice> https://twitter.com/GothAlice/status/582920470715965440 < it'd cost me around half a million $ per month to host my array with Compose.io. ;)
[19:37:02] <gswallow> That $14,400 per month price is about what I'm paying, yeah. I can buy them a *new SAN* every six months.
[19:37:40] <GothAlice> If you host > 1TB, it's actually more cost effective to buy 100% new hardware, colo it, and hire a DBA to manage that one box. ;)
[20:47:56] <dave_den> what's the equivalent of db.stats() using the new 2.x mongo ruby gem? using client.database.command(:stats) just hangs for me
[22:19:02] <Thinh> I'm using the latest pymongo, and this query just seems to throw me an error: col.find().max_time_ms(100)
[22:19:58] <Thinh> "OperationFailure: database error: Can't canonicalize query: BadValue unknown top level operator: $query"