PMXBOT Log file Viewer

Help | Karma | Search:

#mongodb logs for Tuesday the 23rd of August, 2016

(Back to #mongodb overview) (Back to channel listing) (Animate logs)
[03:12:32] <the_f0ster> I am running an aggregate function and can't figure out how to get the data out of the response object.. any suggestions ? https://gist.github.com/f0ster/821fc822521695b80843758ab42b15fb
[03:13:15] <cheeser> https://gist.github.com/f0ster/821fc822521695b80843758ab42b15fb
[03:13:24] <cheeser> grumble grumble
[03:16:22] <joannac> the_f0ster: what's the output of the last line in that gist?
[03:16:38] <the_f0ster> joannac: nothing totally blank
[03:17:13] <the_f0ster> pretty strange
[03:17:19] <joannac> agreed
[03:17:26] <joannac> what happens if you type "test" again?
[03:17:34] <the_f0ster> sometimes it comes back blank
[03:17:45] <the_f0ster> it keeps disappearing
[03:18:19] <the_f0ster> check it out again, https://gist.github.com/f0ster/821fc822521695b80843758ab42b15fb
[03:19:49] <the_f0ster> I need to do this aggregate sum on several collections, then sum the sums
[03:20:08] <the_f0ster> doing a for loop with the aggregate inside.. assume there isnt a convenient way to do an aggregate across multiple collections
[03:20:42] <joannac> try typeof(test) and typeof(test1)?
[03:31:18] <joannac> long story short, I think this is a javascript problem with variables and scoping.
[03:37:45] <the_f0ster> joannac: they both say object
[03:38:04] <the_f0ster> yeah it is strange.. i am going to test writing to a tmp output table and just read from there
[03:42:58] <joannac> oh, i figured it out. aggregation returns a cursor. you've exhausted the cursor
[07:59:38] <hypfer> hi everyone. I have made backups in mongodb with the archive option and now I can't restore them to wiredtiger
[07:59:46] <hypfer> the restore just hangs doing nothing
[08:06:17] <hypfer> the backup was made on a machine which is _not_ set to wiredtiger if that matters
[13:31:30] <onis> guys is this file to much messy? https://gist.github.com/Fen24/8e5b5f4cb46aeab47cc150b95663fb96
[14:35:33] <Zelest> Derick, would be nice if http://php.net/manual/en/mongodb-bson-utcdatetime.todatetime.php could take an argument (timezone)
[14:37:48] <Derick> Zelest: we won't do that with toDateTime, but we are tracking this at https://jira.mongodb.org/browse/PHPC-760
[14:38:36] <Zelest> Ah
[14:38:57] <Zelest> Also, would be nice if toDateTime at least got set to whatever's in php.ini.. like new DateTime() does
[14:39:21] <Derick> Zelest: that would be a BC break now
[14:39:32] <Derick> it's also potentially a bad idea
[14:39:36] <Zelest> why so?
[14:39:50] <Derick> timezone information gets lost storing a UTCDateTime in MongoDB
[14:40:02] <Zelest> Ah
[14:40:08] <Derick> if another driver sets it with a different timezone, you won't get the same one back... potentially
[14:40:15] <Zelest> <td><?= e($item['created']->toDateTime()->setTimeZone(new DateTimeZone('Europe/Stockholm'))->format('Y-m-d H:i')) ?></td>
[14:40:21] <Zelest> so insanely ugly :(
[14:40:24] <Derick> yeah
[14:40:50] <Derick> Zelest: the 1.3 driver, will allow a typemap to your own date/time class though
[14:40:57] <Derick> so you can get rid of all of that everywhere
[14:41:06] <Zelest> oh :o
[14:41:23] <Derick> let me show an example
[14:41:54] <Zelest> or wait, 1.3 driver? :o
[14:42:02] <Derick> https://github.com/mongodb/mongo-hhvm-driver/blob/master/tests/bson-deserialize-typewrapped-utcdatetime-001.phpt
[14:42:07] <Zelest> i run 1.1.8 :o
[14:42:09] <Derick> yeah, we're doing 1.2 first - maybe
[14:42:15] <Zelest> ah
[14:42:18] <Derick> 1.3 is the MongoDB 3.4 support one
[14:42:25] <Zelest> oh
[14:42:36] <Zelest> so for now, the ugly ugly code is a must? :)
[14:42:39] <Derick> but that seems more and more unlikely, so 1.2 would be the one with it in :)
[14:48:23] <Zelest> aha!
[14:48:29] <Zelest> a somewhat cuter solution for now:
[14:48:35] <Zelest> <td><?= date('Y-m-d H:i', $item['created']->__toString() / 1000) ?></td>
[14:48:36] <Zelest> :D
[14:48:48] <Derick> that's uglier :P
[14:48:55] <Zelest> aw, why? :(
[14:49:24] <Derick> it would only work for the default timezone, not for any arbitrary one
[14:49:46] <Zelest> ah, well, yeah..
[14:50:03] <Zelest> in my situation that's more than enough :)
[14:50:09] <GothAlice> Zelest: Also, minor note since it doesn't seem to have been mentioned yet, if your "_id" field is an ObjectId (and it should in most cases), it includes the creation time built-in. No need for an additional field in the majority of cases. https://docs.mongodb.com/manual/reference/method/ObjectId/
[14:50:30] <Zelest> yeah :)
[14:50:54] <Derick> GothAlice: no method in the PHP driver to get it out though.
[14:51:07] <Derick> (on purpose)
[14:51:11] <Zelest> really?
[14:51:23] <GothAlice> Derick: Let me add that to the whiteboard of shame.
[14:51:30] <Zelest> Derick, my issue now is basically "show stuff in local time" .. we don't plan on switching timezones or such
[14:51:40] <Derick> GothAlice: well, ObjectIDs aren't *meant* to be used as creation timestamps...
[14:52:09] <Zelest> yeah, I've read the creation date is there.. but also to "never trust it"
[14:52:46] <GothAlice> Zelest: You can trust it if you can trust your driver. Apparently one can not trust the PHP driver.
[14:53:13] <Derick> uh
[14:53:14] <Zelest> trust no one! foilhat on! :D
[14:53:40] <cheeser> i want to believe!
[14:54:30] <Zelest> paranoia is a gift, not a curse
[14:56:06] <GothAlice> It's not millisecond accurate, but that's "eh?" for most use cases, and there exist no warnings against use (and methods to explicitly support use) in the general shell documentation and Python driver documentation, as examples. I.e. https://docs.mongodb.com/manual/reference/method/ObjectId/ (+ getTimestamp() method) and http://api.mongodb.com/python/current/api/bson/objectid.html (with from_datetime factory and generation_time accessor).
[14:56:40] <Derick> GothAlice: you can't do a simple range search on it though
[14:56:45] <Derick> (ie, between this and this date
[14:56:46] <Derick> )
[14:56:51] <GothAlice> Derick: Yes, yo ucan.
[14:56:56] <GothAlice> s/yo u/you /
[14:57:14] <Derick> not without messing about with the unix timestamp to compare against first
[14:58:40] <GothAlice> Which is why the type provides a method for that explicit use.
[14:58:50] <Derick> sorry?
[14:58:52] <GothAlice> http://api.mongodb.com/python/current/api/bson/objectid.html#bson.objectid.ObjectId.from_datetime
[14:58:56] <Derick> I have two timestamps
[14:59:03] <Derick> start of today, and start of tomorrow
[14:59:09] <GothAlice> Look at the example on that link.
[14:59:12] <Derick> how do I find documents created today without a hack?
[14:59:24] <GothAlice> There is no hack. It's an official factory method for exactly that purpose.
[14:59:29] <Derick> ObjectID.from_datetime is a hack
[14:59:48] <GothAlice> Officially supported features are hacks, now? And cats rule the world. ;)
[15:00:18] <Derick> GothAlice: there is a big warning for a reason there
[15:00:31] <jon1012> (wow I've forgot channels with ops were still a thing on freenode :p)
[15:00:31] <GothAlice> That warning is about inserting those generated values as _ids.
[15:00:41] <GothAlice> The warning has nothing to do with range querying, the intended purpose of the method.
[15:00:46] <Derick> you're abusing a date type for something it's not meant for
[15:01:00] <StephenLynx> I don't get it.
[15:01:10] <StephenLynx> if you want to know when the document was created
[15:01:19] <StephenLynx> why don't you give the document that information?
[15:01:33] <StephenLynx> is not the db responsability to tell you that.
[15:01:40] <StephenLynx> even though it is able to with _id.
[15:01:43] <GothAlice> StephenLynx: The _id isn't generated by the DB. It's generated by your application.
[15:01:48] <StephenLynx> ah.
[15:01:48] <GothAlice> (Common misconception.)
[15:01:52] <Derick> GothAlice: it's generated by the drivers.
[15:02:04] <StephenLynx> still application code.
[15:02:06] <Zelest> note to self, don't ask date related questions. :D
[15:02:12] <GothAlice> ;P
[15:02:17] <StephenLynx> oh, you haven't seen nothing yet
[15:02:26] <StephenLynx> wait until you ask framework related questions :^)
[15:02:39] <GothAlice> Derick: In the face of officially supported, documented, and a with multiple example features, calling it "abuse" → http://s.webcore.io/e7Sk/opinion-man.jpg
[15:02:46] <StephenLynx> kke
[15:02:54] <GothAlice> ;)
[15:04:30] <GothAlice> Now, on the other hand, since I can't len() a BulkOps set to see how many operations have been added, I'm forced to access ops._BulkOperationBuilder__bulk.ops — now *that* is a hack.
[15:05:17] <GothAlice> (Notably, because I explicitly do not want an exception raised in the "no-op" case, as it currently does if you try to .execute() an empty BulkOps set.)
[15:05:29] <Derick> GothAlice: PHP gets that one right: http://php.net/manual/en/mongodb-driver-bulkwrite.count.php ;-)
[15:06:07] <GothAlice> Derick: So does the Python one, if I went one abstraction layer deeper in the first place. Not quite the same things being compared, here.
[15:06:26] <GothAlice> (I.e. I'm using the BulkOperationBuilder wrapper around the BulkWrite object.)
[15:06:50] <GothAlice> In the direct use of BulkWrite case, I'd have direct access to the ops anyway (note the last .ops dereference in my hack example.)
[15:07:34] <GothAlice> No such comparable functionality exists in the PHP driver. ;)
[15:08:01] <StephenLynx> not being PHP is already a functionality on itself.
[15:08:49] <Derick> GothAlice: no, because it belongs in the PHP library. Deliberate choice to keep the driver as lean as possible. Things like creating ObjectIDs from timestamps should *also* go there.
[15:14:02] <cr0mulent> Hello, I have a question about $lookup. What does the “as” Field do? I don’t understand it. https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/
[15:14:47] <GothAlice> cr0mulent: It's the name of the field you want the result of the operation saved into.
[15:15:32] <cr0mulent> GothAlice: Can I write the results to the console?
[15:15:43] <cr0mulent> I don’t want to save the results to anything, I’m trying to do a query.
[15:15:50] <GothAlice> That's entirely separate.
[15:16:12] <cr0mulent> Can I use $lookup with .find()
[15:16:19] <GothAlice> No. It's an aggregate pipeline operation.
[15:16:52] <GothAlice> Unless your aggregate has an $out stage, the results are sent to your client (i.e. the mongo shell, or your application) in a way similar to a .find()
[15:17:03] <GothAlice> Ref: https://docs.mongodb.com/manual/reference/operator/aggregation/out/
[15:17:41] <GothAlice> And: https://docs.mongodb.com/manual/reference/command/aggregate/#dbcmd.aggregate
[15:19:28] <cr0mulent> GothAlice: So my Aggregate query contains $lookup but it does not contain $out. This will output the results to the client, correct?
[15:19:58] <GothAlice> cr0mulent: Try it out. :) (Yes.)
[15:20:14] <GothAlice> cr0mulent: https://docs.mongodb.com/manual/core/aggregation-pipeline/ is the general documentation for aggregation. It's used to process data through a "pipeline" of operations before returning the data to your application or mongo shell. Find gets data, but the operations you can perform on the data server-side are limited vs. aggregation.
[15:21:48] <cr0mulent> GothAlice: Thanks, I’ll review this.
[15:22:13] <phutchins> I'm trying to migrate from one sharded cluster to another and when I start to try to add one of the new mongod's I get "Unauthorized: not authorized on admin to execute command" for the heartbeat request. Anyone know how to get around that?
[15:23:04] <phutchins> I just simply create a user on the admin db?
[16:59:19] <PSUser2> anyone around here?
[16:59:32] <PSUser2> i need some thoughts on how to improve a process
[16:59:57] <PSUser2> i want to transfer some data periodically from sql to mongodb
[17:00:02] <PSUser2> as fast as possible
[17:17:13] <PSUser2> can i index a set of a document?
[17:18:20] <cheeser> of course
[17:21:32] <PSUser2> can you give me an example cheeser?
[17:22:03] <PSUser2> i have a script that uses addToSet to sync mongo from sql, and it takes a lot of time
[17:23:38] <cheeser> oh, by "set of a document" you mean an array?
[17:37:48] <PSUser2> yes
[17:40:53] <cheeser> you can index them like any other field but i don't think that factors in to $addToSet. that will do a full scan of the array to ensure uniqueness of the new vlaue
[17:44:07] <Derick> I think so too.
[17:51:48] <PSUser2> i see
[17:51:54] <PSUser2> can i do anything about it?
[17:51:56] <PSUser2> transactions?
[17:53:28] <cheeser> how would transactions affect anything? even if mongo did support them?
[17:54:25] <PSUser2> there's a "round trip time"
[17:54:35] <PSUser2> for each addToSet
[17:56:45] <Derick> you can add more items in one go though
[17:57:20] <PSUser2> but in the same document
[17:57:22] <PSUser2> right?
[17:58:13] <PSUser2> i'm storing a graph, and its vertices, so it's a two document add per vertex
[17:58:24] <PSUser2> why a graph in a document ?
[17:58:42] <PSUser2> because it's fast for getting back the vertices for a specific node
[17:58:45] <PSUser2> faster than sql
[17:59:07] <Derick> I'd use a graph database for a graph
[17:59:09] <PSUser2> and i bet it's faster than neo4j also, if we're talking about millions of nodes
[18:00:09] <PSUser2> i have done some tests with neo4j too, but haven't compared it on this though
[18:00:16] <PSUser2> but neo4j was slow overall
[18:00:32] <PSUser2> how can i increase the speed of mongoimport?
[18:00:37] <PSUser2> i have a 700mb json
[18:02:21] <cheeser> far be it from me to steer folks away from our product, but: https://www.arangodb.com/ http://orientdb.com/
[18:03:01] <PSUser2> arangodb is way to slow for import
[18:03:08] <PSUser2> *too
[18:03:12] <cheeser> how often are you importing?
[18:03:14] <inspiron> how do i do a count on the number of fields in a collection that is not null or an empty string?
[18:03:38] <cheeser> inspiron: $exists and $ne
[18:04:13] <PSUser2> once a week
[18:04:21] <cheeser> yikes
[18:29:51] <n1colas> Hello
[18:44:42] <paperziggurat> having trouble with a lock on a file that won't let me run mongod without su. does anyone know where the mongo conf file is typically located on OSX when installed via homebrew?
[18:45:41] <cheeser> paperziggurat: /usr/local/etc
[18:46:45] <paperziggurat> thanks cheeser!
[18:47:05] <cheeser> np
[19:05:12] <jamieshepherd> So I'm using Node, connecting to a Replica Set, getting Cannot read property 'collection' of null, wondering where to start
[19:05:24] <jamieshepherd> When using single instance though, this was working fine
[19:19:50] <jamieshepherd> The weird thing is.. the connection string with /admin works
[19:19:56] <jamieshepherd> but my own database does not
[19:20:00] <jamieshepherd> e.g. /mydatabase
[19:20:05] <jamieshepherd> I'll get authentication failed
[19:20:13] <jamieshepherd> but my admin user is global admin