pmxbot IRC Log Viewer

[10:59:06] <jheg> hi all

[12:38:49] <m3t4lukas> hey guys :) Do you know a way to debug morphia aggregation? Like extracting a JSON array from it or stuff like that so I can test it in the console client?

[12:40:57] <m3t4lukas> Another thing I want to ask is whether anyone in here knows when the results of the last wave of exams is going to be published? Please excuse my impatience, I'm excited about it :P

[13:07:01] <m3t4lukas> there is a mistake in the documentation here http://mongodb.github.io/morphia/1.1/javadoc/org/mongodb/morphia/aggregation/Projection.html#projection%28java.lang.String,%20java.lang.String%29

[13:08:12] <m3t4lukas> doc says the second parameter is the new name. In reality the first parameter is the new name and the second parameter is the name of the field before the stage

[13:31:37] <cheeser> m3t4lukas: in reality what now?

[13:32:38] <cheeser> hrm. you might be right. that'd be a bug. i'll have to write some tests.

[13:37:33] <m3t4lukas> cheeser: everythig works for me as it's intended when I switch out the parameters

[13:37:55] <cheeser> definitely a bug then.

[13:40:58] <m3t4lukas> or just wrong doc

[13:41:32] <cheeser> perhaps.

[13:44:26] <m3t4lukas> is it still true that count with filter is slow?

[13:45:17] <m3t4lukas> (if there is an index for the filter)

[13:45:27] <m3t4lukas> *when

[13:48:43] <cheeser> not sure on that one

[14:14:58] <cheeser> m3t4lukas: actually, looking at code/docs, i think it's correct just a bit awkwardly phrased...

[14:15:38] <cheeser> the first param is the name of the field created by the projection and the second param is the source of the value to fetch

[14:16:10] <cheeser> the overloaded versions have more complex value sources, e.g.

[14:24:35] <m3t4lukas> cheeser: okay, I think "projectedField - the new field name" pretty much says that this parameter takes the name of the field after renaming by the stage

[14:24:52] <cheeser> yeah. that's poorly named.

[14:25:01] <cheeser> i'll update the docs

[14:25:18] <m3t4lukas> okay. Anyway it all works properly when I swap out stuff

[14:25:24] <m3t4lukas> thank you :)

[14:26:28] <cheeser> https://github.com/mongodb/morphia/issues/878

[14:26:39] <cheeser> the order won't change. i'll just clarify the docs.

[14:30:55] <m3t4lukas> cheeser: that's great. So it won't break code

[14:31:13] <cheeser> right

[14:33:07] <m3t4lukas> cheeser: what about an overload of the toString() function for AggregationPipeline just like the one for the classes implementing Query? Stuff like the previous error could be found way faster that way

[14:33:21] <cheeser> is there not one now?

[14:34:00] <cheeser> oh, hell. actually i think there is a bug there.

[14:34:20] <cheeser> i'll have to write some tests to validate this.

[14:39:20] <m3t4lukas> unfortunately there is none now :/

[14:42:25] <cheeser> there's some but there definitely could be more

[14:44:11] <m3t4lukas> ah, okay. I just put it into the log and wondered why it just tells me the fully qualified name of the class and the address of the object

[14:44:37] <cheeser> for Projection?

[14:45:22] <pilva> Hi, i moved some data form postgres(~800mb) to mongodb and now i get 10gb database on mongo? is that normal?

[14:45:25] <pilva> 1 collection

[14:45:56] <cheeser> pilva: possibly, yeah. mongo preallocates diskspace so some of that is what you're seeing.

[14:46:08] <cheeser> there are also indexes to consider

[14:46:32] <StephenLynx> and it also depends on your db version

[14:46:53] <StephenLynx> some compress data better than others.

[14:47:05] <cheeser> mongo currently is as space-friendly as it could be because it stores document key names over and over

[14:47:16] <cheeser> currently *isn't*

[14:47:36] <StephenLynx> yeah, rdbs have that advantage, you don't need to store key names.

[14:47:36] <pilva> cheeser: by this way I will full my disk really fast

[14:47:50] <pilva> mongo 3.2

[14:47:58] <pilva> ubuntu 14

[14:48:02] <StephenLynx> maybe not. depends on how much of that space is pre-alocated.

[14:48:03] <StephenLynx> also

[14:48:11] <StephenLynx> what is your data model?

[14:48:18] <StephenLynx> how was it imported?

[14:48:38] <StephenLynx> it might be doing some extreme data duplication to compensate for not having relations.

[14:49:38] <pilva> its not complex, 10 x string <=100 and 1 datatime

[14:49:47] <StephenLynx> show me.

[14:52:33] <m3t4lukas> cheeser: yep, for Projection

[14:52:50] <m3t4lukas> cheeser: sry, for ProjectionPipeline

[14:53:37] <cheeser> file an issue and i'll take care of it.

[14:53:42] <cheeser> por favor. :)

[14:54:01] <m3t4lukas> cheeser: better said the ProjectionPipeline returned by Datastore.createAggregation(class);

[14:54:23] <cheeser> isn't that AggregationPipeline ?

[14:54:32] <m3t4lukas> cheeser: I'll take care of it as soon as I got some time. Which will probably be right after my current project.

[14:54:59] <cheeser> ok. i'm fighting debian package building so i have my hands full, too. :D

[14:55:13] <StephenLynx> change to centOS :v

[14:55:24] <cheeser> already have those packages built ;)

[14:55:29] <StephenLynx> ah

[14:55:34] <StephenLynx> nvm

[14:57:30] <m3t4lukas> cheeser: are you working at 10gen? and if yes, why don't you just let a dedicated package do that stuff? They are a special kind of people :P

[14:57:43] <m3t4lukas> *packager

[14:58:56] <cheeser> i do, yes. and i'm actually working with a contractor on this .deb package. but I still need to understand what he's doing so we can maintain it once he's gone.

[15:00:51] <m3t4lukas> that sounds kinda rational. Have fun ^^

[15:02:26] <cheeser> "kinda" yeah. :)

[15:31:07] <m3t4lukas> cheeser: yeah, it's rational for a paranoid oriented company :P

[15:31:49] <m3t4lukas> If I had to do packaging I'd just ask someone else if the one who made it for me went on

[19:28:35] <bmanturner> Hey everybody. I have an implementation question. Does anyone have a moment?

[19:33:28] <bmanturner> I am building a timeline using timestamped documents. If a user creates a large amount of these documents in a short amount of time, I would like to reduce noise by only returning one of those documents.

[19:34:18] <StephenLynx> findOne

[19:35:45] <bmanturner> Let me clarify: I want to return 10 documents to populate a timeline and they are sorted by a timestamp. If, say, 8 of these were created by a single user over a short period of time, I would like to only retrieve one of those instead of 8, so that the remaining 7 documents are what would normally follow.

[19:36:30] <StephenLynx> so you want to specify intervals and return only one document per interval?

[19:37:02] <bmanturner> Per user, and per another attribute on the document, yes.

[19:37:30] <bmanturner> I'll prepare a pastebin, in case it helps.

[19:37:39] <StephenLynx> no, I get it.

[19:37:55] <bmanturner> Okay.

[19:38:07] <StephenLynx> you can either pre-aggregate and solve this issue on insert-time

[19:38:13] <StephenLynx> or

[19:38:19] <StephenLynx> hm

[19:38:27] <StephenLynx> use aggregate with a group

[19:40:09] <bmanturner> So, in my instance, users are logging their high scores on arcade machines. Each machine has an ID, and scores hold a reference to a userId and a machineId. If a user logs several scores on a single machine within a specified interval, I would like to only return the highest score logged in that interval.

[19:40:18] <bmanturner> Is aggregation still the best solution?

[19:43:12] <StephenLynx> oh that

[19:43:34] <StephenLynx> use group to it will group by user and return only the highest score for said value.

[19:43:42] <StephenLynx> yes, aggregation will work just fine.

[19:43:52] <StephenLynx> like

[19:44:18] <StephenLynx> _id:'$user', score:{forgot the operator:'$score'}

[19:44:40] <StephenLynx> now you have a list where _id is the user and score is his highest score

[19:44:48] <StephenLynx> that on the group operator

[19:45:15] <StephenLynx> you can apply a $before so you only get scores for a certain machine

[19:45:32] <StephenLynx> a $match before*

[19:46:07] <bmanturner> That sounds great. Now to figure out how to do that in Meteor. Thank you for your help.

[19:46:56] <StephenLynx> I strongly suggest you just use the native driver.

[19:47:02] <StephenLynx> meteor is bad.

[19:47:34] <StephenLynx> not only the project itself is an incredible rat nest, but the documentation is poor so as performance.

[19:48:05] <bmanturner> I have about a year's experience using it and I think this will likely be my last project with it, but I'm set on it for this.

[19:49:14] <bmanturner> Looking towards Phoenix or Express after this.

[19:49:22] <StephenLynx> express is crap.

[19:49:30] <StephenLynx> that I know for sure.

[19:49:38] <StephenLynx> even the creator jumped ship.

[19:49:47] <bmanturner> hehe

[19:50:10] <StephenLynx> and can you link me anything to phoenix?

[19:50:26] <bmanturner> http://www.phoenixframework.org/

[19:50:32] <bmanturner> Built on Elixir (Erlang)

[19:51:29] <bmanturner> A lot of Meteor users moved to or are thinking of moving to Phoenix

[19:52:02] <StephenLynx> HAHAHAHAAHAHAHAHAHAH

[19:52:08] <StephenLynx> THAT phoenix

[19:52:16] <StephenLynx> I thought you were talking about node

[19:53:53] <bmanturner> Nah. I'm a Junior dev looking to get a lay of the land, so I'd rather learn something new like Phoenix then box myself into a certain stack.

[19:54:25] <StephenLynx> you are doing exactly that, though.

[19:54:31] <StephenLynx> you are boxing yourself with a framework.

[19:54:49] <StephenLynx> if you really want to get out of the box, you want something that decouples FE from BE

[19:55:20] <StephenLynx> these will just keep you from actually learning how to write web apps.

[19:55:24] <bmanturner> Phoenix is fe agnostic? I was planning on using React/Redux on the FE

[19:55:39] <StephenLynx> >Phoenix is a framework for building HTML5 apps,

[19:55:45] <StephenLynx> it is built to be couples with FE

[19:55:49] <StephenLynx> that is the point.

[19:56:20] <StephenLynx> what if you want to build a B to some mobile app?

[19:56:22] <StephenLynx> BE*

[19:56:43] <StephenLynx> nope, you are boxed with a stack that is coupled with web FE.

[19:56:56] <bmanturner> Cordova and webpack?

[19:56:59] <StephenLynx> no

[19:57:00] <StephenLynx> just node.

[19:57:11] <StephenLynx> it gives you a relative low-level access to HTTP.

[19:57:16] <StephenLynx> so you can build anything to anything.

[19:57:30] <StephenLynx> web,mobile, desktop, dildos :v

[19:57:33] <StephenLynx> its just http.

[19:57:53] <StephenLynx> frameworks are for code-monkeys.

[19:58:04] <StephenLynx> if you keep using them, you won't be more than that.

[19:58:26] <StephenLynx> sure, if you just want to throw some quick hack and could have done it raw, no problem.

[19:58:32] <bmanturner> Super close to the metal, there. As I don't get to use this stuff for work and can only use these things for hobby projects I need something that makes singe-man development a little more manageable

[19:58:34] <StephenLynx> but if can't go without them, thats bad.

[19:58:40] <StephenLynx> not close to metal.

[19:58:47] <bmanturner> *single

[19:58:52] <StephenLynx> I can't see the issue.

[19:59:02] <StephenLynx> gitgud.io/LynxChan/LynxChan

[19:59:08] <StephenLynx> 1 man dev, less than a year.

[19:59:11] <StephenLynx> running smooth as butter.

[19:59:39] <StephenLynx> is up to the developer to make development manageable, not the tool.

[19:59:53] <StephenLynx> gotta go

[19:59:57] <bmanturner> Less than a year versus a couple months

[21:04:35] <ohlol> hi, is it possible to dump a sharded cluster and restore to a standalone instance?

[21:05:02] <cheeser> if you turn off the balancer that should be fine, i'd think.

[21:06:13] <ohlol> i can dump from mongos?

[21:06:21] <ohlol> i tried and it just dumped admin and config

[21:06:52] <ohlol> i didn't do anything w/balancer though

[21:06:53] <cheeser> did you give it a db/collection to dump?

[21:07:08] <ohlol> nah, i have almost 100 databases

[21:07:23] <ohlol> actually make that almost 300

[21:07:55] <ohlol> i assumed it'd dump all without an argument

[21:08:14] <cheeser> on a mongod it would

[21:08:53] <ohlol> should i mongodump from the mongos or each mongod shard?

[21:09:34] <cheeser> if you did it from each shard you'd only get the data on that shard then you'd have to merge it...

[21:09:45] <cheeser> have you considered using copyDb() ?

[21:10:04] <cheeser> https://docs.mongodb.org/manual/reference/method/db.copyDatabase/

[21:10:16] <ohlol> hmm

[21:10:35] <ohlol> it's not clear to me how that would help

[21:11:13] <ohlol> i have three shards, i want to dump all of the data and restore to a single instance

[21:11:27] <ohlol> that is not in the cluster

[21:13:04] <ohlol> https://gist.github.com/ohlol/9c271b37ade465b264fd when i run mongodump, it creates a directory per db but only config and admin have any data

[21:13:27] <ohlol> that is from running mongodump on the mongos

[21:16:13] <ohlol> i find it difficult to believe this has never been a thing

[21:26:17] <cheeser> as i mentioned, give it a database name to dump

[21:26:57] <cheeser> oh, i see what you're saying.

[21:27:44] <ohlol> so upon further inspection, i've discovered that the dbs are not partitioned. i tried dumping them directly from the shard primary but there is no bson

[21:28:46] <cheeser> copyDatabase() doesn't work for you?

[21:29:14] <ohlol> the instances are not on the same network

[21:29:32] <cheeser> is one network visible from another?

[21:29:44] <ohlol> nah unfortunately

[21:31:09] <ohlol> shouldn't this work? https://gist.github.com/ohlol/032776b3773eb8a42868

[21:32:21] <cheeser> and that's a mongod?

[21:33:07] <ohlol> aye

[21:33:17] <cheeser> should yeah

[21:33:35] <ohlol> weird

[21:35:35] <ohlol> i set up an ssh tunnel and got copyDatabase working

[21:35:46] <ohlol> i'll just write a script to run that

[21:41:49] <ohlol> looks like it's working, cheeser. thanks for the help

[22:10:18] <charis_> Hi all, can someone help me with replication set issues? "errmsg" : "exception: need most members up to reconfigure, not ok : test.com:27017", "code" : 13144. I get an error when adding a replica set with ssl turned on. I can connect to it by specifying --ssl in the command line. Is there any way to add ssl to rs.add() ?

[22:11:28] <charis_> mongo version 2.4.14

Log file Viewer

Help | Karma | Search:

#mongodb logs for Tuesday the 5th of January, 2016