[03:43:00] <Saul_Jaramillo> hey guys, quick question, im trying to add a shard from a mongos, im getting this as a response "errmsg" : "could not find host matching read preference { mode: \"primary\" } for set r0",
[03:44:07] <Saul_Jaramillo> and if i do show logs in rs0:primary, i get [initandlisten] connection accepted from ... #9 (3 connections now open)
[03:44:53] <Saul_Jaramillo> and of course the read pref in the rs0 is set to primary
[03:45:50] <Saul_Jaramillo> according to -> printjson(db.getMongo().getReadPref())
[03:46:07] <Saul_Jaramillo> any ideas what could be the problem?
[04:04:52] <Saul_Jaramillo> oh nevermind, i figured it out, now im getting this, "errmsg" : "not authorized on admin to execute command { listDatabases: 1 }", when running the sh.addShard command, but if i go traight from the shell i can perfectly connect to the replica with the specified credentials, is there a specific access or role that needs to be added to this user? (current role is root and db:admin)
[04:11:57] <dman777_alter> if I index a string field, will the lookup on it still be slower than a objectId field?
[05:08:52] <Saul_Jaramillo> ok last question, i keep getting this on my logs in the mongos: SHARDING [Balancer] caught exception while doing balance: Server's sharding metadata manager failed to initialize and will remain in this state until the instance is manually reset :: caused by :: unable to resolve DNS for host ... <-- mongos and configsvr are running on docker containers in aws, the replica set is running at
[05:08:52] <Saul_Jaramillo> home in a windows machine (testing only), any ideas whos missing the dns entry or what to do?
[10:37:22] <amcsi_work> if I index ref_column.$id, is that valid?
[10:37:52] <kurushiyama> dddh: It is not exactly a new article.
[10:38:42] <dddh> kurushiyama: they didn't know about NoSQL ..
[10:45:00] <Sagar> Hello, i am using mongodb and i am running a mongodb query to count db entries using foreach but its taking a lot of time and making pages not to load almost. Anyway i can fast foreach query and make things go alot more faster?
[10:45:06] <Sagar> want to add up the index assuming every user database has a $user['click'] = and i want to add them up foreach $userlinks as $links { $total += $links['click'] and so on...
[10:45:23] <Derick> Sagar: a foreach to do a count means reading all the data
[10:45:30] <Derick> you can just use the "count" command
[10:45:58] <Sagar> Derick: i want to add up the index assuming every user database has a $user['click'] = and i want to add them up foreach $userlinks as $links { $total += $links['click'] and so on.
[10:54:30] <kurushiyama> Sagar: As said, I can barely read this, but it looks to me like a client side aggregation. If said aggregation is done on the MongoDB side, a $sort stage would do the trick. But again – Derick for sure will be more helpful here. It was just something I noticed.
[10:55:20] <Sagar> Yes. i am waiting for him. The reason i chose MongoDB and not MySQL was for scalability and dynamic updates that can easily come up in mongodb.
[10:59:30] <kurushiyama> Sagar: Maybe it would help if you described in non-technical terms what kind of aggregation(s) you are trying to achieve here. A sample doc and the expected output would not hurt, either.
[11:00:27] <Sagar> Kurushiyama: I am trying to count clicks of users. For each unique link they send their users from social media, the click is udpated on their link in the db and a tracker for userip is also inserted.
[11:01:14] <Sagar> So its goes like, u visit a unique link and then the click is updated in the field. { clicks: 1 } and then i have to show on their dashboard who are the top then click givers by using foreach
[11:25:24] <Sagar> Derick: you were going to say the same thing?
[11:25:28] <Derick> Sagar: you're also running an extra query for *every* user, in line 52. That sounds like a relational anti-pattern to me
[11:25:55] <kurushiyama> Sagar: It is one of the most powerful features of MongoDB. Even without replication and scalability, the aggregation framework alone would make MongoDB a good choice for a lot of things.
[11:27:05] <kurushiyama> Sagar: And you can always come to this friendly channel for assistance.
[11:32:12] <Sagar> Derick: the line 52 verifies if the user has links or not
[11:32:52] <Sagar> kurushiyama, Derick: What i do now, please help me with making an easy to use and fast script, what and where should i make modifications?
[11:35:01] <Derick> Sagar: you need to show your data documents first
[11:35:10] <Derick> I don't know how they look like...
[11:36:04] <Derick> a document from both the user and user_links collection please
[11:38:15] <Sagar> Document from user_links: http://pastebin.com/SFWMpqeb
[11:40:09] <Sagar> Document for users: http://pastebin.com/eh8JiMy1
[11:40:19] <Derick> Sagar: a few tips about that one, store date in ISO8601 format (YYYY-mm-dd) (or a MongoDB), and timestamp and generated as a MongoDate
[11:40:20] <Sagar> This is how i want my script to work:
[11:40:52] <Derick> and in that one, dob in ISO8601, and timestamp and registered in MongoDate again
[11:40:53] <Sagar> Derick: i am using MongoDate in php and both date() as well
[11:41:13] <Derick> Sagar: that's not what you store in your documents though. You're storing strings.
[11:41:16] <Sagar> the reason to stoy that way date or dob is their is simple use so i dont want to fetch them
[11:41:41] <Derick> fetch them? I don't understand what you mean
[11:42:03] <Sagar> "timestamp": "2016-01-30T18:36:14.245Z", (It uses new MongoDate())
[11:42:10] <Sagar> basically what i want my script to do
[11:42:17] <Derick> you probably also should not regnerate uids sequentially yourself but use the default objectid
[11:42:48] <Derick> ok, I missed the the timestamp was like that, because you're not showing documents through the shell, but I suppose thorugh json_encode one?
[11:43:10] <Sagar> 1) Find users which has status approved. 2) Then check if user has generated links or not. 3) If generated, get all links of users, and add all of the users links "clicks" as total. 4) then arrange them in decending order
[11:43:43] <Sagar> Derick: showing throught shell? i fetch data using php like u saw in my script
[11:43:57] <Derick> Sagar: will only approved users have links?
[11:44:14] <Derick> so step 1 isn't necessary then
[11:44:26] <kurushiyama> Sagar: I'd do it just the other way around. Do the aggregation over all clicks, save it into a new collection using the out stage, and for the top X, find the according users which are approved
[11:44:34] <Sagar> it is, because user status can be approved,banned,verified,pending
[11:44:34] <Derick> kurushiyama: I was getting there ;-)
[11:44:49] <Derick> Sagar: you just said only approved users have links
[11:45:34] <Sagar> Derick: yes only approved users have links, but it depends if they had generated links or not, its not complasory they have generated a link
[11:45:52] <Derick> Sagar: sure, but in that case, do you want these users in the list at all?
[11:46:52] <Sagar> so first we cannot fetch $Mongo->users->find(); we want to make $Mongo->users-find(array("status"=>"approved")); because only approved users will be able to generate links, once we confirm the user status is approved, we have to check they have generated atleast 1 link
[11:47:29] <Derick> Sagar: yes, but as you say, there is *no* item in user_links for a user, if they are not approved.
[11:47:45] <Derick> so the checking for approved users first isn't needed
[11:48:42] <Sagar> Derick: $Mongo->users-find(array("status"=>"approved")); at first helps to calculate the users which can generate links, if we do $Mongo->users->find(); it will get all users which are not even needed.
[11:50:04] <Sagar> Derick: Line 25 makes it possible get approved users, making sure the item in user_links may or may not have for approved users.
[11:50:29] <Derick> Sagar: i know, I can read the code. But so far, you haven't contradicted this statement yet:
[11:50:33] <Derick> 11:42 <@Derick> but if you query in user_links, you won't find links for users that aren't approved *anyway*
[11:51:10] <Sagar> The reason to get users with approved status because they are the only one with capability of having links. The part could be they can generate links and cannot, so we have to make sure that they have atleast one link.
[11:51:39] <kurushiyama> Sagar: This is well understood.
[11:52:00] <kurushiyama> Sagar: But make a step back for a second. Forget your code. Forget even MongoDB
[11:52:17] <Sagar> So i am doing that part right since i want to first get list of users which has approved status then check if they have a link or not
[11:52:21] <Derick> Sagar: if there is a link in user_links, then the "uid" links to the user that generated the link, right?
[12:13:30] <Derick> so, you can do all of this in *one* query
[12:14:42] <Derick> but with multiple aggregation pipeline steps. Step 1: group all the link clicks into a bucket per user; 2. sort by decending order. 3, lookup the users to go with it
[12:15:02] <Derick> It is best to try this one, by one
[12:18:17] <Sagar> Derick: Help with the script, like i told you i am new to this, i will really appreciate that
[12:47:05] <kurushiyama> synapse: Quick overview: Derick and Sagar were creating an aggregation as replacement for an iteration over all documents client side, and there are in between the stages. Bad thing is that Derick is afk and the code seems to be changed on production.
[12:47:52] <kurushiyama> synapse: And I can not help because of my absence of any PHP skill.
[12:48:14] <synapse> Sagar owns the servers, he was taking advice from someone on IRC and now said person has gone, left him in the lerch and the servers are live?
[12:48:42] <kurushiyama> synapse: Yep, that pretty much sums it up.
[12:48:59] <synapse> Well then I hope Sagar has learned a very valuable lesson
[12:51:04] <kurushiyama> To defend Derick: Sagar only told us that those servers were life after Derick announced his afk
[12:51:42] <Sagar> :| Well i didn't updated anything kurishiyama : all i am saying the issue of iteration is coming, i dont modify codes till the time i can make those work
[12:52:45] <kurushiyama> Sagar: Good luck for you. But you said there were not status updates any more?
[16:08:41] <shlant> hi all. I have mongodump as a cron job and I keep getting emails from cron with output that show this: https://gist.github.com/MrMMorris/9775ab52d223da4825db. Is mongodump sending output to stderr when it shouldn't be? I have it as `mongodump -h bfoo -d bar 1>/dev/null`. Adding —quiet did nothing at all.
[16:08:46] <Sagar> StephenLynx: that "timestamp" is generated by new MongoDate()
[19:08:34] <StephenLynx> nothing to do at my job for nearly 2 months now :v
[19:08:40] <synapse> I've been looking at the source
[19:09:08] <synapse> After taking a look at your code I earned some inspiration out of it. I've started rewriting my project and I'm keeping it procedural and as modular as I can. I just finished writing the first script, mind doing a glance over to critique?
[19:59:40] <StephenLynx> im the only dude here capable of maintaining a vital C project.
[20:00:02] <StephenLynx> and there is a new thing with a client using lynxchan too
[20:00:23] <synapse> But if your permanent contract it's not so bad as every company wants a loyal dev on the team, here in England 'contracting' is very common / popular and people really do come and go in that sector, but the permanent contracts are always the last to ever be affected
[20:01:02] <Doyle> Hey. When a foreground index is being built on a sharded cluster (3.0), I know mongos processes that require a global op like list indexes get stalled, but can those processes block other queries at the mongos level?
[20:01:23] <StephenLynx> synapse, yeah, I am your equivalent of a permanent contract.
[20:01:44] <StephenLynx> here people usually work as if they were a company
[20:01:49] <StephenLynx> of if they were an actual person.
[20:02:03] <StephenLynx> the first gives you more freedom and less security. also less taxes I guess.
[20:02:16] <StephenLynx> the second gives more benefits, more taxes and less freedom
[20:02:29] <synapse> yeah we have a similar set up
[20:02:34] <StephenLynx> you can dismiss a companies services easily
[20:02:41] <StephenLynx> but not so easily with a person.
[20:02:50] <StephenLynx> because there is a fund and shit
[20:03:06] <synapse> In the world of software dev, the "work as a company" is very popular, much harder to get permanent contracts
[20:04:34] <synapse> as here the work as a company option tends to mean you fill 'jobs' on short term contracts, but many times I see "short term" people stay on for years, they get comfy and then bam they're out the door, the upside is they do earn more but it's less secure
[20:05:36] <synapse> Hell at one point here, these short term contracts were sweeping the nation it was so bad as many of them were exploitative contracts, new legislation was needed
[20:06:02] <synapse> Because employers could word things in a way to stop having to pay for sick, holiday, pensions etc etc
[20:06:45] <synapse> The market isn't as crazy as it once was though, it's getting better
[20:12:10] <StephenLynx> there is one thing good about here
[20:12:19] <StephenLynx> is that we had laws to cover that for about
[20:22:41] <synapse> StephenLynx It's crazy to think we didn't have them, but they were never exploited before until recently so there was no real "demand" for the legislation (that's kinda how it works here) if enough people from the right sect of society shouts about something, something gets changed
[20:23:08] <synapse> I emphasize "the right people" too
[22:54:57] <dman777_alter> dddh: on https://docs.mongodb.org/v3.0/tutorial/model-embedded-one-to-many-relationships-between-documents/ example for the first pattern, is the book aray indexed in the publisher document?
[22:55:46] <dman777_alter> sorry, I mean this link https://docs.mongodb.org/v3.0/tutorial/model-referenced-one-to-many-relationships-between-documents/
[22:56:22] <dman777_alter> Is the books: [12346789, 234567890, ...] a multi key indexed array?