Log file Viewer
#mongodb logs for Tuesday the 17th of November, 2015
[00:53:28] <dusky_> I have a json import in a collection, like these: "itens":{ "Sticker | Banana":{ "value":1 }, "Sticker | Bash (Holo)":{ "value":2.46 }, "Sticker | Bish (Holo)":{ "value":2.15 }
[01:00:58] <compeman> dusky_ as i said, use attribute's for each data in your document. this is weird
[01:02:50] <dusky_> look these json, i think it is incorrect and i need to make an attribute like itemName for each item ?
[01:12:00] <dusky_> i have these api thar return a json for me, http://api.ncla.me/itemlist.php, how can i format and add the attribute itemName before each item, example: "item"Sticker | Banana":{ "value":1 },
[01:16:50] <fxmulder_r_u> so I have an index of which isp_domain is the first key, assuming there are less than 50 unique domains, $db->run_command( { distinct => "table", key => "isp_domain" } ); should return pretty much right away no?
[04:29:51] <godzirra> Can anyone help me figure out what's going on? I'm trying to use "mongo --eval" and I'm having a lot of trouble. https://gist.github.com/slooker/85199345eab6d7419790
[04:43:06] <Boomtime> godzirra: bash weirdness, i am not sure what is going on, but it isn't the mongo shell
[05:29:22] <crk> is it viable to implement cookie based retention of my preferred mongodb version for the documentation pages?
[05:30:09] <crk> if this sounds like something that could be done, please direct me to where I must request it officially (bug tracker, request tracker etc.)
[05:32:20] <joannac> rather than just https://docs.mongodb.org/manual/ (which goes to the current version)
[05:32:30] <crk> joannac, more often than not I find myself arriving at mongodb documentation through links from other pages
[05:33:09] <crk> so whenever I get there, I've to set the desired version first, before reading on. forgetting to do so that has led me to trouble a few times :P
[05:33:36] <crk> would it not be smarter if the documentation remember what version I was browsing last, and defaulted to it?
[05:33:56] <joannac> I can see both sides of this. I think it's fraught with danger if I can link you to something and not know if you actually see the same thing I do
[05:44:51] <crk> joannac: thanks for the prompt assistance :) I understand that this is something that needs more thought
[05:47:19] <joannac> yeah, for sure. having to switch versions all the time because one person is on 2.6 so needs 2.6 docs, and another is on 3.0 so needs 3.0 docs
[09:12:02] <kali> dvargek: failover is handled at the replica set level, the sharding stage should barely be aware that something happened
[10:20:17] <dvargek> kali: thanks for the response, I got a weird behaviour, when powering down one of the nodes inside my cluster
[10:21:28] <dvargek> I'm running a cluster with 3 shards, based on 3 replication sets with 3 separate mongos instances
[10:22:09] <dvargek> when I just stop the mongod and mongo-configsrv services on one of the machines, failover works as expected
[10:22:42] <dvargek> but when powering down the same machine, so the ip isn't available anymore, the remaining mongo behaves really slow
[12:58:55] <StephenLynx> mongo doesn't replace relational databases, it just fits different use cases.
[12:59:23] <StephenLynx> if your use case requires a relational database, then you are better using a relational database.
[13:01:41] <StephenLynx> we won't be able to explain it to you, you will have to learn mongodb to see how it fits differently on use cases.
[13:01:59] <StephenLynx> we can't answer you because that depends on information you didn't gave us.
[13:03:55] <Rafibd01717> Actually I see nowadays nosql concept is becoming popular but always wondering what made it so popular over SQL
[13:04:35] <Mattias> Rafibd01717: You don't have to create a schema beforehand, you can just insert data however you want. And it handles JSON natively! So you can search it without problems. Try follow a mongodb tutorial or something.
[13:17:05] <Rafibd01717> should I create a user in "admin" database with "userAdminAnyDatabase" role to enable security in my mongodb server?
[13:17:48] <Rafibd01717> I mean this user will be the super user who can do anything in the mongodb instance?
[13:35:29] <StephenLynx> no, the lack of schema validation is not the only difference that can be beneficial. documents having sub-documents is very useful too.
[13:35:48] <StephenLynx> the ability to add more servers to a cluster without having to turn the db off
[15:51:03] <repxxl> how can i sort multiple mongo cursors by date? i mean i have such a database schema that i cant get all with one query i use multiple collections
[15:55:12] <StephenLynx> since you can't operate with multiple collections at once, it makes no difference on your application how many cursors you are working with.
[15:57:38] <StephenLynx> yeah, at most you could use sub-queries on sql, but your application still only gets one result set.
[16:08:36] <repxxl> can i make the MongoDate less precise ? to be it like the _id timestamp for example i dont need milimicroultra seconds
[16:09:27] <cheeser> though you can normalize them if you want. but the date object will still track those values even if you 0 them out
[16:10:30] <repxxl> cheeser u know because when i sort by _id i getting the correct order but when i sort by MongoDate i just dont get the right order because it is too precise
[16:18:42] <repxxl> cheeser ok so when i will set now my own unique ids to the _id field i dont need to createIndex with unique true right ? its automatically
[16:19:55] <repxxl> cheeser oh i tought i can change the _id so its fixed i cant do anything with it right ?
[16:23:45] <StephenLynx> when a meaningless identification string can be used and you have no other means of identification of something.
[17:14:01] <StephenLynx> I managed to reduce RAM usage on my C code by using chars and shorts instead of ints.
[17:22:26] <repxxl> StephenLynx can i ask you something ? do you use for every ids on your project "the 12 byte long ObjectId" ?
[17:24:49] <repxxl> StephenLynx well, i reading this http://stackoverflow.com/questions/6645277/should-i-implement-auto-incrementing-in-mongodb speciall the answer "42" it starts with "I strongly disagree with author"
[17:25:50] <repxxl> StephenLynx so im concerned in future if my human-friendly unique indexes will not be too large the bytes and ram usage i mean. to do it efficient
[17:26:20] <StephenLynx> it doesn't matter because you can't have documents without that field and that index.
[17:26:46] <yopp> stupid question, is it possible to stepDown the master in "gracefull" way, so no queries are lost?
[17:27:09] <repxxl> StephenLynx i use a random base64 string now but when it will be too long in bytes it will use also more RAM nope ?
[17:27:11] <yopp> Right now I'm seeing that mongos clients are disconnected when master stepped down
[17:28:01] <StephenLynx> as I said, it doesn't matter how much RAM _id uses because you can't not use it.
[17:28:20] <StephenLynx> whatever you implement to substitute it, won't substitute, just add more overhead.
[17:28:50] <StephenLynx> the ONLY scenario where an auto-incremented field can be justified, IMO, is when it isn't global.
[17:29:35] <StephenLynx> and posts have unique auto-incremented id by sub-forum, making it possible for two posts to have the same id, as long as they live in different forums
[17:29:35] <repxxl> StephenLynx yea i got you but wait you use _id like everyone and also a human readable string id like me but the human readable string will take too mutch RAM to get indexed you could only leave with the _id
[17:32:05] <yopp> on the hardware level, there no guarantee that each date sample will be unique, and that it will be the same on all servers
[17:33:18] <repxxl> StephenLynx i have the same for users like you, for my "links" i use _id and a access_id which is something like "RHU5wt2upKMh" a 12 long base64 string which is also indexed
[17:35:10] <StephenLynx> if ALL of the identifiers for something are arbitrary and unreadable, identify by the _id.
[17:36:44] <yopp> StephenLynx, you have sharded mongo, with 2 servers, you're inserted a document. What exact timestamp you will get?
[17:37:46] <repxxl> StephenLynx but if someone is going to share the link on skype or etc looks somekind of crazy the link or ? :D
[17:37:48] <StephenLynx> yopp I am not sure, but I have a guess it will be obtained on a single machine.
[17:40:30] <yopp> repxxl, generally you should not care about the id's in the urls. Nobody cares. Major drawback of the ObjectId is the size. If you have a lot of small records (it's kinda where mongo sucks a lot, even with WT), you might have storage overhead for the _id field and index of the size of the data.
[17:40:48] <StephenLynx> yopp https://docs.mongodb.org/manual/faq/sharding/#how-does-mongodb-ensure-unique-id-field-values-when-using-a-shard-key-other-than-id
[17:41:34] <yopp> StephenLynx, you are still missing the point. ObjectId _can't_ guarantee the order of the records by design.
[17:42:20] <repxxl> yopp and StephenLynx alright i going with the objectid i will use other ids formats really only for human readble ids like "usernames" for example ty for help :)
[17:42:45] <StephenLynx> I can't remember a single time where I needed documents by their insertion order.
[17:44:22] <yopp> you can't reliably work with time on the common hardware even on the milliseconds scale
[17:45:45] <yopp> StephenLynx, it's very complicated task to keep two separate hardware clocks in sync even on ms scale.
[17:48:27] <StephenLynx> but I honestly can't think of a common business scenario where the limitations you pointed are relevant.
[17:51:29] <repxxl> one more question, when i have a field "type" : "real" and "type" : "unreal", now i want to find the "real" documents and i have 1milion documents there are saved only 2 indexes in the RAM since there are only 2 different values "real" and "unreal" or 1 milion indexes will be saved to RAM ?
[17:52:05] <Owner_> if you are connected to a database, but using the oplog, do you only recieve updates for that database?
[18:02:15] <yopp> repxxl, basically, size of that index will be pretty much the same as _id index size
[18:03:27] <yopp> if you'll switch to the boolean value ("real": true | false) you can reduce it in around half
[18:06:21] <repxxl> also i asking myself when people have 2 collections like collection users with user_id field and a users_posts collection with also user_id field basically they share same id references and i have to create 2 indexes for that wich is 2x the size sounds to me also like a nonsense.
[18:06:48] <repxxl> cant there be just created one index and use that for different collections across
[18:07:11] <yopp> StephenLynx, banks are slow, because there not much happening on the regular user account. Ad networks are bloody fast, you have hundreds of 1/1000 ¢ transactions per second
[18:08:27] <repxxl> StephenLynx and for embedded documents and document enlargement i'm not a fan of this because the 16m limit which makes me one day take headache.
[18:09:01] <StephenLynx> why would you even care about that amount of precision on ads to begin with?
[18:09:26] <repxxl> StephenLynx "dont have to index relations ?" what u mean i hope not single document enlargement till one day 16m limit
[18:10:11] <StephenLynx> you have a collection of posts, each post has a field with the id of the user that posted it.
[18:15:07] <yopp> Simple example: you need to stop the ad company immediately when it run out of the budget. You need to track the exact use who was the last paid one. This is small deal for traditional CPV companies, but a huge for CPA
[18:18:25] <yopp> StephenLynx, in CPV, almost nothing, 1/1000th of cent. In CPA companies, it might be hundreds of dollars. Depending on the budget and how they are paid (fixed payment or revenue share)
[18:20:32] <yopp> Let me take a leak, and then I'll explain what the difference between the CPV/CPC and CPA
[18:30:45] <yopp> Ad business is a four way shit: advertisers (pay money), ad publishers (get some money), platform (gets most monies), ad publishers audience (consumes ads)
[18:31:51] <skokkk> by audience you mean victims.. but interesting, listening, would like to know the difference between CPV/CPC and CPA.
[18:33:27] <yopp> But anyway, at the beginning it was simple: publisher got audience, that he can sell. For example 1M of unique views per month.
[18:34:29] <skokkk> StephenLynx, adblock ofc, but sites that force ads and remove their purpose if they detect ads.. then I am a victim.
[18:36:07] <repxxl> StephenLynx why i dont have to index this field ? i mean with index i will get faster the posts of the user if there are also many other user posts or i misundersood something on indexes
[18:36:42] <yopp> Advertiser paid for a fact that ads is shown to this audience. It was not important what will audience do with the ad that this time. It's basically Cost-per-view (CPV), but it's usually called Cost per mil (CPM). Because nobody buys a one view. It's sold in thousands
[18:38:08] <repxxl> StephenLynx yea but is it not a waste ? i mean basically that indexes already exist with the same values in user collection now i have to create the same indexes in posts collection
[18:38:32] <skokkk> yopp, you seem to be very knowledgable in this. How would a platform (such as google ads) prevent bots clicking it 1000x+ on proxies etc..
[18:38:56] <StephenLynx> I am still waiting on his explanation on how a dozen clicks costs hundreds of dollars.
[18:39:23] <yopp> At this point, you usually don't care about precision, because cost of the view is just a fraction of the cent. Basically, when publisher shows a 1M of ads, he will get, let's say $1k. So no point in precise tracking, because potential losses are cents.
[18:39:55] <skokkk> I'm starting to see what he means. You can sell $200 software/hardware and if you have one click out of 12 you will already make a profit.
[18:40:37] <yopp> Then, it became clear, that it just not working. You can spend a fortune, but at the end you will not get any profits from the ads. For example: nobody is buying in your store.
[18:40:40] <StephenLynx> I can't see it. if you expect 12 clicks on a matter of some ms, you expect a shitload over the course of the contract
[18:41:05] <StephenLynx> these hundreds of dollars are still a drop on an ocean of millions of dollards
[18:41:24] <yopp> Then ad business came up with a Cost-per-click. So nobody cares about views anymore. Publisher is get paid when someone clicks on yhe ad.
[18:42:08] <yopp> CPC is more expensive, so you not getting fractions of cents, you are getting cents!
[18:43:13] <yopp> At this point, precise tracking become an small issue. Because you are showing the ads on 1000s of sites at the same time. And it's kinda important to track who will get the last cents.
[18:44:25] <yopp> Because when ad company almost ran out of money, it still will be the page that is already loaded by the user.
[18:46:26] <yopp> It worked for a while, but then internet become a huge thing, and it was like everybody are clicking on the ads (mostly accidentally). And after a while it became clear that CPC is not working as well.
[18:47:57] <yopp> From the ad publisher point of view it's kinda simple: you are getting dollars for the action that user is making somewhere else.
[18:51:30] <yopp> So, you are still showing the same ad to the millions of the people around the world (targeting still sucks), and users are still clicking. And there a simple situation: ad got like last 10$ on the budget. 1M of users are seeing the same ad at the same time, and like 1k of them are clicking at the same moment.
[18:52:56] <yopp> So you need to keep a precise track of the order of clicks. to decide who will get a 10$.
[18:53:27] <StephenLynx> doesn't the clicks go trough a server that redirects the user to the real URL?
[18:54:18] <StephenLynx> and if we are talking about a deployment so large that we might get 1M of users seeing an ad at the same time
[18:54:42] <StephenLynx> how large is the budget of the person publishing the ad in the first place?
[18:55:32] <StephenLynx> so if this 1k users do this action on the advertised site, it was 10k in ads that was paid.
[18:59:18] <StephenLynx> it doesn't matter if they cook between click and action for as long as they do if they complete every X ms.
[19:01:48] <yopp> StephenLynx, it's not about how often they do the action, it's a silly way of tracking who will be paid: first one.
[19:04:15] <StephenLynx> I can see why its stressing to deal with petty people that makes money in a petty way.
[19:06:05] <StephenLynx> it was not that bad, but even him saw how it was silly to trust people to see ads when he used an ad blocker himself.
[19:14:28] <yopp> skokkk, yeah. And the fact that nobody can click or view ads on the 1000/s rate :B
[19:15:31] <skokkk> yopp, now I'm starting to feel bad for always pressing the ad (promoted result thingy) for teamviewer instead of the teamviewer (first result) in google XD
[19:17:43] <yopp> Mostly, all fraud protections systems are build on the outliers detection. When something doesn't fit the "baseline" profile. I'm not expert in that field, my job is to count the monies :)
[20:11:02] <repxxl> is it a good idea to do one collection and have inside something like field Doc_Type : Account , Doc_Type : Post , Doc_Type : Follow so i will have one big collection i could easy query everything without the needs to combine referenced information on the application side
[20:14:08] <repxxl> cheeser i will generally stick with my schemas because i can't allow that document grows behind 16mb, i will just put them in one collection instead of multiple and separate them by this doc_type for example so i can query them better and more efficiently ...
[20:56:51] <d-snp> our cluster just stopped accepting writes, we get this error on one of our collections:
[20:56:54] <d-snp> errmsg: \"requested shard version differs from config shard version for app.requests, requested version is 0|0||00000000000000...\", $gleStats: { lastOpTime: Timestamp 0|0, electionId: ObjectId('564b4199e8aecf48c7ea7db0') } }", "code" : 10429, "shard" : "shard5"
[21:05:19] <bz-> if i want to allow people (an app admin) to update a document arbitrarily, such that the values may change to what ever, so that it can'tt be queried for consistently to identify for the update, should i be using the object id to identify the specific document within the app .. at all times?
[21:20:37] <repxxl> can i somehow control the position of updating a field that does not exists like in the middle of already existing fields ?
[22:17:17] <GitGud> hey. I wanted to know the standard way to do a particular thing. i will be having a database of posts made by a bunch of users that need to be sorted at all times chronologically. and there will be a query from the front page of my webpage that lists the 4 most recent user posts. now my question is what is the most efficient way to sort them and index them and preserve the sorting in the index. and then making sure the query for the first 4 posts
[22:25:57] <deathanchor> hmm... so the answer is yes it works now: https://jira.mongodb.org/browse/SERVER-2677
[23:24:35] <fxmulder> anyone know why this query is taking a long time? http://nsab.us/public/mongodb
[23:26:24] <Boomtime> fxmulder: try increasing loglevel to 1 or 2 on the server and observing what the server says about that command
[23:48:25] <fxmulder> if I set it to 3 I can see that it is doing the query but beyond that is just noise