Log file Viewer
#mongodb logs for Friday the 22nd of April, 2016
[02:11:24] <Troy> I have a list of guid's stored on a document and i want to say give me all records where a guid is part of that documents list of guids. Doable?
[09:38:11] <Ange7> hey all, from https://jira.mongodb.org/browse/DOCS-4999 how is it possible to set socket mongodb with 777 permissions by default ? thank you :)
[10:02:00] <Derick> Ange7: you can use --filePermissions arg as argument to mongodb, but it's also just a config option: https://docs.mongodb.org/manual/reference/program/mongod/#cmdoption--filePermissions
[10:03:10] <kurushiyama> Albeit more often than not, it turns out to be a bad idea to have your application running on the same machine as the mongod.
[10:04:12] <Derick> Ange7: and see https://docs.mongodb.org/manual/reference/configuration-options/#net-options for the YAML config file sttyle for it
[10:04:35] <Derick> kurushiyama: true, but ok for dev, or talking to mongos that is local to say your app server
[10:08:20] <kurushiyama> Derick: Hm. I wonder if it is noticeable, especially given the connection pool maintained by a lot of drivers which eliminates the handshake overhead a great deal by preallocation. But ofc you are right, and for maxing out performance it surely makes sense.
[10:13:13] <kurushiyama> Derick: Now you have me. So you basically say that in case you have a mongos on your app-server, unix sockets should be the preferred communication, which basically means that this setup would be best practise performancewise?
[10:18:37] <Ange7> how is it possible, i have a collection with 608 documents, i try : db.myCollection.findOne({domain: 'www.google.fr'}) (in mongoShell with mongo 3.2.5) took 1/2 minutes... ? and i have an index on domain field.
[11:16:48] <stava> I have a Project model which contains one ProjectType and one ProjectStatus model. ProjectType contains an Array of ObjectIds for allowed ProjectStatuses. Can I somehow create a constraint in Mongoose or MongoDB to validate that, when creating or updating a Project, that the provided ProjectStatus is allowed by ProjectType? Ideally it should not be possible to remove an allowed ProjectStatus
[11:25:19] <kurushiyama> stava: You are trying to put too much logic into the database, here. You do not want allow to remove a used status?
[11:25:26] <stava> kurushiyama: Of course not. The requirement is that projects should be modelled after project types, where you configure facts about the projects that inherit the type
[11:26:01] <kurushiyama> stava: you do not want to allow to remove statuses which are in use, right?
[11:27:44] <kurushiyama> stava: Either way: That is logic which is not put into MongoDB, but your application.
[11:29:00] <stava> I get that. When creating or updating a project, I must fetch the relevant project type and validate the provided status against it?
[11:29:27] <kurushiyama> stava: Well, as far as I can tell from your desciption, this sounds about correct.
[11:30:30] <stava> So a document structure (or "schema") like Project {type: ObjectId, status: ObjectId}, ProjectType {statuses: [ObjectId]} is fine?
[11:32:16] <kurushiyama> stava: I would not do it that way. But then, I would not use Mongoose. The problem is that Mongoose kind of suggests (depending on how you see it) that MongoDB is sort of a limited RDBMS or an object store. Both views are good ways to get you into real trouble when you try to scale.
[11:33:08] <stava> I'd really like to hear how u'd do it, if u have time and motivation to explain it to me :p What do you prefer instead of Mongoose? Using the mongodb package for node directly?
[11:34:20] <kurushiyama> stava: The thing is that with an RDBMS , you identify your entities, their properties and relations, derive the models from that and then bang your head against the wall to get your JOINS right.
[11:35:24] <kurushiyama> stava: The proper way of data modelling with NoSQL in general and MongoDB in particular reverses the process.
[11:36:28] <kurushiyama> stava: You identify your use cases first. Then, you identify the data you need for those use cases. Then, you model your data so that you can query said data required in the most efficient way.
[11:37:19] <stava> So that the documents in mongodb represent pretty much the data you are actually using? Composing and organizing the data in the right way should be done on the application side when doing create/update operations?
[11:38:38] <kurushiyama> stava: As per your example: You'd need to have a second query for your project status when all you want to do is to display a project.
[11:39:09] <kurushiyama> stava: Data modelling is _a lot_ less straightforward than it is with RDBMS.
[11:39:40] <kurushiyama> stava: You have to make decisions. A good rule of thumb: If in doubt, make the more common use case the most efficient.
[11:40:33] <kurushiyama> So, to stay within your example: The more common use case will be to display a project, I assume?
[11:41:15] <kurushiyama> So, have the status explicit: {_id:someId, projectName: "Foo", Status: "NEW",…}
[11:42:40] <stava> The reason why I wanted the status in a separate document though is because a status consists of multiple fields, most importantly "state" and "label", where state is a general constant that is recognized by the system ("new", "complete" etc), and the label may be user defined
[11:43:53] <kurushiyama> Now, with the explicit status, you can display them right away, without additional queries, right?
[11:44:40] <kurushiyama> Now, the user selects a project, we have a _known_ status, for which you can query the details then
[11:45:46] <stava> are you saying we store both the status id, and the label (which is the common use case to display) on the project?
[11:47:12] <kurushiyama> stava: But you are right, ofc you could have the label and the id, that'd be your decision.
[11:49:25] <kurushiyama> stava: But you asked. And you seem to understand. "You have made your first steps into a greater world!"
[11:49:39] <stava> there are two reasons why i want to do this with mongo really, one is that it integrates easily with nodejs, and the other is that its supposedly very fast
[11:50:20] <kurushiyama> stava: Well, then strike Mongoose and put _a lot_ of effort into data modelling.
[11:51:21] <kurushiyama> stava: As per speed: With the redundancy in data we discussed above, you reduced what would be basically a JOIN (in fact it would be multiple queries) to a simple query
[11:53:35] <kurushiyama> Oh, and just to make sure: http://blog.mahlberg.io/blog/2015/11/05/data-modelling-for-mongodb/
[11:56:49] <kurushiyama> stava: That's actually a _very_ good metaphor for data modelling in MongoDB!
[11:58:25] <kurushiyama> stava: "If in doubt, document points in time. As shallow documents in an own collection."
[12:00:05] <kurushiyama> stava: The reason behind this is that you can always use the aggregation pipeline to extract data from it. Conversely, do not preaggregate data too much.
[12:02:08] <stava> thats an entire subject i didnt read up too much on, and perhaps its naive of me to start modelling my data before i know these subjects :p aggregation seems like a kind of JOIN or UNION
[12:02:39] <kurushiyama> stava: As per your example, you'd log working hours as {_id: someId, start: someISODate, end: someISODate, person: dependsOnYourModel, project...}
[12:03:46] <kurushiyama> stava: the aggregation pipeline is a killer feature. You should _really_ learn it!
[12:04:47] <kurushiyama> stava: Which brings us to a another rule of thumb: "If in doubt, prefer the aggregation pipelin over m/r."
[12:08:26] <kurushiyama> stava: Next RoT: "Know what the different levels of writeConcern, readPreference and readConcern do, and know it well."
[12:10:18] <kurushiyama> stava: Which is _extremely_ important in terms of consistency and durability. There are no transaction. Atomicity is only guaranteed on document level out of the box".
[12:10:46] <stava> sounds like i'll have constant "race conditions" or whatever its called, when two processes wants to modify the same document
[12:12:51] <kurushiyama> stava: Hint: ask smart, as in http://www.catb.org/esr/faqs/smart-questions.html
[12:13:55] <kurushiyama> stava: I know this sounds rude. But tbh, I read this essay every once in a while to remind myself.
[12:15:31] <stava> its not rude, i'm just responding with questions, even if i can google them, because i wouldnt feel polite if i didnt respond at all lol
[12:17:05] <stava> it is obvious to me that im rushing things, i need to take time to understand how mongodb actually works and get into the nosql mindset
[12:18:27] <Ange7> kurushiyama: « well, is the server under load » no, i don't think ? but one « findOne » cmd on collection with 608 document which take 4min.... it's very slow
[12:18:56] <kurushiyama> stava: That is a very good approach. If I can help you, do not hesitate to ping me here.
[12:20:20] <kurushiyama> Ange7: Well, there obviously is a problem with your server, as my heavily loaded, 6y old MBA rushes through my test sets of more than 1M docs ;) Do you do those queries from the shell?
[12:23:39] <kurushiyama> stava: No. Never read a book on MongoDB. The docs are extremely well written to get you a jumpstart. After that, you might want to get to school again at university.mongodb.com
[12:27:13] <stava> Yeah, that comes naturally for me though, as I prefer doing dev in linux environments :p
[12:28:07] <stava> for some reason i managed to install an old 2.0 version of mongo, to be rectified shortly
[12:29:55] <Ange7> I was in TCP some days ago, and i passed in unix socket but i had the same problem
[12:31:06] <kurushiyama> Ange7: Well, we need to debug somehow. There is something seriously wrong if a 4Gb MBA with a 1.6GHz CPU beats your monster machine, no?
[12:34:00] <kurushiyama> Ange7: And hence I want to eliminate the differences one by one. First stop: use TCP, and if it is just for debugging. Just want to make sure.
[12:36:58] <Ange7> how to be sure i'm using TCP or Unix Socket ? cause my scripts connect with unix socket, but the mongo shell don't use my scripts ... lol
[12:37:49] <kurushiyama> Ange7: Well, we have to find out. Honestly, I have never used sockets. Gimme a few, need to finish a mail.
[12:57:04] <Ange7> The WiredTiger cache is only one component of the RAM used by MongoDB. MongoDB also automatically uses all free memory on the machine via the filesystem cache
[13:00:04] <kurushiyama> Ange7: In general, MongoDB uses between 85 and 90% of the physical memory. It is said to be 85%, but I have observed up to 90% in some cases. I assume aggregations or map/reduces
[13:03:52] <Ange7> but when i start mongo i don't have warning about numa but it's not started with numa too ?
[13:47:58] <gain_> hello to all, I'm new to mongo and I need to know if there are features to manage the order of documents inside a collection
[13:49:46] <gain_> kurushiyama: as noob, the first method I've though to is to add an "order" field and use it to sort, but I need to know if that's already managed by mongo in order not to add useless data in my documents...
[13:50:40] <kurushiyama> gain_: Explicitly not. Documents are guaranteed to never be fragmented, but their order is totally arbitrary.
[13:52:22] <kurushiyama> gain_: Usually, you'd use an existing field. Say you want to oder appointments, and you'd want the 10 most current ones, you would do sth like db.appointments.find({}).sort({date:-1}).limit(10)
[13:54:26] <kurushiyama> gain_: Well, the order is not as arbitrary as it seems, but enough so that you can not rely on it.
[14:02:35] <kurushiyama> Ange7: well, I have several nodes with a lot bigger databases and only half the RAM
[14:04:49] <kurushiyama> Ange7: Well, time to dig into it: Can you show the output of db.serverStatus(), db.stats() for the db in question and the config file? Please use pastebin
[14:21:58] <kurushiyama> Well, make sure there is a daemon running, otherwise your time can start to differ again.
[14:23:26] <kurushiyama> Ange7: Well, you need to configure it correctly: https://help.ubuntu.com/lts/serverguide/NTP.html Can not help you much more with that, since I do not use or wish to use Ubuntu.
[14:27:33] <kurushiyama> Derick: And local time is shown as "localTime" : ISODate("2016-04-22T13:57:39.340Z"),
[14:30:51] <kurushiyama> Hm, looking deeper into it, the acquireCount and acquireMicros seem to be quite high.
[14:36:56] <kurushiyama> Simplified: Operations require locks, as for WT it is document level locks. acquireMicros denotes how many microseconds were spent while waiting for locks, acquireCount is how often your opertions had to wait for locks and micros/count denootes how long each op waiting for a lock had to wait until it was granted.
[14:38:16] <kurushiyama> Ange7: Since you mostly do updates, as far as I can see, it might well be that there are concurrency issues. However, I am still not convinced that this is the root cause for a simple query taking 4 seconds.
[14:41:32] <Ange7> i have most update with $upset yes. ~5000 update / each minutes in paralels with gearman
[17:23:42] <jasvir> Hi there. I am using mongdb zips: http://media.mongodb.org/zips.json and trying to run a geoNear query. Here is the query: http://paste.ubuntu.com/15986315/. But I run this query is show: http://paste.ubuntu.com/15986338/
[17:30:59] <GothAlice> cheeser / Derick / joannac / Number6: Old link was dead due to Freenode reorganization.
[17:32:04] <zeebee> Hey yall, quick hopefully easy question. how do I start mongod as a service but not use /etc/mongod.conf I wanna use /etc/mongod2.conf
[17:34:06] <zeebee> running mongod --config /etc/mongo2.conf works but i already have a mongo service running that uses the /etc/mongod.conf and I need a second instance
[17:36:06] <GothAlice> zeebee: You'll find that your /etc/init.d or /etc/conf.d (or similar) script which the system uses to spawn the service is referencing that configuration file in the command-line, as you tried doing.
[17:36:39] <GothAlice> One possible solution would be to make a copy of that init script, update it to point at the second configuration file, and register it as a service like the other one.
[17:37:27] <GothAlice> AFIK monogd doesn't automatically reference any configuration files, so all configuration file use is explicit… somewhere. ;)
[17:38:14] <GothAlice> Actually /data means you forgot to specify a different path; I think that one is a default.
[17:40:02] <StephenLynx> you install mongo from mongodb's repositories on RHEL (tested on centOS 7.2)
[17:47:49] <jasvir> Hi there. I am using mongdb zips: http://media.mongodb.org/zips.json and trying to run a geoNear query. Here is the query: http://paste.ubuntu.com/15986315/. But when I run this query is show: http://paste.ubuntu.com/15986338/
[18:30:31] <saml> I want to get distinct keys of err. and count per err.$key. documents are like {err:{Video: ...}} {err:{Image: .., Video:..}}
[18:45:01] <kurushiyama> saml: Can you give us a sample doc? And you want to store the errors we are talking about in the metadata?
[18:49:06] <saml> not sure why i did it as an object. err: [{errorType: "Video", ..}, ..] is more reasonable
[18:49:47] <kurushiyama> saml: Can you give a complete example doc and what you exactly want to do? Oh, and btw, edrocks tends to hang around here, too ;)
[18:50:52] <saml> I run a script. and it just inserts mongodb doc after collecting all errors for each task
[18:51:26] <saml> just wanted to see what type of errors this script found. and how many errors of each kind of error
[18:51:35] <kurushiyama> saml: I am not sure wether you you need to have an array of subdocs. I'd probably go with {errs: ["video not found","metadata invalid"]}
[18:52:18] <kurushiyama> As for the type of error: Then, I'd probably not save an error description, as this is rather a frontend thiny.
[18:52:22] <saml> yah those are types. i also log actual detailed messages like http response code was 404 for video url such and such
[18:53:14] <saml> i put stuff to db instead of logging cause i thought this script will run for days. but actually it finished in 30 mins :P
[18:56:47] <saml> node.js is async out of the box best suited for massive web scale database such as mongodb. async io means it's really fast turning io bound programs to disk bound cause disk is cheap
[18:58:00] <kurushiyama> saml: We could argue on that. But lets come back to your problem. If in doubt, document points in time as shallow as you can. So if an error occurs, you simply log it flat.
[18:59:38] <saml> i did it like {_id:"some/video.mp4", metadata..., ok: false, err: {$errorType: concatenated error message of the type} }
[19:00:12] <kurushiyama> saml: You _can_ do it this way. But I do not see the point in embedding here.
[19:03:02] <kurushiyama> saml: Aye. And when you need the details, you can simply query the log. db.errorlog.find({videoId:"some/…"}).sort({date:-1}).limit(whatever)
[19:03:53] <saml> kurushiyama, is any of your mgo projects open source? it looks like you actually write tests around mongodb as well
[19:07:10] <kurushiyama> saml: https://github.com/mwmahlberg/mongostore/blob/develop/mongostore_test.go
[19:49:15] <_BlackMage_> I have a function saved but when I try to called it in the $where, I get ReferenceError: dateDiffInDays is not defined
[20:56:14] <zylo4747> how come when i run 'show dbs' it says my DB is 7.9 GB but when I switch context to the db and run db.stats() it shows storage size as 1.6 GB
[21:05:41] <jasvir> Hi, I have a query which results states using geoNear. I want to calculate total cities in each of these states. can anyone please help me to do this? Here is my query: http://paste.ubuntu.com/15991581/
[23:43:58] <colwem> New to mongodb. I want to dump a database as is to a file of js statements that can be used to recreate the db. I also want to be able to drop the db. This is for testing.
[23:45:37] <colwem> I guess it's not totally necessary that it's in javascript but I'd prefer it if the dump file is something I can read and edit if I need to.