Log file Viewer
#mongodb logs for Wednesday the 6th of April, 2016
[00:11:19] <greyTEO> GothAlice: any chance development will continue on https://github.com/marrow/task ?
[00:12:35] <GothAlice> greyTEO: Indeed; I'm currently polishing up the cinje template/DSL engine first (ugh, marketing sites are tedious to work on if you're a lib developer ;) then can turn my attention to divorcing task from MongoEngine prior to bundling up a release of that.
[00:14:03] <greyTEO> awesome! im currently re-implementing a queue from a while back and do not marrow.task avalible on pip anymore..
[00:15:29] <GothAlice> greyTEO: https://github.com/marrow/mongo/blob/develop/marrow/mongo/util/capped.py may be of interest.
[00:15:58] <GothAlice> marrow.mongo is going to be my collection of pymongo add-ons, vs. the MongoEngine approach of wrapping everything.
[00:16:48] <GothAlice> And, ah, see the documentation for Python's "warnings" module to disable that "A batgirl has died." warning if you try to patch in production environments. ;P
[00:21:34] <GothAlice> But the function was designed, in this iteration, to match the call semantics of the rest of the new pymongo API, i.e. designed to integrate as a method on Collection.
[01:46:25] <Forbidd3n> Quick question in regards to storing dataset in MongoDB? Would I have three pieces; Cruise Lines, Cruise Ships (linked to lines), Cruise Schedules (linked to ships) - Should I store this all in one collection or should I create three collections and link them by id?
[01:47:26] <Forbidd3n> I was thinking of one collection called Schedules (Line, Ship, Date, Port, Arrive, Depart)
[01:48:03] <Forbidd3n> The only issue with this is I will need to use Cruise Lines and Cruise Ships again and will have other tables using the same references
[01:51:53] <Forbidd3n> Or better yet I was thinking one collection called Schedules it will hold parent (lines -> object/array of ships per line -> object/array of schedules per ship
[08:19:40] <bo_> Hi All, in repica set primary and arbiter nodes can't see secondary node. Return "lastHeartbeatMessage" : "Couldn't get a connection within the time limit". Please help
[08:47:19] <bo_> in repica set primary and arbiter nodes can't see secondary node. Return "lastHeartbeatMessage" : "Couldn't get a connection within the time limit". Please help
[10:15:59] <r3d0x> hey there. I'm having an issue here with an update, concerning the positional operator. Does anyone here have an idea if this is a bug? Thanks http://pastebin.com/h2NGhN8Z
[10:18:57] <Derick> r3d0x: not sure whether it is a *bug* - perhaps not something we support. Does it work if you switch the query clauses for scopes and tags.id around?
[10:45:00] <kurushiyama> @Derick Deja vu: http://stackoverflow.com/questions/36448005/how-to-have-best-performance-with-nosql-database-and-million-documents
[10:46:14] <kurushiyama> @Derick Cant reference IRC, but it is OT on multiple levels, and I opted for too broad ;P
[10:49:24] <kurushiyama> But he is to congratulate. Triple the docs/day from yesterday. The application seems to get massive momentum.
[11:24:36] <kurushiyama> dddh__ Can be one indication. I just want to make sure the basics are in place.
[11:25:14] <kurushiyama> dddh__ I chased flying sheep to often just to find out that it was the freaking grass.
[11:25:31] <dddh__> recently had something similar, had to remove ipv6 from /etc/hosts and reconfig force ;(
[11:28:27] <kurushiyama> bo_ Hm, you have a primary, so where is the problem. You have (more or less) all the time in the world to debug.
[11:42:03] <kurushiyama> bo_ It is really weird, since you ran rs.status() on the secondary. Can you rereun it and use pastebin to show the result?
[11:45:32] <kurushiyama> aiRness Depends on the version you run. If <3.0, that is possible. However you should impose an fsynclock
[11:47:18] <kurushiyama> Hm, you should be able to connect to the mongod inside the docker container, anyway...
[11:47:45] <bo_> <kurushiyama> restart secondary node and get rs.status() http://pastebin.com/W7KeUExx
[11:50:51] <bo_> <kurushiyama> but what is it ? Why mongod have weird behaviour when send HeartBeat?
[11:51:29] <kurushiyama> aiRness a) mongodump is the very least backup solution I use, aside from config servers in a sharded cluster b) The bigger your data gets, the more inconsistent your backup will become using mongodump. c) at least LVM should be possible. It is free and easy to use.
[11:51:59] <kurushiyama> bo_ It sends a heartbeat every few secs, iirc. That is not the cause of any problem.
[11:54:22] <kurushiyama> aiRness Then simply connect to your mongodb via the exposed port and do your dump. Again, doing a mongodump against a running instance is a horrible idea for backup purposes, if we are not talking of cheap data. It _is_ possible, but you really have to put some effort into data modelling to make sure you will get a consistent backup.
[11:54:44] <kurushiyama> bo_ Maybe you should start from the beginning, describe your env and the problems you experience.
[11:55:04] <aiRness> kurushiyama: alright, thanks but regarding connecting to the exposed port I get an empty dump, I can see the data only from inside the container
[11:55:45] <kurushiyama> aiRness https://docs.mongodb.org/manual/reference/program/mongodump/#working-set
[11:57:17] <kurushiyama> aiRness In your _container_ your mongod surely listens on 27017. On the host? most likely not so much.
[12:02:04] <kurushiyama> Which leads to the question how your prod env accesses the data... You'd check that. Now.
[12:03:09] <aiRness> well I didn't configure it but the host machine has multiple 27017 entries open
[12:04:58] <aiRness> kurushiyama: did already, everything works fine apart from the dump from the host
[12:06:34] <kurushiyama> run docker ps -a on the other nodes of the replset. I assume they expose the port.
[12:07:29] <kurushiyama> aiRness If it is not reachable, it is unlikely to be still primary, no matter your designation ;)
[12:08:30] <aiRness> the only issue at the momment is that mongodump is empty, it can see fine the host/port and I can connect to the mongo client
[12:10:04] <kurushiyama> aiRness Well, since you seem to have it all right, I can not help you. Your magical docker env, which does not expose a port probably exchanges the data using quantum fluctuation, which is beyond my understanding.
[12:19:36] <kurushiyama> aiRness Well, we need to find out what's happening... So we need to collect facts. ;)
[12:25:20] <kurushiyama> aiRness Dang, I should have known that. But still I am curious about the quantum fluctuation data exchange.
[12:27:04] <aiRness> well me too but I guess I need to talk with my colleagues because I didn't set it up
[12:27:51] <kurushiyama> aiRness And still, using mongodump as means of backups is something I strongly discourage. You have been warned ;)
[12:39:36] <kurushiyama> aiRness imvho, DevOps is just an expression of CXOs saying "I just have to c&p a few lines, and I have software X installed. How hard can that be?" Overlooking some tiny details.
[12:42:36] <kurushiyama> dddh__ The problem is when data gets big and the mongodump takes long. Hidden or not, you'd run into problems.
[12:43:37] <aiRness> kurushiyama: it will fade away like other stuff did, virtualization, agile, cloud way of doing things, etc.
[12:44:34] <aiRness> nobody understands what it actually means, everyone is missinterpreting each other
[12:44:52] <kurushiyama> aiRness I doubt that. It "saves" money. Those things are going to persist. And I am not too sure agile fades. It is the reason for this devops nonsense the first place, no?
[12:50:41] <Spritzgebaeck> i have a question. we had an application for a mongodb 2.6.11, today we upgrade to 3.2.1 and our aggregates with an $out command are slower than before. if we removed the $out command there perform nearly same
[12:56:49] <Spritzgebaeck> i have a question. we had an application for a mongodb 2.6.11, today we upgrade to 3.2.1 and our aggregates with an $out command are slower than before. if we removed the $out command there perform nearly same
[13:01:46] <kurushiyama> Spritzgebaeck Uhm. You are running a mongod on a windows 8 for production and performance is of concern?
[13:02:52] <kurushiyama> Spritzgebaeck Forgive me, but that is a bit like complaining that your VW Golf is getting slower when you load stones into it, all while trying to get it ready to compete in Formula 1.
[13:03:03] <Spritzgebaeck> it's a developer machine and after the upgrade our aggregates took longer.
[13:03:52] <kurushiyama> Spritzgebaeck Which is understandable. a) WT needs more RAM b) WT needs more CPU c) WT is more reliant on FS performance.
[13:05:38] <kurushiyama> Spritzgebaeck regarding a) Windows takes a lot of RAM by itself. b) Your CPU most likely did no change c) Both ReFS and NTFS aren't exactly known as performance monsters.
[13:08:46] <Spritzgebaeck> kurushiyama: okay. than we have an explanation for our problem. we was uncertain about that problem
[13:10:24] <kurushiyama> Spritzgebaeck I _strongly_ discourage running MongoDB on windows in prodcution. The FS alone is a bottleneck.
[13:11:38] <Spritzgebaeck> clients... there a many of them, all with windows servers and no real admins
[13:14:09] <kurushiyama> Spritzgebaeck Be warned. Running MongoDB on Windows forces you to prematurely scale, most likely because of IO or RAM bottlenecks artificially made worse because of the uderlying OS. We can easily talk of hundreds to thousand/month.
[13:16:06] <kurushiyama> Wow, running an IT infrastructure with no admins. That is the epitomy of devops.
[13:18:47] <StephenLynx> in times like these I am really glad I work in a company where the CEO/owner is a linux user v:
[13:20:44] <dreamreal> I was at a shop like that except they were all mac... and relied on azure/sharepoint for everything
[13:22:34] <Spritzgebaeck> the clients think, they a bought something like ms office and could install it by itself, so no needs for admins.
[13:24:41] <scruz> kurushiyama: i’m building the pipeline in python, so i get some extra tools to work with to create a monster of an aggregation pipeline
[15:21:34] <deshymers> I'm looking at using mongodb to store analytic data, and I was wondering if this https://docs.mongodb.org/ecosystem/use-cases/pre-aggregated-reports/ is still relevant an current?
[15:24:13] <echelon> i'm just wondering if it would add too much latency considering ebs is all networked storage
[15:47:12] <echelon> "Nevertheless, systems running MongoDB do not need swap for routine operation. Database files are memory-mapped and should constitute most of your MongoDB memory use. Therefore, it is unlikely that mongod will ever use any swap space in normal operation. The operating system will release memory from the memory mapped files without needing swap and MongoDB can write data to the data files without needing the
[15:48:11] <Derick> https://engineering.mongodb.com/post/code-generating-away-the-boilerplate-in-our-migration-back-to-spidermonkey/ has the story
[15:49:47] <kurushiyama> @Derick I did not ask the original question, but I was and am interested anyway.
[15:54:52] <kurushiyama> echelon Ok, one step back. You were saying that you are out of memory. What is your RAM utilization?
[15:56:15] <kurushiyama> echelon That _is_ a bit high, admittedly, it should be somewhere between 85 - 90%
[15:57:05] <kurushiyama> echelon But as long as physical mem is still available, I would not be concerned.
[15:58:25] <kurushiyama> echelon With a mem usage that high, it might well be that your queries touch documents not in the working set.
[15:59:44] <deshymers> I am working on storing analytic data in mongodb, we dont need to be very granular only weekly and monthly counts, so I was thinking that this structure would be good for that, https://gist.github.com/deshymers/d86d1d579c32a133e47d2ec227b39315
[16:00:21] <deshymers> is there anything blatantly wrong with, or that could cause problems with that document schema?
[16:00:24] <kurushiyama> echelon And then there might be readahead settings which are not optimal, compression might interfere and whatnot.
[16:02:00] <kurushiyama> echelon Or, to put it the other way around: The best experience I have with MongoDB is on SSDs, enabling raw performance.
[16:03:05] <kurushiyama> deshymers As for time series data, I'd always put it into a flat "schema" like {"date": someDate, "source":whatever, "value": foo}
[16:05:27] <kurushiyama> deshymers You do not need another one. You can calc the counts on demand or from time to time using aggregations, potentially writing those aggregations to a collection using an $out stage
[16:05:38] <deshymers> this is what I was following, https://docs.mongodb.org/ecosystem/use-cases/pre-aggregated-reports/#schema
[16:07:25] <deshymers> kurushiyama: you would probably get better write performance with a flatter document, correct?
[16:09:52] <kurushiyama> deshymers If you need real time data (in the strictest meaning of the word) however, it might not be optimal.
[16:10:44] <kurushiyama> deshymers Then I'd go with flat events, plus aggregations with an $out stage
[16:10:52] <deshymers> I know this talks about down to the second, http://blog.mongodb.org/post/65517193370/schema-design-for-time-series-data-in-mongodb and that is way over kill for my needs :P
[16:11:40] <deshymers> kurushiyama: and filtering on a date index by weekly wont be a big performance hit?
[16:12:46] <kurushiyama> deshymers depends on what we are talking about. But in general we are talking of a DBMS. It would not really stand up to the name if it was unable to answer questions.
[16:14:14] <kurushiyama> deshymers Let me put it that way: I do something similar regularly for testing purposes on 10M records. On my 6 year old MBA...
[16:22:14] <deshymers> so another question, the ObjectID contains a timestamp, is that indexable? or would I be better of having a separate timestamp to query against?
[16:25:36] <kurushiyama> deshymers short answer: Use a separate timestamp. Long answer: There are some rare use cases in which ObjectId is sufficient since it is monotonically increasing. But it may get really complicated with client side generated ObjectIds and alike. All that for a saved index over a 64bit int? I dont think so.
[16:32:01] <Derick> meh, it's not that hard. All you need is stackoverflow and copy and paste skills
[16:34:27] <kurushiyama> Derick Well, since my c++ skills are around 0.1 on a scale of 1 - 100, I'd probably accidentally transcend MongoDB into the Flying Spaghetti Monster. ;)
[18:24:54] <kurushiyama> StephenLynx there is a bug somewhere. Ran into it, and while researching on it, I found out that aggregations are not susceptible to this problem.
[18:30:56] <uehtesham90_> hello, i had a question regarding using flask with mongoengine. i am adding a flask web app on an already existing project which consists of multiple databases. but in flask-mongoengine, we have to define a default database, so should we just provide one of the database names as the default database and internally access other databases through the mongo
[18:30:56] <uehtesham90_> client? i am not sure what the best approach is...would appreciate some help here :)
[18:33:22] <uehtesham90_> i am asking in #mongoengine but there was no response...so i thought i would give it a shot here
[18:34:16] <kurushiyama> uehtesham90_ Well, personally, I do not use either, so my help will be limited.
[18:36:22] <uehtesham90_> StephenLynx: there is a system settings but we can only define one database...but i have currently many databases....so i want sure if i should have a separate database instance referring to each database?
[18:38:31] <kurushiyama> uehtesham90_ From what I can see, the problems might actually lie elsewhere.
[18:39:54] <uehtesham90_> StephenLynx: this is how my project is setup. it is for analysing student data for different online courses. so we stored each course data in separate databases (each with their own collections)
[18:41:41] <uehtesham90_> but what abt individual collections....each course have their own sets of data like user info for a given course
[18:41:57] <uehtesham90_> would i add them in a single collection and define what course they belong to?
[18:42:27] <uehtesham90_> but then each collection would become very large....what is the best way to handle that?
[18:43:05] <StephenLynx> its much easier to manage a single large collection than the same collection on multiple databases.
[18:44:03] <uehtesham90_> can you recommend any good sources to learn more abt good database design...i really want to improve on this
[18:45:00] <StephenLynx> code complete, second edition might give you some general software development insight.
[18:45:53] <StephenLynx> addendum: dynamic collection creation and dynamic field creation are awful too.
[18:46:04] <uehtesham90_> just one more question....based on your experience, when should a separate database be used? like what one should look for when considering using a new database
[18:47:30] <StephenLynx> you have to have this legacy database being used by your software and at the same time have a new database
[18:47:51] <StephenLynx> or you want to isolate the databases physically and it serves a different purpose
[18:48:22] <StephenLynx> usually a different database will serve a different system and you won't connect to the db directly and will instead communicate with the system that handles the second database.
[18:49:51] <kurushiyama> Integration via a database is a Very Bad Idea™ , as we learned the hard way all through the 90s
[18:50:51] <kurushiyama> StephenLynx I was not weakening your arguments. I was just differing on the conslusion ;)
[18:52:27] <kurushiyama> Imho, there is absolutely no reason to have two databases even when dealing with legacy systems. either migrate or integrate. ;)
[18:53:38] <StephenLynx> and any other option means throwing everything that depends on the software to be trashed, including companies.
[18:54:46] <kurushiyama> StephenLynx Whereas it remains to be proven that keeping multiple databases make more sense than doing a migration. No offense?
[18:55:42] <uehtesham90_> kurushiyama: by integration, i am assuming that 'old database' would then become tables belonging to the new database
[18:57:35] <kurushiyama> uehtesham90_ Well... no, not really. Assume you have a legacy system in need of the old database, and you have a new system with some other. Integration would mean that you make those applications communicate with each other to exchange data, rather than to have them communicate by sharing data
[19:01:19] <kurushiyama> uehtesham90_ The problem with communication via data sharing is that two applications become dependent on the same database, and changes in your data structure become really, really hard to be executed without affecting the other system
[19:28:27] <Kaetemi> getting this exception on the target system when doing rs.add, any idea http://pastebin.com/BYFN9fpm ?
[19:33:36] <FuzzySockets> When you create an index with mongoose, what do the 1 or -1 represent... the sort order?
[20:18:04] <FuzzySockets> Anyone know of an accurate json document containing city/state to postal code mappings, preferably for all countries.
[20:57:26] <saml> can you use aggregate to get last action? {timestamp: ISODate(..), action: either 0 or 1, name: thing to group by}
[20:59:32] <saml> i log which user subscribed, unsubscribed over time. i want to get current state of a given user's subscription
[20:59:53] <saml> like, this use is subscribed to these lists (optionally, unsubscribed to these lists)
[21:00:19] <saml> so, if i subscribed to list1 , then unsubscribed more recently, i don't want list1 to be part of the result
[21:02:20] <saml> so i need group by listname sort by timestamp and only use the first listname,action per group
[21:02:27] <kurushiyama> saml "i want to get current state of a given user's subscription" not too precise, then ;)
[21:06:52] <saml> kurushiyama, https://gist.github.com/saml/1d2f050f0440e36cfbe152fa47083399 this is what i'm trying to do
[21:08:32] <saml> db.docs.aggregate([{$match:{user:'saml'}}, {$group:{_id:'$to', action: '$action'}}])
[21:18:35] <kurushiyama> saml db.mailing.aggregate([{$match:{user:"saml"}},{$sort:{"date":1}},{$group:{_id:{user:"$user",list:"$to"},status:{$last:"$action"}}}])
[22:37:07] <Stereo> I'm trying to replace 'http://www' with 'https://www' in any values in all of my documents, and I'm not sure how to do this.
[23:20:40] <dimaj> hello all! have a weird question today... is there a way to access a mongodb through another mongodb? My setup is as follows: Target MongoDB (Network A) <--- Proxy MongoDB (Network B) <---- Application (Network B). I need to access Target MongoDB from my application