Log file Viewer
#mongodb logs for Tuesday the 12th of April, 2016
[03:08:00] <nictuku> hi. I have a single server with a database that I want to move to a server across the atlantic (hetzner.de == cheaper). I'm OK with some downtime, but I wouldn't want to wait for a mongodump transfer. What would you recommend? convert my servers to a replica set, then move data that way?
[03:44:17] <Boomtime> @nickuku: "I wouldn't want to wait for a mongodump transfer." -> you have to transfer the data somehow, sooner or later those bits have to traverse the atlantic - a mongodump could at least be compressed for transmission, with scp or similar
[06:40:15] <lyze> Hello, I'm encountering a problem with morphia, is this the right place to ask for help?
[06:48:57] <Boomtime> @nictuku: what do you mean 'incrementally'? no matter what you do the data has got to be transferred, i'm not sure where you think the efficiency is being improved - the only way to get the data to be sent faster, is to send _less_ - unless you delete half your data, the only other option is to compress it, ala scp etc
[06:55:59] <lyze> And if it is, I've tried creating a simple test class as an entity ( http://pastebin.com/i70SwS2p ) which I try to save into the database. however I'll get a "indexoutofboundsexception" on the "save" method.: http://pastebin.com/r65xKGs3
[08:21:58] <mroman> If I have multiple mongos instances running, do I have to do the sh.addShard() on each of those?
[09:21:36] <mroman> but more importantly: is there a way to convert an existing standalone mongodb server to shards?
[09:23:50] <Keksike> Hey, I need some help updating my mongo on Ubuntu. I have 2.6.12 installed now. I run sudo apt update, then sudo apt upgrade mongodb-org, but it says that newest version already installed
[09:23:57] <kurushiyama> mroman: Set up a config server replset. Fire up a mongos. Add the standalone via sh.addShard()
[09:27:29] <kurushiyama> mroman: Though it is a Very Bad Idea (tm) to have a shard backed by a standalone instead of a replset.
[09:29:01] <gta99_> Hi all, I'm currently trying to install Mongo 3.3 on Debian Jessie, but an apt-get update reports "W: GPG error: http://repo.mongodb.org jessie/mongodb-org/3.3 Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BC711F9BA15703C6" any idea where to get the correct public key?
[09:30:18] <Keksike> kurushiyama: the problem is that although I have only 2.6.12 installed, it is saying that I have the newest version
[09:32:48] <Keksike> kurushiyama: I tried apt update before apt upgrade. Do I need to do something else to add the repo for the newer version?
[09:33:11] <kurushiyama> Keksike: It is NOT a command. You need to add another software repository...
[09:34:18] <kurushiyama> Keksike: https://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/#install-mongodb-community-edition is the result of googling "mongodb install ubuntu" ... ;P
[09:37:13] <Keksike> kurushiyama: do I need to somehow uninstall the previous version of my mongo, im on part 4 now of that walkthrough
[09:38:11] <kurushiyama> Keksike: no, apt should take care of that. Just make sure mongod is stopped.
[09:39:38] <kurushiyama> gta99_: I assume you have run "sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927"
[09:40:06] <kurushiyama> gta99_: Do not care about the ubuntu server, it is just for the signing key.
[09:42:18] <gta99_> - name: deb http://repo.mongodb.org/apt/{{ salt['grains.get']('os') | lower() }} {{ salt['grains.get']('oscodename') }}/mongodb-org/3.3 main
[09:45:23] <kurushiyama> gta99_: Another question is why to install a dev release with some sort of recipe ;)
[09:46:39] <gta99_> I've tried 3.2 wheezy mongodb but it fails with a dpkg configuration error for mongodb-org-server
[09:49:45] <Keksike> kurushiyama: ok, now I installed the 3.2.4 according to the rules you linked. But my mongodb-org-mongos, -server, -shell and -tools are still 2.6.12. How do I update them also?
[09:50:04] <Keksike> I though they would act like dependencies and that updating the mongodb-org would update them too
[09:51:11] <kurushiyama> Keksike: Do it manually, and then remove the old repo. just do an apt-get update mongodb-org-server and so on.
[09:51:55] <gta99_> maybe try apt-get remove --purge mongo* && apt-get update && apt-get install mongodb-org
[09:52:46] <kurushiyama> gta99_: Uhm, I am not super-fit with Debian/Ubuntu, but wouldn't the purpge option also purge the data?
[09:57:51] <Keksike> now im getting this https://gist.github.com/Keksike/2555afb9a117d2461a8c7f59f4457612
[09:59:52] <kurushiyama> Keksike: You might want to change the config file and point dbpath to /var/lib/mongodb
[10:02:52] <kurushiyama> Keksike: Well, good question then. TL;DR A bug in the package assembly manifest, most likely.
[10:04:34] <kurushiyama> Keksike: The longer story: Files need to be marked as configuration files during package creation. There are several ways to do that – and the side effects sometimes are complicated.
[10:05:12] <kurushiyama> Derick: of the top of your head, were there major changes in packaging done lately?
[10:07:35] <Keksike> cihanbebek@ci7:~$ sudo mongod --config /etc/mongodb.conf just doesnt do anything
[10:09:43] <kurushiyama> Keksike: Most likely, you have a permission problem, now, since new files most likely are created for root
[10:09:59] <kurushiyama> Keksike: For various reasons, it is a _very_ bad idea to run a service as root.
[10:10:41] <kurushiyama> Keksike: That https://xkcd.com/149/ is not the way you should think of sudo
[10:14:58] <kurushiyama> Keksike: You might want to stop mongod by doing a "sudo killall -TERM mongod" now, then, you want to run "sudo chown mongodb.mongodb -R /var/lib/mongodb" and then you want to start mongodb via the usual means
[10:18:58] <Keksike> I was trying to run it with mongod but I should have been running it with sudo service mongod start
[10:19:54] <kurushiyama> Keksike: Yep. But you need to go through the procedure I gave you above to fix potential permission problems.
[10:47:06] <SimpleName> If I config a product with serveral images, Should I design images model with mongodb, or use Mysql like this: img1 img2..img10, you can upload at most 10 images to one product
[10:51:00] <mroman> I may be dealing with an architecture where not all servers are really under my control.
[10:51:24] <Derick> "The admin database in a sharded environment will actually live on the config servers."
[10:51:34] <Derick> http://serverfault.com/questions/570032/mongo-sharding-config-server-and-mongo-authentication
[10:55:34] <kurushiyama> @Derick You are correct with the primary shard thing in case the user is stored in the DB.
[10:57:25] <kurushiyama> mroman: Authorization with MongoDB's internal means (and that is the problem here) is not granular enough to guarantee separation of data
[10:57:47] <kurushiyama> mroman: Depending on the data and your jurisdiction, it might even be illegal to do so.
[10:59:04] <kurushiyama> @Derick Personally, in a sharded env, I tend to store the users in the admin database, though
[10:59:56] <kurushiyama> mroman: I know for sure it would be in Germany, and most likely it would be in the othe EU states, as well.
[11:04:09] <kurushiyama> mroman: There are limits defined in what you can do in the BDSG (Bundesdatenschutzgesetz – german for Data privacy act). One of them is that you have to use all means necessary to make sure the data is safe. Putting the data set of multiple companies into the same database is not exactly that. Furthermore in case the data is compromised and multiple companies have potential access, it is very hard to find out who is respo
[11:04:09] <kurushiyama> nsible, making it likely that the provider of said database will be held liable. Each breach of the BDSG might be fined with €50k - €300k, or more, at the judges discretion.
[11:05:07] <kurushiyama> mroman: And iirc, that is per incident. You better have your check book ready.
[11:06:28] <mroman> yeah but how would that be different from collecting all the data in one company?
[11:07:07] <kurushiyama> mroman: Aside of that. When multiple companies share the same database, this is usually done to do some integrative tasks. The industry learned the hard way that databases are a poor tool for integration. With regards to one company
[11:08:47] <mroman> well you could just rent some cloud space and each company uploads stuff to the same cloud space
[11:09:25] <kurushiyama> mroman: And set up an API gateway to CRUD said data. Decouple as much as you can
[11:10:31] <kurushiyama> mroman: I even have a name for said joint data center: "WSOC - Weather StOrage Center" ;)
[11:12:21] <kurushiyama> mroman: At best, that is aggregation. Guess why the framework is called that way.
[11:16:21] <kurushiyama> mroman: Sorta. Iiirc, some poker networks use IVs of pseudo randomly selected sensors in a bariometer network...
[11:17:37] <kurushiyama> mroman: Well, my suggestion still stands. Create WSOC and an API to CRUD the data.
[11:22:05] <kurushiyama> mroman: Hint: more often than not, aggregations are better suited and faster ;)
[11:30:27] <kurushiyama> mroman: Well, sounds reasonable. I was a bit concerned for the throughput, but unless each sensor sends _a lot_ of data/s, even a USB thumb drive should be fast enough. I am not sure wether sharding is necessary from the beginning,then, since adding shards would be to keep the sweet spot for data storage costs. However, you can start out with a single shard, to be sure everything works as expected should the need arise to s
[11:35:08] <kurushiyama> mroman: You should be careful there. I would first identify wether actually a m/r is needed. As a rule of thumb: If you can do it with the aggregation framework, it is usually the better idea.
[11:36:55] <kurushiyama> mroman: To cut a long story short: Create a test env and try to find the best way.
[11:38:01] <kurushiyama> mroman: Have a deep look into the aggregation framework, then. A very deep look.
[11:42:44] <kurushiyama> mroman: Aside from an early match: you can actually use disk space, as m/r does by default. But the aggregation framework offers _really_ nifty functions.
[11:45:08] <Mumla> I got a crazy problem which is summarized here: http://stackoverflow.com/questions/36499924/cant-start-mongodb-as-a-service (I hope you don't hate me for posting such a link here...). I'm quite desperate here at my work :(
[11:48:53] <kurushiyama> Mumla: And you should be. With all the trial and error, your install most likely is pretty screwed.
[11:50:59] <kurushiyama> Mumla: On several conditions. a) Delete the question on SO. It belongs to dba.stackexchange.com b) Reask it there c) Add OS, its version and the mongodb version d) Success not guaranteed.
[11:54:54] <Mumla> when we - so you mean you come back to my dba-queston ;)? i'd also like to stay in touch with this Robert Udah...
[11:55:37] <kurushiyama> We will do it here. I need to ask questions, Inform him that you reask on dba. He will get the message anyway.
[12:00:16] <Mumla> I was just looking for it, here it is: "CONTROL [initandlisten] db version v3.2.4"
[12:29:12] <kurushiyama> mroman: http://blog.mahlberg.io/blog/2015/11/05/data-modelling-for-mongodb/
[12:33:39] <kurushiyama> mroman: Is it? ;) And no, embedding actually does more harm than good for m/r, generally speaking.
[12:35:51] <kurushiyama> mroman: Uhm. Aside from the fact that map/reduce has little to do with actual maps of the source data, the 16MB size limit is hardcoded. So regardless, you need to adapt your models.
[12:38:21] <kurushiyama> mroman: Regardless on what you want and how you put it: the 16MB size limit is given. Model your data accordingly ;)
[12:43:55] <mroman> but I think that a lot of what we want to do might be not well suited for it actually
[12:44:35] <mroman> we don't really know. We just have a shitload of data and we're looking for stuff in it.
[12:44:39] <kurushiyama> Storing and analyzing time series data? MongoDB was developed for storing and analyzing clickstreams. Go figure ;)
[14:03:06] <mroman> Can you aggregate to a temporary collection and then use find or stuff like that?
[14:04:54] <kurushiyama> mroman: If you need to find stuff in your aggregation, use an additional $match
[14:05:26] <crodjer> In MongoDB `db.serverStatus().opcounters` does query mean `find` and `findOne`?
[14:37:18] <kurushiyama> mroman: You'd $unwind them, sort them by "row.rows" descending, and use $first in a group stage, for example.
[14:43:29] <kurushiyama> mroman: > db.rows.aggregate({$unwind:"$rows"},{$sort:{"rows.row":-1}},{$group:{_id:"$id",maxRow:{$first:"$rows.row"}}})
[15:12:40] <basldex> tried to make it listen to 0.0.0.0 now it says it does by default (apparently it doesn't, netstat shows bind to 127.0.0.1)
[15:13:51] <kurushiyama> basldex: And no, not that I know of (which does not say too much, though). OS, version, MongoDB version?
[15:17:19] <basldex> here's the backtrace: https://gist.github.com/anonymous/4f2c2404225aea0f07399793c50c3ab2
[15:18:00] <basldex> I won't mess with my package system on production environments just to get an update, sorry
[15:20:50] <basldex> as I said. I really don't want to mess around with my production machines' package managing
[15:21:14] <Derick> kurushiyama: I thought we compiled that in statically - but could be wrong. If it's an *Ubuntu* packaged mongodb, then, I think you're out of luck anyway
[15:23:10] <kurushiyama> Derick: There is a reason why Ubuntu scares me of like a devil is from holy water.
[15:23:29] <Derick> kurushiyama: it's good as a desktop OS, but they make intersting packaging choices
[15:24:21] <kurushiyama> basldex: You almost always want to use MongoDB Inc's official packages, never failed me, though some of them could be reworked (and yes, I am in the process of doing so) ;=
[15:24:57] <basldex> it is an interesting packaging choice indeed to break important server software in your long term release
[15:26:22] <kurushiyama> Derick: As a desktop OS I find it to be too bloated, and as a server OS? Yeah... "interesting" is probably the politest way to describe it. For example the "interesting" choices regarding the proc fs
[15:29:25] <kurushiyama> Derick: Erm. I do not want to do finger pointing, but there were _extremely_ nasty bugs in wt integration when 3.0 was released. tbh, since then, I always stay a minor behind.
[15:31:41] <kurushiyama> basldex: And by any chance, unless you have _very_ good reasons not to, you might want to migrate to wiredTiger.
[16:03:00] <bros> Does anybody know if there are plans to add a query/$where to $lookup on aggregation?
[16:33:35] <kurushiyama> bros: More often than not, this is an indication of a flawed data model. Just saying. I _never_ missed JOINs
[16:36:06] <kurushiyama> bros: Even more so since it is unlikely that you want to display ALL items of ALL locations SIMULTANEOUSLY.
[16:37:04] <kurushiyama> bros: What is the use case to have _all_ items of _all_ locations gathered together?
[16:38:32] <kurushiyama> bros: Sounds to me like a readthough cache would be more useful to you. But setting that aside: What sort of cache? What are you caching and why?
[16:40:14] <kurushiyama> bros: Set aside the language, but this pretty much explains it: https://github.com/golang/groupcache
[16:41:31] <kurushiyama> bros: And here is what I do not get. Basically, you tend to query either an item and want to find out where to get it. Or, you have a location and want to know what is stored there.
[16:43:30] <kurushiyama> jr3: like you have 10 items in doc A and 20 items in doc B, the answer should be 30?
[16:46:04] <jr3> Would it be better to just have another field arrayCount keep it in sync the index it
[16:47:27] <kurushiyama> jr3: db.yourColl.aggregate({$project:{_id:1,count:{$size:"$yourArray"}}},{$group:{_id:"numberOfItems",count:{$sum:"$count"}}})
[18:35:39] <kurushiyama> dddh_: Beautiful, robost, secure, portable and scalable? Yes, that is funny!
[18:39:59] <kurushiyama> dddh_: We could argue about that, but this is #mongodb and not #languagebash
[19:11:11] <kurushiyama> bros: Would need detailed structure and expected results. But I am preparing dinner, so maybe tomorrow?
[19:44:12] <shlant> hi all. will I run into problems dumping a 3.2 db with a 3.0 client and then importing to a 3.2 db?
[19:44:58] <shlant> and if so, is that also true the other way around? 3.0 db dumped with 3.2 client and imported into 3.0 db
[20:14:38] <mkjgore> hey folks, just did a restore of our mongo cluster and I've noticed that the machines just don't perform as they used to. Even when sitting "idle" (our app isn't accessing the cluster as far as I can tell) there are times when iotop is showing almost constant reads across all the rep sets (sometimes in the Mb for a few seconds)
[20:22:00] <crazyphil> kurushiyama: you weren't kidding about trying to export and import the monster setup I have running right now, it's going on 4 days and the import is only at 34%
[20:45:55] <kurushiyama> crazyphil: Thinking of it, it might be easier/faster to stop and build a temporary replset?