Log file Viewer
#mongodb logs for Tuesday the 28th of July, 2015
[00:27:57] <pylua> I have remove half of the data ,but why does the disk availabe space not increase?
[00:30:51] <cheeser> mongodb doesn't release space back to the OS until you do a repair. at least with mmapv1. wiredtiger is better about that, iirc.
[00:39:05] <pylua> cheeser:if I have removed the data in mongodb ,then when the disk space is not enough ,would mongodb reclaim the unuseful space ?
[00:45:50] <pylua> so i do not need to be worry about the disk space not enough after remove part of the data?
[00:53:55] <pylua> cheeser:my problem is the disk space can not be repaired because the repairDatabse() can not be executed when disk space is not enough
[03:49:18] <pylua> for disk space problem ,can it be resoled by removing the deprecated data from mongodb in a fixed interval ?
[04:07:35] <morenoh149> you could also compute aggregate statistics and store those instead of the original data (would also address disk space usage)
[06:01:58] <m3t4lukas> in the java driver: should I rather use com.mongodb.client.MongoDatabase or com.mongodb.DB?
[06:08:03] <m3t4lukas> I mean, only DB seems to be working, but it seems a lot like unclean code to me :/
[06:38:46] <Jonno_FTW> hi, is mongo the right tool for my job? I need to store ~2TB of documents that have time,location, and an array of objects with ~8 elements and 2 fields. The data won't be updated, just inserted into and queried a lot, more queries than inserts by about a 3 times
[06:39:30] <Jonno_FTW> 2TB is what I have now, but it will be streamed in the future, so maybe up to 10TB on a single machine
[08:47:32] <pdekker> I think my problems are caused by the keyfile. Is there a way to setup a secure replica set without using a keyfile?
[08:57:47] <pdekker> joannac: The problem is the following. If I have an empty db without users, I can use the keyfile option and issue rs.status() or rs.initiate() without problem
[08:58:21] <pdekker> However, if I have a database with users, and then add the replSet and keyfile options, it is not possible to issue rs.status() or rs.initiate() anymore. They give an error "not authorized as admin"
[08:58:45] <pdekker> When I then remove the keyfile option again (but leave replSet), it is again possible to issue rs.status() or initiate
[08:58:59] <pdekker> The problem is somewhere in the combination of using a keyfile and existing users
[08:59:58] <pdekker> I would like to keep my existing database, with existing users, and have a secure way to set up a replica set
[09:00:47] <pdekker> I want to replicate this existing database to a new, empty database, so they become a synced replica set. I do however not want other people to access the data in my existing database by creating a replica set with it
[09:03:42] <pdekker> It seems the existing users do not combine with the keyfile. Is it possible to do replica set authentication based on the users, instead of the keyfile?
[09:06:46] <pdekker> Is this a gap that could be filled up by using a firewall? Only allowing connections from the other members of the replica set, but not other?
[09:11:17] <joannac> Anything in this list that includes "replSetGetStatus" http://docs.mongodb.org/v2.6/reference/built-in-roles
[10:14:04] <sarathms> I see some unexpected behavior with aggregation using the $geoNear operator. There seems to be a limit of 100 items in the result set. I've isolated the problem in this piece of code: https://github.com/sarathms/mongodb-geo-aggregate-test. Am I missing something?
[10:16:43] <sarathms> from location A I'm able to find a doc at B say 20km away. When I added about 100 docs around or at A and repeat the same search, I don't find B anymore.
[10:18:04] <sarathms> works until I add 99 docs around A. From 100 and above, I get only the 100 nearest items in the results. There's no $limit stage in the aggregation pipeline.
[11:00:23] <joannac> sarathms: yes, the part where it says the limit is 100 in the docs ;) http://docs.mongodb.org/manual/reference/operator/aggregation/geoNear/
[11:02:32] <sarathms> nodejs native driver documentation shows a different default value. I happened to see that first :| https://mongodb.github.io/node-mongodb-native/2.0/api/Collection.html#geoNear
[11:04:17] <joannac> sarathms: hrm, I think those docs should say the same thing, i.e. that the default for "num" is 100
[11:21:10] <pchoo> I'm wondering if it is possible to convert an array of objects into an object within the pipeline?
[11:21:46] <pchoo> I have my data in the following format https://www.irccloud.com/pastebin/J46r3mlr/
[11:23:30] <pchoo> I've only just started looking at aggregation like this, so haven't yet grok'd it all properly
[11:26:15] <pchoo> Derick: to reduce client side munging in my meteor app. I wasn't sure if it was possible, but thought I'd ask anyway, thank you :)
[12:15:36] <jamieshepherd> If I have a collection of reviews, I'd like the ability to be able to give a review a "thumbs up", obviously I only want to be able to give one per user account. Should I store this in the users collection, and then just store a total on the review? Not sure the best practice here.
[12:48:12] <aps> Do I need to restart mongod after config change? Is there a way to reload the config file without restart?
[13:00:02] <deathanchor> aps, config changes are good for restarts, but most config settings can be done as mongo commands in the shell
[13:35:10] <pamp> Im using terminal, but I need to analise a lot of logs, and its boring do it in a terminal
[13:45:20] <symbol> Can someone explain this Date behavior in the shell? https://gist.github.com/anonymous/11dfcb5ca4d76efe4ae8
[13:50:25] <symbol> MongoDB doesn't actually create a Date object? Or is it that it just wraps the Date object in the ISODate?
[13:52:01] <symbol> hmm I suppose it's because BSON stores dates in the ISO byte format so the wrapper makes sense.
[14:07:14] <JoshK_> Hello. I have a document that looks like http://www.hastebin.com/irevidimoq.coffee and I would like to insert something into the "logs" object using the java driver - how would I do this?
[14:09:16] <MANCHUCK> in mongo it is db.coffee.update({_id: "foobar", {"$push": {logs: {bar: "bat"}}})
[14:09:28] <pdekker> I have two db's in a replica set which connected well before. I changed the setup of the db which should be secondary: I started running it as a service. Now, the "primary" db can still find it, but denies permission
[14:23:07] <leandroa> Hi, I have a doc like: {date: ISODate(), hours: {"0": 123, "1": 22, ..., "23": 21}.. how can I aggregate to return: {date: ISODate(), hour: "2, value: sum_of_all_values_in_same_date_and_hour} ?
[14:25:07] <lqez> InfluxDB introduces itself - 'An open-source distributed time series database with no external dependencies.'
[14:27:10] <lqez> http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GuidelinesForTables.html
[14:28:55] <lqez> I've used mongodb for inserting-only database very well - but it's up to your data
[14:29:25] <pdekker> lqez: Before, I ran it as local user, with some local directories. Now, I used the CentOS init script and it runs as the mongod user
[14:33:15] <pdekker> lqez: /var/lib/mongo, the standard which is mentioned in the init script. It is owned by mongod. Mongo does not give an error about the data dir. It even connects to the "primary" (actually not yet primary) db, where it is mentioned as STARTUP
[14:33:42] <pdekker> However, the primary says: "replSet couldn't elect self, only received 1 votes"
[14:37:27] <lqez> crised: I think MongoDB is good for lots of irregular, atypical or hierarchical data .
[14:38:35] <crised> lqez: what about a database/cache that can respond to, give me 1 to 50 first rows
[15:02:23] <sachiman> I am using Strongloop loopback to generate REST Apis for models backed by MongoDB
[15:02:46] <StephenLynx> I use the time cut down to the hour to version the stat document of a board
[15:06:11] <StephenLynx> I am working with node (now io, soon node again) with mongo for a year now.
[15:06:43] <pdekker> lqez: Do you have any thoughts? When I change it back to my local version of mongod on the secondary, it works again
[15:07:16] <StephenLynx> ever example of framework of sorts I saw for this environment is nothing but bloated trash that do nothing but hog performance for no good reason.
[15:09:03] <lqez> It looks like primary(yes, not yet) works fine, but secondary didn't setup properly
[15:09:42] <pdekker> lqez: One runs mongod as a local user, with a local data dir and log. The other is an init script, started using "service mongo start". The config is almost the same, only different dirs and the fork option
[15:11:19] <lqez> pdekker: `Failed to connect to ip-address:port` -> did ip-address:port mean primary's ?
[15:16:51] <mskalick> hi, rs.initiate prints "Config now saved locally. Should come online in about a minute." . How to find out that replication is ready? thanks...
[15:21:43] <deathanchor> anyone know how to aggregate $sum { "key" : { "a" : 9, "b" : 7 ...} } not knowing how many a,b,c,d etc.
[15:22:05] <pdekker> lqez: yes that is the address of the primary. I pasted it here: http://pastebin.com/MnPMEcpa
[15:23:10] <pdekker> mskalick: you can do rs.status() afterwards to check. Also, if everything worked out, on the mongo shell's of the replica members you should see PRIMARY or SECONDARY in front of every line
[15:27:53] <lqez> http://stackoverflow.com/questions/30338804/mongodb-keyfile-between-replicas-throws-permission-denied looks similar
[15:48:07] <dj3000> I am trying to use the java api to search a field for multiple regexes. I can't figure out how to do this?
[15:54:53] <StephenLynx> it is pretty standard give an example using the terminal syntax so you can then use it as a reference and use with your driver.
[15:55:40] <pdekker> lqez: It listened on 0.0.0:1234 (because the bindIp option was off). I tried setting the bindip to the external ip address of the primary, but still does not work
[15:56:10] <pdekker> I would say the problem is on the secondary...but I have to keep all options open
[16:00:48] <dj3000> StephenLynx, here is the basic idea: http://pastebin.com/xWZD7LRK . Except I want the fields to be OR'd, not AND'd .
[16:43:37] <blizzow> I'm getting the WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always' warning. I followed the docs and put in the transparent_hugepage=never option for grub, still got the warning. So I did the secondary way and put options in rc.local and continue to get the warning after a reboot. Is there a boot option that I can throw in for the defrag or a way to force rc.local to be run before mongod starts?
[16:49:06] <Doyle> Hi folks. Readahead question. Setting readahead of 32 is recommended, or at least "found to work well". When setting the readahead for an LVM, shoudl the RA for the devices be adjsuted to match? I believe it shouldn't matter since it's the kernel that's accessing the devices, and the app goes through the LVM.
[16:49:47] <Doyle> I think it doesn't... but it's one of those things that i feel like I should verify.
[17:39:23] <Doyle> I have an issue with MongoDB on CentOS7. After boot mongo reports "WARNING: Readahead for /data is set to 4096KB" despite the readahead being set in the lv config as indicated by lvdisplay and specified with lvchange. If service mongod restart, then it's fine. It seems like the mongod process is launching before the lvm configs are in place... somehow...
[17:43:19] <eggsbinladen> The mongo init scripts are woefully inadequate. A) I'm amazed that no mongos init script exists considering a major benefit is sharding and requires mongos processes. B) it's appalling the defaults for readahead, transparent_hugepage and transparent_hugepage/defrag are not set.
[17:45:20] <Doyle> I scripted my mongod build process. The section after the mongod installation is Tuning. Sets the limits up for the monogd user, disables THP, Sets the TCPKeepAlive to 120, and it sets the LVM ra to 32.
[17:46:00] <Doyle> But mongod inits before lvm... somehow... not sure how lvm launches in the boot process
[17:47:52] <ehershey> so if you can't do that tuning earlier in the boot process than the mongod init script, mongod will warn
[17:49:24] <ehershey> what I want to do to put into the official packages is create new init scripts that will run before the mongod init scripts and do that tuning
[17:49:56] <ehershey> because the other reason it's not in the official packages is changing those settings is probably a bad thing for any software package to do
[17:49:59] <Doyle> I was going to put this in at the top of the monogd init. for dev in (dm-0,xvdb,xvdc,xvdd,xvde,xvdf,xvdg,xvdh,xvdi) ; do RA="$(blockdev --report | grep $dev | awk '{print $2}')" ; if [ $RA -eq 32 ] ; then : ; else blockdev --setra 32 /dev/$dev ; fi ; done
[17:51:28] <Doyle> I think it'll only complain for the dm-0, so I could put just that in since the devices will catch up when the lvmetad hits.
[17:53:12] <Doyle> Going to put this in at the top of the start() function. RA="$(blockdev --report | grep dm-0 | awk '{print $2}')" ; if [ $RA -eq 32 ] ; then : ; else blockdev --setra 32 /dev/dm-0
[17:56:59] <Doyle> next thing after this: figure out why centos user has vim highlighting enabled by default while root does not.
[18:08:26] <dadada> would it be possible to use mongodb as a "replacement" for a normal filesystem, I mean, let's say I want to create my own innovative and unique linux distribution, and I want files to be treated very differently from the "norm", I want to treat everything like a huge database, put files into it, put mails into it, put configs into it, allow tags for everything and so on
[18:09:00] <dadada> would mongodb scale in such a way that it's usable for the kinds of loads that filesystems need to handle
[18:09:19] <Doyle> Yea, putting this at the top of the start() function of the mongod init script fixed my issue.
[18:09:20] <Doyle> RA="$(blockdev --report | grep dm-0 | awk '{print $2}')" ; if [ $RA -eq 32 ] ; then : ; else blockdev --setra 32 /dev/dm-0
[18:10:20] <Doyle> RA="$(blockdev --report | grep dm-0 | awk '{print $2}')" ; if [ $RA -eq 32 ] ; then : ; else blockdev --setra 32 /dev/dm-0
[18:13:31] <dadada> saml: I don't even know that I want a classical filesystem hierarchy with folders, maybe optionally
[18:46:52] <kaseano> hi, I'm using mongo with node, and I have to run two unrelated queries one after the other. Should I open two connections so that they can run simultaneously/asynchronously? Or will they do that on one connection? I tried two connections and it still seems to finish the first query before running the second which is weird.
[18:52:42] <StephenLynx> so you don't need to open a second connection if you wish to run these two in parallel.
[18:59:43] <StephenLynx> do you really, really, really need for these to be completely processed in parallel?
[19:00:22] <kaseano> no not really lol just trying to build good habits. If they can be run in parallel I should try to set it up so they will be
[19:00:55] <StephenLynx> a good habit with mongo is to not try to deviate too much from the defaults.
[19:01:07] <kaseano> theoretically in case I ever did have a raid 10 mongo farm with multiple disks/processors
[19:33:08] <m_e> is there some kind of form builder for mongodb? basically i want to define which fields i want and it generates the form and connects any changes to the DB.
[19:59:07] <polydaic> I know its supposed to happen automatically but for some reason it won't change my field from an index to a non-index field
[20:03:01] <polydaic> StephenLynx: sorry I meant a indexing specification for one of my Collections changed
[20:03:27] <StephenLynx> I know you can remove them after they are set, never fiddled with that though
[20:44:04] <symbol> I guess it's valid if you're updating with a full document instead of a single attribute.
[20:45:55] <symbol> Yeah, I suppose it really is. Seems like the upsert property makes more sense for insert in light of that use case.
[21:45:31] <jamiel> Anyone got any idea how this formula works? http://docs.mongodb.org/manual/reference/limits/#Sharding-Existing-Collection-Data-Size
[22:51:20] <shortdudey123> anyone seen mongo complain about perms on a keyfile when run under systemd but works fine when run via a tty
[23:28:15] <stuntmachine> Does anyone have a sensible starting configuration to use in /etc/mongod.conf for a new Mongo 3.0.4 cluster? I know it creates a default one there but I want to make sure I'm using the best options, and I'm fairly unfamiliar w/ Mongo administration.
[23:41:08] <cheeser> well, the defaults are what they are for a reason. they're a reasonable approximation of the general use case.
[23:41:14] <shortdudey123> stuntmachine: probably good to skim this over so you know what it all does though, http://docs.mongodb.org/manual/reference/configuration-options/