Log file Viewer
#mongodb logs for Tuesday the 6th of September, 2016
[00:06:55] <blizzzard> Isn't there a consolidated way to do this say for example, for favorites I have an array of object ID's in schema?
[00:07:16] <blizzzard> I have to submit this task asap and I'm trying to cut corners from anywhere possible
[08:59:51] <cm13g09> morning all - Need to create a replica set out of an existing standalone instance with next to no downtime (a reload is fine, anything else probably isn't)
[09:21:47] <kke> trying to use a TTL index. doesn't seem to work. i enabled TTLMonitor logs and see this : 2016-09-02T08:51:35.257+0000 D INDEX [TTLMonitor] TTL -- ns: dev20160906114743.tokenskey:{ deleted_at: 1 } query: { deleted_at: { $lt: new Date(1472806294257) } } <-- that date is about a week ago, my expireAfterSeconds is 1
[09:22:16] <kke> does it have some setting somewhere that i only want to clean up stuff that is over a week old already or what? :)
[09:46:00] <Derick> cm13g09: https://docs.mongodb.com/manual/tutorial/convert-standalone-to-replica-set/
[09:47:51] <cm13g09> Derick: I couldn't work out if that "just worked" or only worked on a new instance
[09:49:48] <cm13g09> that going to 2.6 was just a case of adding 2.6 node + 2.6 arbiter then wipe and upgrade 2.4
[09:51:02] <cm13g09> My boss' experience of Mongo to date hasn't exactly been stellar (we've had a few random mongodb just dieing for no obvious reason issues)
[09:52:14] <Derick> that works, but it affects performance (you're doing full collection scans for all collection)
[09:52:46] <Derick> tilerendering: plans are that MongoDB might run on ARM64 soon..., but I don't think that's android yet
[09:53:02] <Derick> cm13g09: if you think you can copy the files during down time, that might probably be better/faster
[09:54:00] <Derick> cm13g09: would it be possible to *try* out doing the upgrades on some random extra VMS so that you're familiar with it? like a dry-run?
[09:55:21] <Derick> other servers? For a replicaset set, the three nodes need to be on different VMs - otherwise, it's kinda pointless
[09:55:49] <cm13g09> Derick: we're running 1 in DC1, 1 in DC2 and I'm not entirely sure where to put the arbiter.....
[09:58:32] <Derick> There is also "Major Version Upgrade" at https://www.mongodb.com/products/consulting
[10:00:40] <cm13g09> Just wondering if I can run it on the tertiary NS we have that's not in any of our DC space
[10:04:46] <cm13g09> Machine specs are: 1 vCPU / 512MB RAM / 20GB disk. Currently running at ~50% memory usage, ~10% disk usage
[10:06:46] <cm13g09> Derick: Basically the issue has come about because we need to fix the DC network in DC1, meaning we have to take the whole DC out for a day
[10:08:04] <cm13g09> so I've been told to make the entire platform we have running in DC1, run in DC2
[10:08:14] <nalum> Hello all, has anyone ever gotten an Errno -3 from mongo-connector? I have it setup to sync data between two replica sets
[10:11:18] <nalum> Yeah, I was hoping to get a bit more info as to why it failed. I'm able to connect to the replica set without trouble usign the same connection string
[10:13:53] <cm13g09> Derick: I'm just conversing with the boss about it, he's asking why we shouldn't run the arbiter in either DC1 or DC2
[10:14:36] <cm13g09> Would I be right in thinking that if your site-to-site VPNs went down in the "wrong way", whichever site has the arbiter, wins?
[10:14:56] <Derick> cm13g09: if it runs in DC1, and DC1 goes down, you don't have a writeable set in DC2 as it needs to see the majority of the nodes
[10:16:33] <cm13g09> Derick: of course, I've got my own set of fun anyway, because the core application wasn't designed to be clustered....
[10:19:21] <Derick> cm13g09: your connection string will need the majority of the nodes listed though
[10:22:31] <cm13g09> presumably wait for DC1 to catch up, then shut down DC2 temporarily, forcing an election?
[10:23:28] <cm13g09> equally, I don't much fancy reloading Mongo to enable replica sets in the middle of the day
[10:25:24] <Derick> I'd recommend you test it on your dev environment first if you haven't done it yet.
[10:35:58] <cm13g09> Derick: we're considering spinning up a brand new copy of the service in DC2 now, dumping the data out of DC1 and migrating it over later
[10:37:00] <cm13g09> It'd avoid an interruption of service, at the expense of history going slightly missing for a week or two
[11:08:05] <kke> asking again. i'm trying to use a TTL index and it doesn't seem to work. i enabled TTLMonitor logs and see this : 2016-09-02T08:51:35.257+0000 D INDEX [TTLMonitor] TTL -- ns: dev20160906114743.tokenskey:{ deleted_at: 1 } query: { deleted_at: { $lt: new Date(1472806294257) } } <-- that date is about a week ago, my expireAfterSeconds is 1 there's no other index on the same field
[13:09:04] <deathanchor> what db type is best for doing many <-> many <-> many with possible cyclical matches?
[13:12:52] <deathanchor> I read about neo4j, but since I left my last job which it would have been useful, I never followed up on it.
[13:18:20] <deathanchor> I'm just curious if you think if you would use graph DB soley or RDB or NoSQL, or hybrid for a "recommended items based on purchase" feature?
[13:25:58] <deathanchor> cheeser: yeah, that's what I figured, but some the neo4j docs I read said graphing dbs are "better" for it, but I didn't see how it would efficient unless you precached it to something like mongo and updated the doc in mongo regularly.
[14:04:14] <amcsi> does Linux on the operating level cache the database in RAM so that read/writes happen at RAM speed if the db isn't too big?
[16:08:37] <suicidel> I'm trying to start up mongodb on ubuntu. I get Failed global init: FileNot Oepn failed to open the logfile. I checked the permissions and they are fine. Any ideas?
[16:11:53] <Derick> suicidel: Your error message has typoes in it - can you share the full error in a pastebin.com ?
[16:28:08] <suicidel> @Derick: got it working. It's true . I wasn't starting it as mongodb user, so no access ot log. As a test i chowned to tv and it worked
[16:50:14] <blizzzard> best case would be if we can just have username(string), password(string), favorites(whatnow?)
[16:57:29] <wwwd> I'm trying to start mongdb with a replSet in Ubuntu 16.04. When I issue this command "mongod --replSet zappa --dbpath data/1 --port 27001 --smallfiles --oplogSize 50" I get connection refused...Anyone know why?
[17:06:12] <jokke> is there any performance benefit in using replace_one over update_one when replacing all fields?
[17:31:17] <GitGud> GothAlice, well its specific to you. because i was going to ask if you guys host your own work servers or if you do it remote. because i hear without proper equip, its bad to host it yourself
[17:32:26] <GothAlice> GitGud: We have physical servers managed by another company in a local DC, plus nodes managed in-house for offsite replication / backup purposes. For some clients we use MongoDB Inc.'s cloud services.
[20:37:16] <android6011> I have a collection and each record has a geojson point, i want to run a query that orders them by date_time, and computes distance between each record, to give me a total distance. Not really sure where to start
[20:38:21] <spacecrab> hello folks - i am new to mongodb and have a question about best practices, or rather simple designs... it seems there is a lot of different ways in which i can get off the ground with a database and start working.
[20:38:22] <spacecrab> i'm curious, is it preferred to work in the mongod.conf file, and have all changes defined from there, or should i have my scripts always calling mongod from bash and running with the parameters preferred? it seems they are one in the same but i am not sure which is preferred... any help/corrections/etc appreciated :p
[20:39:18] <spacecrab> essentially i'm simply working to create a 3-member replica set for the first time. i've been watching conference talks and reading documentation about it, but everyone seems to do it a little differently between bash and defining from .conf
[20:58:48] <spacecrab> or further yet, should i just be modifying the init script? i assume the conf should be used for all of this though
[21:14:54] <GothAlice> spacecrab: Production environments should use .conf files; this better protects the security of the configuration itself, given proper file permissions.
[21:16:03] <GothAlice> spacecrab: Command-line arguments can sometimes be exposed via the process list. In other ways, this would let you do things like git-manage the config, ensure configurations are consistent and re-deployable with stock initialization scripts (since most will already look for a /etc/mongod.conf).
[21:16:21] <GothAlice> spacecrab: In development, running mongod through custom shell scripts or directly from a command line is entirely reasonable.
[21:23:13] <spacecrab> GothAlice: i'd rather learn and simply do it from the .conf file(s) then -- understanding the command line parameters was easy anyways it seemed, but the .conf file is actually very specific
[21:23:46] <spacecrab> i'll see if i can set up this replica set strictly through mongod.conf then :) thanks so much for the input! i'll let you know when i succeed
[21:23:53] <GothAlice> The options between command-line and configuration file are equivalent, from what I recall, FYI.
[21:24:48] <GothAlice> Compare https://docs.mongodb.com/manual/reference/program/mongod/ vs. https://docs.mongodb.com/manual/reference/configuration-options/ — they're very close.
[21:25:42] <spacecrab> GothAlice that's what i observed as well -- not 1:1, but the meanings themselves you can derive/associate with little effort
[21:26:16] <spacecrab> i'm reading this right now for the sake of reference/different sources -- is it normal to manually add replica systems via rs.add in the shell? http://www.serverlab.ca/tutorials/linux/database-servers/how-to-create-mongodb-replication-clusters/
[21:26:46] <GothAlice> spacecrab: Yes, or have automation to run that command for you, such as https://www.mongodb.com/cloud
[21:26:48] <spacecrab> i don't understand why someone would want to do that instead of defining it from source/conf... but it seems that everyone favors that option when it comes to tutorials/guides
[21:27:14] <GothAlice> The exact layout of your replica set is a runtime configuration, stored in the database itself, not in a configuration file.
[21:27:55] <spacecrab> GothAlice: oh, interesting. so i should actually be maintaining it with my own bash/python deployment scripts, rather than being concerned about pre-start configuration changes
[21:29:32] <spacecrab> i'll probably dig into it certainly -- but for now i'd like to learn this stuffs :p coming from a history of light oracle DBA stuffs it's very interesting
[21:29:59] <spacecrab> and in a "devops"-y role it's super relevant. haven't had an excuse to touch nosql yet, and from what i've learned so far it's very easy to find one!
[21:30:19] <GothAlice> MongoDB can be… refreshingly lightweight in terms of setup for things like high availability, which can be somewhat more difficult elsewhere.
[21:30:45] <AvianFlu> quick question, if I've got a TTL index on a field, and then update that field before the ttl expires, it resets the countdown, yeah?
[21:30:55] <GothAlice> spacecrab: "Set up three nodes… have them shake hands… and wait, I'm done?" ;)
[21:31:28] <GothAlice> AvianFlu: It's implemented similar to a cron, running once per minute. Any record with a time <= now is cleaned up, so altering the TTL field is a-OK.
[21:32:15] <AvianFlu> so yeah I might lose the race if it's T-1 second when I update it, that makes sense
[21:32:20] <GothAlice> AvianFlu: There are caveats, however. Not all expired records might be cleaned up in a single sweep (it's optimistic), and since it's once per minute, records may easily be stale by up to 59.99 seconds.
[21:32:43] <GothAlice> (So any time you're using a TTL, also check the TTL field when loading data…)
[21:33:23] <GothAlice> cheeser: My crazy auth configs I leave to MMS/cloud. They're pretty… confusing at times, I will admit. T_T
[21:33:56] <spacecrab> GothAlice yeah i was watching some linuxconf talks from a couple years ago, and reading through the docs and i was actually over-thinking that at first... "wait it's that simple?"
[21:34:34] <GothAlice> spacecrab: I've only been a vocal user and advocate until recently. Hopefully my patch will get in sometime in the next few weeks. :)
[21:34:56] <GothAlice> (Allowing negative $push $position indexes to allow insertion from the end of the array.)
[21:35:23] <spacecrab> GothAlice: very cool. well rock on! i've been getting into a couple different projects and it's just as redeeming to see you reciprocating valuable information to new users and potential evangelists, as it is to help out myself!
[21:37:53] <GothAlice> Derick: Some silly bean decided to use size_t for $position offsets. Because a single document can totally store 2^64 array elements. (Hint: it can't. 2 million NULLs is about the limit of a 16MB document. ;)
[21:39:27] <Derick> I don't get the change to INT_MAX either (from std::numeric_limits<std::size_t>::max() )
[21:39:52] <GothAlice> I'm using int, not int_32 C++ mocked structure, so I'm using the C99 INT_MAX def.
[21:42:05] <GothAlice> Derick: In my first local draft I accidentally set that to zero and all my $push w/o $position were suddenly head pushes, not tail pushes. ;P
[21:50:08] <GothAlice> Derick: An array of 2 billion NULLs would take up… a hair under 22.35 GiB — if my math is correct: (4+1+1+1+4+2000000000*(1+10+1)+1+1)/1024/1024/1024
[21:50:50] <GothAlice> (Each number in parens there represents an individual "field" from the BSON spec, e.g. int32 prefix size, CString nulls, etc., assuming a 1-character field name for the array.)
[21:51:36] <GothAlice> (A hair under since 9-digit indexes would take one byte less, 8-digit two, etc.)
[22:04:20] <GothAlice> Derick: Honest question, then, what sizes do customers actually override the 16MB limit to? I'd like to keep that in mind with somewhat accurate figures in the future. ;)