Log file Viewer
#mongodb logs for Saturday the 8th of November, 2014
[00:49:36] <Synt4x`> this is saying takes 2 arguments (3 given), but it seems to be the same as the example I'm seeing
[01:01:33] <Synt4x`> guess I will just avoid it by doing c = db.x.find({date < whatever, date > whatever}), then c.distinct('id') on it
[04:29:18] <ningu> if I wanted to bulk load a bunch of data into mongodb, would a sharded cluster help make it faster?
[04:29:49] <GothAlice> ningu: Sharded? Yes. (Replication alone isn't enough, but sharding should do the trick.)
[04:30:23] <ningu> I am using cayley, which is a graph database with different backends, one of which can be mongo
[04:30:53] <ningu> I want to load about 1 billion triples into it. with its 'bolt' backend this takes about 12 hours on a single machine on GCE (16 vcpus, 60gb ram)
[04:31:42] <ningu> but unfortunately (?) it stores almost everything in two collections -- 'nodes' and 'rels'
[04:32:06] <GothAlice> Yeah; storing graph datasets in MongoDB often comes down to a collection of actual data, and collection storing relationships between those entities.
[04:32:27] <GothAlice> http://blog.zawodny.com/2011/03/06/mongodb-pre-splitting-for-faster-data-loading-and-importing/ can provide some reading on optimizing bulk inserts.
[04:32:55] <GothAlice> This was a topic discussed quite a bit on http://irclogger.com/.mongodb/2014-10-29 — my own contributions can be quickly found by searching for the word "locality".
[04:33:32] <GothAlice> And graph datasets earlier today (follow the irclogger.com link in the channel topic).
[04:34:44] <ningu> this is not exactly 'big data' but what might be considered 'medium data' i.e. I suspect it could fit nicely on 3-4 machines
[04:35:14] <GothAlice> I have billions of records totalling 24TiB, and I don't consider myself to be "big data". (I don't query my datasets the same way as typical "big data" installations.)
[04:36:11] <GothAlice> At work we have a few million records and whenever my bosses use the term "big data" I frown at them disapprovingly. ;)
[04:36:25] <ningu> I think this will fit in 1TB or so (not sure) but I don't want it to take 12-24 hours to load.
[04:36:57] <GothAlice> Sharding with careful key configuration (and ordering of inserted data) a la the first link I provided will speed up those inserts.
[04:38:10] <GothAlice> In the case of graph associations, (i.e. {_id: {a: ObjectId('left'), b: ObjectId('right')}}) sharding on _id.a would spread the associations around reasonably well. (And split the insertions across as many shards as you can throw at it.)
[04:38:34] <ningu> ok, then my other question is, what's the easiest/best way to get up and running quickly with a sharded cluster? I'm only one person... it would be nice if there were some ansible scripts or whatever or automate some of it.
[04:41:22] <GothAlice> ningu: My script provides a clear illustration of the process involved in configuring a cluster like that, if you prefer to read code. It's a combination of http://docs.mongodb.org/manual/tutorial/deploy-shard-cluster/ and http://docs.mongodb.org/manual/tutorial/deploy-replica-set/
[04:41:40] <ningu> I have a bunch of free credits on Google Compute Engine so I was hoping to use that, just to save money, since this is very experimental.
[04:42:19] <ningu> yes, reading code is good. in theory I'd like to just press a button and have it up but I don't know if that's realistic or affordable. I'm willing to do some work to set it up, as long as I don't have to individually configure every server.
[04:42:48] <GothAlice> Install the agent on a few machines, and a few clicks later on the MMS interface, if I understand correctly.
[04:43:14] <ningu> so they don't provision the servers, they just configure them. that sounds like what I want.
[04:43:48] <GothAlice> (I use the monitoring and backup features, myself. Streaming offsite replication is glorious.)
[04:45:33] <ningu> so here's a really basic question, then... they're not going to know the structure of my dataset, so I'll have to set up the shard keys myself, I assume. how does one do that exactly with a cluster... is there a master? do you just connect to any one of them?
[04:48:53] <ningu> yes, I was asking about the replication. I am more familiar with rdbms where there is a master/slave setup.
[04:49:01] <GothAlice> Generally you have sharding "in front" of your replication (i.e. a-f goes to a set of three replicas, g-m goes to a different set of three, etc.)
[04:49:40] <GothAlice> You have "routers" at the very front that your application connects to. These routers speak mongodb, but parse requests and try to analyze where to send the query. Sometimes they need to send the query mutiple places, then aggregate the results.
[04:51:23] <GothAlice> (They speak to "configuration" servers which track the cluster's replication members and everything, generally, self-organizes around your sharding keys.)
[04:51:46] <ningu> so realistically for this graph dataset, which is max 1TB (I mean the size of /var/lib/mongodb if stored on a single machine) how many servers would I need to make the bulk loading faster? I know you need 3 config servers for example.
[04:52:40] <GothAlice> And the configuration servers can piggyback on the ones actually storing the data, to save space. Configservers are extremely light-weight.
[04:53:29] <GothAlice> Your basic setup would need only one machine (whole dataset), and can use as many as you want (one per shard division). Replication (high-availability, backup, etc.) is entirely optional (but a Really Good Idea™.)
[04:54:11] <GothAlice> If you replicate (backup, HA) you'd need three servers per shard as a minimum.
[04:55:06] <GothAlice> How you determine your sharding key, and the order you insert your data will have strong impacts in the speed of insert.
[04:56:00] <ningu> I'm not sure if I need HA and backup right now because this is not the master copy of the dataset, and there won't be a ton of people querying it.
[04:56:20] <GothAlice> http://docs.mongodb.org/manual/tutorial/deploy-shard-cluster/ and MMS are your friends.
[04:57:59] <GothAlice> http://blog.zawodny.com/2011/03/06/mongodb-pre-splitting-for-faster-data-loading-and-importing/ is a really good article about optimizing bulk inserts on sharded clusters, though.
[04:58:30] <ningu> I didn't expect to get such detailed information here, it will take a little while to digest everything
[04:58:43] <GothAlice> ningu: ^_^ Apologies if I'm overwhelming. I like to provide comprehensive assistance.
[05:09:28] <GothAlice> https://en.wikipedia.org/wiki/Acoustic_fingerprint Almost all implementations are purely commercial. :'(
[17:13:12] <giuseppes> I'm trying to set up my mongodb instance on a remote server to be reached from my app, how can i get/ configure usr, password and mongo url?
[22:38:52] <giuseppes> I'm trying to set up my mongodb instance on a remote server to be reached from my app, how can i get/ configure usr, password and mongo url?
[23:30:56] <wsmoak> giuseppes: I haven’t done it, but how about this? http://docs.mongodb.org/manual/tutorial/enable-authentication/
[23:40:52] <wsmoak> that seems to be how you enable authentication … otherwise you can only connect on localhost
[23:48:29] <joannac> giuseppes: your mongod is running on the default port (27017)? Can you connect to it remotely? (telnet yourhostnamehere 27017)?
[23:58:52] <joannac> you should now turn on auth, so not everyone can connect to your instance and look through your data
[23:59:46] <wsmoak> joannac: really? I (briefly) read “localhost exception” and thought that _without_ auth turned on, you could ONLY connect via localhost.