pmxbot IRC Log Viewer

[00:32:37] <Dr{Who}> Anyone have an efficient example or suggestions of tailing a collection that is not capped? prefer C++

[03:27:17] <bcow> how I am querying some documents through the mongo shell on windows, some of the fields are large text fields that are too big to display on the screen, how can I paginate/scroll the resultset from the shell

[05:02:42] <Lapith> Hey all, I am having an odd problem I have mongodb installed on OSX and the php extension installed. When I run new MongoClient(); the connection flag is set to false. I can connect to it in the shell and using umongo.

[05:03:10] <Lapith> I have not seen anything useful from php logs or debugging.

[05:09:49] <Lapith> anyone around to help?

[06:27:16] <llac> thinking about setting up a mongodb instance for a high traffic messaging system, coming from a traditional mysql structure, really don't need the relational overhead and it seems like a great match for mongo

[06:28:47] <llac> messaging entry {user_id:1111, inbox:[{sender_id:2222, messages:[{direction:'to', time:unixtime, body:'message content']}]}

[06:29:28] <llac> I'm wondering about when users get multiple threads with potentially thousands of messages per thread

[06:30:15] <llac> I've tested this structure with test data and it seems plausible

[06:30:34] <llac> but at scale I'm afraid we'll run into a lot of issues

[06:30:57] <llac> we're talking about 4+ million inserts daily and 100+ million reads

[06:31:28] <llac> the reads can have a caching layer (memcached) to keep reads from hitting the cluster

[06:31:52] <llac> also planning on sharding with replicas

[06:31:57] <llac> so...

[06:32:03] <llac> is anyone in here?

[06:33:13] <llac> also your average query will be little more than db.message.find({user_id:1111});

[06:49:20] <_aegis_> llac: what sort of issues do you think you'll hit? :)

[07:20:21] <llac> _aegis_: well I'm concerned with memory usage and indexes

[07:20:41] <llac> _aegis_: also nested updates and queries

[07:30:22] <_aegis_> llac: more servers! :)

[07:33:52] <llac> _aegis_: oh we'll have plenty of servers all of which running 64GB of RAM and SSD's, planning on a 4 relica sets with sharding across them

[07:34:26] <_aegis_> you can write some tests for this and infer some things about scale

[07:35:01] <llac> _aegis_: that's what I'm working on now actually, just hoping to get some insight from others who are less green than myself with the tech

[07:35:41] <_aegis_> mongo's good at queries on nested keys

[07:36:09] <llac> _aegis_: we currently have 100's of millions of rows of message data and it's just a mess, I was hoping to keep data relative to the user to which it belongs

[07:36:55] <llac> _aegis_: so rather than having 100+ million rows we have 10 million rows with a few thousand related messages

[07:36:58] <_aegis_> you could snag a box with a crazy amount of ram and try importing a large chunk of your data

[07:37:03] <_aegis_> they aren't really rows :)

[07:37:18] <llac> _aegis_: I know, I'm just coming from mysql land.. *documents*

[07:38:27] <llac> _aegis_: initially I was looking into each user essentially having their own collection

[07:38:40] <_aegis_> seems excessive :)

[07:38:42] <_aegis_> it's all bson

[07:39:06] <llac> _aegis_: well that and come to find out if one of the config servers drops for whatever reason collections can't be added or removed

[07:39:16] <_aegis_> :)

[07:39:32] <llac> _aegis_: so new users to the system would be messageless, and that just seems like a nightmare

[07:40:07] <llac> _aegis_: if the nested data doesn't scale well I'm having trouble justifying a highly normalized schema in mongo

[07:40:25] <llac> _aegis_: we're pretty good with mysql, this is a pilot project

[07:44:23] <_aegis_> you read up on shard keys?

[07:49:55] <llac> _aegis_: yeah still trying to come up with a solid key pair

[07:50:25] <llac> _aegis_: we have highly active old accounts, hardly used new accounts, etc. was going to shard on userid but that will lead to hot spots for the newest users

[07:50:34] <llac> _aegis_: which are also fairly active

[07:51:10] <llac> _aegis_: might do a secondary shard key based on a modulus of the userid

[07:51:48] <llac> _aegis_: keep users spread across the cluster, but as said, still "thinking on it"

[07:52:41] <llac> _aegis_: also might chunk newer users across a couple replica sets of their own

[07:53:48] <llac> _aegis_: shard1 = [0,200000}, shard2 = [200000,400000}, shard3 = [400000,500000}, shard4 = [500000,}}

[07:54:12] <llac> _aegis_: something like that to ensure the more active servers have smaller datasets that can be kept in memory

[07:55:02] <llac> _aegis_: but there will be active users across the cluster regardless, some accounts will be days old, some months, some years.

[07:55:09] <_aegis_> you really don't need to highlight me every message :)

[07:55:19] <llac> habit.

[08:34:35] <Dr{Who}> do I need to read the entire document to update it or can I just update sub value of a document using dot notation in C++? Cant seen to find any examples of doing an update adding say an array to an existing document and not overwrite the entire doc.

[08:42:16] <_aegis_> Dr{Who}: $set with dot notation?

[08:42:52] <_aegis_> {$set: {'doc.value': []}}

[09:10:04] <Dr{Who}> cool thanks. I think I converted that to BSON c++ shall see when I test.

[10:12:33] <joshua> Mongo support is pretty good. I guess the follow the sun thing helps

[11:52:26] <johnny_2329381> hello, i have a question: my secondary node wirtes in its logs "connection refused because too many open connections: 16000" , but a "lsof -i -P -n | grep mongodb| wc -l" shows me only 60 connections (all established)

[12:01:24] <oskie> johnny_2329381: usign PHP to access mongo?

[12:06:42] <johnny_2329381> no wi am using ruby on rails

[12:07:59] <johnny_2329381> i am wondering about the gap between mongodb.log ~16000 open connections and lsof, that shows me ~60 connections

[13:02:11] <One-Man-Bucket> hi, i'm using the reflectiondbobject for my 2 pojos

[13:02:20] <One-Man-Bucket> and mongo seems to store everything allright

[13:02:39] <One-Man-Bucket> but how do i fetch a document as a MyClass (that extends reflectiondbobject) ?

[13:10:27] <One-Man-Bucket> ah, apparently collection.setObjectClass(MyClass.class)

[17:55:27] <JakePee> If i'm writing a php mongo wrapper for a dev team, should I be abstracting out a base connection (MongoClient instance) that will be used across the session or multiple connections made and allow the connection pooling to sort it all out

[19:30:04] <stoneg> have mongodb running on SSD, but got very poor performance https://gist.github.com/65321285e9ab92ba4f33 , any suggestions /

[20:01:55] <Goopyo> is a count on collection fast? Compared to a count on a query

[20:02:07] <Goopyo> I now 2.3 is going to fix it but in 2.2

[20:40:09] <VinSpee> hey all, how can i remove a collection with a special character?

[20:40:21] <VinSpee> ex: db.new-accounts.remove();

[20:40:30] <VinSpee> obviously doesn't work, but what will?

[20:41:21] <kali> db["new-accounts"].drop()

[20:41:37] <VinSpee> excellent

[20:41:38] <kali> i assume you want to drop the collection

[20:41:39] <VinSpee> thanks kali

[20:41:40] <VinSpee> yes

[20:41:47] <kali> good idea.

[20:42:07] <VinSpee> special chars are a no-no, correct?

[20:42:57] <kali> no_no :P

[20:43:49] <kali> it does work. but they are thing so much more interesting to do than struggle with the syntax

[20:45:13] <VinSpee> awesome. thanks

[20:56:54] <stoneg> exit

[21:27:48] <astro73|mal> is there a way to mount GridFS that doesn't sound like a half-written hackjob or unmaintained?

[21:29:39] <astro73|mal> gridfs-fuse has a large pile of unsupported bits, and last committed 2 years ago

[21:30:00] <astro73|mal> even its forks have nice bits like "password sent through the clear"

[21:32:14] <astro73|mal> gridfsfusepy is new and still has missing features

[21:32:38] <astro73|mal> and apparently GridFS removed some directory support, which i'm not sure the FUSEs handle

[21:33:01] <astro73|mal> even nginx-gridfs has server-stopping bugs

[21:34:45] <astro73|mal> or is that asking too much?

[21:46:38] <jY> 10gen says don't use nginx-gridfs

[21:55:38] <astro73|mal> jY, alternatives? details?

[22:24:02] <thomasreggi> I wanna update multiple separate queries can I do it?

[22:24:22] <kenneth> hey there

[22:24:40] <kenneth> what happens if you issue an update on a record with an empty update object?

[22:25:42] <Antiarc> It would probably clear out the document. Easy enough to test.

[22:26:46] <Antiarc> https://gist.github.com/4457967

[22:36:45] <thomasreggi> Is this asking for a lot? https://gist.github.com/4458046 (Mongo Update $or)

[22:40:19] <thomasreggi> I forgot multi

[22:40:27] <thomasreggi> Yay for figuring it out myself!

Log file Viewer

Help | Karma | Search:

#mongodb logs for Friday the 4th of January, 2013