Log file Viewer

[03:37:47] <bros> is there only an oplog to tail in a replica set?

[03:45:35] <Boomtime> bros: seems to be some context missing - are you asking what secondary members of a replica-set need to do? or are you asking about what the primary must provide? or something else?

[03:45:45] <bros> I just want to tail the oplog.

[03:45:51] <bros> It seems that isn't possible in a non-replica set.

[03:45:57] <Boomtime> ah

[03:46:14] <Boomtime> yes, the oplog only exists for a replica-set - but you can have a replica-set of 1 member

[03:46:29] <Boomtime> which sounds silly, but it has to start somewhere

[03:46:55] <bros> silly is what i'm rolling with at the moment, thanks. :) i'm building an in-memory mongodb oplog change repliactor

[03:46:55] <bros> lol

[03:46:57] <bros> don't ask

[03:47:05] <Boomtime> indeed

[03:47:19] <Boomtime> also -> https://github.com/mongodb-labs/mongo-connector

[03:47:40] <Boomtime> don't roll your own, it's more complicated than you think

[03:47:49] <bros> what purpose does this serve?

[03:48:06] <bros> i have mongo-oplog

[03:48:32] <bros> mongo-memory-javascript-connector, i guess

[03:48:50] <Boomtime> depends on what you need the oplog content for - what are you doing with it?

[03:56:29] <bros> modifying documents in memory

[03:56:34] <bros> i want a memory clone of my mongodb docs

[04:06:55] <Slipspace> Any of you guys have experience with asp.net? I have a C# driver query that seems to only returns results outside of the framework.

[07:23:48] <kurushiyama> niemeyer: Is there a way to have an open GridFile returned from a GridFS query utilizing .Apply(change,&result)? The naive approach, result being a GridFile does not seem to work. As of now, I simply store the result in a map, and do an OpenId, but that does a second query I'd like to prevent, if possible.

[10:39:30] <kenalex> hello

[10:43:51] <alexi5> hello ladies and gentlemen

[13:24:26] <bsdnoob> hello

[15:27:51] <bros> is there anyway to make .remove, .update, and .save wait until the oplog operation has been written

[16:06:49] <kurushiyama> bros: What do you want to achieve? If an operation made it to the dataset, it is made durable to the journal already.

[16:07:03] <bros> kurushiyama: i want to replicate mongo changes in memory

[16:07:56] <kurushiyama> bros: Well, you can not change replication meachanics. MongoDB is eventual consistent. Period.

[16:08:09] <bros> ok. i have another idea then

[16:08:18] <bros> by the time it gets to the oplog, it is boiled down

[16:08:23] <bros> all updates become $sets, basically

[16:08:41] <bros> how can i hook collection.update() and get that translation

[16:08:57] <kurushiyama> bros: Well, operations are split into atomic, idempotent operations.

[16:10:02] <kurushiyama> bros: Again: You can not change replication mechanics, aside from changing the source code. This sounds like you want to (mis-)use a replica set for read load distributions.

[16:10:23] <kurushiyama> With immediate consistency.

[16:11:14] <bros> I have an inventory management app that has a ton of client side data. 85% of client data. However, it makes changes on the server, diffs them to the current cache version, sends deltas to clients

[16:11:25] <bros> It has to pull 10-20k docs per single write sometimes to achieve this

[16:11:32] <bros> so I wanted to record the deltas as they happened

[16:13:00] <kurushiyama> bros: Just that I get it right: Your clients make changes, so much is clear. Who makes "diffs", why and where are they cached?

[16:13:53] <bros> I have a huge SPA with a ton of client side interactions, but instead of doing that typcial "make mobile changes first, then let them make their way to the server"

[16:14:02] <bros> I have API calls that make them on the backend, then push them to all clients

[16:14:52] <bros> I guess one mistake I have right now is everything is stored in redis when I most likely need it all in memory, but that's just about as scary as can be

[16:15:30] <bros> For example, if I change the name of an item, and I have 20k items, it has to: do the update, pull the old cache from redis, pull the new docs from mongo, diff them, store the patch, store the new cache in redis, then send the patch

[16:15:57] <kurushiyama> bros: To complicated, imho.

[16:16:11] <bros> it is, but how else would you achieve client side data like that

[16:16:18] <kurushiyama> bros: Long polls from client, with read-through caches.

[16:16:27] <bros> read through cache?

[16:17:25] <kurushiyama> Ok. Say you have a client, an API server which implements a read-through cache, Redis as caching and MongoDB as source of truth.

[16:19:57] <bros> ok

[16:19:57] <kurushiyama> Now, your methods, instead of accessing MongoDB directly, would use a DAO which checks wether there is a valid cache entry for the doc, loads caches and returns the doc if not, returns the doc from cache if yes.

[16:20:18] <bros> all of my reading is done on the client side

[16:20:30] <bros> i use indexeddb for that

[16:20:33] <kurushiyama> If a change for the doc is made, the cache is simply invalidated.

[16:20:41] <bros> how do i rebuild the cache for 10k docs

[16:20:53] <kurushiyama> You do this doc for doc.

[16:21:03] <bros> i have multi updates, etc.

[16:21:23] <kurushiyama> Aye. You have to take that into account.

[16:22:31] <bros> how?

[16:25:10] <bros> my idea was to tail the oplog, but i need to intercept it

[16:25:56] <kurushiyama> Well, one example would be to include a UUID as transaction ID in the change set, do a db.yourcoll.find({transid:someUUID},{_id:1}) and remove the resulting docs from the cache. This would eliminate large diffs and make the cache more fluid.

[16:26:22] <bros> I was thinking something like a last updated timestamp

[16:27:54] <kurushiyama> bros: Does not identify a change set correctly. If you have concurrent updates, multiple change sets may have the same timestamp – even such you do not really want to invalidate. Hard to say without digging much more into the topic.

[16:28:24] <bros> which you're going to try to spin as paid consulting again, right?

[16:29:03] <kurushiyama> bros: Well, I could equally argue that you try to get services for free. ;)

[16:29:16] <kurushiyama> bros: Beggars are no choosers.

[16:29:22] <bros> I'm just asking questions. You're putting answers behind a paywall.

[16:30:41] <kurushiyama> bros: For the last time. Question: Do you give away your services for free? I doubt that. => Ignorelist

[16:38:49] <kurushiyama> bros: Just to make clear in case you did not get it: you are asking for hours, rather days of digging into your use cases, data models, architecture and code to find the optimal solution for you. I gave you answers you could base your own solution on. If you want me to look into your stuff for several days, I want to be paid for that. If you are not up to that, you have to live with what you have. But you are in no

[16:38:49] <kurushiyama> way entitled to any answer at all, much less for free work. And being both agressive and highly impolite. I strongly suggest reading ESR's excellent essay "How to ask questions the smart way" (http://www.catb.org/esr/faqs/smart-questions.html), so that you get an impression where your expectations and attitude are off. End of line.

[16:42:39] <bros> I thought we were just two programmers bouncing off ideas. I was going to make this into a git repo?

[16:42:44] <bros> github*

[17:33:56] <GothAlice> kurushiyama: Some people just like to watch CPUs burn. :'(

[17:35:14] <GothAlice> That user appears to be reimplementing MongoDB entirely in Node just because he can't figure out how to optimize his queries or structure his data efficiently.

[18:27:19] <StephenLynx> GothAlice, hueeeeeeeeeeeeeeeeeeee

[18:27:44] <GothAlice> StephenLynx: Yeah.

[19:07:27] <verwilst_> Hello! Probably a simple question. Let's say you have a collection 'baskets' and one 'vegetables' containing for example {'id_': 'carrot', 'vitamins': []}, etc. In the collection baskets i reference the vegetables. How would i do that? just have 'vegetables': [ {'_id': 'carrot'}, ...] ? Or the whole carrot as it is in the vegetables collection?

[19:07:42] <verwilst_> i'm leaning towards only keeping a link and maybe some metadata and concatting it, ala mysql, but i want to make sure i 'get' the mongo way :-)

[19:07:53] <verwilst_> concatting it on the fly i mean

[19:18:14] <Mastema> if I use gridfs am I going to have to query out of the db the entire file to just get a part of it? even if it's in bson format?

[19:24:43] <Mastema> maybe it would be easiest to just create and keep track of second, third and so on documents as they reach limit?

[19:28:35] <GothAlice> Mastema: https://docs.mongodb.com/manual/core/gridfs/

[19:28:55] <GothAlice> GridFS is a chunked BLOB storage. You can certainly request subsections, and even adjust the chunking size to optimize for different use cases, such as video streaming, etc.

[19:33:28] <StephenLynx> Mastema, most drivers implement gridfs and allow you to request a part of it.

[19:33:51] <StephenLynx> gridfs is just an abstraction that isn't implemented on mongo itself, afaik.

[19:34:05] <Mastema> Yeah, I'm trying to use the mongo c driver...

[19:34:16] <StephenLynx> i dont think it implements gridfs.

[19:41:07] <Mastema> it does, just not the greatest documentation on how to use the driver in my opinion

[21:46:37] <kurushiyama> GothAlice: If it was only that, I'd probably walked the way of a socratic interview... But being condemned for the fact that _days_ of my time have to feed my family is far beyond what I can accept.

[21:50:22] <kurushiyama> Mastema: You should be able to seek the bytes.