Log file Viewer
#mongodb logs for Wednesday the 25th of February, 2015
[00:00:49] <ra21vi> and will I be benefited with sharded mongo environment for geo queries, or will it just slow a little bit
[00:19:59] <morenoh149> we have 2d-shpere indexes and mongo can be replicated which increases read speed
[00:20:55] <ra21vi> oh nice ... i thought it will slow down. I was choosing postgres gis, but really want to be stick with mongo since I do not need complex geospatial queries
[00:24:11] <morenoh149> I'm not sure but sharding could increase read speed for certain operations. But it's heavily dependent on your read patterns
[00:25:03] <morenoh149> sharding is usually a neccesity not a performance consideration. You shard when you data is too large to fit in a single machine
[00:28:05] <morenoh149> I would rather do more replication in your case and forego the sharding if the data fits on a single machine
[00:28:58] <ra21vi> yes that seems possible.. my data wont be much.. right now in postgres its ~8GB.. in mongo, at most it will be around 12GB.. that I can have on single machine
[00:38:01] <morenoh149> I have an 8gb box at home. I'd rather run that for some of these prices. But there's no redundancy, replication, sharding or failover
[00:42:39] <Boomtime> morenoh149: you could do cheaper by hiring an instance on AWS (or wherever) and managing it yourself
[00:43:37] <menn0> i'm adding a new aspect to our application that's going to be quite write heavy (logs)
[00:44:03] <menn0> i've done some initial benchmarking and there's a signifcant impact on the rest of the application (expected)
[00:44:16] <morenoh149> Boomtime: have you tried azure yet? I'm looking at the more and more recently
[00:45:18] <menn0> both are feasible options within our architecture but i'm wondering how much that would help
[00:46:45] <menn0> morenoh149: separate box is less feasible :) i'm just wondering about getting the most out of a single machine
[00:47:42] <menn0> morenoh149: sure. i'm wondering if there's a win to be had (in terms of locking etc) by using a separate db/mongod instance and if so, which option might be the best to persue.
[00:48:00] <morenoh149> ah that's tougher. What's the effect of running two mongo processes on a single machine? it's probably worse due to the overhead of having two processes instead of one.
[00:48:38] <menn0> morenoh149: i'm going to benchmark this anyway but am trying to save myself some time by asking here first
[00:49:42] <menn0> morenoh149: that makes some sense. i'll start with the separate db approach then and see what that buys me.
[00:50:50] <morenoh149> I think it really depends on the io blocking. If the logging and app queries are constant fighting over the lock then two process on a single machine may help. Really will depend on what the OS does.
[00:51:32] <morenoh149> that being said it'd defintely help if you could just do the logging on a small hosted mongodb instance somewhere else
[00:54:31] <morenoh149> it's a really interesting question. I'd love for others to chime in. Or for you to share your results.
[00:56:02] <menn0> morenoh149: logging to an external service isn't really an option for this product
[00:57:43] <menn0> morenoh149: it's more about the content of the logs. they need to be kept private.
[00:59:01] <morenoh149> how private? the data in something like loggly or mongohq is yours. Unless your passwords are compromised
[01:00:44] <morenoh149> also what are you logging? It sounds like application level events. Because mongodb has facilites for managing it's own logs.
[01:06:54] <bros> Where would you store 10k+ documents of 20kb+ base64-encoded data? In files, then store the locations in Mongo?
[01:11:26] <bloudermilk> Hi, all. I'm currently architecting a solution for the somewhat complex problem of recursively archiving data in MongoDB using NodeJS. I'm soliciting feedback on my architecture if anyone is interested in giving me their two cents. Thanks :) http://stackoverflow.com/q/28709172/29297
[01:11:26] <Boomtime> you don't even need to bother with base64 encoding them, mongodb can store bindata directly
[01:15:29] <Boomtime> bloudermilk: your question is too broad, you don't provide any metrics for how many books/chapters/pages you are expecting, you need to set some expectations on data load before you can architect anything
[01:15:29] <Boomtime> also, this statement: "The data model was designed in a way very similar to what you would expect to see in a relational database" <- if you approach mongodb with a relational mindset you are going to fail hard
[01:15:29] <bloudermilk> @Boomtime thanks for the feedback. Most collections contain many millions of records. The reason I didn't specify is that the solution must be generic enough to work for several places in our model graph, which is ~20 models and would have detracted from the question. Well aware of the fact that the way the data is modeled is not suited to MongoDB–before my time, sadly
[01:19:31] <Boomtime> well, generic questions elicit generic answers - a relational model in a document store will be painful forever, especially given that you have a natural embedded document structure unnecessarily spread over multiple collections
[01:19:32] <bloudermilk> Boomtime: generic answers is all I'm looking for, but changing the data model at this point is out of scope. If you have any feedback on the problem at hand I'm all ears
[01:31:41] <Boomtime> bloudermilk: you should look into the aggregation framework, it can probably short-circuit some of your work
[03:33:50] <menn0> morenoh149: some initial findings. logging to a separate DB (not separate mongod instance) makes a massive improvement for our app.
[03:44:58] <menn0> cheeser: using a separate DB for logging is no problem for this application so this is already a big win for us.
[08:36:23] <rawl79> The repair (repairDatabase) command, can you specific a different path for storing temporary data?
[09:53:05] <Cygn> Hey everyone, i am new to mongo and need a hint in the right direction. I have a table "dataChunk" which contains data like the following: http://pastie.org/9980768, i need to get the "original_value" of the last entry (unit_identifier=10) when items with the same dataId and different unit_identifiers have certain values (f.e. with the unit_identifier 7 has a certain date). How would you solve this in general?
[10:07:07] <jaitaiwan> If I understand it correctly, you just want to sort by descending on the unit_identifier field?
[10:09:08] <Cygn> jaitaiwan: there are 10 entities in the example, all connected via the attribute dataId, i need to fetch the value of the original_value attribute, where the unit_identifier attribute is 10, but only when the (via the dataId) connected entity with the unit_identifier 6 has a certain value (f.e. a date older than today).
[10:10:35] <jaitaiwan> Cygn: that's a VERY relational request... not something mongodb would be well suited to without a large performance hit on possibly the application side.
[10:14:05] <jaitaiwan> bishwa: I would think that they would both write lock and infact probably do the same thing. One is the command line equivalent of the other.
[10:14:49] <Cygn> jaitaiwan: i could also implement the data in another way, since i am fetching it from another (relational) database. Goal was to get more speed by setting up the data to mongodb. How would you structure the data to get the same information using a mongo request… having f.e. the data i have there right now in one row? (unit_identifier would be the column, original_value, the value)
[10:16:35] <jaitaiwan> Cygn: I'd probably combine the data into the one document if that makes sense
[10:20:33] <jaitaiwan> Cygn: if I had a more functional understanding of the data, I could better inform its structure
[10:22:42] <Cygn> jaitaiwan: this is a sale of a product. Contains the market (first entry), the product number, (second entry), the name (third entry) etc. - you would now maybe like to ask why the hell i did it that way in the first place - there will be more data, thousands of entries with different attributes. For example some will have a price, others not (like this one) and i wanted to have a way to handle any amount of additional attributes.
[10:23:47] <Cygn> jaitaiwan: unit_identifier, identifies the type of attribute (1 for market, 2 for product number and so on), original value holds the value and dataId connects the sales attributes to each other… at least that was the plan ;)
[10:23:55] <jaitaiwan> Cygn: in mongodb, the structure isn't set, one document can have a field where another doesn't. You should have a products collection, with one document per product.
[10:24:23] <jaitaiwan> As far as the attribute identifier you could simply use the number as the index
[10:27:16] <Cygn> jaitaiwan: oh my god - this is so sexy i wanna have kids with it. Thank you i will try it out NOW.
[10:28:15] <sabrehagen> hypothetical: i have an object o with three keys a, b, and c that i want to store in mongo. what happens if somebody decides to post data to my api route that saves the object to mongodb with a huge object full of many other keys? surely it gets saved. the only way i can see of stopping this is imposing a schema on the data, which is opposed to the schemaless paradigm
[10:30:38] <jaitaiwan> sabrehagen: Depends on the language your using. Mongodb uses binary objects so depending on the language unless your specifically turning into an object.
[10:41:44] <Cygn> jaitaiwan: MongoDB just increased our execution Time from about 6 Minutes to 20 Seconds. - Thank you very much and keep on the good work !
[10:50:37] <Cygn> jaitaiwan: i am just preparing a giant insert statement, i will give you a look afterwardes
[11:52:59] <Cygn> jaitaiwan: This is an example of the Result by now, in the future there will be datasets with more or less attributes :) http://pastie.org/9981002
[11:57:51] <Cygn> jaitaiwan: Would be the next step, right now i am thrilled to see my first full request working, i am just adjusting the backend. Fine-Tuning Afterwards.
[12:10:03] <Cygn> jaitaiwan: actually, there is a nice out of the box package for mongodb and laravel
[12:20:00] <jaitaiwan> Cygn: I don't know how much you value speed but I know that using phalcon has allowed me to scale at a much lower cost
[13:12:54] <StephenLynx> afaik, you have to do it on your application code with the object you get from the query.
[13:16:23] <iksik> StephenLynx: no way to hack it somehow? i'm just playing around with data inside of robomongo
[13:18:11] <jaitaiwan> StephenLynx: was it you and GothAlice we were discussing "best languages" about? Phalcon is a php "framework" which is a php c extension. Means it helps reduce a lot of the speed issues that most php frameworks face.
[14:56:06] <StephenLynx> If we couldn't eradicate the plague, I don't think we will be able to do it with PHP.
[14:56:21] <GothAlice> (Writing my C10K HTTP/1.1 server, I tried borrowing a C extension from a Ruby HTTP server for header parsing. Turns out the prototype 12 lines of pure Python I wrote were about 60x more performant than the C extension I tried to borrow. And all it was doing was upper-casing the keys, replacing hyphens with underscores, and prefixing the keys with "HTTP_". ¬_¬)
[14:57:11] <GothAlice> (And unwrapping the headers, if wrapped. Yay line length limits in protocols!)
[14:57:52] <StephenLynx> GothAlice it could be for a number of reasons. I heard that V8 has an overhead if you swap from JIT and native code.
[14:59:01] <StephenLynx> I don't know why you had this experience in particular, but I know that C is not slower in general than these two languages.
[15:00:22] <GothAlice> https://github.com/marrow/server.http/blob/develop/marrow/server/http/protocol.py#L139-L151 vs. https://github.com/marrow/server.http/blob/e6468ecfc01808ce66f90411dae836b14f6f773c/marrow/server/http/parser/http11_parser.c
[15:04:44] <GothAlice> The earlier comment about the PHP framework having a C extension. C extension just means part of it is written in C, not that that inherently makes it better or faster. ;)
[15:06:11] <GothAlice> HippyVM (alternate PHP implementation written in RPython) does a pretty good job on the polishing.
[15:13:53] <GothAlice> Instead of eradicating bad languages from the world, I want a common runtime for them all. Then we can have parts written in whatever language people are comfortable with and everyone, regardless of language, could benefit from the good bits. Thus my being such a fan of the RPython interpreter projects. There are already projects implementing Ruby, PHP, JavaScript, Lisp, Smalltalk, Scheme, and IO, with runtime back-ends for C, .NET, JVM,
[15:18:22] <GothAlice> (Compiled to ELF using a compiler written in BF, of course. For binary datastream processing, it does a pretty reasonable job!)
[15:19:09] <StephenLynx> and a single runtime will be nothing but a huge unmaintaineable clusterfuck
[15:20:22] <StephenLynx> not to mention that productivity of people working on projects that use multiple languages would decline because of the context switching.
[15:21:27] <GothAlice> I don't really see a reason why that must be the case. Compilers and runtimes are different things, and as listed above, I've already got compilers for a variety of languages that can emit bytecode (or native code) for multiple extant runtimes. When you're consuming an API typically you have objects, methods, and bare functions. These are pretty common among languages, so as long as your documentation is reasonable, there'll be no real
[15:24:05] <GothAlice> (At work we use Java NLP libraries from our Python code running in the JVM. It's pretty neat to be able to do this.)
[15:30:49] <GothAlice> weka = jpype.JPackage('weka'); self.dataset = weka.core.Instances(java.io.FileReader(datasetFilename)) # Using Java from Python is actually this easy. ;)
[15:46:02] <NoOutlet> Sorry. I'm referring to the logged conversations which -pmxbot- alerts us to upon entering this room.
[16:26:29] <bros> I have a collections called orders. orders has a subdocument which is an array called log. I push timestamps into log. How can I select all orders within a certain time range?
[16:49:02] <valeri_ufo> hi guys! is anyone running mongodb configs with the puppetlabs provider? is anyone aware if the provider can handle auth enabled mongo?
[16:50:58] <elm-> hi, a quick question were I could only 2 year old info online: Is there a way to speed up draining of a shard? I'm waiting already 4 hours for a shard of 20GB to drain, it does so a rate of maybe 2-3 Mbit/s which seems way to low for me
[16:56:52] <bros> StephenLynx, cheers. Quick question. In MongoDB, the fields are dates. Can I simply pass another date object in log.time $gte start_date?
[17:37:29] <freeone3000> I have a SECONDARY whose oplog is not catching up to PRIMARY. What could be causing this?
[18:00:18] <GothAlice> Welp, I know the next marrow project for me to NIH. A MongoDB migration framework based on my five years of migration experience. What's everyone's experience with both early rapid development cycle changes and longer-running datasets?
[18:02:39] <GothAlice> I've been rolling the occasional one-liner like User.update({'oid': {'$exists': True}}, {'$unset': {'oid': True}}, upsert=True, multi=True) or Source.objects({'se': {'$type': 3}}).update(set__settings=[]) when needed. :/
[18:02:51] <GothAlice> StephenLynx: https://github.com/marrow/ < my "open source collective" namespace
[18:04:36] <GothAlice> WebCore is special, it's namespace is just "web". Yes, I'm namespace-greedy. ;)
[18:14:38] <freeone3000> How can I force mongo to ignore intervening oplog entries and simply get the current state of the database to primary, and update itself accordingly? I've tried getting the data from a snapshot, but taking the snapshot took longer than my oplog delay, so pre-seeding doesn't seem to have worked? Is my only option to delete the data and try again?
[18:16:28] <GothAlice> You could increase the size of your oplog and try pre-seeding again. You may even be able to bootstrap (and only have to reconfigure this way) any other existing and up-to-date secondary, if I recall correctly.
[18:18:50] <freeone3000> How can I convert from oplog hours to oplog time? Current state is 10GB == 5 hours, so for 24 hours I'd need 50GB?
[18:20:11] <GothAlice> StephenLynx: Aye. They're "backed" by two companies in Canada here; much of the recent code is exports from commercial work.
[18:21:11] <GothAlice> (Illico Hodes here in Montréal, and Top Floor Computer Systems Ltd. over in BC.)
[18:25:13] <GothAlice> Indeed; it's also pretty much the Python capitol of Canada. So many conferences hosted here…
[18:28:38] <GothAlice> I'm living in Montréal because of PyCon Atlanta, and the CCP after-party. ¬_¬ Presenting at ConFoo for the two or three years I did it was easier once I lived here. XP
[18:38:41] <GothAlice> Indeed; in the last 10 years they changed some of the rules. My parents needed new papers.
[18:39:14] <StephenLynx> there used to be something called workstudy that gave you a working visa, but it was finished.
[18:39:43] <StephenLynx> but once you get an offer, it is pretty fast, AFAIK. 5 years for full citizenship more or less.
[18:42:35] <Nepoxx> I'm actually from Montreal, so it's not an issue for me. I was wondering if it was much easier to get in, than for me tog et in the US
[18:42:51] <Nepoxx> I had a job offer in Seattle, I could have got a TN easily, but not so much for my spouse
[18:46:13] <jimmie_fulton> We have a Brazilian in our office here in San Francisco. He plans on staying...
[18:47:31] <Nepoxx> I'm in Quebec City right now, I absolutely love it, but there's virtually no tech community whatsoever :(
[18:48:13] <Nepoxx> StephenLynx, it really depends where and what you do. I was a Software Engineer for a "big company" in Seattle and I got 125k fresh out of college
[18:48:53] <Nepoxx> I'm making less than half that now, thanks to Canadian (and Quebec) economics :P
[18:50:02] <StephenLynx> i could literally live three decades with what you earned in a single year.
[18:52:40] <StephenLynx> did you know USA is pratically the only country to not have a law giving mothers a license for maternity?
[18:53:04] <ralphholzmann> my database (with journaling enabled) did not shut down cleanly, now the logs say it can't open the database file
[18:53:34] <jimmie_fulton> Yep. 125+. There are so many startups and software companies here, I can't even count: Just off the top of my head... Google, Twitter, Splunk, Zynga, Saleforce (on every single block), GitHub (around the corner from my house), Yelp Square, IGN, Zendesk, Yammer, DropBox, Riverbed, FitBit, Ripple Labs, MuleSoft, Clustrix...
[18:54:13] <ralphholzmann> http://docs.mongodb.org/manual/tutorial/recover-data-following-unexpected-shutdown/
[18:55:33] <Nepoxx> StephenLynx, I know that. I'm not going to comment, we most likely shouldn't discuss politics in the mongodb channel :)
[19:09:55] <doxavore> have a replica set with 1 node is doing a full resync. that crashes during index build (seems it runs out of physical memory without touching swap). short of taking my cluster down to get a snapshot, is there anything else we can do?
[19:10:56] <doxavore> (i've only had bad times trying to get mongo to apply rs config changes with only 2/3 servers up)
[19:11:42] <ralphholzmann> seems like rm -rf-ing the journal folder, then doing a repair, then starting up worked
[19:26:19] <doxavore> freeone3000: i'm not sure, but maybe follow the "resize oplog" instructions, but when creating it, don't provide the capped info, just create the collection?
[19:26:54] <doxavore> afaik you can only set the capped size on collection creation, so if it exists before then...
[19:28:43] <doxavore> in theory, yes. but when a new node comes online missing all of its data, it will start a sync from an existing node, and only needs the oplog to not run out of space between the time it STARTS recovery and the time it finishes and gets caught up.
[19:32:52] <freeone3000> Okay. That's not happening to my nodes - they start recovery, get into STARTUP2, and sit there for days. If I pre-seed with a backup, they get to a point five hours after they started RECOVERY, then their oplogTime doesn't increase.
[19:34:17] <freeone3000> It's at the point where my next step is to launch a new, identical cluster to get the thing into the same state.
[19:35:11] <doxavore> freeone3000: do you see anything in the logs? something about oplog being too old, etc? (i just ran into this the other day...sigh)
[19:35:33] <freeone3000> doxavore: It's more than 6 hours behind and I only have a 10GB oplog, though.
[19:35:52] <freeone3000> doxavore: But that doesn't explain why a force resync (which discards current db state and tries to start fresh) doesn't go past STARTUP2, either.
[19:37:31] <doxavore> i have a node now that can't resync because mongod doesn't handle an index that doesn't fit fully into physical (eg not swap) memory
[19:39:55] <doxavore> so the server you're restoring to syncs up to the point of where it started the sync, and then doesn't do anything but sit in STARTUP2? but mongod process keeps running on that server?
[19:45:01] <doxavore> freeone3000: sorry, perhaps someone else has an idea, but the only time i've seen that it is spitting out messages in the log that it's too old to catch up :-/
[19:46:44] <fastfinger> Hi. I’ve been trying to use $addToSet in a findAndModify query using the node.js api. But I’m receiving an error that the prefixed field $addToSet is not valid for storage. The query works using the mongo shell client, but just not with the api.
[19:48:25] <fastfinger> Basically: db.collection(‘users’).findAndModify({‘username’: ‘foo’}, [], {$addToSet: {roles: “somerole”}}, {upsert: true, new: true})
[19:54:00] <edrocks> does anyone know why my db isnt showing up when I use "show dbs"? I'm using a replica set and I have 2 dbs both with data but one doesnt show up
[20:14:08] <MLM> In my model, would it be better to only have a `dateCompleted` Date field or also add a `isCompleted` Boolean? I feel like the `dateCompleted` Date field by itself is better so I don't have any maintenance complexity
[20:28:47] <MLM> cheeser: Oh, awesome! Haven't used `$exists` yet. http://docs.mongodb.org/manual/faq/developers/#faq-developers-query-for-nulls
[20:30:54] <MLM> I assume you can also use it the way you showed but I can't find any example in the documentation. The use-case I can find is:
[20:32:36] <cheeser> i might've gotten it swtiched around. i always have to look it up. i don't often work in the shell. ;)
[20:47:11] <MacWinner> if I have 3 documents which has a "tags" attribute which is just an array of tag words, and I want to find the documents with the most number of tags matching to a search, is there a simple way?
[20:47:57] <MacWinner> and if there is a matching number of tags, return all the docs.. not sure if I explained that clearly
[20:57:19] <freeone3000> I'm getting "dirname: missing operand" when starting mongod, and starting mongo fails, with no logs. The IGNORECASE fix in AWK is already applied, as well as the --check command to daemon. How should I try to fix this next?
[21:03:57] <morenoh149> MacWinner: you can match by an exact set of tags easily http://docs.mongodb.org/manual/reference/operator/query/#array
[21:04:20] <morenoh149> the part where you don't have a match but want the closest match will be considerably more difficult
[21:24:38] <freeone3000> Okay, turned out it was a bad configuration parameter, reported poorly. Startup script should probably not redirect all output to dev null...
[22:05:25] <wavded> if a property is sometimes an array and sometimes a string, is there a way to update only the non-array properties?
[22:06:00] <GothAlice> wavded: Yes, though this type of mixed-type shared-key situation is strongly discouraged within a collection.
[22:06:38] <GothAlice> Use $type in your query to select for the documents whose given named field is of a certain type. http://docs.mongodb.org/manual/reference/operator/query/type/
[22:06:52] <wavded> GothAlice: i agree, they should be all arrays, I think something happened in the ODM layer
[22:07:09] <GothAlice> I use this during migrations, if I'm trying to find records to update and move over. :)
[22:08:21] <wavded> GothAlice: thanks! I guess I've never done this either but how to you query a field for a value and a type? is there like a $value operator?
[22:13:07] <wavded> db.sales.find({ 'acquisTM.value': { $eq: 'No Consessions' }, $where: "Array.isArray(this.acquisTM.value)"}, { 'acquisTM.value': 1 })
[22:13:08] <GothAlice> Source.update({'se': {'$type': 3}}, {'$set': {'settings': [...]}}) # Common migration pattern.
[22:13:41] <wavded> http://docs.mongodb.org/manual/reference/operator/query/type/#querying-by-data-type
[22:16:35] <codetroth> I am having trouble figuring out how to store a subdocument. Basically I want to have a top level document which has the id of a conversation, and then the messages saved in sub documents under that
[22:18:21] <GothAlice> fewknow: A 16 MiB BSON document can store between 13 and 15 million words. If a single "conversation" reaches that limit, there are other issues. ;)
[22:19:02] <GothAlice> codetroth: This is a pretty common pattern. My own forums store threads this way. Any particular issue or question?
[22:19:37] <fewknow> GothAlice: yeah i agree, but if you want to run stats or anything on the messages in the conversations it is easier with bucketing
[22:21:02] <codetroth> yea I think I will go back to MySQL because none of that made any sense to me...
[22:23:36] <codetroth> Basically I can do the initial write of a document and the first sub document.
[22:24:00] <codetroth> but not how to specify what parent document the child document will attach too
[22:25:00] <fewknow> what is the json structure in mongo that you are getting after the first write?
[22:25:27] <GothAlice> So, in plain mongo shell syntax, you'd do this: db.conversations.update({_id: ObjectId(id of conversation)}, {messages: {$push: {… message sub-document …}}})
[22:28:47] <GothAlice> fewknow: In an array, there's $elemMatch, as well as the various $ operator tricks.
[22:29:21] <fewknow> GothAlice: yeah...but you can avoid if they are all sub docs by just referencing the "key", right?
[22:29:55] <GothAlice> fewknow: Aye, but then you lose any possibility of indexing the data. Having sub-documents with their own ObjectIds is a good approach, and lets you individually update them, etc., etc.
[22:31:32] <fewknow> GothAlice: how do you losing indexing? {_id : id, messages : { message_id : {}, message_id:{} }}
[22:31:39] <GothAlice> {settings: [{key: "google-analytics", value: "GA-373640-2"}, …]} — You can index settings.key and rapidly get documents which have a Google Analytics tracking key set on them. $exists, not so much.
[22:32:23] <GothAlice> Requiring you to use $exists to find the document which contains a message of the given ID. Not indexable.
[22:33:23] <fewknow> i use that structure for stats...but the only query is for the entire document....then the application does the heavy lifting
[22:33:40] <GothAlice> This distinction has bitten us at work. One of our migrations is to convert a {objectid: {…settings…}} mapping to a [{target: objectid, … settings …}, …] array.
[22:35:33] <GothAlice> codetroth: Oh! Say you want to know the average number of back-and-forth messages in a conversation. Pre-aggregation would help this by having each top-level conversation store a value that is the number of messages. (No need to re-count each convo each time you get the average.) When you $push (append) a new message, $inc (increment) the count by one <- baby's first pre-aggregation.
[22:36:18] <fewknow> codetroth: what are you confused about....there is a .push method off the parent.childern
[22:36:19] <codetroth> Under 100, this is going to be used for simple customer service support via chat and sms
[22:36:42] <GothAlice> db.conversations.update({_id: ObjectId(id of conversation)}, {$inc: {"stats.messages": 1}, $push: {messages: {… message sub-document …}}})
[22:37:45] <GothAlice> You would need a collection of "sessions" pairing some form of identity between supportee and supporter. I.e. SMS number to agent ID.
[22:38:27] <fewknow> codetroth: this is the issue with using Mongoose...it tracks the object for you
[22:38:28] <codetroth> I am tying each session to the inbound number as that is what will be unique to each conversation but the same for the length of one conversation
[22:39:24] <codetroth> At the end the agent will close that conversation which will change the status to false
[22:39:54] <fewknow> codetroth: check this out...http://mongoosejs.com/docs/2.7.x/docs/updating-documents.html
[22:39:56] <GothAlice> codetroth: Uhm, there may be legal requirements as to auditing, so "removal" may be a too-simple approach.
[22:40:49] <codetroth> so I am looking for a document where the phone number matches and the status = true
[22:42:04] <GothAlice> It's not, though. _id must be unique, which means you can't have a number with more than one record at any point in time.
[22:42:05] <codetroth> The entire reason I am using mongo is too document it lol. Because I could just accept the incoming connection and pass to the agent via socket.io and save nothing
[22:43:25] <GothAlice> I'd keep _id as an ObjectId… seriously. Then no need for timestamps, you have a unique way to reference every record, and you can have multiple expired histories without issue.
[22:44:00] <GothAlice> (Letting you do date range queries, i.e. everything older than 90 days, on _id alone.)
[22:46:48] <codetroth> I think part of my problem is using mongoose. I may drop it and use mongo directly
[22:48:34] <GothAlice> Mongo's syntax is expressive and powerful, something which I found surprising when first converting over from SQL-land.
[22:49:42] <codetroth> I had found just about the same answer I got from you guys when using mongo directly
[22:51:33] <codetroth> I had the same issue with the twilio library for node.js. It was easier just to queue the rest API directly than try to use helper libraries