Log file Viewer
#mongodb logs for Saturday the 10th of November, 2012
[00:55:43] <John> Hi all. Does anyone know how I could find all geo elements within a bounding box but reduce the result to 1 in a 10 mile radius?
[06:05:51] <hdm> I am looking for the equivalent of multiple emit()'s [m/r] for the aggregation framework, $project isnt quite what I need, any suggestions?
[06:07:26] <hdm> example would a doc with fields {a,b,c}, i need to count how many a/b/c's are seen across all docs matching the query with individual counts per field
[06:08:28] <hdm> could use $project : { a : { $cond : [ "$a", 1, 0 ] }, b : { $cond : [ "$b", 1, 0 ] }, c : { $cond : [ "$c", 1, 0 ] } } to normalize it
[06:09:22] <hdm> but then i still need to count those somehow - grouping would help with _id : { a : $a, b : $b, c : $c }, but it would return one result for every unique combination and sitll have to be added together in the app
[06:09:58] <hdm> if anyone has ideas let me know. the easy fix is to go back to m/r and use multiple emits
[08:40:26] <jammanbo> So, vague question, but is there any chance that even a pre-release that supports a geospatial polygon type/index would become available in foreseeable future (like a year or something), or is such a thing just not even on the radar?
[10:52:26] <navaru> Someone around? I have a quick question about TTL, expires (working with mongoose)
[14:50:12] <bwellsnc> Hello everyone, I have doing some research and thought I would get the opinion of the room. What is the best backup strategy for a sharded cluster. Thanks
[15:44:32] <konr_trab> How should I run https://gist.github.com/941172 in mongo? `mongo [params] < script`?
[17:17:21] <konr_trab> how can I debug a mapReduce script I send to mongo? 'print' doesn't actually print anything inside a function
[18:45:42] <jtomasrl> is it correct if i repeat values of a collection inside a value of another collection?
[18:47:36] <jtomasrl> i have a collection named items with the item name and price and inside another collection named users i have orders that is an array of user orders that include the item name and price from the item collection an a order date, is that correct?
[18:50:18] <jtomasrl> i think im collecting that data on a data warehouse so i'll "empty" those orders once a month or week
[19:04:40] <RONNCC> so...i seem to be writing too fast - what should i do? I have mongodb running on my computer and am writing ~500 mb every few minutes... all the data piles up in my ram and them my swap... ram ~ 2gb, swap ~7 gb... what should i do? --> sharding?
[19:18:54] <hdm> a cluster of 2gb ram systems isnt going to help much, you generally want to tackle resources as a matter of ram -> disk i/o -> cpu, you can use sharding to help with any of those three, but I found it easier to just bump specs on primary systems first
[19:20:14] <hdm> at some point ram stops being a bottleneck (flushing to disk eats your disk i/o), ssds or sharding helps there, or adding more spindles and raid
[19:20:48] <hdm> check your "iostat" output, if your disk is spending all of its time doing reads, you need to reduce your indexes or move to SSDs
[19:20:59] <RONNCC> hdm: this is a corporate env. i can boost ram to 12, but that doesn't matter. on some 10 documents it ate 2 gb of ram, 7 gb of swap o-o. then disk i/o how do i do that... i'm using a corporate vm. i can ask for cores/ram/harddirve..... should i ask for for hd's to mount? can i do something like raid for mongo ... would i have to get a software raid controller >_>" ... and i'm running 6 cpus at the moment. only ~2 get used by mongo w
[19:21:27] <hdm> you may be misreading the stats, it will map 2gb of memory, but only use a small portion of that
[19:24:23] <hdm> the bigger your index gets, the more data it has to read while doing inserts to upate the index
[19:25:27] <hdm> ive got a single box that handles getting on 2 billion rows, but it took a while to find the right balance
[19:27:23] <hdm> the 'res' value of mongostat may be bigger than db.stats() index size and thats fine
[19:28:40] <hdm> example of mongostat output: 0 0 0 0 0 1 0 1432g 2866g 65.6g 0 local:0.0% 0 0|0 0|0 62b 3k 1 13:25:2
[19:31:11] <hdm> RONNCC: do you want to manage a giant cluster and be able to query everything at once, or keep it usable on small systems and break things into multiple queries?
[19:33:23] <hdm> lets just say you dont have a lot of resources then; you need to make sure your indexes stay small, so they fit into ram, and that your lmited disk i/o doesnt drag you down too much
[19:36:00] <RONNCC> so i have ~ 100 files i need to parse. they have ~ 200000 lines in each. so each entry is of the form {time: {dataa:1, data2: ...}}
[19:36:21] <RONNCC> this isn't really a business this is because i have a ton of data that i want to run analytics on....
[19:37:04] <hdm> how you query will determine how you store it, since anything not in an index is slow
[19:37:34] <hdm> you can break the data into multiple databases or collections, but then you have to query each one individually
[19:38:01] <hdm> you can, but it will be slow, so figure out what fields are most important, and what records need to be in the same db or collection for your use case
[19:39:28] <RONNCC> {day: IP: 109.234.78.117, Type: Scanning Host ,Country: DE, Region: cityname, Coordinates: (51.0,,9.0) Misc: (11 ,4,2)}
[19:42:29] <hdm> ok, a month of that data would be about 60Gb (~30m records, take into accoutn the journal overhead, etc) and if you indexed based on just _id (default), id guess around ~8gb of ram for the index, but that isnt a good thing to index on if you want to query it
[19:43:58] <hdm> unless you index /everything/ that should fit into ~12gb of ram and generally be manageable
[19:45:10] <hdm> you wont be able to work with a full month of that data if you download it hourly, you may not be able to even load it into a single db within an hour
[20:49:33] <mattbillenstein> db.cloneDatabase(<otherhost>) returns an error message: query failed ciddb.system.namespaces
[21:02:23] <joe_p> mattbillenstein: https://jira.mongodb.org/browse/SERVER-5577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
[22:21:56] <kgraham> hello, quick question about mongos/configdb's , i have three configdb's and their order in the mongos config file for our worker machines in ec2 (which are autoscaled) has been set for a while, I've recently tried to make it so depending upon what region the machine is launced in, it would connect to that regions' config db to reduce the load, well when I attempted this, I got this error ... http://pastebin.com/PQJibGm8
[23:15:41] <teotwaki> Hi, quick question, I see that the C++ driver is said to use some "core MongoDB code", does this mean that the C++ driver is also licensed under AGPL?
[23:29:58] <teotwaki> ckd: yes, I understand that, but because it uses AGPL files, wouldn't that apply to the whole C++ client driver?
[23:31:09] <silverfix> ckd: in python: cursor = collection.find({'sentiment': None, 'last_access': {'$in': [{'$lt': five_min_ago}, None]}})
[23:38:03] <silverfix> db.labeler.find({'sentiment':null, 'last_access': {'$or':[{'$lt':new Date()},null]}})