Log file Viewer
#mongodb logs for Monday the 20th of April, 2015
[01:00:42] <davidbanham> Hi all. I'm getting my butt kicked by setting up a replica set. I think I have tracked the problem to: "Received heartbeat from member with the same member ID as ourself: 0""
[01:04:02] <clay__> I am kind of in a bind and was hoping someone more knowledgable then me could help :)
[01:04:42] <clay__> error loading initial database config information :: caused by :: Couldn't load a valid config for voipo.cdrtest after 3 attempts. Please try again.
[01:05:21] <clay__> I've done some research online and people ppoint to the config servers as a possible problem but I'm not sure what command to run to "repair" the config server
[01:18:20] <joannac> clay__: look at the logs before that, why can't it load the config? what's the error?
[01:18:34] <davidbanham> joannac: On the master, it was listing the other two nodes as unresponsive. Connectivity between all members is fine. I can mongo --host from on to another without problems.
[01:20:16] <davidbanham> "lastHeartbeatMessage" : "Received heartbeat from member with the same member ID as ourself: 0",
[01:22:17] <joannac> davidbanham: did you by chance rs.initiate() on more than one member or something?
[01:22:53] <davidbanham> joannac: Out of desperation, yes, but it was already doing this before that.
[01:23:48] <joannac> you may have had connection problems before, but you were *not* getting the same error
[01:25:00] <davidbanham> Is removing the contents of /var/lib/mongodb sufficient or is there other state persisted elsewhere?
[01:26:41] <davidbanham> Need to fool around creating the admin users first. Will ping when I'm done, thanks.
[01:30:47] <davidbanham> It won't let you rs.initiate() unless you're a root perms user. It won't let you add a root user without a majority. It's a bit of a dance to start in non-rs configured mode, add the users, then restart in the rs configured mode, auth, etc etc.
[01:32:25] <clay__> I'm seeing ChunkManager::_isValid failed: allOfType(MinKey, chunkMap.begin()->second->getMin())
[01:35:38] <joannac> davidbanham: "mongo --host mongod2 --port 27017" on the host "ip-10-1-17-121" and let me know the output*
[01:36:33] <clay__> then all i see is Assertion failed while processing query op for voipo.$cmd :: caused by :: 13282 error loading initial database config information :: caused by :: Couldn't load a valid config for DATABASE.COLLECTION after 3 attempts. Please try again.
[01:39:41] <davidbanham> joannac: Connects successfully, works like a champ. There's definitely connectivity between all hosts
[01:41:25] <joannac> davidbanham: if that works, but mongo2 still doesn't get the config, then check the logs from mongo2
[01:41:57] <clay__> so weird... i went to all 3 primary in the 3 shards and i can see 10k+ documents
[01:42:09] <davidbanham> mongo --host mongod1 from mongod2 works fine, but not mongo --host ip-10-1-17-121
[01:43:47] <clay__> "$err" : "error loading initial database config information :: caused by :: Couldn't load a valid config for voipo.sms after 3 attempts. Please try again.",
[01:44:29] <joannac> clay__: 1. pastebin. 2. i *know*. if you want help, please just do what I told you
[01:44:39] <davidbanham> There doesn't appear to be an option in the config to specify hostname, unless I'm blind.
[01:45:03] <clay__> you said go check the chunks... how can I do that? I am not certain exactly what you mean by that
[01:52:35] <clay__> i connected to all 3 on port 27019 directly and did that query and 8 documents appear on each... the same ones
[01:53:17] <davidbanham> I think I'll just burn it down and start again with every machine knowing it's externally reachable hostname at an OS level. Thanks for helping me track down the issue, joannac
[01:56:23] <clay__> i would have figured if chunks were missing mongo would just return partial data
[02:01:59] <joannac> if you're willing to lose data since the backup, you could restore from backup, assuming your backup is good
[02:20:49] <clay__> for my reference.. how could you tell by looking at that pastebin that documents were missing?
[06:07:03] <jry123> hey guys, quick question -- i have documents with 15+ language translations which can be quite large (well under the 16mb document size though)... i only need to use the keys for the current user's language, so obviously would be electing to only return those fields.... is this strategy ok? or should i split the translations into multiple documents?
[06:08:13] <jry123> obviously the single document approach is way easier, i just wonder from a memory perspective, there will be millions of records in this collection (converting from an old MySQL DB)
[07:41:29] <gothos> Hello, anyone here familiar with casbah? I'm trying to extract data from an array that has some object in it, it looks like this: "perfData" : [ { "pi" : "3.1" , "minute" : "3"}, ... ]
[07:41:31] <gothos> I'm getting a MongoDBList but can't find a way to get the object within, any recommendations? :)
[08:50:00] <isale-eko> http://stackoverflow.com/questions/29719471/mongodb-mongoid-rails-unable-to-increment-counters-in-a-hash-in-embedded-documen
[09:07:22] <willemb> Hi. I need some advice about running a mongodb replication set across multiple datacentres.
[09:21:29] <sudomarize> If i'm storing categories in mongo, what's the best way to do this? e.g. for a category "House Cleaning", how should i store this?
[09:23:08] <willemb> I have 3 nodes in dc A and 2 in dc B. the B-side's priority is set to 0 so that they don't become master (the services that query this db are all in dc A). However, when dc A goes awol, it is pretty hard to make one of them a primary while there is no primary available.
[09:46:40] <Derick> sudomarize: everybody is checking their mails first on Monday morning - and, it's still sleep time in the US.
[09:47:38] <Derick> willemb: I'd make the DC A nodes have priority 2, and the ones in DC B prioity 1
[09:48:11] <Derick> willemb: but note, that if DC A goes fully down, the 2 (out of 5) in DC B can not see the majority, and hence not elect a primary
[09:53:26] <sudomarize> Derick, do you have any suggestions about the best way of setting up categories in my schema? i'm a bit stuck here
[09:54:06] <Derick> on what you need to store, on how you add the data, and how you need to query it
[09:57:08] <sudomarize> Derick: no worries. So i'm creating a marketplace that allows users to create tasks. In terms of the parameters of listing a task (for the lister), i want to provide most of the structure, i.e. there will be a set of predefined categories and subcategories (3 tiers). But I imagine there will still be significant varying, even between two tasks with the same category tree, so would implementing a tag: {}
[09:57:09] <sudomarize> object in my schema (and then doing a search for both categories and tags) be a good idea?
[09:58:17] <sudomarize> i'm also not sure what the best way of storing that value in my db is, i.e. should i store it just as the formatted string, e.g. "House Cleaning", or as a key-value pair, something like {category: {name: "house-cleaning", formatted: "House Cleaning"}}
[09:59:41] <sudomarize> i've looked on the mongo page for creating categories, which seems relatively complex compared to the SQL solutions which i'm used
[09:59:54] <KekSi> anyone had this problem before? 2015-04-20T09:57:14.038+0000 E - [mongosMain] error upgrading config database to v6 :: caused by :: could not load config version for upgrade :: caused by :: socket exception [CLOSED]
[10:01:03] <sudomarize> and finally i'm a little worried about how making changes to the category structure will affect searches, i.e. is there a way to make changes to my category tree without causing issues when i run queries?
[10:05:43] <sudomarize> Derick: also what way do you think is best of adding categories? should i add a large amount initially, or add them slowly after they are requested by users? basically i want to be able to manage which categories are able to be searched, so only very popular categories are shown
[10:13:19] <sudomarize> Derick: in terms of having a dynamic set of category tiers, i'm still not sure. But generally speaking i want to manage the quality of the category that are available to users, to make search and listings really easy. there's a good guide on the mongo site about setting up product categories (inc a tierd ssystem), so with a few adjustments that can easily be modelled to my particular needs
[10:14:01] <Derick> so you're asking how to store a category with each post? I don't quite understand yet
[10:17:56] <sudomarize> Derick: basically i'm asking 1. what's the best way of storing categories in mongo (e.g. an easy json name and then a formatted name for HTML like {key: "housecleaning", formatted: "House Cleaning}), 2. is there a way to make just a category tree dynamic, that is to say, add, removing or moving categories up or down the tree and 3. Assuming i have a relatively strict category system in place, is the best way
[10:17:58] <sudomarize> of differentiating listings with the same categories using a tagging system?
[10:18:19] <sudomarize> Derick: hope that's not too vauge, still trying to figure a lot of this stuff out at the moment
[10:19:46] <Derick> 2 - storing path structures is not very simple to do. But I think you found the docs on doing that. But I am not sure what you mean by "just make a category tree dynamic"
[10:23:34] <sudomarize> Derick: basically so that i can edit and move categories up and down the category tree, say from tier 3 to tier 2
[10:24:18] <sudomarize> Derick: RE 3: well if a particular user has expertise in a subset of a tier 3 category, then there's no way of users searching for tasks to be able to find it
[10:24:40] <Derick> sudomarize: i think you'll have to do most in your application for the tree management
[10:25:58] <sudomarize> Derick: say gardening was a tier 3 category, and there's a lister who has expertise in growing roses, currently there's no way for them to let users looking for gardeners know this (except in the description of the task, but users can only search with categories <- to make the experience cleaner)
[10:27:25] <sudomarize> mutliple tags or multiple categories, because they would only be able to add multiple categories due to the nature of the category tree
[10:28:52] <sudomarize> Derick: otherwise i'd have to make n categories to fit every particular skill set out there
[10:29:29] <sudomarize> growing roses -> rare temperate roses -> a particular species of rose -> etc
[10:30:03] <sudomarize> i mean 3 tiers is a relatively arbitrary number of tiers to have, but i think it gives good depth while retaining a relatively easy UX
[10:30:50] <sudomarize> Derick: but that's also what i was talking about before with a dynamic tree. there are probably some instances where >3 tiers is required
[10:57:15] <KekSi> right, apparently it was a docker problem (upgraded from 1.5 to 1.6) -- just in case someone else is having odd problems
[12:40:16] <deathanchor> so one of my secondaries fell out of sync beyond the replWindow, and is stuck in recovery with this in the logs: [rsBackgroundSync] replSet not trying to sync from MEMBER it is vetoed for 80 more seconds
[12:43:39] <deathanchor> yeah same problem. lots of data to sync. fresh sync looks like my only choice
[12:43:40] <sudomarize> cheeser, any suggestions you have about how to implement this tiered category system?
[13:06:41] <arussel> I have a replicat set without authorization/authentication. I'd like to introduce it, so: 1: db.createUser({user: "root", pwd: "foobar", roles: [{role: "userAdminAnyDatabase", db: "admin"}]})
[13:07:09] <arussel> 2. db.createUser({user: "myapp", pwd: "foobar2", roles: [{role: "readWrite", db: "myappdb"]})
[13:48:30] <willemb> one problem: bad config for member[3] this version of mongod only supports priorities 0 and 1",
[14:10:38] <greyTEO> Did you ever use cloud server images as backups? http://docs.mongodb.org/ecosystem/platforms/rackspace-cloud/#cloud-servers-images
[14:16:59] <GothAlice> greyTEO: None of my backup mechanisms require locking like that example. Interesting that it's an option, though. (Also: don't get me started on the pricing of this type of "backup".)
[14:18:21] <GothAlice> Backups should be a) as cheap as possible, b) as difficult to restore as possible. "Deep storage" is great. If it takes an hour to get anything out of it, one tries extra special hard to make sure one doesn't need to use it.
[14:29:23] <GothAlice> I also get giggles every time I read through platform instructions that focus on single distributions. "Here's how you do everything you need to do on Rackspace… if you use Debian/Ubuntu."
[14:30:35] <greyTEO> Tis true about about the backups being hard to extract data. PITA is the best motivation
[14:31:37] <greyTEO> I thought it was wierd they have a section for Rakespace...on ubuntu. Shouldn't they just have a section for Ubuntu and stop recommending those specific solutions?
[14:32:20] <GothAlice> Yeah, I wish these sections only had details that actually were unique to that platform. (I.e. the backup thing.) Installation instructions are best left to _installation instructions_.
[14:32:47] <GothAlice> (And there really didn't seem to be anything unique from an installation standpoint.)
[14:37:48] <GothAlice> You could always run multiple dumps in parallel, each dumping different collection sets, potentially from different replica secondaries. The latest version even incorporates parallelism directly into the restore tool, too.
[14:39:15] <GothAlice> And information about indexes is, in fact, backed up. "Backing up" an index doesn't actually make any sense. The location of the data in the restored stripes will differ from the original stripes, so the original index data is utterly useless anyway…
[14:41:11] <greyTEO> the data would be useless but it would cut back on the restore time. depending on the indices and amount of data though
[14:41:31] <GothAlice> You'd be "restoring" bogus data you'd then have to re-iterate and correct anyway.
[14:41:54] <GothAlice> Thus there's literally no point in even attempting to do that; just rebuild the index either as you go, or once at the end, and be done with it. (Iterate once, not many times.)
[14:42:39] <greyTEO> well restoring from a dump would re-import. I am thinking more point in time snapshots.
[14:43:02] <GothAlice> Index data outside of the literal instance that is storing the at-rest data is _meaningless_.
[14:44:49] <GothAlice> There are several types of snapshot; only a filesystem-level snapshot would benefit from preserving index data, and, well, has the added bonus of MongoDB being completely unaware. So yeah, that'd work. (It's also the least efficient method to do a backup.) mongodump with the --oplog option is also a point-in-time snapshot.
[14:45:08] <GothAlice> It's still a re-compacted dump, though, so storing index data would continue to be a waste here.
[14:47:30] <GothAlice> Of course, in production systems "high availability" should be the only backup one needs. Have replication secondaries in-DC, outside the DC, at your office, at your home… so worse-case and you need to switch hosting providers or data centres? Spin up a node, let it replicate, and keep on keeping on. ;)
[14:51:23] <greyTEO> Filesystem might not be the most effecietn but wouldnt that be the best way to get a hot backup without interrupting performance...(given you could run it at your slowest time of traffic)
[14:52:15] <GothAlice> Replication typically has no "instantaneous load" (i.e. it's constantly streaming, so shouldn't have peaks in activity that slow down the rest of the system).
[14:54:00] <greyTEO> Corruption or server failures wont replicate, but deletes and drop collections could.
[14:54:54] <GothAlice> However you can also delay your replicas by however much time you want, to allow for rollbacks in the event of user error.
[14:56:15] <GothAlice> Replication is how you get reliable service in MongoDB. Point in time snapshots / backups are not. (Always remember law #105. Backups are what happens when everything else is on fire.)
[14:58:25] <greyTEO> with mysql http://www.percona.com/ is the best way for hotbackps and PIT. restore is crazy easy and it's all Filesystem based.
[14:59:06] <GothAlice> It can be. Unless AWS decides that if your EC2 instance needs to touch an EBS volume, lock both the instance and the volume. Which _has happened multiple times_ on that service. ;)
[14:59:56] <GothAlice> Also MySQL. (In one of those failures I had to spend the 36 hours prior to getting my wisdom teeth extracted to reverse engineer the on-disk InnoDB format. Managed to recover all the data, amazingly.)
[15:01:22] <GothAlice> greyTEO: I switched to Postgres, and use WALe to stream the write ahead log, i.e. oplog, straight from the Postgres server to S3 and several other destinations. None of my Postgres servers had permanent storage, because like MongoDB, I had them configured to self-repair on startup.
[15:02:58] <digi604> Hi everybody… what is the easiest way to copy a db from a replica set with 2 servers to a 6 server cluster? db.copyDatabase seams not working on clusters…
[15:03:08] <GothAlice> Vitium: http://www.javaworld.com/article/2088406/enterprise-java/how-to-screw-up-your-mongodb-schema-design.html may be useful, if you're up on the formal terms. :)
[15:04:51] <GothAlice> digi604: You do realize mongodump/mongorestore aren't limited to only running locally? I.e. you can run it from a machine with enough scratch space.
[15:06:35] <GothAlice> digi604: Introduce the machine with the data on it into the new cluster, and let it re-balance. That's the hard way. ;)
[15:08:30] <GothAlice> digi604: http://docs.mongodb.org/manual/reference/replication/ and http://docs.mongodb.org/manual/administration/replica-sets/ for some reading.
[15:09:05] <digi604> so if i have 3 shards and 3 replicas…. and i add the 2 existing nodes to it as an additional shard i can later remove the 2 existing nodes (one complete shard)?
[15:09:10] <GothAlice> Vitium: The article I linked should be quite helpful in exploring your current model. In MongoDB, dependant data like the reviews can often be embedded within the "parent" document, since when getting the review you'd likely also be wanting a copy of the product data, and when you delete a product, cleaning up the reviews is a natural second step.
[15:09:45] <GothAlice> digi604: Yes; though when you ask it to remove the node, you'll have to wait for it to shuffle the data off that node before shutting it down.
[15:11:21] <GothAlice> Vitium: Exactly; your reviews would be an array of embedded documents. e.g. product = {_id: …, name: "Walkman", manufacturer: "Sony", reviews: [{author: "Alice", comment: "It's boss."}, …]}
[15:13:06] <GothAlice> When there are so many reviews on something that the record fills up, you need to have a contingency plan. In my forums, where I store replies to a thread within the thread itself, I handle this by automatically starting a new thread, then linking the old one to the new one.
[15:13:59] <GothAlice> Vitium: It's important to not go overboard with nesting things; you can only effectively query one array per document per query.
[15:15:23] <GothAlice> You _can_ have "references" (typically just the ID of the target record, or a DBRef) between collections (tables), but without joins any time you need to gather information from a different collection, you need to perform an additional query.
[15:17:01] <GothAlice> Any time you get the sensation of wanting a JOIN, smack yourself. MongoDB has no relational capability whatsoever, and modelling with such non-existent capability in mind will only lead to slow queries and mangled code. ;)
[15:17:32] <GothAlice> See also: http://docs.mongodb.org/manual/reference/sql-comparison/ and http://docs.mongodb.org/manual/reference/sql-aggregation-comparison/
[15:18:15] <GothAlice> Just repeat this mantra: "My data is more varied and wonderful than a spreadsheet."
[15:20:01] <StephenLynx> What she said. Any time you need to do something that would require a join in sql, you are doing it wrong.
[15:20:30] <StephenLynx> I personally see (few) uses for fake relations, but don't you ever think of them as your first, second or third option.
[15:24:48] <Vitium> Aren't they basically showing a join with the One-To-Many example? http://blog.mongodb.org/post/87200945828/6-rules-of-thumb-for-mongodb-schema-design-part-1
[15:28:33] <GothAlice> Vitium: Indeed, that example is simulating a fake join at the application level.
[15:29:22] <GothAlice> However the full impact of this is difficult to gauge from that example: it implies a second complete round-trip, you currently can't get results back in arbitrary orders (i.e. give me records X, Y, and Z in the order [X, Y, Z]), etc.
[15:31:10] <GothAlice> Unfortunately, that article states "Each Part is a stand-alone document, so it’s easy to search them and update them independently." which would seem to imply that you lose this ability if you embed. You do not.
[15:31:59] <GothAlice> To the point that the entire gaming forums I was converting from phpBB to my MongoDB solution in Python could fit in a single record.
[15:32:24] <StephenLynx> yeah, unless you expect your subdocuments to be really, really, really, huge, size is not an issue.
[15:35:42] <greyTEO> GothAlice, http://s3.amazonaws.com/info-mongodb-com/MongoDB-backup-disaster-recovery.pdf
[15:36:33] <GothAlice> greyTEO: Anything more specific I should be looking for than that? (And numbered page 3, or PDF page 3?)
[15:38:09] <GothAlice> "While mongodump is sufficient for small deployments, it is not appropriate for larger systems. mongodump exerts too much load to be a truly scalable solution. It is not an incremental approach, so it requires a complete dump at each snapshot point, which is resource-intensive." — All true.
[15:38:26] <GothAlice> Also, notably, all things that will vary from deployment to deployment and application to application in terms of acceptable load, etc.
[15:39:59] <Left_Turn> from the docs: db.runCommand(command) ... shouldn't there be a 2nd context/mapping argument for search?
[15:40:00] <GothAlice> Again: the simplest, easiest approach is replication. Replication is love. Replication is life.
[15:40:23] <GothAlice> Left_Turn: The number of times I've had to call runCommand myself can be counted on no hands. What're you trying to do?
[15:45:13] <GothAlice> Manual invocation of runCommand is a good warning sign of something being done more "craftily" than it needs to be, though. Driver support is pretty solid for almost all commands, excluding some pretty low-level administrative ones.
[15:45:58] <Derick> GothAlice: for the new PHP//HHVM driver we're not implementing *any* helpers in the base extension
[15:46:08] <Left_Turn> ah ok ... I'll take the advice... also thanks Derick for all the links.. got a lot of reading to do:)
[15:47:03] <GothAlice> Derick: Well, that does seem to be PHP's modus operandi: expose raw C function calls to the application layer. Which C calls? ALL THE C CALLS!
[16:02:11] <juliofreitas> Hi! Each 15 minutes I get a file with a time series data. The data are structured this way: time (2015-03-27T20:00:01Z), state (string), owner (string). Which is the best way to store and make search with a great performance although my search will be (by day, choosing the interval 15 minutes, 30 minutes, 1 hour...)
[16:04:00] <cheeser> juliofreitas: this might help: http://blog.mongodb.org/post/65517193370/schema-design-for-time-series-data-in-mongodb
[16:21:39] <ParisHolley> “driver is incompatible with this server version”, on 3.0.2 with latest 2.x of mongo native driver and mongoose
[16:22:39] <ParisHolley> no, all my projects use mongoose, did a server migration and thinks just seemed to work find, accept one app
[16:31:55] <snowcode> my query is "AlarmModel.find({geo: { $geoIntersects: { $geometry: {type:'Point',coordinates:[12.506129,41.889117]} } } },".....
[16:40:17] <int3nsity> I'm using mongoose in nodejs, is it recommended or you prefer a more low level mongo driver?
[17:10:38] <snowcode> Derick I've also validated my polygon on http://geojsonlint.com . But it still returns http://pastebin.com/vuRGuXqy
[17:30:35] <Vitium> I was going to learn it but ended up choosing meteor because of how low level expressed looked
[17:31:33] <StephenLynx> I can understand something like the android framework, that is used for you to interact with the android API
[17:32:06] <StephenLynx> and completely changing the workflow, making it harder to get documentation and reference.
[17:32:54] <StephenLynx> people just use them because they don't know any better. I had a dude that thought he needed them for http.
[17:33:25] <StephenLynx> you don't write your own http server, that is already built-in in the runtime environment.
[17:35:03] <StephenLynx> in the long run, using these frameworks is asking to get bitten in the ass hard.
[17:38:01] <StephenLynx> Ive been working with node (now io.js) since september 2014 I guess, using just the RE and mongo driver, I am glad to work with web for the first time in my life.
[17:38:38] <StephenLynx> just released one of my two projects that use a back-end https://gitlab.com/mrseth/bck_lynxhub
[17:39:54] <StephenLynx> so yeah, I suggest you at least trying to not use a framework for once, Vitium
[17:42:00] <christo_m> StephenLynx: i dont want to use db.users.findOne({_id:ObjectId(...)}) just to look at document structure
[17:42:46] <Vitium> I'm learning this for a school project, I don't have enough time to code all that
[17:42:59] <snowcode> okay now there is a "Point must only contain numeric elements" error inserting my GeoJSON Data into mongodb. If I try to insert a simple coordinate point [[lon,lat]] it works
[17:44:32] <christo_m> i want to see if when im adding things that its actually reflected in the database
[17:45:21] <StephenLynx> https://gitlab.com/mrseth/bck_lynxhub/blob/master/doc/model.txt the model documentat that I told about.
[17:51:14] <christo_m> so because id rather use a GUI the implication is that im using the wrong tools?
[17:51:41] <christo_m> its cool if you dont agree with or like mongo-express but saying "just use the terminal" isn't a solution to what im saying
[17:53:10] <christo_m> StephenLynx: http://docs.mongodb.org/ecosystem/tools/administration-interfaces/ i was here and mongo-express was listed there
[18:13:48] <gswallow> were there deprecation notices that simple things like Mongo::Connection.new() would just *stop* working when other peoples' code requires mongo gems, and you released 2.0.0?
[18:16:10] <GothAlice> gswallow: There are still people in existence who don't pin version number ranges? O_o
[18:17:06] <GothAlice> The latest update broke a fair number of things in Python-land, but "pymongo<3" is really, really easy to define as your package's requirement.
[18:20:01] <gswallow> because I'd have thought about it sooner if there were (unless I ignored them, which is equally likely)
[18:26:37] <arussel> is that true: if I don't add the keyFile attribute, then anyone can do anything (if they can reach the host)?
[18:28:25] <GothAlice> But! If you do open it up to connections from anywhere, be sure to either enable authentication (and use good passphrases) and/or set up strong firewall rules.
[18:28:49] <arussel> so 1. I add keyFile to each of my nodes. 2. add an admin user 3.add an 'app' user
[18:29:46] <arussel> do adding keyFile attribute enables authentication, or is it the admin user creation ?
[18:30:07] <arussel> I'm trying to find out when I should take my app down and when to restart it with its new crendentials
[18:33:49] <arussel> if the app is passing credentials but there is not keyFile, would the credentials just be ignored or would this throw an error ?
[18:35:02] <StephenLynx> it only throws an error when you try to do something the user is not allowed.
[18:35:44] <StephenLynx> I could get it to connect and get references to collections fine with a failed authentication.
[18:37:41] <gnu_d> Hi, I'm researching in Sharding, I'd like to know given two mongo servers, let say the first will have the newest data, and the other the data from older than 6 months. Is ok to make a scheduled job that will search for documents older than x time and then relocate them in the other shard, or this is made automatically when configuring the shard, if so can you tell me how ?
[18:39:53] <cheeser> gnu_d: you might consider http://docs.mongodb.org/manual/core/tag-aware-sharding/
[18:41:40] <StephenLynx> that sounds interesing, greyTEO, but I wouldn't trust an automated tool to do something like that.
[18:41:59] <StephenLynx> migrating a db like that is a one-time operation that needs some careful thought.
[18:42:52] <greyTEO> just wondering if any has used it. I am currently handlign this migration with custom code.
[18:43:23] <StephenLynx> yeah, I would keep on doing that. is not much about the process itself, is much more about the design of the new database.
[18:43:46] <StephenLynx> you must consider how you query your data when designing a non-relational database.
[18:45:14] <greyTEO> Makes you think a lot about the product/features when you move away from relational structures
[18:46:46] <greyTEO> Luckily I am making heavy use of elasticsearch, so my database change didnt disrupt too much
[19:01:59] <StephenLynx> but I never see any ruby discussion here. Ive seem more C++ talk than ruby.
[19:20:21] <christo_m> I have a concept of a feed or queue for users. is it better to have it be a nested subdocument?
[19:20:38] <christo_m> its a 1-1 relationship so having it separated out doesn't make much sense.. except for the fact that i dont want to dirty up the user code i have currently.
[19:23:05] <christo_m> im using ionic framework on mobile so i like that ill be able to reuse the services im creating.
[19:27:34] <StephenLynx> https://gitlab.com/mrseth/bck_lynxhub/blob/master/doc/backend_interface.txt
[19:29:00] <christo_m> it lets me know others have established some patterns and that things will scale
[19:30:34] <StephenLynx> dependencies on top of dependencies on top of dependencies on top of dependencies.
[19:33:12] <StephenLynx> I use because It does something that I really need, that is specialized and has lots of details to implement.
[19:33:30] <StephenLynx> when I use imagemagick I use it because doing image manipulation is not trivial.
[19:33:35] <christo_m> im working on something that has deadlines, i dont have time to muck about with configuration when something works out of the box
[19:35:07] <StephenLynx> anything that involves a building with stuff that can be just interpreted is bad.
[19:35:51] <StephenLynx> unless it is some sort of really incredible optimization that you really, really need
[19:39:19] <christo_m> also, data design question, because i dont have much experience with mongo/mongoose
[19:40:32] <StephenLynx> learn from the elders. http://homepage.cs.uri.edu/~thenry/resources/unix_art/ch01s06.html
[19:42:47] <GothAlice> I'm biased, as a contributor/owner to/of several of the projects I'll mention, but MongoDB has replaced a multitude of other services (queues, caches, etc.) for me, I use an ODM called MongoEngine that supplies, application-side, many nice things like efficient fake joins using caching, triggers, reverse delete rules, etc, and build a multitude of packages (under the "marrow" org on Github) that depend on it.
[19:43:36] <christo_m> so when i hit the problems you're talking about, ill consider it, seems to work fine for me right now
[19:44:59] <StephenLynx> that is the problem with these kinds of solutions. is one big stuff. it is heavy and obscure and asks you to shape your problem around its solution.
[19:45:10] <GothAlice> christo_m: The link I provided counters many of the common points others may attempt to use to nay-say MongoDB and can help avoid falling into some of the same traps. :)
[19:46:53] <christo_m> StephenLynx: except it isnt like that at all.. because the components are right there in front of you, there's nothing magical happening
[19:47:17] <GothAlice> MongoDB "eats its own dog food" when it comes to APIs and whatnot; the lack of magic is itself almost magical.
[19:47:38] <GothAlice> (Thus application-side triggers, application-side reverse delete rules, application-side everything, basically.)
[19:47:55] <GothAlice> It's a refreshing approach, vs. relational databases that continually pile on more features into the core.
[20:03:49] <GothAlice> Now here's an interesting question: BSON itself has no particular restrictions on field names. (It's a combined Pascal+C string.) Most drivers seem to enforce no $ in field names. DBRef is a "complex" type (i.e. a compound of simpler types) that uses $ in its field names. Is it OK to create custom types like DBRef for your own use?
[20:09:04] <GothAlice> That's also not exactly what I'm describing, StephenLynx. I don't wish to manually commit embedded documents with $ in their field names, I'm wanting to create a new BSON adapter (like DBRef) that self-encodes other types. Mostly looking for an example of extending pymongo for additional types.
[20:11:20] <GothAlice> And as expected, it explodes when I attempt to use $ in a field name, at the client level. (src/mongo/shell/collection.js)
[20:13:29] <GothAlice> cheeser: It's always dubious how much of the validation goes on server-side. "[object Object]" being a valid collection name always gives me pause to shake my head. ;)
[20:17:21] <GothAlice> Biggest problem I frequently see with Mongoose users: it effectively forces you to throw your hands up and treat all ObjectIds as plain strings. (I.e. the hex encoded binary form, taking up more than 2x as much space, and heaven help you if you mix real ObjectIds and string-form ones.)
[20:20:25] <mortal1> I did a db.collection.insert, instead of an upsert, and i think it may have overridden my collection
[20:21:37] <StephenLynx> on the other hand, update might completely overwrite a document if you forget $set.
[20:23:33] <mortal1> GothAlice: I once had hipchat magically insert a ; in a sql statement, right before the where
[20:27:15] <christo_m> GothAlice: {"message":"Cast to ObjectId failed for value \"add\" at path \"_id\"","name":"CastError","type":"ObjectId","value":"add","path":"_id"} i dont even know what this is doing
[20:27:24] <christo_m> im not doing anything special.. just routed to an empty function with express.js
[20:29:08] <jhoff909> is there a recommended replica set config when running on linux in Azure? and where there are clients external to Azure?
[20:36:52] <StephenLynx> well, not supporting would be having way too much faith on people's stupidity.
[21:06:42] <snowcode> anyone know how to store a GEOJson Polygon into mongodb? It still say to me "Point must only contain numeric elements" but i want to store a polygon, not a point
[21:08:52] <sudomarize> I have Task collection, and each collection has put into a category, e.g. one Task may have a category "iPads" <- "Apple" <- "Hardware" <- "Repair" (where the categories are a mongo tree, each one with a "parent" key pointing to the id of it's category parent)
[21:09:59] <sudomarize> Would it make sense to put the category tree for a particular Task into a {categories: []} array in my Task document?
[21:28:23] <greyTEO> what is the deepest level an object should be nested? speaking in terms of best practices...
[21:33:36] <GothAlice> greyTEO: No more than one level of nesting, unless you are _extremely_ careful.
[21:35:42] <greyTEO> if it's best to flatten, I will extend the address object rather than introducing another.
[21:37:10] <greyTEO> I alwasy use $set on the array and replace rather than update a single...if I need arrays
[21:37:59] <GothAlice> ($pushAll being deprecated these days as the functionality has been rolled into $push)
[23:26:30] <bros> I know you can query for the nth index in an array. (ex. log.0.time) How can you do the last index? -1? $? where clause?
[23:54:55] <GothAlice> bros: Alas, you can't query the last element unless you know how many there are using standard queries. You can project to the last using http://docs.mongodb.org/manual/reference/operator/projection/slice/ and use http://docs.mongodb.org/manual/reference/operator/aggregation/unwind/ followed by an additional match in an aggregate query.