Log file Viewer
#mongodb logs for Wednesday the 22nd of April, 2015
[08:15:19] <KekSi> quick question since i'm not entirely sure about this right now: is it possible to add more shards to an already sharded collection? it looks to me like that wouldn't be a problem and the only problematic thing would be to change the shard key later
[08:28:14] <Boomtime> KekSi: you can add shard any time you like, note that shards are added to the cluster as a whole, sharded collections merely make use of shards as necessary
[08:29:03] <Boomtime> also, changing the shard key is a difficult problem any time, regardless of how many shards you have - you effectively must dump and re-import your entire collection to change the shard key
[10:51:22] <djlee> Hi all, if i have a list of object id's represented as strings, is there a way of doing an IN query against _id without converting them ll to ObjectIds, like : find({_id:{"$in":[ "stringid", "stringid2" ]}})
[11:10:54] <mah0-deh0> I have a visits collection that I would like to "group", i.e.: a user just can have a single visit in a certain day so I need to get all its visits by using the min in_timestamp and max out_timestamp and summarizing each visit duration (which may be available or not..). Which is the best way to achieve it?
[12:27:21] <allaire> Hi. We're using the MongoDB Amazon AMI image for our app. we just upgraded from 2.4 to 2.6.9. sudo sevice mongodb start is not working but sudo mongod --config /etc/mongod.conf is working, any ideas?
[12:32:35] <gabrielsch> for example, I want to find how many products where created for each user in the last 30 days...
[12:32:39] <gabrielsch> I have the following schema: https://gist.github.com/gabrielsch/1d8a0d8125a21378f35c
[12:36:23] <allaire> detailed question here: https://groups.google.com/forum/#!topic/mongodb-user/XpuKeygrrWU
[13:31:11] <Derick> i guess you could write a little script to tail the oplog... but that's not what it's really meant for
[13:32:11] <teejay23> actually when i run tests, usual CRUD. there's no way to actually see data was in mongo and got deleted...
[13:43:44] <amitprakash> Hi, how can I sync my collection data from mongo to EMR? I was looking at mongodb hadoop connector, but the documentation doesn't seem to indicate anything about how information is continuously updated from the latter to hdfs?
[14:07:45] <GothAlice> Oh, HL7 encoding, whoever invented you needs to be taken out back and shot. (Speaking of EMR in the "electronic medical records" sense.)
[14:26:40] <GothAlice> Connector uses a lot of deep tricks to do what it does, including tailing the oplog. If there are _any_ differences in the status information (there are) or operations themselves (there shouldn't be, but could…) then Mongo Connector will have to be updated to recognize it.
[14:28:39] <cheeser> only two issues against wired tiger: https://github.com/10gen-labs/mongo-connector/issues?utf8=%E2%9C%93&q=is%3Aissue+wired+tiger
[15:03:57] <saml> http://docs.mongodb.org/manual/release-notes/3.0-upgrade/#change-storage-engine-to-wiredtiger
[15:04:38] <saml> mongodump, take a node out of replication, change storage engine, restart the node (not joined in replicaset), mongorestore, let it join. repeat?
[15:06:45] <StephenLynx> and saml, from what I heard, is not advisable to use wired tiger on production yet.
[15:08:30] <StephenLynx> so I have a guess you will want to hold on a while before switching your servers to wt.
[15:08:43] <cheeser> saml: the agents open the connection outward so you shouldn't need any firewall tweaking
[15:09:08] <GothAlice> saml: https://jira.mongodb.org/browse/SERVER-17421 https://jira.mongodb.org/browse/SERVER-17424 https://jira.mongodb.org/browse/SERVER-17386 https://jira.mongodb.org/browse/SERVER-17456 https://jira.mongodb.org/browse/SERVER-17542 ancillary: https://jira.mongodb.org/browse/SERVER-16311
[15:11:36] <cheeser> i don't recall what our offical recommendation is. i know mms uses it in production.
[15:12:11] <saml> wiredtiger is default storage engine for mms? or can convert to wiredtiger on demand?
[15:17:03] <cheeser> mms uses WT on some of its node. when you provision via mms, it uses the default engine which is mmapv1
[15:18:02] <GothAlice> Indeed; I tested out my dataset that way. It was fun to watch the elected primary symbol rotate around the nodes during my testing. :3
[15:19:13] <GothAlice> The primary literally couldn't stay up for longer than 30 seconds under my 1K ops/sec benchmark. :(
[15:40:56] <GothAlice> saml: It was ~700MB of data, initially loading the first 500MB then querying that data to generate and upsert the rest (combined reads, getMores, and updates).
[15:41:40] <GothAlice> A typical pre-aggregation workload; the "benchmark" is actually my at-work project's data bootstrap and migration run, part of which re-calculates aggregate click stats.
[15:42:33] <saml> so complicated. i just want to start mongod, no configuration and everything works well up to web scale
[15:43:28] <GothAlice> saml: Slap the MMS agent on a few machines, and yeah, it basically is that simple. (I'm just using somewhat under-sized VMs with tight memory constraints, thus running into multiple issues with WiredTiger.)
[15:45:40] <GothAlice> ^_^ I'm guessing if I threw 64GiB of RAM at my VMs, my test suite won't make them crash. 2GiB, though, and yeah, there be problems.
[16:32:06] <Croves> I'm learning MongoDB and I have a question. My app have two collection: products and tickets. One ticket can store infinite products. Imagine a restaurant that assign many orders to a specific table
[16:32:48] <Croves> I need to relate these two tables. As a mysql user, I'm about to create a products_in_ticket collection and relate them in this new object.
[16:34:21] <StephenLynx> no, there arent foreign keys in mongo, and you never create collections for n*n relations.
[16:42:44] <Croves> StephenLynx: So, my collection should look like something like this? http://pastebin.com/507mKFWQ
[16:43:39] <StephenLynx> duplicate the price because it might change and you won't change it where it has been already recorded anyways.
[16:47:40] <StephenLynx> you would have to get all tickets, which are way more than available products, unwind it, group
[16:48:05] <mah0-deh0> Croves, I'm not the best at mongo, but one of your first steps should figure out if u really need mongoDB or a SQL one as your data is relational
[16:48:44] <StephenLynx> well, you can do with a few relations in mongo. but if you need a join, then you really can't use mongo.
[16:49:09] <StephenLynx> if your amount of queries increase with the amount of results, you can't use mongo.
[16:52:48] <StephenLynx> for stuff that requires a lot of reading and writing and expect to have many users.
[16:53:47] <StephenLynx> as I said, you can go with fake relations, if you understand how that limits you.
[16:55:22] <StephenLynx> if you can use fake relations without performing additional queries to the db, you don't really lose much in performance, or anything at all. and if you would need to perform an unwind on a large set of data to perform a sort, you would even gain, IMO.
[16:55:25] <mah0-deh0> Croves: you should google "mongodb vs mysql" to figure out what do you need, there are A LOT of articles covering this topic already
[17:00:52] <StephenLynx> I don't think stuff like this are designed with that kind of hardware in ming.
[17:01:37] <Croves> StephenLynx: I have a software that manages a entire restaurant. Now I'm building a new version of it, so the waiters will use iPads to note customer's orders and so on...
[17:02:06] <Croves> But for many reasons, I can't keep this software on the web. I really need a physical server into my customer's restaurant
[17:02:59] <Croves> So, I was readinh about RPi and ARM and I don't think I'll be able to run on it... Probably I'll have to switch for a micro-itx PC
[17:03:48] <StephenLynx> a database is kind of an overkill for a system that runs only locally and is not heavy on data load.
[17:04:32] <GothAlice> I've used SQLite on small apps before, for example, an online massage booking system.
[17:04:48] <Croves> Actually, I just have a simple RESTful server built in with NodeJS. Everything else is processed by the browser with Angular
[17:05:31] <StephenLynx> btw,if you use node, I suggest migrating to io.js. its a fork, but much, much better.
[17:06:27] <StephenLynx> a fork of node. all the contributors that are not on joyent's payroll migrated to it.
[17:07:34] <StephenLynx> just released a project that uses it, https://play.google.com/store/apps/details?id=com.cheshire.lynxhub
[18:25:25] <juliofreitas> I've another problem. I'm using: mongodbimport -d testParser -c myCollection --type tsv --file myfile.tsv --headerline and my collections there is a key value like this "" : "" . How can I remove it?
[18:26:15] <GothAlice> db.docs.update({"": {"$exists": 1}}, {"$unset": {"": ""}}) — might work, juliofreitas.
[18:26:54] <GothAlice> The query portion? It's an optimization to avoid needless updates against records not affected.
[18:28:33] <GothAlice> mongoimport is only doing what you're telling it to do, so it really thinks you have an extra field in there named "".
[18:29:01] <GothAlice> If you need to do processing, you're going to have to write an import tool yourself, alas.
[18:31:48] <juliofreitas> GothAlice, db.collection.update({"": {"$exists": 1}}, {"$unset": {"": ""}}) works at mongodb 3.x?
[18:33:41] <GothAlice> StephenLynx: https://gist.github.com/amcgregor/22b70fae30cdfce3e4ed for your benefit, it's not a "showterm". ;)
[18:34:02] <GothAlice> Seems a bit strange to let one insert an empty field name, but then never be able to manipulate it. XD
[18:34:55] <GothAlice> I'll dig around and see if there's an existing JIRA ticket for it after work. :)
[18:36:49] <StephenLynx> so he will have to manually import / export it if he really don't want the empty field.
[19:40:23] <StephenLynx> hey, if a json object has a key with ":", how do I read it? object['lala:lolol']?
[21:57:04] <GothAlice> miceiken: Unless you are aware of the crypto security issues involved, I recommend storing scrypt hashes rather than rolling your own (i.e. "just sha it").
[21:58:48] <GothAlice> Heh; sorry 'bout that. Sometimes picking up my laptop triggers my OTP token. XD Time to generate a new one somewhere else…
[22:01:14] <GothAlice> Still; scrypt handles things like salting for you, it's also designed purposefully to be slow. (And memory hard in addition to being CPU hard, to prevent parallelization attacks. Literally the result of every previous loop iteration, internally, is needed to produce the next iteration.)
[22:02:47] <GothAlice> This differs from bcrypt in that bcrypt uses symmetric encryption to make itself slow, and doesn't have that type of feedback setup; increasing the number of needed iterations increases CPU difficulty, but not memory.
[22:04:41] <cheeser> GothAlice: interesting. i'll see if there's an implementation for the jvm. surely there is.
[22:05:16] <GothAlice> miceiken: https://github.com/bravecollective/core/blob/develop/brave/core/util/field.py#L14-L48 < I use the MongoEngine ODM on top of pymongo; if you do, too, you can borrow my PasswordField implementation. (MIT license.)
[22:06:58] <miceiken> yeah, I heard there was something going on. Anyway, I am literally writing a microservices thing now because I got so inspired by the brave collective IT
[22:07:32] <GothAlice> miceiken: For general web dev chat, there's ##webdev, which I also camp in, BTW.
[22:08:03] <GothAlice> The sheer volume of drama caused in EVE Online by people not paying bills is kinda insane.
[22:09:02] <GothAlice> cheeser: The result of that, BTW, was doxxing and some rather tedious conversations with my employers. (They agreed that gamers getting that upset over volunteer services going away are, themselves, pants-on-head.)
[22:10:06] <cheeser> reminds me of a t-shirt i wish i'd bought years ago: do not taunt me for I am root and have not the need for subtlety.
[22:11:29] <GothAlice> cheeser: I use lines similar to that fairly frequently. http://s.webcore.io/image/3A313C1q1A0y < side effect of camping ##webdev with a female nick.
[22:14:32] <GothAlice> cheeser: Could be worse. Incidents like https://gist.github.com/amcgregor/1447023efb76b97af637 aren't uncommon, either.
[22:17:54] <GothAlice> Yeah. I'd say it destroyed my faith in humanity, but I had little enough going in. ;)
[22:18:22] <cheeser> https://s-media-cache-ak0.pinimg.com/736x/22/34/fa/2234fa431771307180c6071e21080fa2.jpg
[22:22:47] <Feadurn> Is it possible to query a database to find all the documents that have a specific value to a field and doing it again with the next different one
[22:23:08] <Feadurn> I have seen that distinct can output all the distinct value but the dataset contains millions of row
[22:23:29] <Feadurn> so how can I do that without knowing in advance all the distinct value and not parsing them several times