Log file Viewer
#mongodb logs for Wednesday the 14th of January, 2015
[00:03:59] <harttho> Kind of a silly question, but what's the standard way to name Databases/collections
[00:06:12] <lexi2> wanted to know with tag aware sharding if i update a doc thats using tag aware sharding and the update causes it to move to another tag or range. whats the process that taks place
[00:07:34] <Boomtime> lexi2: tag aware is a distribution of the shard key, and the shard key for a document cannot change
[00:08:36] <Boomtime> you can try, but any update that attempts to change the shard key of a document will be rejected
[00:09:27] <Boomtime> if you change the tag ranges themselves though i believe that it just triggers regular migrations.. but i'm not certain actually
[05:13:35] <speaker1234> any suggestions on how to detect duplicate records? The only thing I can think of is add an sha1 of all the variable fields and search on that.
[05:41:45] <Boomtime> speaker1234: can you define what you mean by duplicate records? _id must be unique across all documents, so it is theoretically impossible to have precisely duplicate records
[05:42:55] <speaker1234> The situation is that a customer of mine is sending out batches of records and every so often they send out a duplicate batch because their side burped
[05:43:16] <speaker1234> I have to do data normalization on what they send me so I get to handle all the fun filtering. :-)
[06:24:45] <speaker1234> Boomtime, the only way I can tell if a record is a dupe is if certain fields are the same.
[06:27:37] <Boomtime> speaker1234: if certain fields being the same indicate a duplicate, then make a unique index on the combination of those fields
[06:28:05] <Boomtime> mongodb would not allow the duplicate to enter the database, specifically giving you the error "duplicate key"
[06:28:24] <sijojose> Hi, I've a collection called trainings, inside that there is a filed courses:[{id:1,name:'abc',},{id:2,name:'def'},{id:3,name:'fgh'}] , how can I query for a specific course by using its Id....?
[06:33:03] <Boomtime> sijojose: db.trainings.find({courses.id:<id>},{"courses.$":1}) or something like that
[06:37:26] <sijojose> Boomtime: one more thing for inserting values this approach courses:[{id:1,name:'abc',},{id:2,name:'def'},{id:3,name:'fgh'}] correct right ..?
[06:41:11] <sijojose> Boomtime, in training collection courses is a field.. in courses field I'm inserting values like that... in sql courses will be another table with foreign key reference to trainings table
[09:00:18] <optiz0r> Morning all, I've been banging my head against an authentication issue for a few hours. Could anyone help me with https://gist.github.com/optiz0r/f8ba0b8d382ab0884191 I can login via the shell with the user's credentials and run show collections but attempting to do the same via mongoengine gives an authorisation failure
[09:07:45] <optiz0r> looks like there was quite a lot of change relating to authentication between 2.4 and 2.6 so I should mention this is using 2.4.6
[10:56:58] <optiz0r> I also tried creating the user in the admin database, then adding it with userSource:"admin" in the pulp database; and adding it in the admin database with otherRoles:{"pulp":["readWrite","dbAdmin"]}. Both resulted in the same error message as when the user was defined in the pulp database directly
[11:00:51] <joannac> can you see the connection from your test program authing successfully in the mongod log?
[11:03:43] <optiz0r> joannac: I see the connection but not an auth attempt logged (successful or otherwise) https://gist.github.com/optiz0r/f8ba0b8d382ab0884191#file-mongodb-log Do I need to bump up the logging verbosity to see that?
[11:26:59] <optiz0r> joannac: sorry had to pop away from the desk for a few moments. you're right there was an extra line I omitted (there are replication heartbeats every few minutes so the log is somewhat noisy. I'm slightly confused that it's attempted authentication as __system with a key. I do have a keyfile for replication, but I'm explicitly attempting user/password authentication from the code
[11:27:13] <optiz0r> https://gist.github.com/optiz0r/f8ba0b8d382ab0884191#file-mongodb-log updated with the extra lines
[11:36:41] <optiz0r> ok ignore that, I was grepping on the connection id and pulled in a line from another day by mistake. Even bumping up verbosity to 4 I don't see an authenticate line from my test script
[11:55:07] <winem_> and I told my colleague to download the tar.gz from the website because the apt repos only had 2.4.x this morning
[12:23:16] <vinnie_is_in_da_> One of my slave broke few days ago and I didnt realize it - I switched it back on today
[13:41:34] <jmacdonald> As a beginner, it seems there are enough subtle administrative differeences between 2.4.9 and 2.6.x ... that I should deff go with 2.6 for updated docs. is this assumption correct? i ask cause ubuntu 14.04 has 2.4.9 by default and i have to source up a 3rd party repo to get 2.6
[13:43:10] <jmacdonald> I don't actually use mongo, but i'm being asked to admin a dev setup of it, so i'm more wrapping my mind around roles for doing backups and whatnot.
[14:28:38] <Folkol> Hello. Mongod just filled up my disk and crashed. The data is not important, so I would like to drop the database to free some disk space. But I can not start mongod due to no disk space for the journal... Can I remove the datafiles manually, or will that leave mongo in an inconsistent state?
[14:36:08] <jiffe> is there a way I can start a replica member and have it skip the first operation it will try to sync?
[14:52:07] <dschneider> I tried to update to mongodb 2.6.7 (with yum) but the rpm repository seams to be broken: http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/repodata/primary.xml.gz: Metadata file does not match checksum
[14:53:55] <StephenLynx> http://docs.mongodb.org/manual/tutorial/install-mongodb-on-red-hat-centos-or-fedora-linux/ this is what I use.
[15:01:23] <dschneider> StephenLynx: I have the same mongodb.repo configuration as in the tutorial and It was working before. I also did a "yum clean all" before I tried to update.
[15:35:22] <catphish> i'm planning to store power consumption of a large number of electronic devices (10s of thousands) at 1 minute intervals, i have a couple of basic questions about how to organize the data
[15:36:34] <catphish> firstly, is it common to use separate collections to shard data like mine on a per-device basis, or would you normally throw them all in together into one table as i would have done with a sql database?
[15:37:18] <catphish> and secondly, given that my data set will very quickly exceed available storage, does mongo offer anything to automatically aggregate averages of old data (similar to rrd)
[15:38:45] <StephenLynx> afaik, as long as you don't use $group, a sharded deploy will not have issues with a collection that is distributed among the machines
[16:13:08] <Torkable> can anyone comment on experience with the two-phase commit pattern or any alternatives?
[16:13:27] <Torkable> http://edgystuff.tumblr.com/post/93523827905/how-to-implement-robust-and-scalable-transactions
[17:08:33] <proteneer> my secondaries are stuck in { "$err" : "not master and slaveOk=false", "code" : 13435 }
[17:10:48] <Chubbs> I am building an API where I need to append a few fields to a mongo result (in Node.JS) before sending it as JSON, however whenever I try and add a key value pair to my mongo object it does not appear. I can change the value of an existing field without issue however. Is there some way to make this object modifyable or do I need to clone the result into a new object before I can alter it?
[17:15:09] <StephenLynx> they can cripple your performance because of what they do behind the scenes
[17:16:27] <StephenLynx> Torkable it is anoter travesty for people who want to shoehorn a design other than the one the creators conceived.
[17:22:41] <StephenLynx> you are just adding a layer of complexity and a generic system to your code.
[17:23:17] <StephenLynx> and of course, then there is a sort of dialect "Immutable.Vector.prototype['@@append']"
[17:25:31] <StephenLynx> if people were to consider library code part of their project, they would stop using this much bloat.
[17:50:22] <matejjjj> hello, if I have array:[1,2,3,4,5] how do I query collection for array that has [1,1,1,1,*] where * is any number?
[18:06:20] <StephenLynx> matejjjj I am not sure it is possible. have you read and experimented with $elemMatch and $in?
[18:06:52] <StephenLynx> because not only you are looking for elements in a sub-array (which can be done) but you are looking for a specific sequence.
[18:07:57] <StephenLynx> http://stackoverflow.com/questions/14769355/in-mongodb-how-to-search-in-arrays-for-sequences-and-complex-searches
[18:10:45] <StephenLynx> that is "XY: the article". the author complains about events, but events are just syntax sugar for callbacks.
[18:10:58] <StephenLynx> so he is addressing a solution to the problem instead of the problem itself.
[18:14:20] <StephenLynx> matejjj yeah, mongo won't do that internally, you will have to process the result in your code.
[18:14:39] <StephenLynx> There's a whole stack of ideas that combine to make channels and transducers valuable, but I'll pick just one: events are a bad primitive for data flow."
[18:15:24] <Torkable> thats like saying case statements and ifs solve the same problem so you should never even consider case
[18:15:45] <matejjjj> collection of {array:[1,2,3]},{array:[1,2,4]},{array:[2,1,2]} and to return all objects where array has [1,2,*anyNumber] I didnt sucseed, it allways returns random positions and nubmers
[18:16:15] <StephenLynx> that you can safely just do a boolean comparison instead of an equality check.
[18:16:37] <StephenLynx> with events and callbacks it is the same thing. you have a key and a function.
[18:16:51] <StephenLynx> but with events you have a string and an object instead of a property in a function
[18:18:47] <StephenLynx> so how do callbacks operate differently than callbacks on a fundamental level?
[18:21:28] <StephenLynx> and my point is that the link does not addresses the problem, but a solution to it.
[18:23:19] <StephenLynx> im reading, and I'm addressing the article's part that says "What's the problem?"
[18:23:28] <matejjjj> http://stackoverflow.com/questions/27949764/query-array-particular-array-elements-in-mongodb
[18:24:07] <Torkable> it addresses pros and cons of several different ideas I believe, including callbacks, promises, and FRP
[18:29:20] <StephenLynx> when a functionality has a reason to exists, you can't just "not use it". there will be a case where you will have to use it.
[18:37:34] <Bico_Fino> Hello. I creating a mongo replica set(3 servers). My question is, I need the same size of disks on the 3 servers?
[18:40:39] <Bico_Fino> I can add more replica sets later right? (I'm starting with primary/secondary and arbiter)
[18:46:01] <cheeser> so if your cluster has 5 shards, each shard can be the primary of its replica set for a total of 15 nodes in the cluster.
[18:46:27] <cheeser> data will balance between the shards and that balancing will be replicated in each replSet
[19:16:11] <bmbouter> I've verified that conf file is in use when I start the daemon, and I increased verbosity and started it in the foreground
[19:23:09] <jiffe> is there a way I can start a replica member and have it skip the first operation it will try to sync?
[19:36:47] <Mmike> Hi, lads. When I do rs.initiate() that command returns 'ok' (assuming I didn't do syntax error or provided bogus rs.config or such). But the replicaset is initialized only later. Subsequent calls to replSetGetStatus shows that Mongo goes from throwing OperationalFailure with 'replset being intialized' text, then it is in state startup->startup2->recovering->secondary->primary.
[19:42:56] <bmbouter> and yet in the verbose output I still see 'note: no users configured in admin.system.users, allowing localhost access' in the output
[19:56:57] <bmbouter> and yet I can use 'mongo' and then switch to the database and it lets me pass without auth because I'm localhost
[19:57:48] <harttho> http://docs.mongodb.org/manual/tutorial/add-user-administrator/#authenticate-with-full-administrative-access-via-localhost
[19:59:40] <bmbouter> http://docs.mongodb.org/v2.4/tutorial/add-user-administrator/#authenticate-with-full-administrative-access-via-localhost
[20:06:11] <bmbouter> well I have auth now, but I need this to have certain roles on the pulp_database
[20:06:51] <bmbouter> but do I make it in the admin db like we did here, or do I add it into my specific db of interest 'pulp_database'
[20:06:59] <harttho> I'd have to brush up, but I think if you have the second user within the pulp_database, you can restrict things within that database under the umbrella of the admin user
[20:07:28] <bmbouter> do you always configure users in 'admin' or on the database itself 'pulp_database'
[20:08:08] <harttho> admin users go in admin, other users should be put into the pulp database would be my recommendation
[20:08:58] <harttho> "Authentication requires at least one administrator user in the admin database. You can create the user before enabling authentication or after enabling authentication."
[21:18:14] <jayjo> If I'm running a mongo daemon on my server to hold my data, what is the best way to ensure that data's integrity? Do I copy it to a local machine every night with a cron job or spin up a separate server for that same task?
[21:22:07] <joshua> I think that depends on how important the data is and what your resources are. You can do a database dump during off peak hours if you have a period where it won't interfere, you can run a replica set and use one of those nodes to do a filesystem snapshot, you can snapshot the filesystem where you have it now.
[21:33:55] <joshua> Yeah if you can leverage MMS, go for it. Saves having to figure out all the stuff on your own
[21:59:39] <theRoUS> is there a way to apply .distinct() to a .find() resultset? i.e., i want to find the distinct values for a field within a subset of the collection rather than the whole collection
[22:16:44] <FunnyLookinHat> If I have a bunch of items in a collection with the same indexed key - is there a fast way to update each to have a new value for a different key rather than looping each record?
[22:17:44] <FunnyLookinHat> Ah I just wasn't googling right: http://docs.mongodb.org/manual/reference/method/db.collection.update/
[22:20:10] <catphish> i am looking to store large quantities of time-based numerical data, too much to reasonably store, so i'd like to store hourly, daily averages automatically and delete older data, is this easy to achieve?
[22:21:35] <kexmex> i looked through it, and besides save(), i see only updates with $set on specific fields
[22:22:28] <catphish> i know i can set a ttl on records to auto-delete, but i'm not sure how to handle the automatic aggregation
[22:23:33] <catphish> it's the aggregation query i'm unsure about, given the large quantity of data, i'm not sure what it should look like
[22:24:41] <catphish> the data is quite simple {"device":1, "timestamp": "2015-01-01 00:00:00", "value":100}
[22:25:07] <catphish> this will be stored at approx 30 second intervals for many thousand values of "device"
[22:25:49] <catphish> but i'd also like to store an average of the data for each hour (obviously kept for longer) and each day (kept indefinitely)
[22:26:28] <catphish> so my main question is how to build the command to build the hourly / daily data
[22:27:34] <Torkable> create a small program in the language of your choice that runs the aggregation query
[22:28:59] <catphish> i'm just trying to get my head around what that aggregation query will look like
[22:29:51] <Torkable> but you could store the result object in a new collection so you have a log of the data through the day
[22:30:44] <joannac> do you have profiling on, or is this a replica set? do you have verbose logging?
[22:32:35] <catphish> Torkable: thanks for your help by the way, i'm afraid i'm quite new to mongo so i hope i'm explaining myself clearly
[22:32:59] <catphish> as said, my input data is {"device":1, "timestamp": "2015-01-01 00:00:00", "value":100}, for many values of timestamp and device, and i will delete data in this collection older than approx 72 hours, i then want hourly averages for each value of device (i assume in another collection), however it is probably insane to completely regenerate all the hourly averages every time
[22:33:23] <kexmex> but maybe other fields that are present in this bad document, are missing elsewhere
[22:36:27] <catphish> Torkable: thanks, am i right in thinking i can just overwrite data in my "hourly_averages" collection at regular intervals, looking only at data from he main table from the previous 120 minutes, running the query every 30 minutes, to ensure everything is correct?
[22:37:13] <joshua> You can generate another date for when the average date was added, and put another TTL on it
[22:46:25] <kexmex> i am guessing i should have a writer thread for this and i should just queue commands and have the writer thread pick them off one by one in proper order
[22:52:47] <kexmex> so when i do collection.Save(), i am handing it over, and if the thread that called .Save() dies, that shouldn;t prevent the insert from going through, should it?