Log file Viewer
#mongodb logs for Friday the 6th of June, 2014
[01:25:31] <dman777_alter> http://bpaste.net/show/4I7LDFQa3kKxbcP5kzrn/ I'm new to mongoose and mongodb. I have a static field metrics : [ { duration : "15m"...when writing the function that creates the document should I include it also as I have in the paste?
[02:17:07] <dman777_alter> http://bpaste.net/show/kAcBu9SK1kDZDxM1jku3/ ok, my first attempt to save a document. but it's not saving :(
[02:21:27] <dman777_alter> joannac: no...and I placed a console.log("hi"); message inside of cpu.save() which doesn't show anything ether.
[02:24:49] <dman777_alter> joannac: no, because none of the cpu.save() is not running. I also went into mongo shell and db.cpu.find() didn't return anything
[04:08:57] <Cache_Money> I'm on a MBP. I usually start the Mongo daemon using $ mongod But now I'm getting this error: ERROR: dbpath (/data/db) does not exist.
[04:25:44] <Cache_Money> I know I used to have other collections in my mongodb before but oh well...
[07:26:01] <gancl> Hi! How to update or insert subdocument's array? http://stackoverflow.com/questions/24075910/mongoose-cant-update-or-insert-subdocuments-array
[07:52:27] <narutimateum> 2nd question... do i put reference in subdocument like this "_id" : ObjectId("538ef097576f1f7d505c2ebc"), or "_id" :"538ef097576f1f7d505c2ebc",
[08:06:24] <narutimateum> so i have a problem where i dont know actually i get this objectId type to register into mongo
[09:21:53] <djlee> Would i be right assuming that if i wanted to scale mongodb horizontally on AWS EC2, i would have to have my application code write to the master mongo/ec2, and place the slaves behind a load balancer and have my application do reads from the load balancer
[09:35:07] <djlee> kali: Ah, just noticed primary and secondary replica set members in the PHP MongoClient class, must be what you are referring too. Seems to only relate to reading rather than write, but i'll ask google, as that must be the stuff i need to look at
[09:36:45] <kali> djlee: you should not have to interact with these classes anyway, but use the ReadPreferences system instead: you can specify at the client scope, database, collection or even query scope where you want to read to go
[09:37:17] <kali> djlee: and DONT try putting a replica set behind a tcp load balancer. it will not work
[09:38:32] <_boot> if I start updating shards/replica set members to 2.6 from 2.4.9, is there likely to be any problems?
[09:40:18] <djlee> kali: ah right, load balancer idea is out the window then. No doubt i need some sort of sentinel/manager, which monitors slaves and i can use to query and return a suitable slave IP (dont worry, i'll happily google it, clearly got a lot to learn quickly :P)
[09:41:40] <djlee> kali: i'll read the mongodb docs on replication e.c.t. clearly they've implemented most of the stuff i thought id have to manually manage!
[09:41:47] <kali> djlee: the only thing you need is some kind of out-of-band monitoring to allow you to know that your replica is degraded
[09:43:18] <djlee> kali: yeah im still new to the nosql world. Spent my entire career on single MySQL hosts. Now i've got an api, mongo and redis instance that all needs to scale well, in the cloud, by next friday, and have been load tested (god i love client deadlines)
[09:51:46] <Nodex> strangely I employ a similar tactic for Mongo to solr mappings and de-normalisation
[09:52:29] <Nodex> all my mappings are simply json objects that are run through a parser that cleans / maps / parses things to build an eventual object for solr/mongo
[09:53:48] <Nodex> This work is free. You can redistribute it and/or modify it under the terms of the Do What The Fuck You Want To Public License, Version 2, as published by Sam Hocevar. See the COPYING file for more details.
[09:54:15] <Nodex> I run it as an API, I simply throw an object to it and get back two objects - one for mongo, one for solr
[10:21:53] <wenxueliu> I make a shared library which call by lua. but get ./client.so: undefined symbol: mongoc_collection_destroy
[10:58:30] <rainerfrey> I get errors on the mms support form, so I ask here: what's up with MMS UI requesting to set up two-factor auth? I used to be able to access my MMS account without, but all links are redirecting to the setup form ...
[12:27:08] <gancl> Hi! How to let mongoose update or insert subdocument's array? http://stackoverflow.com/questions/24075910/mongoose-cant-update-or-insert-subdocuments-array
[13:07:44] <gancl> nemux: I know it. But it's in the Array, it returns all items in a array, I just want to modify only one item
[13:26:46] <dhm_edx> gridfs and I have been fighting for 2 days. I issue queries which take hours but show almost no activity other than locks.
[13:28:26] <dhm_edx> what I'm trying to accomplish is that we changed the key order in our subdocs we use for files_id and I'm trying to find the chunks with the intermittentlly screwed up order and fix their files_id
[13:29:14] <kali> dhm_edx: an index on "n" should help this query. also try using count({..}) instead of find({ ...}).length
[13:29:59] <dhm_edx> i'm going to iterate over extra. I just wanted to see how many records I'm dealing w/ as they tell my users how many images are broken in current courses
[13:35:58] <dhm_edx> my mongo monitoring console shows ~ 100% lock ratio but only about 3 command/sec and about 1 write/sec (I have another query updating 284 of the entries)
[15:36:47] <dgarstang> Can anyone recommend a way to deploy mongodb cluster automatically? The chef cookbook for it kinda sucks
[15:45:37] <_dougg> Can anyone recommend a way to deploy mongodb cluster automatically? The chef cookbook for it kinda sucks
[15:50:18] <mischat> does anyone have a feel for how long releases stay in rc before being shipped ?
[15:51:34] <_dougg> i'm still waiting on finding out if there's a way to easily automate deployment of a mongo cluster.
[16:03:05] <saml> db.urls.find({url:{$not:{$regex:/\/fashion\/fashionshows\/season/}}, type:'fashion_archive'})
[16:03:25] <saml> i want to find docs whose .url does not match regex but whose .type is fashion_archive
[16:04:16] <_dougg> i'm still waiting on finding out if there's a way to easily automate deployment of a mongo cluster.
[16:04:32] <saml> db.urls.find({url:{$not:/\/fashion\/fashionshows\/season/}, type:'fashion_archive'})
[16:13:00] <jigax> collection.aggregate({ $match: { techId: req.user.techId } },{$group : {_id: "$Order Class", data: {$sum: 1}}}
[16:19:20] <_dougg> i'm still waiting on finding out if there's a way to easily automate deployment of a mongo cluster.
[16:26:45] <federated_life1> _dougg: its pretty easy already , there should be an announcement at mongo world with some automation tools similar to MMS
[16:40:51] <achiang> hello, i have successfully configured a collection to have a 2dsphere index, and geospatial queries such as geoNear already work. question is -- is there a "normal" search to see if the collection already contains a coordinate pair?
[16:41:31] <achiang> use case is preventing duplicate data from being inserted, so search collection for the coordinates first, if they exist, then do not insert
[16:47:56] <heewa> another aproach is to insert, check for dupe & remove yours. easier in some cases, if you don’t mind having dupes for a short period. (like in complex, multi-document, multi-collection scenarios)
[16:49:20] <dhm_edx> @NodeJS_ all docs and subdocs are ordered; however, not all language impls provide access to the key order
[16:52:21] <NodeJS_> dhm_edx: Let's say I have the following documents in collection: {_id: ObjectId(1), title: ''}, {_id: ObjectId(2), title: ''}, {_id: ObjectId(3), title: ''}, how can I get it in specific order?
[16:52:58] <_dougg> Is there a good way to deploy an entire cluster? The community chef cookbook designed for the task is pretty flaky. Enterprise app... no way to automagically deploy?
[16:53:29] <dhm_edx> ah, by definition, order of persistence for separate objects does not guarantee order of retrieval; however, since ObjectId encodes the time stamp, you can sort by the implied time stamp I believe
[17:02:57] <moleWork> but i'm spining up mongodb replicaset's with it on opsworks http://netinlet.com/blog/2014/01/18/setting-up-a-mongodb-replicaset-with-aws-opsworks/
[17:05:34] <moleWork> i also have a similar vagrant setup so i can spin up dev clusters that closely match production cluster on aws
[17:09:20] <og01_> Hi, i've come across something that i don't know how to do, i've pastebined an example: http://pastebin.com/BTQPgBvB
[17:09:20] <_dougg> moleWork: you aren't having problems with it? I can reproduce bring up 2 replicas in a shard, but bringing up the third... chef fails
[17:09:47] <_dougg> moleWork: i have to add the third node manually in the primary replica, and then go back and run chef
[17:09:57] <moleWork> i don't have any shards sorry.... just replicaset.... i do have a 3 node replicaset no problem though
[17:10:04] <og01_> I want to aggregate on the max time of one field while pulling out the another field from that max
[17:10:53] <og01_> please take a look at my paste, I might not be explaining my question well, i think the paste makes it clearer
[17:13:10] <tscanausa> _dougg: I actually rebuilt the chef cookbooks for my needs for both replica sets and shards.
[17:14:25] <moleWork> and chef seems pretty complicated soo many subtlties to timing and how it's works
[17:15:00] <_dougg> well I'm open to alternatives. Mongo is supposed to be an enterrprise app. An enterprise app should have a way to automate deployment of a cluster
[17:15:16] <moleWork> well there's the main edelight one... then the parsops one https://github.com/ParsePlatform/Ops/tree/master/chef/cookbooks/mongodb
[17:15:37] <_dougg> moleWork: the Edelight one is the one I started with. I've forked it to fix issues, but it still has more
[17:17:08] <moleWork> it's based on an old version of the edelight one and has it's own issues and compatbility problems
[17:17:46] <_dougg> the installation of mongo is easy... it's the orchestration and config that's a pain
[17:18:15] <moleWork> opsworks makes that a bit easy since you have access to all the instances within the built in opsworks attributes
[17:18:40] <moleWork> i want to use docker/ansible but i don't want to customize my whole aws stack to do it
[17:25:16] <_dougg> I think the problem with the Edelight cookbook is that it works with 2 nodes because the first one has nothing to do, the second one becomes the primary, and the third one... well I just think they never tested that
[17:27:35] <og01_> Im sure what im trying to do must be possible, can someone take a look at my pastebin example? http://pastebin.com/BTQPgBvB
[17:27:50] <moleWork> https://github.com/edelight/chef-mongodb/blob/master/definitions/mongodb.rb would try to debug whether or not your query on line 204-210 brings you back all three nodes
[17:28:12] <og01_> im trying to think of ways to project the data, or use multiple stages in the aggregation pipeline, but i can't work it out
[17:28:25] <_dougg> moleWork: reading the docs on mongodb. even that says you had to add nodes via the primary. That's never going to work with chef.
[17:29:05] <_dougg> moleWork: you add the third node, and bring it up. Chef fails because it's trying to run rs.add on a non primary node. You have to go back to the primary and run the add by hand and then chef will run ok on the third node.
[17:31:39] <tscanausa> that command will run the other 2. its really not difficult. you just need the master to run the add commands
[17:33:57] <moleWork> it's ugly code but i'm pretty sure that code in that cookbook i initialially pasted works
[17:35:16] <tscanausa> this is the ruby code I use to manage replica set https://gist.github.com/anonymous/1910558ada1354d101c9
[17:35:42] <moleWork> https://github.com/netinlet/chef-mongodb_replicaset/blob/master/mongodb/libraries/mongodb.rb if you look at line 138 onwards in this file
[17:36:35] <_dougg> it cant configure the replica set. The second node is the primary, and the third node is not
[17:38:04] <dhm_edx> NodeJs_ once again, afaik, position is a nonsensical notion wrt dbs; however, if you have the notion of an order, then just add a field to represent that order and give it the serial # you want.
[17:38:40] <dhm_edx> if you need to insert between things frequently, then use floats or leave enough room for your upperbound # of inserts
[17:40:09] <fschuindt> hey guys, there is a way to set a default mongo config file? I always need to type $ mongod --config-file ~/mongo/mongodb.conf I want just to type mongod then go
[17:40:43] <_dougg> moleWork: it's just calling connection = Mongo::Connection.new("localhost", node['mongodb']['config']['port'], :op_timeout => 5, :slave_ok => true, :ssl => true, :ssl_cert => "/etc/ssl/mongodb-#{node.chef_environment}.pem")
[17:40:57] <moleWork> when you connect to it as a replicaset, even with partial host list i believe it seeds the connection so you get to the primary (i think)
[17:41:21] <_dougg> when I connect with the same options via the cli shell, I get local, not primary
[17:44:40] <moleWork> yeah this code big and ugly with huge else branches but if you read the code starting with " # remove removed members from the replicaset and add the new ones" you can see how it reconfigures the replicaset starting with the old members that already exist
[17:46:18] <_dougg> oh shit. it's connecting to the first node, which is second, not the primary. WTF?
[17:48:12] <moleWork> if you bring up the nodes with explicit priorities you can then know exactly which one will come up as primary
[17:49:34] <moleWork> i haven't played with that yet cause i ditched that cookbook too because it didn't do what i wanted :(
[17:49:41] <_dougg> but then i have to manage node attributes, which sucks. i wonder how that works if we have to promite a slave to primary too
[17:50:31] <tscanausa> you only need priority if you are having replica sets out side of your normal "cluster"
[17:50:34] <moleWork> if you need to do any thing it basically has to be done from primary so setting priorities makes it so you don't have to guess at which one is primary
[17:50:51] <fraterlaetus> Hi. Is it possible to use mongodump or a similar utility to dump only the last 24 hours of data? I'm trying to reduce the size of the dataset I use for testing.
[17:54:54] <tscanausa> kinda a problem with the system design and lack of quality effort as in there are a lot of broken cookbooks
[17:57:05] <moleWork> https://github.com/opscode-cookbooks/aws/blob/master/resources/ebs_raid.rb this particular resource is causing me problems
[17:57:10] <_dougg> i'm not even sure this shitty cookbook takes the priority into consideration when selecting the primary
[17:57:50] <_dougg> cmoleWork: i know but the cookbook seems to do a chef search and then pick the first
[17:58:37] <moleWork> yeah by setting priority or removing all other members and reconfiguring it so that it's the only member
[18:02:39] <moleWork> i don't think mongodb allows you set the master other than implicitly through priorities
[18:06:15] <_dougg> even if it works, this makes me nervous. I don't want to manage a prod db cluster with a flaky chef cookbook
[18:06:44] <moleWork> yeah that's why i'm writing mine from scratch, there is too much of a mess of horrible ruby code in this stuff
[18:07:24] <_dougg> i understand chef... I've been knee deep in it for months. it's ruby that I hate
[18:08:11] <moleWork> i don't hate ruby i just wish it didn't exist and that this automation stuff wasn't such a mess
[18:10:45] <tscanausa> I think your problem with chef is you are using the features that were added months to year ago that no really uses
[18:11:46] <tscanausa> moleWork: said he was use the ebs raid resource. i have never heard of anyone else using it
[18:13:19] <tscanausa> ya. that could be another thing. opscode isnt exactly chef so the code resources may or maynot work there.
[18:16:34] <moleWork> which should work cause i only need persistence as long as the recipe is running
[18:18:12] <moleWork> and it doesn't look like this is reusable https://github.com/aws/opsworks-cookbooks/blob/release-chef-11.4/ebs/recipes/raids.rb
[18:18:54] <tscanausa> moleWork: i think set is not chef solo capabile as that should update the server.
[18:20:13] <moleWork> but you are pretty much right on this issue... "this cookbook does not work on opsworks"
[18:24:25] <moleWork> [2014-06-06T16:17:19+00:00] ERROR: aws_ebs_volume[sdi1] (/var/lib/aws/opsworks/cache/cookbooks/aws/providers/ebs_raid.rb line 339) had an error: RuntimeError: Volume no longer exists
[18:24:25] <tscanausa> I make the statement that a lot of the mongo cookbooks out there are crazy complicated because of mongodb's system architecture.
[18:24:51] <moleWork> just trying to bring up a raid on opsworks through chef config and not through the web ui
[18:25:00] <tscanausa> moleWork: I am saying go into the #chef irc and ask "Does aws/provider/ebs_raid work on opsworks?"
[18:27:35] <moleWork> i think i'm just gonna give up on automating the raid on opsworks with this recipe
[18:28:25] <moleWork> it'd be nice to understand why it doesn't work though cause, after staring at this for almost 3 days now i can't figure it out
[18:28:36] <tscanausa> I know that feeling right now aws php sdk is not working and I cant find out why
[18:39:47] <moleWork> yeah it's not ontopic... random convos.. question didn't peak anyone's interest it looks like
[18:40:54] <moleWork> i'll figure it out eventually but i've already wasted too much time trying to save some time
[18:51:56] <moleWork> something in those cookbooks must rely on some "passing of state" that simply can't work with chef-solo
[18:55:01] <moleWork> one thing i actually should of check if the source for the aws cookbook in the cache matches that of the one on the opscode-cookbook repo
[18:56:00] <tscanausa> moleWork: ya that one of the fun things my infrastructure runs on 3 clouds so I have to stay away from the proprietary stuff.
[18:56:20] <_dougg> moleWork: i wouldn't do it that way. unless your careful, chef could blow away your disks. I know from experience. I'd write a script that runs on boot and never runs gain
[18:56:38] <moleWork> yeah, i mean i'm fulling willing to role custom if my boss gives me goahead but i don't want to support something i don't have to if something exists that "just works"
[18:57:16] <moleWork> _dougg, well, i only think "setup" runs once and the cookbook checks to see if the volume already exists
[18:57:39] <moleWork> and technically nodes should be ephermeral so if it blows out my disks... it'd be a good test to see how i handle it
[18:58:17] <moleWork> http://docs.aws.amazon.com/opsworks/latest/userguide/workingcookbook-events.html
[18:59:11] <moleWork> the configure event is the one that bbugs me cause i'm worried it'll force mongodb primary to drop all connections on a reconfigure in some situations
[19:23:45] <Viesti> maybe in an odd way, using reduce-kv for just iteration, and doing mutation while traversing the transient map: http://pastebin.com/EA2yhtcD
[19:40:14] <modcure> I created a sharded cluster and a database with some collections. i restarted the server and started the sharded cluster the data is gone.. any thoughts?
[19:49:54] <_dougg> This edelight mongo cookbook is such a POS. If I run chef twice on the first shard node, it becomes the primary.
[20:06:53] <q85> Upgrading to 2.4 here. We have our config servers running on the same host as our replica sets. The upgrade instructions state you should upgrade the config servers first then the mongod instances. My question is can I replace the mongod binary while mongod is still running?
[20:09:05] <q85> cheeser: thanks for the reply. We are running linux though. I should have mentioned that.
[20:55:12] <_dougg> Might as well ask again... is there a way to deploy an entire mongo cluster in an automated fashion?
[21:11:13] <ehershey> _dougg: https://mongodb-documentation.readthedocs.org/en/latest/ecosystem/tutorial/automate-deployment-with-cloudformation.html
[21:13:51] <NodeJS___> Does anybody could resolve my problem which I'm faced in MongoDB http://stackoverflow.com/questions/24089031/mongodb-how-to-make-ordering-of-documents-but-not-array
[21:23:56] <fraterlaetus> mongodump -d databasename -c collection -q "{ created_at: { \$gte: new Date('06/01/2014') }}"
[21:27:44] <ehershey> fraterlaetus: I don't think you can create a date object in your query like that
[21:35:01] <_dougg> ehershey: that doc talks about installing a single node I think.... I'm trying to deploy an entire cluster
[21:37:19] <amcrn> _dougg: http://docs.mongodb.org/ecosystem/_images/MongoDB_ReplicaSetStack.template
[21:42:31] <moleWork> _dougg, this is totally working for me... except i slightly modified it to pass in rs_nodes instead of querying it.. and i'm not doing any of the sharding... just replicaset but i can bring up 1,2,3 doesn't matter... so far it has "just worked" https://github.com/netinlet/chef-mongodb_replicaset/blob/master/mongodb/definitions/mongodb.rb <---
[21:43:14] <fraterlaetus> hrm. so assuming "created_at" is an ISODate, why would -q "{ created_at: { \$gte: ISODate('2013-09-10 21:50:05.844Z') }}" fail to parse?
[21:44:21] <fraterlaetus> mongodump -d db -c collection -q "{ created_at: { \$gte: ISODate('2013-09-10T21:50:05.844Z') }}"
[21:44:37] <fraterlaetus> assertion: 16619 code FailedToParse: FailedToParse: Bad characters in value: offset:21
[21:45:54] <moleWork> https://github.com/netinlet/chef-mongodb_replicaset/blob/master/mongodb/recipes/opsworks_replicaset.rb
[21:46:05] <moleWork> i also modified this cause i didn't like that he was using the replicaset name as the layer name
[21:47:38] <_dougg> moleWork: if you use cloud formation, you can change the template, and call update() and it .... should.... work....
[21:48:23] <moleWork> which is what i'm currently using, but now researching how i could run my own clusters
[21:49:14] <moleWork> opsworks seemed apealing but it's killed me for over a week now because of small gotacha all over the place
[21:51:36] <moleWork> on thing that's cool is that i can add a node to my mongodb replicaset and then the app can magically be reconfigured and reseeded with the new hosts of the replicaset
[21:52:04] <moleWork> i wasn't too sure how i was gonna have my app "discover" the new database servers before
[21:53:05] <_dougg> moleWork: your code... epic fail ... "uninitialized constant Chef::ResourceDefinitionList::OpsWorksHelper"
[21:55:30] <moleWork> okay... i don't know much about chef but .... https://github.com/netinlet/chef-mongodb_replicaset/blob/master/mongodb/libraries/opsworks_helper.rb i changed "class Chef::ResourceDefinitionList::OpsWorksHelper" to "module OpsworksHelper" then the https://github.com/netinlet/chef-mongodb_replicaset/blob/master/mongodb/recipes/opsworks_replicaset.rb line ::Chef::Recipe.send(:include, MongoDB::OpsWorksHelper) i changed to "::Chef::Recipe.send(:include, Opswo
[21:56:19] <moleWork> not sure why people define their libraries as ResourceDefinitionLists.... but i'm not a ruby expert... it looks like they are sorta doing a HWRP as a library but i don't know what a ResourceDefinitionList is
[21:57:43] <moleWork> the key is... the actual mongodb resource definition logic in this seemed to work for me... once i got passed all of the nuances of the rest of the recipie