Log file Viewer
#mongodb logs for Tuesday the 20th of May, 2014
[00:12:02] <jamesaanderson> If I have a users collection that looks like this: http://pastie.org/9191184. I can't seem to figure out how to select a post by id and return only that post. What am I missing?
[00:18:19] <joannac> no, I take that back. The positional operator http://docs.mongodb.org/manual/reference/operator/projection/positional/
[00:18:30] <dgarstang> dangit. now I can't add a shard. ""errmsg" : "couldn't connect to new shard socket exception [CONNECT_ERROR] for rs1/mongo-shard-shard0-rs1.prod.foo.com:27017""
[00:20:34] <dgarstang> the mongos server obviously needs to be able to reach the shard on port 27017. Does the shard need to communicate back to the mongos box?
[00:24:14] <dgarstang> joannac: i mean... do I have to open a firewall hole to allow shard to communcate to mongos in addition to allowing mongos to reach shard
[00:24:22] <federated_life> dgarstang: yea..once you get it up and running, you'll notice that each mongoD of the replica set of each shard keeps connections open to the config servers , the mongoS
[00:25:11] <dgarstang> federated_life: in this particular case, I'm running the addshard command and getting a connect error. Running tcpdump on the shard box though shows that it's receiving the traffic from the mongos box
[00:32:32] <dgarstang> joannac: as I said above, tcpdump shows that the mongod shard is receiving traffic from the mongos box, so it's not a firewall/connectivity issue. It's also logging the connection but nothing else.
[00:34:16] <dgarstang> joannac: yep... mongo --host mongo-shard-shard0-rs1.prod.foo.com --port 27017 works
[00:35:26] <dgarstang> but again, running sh.addShard("rs1/mongo-shard-shard0-rs1.prod.footest.com:27017") results in "errmsg" : "couldn't connect to new shard socket exception [CONNECT_ERROR] for rs1/mongo-shard-shard0-rs1.prod.footest.com:27017"
[00:43:32] <dgarstang> joannac: actually, no. There's nothing in the mongo doc at http://docs.mongodb.org/manual/tutorial/deploy-shard-cluster/ about that. I did see someone saying you had to run rs.initialize() first but when I did that it complained mongod had not been started with the replset option.
[00:45:18] <dgarstang> there's really nothing about initialising at http://docs.mongodb.org/manual/tutorial/deploy-shard-cluster :)
[00:45:32] <joannac> okay, I interpret "adding a replica set" as having the pre-condition "set up a replica set"
[00:46:06] <dgarstang> I was under the impression, based on the doc, that running addShart() did that.
[00:48:08] <dgarstang> do I need to start mongod on the shard with the replset option (even though it's not mentioned in the doc?)
[01:20:32] <joshua> 5-6k documents deleted per second, this should only take another couple hours. heh
[01:31:52] <nullne> i am running fedora 20,and installing the latest version of mongodb, I was able to run the mongod daemon as a service without error the very first time but when I shut down my machine and came back, the service refused to start due to some failure.
[01:33:07] <nullne> ERROR: Cannot write pid file to /var/run/mongodb/mongod.pid: No such file or directory
[01:34:33] <nullne> when i start the mongos :sudo service mongos start ,i got this erro message : Job for mongod.service failed.See 'systemtcl status mongos.service' and 'journalctl -xn' fro details
[01:35:38] <nullne> i follow this resolution http://stackoverflow.com/questions/23086655/mongodb-service-will-not-start-after-initial-setup, and remkdir the mongodb
[01:41:16] <cheeser> check permissions on that directory and the ownership. make sure it works for whatever users mongod runs as.
[05:56:42] <Soothsayer> Does the size of the field name matter from the point of view of sorting on that field name?
[07:38:27] <stevenm> Hey, anyone know of any prebuilt ARM (compatible with say a Raspberry Pi) packages?
[07:41:52] <stevenm> rspijker, well I've found plenty of guides showing it being compiled on a Pi and working
[08:20:33] <justind> rspijker: Is there another function that will return the difference of two arrays of ObjectId?
[08:24:17] <justind> The returned array should contain all ObjectId's in array 1, if that ObjectId is not in array 2.
[08:28:19] <rspijker> what you want to comapre is the string content of the ObjectId and not the actual Object
[08:31:42] <rspijker> then you’ll end up with a list of stirngs though, which might be ok, again, depends on what you want exactly
[08:32:30] <justind> Yeah I was thinking of doing that, but I was hoping there would be a more elegant solution.
[08:33:21] <rspijker> I’m not 100% up-to-speed on Lo-dash, so there might be some fancy-ness I don’t know about
[08:33:53] <rspijker> but I’m not aware of any differenceBy function or anything like that, where you can pass in a comparator-function, which is kind of what you want…
[08:47:36] <jacksmith> I need some advice, I have a big data coming from some GPS devices, which i need to store it for months and analyze it, currently i am using MySQL but it is very slow for this work, can mongoDB can helpfull to me in this case ?
[08:49:48] <kali> jacksmith: mongo is good for small, fast, targetted requests. it should be fine for collecting the data. but if you plan big scala data analysis, you may want to complement mongo with something from hadoop (like hive or pig) or spark/shark
[09:03:11] <jacksmith> I need some advice, I have a big data coming from some GPS devices, which i need to store it for months and analyze it, currently i am using MySQL but it is very slow for this work, can mongoDB can helpfull to me in this case ?
[09:22:30] <rspijker> indices is more prevalent in the rest of the world. Both are correct plurals of index though
[09:26:02] <rspijker> then again, the shell provides both db.coll.getIndexes() and db.coll.getIndices() :)
[10:38:27] <talbott> so they need some additional role to be able to authenticate to the db in the first place, instead of having to authenticate against admin and then change
[10:58:06] <rspijker> I think you can set up users on specific dbs as well that inherit from the admin db, but you would have to check the docs for details
[11:00:45] <netQt> Hi all. I need to traverse very large db and make updates on each row. It would take to much time to select all data then process. Is there a way to process as soon as selecting?
[11:04:44] <netQt> I want something like iterator. Without having to select all data before starting to make updates
[11:06:03] <netQt> Assume we have 1 million documents in the db, and we need to update one of the fields in all of them
[11:07:32] <netQt> I read info from each doc, according to the data I make request to some services, get data, and update it int db
[11:10:15] <rspijker> it will perform the select first (I believe), so the query will still take time. But it won’t get all of the documents from the database, it just gives you this cursor into the resulting set
[11:15:14] <rspijker> that is, if you iterate over the cursor, technically you would never have to have more than 1 document in memory at the same time
[11:17:31] <rspijker> but if your indices are bad, you’ll still end up with a large memory footprint
[11:19:43] <rspijker> or if you have a field that you know to be monotonically increasing (like an ObjectID), you could use findOnes with a $gt on the id field…
[11:20:11] <rspijker> but this is all very contrived… You should just let the kernel and mongod deicde what to do, trust me when I say they’ll be better at it than you
[11:27:13] <netQt> I found this query streaming http://mongoosejs.com/docs/2.7.x/docs/querystream.html
[11:32:20] <rspijker> hmmm, apparently cursors behave a bit better nowadays: http://docs.mongodb.org/manual/core/cursors/
[11:47:21] <Fiibar> hi, how can i check for double uniqueness? i have a mongodb document in my symfony project. i want to have a check on two fields for uniqueness. how? mongdb/unique can be done on one field by default, right?
[11:50:36] <Fiibar> * @MongoDBUnique(fields = {"content", "people"}, message="exists.already") >> not workin'
[13:05:00] <Industrial> Hi. I'm trying to build an export feature (to CSV) by merging several streams that come from mongodb aggregations. I'd like to support progress reporting while the export is running in the browser, so I need to know howmany documents will be streamed from the aggregation and at what point it's currently at (So I can send 20% 30% 40% etc to the browser). How would I do this?
[13:36:29] <Industrial> http://mongodb.github.io/node-mongodb-native/api-generated/collection.html#aggregate
[13:36:44] <Industrial> http://mongodb.github.io/node-mongodb-native/api-generated/cursor.html#count
[13:37:35] <Industrial> But the Stream object I'm getting from Collection#aggregate doesnt have a count method
[13:43:53] <Industrial> tscanausa, so the method .count() is only put on the instance of Cursor after .. what event?
[13:58:41] <Industrial> its a stream, so its already sending output before it is done, instead of loading everything into memory, right?
[13:59:23] <Fiibar> In Symfony config for MongoDB Doctrine bundle there is an option for "default_database" is there an easy way to set another database? Like different databases for each user for example.
[14:03:46] <tscanausa> Industrial: in my experience the aggregation is already done by the time you get the cursour
[14:15:18] <tscanausa> when I remove a shard, my mongos server complain about not being about to connect to the old shard. Is this expected?
[14:33:32] <wickwire> is there some way in mongodb, to omit the results enclosure for the documents returned by an aggregation framework query...?
[14:38:54] <rspijker> wickwire: just return xxx.result instead of xxx to whatever consumer you’re using?
[14:40:05] <wickwire> I think it's ok now, using find and the regex, I get the filtered documents and the same find() structure
[14:40:42] <rspijker> if all you were doing in the aggregate was a match, then you can do the same with a find
[14:46:17] <q85> Do the mongod instances (replicasets) have to be running to perform the meta-data upgrade when moving to 2.4.x?
[14:49:30] <tscanausa> q85: generally you leave everything running and can do a "liveish" upgrade but doing one server at a time
[14:50:53] <q85> tscanausa: but if a mongod was down at the time the mongos --upgrade was run would the upgrade complete as normal.
[14:52:42] <q85> I'm curious if the config servers are all up and a mongod is down, will the upgrade to the meta-data still complete ok.
[14:56:03] <tscanausa> from the release noted it looks like the upgrade just changes information on the config servers.
[15:25:46] <q85> tscanausa: that's also what I'm thinking. I'm going to assume the upgrade was completed successfully.
[16:42:24] <zerooneone> hi there, i'm having errors trying to build the 26compat cxx driver on ubuntu 12.04 with errors like 'make_shared' is not a member of 'boost'
[17:18:15] <jimpop> with FTS (2.6.1), is it correct that "\"batmna\" joker" should only find documents containing both batman AND joker (logical AND)?
[17:41:32] <dgarstang> Argh. I'm just not quite getting sharding and replicasets. If I have a shard, split across two boxes... what do I have? I have a single shard, correct? How many replicasets would I have? one or two?
[17:45:29] <dgarstang> cheeser: so I'm looking at the last diagram at http://www.severalnines.com/blog/turning-mongodb-replica-set-sharded-cluster
[17:46:45] <dgarstang> i thought the shard just had some number of replicas. Why the primary/secondary stuff?
[17:47:27] <dgarstang> one replicaset. So, all the boxes in the same shard would have in mongodb.conf replSet = samething
[17:48:16] <pmercado> find( { _id:ObjectId("53753204abce5effbeb3ffa6") } ); <--- in console, but using java driver ?
[17:49:24] <dgarstang> cheeser: implementations I've looked at earlier only have two nodes in the same shard. That's the minimum? one primary and one secondary?
[18:03:42] <boomy> Hey guys, I would I calculate the rank of each item in a collection and add that as a field to the entry when I retrieve them?
[18:48:24] <dgarstang> Next! New mongo issue.... "[rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)" ... what's that mean?
[19:20:31] <dgarstang> I just set up a shard... and I think it's working but sh.status() is showing me nothing...
[19:31:50] <Frozenlock> Will a capped collection be faster than a normal collection where I manually remove the oldest element? (say after 2-3 complete rewrite)
[19:32:04] <dgarstang> http://pastebin.com/zMbMV8bc .... that shard isn't the right one. it's from an earlier attempt. the replicaset is called rs0
[20:26:18] <pmercado> https://github.com/mongodb/morphia/wiki/QuickStart#prepare-the-framework <--- here in code "Mongo mongo = ....." means MongoClient ?
[20:35:54] <dgarstang> I'm running sh.status() on my mongos, and eventhough I think sharding has been set up, it's not showing me anything
[20:38:00] <dgarstang> nothing but pain mongo, nothing but pain "{ "ok" : 0, "errmsg" : "Can't remove last shard" }"
[20:43:17] <cheeser> dgarstang: http://www.meetup.com/San-Francisco-MongoDB-User-Group/events/148827432/
[20:46:52] <dgarstang> because... every time I touch what I have, it breaks. Either it's fragile, or I dont have a clue
[20:49:37] <schimmy1> hmm - I figured that would be some low-hanging fruit to use the batch API with
[20:51:17] <schimmy1> My own personal use case is that I would trade off update latency and frequency for less write lock time on the replica
[20:51:48] <schimmy1> I have writers to the db, but there's no strict SLA on when those writes need to go through
[20:52:07] <schimmy1> and I have an API reading from the replica, which is ideally as low-latency as possible
[20:54:23] <schimmy1> hmm - not sure that would work for it either, as the writes are fairly constant
[20:54:58] <schimmy1> the delayed replica would still have a stream of writes locking, just an hour-delayed stream, etc
[21:13:41] <pmercado> where can I found more info about query sintaxys ? fourStarHotels = ds.find(Hotel.class).field("stars").greaterThenEq(4).asList();
[21:21:49] <proteneer_osx> yay i finally deployed my 3 member replica set with SSL support, upstarts, and all that shebang on digital ocean
[21:25:12] <dgarstang> both shard boxes _Seem_ to beconfigured but sh.status() on the mongos shows no shards. Arg.
[21:27:21] <dgarstang> when i run rs.initiate() on the shards, it says "already initialised" but the mongos shows no shards. Any ideas?
[21:29:10] <dgarstang> Derick: well... rs.initiate() on the mongos gets me "{ "ok" : 0, "errmsg" : "no such cmd: replSetInitiate", "code" : 59 }" Ugh
[21:33:22] <dgarstang> can't remember. my mind is jello after spending days trying to get this to work. someone here was going to open a doc bug for me yesterday against one of the issues
[21:35:48] <Derick> until your working set fits in memory, you probably don't need it; unless your app gets limited due to write contention
[21:37:22] <Derick> ap4y: yes, for config you need to be careful and make sure they don on different machines though...
[21:41:35] <Derick> http://www.kchodorow.com/blog/2010/08/09/sharding-and-replica-sets-illustrated/ is a good minimal setup
[22:05:39] <nukulb> joannac: thoughts on a good way to workaround this - https://jira.mongodb.org/browse/SERVER-1243. I can loop over the array items and update manually but I am wondering if its set to P4 then there might be exist a better work around