pmxbot IRC Log Viewer

[01:17:16] <recidive> hello, I've introduced a 'counter' field in one of my collections, I want to update it with the number of occurrences in another collection, what's the easiest way to do that from the command line?

[05:16:30] <hdm> mother@#$ Wed Jul 18 00:08:58 uncaught exception: count failed: {

[05:16:30] <hdm> "errmsg" : "10334 Invalid BSONObj size: 1819112552 (0x68746D6C) first element: \\r\\n\\r\\n: ?type=62",

[05:16:49] <hdm> complete ETL reload and hit another bson corruption issue

[06:45:54] <ron> mother? :)

[07:35:18] <[AD]Turbo> hola

[13:11:01] <patito> hi all

[13:11:22] <patito> I'm using replicaSet and I've a litle problem

[13:11:24] <patito> =)

[13:11:38] <patito> I read some links, but nothing helps me

[13:12:07] <patito> I'm trying to reconfigure my replicaSet. I want change the host

[13:12:16] <patito> var c = rs.conf();

[13:12:31] <patito> c.members[0].host = "XX.XX.XX.XX";

[13:12:40] <patito> rs.reconfig(c);

[13:12:56] <patito> is wrong?

[13:13:05] <patito> my message error: "errmsg" : "replSet error loading set config (BADCONFIG)",

[13:33:42] <Bofu> Is there a simple way to get the index size of a single index within a collection? Not just the total?

[13:37:15] <Bofu> Nevermind, found it on stats

[13:39:47] <Bofu> (as in db.<x>.stats() if anyone else is looking)

[13:44:31] <jamescarr> hey, what is a good way to find duplicate entries in a collection where one field is the same?

[13:49:14] <Lujeni> jamescarr, distinct?

[13:49:22] <jamescarr> yes

[13:50:28] <remonvv> jamescarr, use the AF

[13:52:44] <jamescarr> AF?

[13:59:56] <houms> can configsvr be specified in the config file? if so is it just configsvr = true?

[14:00:09] <remonvv> aggregation framework

[14:01:10] <Bofu> For compound indexes if you index 3 fields and only search using 2 of them for a query, will it still use the index?

[14:01:27] <remonvv> restore your data on a 2.1 server, run an aggregation to find your dupes, dump the _id values of the conflicting documents and remove them from your production database. Alternatively build a JS script that runs a distinct on the identifying field and iterate over all distinct keys by doing a count for that field value, if the count > 1 do something

[14:03:23] <jamescarr> any idea why this isn't giving me a collection of the duplicate items I'm looking for? https://gist.github.com/3136381

[14:09:15] <jamescarr> dammit

[14:09:23] <jamescarr> is there someway to debug my map reduce?

[14:12:48] <jamescarr> my reduce never gets called

[14:16:26] <houms> is it possible to run configsvr on the same servers running mongod? we have 3 servers and just want to test replication.

[14:19:39] <remonvv> yes it is

[14:19:54] <remonvv> config servers require very little resources

[14:23:05] <jamescarr> how do you run a map reduce using a file?

[14:25:42] <jamescarr> please?

[14:26:36] <remonvv> hehe, google broken, it's the top hit

[14:26:50] <remonvv> http://www.mongodb.org/display/DOCS/MapReduce -> Ctrl+F "file"

[14:29:56] <patito> can I remove files from /data/db/local.*?

[14:32:38] <houms> remonvv i cannot seem to get it going. I first had all 3 servers running using replSet = true in config. but the docs say run 3 config servers. I figured the 3 servers could run both config and mongod but with replication. but the docs say do not run replSet with configsvr at the same time

[14:33:37] <jamescarr> actually I got it...how can I convert a date to an ISO date?

[14:33:54] <jamescarr> I have a date like this: Tue, 17 Jul 2012 23:32:53 GMT

[14:34:01] <jamescarr> need to search for an ISO date like that

[14:40:47] <remonvv> hoodow, you have to run 1 or 3 config servers. Where they are running is not relevant.

[14:41:41] <remonvv> hoodow, you said you were testing right? Then just use 1 config server and run it on any of the instances available to you. The reason you should want 3 for production environments and them not running on data nodes is for availability reasons.

[14:42:53] <houms> so i do not need replSet correct? at least the docs seem to imply that running three config servers that not to run any conig servers with replSet because replication is automatic between them but that is done but mongos correct? http://www.mongodb.org/display/DOCS/Configuring+Sharding#ConfiguringSharding-ConfigServers

[14:43:43] <houms> so in theory the minimum recommended dbs in prod. is 6 if using replication?

[14:43:58] <houms> i mean 3 config servers 3 mongod

[14:44:04] <remonvv> jamescarr, this isn't a glorified google sir. Your questions are all first hit google questions. http://www.mongodb.org/display/DOCS/Dates

[14:44:20] <remonvv> hoodow, you need replSet if you're testing replication sets

[14:44:51] <houms> so i set replSet in mongo conf file correct?

[14:45:01] <remonvv> houms, as documented, yes.

[14:45:23] <remonvv> minimum production environment depends on your availability requirements.

[14:45:41] <remonvv> For example it's relatively safe to have a 3 member repset with 3 config servers on each instance.

[14:45:42] <houms> once i do that how do i start the configsvr option ? that cannot be put in the conf file so i assume it also cannot be run on even one of the mongod servers that are usin replSet

[14:45:56] <remonvv> you start a mongod as config server

[14:46:09] <remonvv> again, this is all completely documented from start to finish

[14:46:11] <houms> remonvv the example you describe is what i am trying to achieve

[14:46:31] <Bofu> Any ideas on getting a "BufBuilder grow() > 64MB" while trying to create an index? I know there's the 64MB per-item limit, but ... never thought I'd get that when trying to build an index.

[14:46:44] <houms> the part i do not understand is the docs say you cannot specify configsvr and replset on the same box

[14:46:48] <remonvv> houms, so follow the instructions but adjust where the instances are running

[14:46:54] <remonvv> well you need two mongod processes

[14:47:04] <remonvv> where they run doesn't change anything

[14:48:31] <remonvv> repset members start with mongod --replSet, config servers with --configsvr and then run your mongos with the appropriate config server parameters and you're done

[14:48:38] <remonvv> where these processes are running is not relevant

[14:49:00] <houms> so when the docs say replication between configsvr is automatic so do not use replset, they mean just use configsvr in the config file and then have mongos do the synch between them?

[14:49:11] <remonvv> it's completely seperate

[14:49:39] <houms> so mongos just synchs the config server not the dbs

[14:49:44] <remonvv> IF you run 3 config servers they use replication amongst eachother (through a 3-phase commit approach iirc) to keep the cluster metadata consistent across the config servers

[14:49:49] <remonvv> it has nothing to do with your rep set

[14:50:37] <remonvv> houms, no, config servers sync config servers, repset members sync amongst eachother and mongos pulls repset metadata from the config servers to be able to reach the repset members

[14:52:48] <houms> so if running them on the same box you should be running two mongo processes

[14:56:05] <Bofu> yes.

[14:58:02] <houms> is there an elegant way to run a second daemon or is it just a matter of copying the init scripts, etc.

[14:58:37] <Bofu> that's up to you

[14:58:41] <Bofu> depends on how you want to run them

[14:59:42] <daveriess> in the docs on choosing a shard key, a note says that when prefixing a shardkey with yyyy-mm, performance hits at the beginning of the month can be avoided by creating ranges ahead of time.. can someone expound this?

[15:00:22] <houms> well we are in testing but our testing generally consists of a more accurate prod. setup. granted in prod. we may not run them on the same box we may not totally count that out. as it seems the mongo docs dont necessarily discourage this since the config server is not very resource intensive

[15:01:16] <houms> remonvv and Bofu thank you for your insights and putting up with my questions.

[15:01:31] <Bofu> Was all him, I was just here for the end heh

[15:03:01] <cdave> in my test db box, i filled up my disk and now i get ""assertion" : "Can't take a write lock while out of disk space"," if i try to drop db or collections, is there any way out of this catch22 ?

[15:05:31] <NodeX> stop the db, make some space

[15:05:36] <NodeX> start the DB and drop

[15:05:44] <rguillebert> hi

[15:05:56] <rguillebert> how can I know if my data is journaled ?

[15:06:50] <NodeX> check if journaling is enabled in the config

[15:06:54] <NodeX> it is by default iirc

[15:07:10] <Bofu> Any ideas on getting a "BufBuilder grow() > 64MB" while trying to create an index? I know there's the 64MB per-item limit, but ... never thought I'd get that when trying to build an index.

[15:07:36] <Bofu> NodeX, I think only on 64bit isn't it?

[15:07:38] <NodeX> jeez, how big are your indexes

[15:07:49] <Bofu> Not that big I didn't think =/

[15:07:49] <houms> can distributed reads be secified in config or is it strictly a shell operation?

[15:08:00] <Bofu> It's a timestamp field so there's definitely a lot of variations

[15:08:21] <NodeX> but 64mb per document is huge

[15:08:32] <Bofu> that's why I was confused, no docs in here should be over 64mb

[15:08:37] <Bofu> the index would definitely be, though.

[15:08:48] <NodeX> is it on the shell you get the error ?

[15:08:51] <Bofu> yes

[15:08:55] <rguillebert> NodeX, there's no way of knowing that through the mongo shell ?

[15:09:10] <NodeX> rguillebert : no

[15:09:18] <NodeX> (not that I'm aware of)

[15:09:27] <NodeX> why can't you open the config and check?

[15:09:37] <rguillebert> I can

[15:09:41] <cdave> NodeX, how do i make some space ?

[15:09:43] <rguillebert> it's just that there's nothing

[15:09:50] <cdave> can delete those data files

[15:09:51] <rguillebert> (so it means enabled)

[15:09:57] <NodeX> http://www.mongodb.org/display/DOCS/Journaling#Journaling-Disabling%2FEnabling

[15:10:01] <rguillebert> but the journal directory is empty

[15:10:20] <Bofu> cdave, you can't delete anything else on the drive anywhere?

[15:10:22] <NodeX> cdave : dont you have anything on the drive you can delete?

[15:10:32] <cdave> nop

[15:10:38] <Bofu> not even log files?

[15:10:41] <NodeX> can you move a data dir to another partition?

[15:10:49] <houms> if you cannot delete the logs then compress them

[15:10:53] <NodeX> (not the whole dir just one)

[15:11:27] <Bofu> I would check shit like /tmp, /dev/shm, /var/log to see if there's literally anything you can afford to wipe

[15:11:53] <cdave> all my data is in /var/lib/mongodb

[15:12:00] <cdave> which is 100% full

[15:12:11] <NodeX> is /var/lib/mongo it's own partition ?

[15:12:18] <cdave> i see files like dbname.[0-1]

[15:12:39] <cdave> yes, it has it's own partition

[15:12:49] <NodeX> stop your mongo

[15:13:36] <NodeX> move a set of files - foo.0 ... foo.ns to a separate partition

[15:14:03] <NodeX> start your mongo, drop your collection, stop your mongo, move the files back, start your mongo

[15:15:31] <Bofu> NodeX, is it because the index itself is over 64MB or ... what aspect of it would be triggering that?

[15:16:20] <NodeX> no, the index can be massive

[15:16:43] <houms> can distributed reads be secified in config or is it strictly a shell operation?

[15:16:48] <houms> rs.slaveOk()

[15:17:35] <Bofu> Trying to figure out what it could be for it, I can't imagine it's that bad considering it's literally just epoch timestamps.

[15:18:59] <NodeX> apparently index's do have a max size

[15:19:04] <Bofu> uh oh

[15:19:33] <NodeX> the keys at least

[15:19:44] <NodeX> so it wont be that

[15:20:12] <NodeX> can you pastebin/paste your ensureIndex command?

[15:20:16] <Bofu> yeah sure

[15:20:51] <Bofu> db.entries.ensureIndex({created_at:-1})

[15:20:53] <Bofu> heh

[15:21:06] <NodeX> can you pastebin an example document?

[15:21:10] <Bofu> at first it was a 4 field index, then found out I couldn't do 2 arrays in it - so it was a 3 field, got that. Took it down to just this, and it happened

[15:21:11] <Bofu> yeah sure, 1 sec.

[15:22:09] <Bofu> PM'd

[15:23:02] <NodeX> do you have any indexes on it already ?

[15:23:20] <Bofu> just on the entry_hash

[15:23:25] <Bofu> (and the default _id)

[15:23:30] <cdave> NodeX, that worked, thanks.

[15:23:45] <NodeX> ;)

[15:24:07] <NodeX> drop the entry_hash one and try with the created one first

[15:24:58] <Bofu> k, 1 sec.

[15:25:02] <Bofu> may take a few minutes

[15:25:15] <NodeX> are you 32bit or 64bit?

[15:25:20] <Bofu> 64

[15:25:23] <NodeX> and what mongo version

[15:25:40] <Bofu> db version v2.0.6, pdfile version 4.5

[15:26:54] <NodeX> http://grokbase.com/t/gg/mongodb-user/11bpzfh4f4/strange-assertion-13548-bufbuilder-grow-64mb-2-0-1

[15:27:03] <NodeX> that suggests corruption ..

[15:27:59] <NodeX> the guy who reported it said a repair fixed the problem

[15:28:10] <Bofu> want me to try that before dropping the other index?

[15:28:34] <NodeX> http://www.mongodb.org/display/DOCS/Durability+and+Repair

[15:28:37] <NodeX> might be an idea

[15:29:00] <NodeX> is it possible that one created_at snuck through with a massive field ?

[15:29:08] <NodeX> massive value **

[15:29:41] <Bofu> It's possible...

[15:29:45] <Bofu> Let me check.

[15:44:31] <houms> is there a place to view all mongo conf options?

[15:54:15] <Bofu> Repairing... yay cursor locks....

[16:46:59] <jmccree> Quick question, does anyone know if the php mongo extension supports ssl connections?

[17:24:52] <jbenet> Hello! Question: when using pymongo + gevent with db.authenticate, does (1) each greenlet need to call db.authenticate() separately, or (2) the connection is authenticated once, and this auth works for all greenlets?

[17:27:42] <jbenet> some evidence in support of (1) http://stackoverflow.com/a/9193682/346736 + http://api.mongodb.org/python/current/examples/gevent.html#using-gevent-without-threads.

[17:28:32] <jbenet> some evidence in support of (2) http://api.mongodb.org/python/current/api/pymongo/database.html#pymongo.database.Database.authenticate ("all threads will share the authentication")

[17:29:49] <jbenet> these are inconsistent. I'm inclined to trust (2) more, except for the fact that i'm getting odd OperationFailures. :/

[17:48:47] <jbenet> thoughts anyone ?

[18:04:09] <jbenet> Argh!! nevermind. different venv was active, with pymongo 1, not 2. Sorry for the confusion. Will add pointers to above SO answer.

[18:42:53] <Dreamer3> anyone have issues with half saved files in GridFS?

[18:43:04] <Dreamer3> i have orphan chunks with no file

[18:43:32] <Dreamer3> yes as far as i know no error was raised

[19:23:48] <nick_h> for db.users.find({name: "nick", age: "31"}) , is there a difference between these 2 indexes?

[19:23:52] <nick_h> db.users.ensureIndex({"name": 1, "age": 1});

[19:23:56] <nick_h> db.users.ensureIndex({"age": 1, "name": 1});

[20:43:19] <edvorkin> Hi, I have a problem accessing mongodb instance on amazon - works locally on port 27017 but can't access remotely. Security group allow access to all on port 27017. I also use elastic IP on the instance. Any suggestion?

[20:45:18] <sirpengi> edvorkin: firewall?

[20:45:55] <sirpengi> I suspect either firewall or mongodb isn't listening on the right IP

[20:46:19] <edvorkin> no. I can't get to it from another AWS instance as well

[20:46:52] <edvorkin> how I can make it listen to right ip? build_ip?

[21:03:37] <danryan> hey folks, how do I determine the progress of a new replset member that's recovering?

[21:31:50] <jiffe98> how do I use the perl module with databases with _'s in their name?

[22:06:02] <linsys> danryan: rs.status

[22:07:30] <danryan> linsys: thanks, that says the member is in a recovering state. I'm looking for a way to determine how far along this recovering state is

[22:13:37] <Deathspike> Is async operations in C# something that is planned?

[22:20:27] <asdfqwer> oye, how does one do intergration, staging, development enviroments with clients?

[22:20:40] <asdfqwer> given massive amounts of production data

[22:21:49] <asdfqwer> we write our tests and develop locally and tunnel into prod for everything, except it becomes a problem if clients what their collections staged independently of prod

[22:22:18] <asdfqwer> wondering if there is a silver bullet for such a problem

[22:53:59] <toddnine> Hey guys. Are there any examples out there using the java client and building queries for $and or $or

[22:54:12] <toddnine> it's not clear to me how the client should be used to build the array of params

[22:59:57] <toddnine> or if I can do $and [$or (exp), $and(exp)]

Log file Viewer

Help | Karma | Search:

#mongodb logs for Wednesday the 18th of July, 2012