pmxbot IRC Log Viewer

[00:27:13] <SpanishMagician> hello

[00:27:20] <SpanishMagician> anyone awake?

[00:29:18] <bikcmp> nope

[00:30:04] <SpanishMagician> that's what I thought

[07:16:44] <pomke> Hello, quick question, is rs.initiate() destructive? if you already have data on one node, then add two more and set them up as a replica set, will that data be replicated to the other nodes?

[07:34:21] <[AD]Turbo> hi there

[07:53:32] <Gargoyle> Morning all

[07:55:37] <Gargoyle> Are these messages something to worry about? [rsSync] info DFM::findAll(): extent e:7061e000 was empty, skipping ahead. ns:local.replset.minvalid

[07:56:26] <Gargoyle> or serverStatus was very slow: { after basic: 0, middle of mem: 0, after mem: 0, after connections: 0, after extra info: 0, after counters: 0, after repl: 0, after asserts: 0, after dur: 1040, at end: 1040 }

[08:05:52] <cmex> good morning!!!

[08:06:16] <Gargoyle> o/

[08:06:37] <cmex> Gargoyle: stil PHP'ing?

[08:06:43] <cmex> *still

[08:06:45] <cmex> :)

[08:07:01] <jwilliams> if i have mongodb v2.0.1, writing data with java driver 2.6.5. does it support continue on write feature (if duplicate happens, the rest of record will not be affected)?

[08:07:02] <cmex> Gargoyle:kiddin :) good morning

[08:07:08] <Gargoyle> No. After your convincing argument yesterday, I decided to give it up! ;)

[08:07:14] <Gargoyle> Switched to VB!

[08:07:30] <cmex> Gargoyle: vb 6 i hope?

[08:07:31] <cmex> :))

[08:07:44] <Gargoyle> cmex: Indeed! The best version IMO

[08:07:44] <kali> irk

[08:07:46] <jwilliams> i read doc saying mongodb supports at server side, but when testing i notice data seems not be inserted if not duplicate.

[08:07:57] <cmex> Gargoley:now u talking :)

[08:10:55] <cmex> Gargoley: this agrgregation thing is realy faster then mapreduce

[10:11:46] <superball> Hello! I'm planning on storing SHA256 hashes as an ObjectID (as BinData-byte-array ofc), now my question is; is there any way for me to tell the _id-index to only index X bytes, (and then open up the object to check the rest of the byte-array if the index has dups?)..

[10:14:45] <superball> Or at all - when indexing, if not using _id, but a custom column, is it possible to say how "wide" the index should be? Can't really find an option for it. Like in MySQL where you can CREATE INDEX idx_name col_name(index_length) <- is it possile to define an index length like that in mongodb? :-)

[10:21:32] <algernon> superball: not possible to do that, unless you copy the first N bytes of the SHA256, and index that field too.

[10:50:28] <Null_Route> Hey guys! How do I see that my 3 config servers are working well?

[10:50:38] <Null_Route> is there a command which shows me their status?

[10:55:48] <gigo1980> sort on objectdata

[10:56:32] <gigo1980> how can i make an sort after map reduce if i have inline out put ?

[11:08:50] <Null_Route> gigo1980: was that directed at me?

[11:15:23] <hallas> Can I limit the values that my field can be?

[11:15:42] <hallas> Like, { type: String, values: [ 'A', 'B', 'C' ] } ?

[11:16:55] <wereHamster> hallas: enum

[11:20:54] <hallas> wereHamster: can't find any docs on it

[11:21:43] <kali> hallas: mongodb does not do schema constraints validatin of any sort. you need to implement this application-side

[11:22:08] <hallas> kali: how about min and max on number types? thats validating

[11:22:28] <kali> ha, you meant in as a query ?

[11:22:30] <kali> it

[11:22:42] <hallas> Nope

[11:23:16] <kali> hallas: well, it's the same. no validation

[11:23:23] <hallas> alright

[11:26:54] <zeppo> hi guys, i have a collection that has a single quote in its name. that i can't query. how would I go about to access it?

[11:26:58] <zeppo> can't even do a findOne on it

[11:29:27] <emocakes> how about searching with " "

[11:29:27] <emocakes> ?

[11:29:37] <emocakes> or escaping the '?

[11:33:35] <kali> try db["the_collection_with_a_'_in_it'].findOne()

[11:33:49] <kali> ot db["the_collection_with_a_'_in_it"].findOne()

[11:33:51] <kali> or !

[11:33:53] <kali> rhaaaaa !

[11:34:30] <NodeX> why would you put a single quote in a collection name?

[11:35:18] <superball> If I understand it correctly, by indexing 32B byte arrays instead of using normal MongoID's, the index generally will become larger(?) and in that case, how much larger? How can one calculate the index size for, say one million (unique & random) 32-Byte ObjectIDs?

[11:35:41] <NodeX> well Normal oid's are 24bytes iirc

[11:35:48] <superball> Yes,

[11:35:52] <NodeX> so one million multiplied by 8

[11:36:08] <superball> But is it really that easy to calculate the -index- size?

[11:36:28] <NodeX> why not?

[11:36:40] <wereHamster> hallas: http://mongoosejs.com/docs/api.html#schema_string_SchemaString-enum

[11:36:59] <superball> I'm thinking of the index structure (from which I have no idea how it looks - if you have any hints on docs how it works, I'd be glad!)

[11:37:10] <NodeX> btree's iirc

[11:37:30] <superball> :)

[11:37:34] <NodeX> is there a real reason you're changing from the default mongo _id

[11:37:46] <superball> YEs

[11:37:47] <superball> Yes*

[11:38:10] <superball> Gonna do hash-lookups (and they're going to be unique anyways)

[11:38:27] <NodeX> hash lookups ?

[11:38:38] <Zelest> is it safe to upgrade mongodb from 2.0.3 to 2.2 ?

[11:38:45] <superball> Uhh, like "Give me the data based on this hash"

[11:39:02] <superball> they don't need any mongoId, so I'll use the _id-field to store the hashes in

[11:39:05] <NodeX> and how does that differ from an ObjectId() ?

[11:39:54] <superball> Well iirc the objectId has a limit of 24B

[11:40:04] <superball> and I'm going to store a sha256-hash, which is 32B :)

[11:40:32] <NodeX> I dont understand the problem in storing the sha in another field and why it's better to store it in the _id field

[11:41:00] <superball> because then I'll have two indexes for no reason. I'm never going to refer to the MongoId

[11:41:14] <NodeX> rihgt

[11:41:15] <superball> Mainly, size is the problem

[11:41:16] <NodeX> right *

[11:41:27] <NodeX> so what's the problem then?

[11:41:35] <NodeX> 24b to 32b ...

[11:41:43] <NodeX> it's 8 difference

[11:42:28] <superball> Yeah, but I was kind-of thinking it would behave exponential when it came to the index itself, so nevermind.. Thanks for answering btw! :)

[11:42:40] <NodeX> index is an index tbh

[11:42:57] <NodeX> the added feature with an ObjectId is it's timestamp based

[11:43:07] <NodeX> and you get a free sort on it as it's already indexed

[11:43:14] <NodeX> and you can range query with it too

[11:43:47] <superball> ah =) nice. well I'm not going to do that on that certain collection anyways, but thanks for the tip :)

[11:46:41] <Gargoyle> Few more peeps here now, so I'll ask again. Should I be concerned about extent e:7061e000 was empty, skipping ahead. ns:local.replset.minvalid repeating over and over in my log files?

[11:47:17] <kali> Gargoyle: nope

[12:01:01] <Null_Route> Does anyone here know how to verify that all 3 config instances are all healthy?

[12:08:02] <Lujeni> Null_Route, MMS agent ?

[12:08:48] <Null_Route> Lujeni: Good answer! Thanks!

[12:09:07] <Neptu> yo have a problem with the mongodb python driver... How I encode floats??? or everything is an string??

[12:28:23] <NodeX> Neptu : how do you cast in python ?

[12:28:32] <NodeX> because that's your answer

[12:38:13] <Neptu> NodeX: float(sadf)

[12:38:47] <Neptu> NodeX: now works it was not a casting problem... used float long and int and now is dumping to the database so far 4M register no problemo

[12:40:24] <NodeX> I dont understand what that means but sounds greaty

[12:40:26] <NodeX> -y

[12:40:47] <Neptu> yep

[12:41:37] <Neptu> 11M registers and no problemo

[12:41:58] <NodeX> I dont know what you're talking about

[12:42:21] <Neptu> inserting in a collection

[12:42:34] <Neptu> dumping todays data only

[12:42:36] <Neptu> ;)

[12:43:18] <NodeX> ah, I was supposed to be impressed by the number of docs, cool ;)

[12:43:46] <Neptu> no, u should be impressed that so far no errors

[12:43:52] <Neptu> 12M registers for mongo is nothing

[13:13:42] <NodeX> Gotta love my new bounce rate. 3.4% lol

[13:24:36] <gigo1980> hat can i do that my mongos process also use the slaves of my shards (each shard contains an replicaset)

[13:30:56] <Rhaven> gigo1980: slaveOK command doesn't do that ?

[14:25:33] <yawniek> suppose i have a number of embedded documents that all have a integer property. is there a way to get the maximum of just one document's embedded docs ?

[14:26:01] <NodeX> aggregate framework

[14:26:28] <NodeX> well I suppose you could sort on the embedded integer and pull out the top/bottom embedded doc

[15:15:23] <kali> grmbl

[15:15:24] <kali> https://jira.mongodb.org/browse/SERVER-6909

[15:15:24] <kali> this is proving a real pain.

[15:23:14] <kali> sigh. writting test is good. running them once in a while is usefull too

[15:29:14] <Rhaven> Hi, I would like to know if routing slaveOk options redirect all queries to slave member or make something like balancing between primary and slave members ?

[15:32:24] <jtomasrl> why mongodb connection doesnt require user authentication?

[15:35:21] <Rhaven> jtomasrl: I don't know, maybe for performance issue. But you can enable it by your self in config file

[15:35:40] <fg3> ease of use

[15:38:29] <Almindor> is there a "string to query" functionality in the basic java library for mongodb?

[15:38:37] <fg3> how can I collection.find to get a unique list of a field

[15:38:55] <Almindor> something to translate {"some":"value"} into new BasicDBObject("some", "value); basically

[15:41:01] <Almindor> ow, found it ;)

[15:43:33] <jtomasrl> why when i connect to my DB i get 5 connections? [initandlisten] connection accepted from 127.0.0.1:50119 #1 #2 #3 #4 #5

[15:43:57] <fg3> found it too -> db.names.distinct('age')

[15:52:18] <jtomasrl> how can i trace the default config file?

[15:52:37] <jtomasrl> my config is not using /etc/mongodb.conf

[15:58:54] <enw2> Hola. I'm having trouble connecting to a replicaset secondary. Looks like auth is enabled in the config file but I don't think that any credentials are set. I see this when I try to connect -

[15:58:57] <enw2> error: unauthorized db:corpinfo lock type:-1 client:10.253.181.182

[15:59:24] <enw2> Look familiar? btw, I'm using the node mongo package.

[15:59:51] <enw2> (not the most recent one but I can udpate if needed)

[17:43:59] <yati> Hi. I have been using mongodb for months now and I am interested in the internals of a document oriented database. The data structures and design considerations and all. Any insight?

[18:10:35] <nickhuanca> Hello all, what types of data does MMS collect and is there a privacy policy for information collected by MMS?

[18:10:57] <Jayflux> Is it possible to have users only view their own database? When i create a new user he can see all databases

[18:30:50] <edmundsalvacion> Hey all. I was wondering if anyone had any insight or advice to a bit of a scalability problem we are facing.

[18:31:38] <edmundsalvacion> we have a collection with an array field which is indexed used for user feeds

[18:31:56] <edmundsalvacion> each record has up to 1000 user ids

[18:32:25] <edmundsalvacion> if a user has 600k+ followers, we seem to be getting performance issues

[18:33:15] <edmundsalvacion> from looking at the mms charts, it seems as if the user asserts plus btree accesses/hits are spiking

[18:33:34] <edmundsalvacion> and the lock% is spiking as well

[18:38:14] <skot1> edmundsalvacion: probably best to breakup the followers into multiple docs so when you add new ones you don't touch *all* of the old ones.

[18:39:31] <edmundsalvacion> skot1: it is currently broken up into chunks of 1000 users per doc. any new followers will create additional docs.. so old docs are never touched

[19:01:02] <edmundsalvacion> can safe writes affect lock%?

[19:19:11] <skot1> the safe part doesn't affect lock %, no

[19:19:49] <skot1> the safe part just waits for the write to finish from the clients perspective but doesn't take any time in the lock (the safe part is calling getLastError internally which is a lockless operation).

[19:33:11] <svenstaro> are there any mongodb devs here?

[19:33:16] <svenstaro> I need someone to commit something for me

[19:36:15] <skot1> best to put in a pull request.

[19:36:41] <svenstaro> there is a patch in a bug report that's being ignored

[19:36:44] <svenstaro> but it is really important

[19:36:56] <svenstaro> you can't compile mongodb on modern systems with recent boost

[19:37:00] <svenstaro> such as osx or arch

[19:37:03] <svenstaro> and fedora

[19:37:05] <svenstaro> and sid

[19:38:20] <svenstaro> this is a lot more important than the devs make it

[19:45:04] <gigo1980> hi all is there an option to make an sort after i make an map reduce to some data ?

[20:04:43] <Jayflux> Is it possible to have users only view their own database? When i create a new user he can see all databases

[20:07:44] <_m> Jayflux: Check the docs for Security And Authentication

[20:07:45] <_m> http://tinyurl.com/9x25gg9

[20:08:05] <Jayflux> Yeah i followed that, addded the auth option

[20:08:09] <Jayflux> still could access other dbs

[20:13:43] <quuxman> is there a way to test if a given spec matches a given dictionary, without involving the database with pymongo?

[20:16:02] <skot1> svenstaro: boost is now in the source tree : https://github.com/mongodb/mongo/tree/master/src/third_party

[20:16:19] <svenstaro> skot1: I know and that is bad

[20:16:41] <svenstaro> it means that if you link something that uses boost.filesystem and mongodb you'll get symbol conflicts

[20:17:00] <skot1> Yep, means you need to use the version from mongodb

[20:17:07] <skot1> what are you linking against mongodb?

[20:17:16] <svenstaro> 1.50

[20:17:47] <skot1> no, I mean what code are you linking with mongodb?

[20:17:58] <skot1> Are you using the c++ libs for your app?

[20:18:29] <quuxman> I want the functionality of Collection.find except I pass it the spec and the document

[20:19:06] <skot1> quuxman: the server does the matching. What you are asking for is the server code to be written in python.

[20:19:19] <skot1> It doesn't exist.

[20:19:23] <svenstaro> skot1: Oh, I'm not linking anything. I'm fixing a bug report of one of our users

[20:19:29] <svenstaro> skot1: I'm the mongodb packager for arch linux

[20:19:41] <quuxman> skot1: yes, exactly. It's not that complicated...

[20:19:49] <skot1> svenstaro: ah, and the bug requires boost 1.5.0?

[20:20:19] <skot1> quuxman: doesn't exist, go for it. The code you want in in Matcher.cpp

[20:20:51] <svenstaro> skot1: no, the user links using the system version of arch linux

[20:20:58] <svenstaro> skot1: but the user also uses mongodb

[20:21:03] <svenstaro> so you get symbol conflcits

[20:21:05] <svenstaro> which sucks

[20:22:23] <skot1> why are they linking with boost? Is this a scons build problem? Maybe I'm missing something.

[20:22:45] <skot1> The boost source is statically compiled in to mongodb, no?

[20:23:32] <svenstaro> because their software requires boost

[20:23:40] <svenstaro> and as such they link with boost

[20:24:02] <skot1> yeah, which works fine. What does mongodb have to do with that? They are including the mongodb source as well?

[20:24:14] <svenstaro> mongodb already includes boost symbol

[20:24:16] <svenstaro> but old ones

[20:24:22] <svenstaro> and this leads to conflicts

[20:24:29] <svenstaro> https://bugs.archlinux.org/task/31408

[20:24:59] <svenstaro> you cannot link 2 different versions of the same library into the same executable

[20:25:43] <skot1> i understand but when I asked what software you are building you didn't mention they were using the mongodb libs, so I didn't see the connection.

[20:26:42] <skot1> changing boost to 1.5.0 will require some testing and significant changes which will not be back-ported to 2.2 branches basically.

[20:27:12] <svenstaro> but I already did that work!

[20:27:13] <skot1> since boost changed their interfaces in a breaking way it is not easy to support multiple in the same process

[20:27:18] <svenstaro> yes dude I know

[20:27:23] <svenstaro> I just want someone to commit my stuff

[20:27:36] <skot1> If we change to 1.5.0 then we will have the same problems with older versions, no?

[20:27:51] <skot1> what jira issue is your patch on?

[20:28:09] <svenstaro> non yet, the one that is posted on jira is old

[20:28:11] <svenstaro> hang on

[20:28:15] <svenstaro> https://jira.mongodb.org/browse/SERVER-4314

[20:28:20] <svenstaro> that is for 2.0.6

[20:28:31] <svenstaro> and that one indeed would give you trouble with old boost version

[20:28:42] <svenstaro> however, I can add compile time version checks or the boost versions to make it compatible

[20:42:47] <svenstaro> skot1: you got commit rights?

[20:50:57] <enw> Hi. I

[20:51:14] <_m> HAII!

[20:51:53] <enw> :) I'm trying to get connected to a ReplicaSet SECONDARY. & get one of these errors when I try to find() - error: unauthorized db:corpinfo lock type:-1 client:10.253.181.182

[20:52:26] <enw> I don't have full control over this server & the guy who set it up is on holiday.

[20:53:12] <enw> It *was* running with auth = true in the config. We turned that off but there are still users in the users collection of our database -

[20:53:13] <enw> SECONDARY> db.users.find()

[20:53:13] <enw> { "_id" : ObjectId("502ab49d19d29ff7b6c289f9"), "username" : "corpuser" }

[20:53:53] <enw> Is MongoDB doing anything special with the users in this collection other than auth? Is it cool if I drop the users collection?

[20:59:43] <_m> enw: Did you issue the rs.slaveOk() command?

[21:02:33] <enw> @_m Yes. That's getting called. I'm using the node native package.

[21:03:54] <enw> I'm running db version v2.0.7, pdfile version 4.5

[21:04:02] <_m> Mongo should only care about the 'db.system.users' collection and the 'admin' database

[21:06:16] <enw> Any recommendation for working through the "unauthorized db:corpinfo lock type:-1" error? ('corpinfo' is the database name, btw)

[21:07:39] <enw> Unfortunately I'm not able to reproduce this locally. "unauthorized" sounds like an authentication issue.

[21:08:25] <_m> Yeah, if you're using an unauthorized user, that's the error.

[21:09:36] <_m> enw: Do you have valid credentials?

[21:09:39] <enw> is 'auth' managed at the server level or at the ReplicaSet level?

[21:10:45] <enw> @_m - that's a great question. I had them disable 'auth' on the SECONDARY. My client is not authenticating.

[21:12:24] <enw> @_m - I just looked and the PRIMARY config file still says "auth = true"

[21:12:45] <enw> @_m - but the SECONDARY say "noauth = true"

[21:14:08] <_m> Hrm. I would attempt to manually authenticate/query in the Mongo console to see how that works out.

[21:14:54] <enw> @_m Yeah. I'm able to manually query w/o authenticating - just db.getMongo().setSlaveOk() first and I can see all replicated documents.

[21:17:32] <enw> @_m - perhaps the corpinfo.users collection is left over from an older build of mongodb? the admin database is empty.

[21:20:00] <enw> @_m - so is it okay if PRIMARY has different auth settings than SECONDARY servers?

[21:21:43] <_m> enw: The corpinfo.users collection would be a project-level collection

[21:21:52] <_m> You would want the db.system.users collection for users

[21:22:18] <_m> As for the user replication across the sets, I honestly don't know the answer.

[21:24:25] <enw> @_m k. thanks.

[21:58:23] <timeturner> what is the benefit os using $exists and returning the document vs using findOne() and checking to see if the document is empty or not

[21:58:32] <timeturner> aren't those the same thing?

[22:02:04] <Gargoyle> timeturner: You can fetch docs where something doesn't exist

[22:19:57] <Gargoyle> Derick: You still up? (I noticed a github notification a little while ago)

[22:33:03] <timeturner> Gargoyle: I still don't understand the difference?... :/

[22:34:08] <Gargoyle> timeturner: collection->find({'images':{$exists: false}}) would return all the docs where the key 'images' wasnt set.

[22:47:23] <svenstaro> skot1: you with me?

Log file Viewer

Help | Karma | Search:

#mongodb logs for Tuesday the 4th of September, 2012