pmxbot IRC Log Viewer

[01:49:22] <dbousamra> Hi all. Can anyone help me with an aggregation query. Here is my query and result: https://gist.github.com/dbousamra/14ca3ff7733fe0952a67

[01:56:23] <dbousamra> I am trying to get rid of those key names. I just want an array of snapshots, without keys

[02:01:37] <Boomtime> can you give an example of the result you want?

[02:01:45] <Boomtime> oops, nm, scroll

[02:02:24] <Boomtime> do the "keys" not have a consistent name? i.e are the key names a datestamp or something?

[02:05:09] <dbousamra> They are the timestamps. It's a shitty key system I know, but thats the way it is (I can;t change it)

[02:05:38] <dbousamra> The keys are thus, unique

[02:18:22] <dbousamra> So yeah. Not sure how to remove keys

[11:13:48] <cnu> Hi, Is there some documentation on what is the range for the score calculated for each document, when one does a full text search?

[11:17:38] <rspijker> cnu: isn;t score just the weighted sum of frequency?

[11:19:25] <cnu> Is there a limit of minimum score and maximum score? Need to use that score with a number from another program and get a normalized total score.

[11:19:41] <cnu> If I know the range, will be helpful to normalize it.

[11:23:07] <rspijker> cnu: hmmm, good question

[11:23:10] <rspijker> I don’t know

[11:23:15] <rspijker> never seen it in the docs anywhere

[11:23:36] <cnu> Yes. I too searched for the entire day. couldnt find any place which mentions this.

[11:23:46] <cnu> next was beginning to look at the code.

[11:36:39] <xdotcommer> I want the update for this particular field to only go for each hour through first time

[11:36:39] <xdotcommer> $update[$set]['hours']['0']['open'][24];

[11:36:39] <xdotcommer> $update[$set]['hours']['1']['open'][24];

[11:36:41] <xdotcommer> kind of like $first with aggregation framework but that will work on update $SetOnInsert wont work because its for an entire document.

[11:37:04] <xdotcommer> ie.. if the field is already set I dont want to update it

[11:42:31] <boo1ean> Hi folks! I'm experiencing issues with 2dsphere index. I've a large collection (about 50 GB) and each doc has location specified as multipolygon. And I'm using search by point at on this collection and it works blazing fast BUT for same rare cases it works very-very slow. I've noticed that some of slow cases are docs with big-area-polygons. Also there cases when mongo lock all reads for more then 90 seconds. What is the reas

[11:42:31] <boo1ean> on of slow search for some cases? is it just worst case for search algorithm? How can I workaround this issue?

[11:48:55] <rspijker> xdotcommer: make the query part $exists:false ?

[11:49:44] <xdotcommer> rspijker: this is an insert/upsert

[11:50:53] <xdotcommer> and it includes many other fields as well

[11:51:46] <rspijker> xdotcommer: and you only want the one specific field to behave this way?

[11:51:59] <xdotcommer> rspijker: exactly the others get $set or $inc

[11:52:58] <rspijker> so why won’t setOnInsert work then?

[11:53:34] <xdotcommer> rspijker: because setOnInsert works on entirer document

[11:54:00] <rspijker> eh.. what?

[11:54:36] <xdotcommer> well if document exists then it wont work

[11:54:47] <rspijker> isn’t that exactly the point?

[11:54:48] <xdotcommer> the entire document not just a subdocument

[11:54:48] <rspijker> o

[11:54:50] <rspijker> I see

[11:55:12] <rspijker> yeah, setOnInsert will *never* work if it’s an update, only on insert

[11:55:24] <xdotcommer> And I prefer not to do aggregation every time

[11:55:37] <xdotcommer> would be nice if there was a SET if does not exists... feature

[11:55:47] <xdotcommer> setIfEmpty

[11:56:26] <xdotcommer> or a reasonable work around

[11:56:54] <rspijker> yeah, I can’t see a single-query answer

[12:00:08] <xdotcommer> dual query i guess or aggregation .. not clean :(

[12:00:36] <rspijker> xdotcommer: vote for this, I suppose… https://jira.mongodb.org/browse/SERVER-6566

[12:00:39] <xdotcommer> since this is for financial data.. I also prefered atomic

[12:01:03] <xdotcommer> rspijker: nice thanks... somebody worded it better :)

[12:01:30] <xdotcommer> lol @ first comment "This is a feature that is sorely lacking in Mongo. We have to do > 25 queries in some cases to work around the problem."

[12:03:25] <rspijker> yeah, if you happen to have a use-case where it’s needed then I can imagine it’s quite a big gap...

[12:05:44] <xdotcommer> rspijker: this is for financial data "open" value... so when you are inserting trades... and splitting them up by second/minute/hour... you have the typical open, high, low, close ... the last 3 work great with $max, $min, $set respectivly but first one is missing

[12:06:15] <xdotcommer> ie.. open... would keep getting overwritten

[12:07:01] <rspijker> well… can’t you guarantee that when you create the document you fill in the open value?

[12:07:12] <rspijker> or is this embedded somehow with many opens?

[12:11:58] <xdotcommer> rspijker: yea the document contains data from hour to second resolution

[12:12:25] <gnaddel> Hi there, question about the bindIp option in /etc/mongodb.conf: Can I use wildcards in that? I would like two things to be accepted: 1: localhost and 2: my Universities IP-Range. So would "127.0.0.1,my.uni.range.*" work?

[12:13:11] <xdotcommer> so it will have "open" for 60 times in each minute multiplied by 60 so 3600 opens :)

[12:16:07] <rspijker> gnaddel: bindIP is what you bind on… not where you allow connections FROM...

[12:16:50] <rspijker> unless you dynamically get an ip address at uni every time and you always want to bind to that regardless of what it is

[12:16:58] <rspijker> in which case, you’re probably out of luck…

[12:17:15] <rspijker> xdotcommer: I see

[12:18:48] <gnaddel> rspiker: I was under the impression that it would only accept connections from hosts it is bound to: "Specifies the IP address that mongos or mongod binds to in order to listen for connections from applications. "

[12:19:06] <gnaddel> ah I see my error

[12:21:25] <rspijker> gnaddel: yeah, I can see how it might be misleading. You define there which interfaces to listen on, roughly speaking. If you want to limit access from external sources, you need to set that up in your firewall

[12:46:36] <ernetas_> Hey guys.

[12:48:33] <ernetas_> So regarding yesterday's issue of high CPU usage after MongoDB upgrade, we filled a bug report: https://jira.mongodb.org/browse/PHP-1164 Is anyone else able to reproduce it?

[13:09:59] <topwobble> mongo docs down :(

[13:10:12] <topwobble> actually all of http://mongodb.org/ is down

[13:11:12] <cheeser> wfm

[13:11:24] <cheeser> there was a DNS change yesterday. try flushing your dns cache

[13:12:29] <topwobble> still nothing. can you reach it cheeser ?

[13:12:55] <tscanausa> I can reach it

[13:13:05] <topwobble> I used “sudo killall -HUP mDNSResponder” to flush it

[13:16:17] <topwobble> can you tell me the parameters for db.dropIndex() ?

[13:17:06] <cheeser> run this: db.dropIndex

[13:17:09] <cheeser> (with the parens)

[13:17:25] <cheeser> iirc, it just takes the index name.

[13:17:32] <cheeser> db.collection.dropIndex("someIndex")

[13:17:43] <topwobble> w/o parameters, I get : assert failed : need to specify index to dropIndex

[13:17:50] <topwobble> ah ok

[13:18:06] <topwobble> thx cheeser

[13:18:10] <cheeser> db.dropIndex() isn't a thing afaik

[13:18:25] <topwobble> ok. thought you could drop all indicies

[13:20:33] <cheeser> db.collection.dropIndexes()

[13:20:36] <cheeser> http://docs.mongodb.org/manual/tutorial/remove-indexes/

[13:20:48] <topwobble> :D ^^ still doesn’t load for me

[13:22:05] <cheeser> did you flush your dns cache?

[13:22:23] <tscanausa> topwobble: what answer are you getting from dns?

[13:22:58] <topwobble> Yes, I flused it via CLI: dscacheutil -flushcache;sudo killall -HUP mDNSResponder

[13:23:13] <topwobble> tscanausa: what do you mean?

[13:23:27] <tscanausa> what is your hosts os?

[13:23:34] <topwobble> OSX Mavericks

[13:23:45] <topwobble> PING mongodb.org (184.168.221.53) == timeout

[13:24:48] <tscanausa> topwobble: that is the old dns response.

[13:24:53] <cheeser> PING mongodb.org (75.101.156.249): 56 data bytes

[13:25:10] <topwobble> ok. so not flushed. thx

[13:25:36] <topwobble> idk why it isnt flushing then

[13:26:21] <cheeser> reinstall your OS,i guess ;)

[13:26:42] <topwobble> sudo rm -rf /

[13:30:05] <tscanausa> topwobble: which part of the world are you in?

[13:30:37] <topwobble> USA; washington dc

[13:31:58] <tscanausa> for know ( and this is frowned apon ) you could edit your /etc/hosts to point docs.mongodb.org to 75.101.156.249

[13:32:38] <topwobble> hah. yep

[13:32:52] <topwobble> nice call. its ok. i got the index dropped ;) ill just try to get DNS working again

[13:52:59] <_boot> if I shard a collection, will the record allocation strategy be communicated to the individual shards?

[13:53:32] <kali> _boot: to the mongos

[13:54:05] <_boot> right, but will the shards the mongos talks to end up with a collection using the correct strategy or will they just use the default?

[13:54:37] <kali> _boot: you're referring to the usePowerOf2 flag ?

[13:54:54] <_boot> yep

[13:55:26] <kali> that's actually a very good question :)

[13:55:35] <_boot> i guess Ill find out late tonight/tomorrow

[13:56:06] <Derick> usePowerOf2 is default in 2.6 though

[13:56:17] <Derick> but I would expect it to be communicated, just as indexes would

[13:56:32] <_boot> sounds good then :D

[13:57:10] <_boot> i've turned it off on the collection I'm about to shard

[14:17:08] <abusado> foreach ($display as $document){ print_r($document); <-- how to make it human readable? the result is gibberish

[14:17:32] <abusado> I think i got the wrong channel I think it's in php channel

[14:17:43] <Zelest> o_O

[14:20:20] <Derick> abusado: what's the output?

[14:20:30] <abusado> Array ( [_id] => MongoId Object ( [$id] => 53ea1fb87ba5d18418000029 ) [book_id] => 1234 [title] => Kas [description] => Adsa [category] => Asad [author] => Sdasdsa )

[14:20:43] <cheeser> so, standard output

[14:20:47] <Derick> why is that gibberish?

[14:20:58] <abusado> I want it to show just the title description and category

[14:21:12] <Derick> then you need to select only those fields

[14:21:30] <Derick> you can do that with the fields argument (2nd argument) to find

[14:21:43] <abusado> print_r($document->title);

[14:21:45] <abusado> ?

[14:21:47] <rspijker> lol, “I put gibberish in my DB and now when I ask what’s in it, it gives me gibberish. THIS IS BROKEN!”

[14:22:06] <abusado> lol pardon im just new in programming

[14:22:08] <Derick> abusado: no, it's an array, so print_r($document['title'])

[14:22:17] <rspijker> no worries abusado :)

[14:22:49] <abusado> there you go... thank you very very much

[14:36:27] <Sjimi> Is it possible when using aggregate to project a String to an Int64 by any chance? Or is there another solution that doesn't involve fiddeling with the data?

[14:38:47] <tscanausa> Sjimi: why do you need to?

[14:39:48] <Sjimi> The $sum group accumulator does not work with Strings; unfortunately it is practically impossible to convert the data itself. Thus projecting to Int would be great.

[14:40:31] <tscanausa> I do not think aggregate will allow but map-reduce probably will

[14:42:25] <Sjimi> Okay I'll have a deeper look at it tscanausa, thanks.

[15:28:09] <rickitan2> Hey guys can any one give me a hand with this:

[15:28:09] <rickitan2> http://stackoverflow.com/questions/25267738/mongoose-mongo-wrongly-creating-id-property-inside-array-objects

[17:37:22] <piotrkowalczuk> hi, is there anybody?

[17:59:01] <pgentoo-> i am doing around 3000 queries (by _id only) on a large mongo collection (~500M documents), and am seeing upwards of 85% lock percentage on that database. I thought reads in this fashion wouldn't lock things up. Is this normal?

[18:00:39] <wc-> hi all, i am hitting a lot of errors i havent seen before as i attempt to do a query with 2 different 2d geo query operators

[18:01:02] <wc-> i have two polygons, (one happens to be a box) and i want to return objects taht are only in their intersection

[18:01:16] <wc-> so i tried an $and with two different geoWithin queries inside it

[18:01:25] <wc-> and im getting unner error: Overflow hashed AND stage buffered data usage of 33563702 bytes exceeds internal limit of 33554432 bytes

[18:01:58] <wc-> im running mongo 2.6.4

[18:07:49] <wc-> here is a pastie of the query and the error

[18:07:50] <wc-> http://pastie.org/9467630

[18:08:19] <wc-> does anyone know how i could go about returning only documents that satisfy both these polygon requirements?

[18:12:04] <pgentoo-> Ah ha, i found an update that i didn't think was happening (relying on mongo query to limit whether an update happened, instead of doing in my client code to prevent the call from happening). :)

[18:31:05] <abusado> $display=$list->find($_POST["search"]); <--- is that correct?

[18:39:44] <abusado> anybody?

[18:39:48] <abusado> $display=$list->find($_POST["search"]); <--- is that correct?

[18:41:08] <BurtyB> correct in what way?

[18:42:25] <abusado> Got it

[18:42:48] <abusado> $authors = array('author'=> $_POST["search"]);

[18:42:48] <abusado> $display=$list->find($authors);

[18:42:49] <abusado> thanks

[20:29:52] <culthero> Anyone got any insight as to whether or not a specific improvement will make it in to 2.7, or how they are selected?

[20:29:54] <culthero> https://jira.mongodb.org/browse/SERVER-13801

[20:37:43] <cheeser> culthero:

[20:37:45] <cheeser> Fix Version/s: 2.7 Desired

[20:39:37] <culthero> I read that, desired seemed ambiguous and wondered for other desired major features how many of those really make it in on development relasese on time

[20:40:39] <culthero> It just happens to specifically impact some decisions on.. a potential product

[20:42:41] <cheeser> generally speaking, until a feature is delievered it's risky to base any decision on it.

[20:45:27] <culthero> Of course. My general approach for what I am using isn't going to change, however avoiding having to think about deploying something we might start offering as a service as a mongo sharded cluster would be.. nice.. The range limitation conceptually in my head means that fulltext indexes on timestamped data could be fast if done in small enough chunks on a very limited computer

[20:47:26] <culthero> You could let the performance characteristics of the server decide how your application should be configured, and having 1 server is an easier sell then having x servers with two different kinds of (mongo + elasticsearch / solr / whatever)

[21:20:03] <q85> I have a test cluster with a primary, a secondary, an arbiter, one config server, and one mongos. All components are 2.4.9. The secondary is hidden with priority 0. I have a script connecting to mongos with a read preference of secondaryPreferred. I would have assumed that the since the secondary is hidden, mongos would route the queries to the primary. Instead, it is routing queries to the hidden secondary. Is this behavior expected? If

[21:38:11] <cbuckley> hi there, I'm running into some issues with importing a dump on mongo 2.4.9

[21:38:13] <cbuckley> Btree::insert: key too large to index

[21:38:26] <cbuckley> Everything I've looked online has solutions for 2.6, but I'm struggling to find anything for 2.4

[21:38:30] <joannac> q85: huh, that's weird

[21:38:38] <joannac> cbuckley: um, make your keys smaller?

[21:39:41] <cbuckley> thats not currently an option

[21:40:31] <joannac> cbuckley: well, I'm not sure what you expect then

[21:41:51] <cbuckley> i'm not expecting anything except for some assistance

[21:42:01] <joannac> assistance with what?

[21:42:11] <joannac> do you want some explanation about the implications?

[21:42:23] <cbuckley> in 2.6 i see there is a paramater failIndexKeyTooLong

[21:42:27] <q85> joannac: Would you say that behavior is unexpected?

[21:42:32] <cbuckley> which you can set to false

[21:42:35] <abusado> how to create inner array? or multinest array? i cant create another field within a field

[21:42:53] <cbuckley> i'm wondering what exactly that means, as it says it is dictating pre 2.6 behavior

[21:42:56] <joannac> cbuckley: okay. in 2.6 doing that will cause the server to crash

[21:43:08] <joannac> the parameter is to revert to not crashing

[21:43:09] <cbuckley> and yes, some insight into implications would be good

[21:43:13] <cbuckley> ahh i see

[21:43:13] <cbuckley> ok

[21:43:16] <joannac> either way, the key is not indexed

[21:43:21] <cbuckley> got it

[21:43:29] <joannac> so, if you run a query and use that index, the document won't be found

[21:43:35] <joannac> because it's not indexed

[21:43:57] <joannac> so you now have an index that doesn't actually contain all documents

[21:44:10] <cbuckley> yep ok

[21:44:58] <joannac> q85: yes, although I can't see how it would happen. How did you tell the queries were going to the hidden secondary? Could you pastebin your rs.conf()?

[21:45:50] <q85> I have full profiling turned on. Yes, one moment.

[21:46:08] <joannac> abusado: not sure what you want. db.foo.save({a:[1, 2, ['a']]}) ?

[21:47:43] <joannac> cbuckley: actually, i should correct a detail. in 2.6 the insert or update will fail if the key is too long. the server will *not* crash. not sure where I got that from

[21:48:13] <joannac> cbuckley: the behaviour in 2.4 is that the insert/update succeeds but the index(es) are just incomplete

[21:48:29] <cbuckley> ok cool thank you

[21:49:12] <joannac> no probs

[21:49:45] <q85> joannac: http://pastebin.com/vxhHF8B1

[21:56:11] <joannac> q85: pastebin your connection string?

[21:56:39] <joannac> q85: also the output of db.isMaster()

[21:56:43] <joannac> I have a theory :)

[21:56:52] <abusado> How to create new field within a field (for comments)

[21:57:34] <abusado> I use update and $set $push it doesnt work for me

[22:00:18] <abusado> Ok, I give up

[22:01:47] <q85> is master: http://pastebin.com/xgrBZWQv

[22:02:16] <q85> profiler: http://pastebin.com/BrY2aLYK

[22:02:39] <q85> script with connection string: http://pastebin.com/ppLikkCu

[22:09:04] <q85> joannac: and here is part of the log for the secondary showing config version 5 being applied half an hour before I ran the script: http://pastebin.com/28agwJxK

[22:12:08] <joannac> Tue Aug 12 15:54:03.295 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 5 5

[22:12:24] <joannac> the secondary definitely has the same config?

[22:14:41] <q85> both are on version 5 and both list the secondary has hidden with priority 0.

[22:14:50] <joannac> i've been testing for about 15 minutes, and I can't reproduce it

[22:15:12] <joannac> :(

[22:15:37] <q85> interesting. this is happening in our prod environment also.

[22:17:18] <q85> My understanding is that only read preferences of nearest would get around a hidden member, which we don't have anywhere.

[22:17:36] <joannac> what version are you on?

[22:17:41] <q85> 2.4.9

[22:18:04] <q85> all components in the test environment and prod are on 2.4.9

[22:19:39] <q85> The version 5 warning is because I stored rs.conf() to a variable to make the changes to the member, but didn't remove the version.

[22:25:43] <q85> joannac: in testing now, I'm unable to hit the hidden secondary. Does mongos issue an isMaster command to the primary to receive the list of members for each query it directs?

[22:27:30] <joannac> q85: only at the start of a connection

[22:31:06] <joannac> gtg, back in 40

[22:54:43] <q85> joannac: I was able to reproduce it again. It only happens if I do not remove the key "version" when I apply the new config AND the value of the version key is equal to the current version. The new config is still "applied" however, I'm also still able to hit the hidden secondary.

[23:10:33] <q85> joannac: I've now been able to reproduce the situation as long as the new config includes the version key (any value). Even if the value is correct. The config is applied, it corrects the value of the version key, but the hidden member still receives queries.

[23:12:34] <q85> back in 10

[23:20:39] <q85> back

[23:23:48] <joannac> q85: I think that's the same thing I tried, but I didn't get the same behaviour

[23:24:18] <joannac> I suggest you open a SERVER ticket with your repro steps, and then one of the mongodb folks can try and reproduce it

[23:25:17] <q85> ok, thank you for your time joannac!

[23:25:49] <joannac> no probs!

[23:35:30] <elux> hello

[23:35:48] <elux> any tips checking if any of my indexes need to be rebuilt..?

[23:36:09] <elux> or maybe im missing some indexes.. im finding a lot of slow queries, yet i have 15GB of ram and my data set is 19GB

[23:36:21] <elux> that doesnt make sense.. on 2.6.4

[23:44:00] <joannac> example slow query?

[23:44:18] <joannac> add .explain(true) to the end of your query and look at the explain output?

[23:46:25] <elux> https://gist.github.com/pkieltyka/837f785bb03e454e7496

[23:46:27] <elux> this is it..

[23:46:53] <elux> i see it has a planSummary with IXSCAN .. assuming those are the indexes its going to try to scan

[23:47:51] <elux> i see a lot of these.. some > 700ms .. lots in the 500ms

[23:47:53] <elux> so strange

[23:48:07] <elux> it should be like 1ms

[23:48:08] <elux> or less

[23:51:09] <elux> it makes like 5 different plans .. :P

[23:52:52] <elux> lol or how about this nice 7 second query ... yikes :/

[23:52:54] <elux> something is very wrong

[23:54:33] <elux> so shitty

[23:57:05] <isthisreallife> http://pastie.org/9468254 how can i do it with mongoid? it doesnt support has_many through relation right?

Log file Viewer

Help | Karma | Search:

#mongodb logs for Tuesday the 12th of August, 2014