pmxbot IRC Log Viewer

[01:21:14] <AnonHealthEmploy> Does anyone have experience using MongoDB for a use case that requires high durability?

[03:16:53] <Tobsn> what exactly is unstable at mongodb-10gen-unstable - anyone know?

[03:21:30] <acidjazz> in mongo+php is it possible to update the actual bytes of a file in mongogridfs

[03:21:36] <acidjazz> or do i need to remove() and then storeBytes()

[03:25:47] <Tobsn> you should ask kristina about that, she should know ;)

[03:33:53] <acidjazz> who the..

[05:01:02] <richardr> If I have a million documents, and would like to pull out a specific 100 by IDs, what is the best approach? Thanks

[05:12:57] <carsten_> hi creatures

[06:11:59] <skot> richardr: {_id : { $in : [<ids>]}}

[07:32:11] <[AD]Turbo> hola

[08:45:29] <kali> ca a l'air fun le tgv aujourd'hui

[08:45:35] <kali> oups, sorry guys :)

[09:47:51] <NodeX> Pure power of indexes... I was (testing) somehting with upserts and the import was taking about 5 minutes for 85k documents .. about 40k documents through the second test I ensuredIndex on the upsert field and it finished in 3 seconds - if that's not testement to the speed of indexes I dont know what is!

[12:27:30] <abrkn_> what's a quick way to add a field to each document in a collection that's the insert time of the document in iso 8601 (extended)? i've kept the original _id but not familiar with their timestamps

[12:27:52] <NodeX> your app will have to

[12:28:24] <abrkn_> ok, that was a bit unclear. i've modified my app to add that for -new- documents, but need to fix the old ones. only like 4k docs

[12:29:14] <NodeX> db.foo.update({field: {$exists :false}},{$set : {field : ''}},false,true);

[12:29:33] <NodeX> where field is the field you want to add

[12:30:01] <abrkn_> what's the timezone of _id.getTimestamp()

[12:30:06] <abrkn_> ?

[12:30:56] <NodeX> whatever your server TZ is

[12:32:59] <abrkn_> k

[13:15:38] <maxmc> hi. does someone know how to sort queried docs by the closest date from now, but only future dates? thanks. I'm using mongo 2.02

[13:22:04] <carsten_> and why only for future dates? what do you lose when you sort everything?

[13:22:17] <carsten_> honestly, a pretty weird requirement

[13:27:10] <chandru_in_> I understand that a normal update can yield the write lock if a page fault occurs. Does the same apply to findAndModify?

[13:33:57] <maxmc> @carsten i need to display upcoming events. next event first and so on. i dont think this is weird ;)

[13:34:19] <carsten_> why only sort on certain part of the data?

[13:34:29] <carsten_> the partial sorting is weird

[13:35:28] <maxmc> no no misunderstanding, i was describing the result. to compute it i obviously need to sort all data

[13:35:52] <carsten_> so the problem is?

[13:36:01] <maxmc> but the result should yield the next date first and put past dates to the end

[13:36:08] <maxmc> ..the query

[13:36:56] <carsten_> perform a range search...

[13:37:42] <maxmc> ah

[13:38:02] <maxmc> great! thanks for the hint. i think i can make it now :)

[13:38:33] <spillere> if i have {user: name, photo: [{foto2}, {fto2}]... how can I get how many photos a user have?

[13:38:54] <carsten_> count on the client side

[13:39:09] <chandru_in_> Does findAndModify hold the write lock even if there is a page fault?

[13:42:17] <therealkoopa> I'm trying to figure out the best way to store a changeset/versioning system on some objects that are persisted. Is it better to store a changesets: [] on a document, or create a separate history collection ?

[13:50:43] <wereHamster> depends

[13:54:16] <therealkoopa> I'm thinking it's better in a separate collection since it won't be accessed frequently.

[13:55:44] <carsten_> the 16MB document limit must be considered

[15:18:32] <mikerj> hi guys

[15:19:09] <mikerj> i've got a sharded, authenicated cluster using the --keyFile arg

[15:19:19] <mikerj> it has multiple db's and multiple users

[15:19:31] <mikerj> when i add a user to a database, and auth that user

[15:19:55] <mikerj> the password is accepted (a 1 returns), but the user can't issue any finds due to unauthorized errors

[15:19:57] <mikerj> any thoughts?

[15:22:53] <therealkoopa> I'm using mongoose (node) and defining some schema/models that reference each other. If you have a BlogPost with comments: [Comment], each comment has an objectId. Is this necessary? Can it be done without an objectId in each embedded document

[15:58:38] <solars> hey, are there any tools to get an overview of the slowest queries? not just logging but a nice overview, etc?

[15:58:46] <NodeX> why would you not want an Object id therealkoopa ?

[15:59:01] <NodeX> sorry, I misread your question

[15:59:43] <carsten_> solars: the standard profiler is not good for you?

[16:00:24] <Lujeni> solars, profiler is great, also you can use "explain" method for specific query.

[16:00:51] <solars> I'm looking for a tool that analyzes the logs of a running app and filters out the slowest one, I'm not yet interested in particular queries

[16:01:37] <carsten_> that's what the profiler is for

[16:02:26] <solars> I know the profiler, but I'm looking for a tool that uses it to create a report based on various data (days, etc)

[16:03:02] <carsten_> dream on

[16:03:24] <solars> it was a question

[16:03:39] <carsten_> write one or take the profile

[16:03:41] <carsten_> r

[16:04:22] <solars> how about just answering along 'no I don't know any' or nothing at all?

[16:04:58] <carsten_> you had a bad sleep?

[16:05:19] <solars> I slept very well

[16:07:11] <NodeX> solars : why not write one ?

[16:07:17] <NodeX> won't take long

[16:08:01] <solars> I just wanted to know if anyone knows a tool like that already, as there is one e.g. for rails

[16:08:41] <carsten_> NO, NO SUCH TOOL

[16:09:12] <solars> do you have some kind of problem carsten_?

[16:09:22] <carsten_> *ignore*

[16:09:25] <solars> apparently there are such tools, besides that I was explaining why I'm asking

[16:09:34] <solars> thanks

[16:09:56] <NodeX> https://scoutapp.com/plugin_urls/291-mongodb-slow-queries

[16:10:01] <NodeX> google is your friend

[16:10:19] <NodeX> ignore carsten - he's an idiot

[16:10:35] <solars> yeah, I realized already a couple of weeks ago

[16:10:36] <NodeX> he likes to be a bad kid since MacYET got banned

[16:11:01] <solars> NodeX, I saw the scout plugin, will give it a try, but as new relic it's also hosted

[16:11:11] <solars> just wondered about the alternatives

[16:11:13] <carsten_> well said by the channel jerk nodex

[16:11:31] <NodeX> I don't know of any - I wrote my own but it's not in Rails

[16:11:47] <solars> I think I'll also write a short report script

[16:12:03] <NodeX> perhaps scout can give you some ideas ?

[16:12:15] <Goopyo> is there a way to keep a collection in memory?

[16:12:17] <solars> I'll check the plugins, yeah

[16:12:20] <solars> looks nice

[16:15:28] <Goopyo> I have to query all rows in a collection (testing 80,000) and it seems to take 7s. Should I unload this stuff to a cache? or is there a way to make it faster by indexing the whole collection

[16:15:50] <carsten_> create an index...

[16:17:49] <Goopyo> carsten_: yeah I get that, but on the whole db?

[16:17:55] <Goopyo> I mean collection?

[16:18:36] <carsten_> did you ask this on the list yesterday?

[16:18:49] <Goopyo> nope

[16:19:41] <carsten_> then look at the archieve - this has been asked and answered yeterday

[16:19:46] <Goopyo> http://pastebin.com/Dv9ZBU8K

[16:20:15] <Goopyo> is this the google group? Never used it gonna go look for it

[16:28:17] <Goopyo> carsten_: https://groups.google.com/forum/?fromgroups#!topic/mongodb-user/nhT7yLOs2Yo

[16:28:18] <Goopyo> that?

[16:28:50] <carsten_> eactly your question

[16:29:10] <Goopyo> yeah doesnt really apply to me though

[16:29:11] <Goopyo> m

[16:29:11] <Goopyo> y i

[16:29:38] <carsten_> then i can not help

[16:30:01] <carsten_> somem other poor creature may know

[16:30:04] <Goopyo> my issue is that I want to write a ranking algorithm that requires me to run calculations on all the rows in a collection (i.e for all-time rankings). How can I do this fastest?

[16:30:46] <carsten_> then don't ask for in memory collectioins

[16:31:21] <carsten_> as my coworker said: stay out of #mongodb - full of icompetence

[16:32:08] <solars> good that the incompetence just left

[16:44:41] <Goopyo> jesus what a douche

[16:54:21] <NodeX> lol

[16:54:26] <NodeX> I have him on perma ignore!

[16:54:39] <solars> heh

[16:55:33] <NodeX> imo civility and courtesy cost nothing so you should give them for free

[16:55:48] <NodeX> and in the interwebs you can be anyone you like so dont be a butthole!

[16:55:56] <NodeX> (not you - people in general)

[16:57:27] <solars> thats right

[16:58:19] <tystr> lol

[16:58:26] <tystr> wow

[16:59:16] <tystr> question: I've got a collection with an array of embedded documents that contains an array of embedded documents

[16:59:41] <tystr> how would I update a doc that's embedded that deeply?

[16:59:51] <NodeX> you have to walk the arrays

[17:00:02] <NodeX> foo.arr1.arr2.bar

[17:00:02] <tystr> I found myself needing to use the $ positional twice, which obviously doesn't work

[17:00:29] <NodeX> do you know the position of either one of the arrays ?

[17:00:46] <tystr> no

[17:01:08] <tystr> basically the deepest embedded doc is in another collection, and when it's updated there, I wanted to update it everywhere it's embedded

[17:01:36] <NodeX> I dont think it's possible with a direct query

[17:01:48] <NodeX> you will have o bring it into your app and modify it then put it back

[17:02:04] <tystr> hmm ok, that's kinda what i was thinking, just wanted to make sure I wasn't missing something

[17:02:21] <NodeX> I stopped embedding after 1 depth for this reason back in 1.4 /1.6 or something

[17:02:43] <tystr> ya, I could do it with $set if it were only 1 deep

[17:03:17] <NodeX> if you knew the position you can number the arrays

[17:03:36] <NodeX> foo.arr1.0.bar=

[17:05:40] <NodeX> https://jira.mongodb.org/browse/SERVER-831 <---- suggested patch for 2.3

[17:06:59] <tystr> yeah

[17:07:56] <tystr> but this doc could be embedded a bunch of times, i just want to do a multi update on it everywhere in the collection

[17:08:02] <tystr> guess I'll have to rethink this a bit

[17:12:31] <tystr> gah that feature is exactly what i need! lol

[17:13:48] <NodeX> !!

[17:14:35] <jasiek> hey, i think we've found what looks like a bug in mongodb

[17:15:28] <jasiek> when you have a unique index on an array of items, it won't ensure that the elements in the array itself are unique

[17:15:38] <jasiek> now is this a bug or is this by design?

[17:15:42] <NodeX> by design

[17:15:51] <NodeX> $addToset pushes unique values

[17:19:13] <Goopyo> made a json dump of a collection to a file, the read operation is 4x faster than reading the collection off mongodb

[17:20:12] <jasiek> NodeX: thanks

[17:20:39] <NodeX> ;)

[17:23:54] <ranman> Goopyo: how big was the collection and how many times did you run that test?

[17:24:29] <Goopyo> how many times do you think a good test should be run?

[17:24:35] <Goopyo> ill run it real quick

[17:24:56] <Goopyo> ranman: ^

[17:25:15] <infinitiguy> Hi all, Are there any tips to improve performance on a mongo DB when doing a lot of deletes while under a high read/write load? We saw our performance (requests per second) tank when starting deletes

[17:25:27] <infinitiguy> we were doing 2000-3000 r/ws and it fell to about 300ish

[17:27:17] <Goopyo> infinitiguy: whats the data structure like? i.e are old ones being deleted or is it random?

[17:27:54] <infinitiguy> I think it was all that matched a particular ID

[17:27:57] <Goopyo> if its random you can always que up deletes so you do the operation less often

[17:27:59] <infinitiguy> we had a value cid = 432

[17:28:05] <infinitiguy> and were deleting everything that matched that

[17:28:13] <infinitiguy> so I think that wouldn't be random

[17:28:51] <dstorrs> infinitiguy: do you need to actually delete them, or can you just mark them as obsolete until load decreases?

[17:29:07] <Goopyo> yeah can you que them for delete and just delete later?

[17:29:45] <Goopyo> if its timeseries data you can used tiered collections (i.e drop a collection which is way faster than deleting the documents)

[17:29:47] <infinitiguy> well - I guess that's the tough part. This is an environment where there usually isn't a low load period. I suppose in the middle of the night load will be a little less - but the application is worldwide so it's never truly quiet

[17:30:14] <dstorrs> hm.

[17:30:26] <dstorrs> well...you're using replica sets, yes?

[17:30:43] <infinitiguy> not yet - this is a new app - we saw the delays during testing

[17:30:47] <dstorrs> fallback, if there's no way to speed it up, would be to rotate machines out of the set, do the delete, add it back.

[17:30:50] <infinitiguy> we were thinking about replica sets

[17:30:59] <dstorrs> but that's a fallback. There out to be a way to do this.

[17:31:15] <infinitiguy> how do replicas work with reads? does mongos automatically send reads to the replicas and have writes/deletes on the master?

[17:31:18] <dstorrs> *ought to be

[17:31:43] <dstorrs> unless you set "slaveOkay : 1" in your read, it always goes to master.

[17:31:44] <infinitiguy> disclamer - I'm also extremely new to mongo - so forgive any stupid things I say :)

[17:31:51] <dstorrs> the slave is purely for redundancy

[17:31:54] <infinitiguy> ok

[17:32:00] <dstorrs> writes / deletes always go to master

[17:32:02] <ranman> Goopyo: depends on how long it takes, the bigger the sample size the better, at least a 100 times for something that takes 1s

[17:32:43] <Goopyo> it was 80k documents, list(collection.find()) took 8s, dumping that list to a json file and loading it took 2s. Just tested 10 times with the same thing

[17:40:12] <Goopyo> test finished 12.2 seconds per 80k mongodb read (10 times), 1.53s for a json load from file (10 times).

[17:48:30] <zenocon> hi, can i create a compound index that ignores null values?

[17:49:15] <Goopyo> zenocon: look into sparse indexes

[17:49:25] <zenocon> e.g. { a: "foo", b: "bar", c: null }, { a: "foo", b: "bar", c: null } will not generate a dupe error, but if c has the same value on both, then it will

[17:49:44] <zenocon> i was trying to use sparse, but somehow i don't have it right yet...still getting dupe error

[17:49:48] <Goopyo> well you can also do a non unique index

[17:50:11] <zenocon> Goopyo - non unique index?

[17:50:22] <Goopyo> yeah indexes dont have to be unique

[17:50:37] <Goopyo> i.e indexing 'c' with two nulls int an issue

[17:50:42] <Goopyo> http://www.mongodb.org/display/DOCS/Indexes#Indexes-unique%3Atrue

[17:50:57] <Goopyo> infact unique is optional, not default

[17:51:08] <zenocon> well, i have a compound index on three properties. if all three are identical, then i want it to throw a dupe error, but if only 2 of the 3 are identical i want to allow it

[17:54:52] <zenocon> Goopyo: here's what I'm trying to do: http://hastebin.com/peqotocaxu.sm

[17:55:04] <zenocon> using mongoosejs, but principles are the same

[17:55:35] <zenocon> i want to allow dupes that have the same raffle id and number properties that have a winningTicket == null;

[17:55:52] <zenocon> but not allow the same winningTicket number to be assigned to more than one prize document

[17:56:49] <ranman> Goopyo: could it just be an IO buffer for the printing?

[17:57:10] <Goopyo> ranman: not printing anything

[17:57:53] <ranman> anychance you have a gist of the code you're using?

[17:58:44] <Goopyo> ranman: can you run python?

[17:58:48] <ranman> yes

[17:58:58] <Goopyo> one sec lemme clean it

[17:58:58] <ranman> are you doing this in the interpreter

[18:01:36] <Goopyo> nope running it as a scirpt

[18:15:38] <Goopyo> ranman: still here?

[18:15:51] <mklappstuhl> Hey

[18:16:12] <eloy__> Hi, When I use Doctrine-mongoDb lib, this library create new query for Database Reference, don't use getDbRef. Any suggestion?

[18:26:47] <ranman> Goopyo: sorry @10gen work takes me to and from channel

[18:27:05] <Goopyo> all good heres the simulation

[18:27:07] <Goopyo> https://gist.github.com/2961392/753700ad4f10af6daa2bc50f46a47955deae8a66

[18:27:26] <Goopyo> just change 7-9 to your desired values (9 is your collection) and run it

[18:31:48] <ranman> Goopyo: kk, I'll take a look now

[18:33:16] <ranman> you're using pymongo?

[18:33:24] <Goopyo> yup

[18:46:09] <Goopyo> ranman: any luck?

[18:47:21] <ranman> Average: 1.17445559502

[18:47:23] <ranman> Testing: test_json_read

[18:47:23] <ranman> Average: 0.61522860527

[19:29:24] <kuzushi> Is it possible in the log file to observe all the queries being passed to the server? from what I've seen the logs only show that a query is being performed, even with a -v start of the server?

[19:33:07] <matthewvermaak> i am trying to do an atomic update to a field inside an embedded document, and seem to be tripping up, is there examples of that?

[19:33:21] <matthewvermaak> on the mongodb home, i see examples on documents, not embedded documents

[20:31:34] <meonkeys> Would someone mind approving my post to mongodb-user? I think a member's first post is moderated or something like that.

[20:45:36] <infinitiguy> what's the side effect of having a really small chunkSize?

[20:45:40] <infinitiguy> in a sharded environment

[20:45:48] <infinitiguy> could that possibly affect delete operations?

[20:46:08] <infinitiguy> I was testing with a heavy load (thousands of read/writes) per second with a 1MB chunkSize set.

[20:46:42] <meonkeys> Anyone know what the RSS/atom feed URL is for https://groups.google.com/forum/?fromgroups#!forum/mongodb-announce ?

[21:05:14] <meonkeys> Whomever just approved my post(s): thank you!

[21:45:08] <timkuijsten> i really don't like to specify a password on the commandline (and in my import scripts that are under version control). so how can i let mongoimport read a config file that contains a password on the server?

[21:47:15] <timkuijsten> doesn't look like mongoimport has any support for this (neither the mongo client) https://github.com/mongodb/mongo/blob/master/src/mongo/tools/import.cpp :(

[22:12:32] <ranman> you could use an environment variable

[22:12:37] <ranman> timkuijsten: ^

[22:13:30] <timkuijsten> ranman: well, it's a start, but then i still have to save the password in my import scripts..

[22:13:47] <ranman> I mean what other solution would you like?

[22:13:52] <ranman> you can use ~/.mongorc

[22:15:08] <timkuijsten> can i?

[22:15:21] <ranman> I'm fairly sure yeah… let me find you a link

[22:15:21] <timkuijsten> ranman: that's exactly what i'm looking for

[22:15:26] <timkuijsten> please..

[22:15:52] <timkuijsten> at least for mongoimport, and preferably for `mongo` as well..

[22:22:36] <ranman> well ~/.mongorc.js always works for mongo

[22:22:42] <ranman> it does not work for mongoimport

[22:27:39] <ranman> http://www.mongodb.org/display/DOCS/Overview+-+The+MongoDB+Interactive+Shell#Overview-TheMongoDBInteractiveShell-.mongorc.js

[22:30:36] <timkuijsten> ranman: cool tnx, i'll file an issue in jira for mongoimport and the other tools

[22:30:52] <ranman> timkuijsten: don't actually I'll explain the reasoning in PM

[22:31:09] <timkuijsten> PM?

[23:26:52] <matthewvermaak> anyone using mongoid in here, have time for a Q?

[23:32:27] <meonkeys> I guess not. :( It's been pretty quiet in here today.

[23:47:50] <BurtyB> tbh prob best to just ask the Q and wait rather than asking to ask something people don't know about

Log file Viewer

Help | Karma | Search:

#mongodb logs for Wednesday the 20th of June, 2012