PMXBOT Log file Viewer

Help | Karma | Search:

#mongodb logs for Monday the 10th of September, 2012

(Back to #mongodb overview) (Back to channel listing) (Animate logs)
[04:12:49] <duncan_donuts> I have read the docs and tried multiple things and can't figure this out… hoping someone here can give me a quick pointer….
[04:13:50] <duncan_donuts> .. if I have an ObjectId from collection A, and I want to find documents in collection B which contain the ObjectId in a list field.. what is the query syntax?
[07:27:03] <augustl> what's a good way to find the position in a collection of a single record by Object ID when sorting by a specific attribute?
[07:32:54] <augustl> only way I can think of is to actually fetch all records and find the positin manually in memory
[09:07:28] <gigo1980> hi where can i set the size of the oplog for an new shard ?
[09:43:16] <Rozza> gigo1980: there is an oplogSize arg for mongod
[09:43:45] <Rozza> the docs about sizing are here: http://docs.mongodb.org/manual/core/replication/#replica-set-oplog-sizing including a link to how to resize the oplog if needed
[10:14:47] <Bastian_B> Hello, I am using python 2.7, each time I try to import pymongo I got an error: 'from bson.errors import * ImportError: No module named errors'. Versions: pymongo==2.0.1, mongokit lastest, any idea?
[10:56:57] <NodeX> Afternoon mongoers
[12:05:54] <darklrd> hi, if I want to find the list of all collection names beginning with a specific keyword, how can I do that via mongo shell?
[12:06:32] <Vile> NodeX: I was under impression that you are in UK, was that wrong?
[12:06:59] <NodeX> Vile : that's correct
[12:09:04] <Vile> I see. :) ==> [11:55:53] NodeX: Afternoon mongoers
[12:09:28] <NodeX> 4 mins 7 seconds out lol
[12:09:57] <darklrd> I was wondering if there is any command which would do that
[12:10:15] <NodeX> darklrd : no, you'll have to script it
[12:10:47] <emocakes> http://color.method.ac/
[12:11:03] <darklrd> NodeX, thank you for replying, so I would have to loop through all collection names in the db? No direct command?
[12:11:44] <NodeX> not that I know of darklrd
[12:12:00] <darklrd> NodeX, thanks!
[12:15:33] <Vile> Guys, I need some help with brainstorming
[12:19:13] <Vile> I have a collection with data recorded using timestamps of change
[12:20:22] <Vile> i.e 00:00:05 => 5; 00:00:55 => 6; 00:01:20 => 7; 00:02:04 => 8 , etc
[12:20:55] <Vile> I need to be able to calculate averages for fixed periods of time
[12:21:11] <Vile> In case of the example above - per minute.
[12:22:17] <Vile> Bu in order to calculate average for interval [00:01:00, 00:02:00) I need to know one value outside of that interval
[12:22:31] <Vile> in this case it would be 00:00:55
[12:23:32] <Vile> Question is - how can I do that?
[12:30:40] <Vile> at the moment I'm doing findOne( ) within the map() function. I would prefer to avoid any sub-queries or do them in finalize at least
[12:30:54] <Vile> any ideas?
[12:34:44] <NodeX> you're going to have to change the format of the time unless it's an iso date and do range queries in the aggregate framework
[12:34:52] <NodeX> else you'll have to do it app side
[12:55:01] <Vile> NodeX: ISODate. As far as I've seen aggregation framework does not allow date operations
[12:55:09] <Vile> except very basic
[12:59:44] <Derick> mongodb could do with some better date/time handling. But then again, I might be a bit biased.
[13:00:25] <Lujeni> Hello - I try to choosing my Shard Key. I only use month field (ISODATE) for all query. month field + _id field seems good choice ? Thx
[13:02:13] <NodeX> Vile, I would suggest you do what I do when I need to aggregate on time ranges
[13:02:29] <NodeX> save YmdHMS and range on that
[13:15:07] <Vile> NodeX: can you explain a bit about range queries?
[13:15:29] <Vile> I'm doing ranges now for the values within the interval
[13:17:26] <Vile> but can not decide on how to do that for values outside of he range. Because I don't know how far from the range it could be
[13:20:50] <Vile> I can probably change values from ISODate into timestamps (msec)
[13:22:16] <Vile> then, some of the operations from aggregation framework should be able to work.
[13:22:51] <Vile> By the way, can i change aggregation output to collection?
[13:23:29] <NodeX> I dont think you can
[13:23:45] <Vile> This way i could use aggregation as kind of "emit" query and do m/r on the data afterwards
[13:23:52] <NodeX> with the aggregate funcitons you can group on the minute number for a count / $project
[13:24:25] <NodeX> for example you can gather all users today and break it down into minute by minute registrations
[13:24:47] <Vile> NodeX: sure. but i can do pretty much the same with m/r :)
[13:24:56] <NodeX> then use map/reduce
[13:25:14] <NodeX> what's the problem?
[13:25:32] <Vile> access to values outside of the grouping key
[13:25:46] <NodeX> then use aggregate functions
[13:25:53] <NodeX> and $project
[13:26:39] <NodeX> "Use $project to quickly select the fields that you want to include or exclude from the response. Consider the following aggregation framework operation."
[13:27:02] <NodeX> good luck
[13:27:06] <Vile> NodeX: those fields are in the other document
[13:28:00] <NodeX> then as I have said to you about 10 times in the last few weeks... you cannot do it in one operation
[13:28:04] <Vile> i.e. if i'm calculating averages for let's say last year, i need to get some values from the year before that
[13:28:11] <NodeX> at some point this has to register in your brain
[13:28:32] <Vile> NodeX: registered. long ago
[13:28:50] <NodeX> then why are you still trying the same solutions?
[13:29:15] <Vile> no. i'm trying to figure out better one.
[13:29:36] <NodeX> which I am offering but your schema will have to change
[13:29:53] <NodeX> or you will have to do it in 2 operations
[13:30:34] <Vile> NodeX: schema change is fine. but what you suggesting is actually external application. because aggregation does not have persistence
[13:31:01] <NodeX> no but your app does have persistence
[13:32:15] <Vile> NodeX: at the moment all teh aggregations are done on DB level. app just uses them
[13:32:46] <Vile> i have number of intermediate collections with pre-aggregated values
[13:32:57] <NodeX> which is the correct way to do it
[13:33:06] <Vile> what i'm trying to figure out - the most effective way to build those
[13:33:33] <Vile> incrementally, in parallel, etc, etc
[13:34:52] <Vile> i.e. => get rid of "find" inside of aggregation procedure or at least reduce number of those.
[13:35:34] <Vile> actually i've just thought of one idea. i need to store previous value together with current value in the same document
[13:35:53] <Vile> of course it would cause data duplication
[13:36:09] <Vile> but should be much better performance-wise
[13:37:12] <Vile> then, on the next step i should be able to build pre-aggregated collections using this pre-recorded "previous" value
[13:40:57] <Vile> really, aggregation framework output as a source for reduce procedure would be nice :)
[13:42:54] <NodeX> I do somehting similar, every 5 minutes I grab the latest data for today and process it
[13:43:43] <NodeX> I upsert my aggregates for later aggregation and plsit them into minutes from days, I get good performance and a semi realtime feel to it
[13:45:16] <NodeX> plsit -> split
[13:53:38] <Vile> At the moment i have around 3-4K values per second processing. need to boost it 5 times at least :(
[13:54:27] <IAD> Vile: try use bulk inserts, if posible
[13:59:56] <Vile> IAD: Already
[14:01:26] <Vile> Would like to switch to sharding, but unless I get rid of "find" inside the map() function, it will not offer any additional benefit
[14:04:27] <dominikzogg> jmikola. Can you link me an example how to use the querybuilder with reference many document? http://docs.doctrine-project.org/projects/doctrine-mongodb-odm/en/latest/reference/query-builder-api.html
[14:07:36] <jmikola> http://www.doctrine-project.org/api/mongodb_odm/1.0/class-Doctrine.ODM.MongoDB.Query.Expr.html#_includesReferenceTo is the only relevant method afaik; also, this channel really isn't the place for such questions. #doctrine would be more appropriate.
[14:11:20] <dominikzogg> <jmikola> thx
[14:26:46] <sirious> anyone know of a way to determine which port your instance of mongo is running on within mongo shell?
[14:26:58] <sirious> db.serverStatus() only has hostname, but no port
[14:30:50] <algernon> sirious: db.serverStatus().repl.me perhaps?
[14:31:19] <algernon> though, that possibly is only available when using a replicaset
[14:32:30] <algernon> alternatively, db.getMongo()
[14:33:05] <sirious> algernon: is that 2.2 specific?
[14:33:26] <sirious> .repl has no properties on my 2.0.6 sharded instance
[14:33:36] <sirious> and getMongo() only returns the IP
[14:33:49] <algernon> for me, db.getMongo().host returns the ip and the port aswell
[14:34:31] <sirious> odd
[14:34:44] <sirious> db.getMongo().host returns "127.0.0.1" and that's it
[14:34:55] <algernon> oh..
[14:35:16] <algernon> that returns the ip/port you connected to, as-is, and I did mongo localhost:27017
[14:35:31] <sirious> ahh
[14:35:36] <sirious> i think that's the problem
[14:35:40] <sirious> if you ran just 'mongo'
[14:35:59] <sirious> it probably won't do that i'd guess
[14:36:15] <algernon> yep, then it just prints 127.0.0.1.
[14:36:43] <sirious> sweet. i'm not crazy! lol
[14:36:56] <sirious> trying to work on my .mongorc.js
[15:26:44] <Nopik> hi all.. i have some race condition in my app, i'd like to know what is the best way to solve it. Lets say we have User documents, each user has unique name property. So, in my code I have something like register_user( name ), which need to create new User document with given name, but only if name is not taken yet. Of course calling User.find({name:name}) and calling User.save() after getting 0 documents from find() is *bad*. How that
[15:26:44] <Nopik> should be addressed instead?
[15:26:57] <Nopik> there is upsert, but I do not modify existing record, if it is existing
[15:27:33] <algernon> insert with safe mode
[15:27:46] <algernon> if it returns an error, the user already existed
[15:34:36] <Nopik> algernon: and that is guaranteed only if I have unique index on name field, right?
[15:35:02] <Nopik> algernon: i.e. unique filter on the field is the source of error which I'll get after invalid insert
[15:37:20] <algernon> Nopik: yep
[15:37:28] <Nopik> great, thanks
[15:49:16] <gigo1980> is anyone here how use the mongopool in php ?
[15:50:56] <NodeX> mongopool ?
[15:55:24] <NodeX> Ah Derick, any news for the socket connection?
[15:55:52] <Derick> i need a little bit more context :-)
[15:56:06] <NodeX> connecting to Mongo via a unix socket in the php driver
[15:56:18] <NodeX> it got ~"missed out" in the 1.3
[15:56:26] <Derick> ah
[15:56:26] <Derick> yes
[15:56:27] <Derick> I do
[15:56:34] <Derick> it's been added to the spec
[15:56:43] <Derick> I will probably not make 1.3, but 1.3.1 instead
[15:56:59] <NodeX> ok kewl, when is 1.3.1 due for release?
[15:57:12] <Derick> 1.3.0 first :-)
[15:58:55] <NodeX> ok brilliant
[15:59:00] <NodeX> so a couple of months then?
[16:00:37] <Derick> no
[16:00:45] <Derick> I expect 1.3.1 to come out fairly soon
[16:03:04] <NodeX> ah great, thanks
[16:11:05] <Almindor> hello
[16:11:12] <Almindor> how do you do "array in array" query?
[16:11:36] <VultureZ> http://www.pastebin.ca/2202902 trying to create a document with sub-documents that have a unique index field that is being enforced even though no sub-document entries are added.
[16:11:52] <VultureZ> Any ideas on how to resolve this?
[16:11:56] <Almindor> say documents have an array field and you're looking for "any of [1,2,3,4]" inside of them (so it matches all documents which have any of those 4 inside their arrays)
[16:12:41] <VultureZ> Almindor,can't you use a regex query to look for any of those values within that array?
[16:13:19] <Almindor> VultureZ: it's numbers and non-continuous
[16:13:24] <NodeX> Almindor : use an array
[16:13:43] <NodeX> db.foo.find({your_array:[1,2,3,4,5]});
[16:13:56] <Almindor> NodeX: and that looks for "or" values?
[16:14:03] <NodeX> if youi want all docs with 1 or 2 or 3 or 4
[16:14:09] <Almindor> ah good
[16:14:16] <Almindor> I thought it was "and"-ing in that case
[16:14:20] <NodeX> do this .... db.foo.find({field:{$in:[1,2,3,4]}});
[16:14:21] <Almindor> is it particularly slow?
[16:14:36] <NodeX> $in = it will match any of those
[16:14:41] <Almindor> mmhm
[16:14:54] <Almindor> NodeX: thanks
[16:14:57] <NodeX> ;)
[16:16:20] <VultureZ> If you have a sub-document and you have a field that is unique and required, if you create the parent document without any of the subs, how can you prevent the sub's unique field from being enforced?
[16:16:43] <NodeX> you can't force it
[16:17:04] <NodeX> or unforce it, unless you add the part of the document that it's being enforced on
[16:17:23] <NodeX> null is a valid value for an index and therefore it will throw a dupe
[16:17:26] <VultureZ> okay, even if it doesn't require it to have that sub-document added?
[16:17:30] <VultureZ> okay
[16:17:39] <NodeX> if your unique index requires it
[16:21:43] <ppetermann> VultureZ: maybe the term subdocument is misleading you there, its not really a document stored in a document, its sort of part of the document
[16:41:33] <jmpf> http://pastie.org/private/5tsydgxkf2y9zmd03okb1q <--am I doing something wrong here? this takes forever to run - on mysql it runs in less than 80ms on a 100k row set - indices are present on created_at
[16:48:22] <mrpoundsign> jmpf: what's explain look like?
[16:48:39] <DarkSector> Can anyone please help me out with this? http://pastie.org/private/oyq7nymziuqh1pfdpuxg
[16:48:41] <DarkSector> Pymongo
[16:49:54] <mrpoundsign> DarkSector: Looks like a python strings issue.
[16:50:21] <DarkSector> yeah but I am inserting a dictionary
[16:50:40] <mrpoundsign> DarkSector: But you're forming a string to insert.
[16:50:49] <mrpoundsign> and it's complaining about the string.
[16:51:14] <DarkSector> mrpoundsign: but when I check the type it returns dict
[16:51:16] <DarkSector> not str
[16:53:04] <DarkSector> mrpoundsign: how should I solve it? I mean I need to insert every row in the db
[16:53:24] <mrpoundsign> DarkSector: Not a python expect. But you're putting a string in the dict, right?
[16:53:32] <DarkSector> ah yes
[16:53:38] <DarkSector> a string in the dict
[16:53:39] <mrpoundsign> expert*
[16:53:49] <DarkSector> but how is that a problem?
[16:53:58] <mrpoundsign> DarkSector: ok, try just that line where you're adding things together to make them a string.
[16:54:21] <DarkSector> mrpoundsign: try how? print?
[16:54:30] <mrpoundsign> DarkSector: then check out why python is returning a UTF-8 error on that -- it's really not a mongo issue.
[16:54:35] <DarkSector> oh
[16:54:55] <mrpoundsign> foo = str(row[2] + ',' + row[3])
[16:56:35] <mrpoundsign> DarkSector: I'd try "%s,%s" % row[2] row[3]
[16:56:47] <mrpoundsign> but my python experience is from quite a while ago
[16:56:52] <DarkSector> okay let me try that :)
[16:57:10] <camonz> is there a way to step-through or make a mapReduce call be fully explicit on what's being emitted and reduced ?
[16:58:31] <mrpoundsign> might be more like ("%s,%s" % row[2],row[3])
[17:01:12] <DarkSector> mrpoundsign: stags.insert({'states': row[0], 'districts' : row[1], 'disaster' : row[2], 'climates' : row[3]}) also yields the same error
[17:01:21] <DarkSector> now this is a proper dict isn't it?
[17:17:37] <mrpoundsign> DarkSector: what are the types of all those rows, and are you trying to insert a non-string as a string?
[17:17:54] <DarkSector> mrpoundsign: I think its because of the csv
[17:17:57] <Almindor> if you do collection.save(doc) in java does doc get the new _id and default fields filled in or do you have to query for it?
[17:18:04] <DarkSector> because the csv doesn't support unicode encoding
[17:18:51] <mrpoundsign> DarkSector: I don't know enough python to fox that. have you tried the python channel about the conversion to a string?
[17:19:04] <DarkSector> Yes, but nobody seems to be able to help
[17:19:16] <DarkSector> I got an error on UnicodeEncoding now, I've asked it
[17:20:09] <mrpoundsign> http://stackoverflow.com/questions/3305926/python-csv-string-to-array
[17:51:05] <Gargoyle> Is it possible to rename a database?
[17:52:50] <sqwishy> Where do I go to complain about bugs in the mongo-pyton-driver project?
[18:24:22] <toddnine> Hi guys! Dumb question, but… Is there a quick way using the Java driver to find a document by it's "_id" or do I need to use the query syntax?
[18:29:00] <jmpf> trying to group rows by day of year - http://pastie.org/private/tjrk8snurcqh8ssupzb9qq - what's wrong w/this aggregate query?
[18:38:23] <skot> toddnine: you need to use the query syntax.
[18:39:39] <skot> sqwishy: http://jira.mongodb.org/browse/python
[18:39:59] <skot> Gargoyle: no, but you can dump/restore to a different db
[18:40:11] <Gargoyle> skot: Done that.
[18:40:41] <Gargoyle> I assume that once the old db is not used, it will get unmapped from memory? (not quite ready to drop it just yet)
[18:41:03] <skot> yes
[18:41:16] <skot> or that is to say the new stuff will replace it if memory is tight.
[18:42:23] <Gargoyle> 9GB on disk (dump dir usage), 32GB system RAM
[19:05:05] <tlipcon> hey folks. random question: has anyone re-run TPC-H benchmarks on mongo after the latest aggregation framework changes?
[19:05:11] <tlipcon> similar to http://www.ifi.uzh.ch/dbtg/teaching/thesesarch/VertiefungRutishauser.pdf ?
[19:10:35] <Venemo> hi
[19:43:28] <jaraco> I have a continuous integration server where our MongoDB instance will no longer start.
[19:43:38] <jaraco> I upgraded it this morning to 2.2.0.
[19:43:58] <jaraco> It's a Linux Lucid box using the 10 gen apt repo.
[19:44:24] <jaraco> So I've uninstalled MongoDB. Deleted /etc/init/mongodb.conf. Reinstalled. Still, the service doesn't start.
[19:44:29] <jaraco> In fact, there's no upstart script now.
[19:44:45] <jaraco> But /etc/init.d/mongodb still references upstart.
[19:44:56] <jaraco> Any suggestions?
[19:50:33] <jaraco> I think the 10gen Lucid build of MongoDB 2.2.0 packaged for upstart doesn't properly install the /etc/init/mongodb.conf.
[19:56:37] <jaraco> I upgraded my Precise host this morning without any issues, so if the issue is with the distro, it's specific with Lucid.
[20:13:20] <jaraco> I just installed to a clean system, and it installed the file without a problem.
[20:13:21] <jaraco> :/
[20:21:36] <Spaceghostc2c> Is there a preferred way to use multiple reducers in the mongodb community, or is it safe to assume I'm the local expert for my own problem?
[20:36:08] <sysdef> hello. i read there is no version for android. there is (still) no version?
[21:07:29] <ribo> is there any way to have mongos route database reads to a specific replica set? and writes to another?
[21:07:47] <ribo> e.g. writes to the primary, reads to secondary?
[21:18:51] <camonz> is there a way to apply a new reduce to a field in a collection that's the result of a map/reduce ?
[21:26:33] <AlessandroD> Hello channel! I am new to MongoDB and I'd like to know what is the recommendation on using the Mongo-generated __id as a public external identifier in my application's object model. Is that somehow discouraged? Is there somewhere with information about it? Thank you in advance!
[21:27:36] <ron> I don't see a problem with it. Foursqaure uses it that way.
[21:31:51] <AlessandroD> hmm, is that so? interesting
[21:35:11] <crudson> http://www.mongodb.org/display/DOCS/Object+IDs for general info re. _id and ObjectID. Make yourself aware of what the generated id represents and decide if you're happy using it that way.
[22:58:59] <sqwishy> Is there a way to submit bug reports without creating an account on the jira thingy?
[23:02:01] <skot> nope, not really.
[23:16:16] <RomyK> the aggregation framework, why it is better to use it instead of map reduce?
[23:29:15] <skot> native code, and not javascript, better concurrency and less memory usage (most likely)… but it has limitations as well
[23:29:39] <RomyK> ahh.. okay
[23:40:08] <BP_> Hi, I keep trying db.loadServerScripts() but I get TypeError. Tried on both 2.2.1-pre- and 2.3.0-pre-