Log file Viewer
#mongodb logs for Wednesday the 5th of March, 2014
[00:20:07] <niftylettuce_> Right now, per that link to StackOverflow, you have to do `docs.filter` in order to filter out any tags that were not populated (e.g had null or undefined values). What I want to do, is instead of having to do this extra step with `docs.filter`, I want to add something like `.where({ tags: { $ne: undefined })` or something to the main query before `.exec`
[00:20:08] <niftylettuce_> is called. However it appears this feature is not supported or something.
[00:24:00] <niftylettuce_> e.g. by having .limit and .sort in the main query, not at/after the .filters part as shown in S.O., it makes it fast
[01:13:30] <Asterfield> Anywho, question time. I _really_ love the idea of mongo, but my workplace uses MSSQL exclusively, so my brain has a lot of table-based habits
[01:16:05] <Asterfield> I have a `users` collection and a `conversations` collection. Right now, I'm trying to query all the conversations that have a certain user's `_id` as `userA` or `userB`.
[01:17:13] <Asterfield> This seems like it's going against the spirit of mongo, but I don't know the "correct" way of structuring things
[01:33:53] <Asterfield> What is the appropriate way to structure conversations and users in that a conversation can be queried by the user inside it?
[01:43:36] <Asterfield> Is monk's querying ability crippled? I can't get a simple `or` or `in` to work
[02:27:03] <alsotest> why does mongoimport not want to import this? https://raw.github.com/shuw/ies-twitter/master/data/crunchbase/service-providers.js
[04:35:31] <Fandekasp> normally, clicking on the addon redirects to the proper interface, but today I am redirected to partners.mongohq.com and I have to login with unknown credentials (those setup by heroku are username&password, partners.mongohq.com asks for email&password
[04:35:57] <Fandekasp> Any idea why this only happened today? and what email am I supposed to use there?
[04:36:53] <ranman> Hi Fandekasp: this is a mongodb channel not a mongohq channel but have you tried just using your existing credentials?
[04:38:32] <Fandekasp> ranman: oh right, sorry. and yes, I tried to fill the email field with the username "heroku", without success. I then tried to register with the same email I use in heroku, but it didn't find my heroku mongodb
[04:42:56] <ranman> I thought you tried to use the username "heroku" which was not your personal username I assumed
[06:02:40] <safani> Im getting an error Failed to load c++ bson extension, using pure JS version , any ideas
[08:17:19] <Repox> Hi guys. I'm having an issue with updating some members. I'm trying the following command which reports an error (which can be seen in the same pastie). Could you help me figure out how I can achieve what I'm trying to do? http://pastie.org/private/fyyehbo0ifstlappxvevw
[08:47:03] <kali> Repox: i assume what you want is to use $set. the second update argument can be either - a full document that will replace the matched document, - or a combination of $xxx modifiers (like $set, $addToSet, $push, etc)
[09:26:02] <Striki> we need to change the format of the shard key. we must do it manually by deleting the documents, apply the hashing in client code and insert them again. the number of documents that we need to change are around 100-200 million in a 530 million document collection. It's distributed over 4 shards. Should we stop balancing while running the program that takes care of the changes?
[09:26:31] <Striki> the thing is, we started out sharding the collection without hashing. then we changed it to be hashed but we haven't changed the "leftovers"
[09:26:49] <Striki> now we need to change the leftovers because we are experiencing hot shard because of this mess
[09:27:03] <Striki> even though "only" 100-200 million out of 530 million of those documents do not have a hashed shard key
[09:27:40] <Striki> just wondering if I should turn off the balancing while making these big changes
[09:28:11] <Striki> running that program will take some days as well and the system will be kept running as well. the collection is increasing by around 3-4 million documents a day IIRC
[10:05:59] <ahawkins> Hello everyone, I have an array of strings. How can I query documents where an array element matches that regex?
[11:14:55] <Rhaven> Hi, is there a way to add an arbiter in mms console? I tried this solution : https://jira.mongodb.org/browse/MMS-1158, but the mms agent throw an error : "ERROR NonBlockingMongoStatsThread-xxx:27016 Problem collecting non-blocking data: xxx:27016 is an arbiter"
[11:39:56] <muraliv> hi group, I'm trying to enable authentication in a replication setup. I've --keyFile enabled. after creating admin and normal user, restarting mongodb with --auth option, replication by itself is not happening.
[11:45:14] <muraliv> do I have to create users and set password in every mongodb instance, to enable authentication in a replication environment?
[11:57:25] <muraliv> _boot: yes, all the replicas have the same file infact, all the repl instances are running in my machine only.
[12:00:33] <muraliv> _boot: yes. before enabling auth, I logged in and created admin user in admin db and a normal user in my db. only after that I enabled --auth in command prompt
[12:01:26] <muraliv> no. not on all replicas. only on the primary I created both the users. because, if I create a user in primary, the collection and document is replicated in the secondary too.
[12:03:56] <muraliv> ohhh.. but is my approach correct? i.e. I've first created replicas. and then added users in primary.
[12:04:34] <muraliv> or should I create users individually in every instance, before replication. then restart mongodb with -auth and then enable replication?
[12:07:30] <muraliv> :) anyways, thanks a lot for your time. if I came to find it out, will update you . thanks.
[13:56:44] <OliverJAsh> i'm console logging my mongodb queries, however they are converted to JSON which means object IDs just become strings, which means i can’t copy the query straight into the mongoDB client/REPL for inspection.
[14:19:04] <OliverJAsh> error: { "$err" : "bad query: BadValue unknown operator: $oid", "code" : 16810
[15:04:31] <ryansb> Hey, I have a question about MMS in multi-tenant deployments. It says here http://bit.ly/1cuaKQV that a single mms agent should be able to access all of your mongodb nodes to monitor them
[15:06:04] <ryansb> In my case I want to have one mms agent per cluster (separate config servers, etc)
[15:11:05] <cheeser> isn't done by creating separate entities in mms and then attaching agents with those entity IDs?
[15:14:09] <ryansb> Do you mean creating an entirely separate group (e.g. myco.com1 and myco.com2) or within the group registering separate clusters?
[15:14:36] <cheeser> probably a separate group. i haven't used mms much it seems like that's the approach used.
[15:16:38] <ryansb> joannac: thanks. I see there's a ticket to change that https://jira.mongodb.org/browse/MMS-1551
[15:16:42] <odc> Hi there. Question: let's say I have a collection with plenty of small text fields and one big blob field. To get the best read performance, should I stay like this or store the big blobs in a separate collection or database?
[15:17:42] <joannac> are you going to access both the small fields and the large blob at the same time?
[15:26:31] <odc> Nodex, but even with projection, mongodb will first have to load the full document, and then only copy the selected fields in the response buffer
[15:38:05] <Nodex> odc, I don't understand what you're asking. Are you saying it's quicker to do 2 queries or somehting?
[15:39:13] <joannac> but if you have a small field and a large field, and the large field is rarely needed, you don't want to incur the cost of reading the large field in all the time
[15:39:53] <Nodex> I know it's not quicker, I was asking for confirmation of what he / she was asking :)
[15:41:19] <odc> yeah, i'm wondering if mongodb internally reads/loads the whole document, even if i only selected a few fields
[16:45:20] <dsdsf> hi all ! I have mongodb in my server with 3 replication. I delete by accident one collection (remove documents) from 1 replica. How can I know that there is backup in other replica? And how do I backup from another replica?
[16:45:56] <hydrozen> Hi there. I’m getting the following error on a query sometimes: https://gist.github.com/hydrozen/9370933. I do have an index defined on the updated_at field. Any ideas what I can try to improve things?
[17:02:16] <joannac> I guess *in theory* you could go into your db files and look for BSON and see if you can recover anything?
[17:39:16] <sgfny> I have three collection A, B, C with multiple Cs having a field with DBRef to B and multiple Bs having a field with DBRef to A. Is there a way to do a mongo aggregate query on C using a reference to A?
[18:21:12] <joannac> mboman: do you run something sufficiently non-critical that you can afford to use glacier?
[18:22:33] <mboman> joannac, so it is more important that I can get the data back then getting it back quickly
[18:23:51] <mboman> joannac, to be fair, it will probably take me longer to replace the drives then suck the data back from Amazon...
[18:27:24] <kali> mboman: it's very simple actually. put your backups in an s3 bucket, and you setup a few rules with the aws console to get them to the glavier vault
[19:16:44] <nukulb> hey , quick question about Mongo, I have a product Schema, that has a details object. Each Detail is simply a name value pairs, like price: 25. size: 7. Would it better to store details as an array or simply objects. As in is this better http://pastebin.com/HKcRd8xg or http://pastebin.com/i2acpxBW. Remember that details are different between different products and not a standard list
[19:35:12] <mboman> How can I see how large (storage requirements MB/GB) a collection is? 'show dbs' tells me the whole DB, but not the individual collections....
[19:52:22] <nukulb> joannac: in my current testing so far that seems to be significantly slower than [ {'k': 'nukulb'}, {'type':'irc user'}]
[19:53:16] <nukulb> my writing test isn't great though, since I am writing the entire product, not individual items in the array
[20:16:21] <nukulb> and despite the index , its slower get product details with this style - {details: [{k:'name', v: 'nuklb'}, {k:'type', v:'irc user'}}
[21:08:13] <module000> Chepra: if you can take down your DB, you can dump/restore to reclaim the space
[21:09:09] <Chepra> I am not at 100% disk space :) I some gbs free. I just deleted a bunch of collections and want to free this space up :)
[21:11:52] <Chepra> the actual working set of the database is about 250mb, but the disk size is about 15gb :(
[21:27:11] <cheeser> check for external files and clear up some space enough to allow repairDatabase to run
[21:35:28] <nicholaswyoung> I'm designing a geoanalytics system on top of Mongo, and could use input from someone more knowledgable. Any insight on this thread would be greatly appreciated: https://groups.google.com/forum/#!topic/mongodb-user/rs4pgTS_bQI
[21:36:32] <nicholaswyoung> I've worked out how to use $addToSet, but incrementing the embedded counters... I'm still lost. Maybe I'm going about this entirely wrong, too. So rebuke I'm going astray!
[21:40:23] <michael_swe> nicholas: If you're going to enumerate all countries and regions your document will get really big. Why don't you create one document per counter?
[21:42:31] <nicholaswyoung> michael_swe: That's a very valid point. I'm already doing some document partitioning based on URL and day of the month, and creating collections by month number.
[21:43:38] <nicholaswyoung> michael_swe: I think with those partitions already in place, it will scale well enough, unless you see something else wrong. Essentially, in the current draft, there's a JSON structure (like I showed in the gist) for each 24-hour day.
[21:46:16] <nicholaswyoung> michael_swe: But then again, I'm no expert. What do you think? Am I relatively safe?
[21:46:57] <michael_swe> I'm new to noSQL (comming from SQL Server), so please take my words lightly :) I'm still wondering though.. how many countries/regions will be enumerated. You can't index that kind of structure.
[21:49:01] <nicholaswyoung> I'd say, at max, 150. That's not terrible, given the performance I've seen from Mongo thus far.
[21:49:31] <nicholaswyoung> The biggest issue is incrementing those counters, and so far, the documentation hasn't helped much.
[21:50:30] <nicholaswyoung> I've tried to comprehend $elemMatch, which I think I need to select the appropriate title="" key, but I don't understand it yet. I'm hoping someone else does!
[21:51:38] <nicholaswyoung> and given that I'll be pulling everything back one document at a time (since documents are already created one for each day), indexing shouldn't be too horrid.
[21:52:56] <michael_swe> You could loose one depth level by naming your keys "countrycode.region", i.e. "US.California". Wouldn't that make incrementing the counters easier?
[21:55:23] <nicholaswyoung> I did try that, but encountered some issues with international naming schemes. (I have a trial of that system running here: http://api.machine.fm/episodes/5313a16a3230350a20010000/accesses)
[21:56:56] <nicholaswyoung> I guess I should also mention that I'm not concerned with the complexity of queries needed to achieve the initial schema design, because I can handle 'em in a queue. This isn't a realtime system.
[22:03:13] <nukulb> nicholaswyoung: not to interrupt your conversation , but I am currently using $elemMatch, like this - http://docs.mongodb.org/manual/faq/indexes/
[22:03:37] <michael_swe> How many days will you save in one document? You can't go on forever (mind the 16MB limit). I'm sorry, but I have no clue of how to increase those counters in an effective way. Why don't you make a "days" collection and reference the book/media or whatever from each document? You'll have to make some bucketing anyway when more days goes by.
[22:04:35] <nicholaswyoung> nukulb: you're not interrupting at all. Let me take a peek at that link, because I don't think I've seen it before.
[22:05:01] <nicholaswyoung> michael_swe: It's only been up for a few days. That's the test environment, where I'm prototyping this idea.
[22:06:10] <michael_swe> One collection per month.. That's a new concept to me. I that legit in the noSQL world? Would be a no-no in SQL Server.
[22:07:03] <nukulb> nicholaswyoung: look under this heading "How can I effectively use indexes strategy for attribute lookups?"
[22:07:05] <nicholaswyoung> I don't know about the larger world of nontraditional databases, but it's totally cool in Mongo, since collections can be created on the fly without a real performance penalty.
[22:08:30] <nicholaswyoung> nukulb: This helps a lot. Is there any documentation on incrementing counters for the matched keys?
[22:09:05] <nicholaswyoung> sflint: Here's the schema I would like to have, on a per-document basis: https://gist.github.com/nicholaswyoung/475eeaa49800db1d2f75
[22:09:48] <nicholaswyoung> just total. Each day has a new document, and that's as specific as I get.
[22:10:53] <nicholaswyoung> That seems a little extreme. I plan on expiring the monthly generated collections, as I go.
[22:11:45] <sflint> right but if you don't use 1 DB for each month you will fragment and not get the space back
[22:13:16] <nicholaswyoung> FYI: the whole "collection per month" thing came from one of the MongoDB early adopters, John Nunemaker of http://gaug.es. That's who started me down this path.
[22:13:48] <nicholaswyoung> I tried that approach too, but the international naming for regions etc were a massive pain to include in keys of any kind.
[22:14:27] <nicholaswyoung> You get weird characters and stuff, like characters we don't see in the normal alphabet.
[22:15:06] <nicholaswyoung> Gotcha. Well, I'm going to do some more digging and see what I come up with. Thanks for the different perspective.
[22:15:15] <sflint> the collection per month is okay but the OS doesn't get the space back if you remove the collection
[22:15:29] <sflint> if you use a seperate db for the month when you drop it the OS gets the space back
[22:15:38] <nicholaswyoung> I wonder how everyone else has been getting away with the technique then.
[22:16:08] <nukulb> nicholaswyoung: I haven't seen any where you are incrementing counters for matched kets
[22:16:17] <sflint> or they don't take on a lot of data....i am used to having a lot of datat coming in
[22:37:27] <pomke> Would anyone know where I might be able to buy some mongo stickers for my laptop?
[22:38:47] <Joeskyyy> not sure the legality of redbubble's use here but you can buy some here apparently? http://www.redbubble.com/people/bape/works/11148916-mongodb-2?country_code=US&p=sticker&utm_campaign=shopping&utm_medium=google_products&utm_source=google&gclid=CJv105C6_LwCFYuXOgodJHsAIw
[22:41:29] <Joeskyyy> might actually come to mongo world actually, y'all sent me a tshirt already so I'm just begging :P
[22:42:36] <Joeskyyy> Nah, I work at Rackspace, so we host some mongo events with you guys sporadically with objectrocket and stuff
[23:07:59] <Fallout2man> I'm setting a date of Febuary 18th at 12:00 P.M. PST and getting ISODate("2014-03-18T19: 00: 00.0Z") when it should be ISODate("2014-03-18T4: 00: 00.0Z") what with PST being eight hours behind UTC
[23:30:58] <rafaelhbarros> should I run the app on the beanstalk and the db on 3 ec2? any suggestions?
[23:32:32] <MacWinner> hi, I have imported a document into a collection.. one of the fields is something like this {"timestamp" : "2013/08/26 17:36:22" } can I make mongo treat this as a normal datetime field?
[23:47:40] <Fallout2man> Using NodeJS and I'm getting problems with setting/retrieving dates to my mongo server. I set a date at for instance Feb 18th 12:00 P.M. PST in javascript. I record the date as a date object, I get ISODate("2014-03-18T19: 00: 00.0Z") when it should be ISODate("2014-03-18T4: 00: 00.0Z")
[23:47:58] <pasichnyk> hey guys, i'm working with aggregation framework and curious if i can insert a property value from the result documents into a proeprty name of the projection. Basically i'm getting information back grouped by users, and i want to build a document that has a schema like { date: {some date here}, summary.$user.total: $total, summary.$user.sales: $sales, ... }. is that possible, or am i always
[23:47:58] <pasichnyk> goign to end up with a document per "result" that comes back from the aggregation?
[23:48:44] <changerOfSea> hey all, got an aggregation problem here. in my $group clause I have 3 vars that I want to group by, they are all part of the same sub-object, an id, name, and slug. However, only the id is showing up, any ideas?