Log file Viewer
#mongodb logs for Tuesday the 23rd of December, 2014
[03:33:40] <Boomtime> a sharded cluster is usually formed from multiple replica-sets.. that is, each 'shard' is expected to be a replica-set
[03:34:46] <Boomtime> the usual upgrade path is to expand from a single replica-set to a sharded cluster, by introducing a new replica-set to be your second shard
[06:29:45] <hemangpatel> joannac, I have this record in mongodb, https://gist.github.com/Chintanvpatel/22b493c04ba2430cc131
[06:43:21] <Boomtime> ignoring the question of why you are using a database for a single rule engine, your example is faulty: "/web/*" does not match "domain.com/web/index.html"
[06:48:39] <Boomtime> you are not matching something that exists in mongodb, you are asking mongodb to do a fuzzy-search and find a "best fit" based on knowledge it does not have
[06:50:11] <Boomtime> if the URL "/domain.com/web/index.html" existed in mongodb then you could query for the pattern "/web" but you want to do the reverse
[09:02:33] <borjagvo> Hi. I never found answer for this question: http://stackoverflow.com/questions/27381041/how-does-mongodb-text-search-work (see last comment). Does anyone know exactly how this works? Thanks!
[09:42:11] <BlackPanx> so i just thought of adding another member and then remove the arbiter like i would remove any member
[09:42:32] <BlackPanx> or is this wrong, and i should remove the arbiter first and add a member afterward
[10:00:22] <BlackPanx> i think asking a question will get you an answer faster than asking if anyone is here...
[10:10:56] <rcombs> don't know any off the top of my head, just that they exist and can be found in 5 seconds of googling
[10:15:38] <amitprakash> Whats wrong with this statement ?mongodump -h mongo-04.delhivery.com -c bags -d delhivery_db -q '{cd: {$gte: ISODate("2014-12-04")}, "cs.ss": "Complete"}'
[10:33:44] <BlackPanx> is moving from arbiter to full member replica set best to remove arbiter and then add member, or is it better to add member and then remove arbiter?
[11:02:13] <neo9_> Can any body please suggest in what way we can set the replica to the mongodb to only partial db.
[11:12:13] <Neo9> chetandhembre: we are not yet set any configuration for replica. we have to do that. (right now their is no slave).
[11:14:04] <Neo9> chetandhembre: even thier is no sharding cluset. we have a mongo server with a simple architecture.
[11:15:29] <Neo9> chetandhembre: mongo server that simpley stores & server the clients. their is no certain architecture or method we followed for the mongo db
[11:16:26] <Neo9> chetandhembre: initially we have a requirement of quick setup of mongo, so thier is no specific architectures or deployment methods for the existing system.
[11:17:38] <Neo9> chetandhembre: http://www.codeproject.com/KB/install/plasticdbbackend/firebirdserverdeployment.png
[11:28:26] <Neo9> chetandhembre: ok. thanks for spending time for me. And please share me your suggetions if you have any...
[13:57:10] <skunkNode> Hello all... I have MongoDB installed and running as a service on CentOS7. I have added a couple of users to the admin database with roles adminAnyDatabase and userAnyDatabase. I then created a database called gdex and added the needed admin user for that database to it. However, now I keep getting auth failed... Is there a way to automate the user process similar to MariaDB/MySQL wherer you Create 'Usere'@'%' Identified by 'pwd'; Grant
[13:58:18] <skunkNode> I am trying to access the database using the following uri: mongodb://user:password@mongodb.hostname.net:27101/gdex and keep getting auth failed.
[14:02:33] <iancrowther> my mongoose schema's custom validator is being called twice and i cant work out why, can anyone explain what might be going on?
[16:13:54] <dyl> Or why it outputs noisy to the command line by default, despite being a data dumping tool.
[16:16:35] <skunkNode> dyl: Good questions. I can't authenticate into a db using a URI at all. :/ I'm stumped.
[16:21:34] <skunkNode> Question: Everytime I try to login as a particular user from the mongodb's server such as " mongo -u dexsa --authenticationDatabase admin -p" I get prompted for the password, but then get authentication failed.
[16:22:11] <skunkNode> I can get into the mongo shell by just typing 'mongo' but I can't authenticate as the particular sa user I am trying to use.
[16:54:27] <skunkNode> Anyone here using mongodb on a centos7 server? I'm trying to figure out how to make it start as a service, generally I would use systemctl enable mongod, and then systemctl start mongod, but that doesn't work. is chkconfig --levels 235 mongod on good enough?
[17:16:38] <toter> Hi everybody... Using PHP, I'm trying to insert 91 documents into a collection, each with a unique _id, but the way I'm doing is all 91 documents are being inserted into an object id, like this: http://bit.ly/1t7EpYh
[17:18:09] <kali> toter: show the code, somebody who know php and the php driver should be able to help
[17:20:55] <brianseeders> looks like you're inserting it as one big document instead of multiple documents
[17:23:09] <brianseeders> or possibly just foreach($json as $document){ $collection->insert($document, array('w' => true)); }
[18:31:36] <Alittlemurkling> Hello #mongodb. Is there a way to make geospatial queries return all matched documents, instead of just the first 100?
[18:43:51] <vacho> anyone using mongohub? I am trying to connect but having trouble. What should I put as bing address?
[19:21:31] <vacho> I have a collection called shows..how would I go about adding another season to this collection? http://pastie.org/9796350\
[20:12:32] <proteneer> for fucks sake mongodb really needs to support multiple document transactions
[20:14:01] <proteneer> is there a compromise between SQL query shittyness but nice transactions and NoSQL query awesomeness but no transactions?
[21:26:56] <asturel_> hi, im trying to make my collection capped but i get "errmsg" : "cloneCollectionAsCapped failed: { errmsg: \"exception: can't map file memory - mongo requires 64 bit build for larger datasets\", code: 10084, ok: 0.0 }"
[21:42:46] <GothAlice> hicker: Effectively, yes. Run the query portion of your update as a find() to see which records will be modified.
[21:43:36] <GothAlice> asturel_: You're not running a 64-bit mongod, which is highly not recommended.
[21:45:15] <GothAlice> If you gracefully shut down the host, you can mongodump straight from the on-disk files.
[21:46:39] <hicker> If I first create a child doc that needs to be referenced by a parent doc, then try to create the parent doc which fails, now I have an unreferenced child doc.
[21:47:54] <hicker> Is there any way around that? I guess I could delete the child if the parent fails.
[21:48:44] <GothAlice> hicker: MongoDB isn't relational and does not offer server-side triggers. Some client drivers do offer simulation of triggers (a la MongoEngine's signal support) but this is generally a bad way to do things, depending on your exact data model requirements.
[21:49:03] <GothAlice> For an overview of the patterns involved: http://www.javaworld.com/article/2088406/enterprise-java/how-to-screw-up-your-mongodb-schema-design.html
[21:49:51] <GothAlice> The pattern I tend to use for such parent/child data is direct embedding. I.e. the replies to a forum thread are embedded within an array of the thread itself.
[21:50:27] <asturel_> is it normal that the mongodump file is 1/2 large comparing to the dbpath files?
[21:50:49] <GothAlice> asturel_: mongodump files are "compacted" vs. the potentially sparse on-disk stripes used during normal operation.
[21:51:24] <GothAlice> Note that the 32-bit limitation applies to the on-disk stripe sizes, not actual allocated data sizes AFIK.
[21:51:49] <GothAlice> (mongo needs to memory-map all of the stripes, but will fail when they exceed 2GB total on 32-bit platforms)
[21:55:04] <vacho> http://pastie.org/9796350 can someone help me write an update that will add another episode?
[21:55:13] <hicker> So let's say I want to create a new Store document with a Manager... I'm trying to create both in one call... but maybe I could easily solve my problem by making them separate calls?
[21:56:39] <hicker> Because a Manager's going to be referenced by multiple schemas... I'm not sure I want to embed it in Store
[21:57:20] <GothAlice> hicker: In your case, those would appear to be different collections of data. And yeah, you'd have to make those separate calls.
[21:57:44] <GothAlice> In my forums example, I still keep the actual threads "within" a forum separate from the forum document itself.
[21:58:20] <GothAlice> vacho: You're looking for $push: http://docs.mongodb.org/v2.6/reference/operator/update/push/
[21:59:49] <vacho> GothAlice: you are awesome thanks so much! i will look into that.. but how would I update a nested document?
[22:00:29] <GothAlice> With a tiny bit more difficulty and the $elemMatch search operator: http://docs.mongodb.org/v2.6/reference/operator/projection/elemMatch/
[22:01:22] <GothAlice> Basically, search for the season you are interested in, then use the $ (positional match element) a la http://docs.mongodb.org/v2.6/reference/operator/update/positional/
[22:01:43] <GothAlice> Well, you need to find the nested record you want to update first. Then you can append something to that nested structure.
[22:02:12] <GothAlice> I typically don't nest things more than one level deep, though, since it becomes increasingly hard (and eventually impossible) to formulate sane updates. :)
[22:02:32] <vacho> GothAlice: I am not being lazy or anything, but if I actually saw an example for my structure, I can figure out the rest. Can u help me just write the part of updating a episode? I can provide what I have so far
[22:03:44] <GothAlice> That's not just a bad idea, that won't be functional in the typical sense of being query-able, update-able, etc. $elemMatch only works on a single array in a query, and you can only have one of them.
[22:05:12] <vacho> GothAlice: I am pretty new to this.. and I liked the fact of nested data, since it goes well with my data e.g. shows->seasons->episodes->talents
[22:05:17] <GothAlice> Yes. There's some pivoting you should probably do to your data. Have a document per show containing the season meta-data for all seasons. Then have a separate collection of episodes embedding the talent for each episode. That way each document (you now have two) only has one level deep of nesting.
[22:05:48] <vacho> GothAlice: but then I have a relational database I feel..might as well use MySQL?
[22:06:08] <GothAlice> Generally you should ask yourself: if I load up a show, do I always care about every single talent? Every single episode?
[22:06:29] <vacho> GothAlice: if I load up a show, I care about every single episode in the last 2 seaons :)
[22:06:30] <GothAlice> In the forums example, yes, when you load up a thread generally you always want the replies at the same time. When you load up a forum, you don't want all of the replies.
[22:07:25] <GothAlice> Using my suggested split you'd load up the show, identify the last two seasons ($slice projection can make this efficient), then perform one additional query to find the episodes for those two seasons.
[22:08:04] <GothAlice> Where relational databases encourage construction of giant monolithic join queries to get "everything back all at once in the way you want it", MongoDB encourages lighter-weight querying, thus slightly more querying than one might be used to.
[22:08:40] <GothAlice> Having the episodes loaded in a second query will also make it much easier to paginate, if you desire that. :)
[22:08:41] <vacho> GothAlice: I see.. I don't care that much about efficiency, this tool im buiding will only be used for 2-3 users concurrently .. and the data set is not that big.. maybe 50-100mb at most any given time
[22:09:54] <vacho> GothAlice: query a document, get the ID's ... and then use the ID's to query another document. I feel that's a join
[22:10:33] <GothAlice> It's the difference between "give me every season for this show" and "give me the last two seasons for this show". db.shows.find({name: "Dr. Who"}, {seasons: {$slice: -2}})
[22:11:46] <vacho> GothAlice: in your example, the show and seasons and episodes are all in the same document right?
[22:12:37] <vacho> GothAlice: I wish I could PM with you a bunch of questions.. I am so new to this.
[22:12:39] <GothAlice> They key being that if you perform top-level array projection, you're screwed if you want to perform projection on the items nested below it.
[22:12:52] <GothAlice> Well, this channel is dedicated to MongoDB, so feel free to ask anything MongoDB-related here. :)
[22:13:40] <vacho> how many documents would you create for my data set? I want to store shows, seasons and episodes and meta data related to each layer
[22:14:23] <GothAlice> I suspect this will give the broadest capability for asking questions of the data.
[22:14:52] <vacho> so each show would be a document, e.g... I would have "Big Bang Theory", "Gossip Girls", "Users", etc ... ?
[22:15:46] <vacho> GothAlice: that would be give all documents, e.g. "User Action Log", "Users", etc
[22:16:16] <GothAlice> If you only cared about the ID and name of each show, you'd do this instead: db.shows.find({}, {name: 1})
[22:16:28] <vacho> GothAlice: also I don't think a document can have special characters like ", ! *"
[22:16:47] <GothAlice> Don't store different types of data in the same collection… that way lies madness.
[22:17:06] <GothAlice> Have a collection for users, a collection for the log, a collection for shows, and a collection for episodes.
[22:17:20] <vacho> GothAlice: one collection for everything APP related..and one collection for my Shows
[22:18:17] <GothAlice> While you *can* store multiple models of data in one collection, this is typically only sane when your data describes a "kind-of" relationship. I.e. if you have two different types of users: user and staff. Staff is just a "kind-of" user, potentially adding data specific to staff and dropping data that doesn't apply, but it's still a "kind-of" user.
[22:18:59] <vacho> GothAlice: so to confirm, I would have a single document for each show, and then a document for all episodes?
[22:20:06] <vacho> GothAlice: do you mind must pseudo writing how a episode document would look? I want to store episode number, title and air date e.g.
[22:20:09] <GothAlice> E.g.: show = {name: "Adventure Time", seasons: [{season: 1, episodes: 27, aired: 2010}]}
[22:21:08] <GothAlice> episode1 = {show: ObjectId(ID of Adventure Time), season: 1, episode: 1, title: "Slumber Party Panic", aired: Date("April 5, 2010"), …}
[22:23:43] <vacho> GothAlice: what's the benefit of doing this is mongo instead of SQL besides efficiency, I can run a query to get this data in MySQL in a few MS
[22:24:28] <vacho> GothAlice: yes I understand, but I wanted to go the mongo route just because of the nesting, I felt my dataset was perfect for nesting.. but I feel it's not nested now so I might as well use MySQL
[22:24:52] <vacho> GothAlice: what's the point of nested documents if it's only good for a single layer.. ? I feel it has no strong purpose
[22:25:12] <GothAlice> You mentioned having nested data under the episode, though. I.e. {season: 1, episode: 1, title: "…", crew: [{role: "director", name: "Bob Dole"}, …]}
[22:25:41] <vacho> GothAlice: I don't need talents.. I just added it there to make it future proof.. at this time,, talents are not needed
[22:27:34] <GothAlice> The "why use this over something else" question is unfortunately broad. I could have written my forums using a typical relational database approach: table of forums, table of threads, table of replies. Rendering a given page of replies, though, would still require SELECTing the forum, the thread (title, other metadata), and finally a query of the replies limited to the current page. (Three queries, maybe two or one depending on how much of
[22:28:05] <GothAlice> In MongoDB, the same thing is exactly two queries: load the forum title and then load the thread record, $slicing the replies to the current page.
[22:28:37] <GothAlice> It's still semi-relational, of course. Forums have threads. Can't escape that. :)
[22:29:04] <vacho> GothAlice: I was really excited about mongo because of it's nesting.. but it seems it's not as I thought.
[22:29:42] <GothAlice> Nesting can be useful, but like anything useful, it should be used in moderation and with consideration for the side-effects.
[22:30:09] <vacho> GothAlice: so even a two level nest is not recommended? e.g. a document for a single show that contains all seasons and all episodes
[22:30:44] <GothAlice> You could do that, but you'll end up with (to aid querying) what appears to be data duplication. I.e. the episodes will still need to include their season number, even if they're embedded in their season.
[22:31:42] <GothAlice> Yes, but if you're looking for a specific episode at some point, you can't easily query across nesting levels.
[22:32:34] <vacho> GothAlice: no way of saying, give me all episodes of season 3 and filter out episode 3?
[22:33:02] <GothAlice> vacho: Everything is limited. People just have different tolerance levels for different forms of limitation.
[22:34:00] <GothAlice> Take this example: "find me the show that has a second-season episode entitled Mystery Train". db.shows.find({'season.season': 2, 'season.episode.title': "Mystery Train"}) < this will find any show that has a second season, and has an episode (in any season) entitled Mystery Train. This is not the answer to this query.
[22:35:07] <GothAlice> db.shows.find({'season.episode': {$elemMatch: {season: 2, title: "Mystery Train"}}})
[22:35:26] <GothAlice> ^ This will work as intended, but the season number needs to be bundled with the episode.
[22:36:49] <GothAlice> (For example, I could have fit the entire legacy phpBB forums in one MongoDB record if I so chose—all 10K threads or so.)
[22:39:18] <GothAlice> Consider how much space 16MB is. BSON has some overhead (I'll go with a conservative 32 bytes for a record containing nothing but a giant string). Assuming an average of 5.5 letters per word (including the space) this would then allow: (16*1024*1024-32)/5.5 = 3.05 million words.
[22:41:00] <bmillham> vacho: I'm also new to MongoDB, and the more I play with it, the more I'm liking it.
[22:41:40] <bmillham> Live a forgot a field in my original document design, so I just added the new field to my model. No mucking around with the database.
[22:42:03] <vacho> I was hoping I could do something similar to showname.season5.episode4 and just get what I want :)
[22:42:09] <GothAlice> For my own projects, MongoDB replaced a multitude of other systems, allowing me to focus on scaling only two things: MongoDB and my application. It replaced: ZeroMQ, RabbitMQ, Redis, Memcache/Membase, and Postgres on one project alone. (MongoDB capped collections = queue replacement, TTL indexes = automatic cache expiry, etc.)
[22:42:59] <GothAlice> vacho: While you could use fields to represent these things, the side-effect is that you basically can't use indexes.
[22:43:18] <GothAlice> It really seems that you had a preconceived notion of MongoDB that turned out to not align with reality. :/
[22:43:32] <vacho> GothAlice: I don't care about indices, my dataset is so small... it will be at most 20-30mb
[22:44:13] <GothAlice> vacho: If one truly cares not about anything, one might recommend SQLite. ;^)
[22:44:46] <Boomtime> vacho: for what you've described you could just be using a single XML file and then locate your stuff by XPATH
[22:45:34] <Boomtime> a database provides a way of finding / collating /updating data based on multiple criteria - you don't want to do any of that
[22:46:01] <GothAlice> The question then becomes: "why use a database", not "which database should I use".
[22:48:31] <vacho> Boomtime: yes true, but I want to make some updates and track stuff related to each season and episode.
[22:52:53] <Boomtime> if i were you i'd just start with a single document per episode, you can filter whatever you don't normally need - a show is just every episode, and a season is just a slightly tighter filter of that
[22:53:06] <vacho> GothAlice: last question, what are the drawbacks of having a document for each show with nested easons and episodes.. what is there I cannot do with that structure that will limit my application?
[22:53:31] <GothAlice> vacho: Above I gave an exact query that is negatively impacted by that type of nesting.
[22:54:19] <Boomtime> aggregation could do it but that makes most of your simple queries aggregations
[22:54:29] <GothAlice> Deep nesting = basically destroying your ability to query the data in a sane way.
[22:56:07] <GothAlice> "Show has a second season" and "show has any episode entitled "Mystery Train" — returns a show. Not even the show you're looking for, potentially, as those two criteria are wholly unrelated to each-other in that example.
[22:56:08] <Boomtime> just because your query was for a specific aspect of that show (that it has an episode called "Mystery Train") was merely a match criteria
[22:57:13] <Boomtime> i can't imagine too many shows have a second season episode named "Mystery Train" but the result is still the entire show, all episodes etc
[22:57:54] <Boomtime> you can use a selection filter to cut down to just returning what you want, probably.. but that will only take you so far
[22:58:33] <GothAlice> Boomtime: I know of three shows that would match that bad example: Adventure Time, Blossom, and Ghost Train. Only Adventure Time has a literal second-season episode entitled that, the others have two seasons and an episode entitled that; very different.
[23:00:28] <vacho> GothAlice: I really thought mongo had a similar query system to xpath. It would make total sense to have it.
[23:02:22] <relgar> equivalent to XPath for the JSON world. Quite nice, good libraries for the major languages
[23:03:21] <vacho> GothAlice: I read about how you can nest documents, and you would think if a DB technology supports nesting, it would also allow for easy quering a nested document.. :)
[23:03:39] <relgar> if you wanted some kinds of standardized syntax. of course mongo doesn't support it, but could do the bulk filtering in mongo, and finer filtering in the app server i guess
[23:05:50] <vacho> GothAlice: if I create a document for each show.. what would the document be called for show "$!#%t my dad says"
[23:07:11] <GothAlice> "called" — it wouldn't be "called" anything, it'd contain a field that marks the title.