Log file Viewer
#mongodb logs for Tuesday the 5th of November, 2013
[02:06:58] <pzuraq> in general is it a good idea to normalize hasMany relationships in a collection? Seems like given the cost of reallocation
[02:10:12] <cheeser> if you're always going to need those relationships everytime you fetch the parent doc, it can make sense to nest them.
[02:10:48] <cheeser> if there are going to be a *ton* of them, it might make sense to break them out and fetch them separately.
[02:11:36] <cheeser> especially if you're going to only want a subset e.g. for pagination, a separate collection would be better
[02:12:50] <pzuraq> cheeser: So lets say I have Posts, and every post has many Comments. When listing all posts you can't see the comments, but when looking at any post you will see all comments w/o pagination, and there can theoretically be an limit of several thousand comments
[02:15:47] <pzuraq> so an order can have many products, but it's not likely to have them change much or have a huge number
[02:28:55] <pzuraq> cheeser: Would you do that even if you had places elsewhere in the app where you need the products on there own?
[04:53:59] <jkitchen> mun: I'm unsure. I haven't vetted any of those suggestions, was just trying to point you in the right direction
[06:02:07] <brucelee1> i have 3 clusters but why cant one cluster work by itself, and needs 2 clusters?
[06:18:38] <brucelee1> joannac: but it makes the app unusable though since the db cant be written to
[06:19:16] <brucelee1> joannac: so to have a 3 node setup, where 2 can fail and still have teh whole thing functioning, it would need to actually have 3 nodes, and an arbiter right?
[06:25:08] <joannac> You need a majority of votes. 2 votes out of 4 is not a majority. 3 votes out of 5 is a majority.
[06:28:16] <brucelee1> joannac: but in a 3 node cluster, it makes sense that if 2 fails, 1 is still there
[06:29:04] <brucelee1> joannac: infact this is what happens (it will fail after 2 nodes fail) in our 3 node setup
[06:29:04] <joannac> brucelee1: Correct. THe remaining node can't tell the difference between the other 2 nodes dying, and a network partition.
[06:29:30] <joannac> FOr all it knows, the other 2 nodes are running happily on the other side of the network partition.
[06:29:44] <brucelee1> joannac: for the purposes of high availability for application, whats the point of going 3 nodes, vs 2
[06:30:11] <joannac> If you have 2 and 1 dies, or there's a network partition, you will have no primary.
[06:33:04] <brucelee1> for a 3 node cluster, i presume everybody wants it set so it can support up to 2 failures (other they can just use 2 nodes and 1 arbiter)
[06:33:47] <brucelee1> if resources/money is no prob, ideally everyone would run a lot of real nodes, rather than arbiters
[06:34:12] <brucelee1> and with 2 nodes, and 1 arbiter, you get the same amount of high availabibility as 3 nodes
[06:35:03] <brucelee1> so i presume if someone goes from 2 nodes and 1 arbiter, they get the added redundancy by upgrading to 3 nodes
[06:35:38] <brucelee1> so with 3 nodes, would it ever note make sense to add in 2 mroe arbiters to make it able to withstand 2 node failures
[06:36:46] <joannac> (you can restore from backup but still need to sync from backup time to current time)
[06:36:54] <brucelee1> joannac: so i guess my final question is, any reason not to add 2 more arbiters to make a 3 node able to withstand 2 nodes down
[06:37:24] <hkonkhon> sorry to be such a newbie, but what's the etiquette for asking a new question? do I just type it here?
[06:37:46] <jkitchen> hkonkhon: yup, ask away. don't paste to the channel, use a pastebin type service
[06:37:56] <brucelee1> joannac: losing 2, have 1 left, the other 2 needs to be completely sync'd up AND take normal loads when it comes up
[06:38:04] <joannac> you need to repair 2 data nodes from 1 remaining data node. hope the last one doesn't die. keep up with app load.
[06:39:19] <brucelee1> if theres 2 arbiters and 3 nodes, theres 5 voters altogether, if one dies, theres 4 voters, unable to make even vote right?
[06:39:41] <joannac> brucelee1: like you said, sometimes you don't have the resources to run as many nodes as you want. You can make the decision about what kind of risks you're willing to take
[06:47:18] <joannac> brucelee1: I mean, this is all very worst-case. But High Availability is pretty worst-case thinking anyway :)
[06:50:35] <jblack> It's all single point of failure anyway. one meteor can take out the whole planet
[07:10:23] <rahulkmr> What happens when I add a mongodb node as a replica to another node? I understand it ships the oplog but will the initial sync wipe out the data on the just added replica. I have a primary containing a db say Foo which I want to be synced with the secondary. Secondary already has another db(Bar) and I don't want Bar to be wiped out. I just want Foo to be synced to the secondary so that I can run both Foo and Bar from the secondary
[07:28:33] <brucelee1> how would i go about upgrading monogdb schema without affecting application in a 3 node set up?
[07:36:56] <k_sze[work]> I don't quite understand how querying works in MongoDB. I can't return only parts of a document or querying only a part of a document?
[07:41:23] <k_sze[work]> Can I extract only the education objects that are of postgraduate-level, for example?
[07:50:08] <GeertJohan> Question, is GridFS often being used with filenames like "/profilepics/<username>/firstUpload.png" ?
[07:50:44] <GeertJohan> k_sze[work]: yes, http://docs.mongodb.org/manual/reference/method/db.collection.find/#query-subdocuments
[08:47:06] <st0ne2thedge> How full must a collections allocated file become before it gets a new allocation? ^^
[08:47:46] <Derick> The moment there is a write in filename.n, the file with filename.(n+1) gets created
[08:55:44] <st0ne2thedge> Derick: Right, so Looking at the .stats().size and comparing it to .stats().storageSize isn't going to give me an idea when the mongodb will grow in size?
[09:25:58] <st0ne2thedge> Derick: How would you advice going about it then? Right now I'm looping through every collection in every database and asking for die size variables
[09:36:51] <k_sze[work]> Suppose I have a rich_people collection, each document in rich_people has an array of cars (which are subdocuments in MongoDB parlance, I suppose). Would it be possible to use MongoDB API to first get Bill Gates out of the rich_people collection, and then return the blue cars that Bill Gates owns?
[09:38:11] <st0ne2thedge> Derick: I'd like to be able to calculate how full the allocated fysical file is with actual data. That way I'd be able to guess when a new allocation will occur, making me able to guess when my partition will be full ^^
[09:39:07] <st0ne2thedge> Derick: What I had understood from reading some documentation I was convinced looking into the stats() of the collections would do the trick :P I appear to have been wrong ^^
[09:39:38] <Derick> it's difficult to guess file allocation as fragmentation of data files plays a big role in that
[09:40:15] <Derick> if you're not in a really important production environment, then you can start mongodb without preallocation
[09:42:08] <NodeX> k_sze[work] : you've said that your rich_people document has a sub document so you already have the list of cars he owns
[09:48:39] <Derick> you tend to always get the whole document back (with or without some top-level fields)
[09:49:15] <NodeX> k_sze[work] : a document is more like a Row in a traditional RDBMS, you seem to be treating it more like a table
[09:50:55] <tilya> good afternoon. is this the right place to ask questions about mongodb administration?
[09:51:25] <k_sze[work]> I thought the whole idea about schemaless document-based database is that each document would be self contained.
[09:52:04] <NodeX> k_sze[work] : each doc is self contained - however your pastebin describes ONE document
[09:53:30] <k_sze[work]> NodeX: but by taking the cars out of Bill Gates, and flattening it, I would need to query multiple collections to reconstruct a profile of Bill Gates.
[09:54:10] <tilya> i have a problem since 2.4.6. i have three nodes in replicaset, primary, secondary and arbiter, with authentication enabled. for the last few versions unfortunately, there's a problem with authentication on the arbiter node. is there a way to solve that, or is it on purpose?
[09:55:32] <Derick> k_sze[work]: yes, a whole profile belongs together, but if a query shows that it's not efficient, change it.
[09:56:20] <Derick> NodeX: but yes, I do see a document level too much in there. Perhaps just badly pastebinned though... as the terminology that k_sze[work] uses in it is right.
[10:00:18] <Derick> k_sze[work]: let that be a lesson for you too, don't wrap it in an extra layer in examples :P
[10:00:23] <tilya> Derick: it seemed as a good idea for monitoring reachability over network, back in times it worked :)
[10:01:19] <NodeX> you can use positional operator to return parts of the array you're interested in
[10:01:53] <tilya> Derick: it somehow worked until 2.4.6. but well, if that's intended and not a bug, then i will dump this dumb monitoring and do something else. thanks.
[10:12:56] <st0ne2thedge> Derick: Can one run an existing database without preallocation? Is there any danger to existing data? We are currently storing our packaging into the database (pulpproject)
[10:18:12] <st0ne2thedge> Derick: you said you'd advice thresholding the mongodb's partition at >2G remaining space right? ^^
[11:01:51] <robertjpayne> the "auth" setting isn't affected in any way by latency or network conditions is it? I'm struggling getting it working on a remote mongo but works just fine in a local vagrant mongo
[11:09:25] <robertjpayne> Gah mongodb host has to be an IP address always not a domain name it seems?
[11:10:22] <robertjpayne> yea hmm now trying to use my vagrant to connect to the one on the server it only works with IP but may be because of network briding and sorts
[11:11:56] <rickogden> does anyone know of any MongoDB-as-a-service providers they recommend for open source projects?
[11:12:41] <robertjpayne> rickogden: MongoHQ/MongoLab are the two big ones -- not sure they give any discounts for OSS
[11:14:09] <robertjpayne> rickogden: I'm sure there are others too but those are the two big ones I know of
[12:16:10] <HashMap> Hi there people.. I am just curious, I am doing the MongoDB for DBAs course and currently there is a week about sharding.. And my question is, for example there are three config servers.. does this usually in production means three separate physical machines?
[12:18:44] <robertjpayne> HashMap: Sharding normally will happen across separate physical machines to gain more durability in the event of hardware failure.
[12:19:11] <robertjpayne> HashMap: That's not to say it's possible to run the shards in LXC or similar virtualized containers
[12:20:59] <st0ne2thedge> I'm reading up on mongodb writes in mongodb v2.0, where as soon as a write has been buffered in the outgoing socket buffer of the client host, the insert is 'completed'. Is this fixed in later releases/
[13:16:48] <st0ne2thedge> kali: Im reading through the release notes of 2.2 but having trouble finding anything about WriteConcern though
[13:57:09] <alFReD-NSH> Any one know here what might cause "Assertion: 10307:Client Error: bad object in message: invalid bson type" on node js driver?
[17:48:52] <JEisen> Using collMod() to set usePowerOf2Sizes is a non-blocking operation on a primary, right?
[19:49:16] <astropirate> Tue Nov 5 12:46:42.358 [conn39] query myDB.rankings query: { $query: { poll: ObjectId('51daef3076135c762d030238') }, orderby: { rank.weighted: 1 } } ntoreturn:100 ntoskip:0 nscanned:2781 scanAndOrder:1 keyUpdates:0 locks(micros) r:8222 nreturned:100 reslen:23671 8ms
[19:49:54] <astropirate> Tue Nov 5 12:46:48.771 [conn30] query myDB.rankings query: { $query: { poll: ObjectId('51daef3076135c762d030238') }, orderby: { rank.weighted: 1 } } ntoreturn:100 ntoskip:0 nscanned:0 keyUpdates:0 locks(micros) r:86 nreturned:0 reslen:20 0ms
[20:02:07] <JEisen> Lock percentage going up and if your working memory set is close to physical mem are your two biggest indicators.
[20:02:52] <tystr> ya we had some issues the other week with 100s of page faults and disk io maxing out
[20:03:26] <tystr> but the servers weren't configured correctly with the suggested optimizations (ulimit, readahead, etc)
[20:03:49] <JEisen> you'll want to be really careful about whether sharding makes sense for your app in general. there are some major caveats.
[20:04:28] <JEisen> I would recommend upgrading the servers before sharding if that's an option, unless it really makes sense.
[20:04:32] <tystr> I'm just trying to understand if the issues we had were soley related to the misconfiguration, or if we're beginning to reach a bottleneck
[20:07:44] <astropirate> Anyone know what is going on with my problem? I couldn't drop the index, it says db is currupt. I did a db repair, again it worked for the first query and then returns 0 values for subsequent queries
[20:09:40] <joannac> astropirate: you run the exactsame query twice in a row, and the first time you get results, the second you don't?
[20:12:02] <joannac> astropirate: If you really did nothing in between... i dunno, let me think about it.
[20:15:46] <JEisen> tystr: Depending on your usage patterns, yeah, sharding might be valuable. But I'm definitely not saying "Go shard."
[20:21:58] <J-Gonzalez> these events have options for purchasing tickets. When a user purchases tickets, an order gets created in an order collection
[20:22:28] <J-Gonzalez> the one thing I can't seem to decide on is where a TICKET should be create (think like a hard ticket when you go to sporting events or concerts)
[20:22:49] <J-Gonzalez> Should tickets be it's own collection that has a reference to the order and event
[20:23:15] <JEisen> how often are you going to be querying on just the tickets vs. as part of the order,for example?
[20:23:56] <J-Gonzalez> The tickets get queried semi-often, during event checkins at different venues
[20:24:26] <JEisen> you'll need to weigh the disadvantage of having multiple round-trip queries vs. having to transfer/sort through a lot of unrelated data.
[20:25:35] <JEisen> but, say, if when you get the ticket you also want to know the order that generated it and so would use that data anyway… that might not be so bad.
[20:26:38] <J-Gonzalez> The orders don't necessarily get queried as often, and are more for accounting purposes, say a user wants to see all their past orders
[20:28:54] <J-Gonzalez> yea the orders are pretty separate - basically, we just need the easiest way to pull up tickets when customers bring them in. And either having them as subdocs within the event that is being checked in, or querying the entire tickets collection
[20:29:59] <JEisen> my perspective is, don't use subdocs unless you automatically want that data when you get the parent data. but that may be hardline.
[20:33:13] <JEisen> anyone happen to know if collMod() is blocking or not? (looking at turning on usePowerOf2Sizes)
[20:39:25] <vicTROLA> Does anyone have any advice on best practices for paging through large datasets and 'maintaining state' (for lack of a better phrase)? My app is supposed to take the user through potentially thousands of individual documents. I'm unsure how to preserve offset state in the next request to avoid showing duplicates
[20:40:00] <vicTROLA> I'm thinking about storing limit skip offsets in cache and modifying them on every request. Is there a better way?
[21:19:28] <flatr0ze> I'm storing binary strings in mongo (both files and encrypted text), and getting console.log() output of "\u0012\u0032..." mixed with glyphs and symbols in the output. Can it be that it's just the output, and I'm not really saving it as UTF-16? I'm really trying to save space and store only bytes, no Unicodes... Using Node.js w/ MeteorJS framework, GNU/Linux.
[21:22:07] <cirwin> flatr0ze: unfortunately that's not going to work very reliably, I don't think the node mongo driver has good support for buffers
[21:22:41] <cirwin> you should also be careful because not all byte strings are technically valid unicode, but as far as I'm aware neither mongo nor node enforce that strictly
[21:23:45] <flatr0ze> cirwin: you think there's any way to store just byte array? like in the ol' good times? I'm getting pretty sick of having utf where I don't need it at all
[21:25:36] <cirwin> flatr0ze: I think there's a way of putting raw data into BSON, but I've not tried super-hard
[21:37:04] <ramsey> Derick: what's the alternative to using setSlaveOkay() in the PHP Mongo driver? We upgraded and now we're getting deprecation notices.
[21:42:32] <ramsey> Derick: thanks. I think our problem is that we don't understand what the read preference should be :-)
[22:33:45] <tab1293> I'm trying to insert an integer value of 5368709120 but whenever I try to insert it into a field it stores in the db as 1073741823
[22:40:27] <tab1293> its weird cause if I insert the number with mongo shell it works, but with php it doesn't
[22:52:21] <tab1293> and also could this be the issue http://php.net/manual/en/mongo.configuration.php the native long variable
[23:30:20] <bjori> tab1293: in your php.ini file, or mongo.ini file if your distro splits extensions like that
[23:30:53] <bjori> tab1293: it'll tell you which php.ini is loaded, and if other ini files are loaded. if you see a "mongo.ini" then use that, otherwise just add it to the main php.ini
[23:54:16] <tab1293> bjori: Do you know anything in regards to when summer intern applicants should hear back from you guys? I just put in an application last week