Log file Viewer
#mongodb logs for Monday the 8th of June, 2015
[00:35:46] <maumercado> kind of like a findModify but if it finds and object, then it should not modify nor insert
[01:39:59] <luminux> Hi, I’m new to version 3. Is it still necessary to shard a database to allow better write performance on multicore systems?
[06:35:42] <roelof> If I have this code (http://lpaste.net/134076 ) why do the last 6 entries not get sorted ?
[10:14:11] <donguston> I have thousands of JSON files that I have exported from a websites API and I want to turn them into a database that I can query to output stuff in the format that I require. Some of them have syntax errors. Is there a way for MongoDB to read them all? Is there anyhthing I can use to attempt to automate the syntax fixes
[12:31:39] <DennisGer1> hello is it possible to perform a query projection on a specific Json array index ? I tried this without any success
[13:08:22] <StephenLynx> but DennisGer1 if you are hard hard coding indexes of arrays, probably something is wrong.
[13:10:02] <StephenLynx> I can understand when its something like I had to face recently, with generated HTML pages and forms, where your options are limited. but as soon as that 'something-id' hit the application, you should store each part in their proper fields.
[13:18:30] <DennisGer1> not working as expected, Not possible to query projection like this : arrayname.subarrayname[index]
[13:26:30] <DennisGer1> mongodb is not suitable for deeply structured json array ? component has a component which has another component and so on
[13:30:44] <StephenLynx> if you are using hard coded indexes of arrays, you could use objects with keys.
[13:35:28] <DennisGer1> for example : server_monitoring [server 1 [mainboard [ graphicard, cpu, harddisk] , server 2 [mainboard [...] , server 3
[13:36:49] <DennisGer1> then how can I releate this graphiccard to the specific mainboard which belongs to the specific server
[13:37:12] <StephenLynx> you can do that either way because mongo doesn't enforce relational integrity.
[13:38:23] <DennisGer1> this is hardcoded.. I know how the serverfarm structure will look like and it will not be changed anymore afterwards
[13:40:24] <StephenLynx> arrays of arrays where an index represents a specific value is not a good design anywhere.
[13:44:01] <DennisGer1> db.getCollection('serverfarm1').find( {}, {"servers.server.1":1}) is not returning all the sub arrays ...only till mainboard level
[13:56:50] <DennisGer1> still not getting all levels....but thanks so far Stephen. I think I really need to redesign the JSON structure
[14:48:36] <gabrielsch> is there any way to query and retrieve nested collections? here's my "schema": https://gist.github.com/gabrielsch/170736a9682915e63fed
[14:51:39] <GothAlice> Instead of {supplies: {some_id: {productId: some_id, …}, …}} if you pivot your data like this, it becomes actually usable: {supplies: [{productId: some_id, …}, …]}
[14:52:05] <GothAlice> db.Supplier.find({"supplies.productId": some_id}) < and you can index on supplies.productId to make this efficient.
[14:53:14] <GothAlice> http://www.javaworld.com/article/2088406/enterprise-java/how-to-screw-up-your-mongodb-schema-design.html < give this a read
[14:55:04] <GothAlice> It more comes down to data growth (growing records up can be expensive) and patterns of use. Will you always want the parent record's data when getting a single child? If you delete the parent, will you want all children cleaned up automatically? Etc. The article I linked explores a few of the criteria for embedding.
[14:55:06] <saml> db.supplies and db.suppliers but then if you need suppliers data... you might want to denormalize
[14:57:52] <gabrielsch> saml: when I designed this, I thought that supply is an internal detail of supplier
[14:57:59] <saml> so, I think this makes sense for you: db.supplies {_id: productId, supplier: {name: 'webscale', address: '1st west nyc'}, some other product info}
[15:00:33] <GothAlice> Uhm, like any database I hope you aren't trusting user-supplied data for your query source.
[15:00:49] <nawadanp> There is two commands to manage the balancing of a collection : sh.disableBalancing(namespace) and sh.enableBalancing(namespace). But how can I see if the balancing is enable or not on this collections ?
[15:00:57] <GothAlice> I.e. your application code should be constructing the query, and not through JSON serialization. ;P
[15:01:41] <GothAlice> nawadanp: Ref: the Databases and Sharded Collection sections of http://docs.mongodb.org/manual/reference/method/sh.status/
[15:03:33] <nawadanp> Also, when I disable the balancing on a collection, how can I be sure that there isn't any balancer process on it ? Currently, I just check that this return nothing : config.locks.find({ '_id': namespace, "state": 2}).count()
[15:05:15] <saml> gabrielsch, so.. maybe db.products and each product is tagged with one or more supplierIds
[15:11:17] <GothAlice> gabrielsch: It's also very important to remember that having truly dynamic field names (i.e. {foo: {"27": {some_id: "27", …}}} — "27" is being used as a field name here) is in general a very bad way to store data, as you can't index it, you can't search across several nested values at once, only one at a time, etc., etc.
[15:12:14] <nawadanp> About the s.status(), I expected a command more specific, like for example getBalancingStatus(namespace)
[15:23:05] <lllama> hello all. I'm upserting a bunch of docs from node but I'm closing my connection before they've all been written.
[15:23:36] <lllama> Is there a 'drain' event I can listen for, or similar? (using something like that with postgres).
[15:29:10] <lllama> GothAlice: thanks. Using bulks might need a rework, as I won't know the number of docs until runtime.
[15:29:36] <GothAlice> lllama: The bulk operations manager will do the chopping up into discrete units of work for you, or should.
[15:30:16] <GothAlice> I.e. if you add 1001 operations to a bulk set, it'll do two units of work. (The first 1000, then the next 1.)
[15:43:30] <lllama> GothAlice: cheers. I shall experiment further. ATM {w:1, j:1} doesn't seem to help
[15:44:05] <GothAlice> lllama: If your JS code is async, you'll need to be careful of callback hell, and make sure you only let your app actually quit when all operations are complte.
[15:44:11] <lllama> I can bodge it by delaying the db.close call until all my update callbacks have fired.
[15:45:17] <lllama> GothAlice: Had the same issue inserting into postgres but its driver has a "drain" event that I can listen for.
[15:46:34] <GothAlice> lllama: As a note, I've never once in my life explicitly closed a MongoDB connection.
[15:47:02] <GothAlice> lllama: Given the client drivers do connection pooling, closing connections seems excessive. ;)
[15:47:47] <lllama> GothAlice: my inserts are being done from a script, so I want the program to exit once I'm done.
[15:48:44] <GothAlice> In those scenarios I still don't explicitly close. The connection will be cleaned up when the script exits, regardless. (Not sure if Node has an "atexit" callback registry, though.)
[15:59:38] <deathanchor> you only need to exhaust cursor that you explicitly say never to timeout, other than that you don't need to close anything
[17:25:50] <StephenLynx> funny thing, "but it replaces 1970s B-tree indexing with modern Fractal Tree® indexing technology." why doesn't mongo implement this on their engine?
[17:29:16] <deathanchor> also if the doc is big, it takes slightly longer to do updates because it has to recompress
[17:30:22] <deathanchor> nice thing it does to document level locks, so there is much less write blocking
[17:31:54] <deathanchor> there is no "repairDB" if you corrupt your files somehow (less reselient than mongo).
[17:45:25] <deathanchor> I use ansible for now, we may be forced to use some other paid software in the future, but for now everything is too complex for our needs where ansible fils that role easily.
[17:53:31] <bjorn`> Say I wanted to store sensor data (or any data for that matter) dumped once a minute, but I have no use of down-to-the-minute data weeks back, say I need minute data for the last day, per-hour for the last month and bi-daily for everything before that; What would be the best way to design this? I've though of separate collections for each time series, but that seems a bit unflexible. Better suggestions? A
[17:55:34] <StephenLynx> and about the different means of accessing the data, I would pre-aggregate these.
[17:58:04] <StephenLynx> the least data I have to pre-aggregate and the more often I have to access it, the more I am willing to pre-aggregate it.
[18:28:09] <shlant> hi all! is it recommended you use WiredTiger in most cases? I have a read heavy app on EC2
[18:31:20] <GothAlice> When it is, it'll be set as the default engine for MongoDB. Before that, stick to MMAP.
[18:31:47] <shlant> will do. And how would I go about determining whether or not I should go with PIOPS?
[18:33:43] <jr3> also, is there any way to just save specific fields back? I'm trying to reduce data going over the wire from mongo to my server
[18:38:30] <deathanchor> you can use it for update() or findandmodify() if you need the doc before change.
[18:39:18] <jr3> right now I'm basically retrieving all properties on the models, then save() back the entire thing
[18:39:34] <jr3> which I think is slowing it down, because we have some really large lists on the model
[18:40:17] <deathanchor> most of my ops are ( { query this }, {$set : { newOrExistingField : toValue } })
[18:41:29] <deathanchor> this blows, can't figure out why this driver won't send the query to secondary.
[18:45:49] <StephenLynx> anyone know how I contact the guy behind the node.js driver? I wrote a fix.
[18:46:39] <jr3> deathanchor: one last question, should I consider some kind of bulk save right now I have loops doing individual saves
[19:26:39] <crised_> in java, connecting to mongoclient with an URI that already have the database name in it
[19:27:04] <crised_> To get the database, does this line makes any sense? MongoDatabase database = mongoClient.getDatabase("mydb");
[19:32:02] <cheeser> afaict, that DB reference in the URL is largely for certain systems to be able to configure which database to deal with by inspecting the URL which can come from, say, stage specific config files (dev vs. staging vs. prod_
[19:33:09] <crised_> cheeser: oh ok, I don't have to worry about reconnects or anything, that should be done under the hood, right?
[19:34:21] <GothAlice> Additionally, the db provided in the URL is used as the authenticationDatabase for URI-based configs.
[19:34:33] <GothAlice> Thus you can include credentials in the URL, then switch to another database later.
[19:38:53] <cheeser> crised_: right. you might see some churn during a reelection but once the dust settles the driver will pick up the new master for you
[20:02:33] <deathanchor> weird... profiler writes to magic collections on secondaries, they vanish once you switch dbs.
[20:04:03] <deathanchor> so I'm having a strange time trying to get a distinct query to run on secondaries, but it seems to keep running on the primaries no matter what.
[20:09:40] <StephenLynx> "<deathanchor> so I'm having a strange time trying to get a distinct query to run on secondaries, but it seems to keep running on the primaries no matter what."
[20:11:18] <deathanchor> so yes, working with a dev who is doing the code changes. I requested the change because a user clicks something which triggers this distinct query, which locks up the DB, which backlogs a bunch of stuff which is my job :)
[20:11:52] <deathanchor> basically I wanted the distinct moved to secondary so it doesn't stop write ops
[20:26:25] <deathanchor> hmm.. anyway to tell mongo "run this query, but stop if it takes more than 30 seconds?"
[21:13:41] <cheeser> there's a lock taken out but it yields from time to time. you won't want to do it during peak times if you have a lot of data.
[21:23:34] <shlant1> is memory usage for a mongo server determined by size of DB? or by reads? or something else
[21:52:19] <Owner_> well i assume that --out will supress the stuff it spammed earlier when i let it dump to a directory
[22:09:17] <Doyle> Hey. Is there an upper limit to the size of the DB where performance becomes unmaintanable?
[22:10:05] <Doyle> I'm looking at the MongoDB at Scale page and I have to assume that some of these companies have hundreds of TB in their sets.
[22:17:44] <crised_> I'd like to have 2 rows as primary key, one is an number id, and the other is a date timestamp, the number id gets repeated through time many times. Can I do complex queries on attributes only on the last occurence of each id?
[22:59:29] <pqatsi> the build.log is https://bpaste.net/show/20b0d487e1a2 and the patch applied is https://bpaste.net/show/e3bc19c3c6be