Log file Viewer
#mongodb logs for Monday the 2nd of March, 2015
[00:18:01] <obeardly> boutell: if you're still here, I'm fairly certain at this point it's not a MongoDB issue at all, but a supervisord issue.
[03:05:08] <MLM> Why are the objects passed by from a Mongoose find query not working with `.hasOwnProperty(key)`?
[04:13:13] <preaction> the best way is the one that maximizes storage efficiency and makes querying as simple as possible
[04:17:02] <aliasc> and link documents with ids, i think this is not the way mongodb is meant to work right ?
[04:19:44] <preaction> but if videos have comments, you could make those as an array of inner documents
[04:20:53] <aliasc> embedding is good for small documents that dont grow fast and do not change regularly
[04:22:32] <aliasc> its not even a video sharing project, im using mongodb as cache db to compare and control some data
[04:40:54] <MacWinner> HI, with gridFS or with the gridfs files collection, is there a way I can find all files that are under a prefix? like filename: "/mypath/subdir/*"
[04:41:18] <MacWinner> i have all my files stored in gridfs with a filename key that corresponds to a linux file path
[05:07:55] <MacWinner> running into weird issue with php driver. where findOne() returns a document.. but find() does not with the same exact query: eg: $files = $collection->findOne(["md5" => "d41d8cd98f00b204e9800998ecf8427e"]);
[05:58:14] <guest999> thanks. just starting to get into Document -oriented databases (i come from relational). im a developer
[07:13:58] <daidoji> hello I'm trying to do a bulk import from the first method on this page http://api.mongodb.org/python/current/examples/bulk.html
[07:29:48] <daidoji> oh well, its time for sleep at this point but its frustrating when something that should work according to docs doesn't :-(
[07:32:02] <daidoji> but yeah it seems to be limited to generators, if I instantiate the list it seems to work
[10:20:12] <amitprakash> Hi, while attempting to create a user for a db, i am getting a No role named userAdminAnyDatabase@database
[10:28:06] <amitprakash> joannac, what role should I be specifying? mongo docs tell me this is one of the default roles
[10:30:02] <amitprakash> db.createUser({user: "username", pwd: "password", roles: [{role: "userAdminAnyDatabase", db: "application_db"}]})
[10:31:27] <joannac> the default role is called "userAdminAnyDatabase" and it's only valid against admin
[11:20:47] <bowlingx1> hey, I have a question to a gist: https://gist.github.com/BowlingX/2b5e4420f1da73decb4d#file-gistfile1-js-L14-L16. I have lots of Data that I'm going to iterate with this script (for a migration).
[11:21:28] <bowlingx1> I have 34 documents in the given collection, but for only 4 is the "update" callback method called
[11:22:10] <bowlingx1> but everything is written correctly…Only my futures are not resolved correctly (line 25) because not all callbacks are called
[11:25:51] <pamp> i have amillions of record...and in few cases v is an array with a lot of properties
[12:24:01] <rosenrot87> I do have a question about a weired behaviour of the MongoDB Driver for C#, it works flawlessly if I do not use the replSet option. Once activated this option, I get a connection refused from server. What do I miss?
[12:30:19] <rosenrot87> I use the oplog.rs to get notified if there are changes within my database. This is the only reason why i use replica sets
[12:34:15] <rosenrot87> Exception:Caught: "No connection could be made because the target machine actively refused it" (System.Net.Sockets.SocketException) A System.Net.Sockets.SocketException was caught: "No connection could be made because the target machine actively refused it"
[12:43:12] <rosenrot87> Waheedi: I want to access the database as before from a c# application. Do I need to specify a member?
[12:43:51] <Waheedi> if you want to have a replica set you need to add members to the replica set :)
[12:46:06] <Waheedi> because it is a software joannac. it should notify me it can't and act as a read only or something
[12:47:10] <joannac> rosenrot87: can you connect to the mongod from the mongo shell, where you app server is?
[12:48:25] <rosenrot87> joannac: I can connect from everyelse, even from my shell on my client without any problem
[12:48:44] <GothAlice> Waheedi: Except that it detected that its data is wrong. Unless you _want_ wrong data coming back in answer to queries, you pretty much want it to "crash".
[12:49:23] <rosenrot87> joannac: it is only the mongodb driver which tells me connection refused. this is why I'm here :)
[12:49:29] <guest999> hi. im using the windows dirver for mongoDB, but cannot resolve the 'Mongo' class in Visual studio. I have referenced MongoDB.Driver.dll and MongoDB.Bson.dll and have the necesary 'using' clauses
[12:49:49] <GothAlice> Simply have automation to detect removal of a member which spins up a replacement. High-availability means you need monitoring and automatic resolution of certain issues.
[12:51:17] <rosenrot87> joannac: I can connect from my windows gui app, my windows shell, I did not change anything on the client side, only the replset within the conf file on the server running mongodb
[12:51:24] <guest999> rosenrot87: yes. .Net 4.5 and yes using Nuget. The dll files don't seem to have a class called 'Mongo' anywhere that i can see !?!
[12:52:43] <rosenrot87> joannac: if i remove the flag from the config I can connect again....any ideas?
[12:52:58] <guest999> rosenrot87: yes i can see thoes. the tutorials im reading refer to 'Mongo.Connect()'
[12:53:30] <rosenrot87> guest999: look for another tutorial, there were a lot of changes. keep to the latest ones 2.6.8
[12:55:02] <rosenrot87> guest999: I also experienced this. Even the tutorial of the MongoDB Driver website says you should use client.GetServer(), but this is already deprecated
[12:56:20] <rosenrot87> joannac: because i just started with mongodb and thought for something basic like connecting to a database even a beta driver should work
[12:57:18] <rosenrot87> I mean I do not want to do fancy stuff. Connecting to a database should work in a beta, at least from my point of view :)
[12:58:15] <rosenrot87> Maybe there is one of the developers here which could verify if this is a bug maybe?
[12:59:02] <GothAlice> Both represent "pre-release, use at own risk". Neither should be assumed to do anything other than explode at the least opportune time.
[13:01:14] <joannac> rosenrot87: I would also like to see the output of the working cases i.e. windows gui, mongo shell
[13:03:07] <joannac> i highly doubt a driver would have gone out not being able to connect to a mongod, like you said it's a pretty basic feature :)
[13:05:49] <guest999> all tutorials i see tghem use the statement: var m = MonoDB(); but in MongoDb 2.6.8 using driver 1.10, the website says http://docs.mongodb.org/ecosystem/drivers/csharp/ its ok to do it, but i cannot see this 'MongoDB' class.
[13:06:19] <Waheedi> so if this db.terms.remove(d._id) is taking more than 40 minutes to execute that means there is definitely something wrong in there right?
[13:06:40] <guest999> is there a list or something showing the API and/or whats deprecated ? (this is all very confusing for me)
[13:06:50] <rosenrot87> joannac: both the shell and the gui application work right now...i can use them es usual
[13:08:07] <rosenrot87> joannac: now I reverted to the stable driver and I get the error: "No such host is known"
[13:11:07] <rosenrot87> joannac: sure, this is just the case if the replset is set and within my c# app
[13:13:04] <Waheedi> joannac: I'm using a hosting company famous one :) and few cloud block storage volumes lost connection. and this machine was a primary node in replset
[13:14:29] <Waheedi> but unfortunately that node was really outdated and it didn't shutdown or crash
[13:19:18] <GothAlice> Considering "id" isn't a query, I'm not sure what you'd even expect that to do. :/
[13:23:41] <GothAlice> Waheedi: I'm certainly something, but classification appears to elude you. You seem very obstinate for one requesting assistance; writing p-code describing what's going on is far less useful than providing logs demonstrating without modification or the need for assumptions what is _actually_ going on.
[13:24:14] <rosenrot87> joannac: I can connect from the shell when replset is OFF and ON, onces it responses with ">" and the other time with "rs0:primary>" :) Like it should
[13:24:53] <joannac> rosenrot87: yes, I want to see the output, including what options you give the shell
[13:27:38] <rosenrot87> joannac: mongo.exe -u mongo -p pass 111.111.111.111:27017/db MongoDB shell version: 2.6.8 connecting to: 111.111.111.111:27017/db rs0:PRIMARY>
[13:28:07] <Waheedi> Do you guys really tried this db.terms.remove(ObjectId("54f0e60931326c1e3d000d6b"))??
[13:28:56] <GothAlice> Waheedi: http://docs.mongodb.org/manual/reference/method/db.collection.remove/
[13:32:22] <joannac> rosenrot87: paste your connection code. paste the output of the telnet command I gave you earlier
[13:34:49] <spuz> Hello, is it possible to update a document only if a certain field matches a certain value?
[13:36:16] <GothAlice> Waheedi: RTFM and see that even if it works, it's not how you're supposed to do it. The fine manual does not mention anywhere in the documentation for the command that use, thus the ability to use it the way you are is a fluke, unsupported, and subject to change without notice.
[13:39:35] <GothAlice> rosenrot87: I haven't been fully following your issues, but in your terminal are you able to execute "db.getCollectionNames()"?
[13:40:08] <cheeser> at least the java driver (which shares DNA with the .net driver) lazily connections.
[13:40:38] <cheeser> so even though the new "MongoClient" "succeeds" you're not actually connected until you try to talk to the db
[13:40:39] <joannac> rosenrot87: when I say "I want the output of the telnet command", I want the DAMN OUTPUT
[13:43:13] <joannac> cheeser: but the server.getDatabase should make the connection actually instantiate, right?
[13:44:22] <rosenrot87> telnet server port -f output.txt should write everything to file.....file is just empty
[13:49:13] <amitprakash> Hi, I have a collection package reflecting manifested shipments ( assigned waybills w/o information such as consignee/address etc )
[13:49:46] <amitprakash> When we get the actual shipment information, we go and individually update each document in the collection
[13:50:23] <amitprakash> i.e. when _id = blah, update with this and when _id = blahblha update with this_ and so on
[13:50:39] <GothAlice> amitprakash: Technically, yes, there do exist bulk operations. However they don't operate the way you're wanting, really.
[13:51:45] <joannac> that's your mongo shell's connection to your mongod dying and getting re-established
[13:55:04] <GothAlice> rosenrot87: Make sure all DNS names, both fully qualified and bare, resolve, even if you have to add them to /etc/hosts (%SYSTEM32%/drivers/etc/hosts on Windows, AFIK).
[13:55:16] <amitprakash> GothAlice, would it be advisable to go with bulk updates as opposed to updating one by one?
[13:55:40] <rosenrot87> joannac: Could it be the problem that the host and primary entry "phoebe:27017" can not be resolved?
[13:56:21] <rosenrot87> GothAlice: Could I replace the "phoebe" with the plain IP address of the server?
[13:56:55] <GothAlice> amitprakash: It can be worth it, but to truly answer the question would require writing it both ways and comparing performance.
[13:57:19] <amitprakash> GothAlice, we've already done the one by one approach and its already taking a lot of time
[13:58:03] <GothAlice> amitprakash: Perfect, then you have a point of comparison. Optimization without measurement is by definition premature. :)
[13:59:41] <joannac> rosenrot87: can you make the hostname "phoebe" resolve from the machine where you're testing the c# driver and try again?
[14:09:33] <GothAlice> Two points: first, I don't know. Second, it's not something I'd ever do. DNS allows for reconfiguration, potentially without downtime or reconfiguration steps (other than adjusting DNS resolution), so this is a great feature, not a bug.
[14:10:52] <arussel> should I expect my driver (reactivemongo) to use a snapshot when doing a query ?
[14:13:22] <GothAlice> arussel: See the "snapshot" method reference on http://reactivemongo.org/releases/0.10/api/index.html#reactivemongo.api.collections.GenericQueryBuilder
[14:13:36] <GothAlice> arussel: Out of the box, likely not. No driver I know of snapshots by default.
[14:23:34] <arussel> GothAlice: thanks, not easy to google snapshot as we get all the result about version SNAPHSOT
[14:30:05] <menger> and its in the repo: http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/RPMS/
[14:30:36] <GothAlice> The trick, I find, is in the balance between evaluating what the client wants, and giving them what they need.
[14:30:42] <GothAlice> http://docs.mongodb.org/v2.2/tutorial/install-mongodb-on-red-hat-centos-or-fedora-linux/ < these are the docs from 2.2, no guarantee it'll work.
[14:34:46] <GothAlice> menger: https://jira.mongodb.org/browse/SERVER/fixforversion/12313 < browse forward from here (navigation in upper right) to see what has improved over time up until 2.2.7.
[14:36:00] <GothAlice> (Or "Release Notes" button in top right for a breakdown by category for any release.)
[15:05:39] <Cygn> Anyone ever used the jenssegers laravel plugin for mongodb and figured out how to use map/reduce there?
[15:37:55] <calmbird> hi, could you help me: http://gyazo.com/a6d611265928b046c91f41edacb81877 , mongoose: how could I deal with category = All in my case?
[15:39:14] <GothAlice> calmbird: Progressive query refinement. Basically, store the intermediate result of Idea.where('author').equals(author) somewhere, then only conditionally restrict on category through re-assignment of the query variable you are building up.
[15:43:07] <GothAlice> Insert obligatory rallying cry of "MONGOOSE!" and fist, raised to the sky. That is some ugly, ugly stateful syntax. ^_^;
[15:45:23] <GothAlice> Collection.where('field').equals(value) — How is this "easier" than Collection.find({field: value})? What the hojek does .where() even return? Can you save _that_ for later? (I wouldn't try it…) I love libraries that give more questions than answers. XD
[15:47:33] <GothAlice> The majority of support cases I deal with in here are mongoose-related injuries.
[15:47:34] <d0x> To make our "operational" data (collected over the day) available in a dashboard, I used a daily Mongodb-MR job is converting it (joining collections). I used MR because i don't like to have another "infrastructure" to do this job. (in sum we have around 400GB). Is it common to use MR for this kind of tasks? Because I have hit already limitations like calling `db.xxx.find(this.xxx)` in the map stage. As workaround i simply loaded the wh
[15:47:34] <d0x> ole xxx database into the scope... Which is working because joined collections aren't huge.
[15:48:46] <calmbird> Oh I can't find clear answer in the internet. Can we start mongod with some parameter, that will ensure data will be writen to disk, before answering?
[15:49:33] <GothAlice> d0x: We took a different approach for our event analytics. We pre-aggregate the relevant statistics, allowing us to produce whole dashboards a la http://cl.ly/image/2W0a2D3I370F that generate, from live data, in less than 100ms.
[15:49:54] <GothAlice> d0x: See: http://www.devsmash.com/blog/mongodb-ad-hoc-analytics-aggregation-framework
[15:50:43] <GothAlice> calmbird: There are many different options you can choose when telling MongoDB to read and write data. What you're looking for is the ability to set a default "write concern" for your application connections.
[15:52:00] <calmbird> I just want to be sure, that mongodb will write data to disk before answering ok.
[15:52:45] <GothAlice> calmbird: By default these days, MongoDB will wait for confirmation of receipt by the server. If you want disk-persistence guarantees, enable "journalling" in your write concern. If you want to avoid a single-host failure nuking the data, you can specify how many hosts you want the data to appear on before the write is considered complete.
[15:54:50] <calmbird> I'v seen video https://www.youtube.com/watch?v=JWaDa8taiIQ, that guy is saying mongodb is bad for criticall data, because you are not sure, data will be writen to disk, but mongo will always answer yes. But I'm guessing that it has been fixed for now.
[15:55:35] <GothAlice> calmbird: As a fun note, I actually _lower_ the write concern for many of my inserts. Centrally aggregated logging records need to be fast, it's OK if a few get lost during high load. (There are machine local files which contain any missing records.)
[15:56:43] <krisfremen> funny how people believe everything the read on the internet without doing any testing for themselves
[15:58:43] <calmbird> I can just use mature database like mysql , postrI can just use mature database like mysql , postgrees etc, that everyone using. Or get lucky shoot with mongodb, and then cry, that 10k Euro transfer wasnt writen in DB. :)
[15:59:02] <GothAlice> calmbird: I have 26 TiB of data in MongoDB, and I've had it in MongoDB for 8 years…
[16:01:22] <GothAlice> Okay, 6. My "nearest even number" rounding failed in this instance. calmbird: I migrated my data out of MySQL as soon as a viable alternative was presented. MongoDB fit the bill for my requirements, even back then.
[16:02:07] <GothAlice> Though I did have to add full text indexing and compression at the application layer.
[16:02:45] <GothAlice> calmbird: If you've ever had a failure in MySQL that required you to reverse engineer the on-disk InnoDB table structure… BSON is a joy to work with by comparison.
[16:03:55] <GothAlice> Derick: It was a bit of a shock that even with directoryPerDB enabled, InnoDB stored critical table structural information in a common pool a directory up… quite the surprise indeed.
[16:04:18] <calmbird> GothAlice: : In mongodb we need to join data on server side, in mysql we can join data in database side, is it any issue of mongodb? is it slower etc?
[16:04:47] <GothAlice> calmbird: In MongoDB you don't design your data models as if they were relational, so in general the difference is a non-issue.
[16:05:12] <GothAlice> There are other ways of storing related data that make more sense when you have document storage, for example, I store all of the replies to a forum thread within that thread's document.
[16:05:39] <GothAlice> Need to move an entire thread to a different forum? $set one value in one document and you're done.
[16:05:51] <calmbird> GothAlice: Well in mongouniversity course, they said that for larger data, we should add more collections. So then we need to join some data.
[16:05:59] <Derick> GothAlice: what sort of "tricks" did you do to make sure you're not hitting the 16MB doc limit that way? Or, don't you have that many replies?
[16:06:16] <GothAlice> Indeed. Highly relational data, data where you have transactional requirements, or data where you are performing deep graph traversal are all examples where MongoDB might not be the right solution for you.
[16:07:14] <GothAlice> Derick: 16MB doc limit = (16*1024*1024*1024/6.5) ~2.5 million words if the average length is 5.5 and counting spaces.
[16:07:34] <GothAlice> Derick: But, in general, it's trivial to have a "comment" at the end that links to the continuation thread, then the initial thread is locked.
[16:07:50] <Cygn> Hey Guys just a general question, if i have a collection, let's say with customers, each customer did multiple sales, would you say this is something that should like ALWAYS be done with a relational database? Or could somebody for example say i insert the sales as a sub-array inside a customer collection without ruling against any best practice rule?
[16:09:00] <calmbird> look: http://gyazo.com/9094325c7d11126a3cb65e261200a670, this is from mongouniversity.com course, they are making related data here
[16:10:12] <Cygn> Derick: I need to fetch sales, by ONLY counting them, using their date and market (would be saved in the subarray). Before i do that, they would be filtered using attributes of the customer. (For Example, i could need all sales of an iPad (article name in the subarray), but only if the customer is from USA(country in the customer data) )
[16:10:13] <calmbird> yeah we have something like 15MB limit, cant remember, if we expect documents to have large amount of data, we should split it to more collections and eventualy join it
[16:10:56] <Derick> Cygn: i would not create subdocuments in that case then. Have each document store date, market, customer attributes that you need to query on - perhaps
[16:11:02] <GothAlice> http://www.javaworld.com/article/2088406/enterprise-java/how-to-screw-up-your-mongodb-schema-design.html
[16:11:34] <calmbird> Derick: : I understand, but subdocuments are part of document right? and if we will have realy alot subdocuments, we can extend ~15MB limit
[16:11:52] <Derick> in the case of blog comments f.e. - if it's something like my tech blog, it's not a problem as I rarely have any comments at all. If you're hackernews, you might run into issues.
[16:12:04] <Cygn> Derick: But that would mean i should rather use a relational database? Because i was very happy about the speed that mongodb delivers right now, but that was BEFORE i had to filter it by the other filter criterias.
[16:12:46] <medmr> is there any way you can fetch related documents from another collection with out first looking up the salesperson
[16:13:27] <calmbird> i have a game, document user with subdocuments, mapobjects, mapblockers, friends everything, and I started to worry about 16MB limit actualy
[16:13:40] <GothAlice> calmbird: Consider that I when I was writing those forums I had to import all of the old forums for a gaming group with 14,000 people. The entire old forums could have fit in a single 16MB document.
[16:13:53] <Cygn> Derick: Actually right now i don't see how to get this without joins… medmr: in every sale there is an attribute which declares the salesperson
[16:14:17] <GothAlice> medmr: I previously described, for the forums example, a method of having continuations.
[16:15:26] <Cygn> medmr: So you would just fetch all sales that fit and all customers that fits and then iterate through it and see which fits together?
[16:15:33] <medmr> if the queries can be done in parallel its not significantly slower than looking up one monolith document
[16:17:09] <Cygn> The Problem is this could result in quite a high amount of data… let's say a few million sales and a few million customers, and i would have to fetch nearly 90% just to sort it out… does not feel like the most clever way :/
[16:18:13] <Cygn> medmr: So you would first fetch all customers that are fitting and afterwards ask for all sales that belong to them ?
[16:22:06] <Cygn> medmr: okay, most of the time, the question will f.e. be "how many sales in june" - i don't need the customer for that, only the sales (data is in the sales). But then there will be questions like "how many sales in june by vip customers" - that's where i will have to start filtering the results by the customer criterias
[16:22:48] <Cygn> medmr: Actually i have to in a way (because i always have to count sales, filtered by customer criteria) but no i don't have to display the customer information by itself.
[16:24:03] <Cygn> GothAlice: That was just one example without digging to deep in the actual topic. The Customers we are talking about here have very much specific criteria (because actually it is not a customer buying but a sales unit, which contains information about series, headunit, country, model etc.)
[16:25:16] <GothAlice> Cygn: The trick with MongoDB is that your "schema" isn't sacrosanct, it should instead adapt to meet your query needs. Pre-aggregation (i.e. including "vip" in the sale) isn't data duplication, it's making your data queryable.
[16:26:06] <Cygn> GothAlice: Yeah i'm kind of getting this idea right now, but that would mean i would have to update every single sale of the customer when the customer changes, right?
[16:26:31] <GothAlice> Potentially, but again, an index on cust_id will make that type of update fast.
[16:26:42] <Cygn> medmr: by flags you mean just insert the customer attributes into the data, because i just googled "mongodb flags" ;)
[16:28:43] <Cygn> GothAlice, medmr: Just one last backquestion. If i would like to insert the data. My structrue should be Customer1 { id: 1, sales: { sale1, sale2, sale3 }}, Customer2{id: 2, sales {sale1, sale2, sale3 }}, and an index on the customer id, correct?
[16:30:31] <Cygn> GothAlice: Okay, but (sorry for asking again) then i don't get how i should filter the sales using the customer data, without fetching the customer first in my client/server-side code?
[16:30:44] <GothAlice> Indeed. We also pre-aggregate the related data and embed it all, since our invoices need to be involatile.
[16:31:13] <Cygn> GothAlice: But by pre-aggregation you mean aggregate on database-side or on server-side?
[16:31:39] <GothAlice> Cygn: Question: if a customer changes their billing address, do you want old, archived orders to change to reflect the new address?
[16:32:04] <GothAlice> (The only correct answer is: no. Archived financial information should not change.)
[16:33:06] <GothAlice> As such, we clone pretty much every detail about the user placing an order, and the company the order is being billed to, within our invoices. (All that make sense to store, at least. Login history from the user doesn't need to be there. ;)
[16:33:36] <GothAlice> Querying our invoices for company or user-related information is then easy; no join, since the data is "pre-joined".
[16:34:18] <Cygn> GothAlice: Sure, actually in our case we are storing the sales only for statistic use. But anyway, if i get you right, you would aggregate the two collections in the moment you would query for it, like creating a View in the classic sql world?
[16:36:12] <Cygn> GothAlice: You would insert all the customer data in the sale just when you save the sale? But didn't you say invoices are their own collections? Sorry, i really try to get it.
[16:36:42] <GothAlice> User = {name: "Bob Dole", email: "bdole@whitehouse.gov"} Company = {name: "Some Company", vip: true} Invoice = {creator: {name: "Bob Dole", email: …}, company: {name: "Some Company", vip=true, …}, line_items: […], …}
[16:37:08] <GothAlice> Indeed. If a user changes their e-mail address, or a company's VIP status changes, in our case we still want those archived orders to say "yeah, this order was a VIP order".
[16:38:21] <Cygn> GothAlice: I won't have this situation (since our "customers" are sales units for statistical data, if they change their status, it should be changed for older sales also), but anyway i could think of doing it that way anyway.
[16:39:22] <Cygn> Now i just have to find out how to fetch for subarrays of an entity using a condition for only specific subarrays ;)
[16:39:35] <mikeputnam> can someone share or point me to pragmatic methods to handle data migrations/deployments as part of a workflow? devs create code + data/schema changes => i orchestrate those changes into deployments that get applied to staging, production
[16:40:26] <mikeputnam> in the mysql world, i've done this with capistrano and incrementally numbered .sql files that get applied sequentially.
[16:41:23] <mikeputnam> i realize this. but that is a symantic argument. the concept of a schema still exists.
[16:41:40] <StephenLynx> one can write code that perform queries that rename fields and adapt data to a different logic.
[16:42:43] <StephenLynx> when I am writing code I always assume things may not be there and write defensively around that.
[16:44:09] <StephenLynx> so unless you are working with a tool that implements a pseudo-schema, like mongoose, the developer will have to develop it from scratch.
[16:46:56] <mikeputnam> i see. that makes sense. from my perspective, i will likely be adding a "db_changes" step to my deployment process that just runs whatever the devs come up with.
[17:04:16] <Cygn> StephenLynx: Do you have an url with a best practice example for how to query for items using attributes of the subarrays for me?
[17:05:01] <StephenLynx> but I can link you a project I did using mongo and try to learn the pattern I work under.
[17:07:14] <StephenLynx> I try to stick to basic rules: don't make multiple queries for one thing, don't avoid sub arrays too much
[17:08:02] <GothAlice> mikeputnam: https://gist.github.com/amcgregor/dda39062f8b4f74f5e1a < my method of doing "migrations"
[17:08:19] <Cygn> StephenLynx: Just a general question before i start digging in: Do you use elemMatch for the Subarrays?
[17:08:51] <StephenLynx> specially when I just need to know if the array contains a specific element
[17:09:32] <GothAlice> mikeputnam: The runner searches a given Python package namespace for scripts with main() functions (sorta, it's a bit indiscriminate at the moment) and runs them, using docstrings to output progress. The migrations themselves are written so that multiple execution is safe, i.e. they check preconditions.
[17:13:47] <Cygn> StephenLynx: in my case it could be really useful since i never need the full data, i only need to count the subarrays using one of their criteria
[17:18:39] <mikeputnam> GothAlice: thanks! this confirms the direction StephenLynx suggests. generic runner runs specific developer-created code that changes mongo appropriately.
[17:19:34] <mikeputnam> in the past i've come upon suggestions that documents should have a version number to enable rollbacks. any thoughts on this?
[17:20:28] <GothAlice> Well, that alone doesn't enable rollbacks, but consider: why does one really need to move backwards?
[17:22:04] <d0x> GothAlice: Thx for the references. Our application is using serval collections which are designed for the daily business. And to "pre-aggregate" them for the Dashboard i utilised MR (because the Aggegration Framework had some limitations for us (like missing string functions, accessing data from other collections, ...)). After the MR is executed i use the aggregation framework as well to get fast responses.
[17:22:04] <d0x> How would you "pre-aggregate", "pre-join", ... the data? I thought the best is to utilise mongoldb as well. And the only option was for me is MapReduce. As alternative i could do smth. like db.xxx.find().forEach(...). But that doesn't scale.
[17:23:14] <mikeputnam> my perspective is that of a system administrator. i'm concerned about late-at-night code+dat migrations that fail due to unforseen data condtions (or whatever) and i'm left holding the bag. as a non-developer with 15 developers generating change, this has me concerned about the stability of production.
[17:25:53] <mikeputnam> we are moving toward an always up/no downtime deployment model (blue/green) but this introduces more concerns for the developers writing the db changes correctly. -- at least in this model, i can recover by just reinstating the previous version.
[17:26:07] <GothAlice> mikeputnam: Standardize your deployment procedures. We have a random YouTube music video go up on the projector when the automation gets triggered, and the developer deploying must wear a silly hat.
[17:26:33] <GothAlice> (These two things generally attract attention, which means oversight of whatever that developer is doing in production.)
[17:28:07] <GothAlice> For example, we also deploy first to a clone of production called 'staging'. And only if everything goes well there do we repeat the automation on the real production environment.
[17:29:07] <daidoji> if inserting with continue_on_error=True, is there a way to get all the exceptions?
[17:29:17] <daidoji> or is it like the documentation says, I only get the last error that occurred in the batch?
[17:30:36] <daidoji> hmm, then thats kind of pointless for my use case then. Guess its better to use the old style bulk_import functions if I need all the errors?
[17:31:15] <GothAlice> Indeed. You'll get reduced throughput (inserts/sec) by querying for the result of each, but if that's what you need, that's what you need.
[17:34:50] <daidoji> also out of curiosity, what's the fastest way to load data in your experience? python -> pymongo or python parse -> stdout -> mongoimport
[17:35:32] <GothAlice> I've never really bothered to benchmark bulk loads. I do the smallest number of hops, so pymongo direct.
[17:35:34] <daidoji> I ask because we pipe-delimit here like all god-fearing people and mongoimport doesn't support choice of delimiters for some reason :-(
[17:36:08] <GothAlice> If you "delimit" at all, there's something wrong, 'cause MongoDB does deeply structured records, not flat ones. ;^)
[17:36:38] <daidoji> awww, I know that and if I were in charge things would be a lot different, but I jump when they say jump at the moment :)
[17:37:05] <daidoji> currently, I'm about bulk-loading large datasets that change constantly which is why I ask
[17:37:08] <GothAlice> Hint: YAML, of which JSON is a sub-set. (YAML also has a natural way to express multiple records per file.)
[17:37:52] <daidoji> unfortunately these formats are out of my control and are pipe delimited csvs :-9
[19:04:04] <keeger> i am trying to design a database for a program in mongo, and i have a question about document arrays
[19:04:19] <keeger> i would like to store an array of sub-documents sorted by timestamp inside a document
[19:07:33] <StephenLynx> but you can, from time to time, use an aggregate and update the array with the result of the array.
[19:08:18] <StephenLynx> the downside with this option ,though, is that if the array is empty, you will get nothing, if I'm not mistaken.
[19:08:42] <GothAlice> keeger: For the most part attempting to keep elements within a nested array in any order other than insertion order (i.e. always append or always prepend) is difficult.
[19:08:51] <StephenLynx> unwind splits the array into a series of documents with an element of the array each.
[19:11:08] <GothAlice> keeger: Regardless of how you do it (aggregate unwind, filter/sort, group, or within your application code) it's expensive.
[19:16:31] <GothAlice> In that instance, I'd just sort them at the display layer. I.e. in your application code.
[19:17:18] <keeger> sounds like it. then when i delete one, i can just delete it by index in the array
[19:18:11] <keeger> i'm also struggling a little conceptualizing horiznotal scaling on webservers and mongo
[19:18:48] <keeger> if i have 2 web servers, and 2 people attempt to write to the same document, does it block the 2nd one?
[19:20:57] <keeger> i am thinking of putting in a simple lock system, because i need to loop through this array and update other parts of the same document. but it could take 100 ms who knows
[19:21:43] <keeger> so i was thinking of putting a document.LockId entry, and then the clients would read, look for lock, if missing lock it with a unique Id, read it back and confirm the Id matches, and it's theirs
[19:22:08] <GothAlice> keeger: MongoDB doesn't ever really do two things to the same data at once. Instead, MongoDB reduces operations down to individual atomic operations. So operations A and B on the same data will either resolve A, B, or B, A.
[19:22:52] <GothAlice> mordonez: Or, if you don't already have a replica set, see: http://docs.mongodb.org/manual/tutorial/convert-standalone-to-replica-set/
[19:24:32] <GothAlice> keeger: Update operations (A and B in my continuing example) may have conditions that need to be true before the update can be applied. This would be the query that is passed as the first argument to .update() (normally). You can check for ownership of a lock and if nUpdated=0 (in the returned result) the condition failed.
[19:25:40] <keeger> i think that's more straightforward and lightweight than trying to a network level semaphore
[19:26:57] <GothAlice> keeger: As a note, so far I haven't ever required a dedicated field for locking except for my distributed task runner, which locks based on current worker. Simple field updates and update-if-not-modified (conditional) updates are more than sufficient for most cases.
[19:28:14] <keeger> but i do have one process that i don't know how long it will run, and it can only run once. i'd like to lock it at the db level cuz of the horizontal ness
[19:30:48] <MLM> How do I find documents that have a certain string in one of their array of string fields? Mongoose `Model.find({ usersIds: req.user._id })` isn't pulling up any results (confirmed that they are in the collection)
[19:57:29] <sivli> Well I am rather new to geo-location data but as I understand it mongo as good geojson support?
[19:59:28] <sivli> Was wondering if anyone use the geospatial indexes and if they know if topojson is supported?
[20:01:00] <sivli> We are not doing anything to crazy right now, just points, but there are plans to expand later and topo looks much more efficent.
[20:01:32] <sivli> Any idea if this is being considered or is it jsut too early in the geo game to know?
[20:03:21] <sivli> Agreed. Ok wel; thanks, I just wanted to be sure before I told the boss man no can do for topo. Not worth losing the mongodb support.
[20:19:23] <MLM> Running `db.queues.find({userIds: '54effc576676066c20293010'})` returns some results in the mongo shell. Running the same query with Mongoose `Queue.find({ usersIds: '54effc576676066c20293010' })` gives no results. What is the best way to debug this?
[20:20:06] <GothAlice> MLM: It's actually a critically important distinction: are you storing the hex-encoded string version of the ObjectIds, or are you storing actual ObjectIds?
[20:21:15] <GothAlice> Less cool, but consistency is key. You can't mix the two and get sane results back. It's less cool because you're wasting 12 bytes for every single reference stored, but…
[20:21:32] <GothAlice> So you're mixing, and your ability to query that field goes out the window. :(
[20:23:33] <MLM> I will convert over to full string representation consistency but why does it return results in the Mongo shell
[20:24:10] <GothAlice> That's a very good question. Likely the ODM-side of things recognizes that that field is an ObjectId reference and is attempting to automatically cast the (valid) string-version you are supplying.
[20:24:34] <GothAlice> Real ObjectIds store some interesting data that is useful to be able to access. And halving the storage space is nice.
[20:25:28] <MLM> Will do (convert to consisten ObjectId). Need to look up again the quirks with comparing them. Thanks for the help
[20:58:22] <MLM> The mongo shell is returning the documents with this query `db.queues.find({userIds: ObjectId('54effc576676066c20293010')})` but I can't get Mongoose to return it. I tried converting the string to a ObjectId in the query itself even though I have read that it should be auto casted.
[20:58:27] <MLM> `Queue.find({ usersIds: '54effc576676066c20293010' })` or `Queue.find({ usersIds: new require('mongoose').Types.ObjectId('54effc576676066c20293010') })`
[21:02:58] <obeardly> MLM: I'm a noob to Mongo, but I would couldn't you just: ObjectId("yourstringgoeshere").toString()
[21:03:16] <MLM> GothAlice :/ - I know we talked before about native Mongodb driver and I was tempted and tried moving over but I kinda like the enforced Schema system which I would need to find somewhere else or write custom
[21:10:34] <MLM> Here is a full barebones test snippet that demonstrates the issue if anyone is interested. If it is not my fault then I'll make an issue: http://pastebin.com/enUPT03A
[21:15:47] <GothAlice> MLM: There's a schema, it obviously says "this is an array of ObjectIds", doesn't cast when querying. Looks like a bug, or, a feature, depending on how generous the Mongoose developers want to be.
[21:16:29] <mspro> hello, i think i managed to create an new collection with the name [object Object] when i used the copyTo command, now i can’t drop that collection, anyone an idea?
[21:17:35] <GothAlice> mspro: In the MongoDB shell you can access collections as attributes of the database (db.foo) but also as associative array elements (db['foo']) — this latter approach should let you clean up that mis-named collection.
[21:17:55] <GothAlice> mspro: Also love the combination of your IRC handle and having that particular issue, BTW.
[21:18:51] <GothAlice> calmbird: That's the typical approach. Lighter-weight approaches involve "update-if-not-different" mechanisms; two-phase is most useful for coordinating changes to multiple documents, on single documents the simpler approaches will likely be better.
[21:19:07] <MLM> GothAlice: Not totally understanding what you are saying. Is it just the way I am trying to query it or a true bug?
[21:20:16] <GothAlice> MLM: Unfortunately you'd have to ask the Mongoose developers. Whenever Mongoose comes up I'm reminded of a quote which I will modify appropriately: I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a design. I.e. they may consider the behaviour you are running into to be by-design.
[21:22:41] <MLM> Made an issue: https://github.com/LearnBoost/mongoose/issues/2728 - If you see any way to improve the clarity/description of the issue, then I'm intersested
[21:28:15] <GothAlice> obeardly: One stores ObjectIds as ObjectIds to save 17 bytes of space (hex encoding doubles the number of raw bytes, plus requires a terminating CString null and leading 4-byte PString length), to maintain numeric comparison capability (i.e. you can query $gt/$lt ranges to filter based on record creation time), as well as maintain the client-side capability to examine the fields of the ObjectId (timestamp, host, PID, sequence number) without
[21:30:24] <mspro> GothAlice: i can access my other collections with the [‘foo’] syntax, but when i do a ‘show collections’ i get a list with my collections + on top of that list within square brackets i get the [object Object] that wasn’t there before? It’s like when you want to print an object
[21:33:06] <mspro> GothAlice: your a genius and i still have a lot to learn, so my collection was literaly named [object Object], i did not expect that one :) thanks
[21:33:49] <GothAlice> mspro: MongoDB would "stringify" any value you try to use as a collection name. "show collections" is less useful to see what's going on than "db.getCollectionNames()" is in this instance.
[21:35:21] <keeger> so if I have a natural key for my document, would it be better to use that over the ObjectId?
[21:35:36] <GothAlice> keeger: If it's unique and can be generated without querying the existing dataset, yes.
[21:35:37] <mspro> yes, i tried that second one but still didn’t get that it was listing the literal name
[21:36:25] <keeger> GothAlice, when you say generated without querying, you mean, it's not like Id = Max(id) + 1?
[21:36:53] <GothAlice> keeger: Indeed. Or, say, something more complex like YYYYWWNNN where NNN is the invoice number for the week WW in year YYYY. (Our invoicing scheme at work.)
[21:37:16] <GothAlice> Any time you need existing data to insert a new record you run into race conditions.
[21:37:46] <GothAlice> (This is why MongoDB uses ObjectId and not auto-increment. ObjectId scales to multiple independent processes quite well.)
[21:39:06] <GothAlice> (This is something Facebook and Twitter had to learn the hard way… Twitter even rolled an entire separately-scaled software service for the purpose of generating IDs without conflict. ;^)
[21:39:49] <keeger> i dont want to have to do joins, so i'm thinking the location is inside the person document
[21:40:25] <GothAlice> Setting ownership in a way that there can be only one victor (first person to claim wins) is easy using update-if-not-modified. db.locations.update({_id: …, owner: null}, {$set: {owner: ObjectId(…)}})
[21:40:55] <keeger> more of a do i store a collection of Locations, and a person holds a reference to it?
[21:40:57] <GothAlice> The record with the given ID won't re-set the owner if one has already been set. (The result, returning nModified, lets you know if the assignment worked, or there was a conflict.)
[21:43:29] <GothAlice> Updates to a single record will always be resolved in a linear fashion. So two updates, U1 and U2, will never really "conflict" (other than trampling each-other's data), and the "find" part of that update handles the trampling by requiring the field to be empty (null) or the $set won't be applied.
[21:44:25] <GothAlice> If your application layer issues the updates in the same microsecond it may seem semi-random which one will win. But with an update like the above, only one will win. The other will update nothing and be informed of this fact.
[21:44:39] <dscastro> what is that mean: failed with error 57: "The dotted field 'haproxy-1.4/web_proxy' in 'pending_op_groups.0.pending_ops..sub_pub_info.haproxy-1.4/web_proxy' is not valid for storage."
[21:47:36] <GothAlice> dscastro: No, likely what you were doing wasn't expressly checked or forbidden before, but was still documented as being a no-no. (I.e. it let you get away with it, but you're still a bad person for trying to get away with it. ;)
[21:48:36] <GothAlice> dscastro: Someone using MongoDB should follow their language's variable naming convention. For most languages, this means the regex: [a-zA-Z_][a-zA-Z0-9_]*
[21:49:08] <GothAlice> With your field, you can't access it using attribute notation. (foo.haproxy-1.4/web_proxy would be interpreted as foo.haproxy minus 1.4 divided by the contents of the web_proxy variable.)
[21:53:17] <GothAlice> So, uh, the layer designed to abstract things and make your life easier is, in this case, making your life harder.
[21:55:13] <GothAlice> Little-appreciated fact: MongoDB, being schemaless, must store the _names_ of every field in each document that uses that field. Your key adds 20 bytes (beyond a single-letter key) per document that uses it. Something to consider. ;) (I use one- or two-character field names and let my ODM abstraction layer do its thing.)
[21:55:58] <GothAlice> dscastro: Moving forward, fixing your situation will require $rename'ing the fields, then trying to upgrade again.
[21:57:24] <GothAlice> dscastro: Are any of the freakishly-named fields indexed? If so, you're going to _have_ to reindex after re-naming in order to upgrade. Bogus field names would remain bogus, and current versions aren't missing the assert().
[22:17:17] <dscastro> GothAlice: what about this: /usr/bin/mongod: symbol lookup error: /usr/bin/mongod: undefined symbol: _ZN7pcrecpp2RE4InitEPKcPKNS_10RE_OptionsE
[22:18:26] <GothAlice> dscastro: Hmm, the symbol name is gibberish to me, but that typically means you have MongoDB compiled for a library not present on your system. I.e. you installed the Ubuntu version on Debian, etc. I compile MongoDB directly for my cluster, so that's rarely an issue for me.
[22:20:33] <GothAlice> (Also because I want SSL support, and am too cheap to buy MongoDB Enterprise. ;)
[22:21:58] <GothAlice> mocx: Indeed. As with the 16MB document size limit, if you're hitting it, you're probably doing something wrong in the design of your schema.
[22:22:37] <GothAlice> mocx: Correct. See also: http://www.javaworld.com/article/2088406/enterprise-java/how-to-screw-up-your-mongodb-schema-design.html
[22:31:12] <bendyeraus> hello, I’m optimising my queries, and was wondering to achieve a covered query why do I need to supress _id? sure the _id is in the index? how else is the index linked to the documents in the heap?
[22:33:09] <GothAlice> bendyeraus: Specifically, you need all of the returned fields to be stored in the single index that gets used for the query. Unless you're including _id in your "covered" index, you'll need to exclude the field during projection.
[22:34:34] <bendyeraus> Hi GothAlice, thanks, my question was more to do with if _id is not automatically in the index, how is the index linked to the documents in the heap?
[22:35:09] <ParkerJohnston> public static MongoCursor<Document> getAllActiveHistoryByCaseNumber(String caseNumber) {
[22:35:10] <ParkerJohnston> return caseInformation.find(and(eq("casenumber", caseNumber), eq("history.active", true))).iterator();
[22:35:30] <GothAlice> bendyeraus: Behind-the-scenes is a complex BTree bucketing process. Yes, there is some form of association between an individual index value and the record it came from, but it might not be the ObjectId. (It might be the stripe ID and index into the on-disk stripe of the record… might not be.)
[22:35:31] <ParkerJohnston> this is returning all embed docs in a single item not in a hasNext format so i cant iterate over them
[22:35:44] <GothAlice> bendyeraus: Thus unless you explicitly include _id in the list of fields, don't expect it to be there.
[22:39:24] <cheeser> a collection has documents and when you fetch that document you get everything inside it regardless of whether it could be considered an embedded document or not.
[22:39:26] <ParkerJohnston> but they are stored as an array why cant i get them back out the same way
[22:40:04] <cheeser> can you pastebin what you're betting back and how it doesn't match expectations?
[22:43:55] <bendyeraus> While I’m here one other question I was thinking about, for optimal count queires, should I be doing a find query, projecting the fields I know will cause a covered query and then calling count, ie. find({x:123}, {x:1,_id:0}).count(). or is count smart enough to only access the index ie. is count({x:123}) enough?
[22:50:40] <ParkerJohnston> the issue is that when i get the results and try to use history.next() to print out the results it does not print ot two lines it is one large cluster
[22:53:05] <cheeser> you're getting the document from the collection. the history array would be nested inside that document.
[22:57:37] <ParkerJohnston> MongoCursor<Document> history = History.getAllActiveHistoryByCaseNumber("2015-REP-01-0001");