Log file Viewer
#mongodb logs for Thursday the 14th of July, 2016
[01:42:14] <ARandomFurry> I'm getting a max call stack size exceeded error when calling myCursor.next(next); It bounces between serializeInto and serializeObject. Any thoughts?
[01:50:41] <Trinity> Also if I am starting with a fresh DB should I allow my application to enter the first documents or should I enter test documents first?
[01:56:45] <joannac> indexing earlier means queries can be satisfied with an index, but your writes will be slower because the document and index entries need to be written. It's a tradeoff and is your own decision about when it would be appropriate
[01:57:17] <Trinity> joannac, so if I index later will the indexing just take longer or will older documents not be indexed?
[01:59:03] <Trinity> joannac, also, in the documentation atomicity and transactions mongo docs states, Using two-phase commit ensures data consistency, but it is possible for applications to return intermediate data during the two-phase commit or rollback.
[02:00:01] <Trinity> from what I understand, two-phase commits are basically writing atomically to a journal entry of the next operation you'll be doing
[02:00:01] <joannac> well, if you're halfway through updating, a clinet could theoretically see the intermediate data
[02:02:58] <joannac> so nother client can connect, and read after write 2 and before write 3 and hence - intermediate data
[02:03:15] <Trinity> joannac, okay, and to confirm messing up usually means the database crashed or the server it's hosted on is down right?
[02:46:17] <Trinity> joannac, If multiple clients write to a standalone db they are queued up right?
[03:25:42] <troydm> hey all! I've created my function but when I call it I get myFun is not a function, what's wrong?
[03:27:14] <Trinity> troydm, doesnt seem like a mongo thing and more so a javascript or another language thing
[03:29:34] <troydm> Trinity: nvm, figured it out, well now I'm trying to use sqrt but it seems mongodb doesn't has this function
[03:42:56] <charginghawk> can I have an '_id' like 'blog', or does it have to be a string of certain length, etc
[03:44:46] <Skaag> Hi guys, I'm trying to check what type of object is a certain field. I call: typeof db.test.age and it tells me it's an object
[04:07:24] <Boomtime> @Skaag: you are experiencing a javascript limitation (just FYI, this is not a MongoDB question in any way) -- typeof tells you the javascript type, and since javascript has no classes there are very few actual types
[06:30:34] <Boomtime> -> "This page assumes prior knowledge of TLS/SSL as well as access to valid certificates."
[06:30:54] <Boomtime> if this is the first time you've done anything with SSL/TLS then you're going to have a bad time
[06:31:27] <Boomtime> you need to go and read up on how exactly SSL works and then you'll have a chance of understanding the docs on how to set it up in mongodb (or anything else for that matter)
[06:33:33] <Boomtime> when you can generate a certificate, then search for "how to validate certificates with openssl"
[06:34:09] <Boomtime> once you get pretty good at doing those steps with the openssl toolkit you'll pretty much understand what all the parameters are to mongodb ssl
[06:35:01] <thapakazi> I have it all I guess, a generator too(yesteray I made one) https://gist.github.com/4b7d6f85d4e39bb06825e5457f9c4e1c
[06:35:57] <Boomtime> that's a self-signed certificate - you could use that as your own CA - search goolge for "how to sign your own certificates with openssl"
[06:41:01] <JeansenVaars> Hi All. Trying to test the Scala Driver here in Linux. Mongo server is running and I can connect through shell, but failing to connect through Scala code. Any assistance appreciated
[06:45:26] <thapakazi> i wish I know Scala JeansenVaars, provide more context, if you are connecting with authentication or its open like stuffs for more appropirate helps
[07:05:42] <Trinity> why doesn't mongo offer transactions? it seems pretty easy to implement. This blog I came across is the best implementation i've seen so far http://trendblog.net/fix-soft-bricked-android-device-first-aid-guide/
[07:09:20] <Boomtime> @Trinity: your statement of claim is not based in reality, nor is the URL even remotely related from the look of it
[07:09:43] <Boomtime> also, MongoDB is an open source database, if transactions are easy to implement you should contribute a patch
[07:14:35] <Trinity> Boomtime, well i'd think the basic idea would be similar to how two-phase commits work right now. Start a transaction via driver method, add transactional steps, commit the transactional steps so it's written to some collection in database atomically. And run through each transactional step. If an error occurs it's all logged into the relevant document inside the transaction collection
[07:15:07] <Trinity> so you can rollback easily. I guess what i'm talking about is a library to abstract away two-phase commits
[07:18:13] <Boomtime> apparently you thought i didn't know how transactions work -- anyway, you should implement it and make a pull request, it's easy right?
[07:18:42] <Trinity> Boomtime, why are you so offended? Nowhere did I imply that you did not know how transactions worked.
[07:18:50] <Boomtime> also, are you aware that mongodb is a distributed database? how will you rollback a partial transaction between potentially thousands of nodes?
[07:19:52] <Boomtime> i'm offended because generally anybody who says in their opening statement that a specific feature implementation is "easy" invariably has no idea what they're talking about -- be less arrogant and i'll be more friendly
[07:20:53] <Trinity> I started off with a question, implying that I did not know why something was not done. There was no arrogance there. The only arrogance is the implied arrogance that you brought to the table
[07:21:38] <Boomtime> -> you're seriously claiming this was a simple question -> "why doesn't mongo offer transactions? it seems pretty easy to implement."
[07:23:11] <Boomtime> mongodb is a distributed system -- theoretically it would be possible to implement transactions in a constrained way
[07:24:15] <Boomtime> but in a useful way, like the open ended description you gave, would not be realistic -- admittedly it could still be done, but it would be dog slow due to having to coordinate so many interactions
[07:24:50] <Boomtime> fact of life is that the pacific is big, the speed of light exists, and mongodb can scale across continents
[07:25:36] <Boomtime> a transaction that could pull in and involve multiple members synchronising a commit across that distance would more often fail than succeed -- rollbacks would suck up half your resources
[07:26:06] <Boomtime> again, it could be done -- but a slightly careful design by the application can usually avoid any need for transactions
[07:26:55] <Boomtime> transactions are 99% of the time lazy programming -- there are some use cases that would be useful, that's a pain point but application document locks can get around it
[07:27:31] <Trinity> Boomtime, would you mind elaborating? How could you guarantee atomicity/ability to rollback without transactions and through careful design?
[07:28:38] <Boomtime> you make your changes idempotent -- if you can't do that then you keep a ledger
[07:29:12] <Boomtime> i.e let us say you want to update a balance by "subtracting 5 dollars" -- your dependency on setting a new balance should be your understanding of the current balance
[07:31:04] <Boomtime> it isn't -- you can implement this -- but if you do it in a general way (say in a driver) then you don't know where the data resides
[07:32:14] <Trinity> Well with the driver it would just be abstracted away right? Not something you need to be concerned about. And even then the driver has to store that information somewhere albeit in a hidden collection or something
[07:32:23] <Boomtime> generally, a two-phase commit needs to use a very pessimistic model and performance is based on the idea that everything is happening in a close-by location -- btw, it can be mathematically proven that two seperate systems can never come to an agreement
[07:33:14] <Boomtime> two phase commit, in the way you're using it, is intended to solve this problem: https://en.wikipedia.org/wiki/Two_Generals%27_Problem
[07:33:17] <Trinity> I dont follow on that last statement. Everything is happening in a close-by location?
[07:33:47] <Boomtime> if you have a ledger in one system, and the updated record in another system, like say in a distributed system like mongodb...
[07:34:30] <Boomtime> you can push the system into the future, but you can never guarantee that two systems commit a singular datum at the same time
[07:35:50] <Boomtime> but if these are distributed, as could happen in a general solution implmented in the driver of a distributed database product, it is mathematically impossible
[07:36:34] <Boomtime> and you can implement it yourself, if you're careful and you understand your data model well
[07:36:41] <Trinity> so the gist of what you're saying is that it's possible but by abstracting it away you lose performance
[07:37:18] <Boomtime> that would be understating it significantly -- you wouldn't have a useful system at the end of it
[08:22:17] <kurushiyama> Trinity I have yet to find a use case where transactional integrity could not be achieved by clever data modelling – remember, document level operations are atomic.
[08:23:14] <Trinity> kurushiyama, would you mind giving me an example? Everything I can think of revolves around the concept behind two-phase commits
[08:24:00] <Trinity> For example my current issue: I have a physicalServer that can have 4-8 virtualHosts depending on how many cpu cores are available
[08:24:07] <kurushiyama> The general example in the SQL world does this: it deducts an amount of account X and adds it to account Y
[08:24:43] <Trinity> virtualServer and physicalServer are in different collections and each "server' would be a separate document
[08:25:42] <kurushiyama> Trinity This typical SQL approach lacks a crucial information: When was which amount transferred?
[08:26:46] <kurushiyama> Trinity So what you would do is to _document_ the transaction: {time: someISODate, amount:100, source: accountX, destination: accountY}
[08:27:37] <kurushiyama> Trinity Now, to get the balance of an account, you'd do an aggregation over the transactions. Problem solved. Ofc, this is a bit simplified, but it should be sufficient to get the picture.
[08:28:34] <Trinity> Isn't that a basic two-phase commit? Also if you wouldn't mind me asking how did you see my post from before O.o you joined recently
[08:29:38] <kurushiyama> Trinity db.transactions.insert({time: someISODate, amount:100, source: accountX, destination: accountY})
[08:30:51] <Trinity> kurushiyama, right, but wouldnt you still need to modify the documents of accountX and accountY (assuming they are stored in separate non-embedded documents) to reflect the balance?
[08:31:08] <kurushiyama> Trinity Also my example was a simple start to showcase that by smart data modelling, you can achieve a lot.
[08:31:59] <kurushiyama> Trinity You would not store the balance – you'd calculate it. Ever wondered why ATMs need some time before cashing out the money you asked for? Because this is what banks do, even with SQL databases. They calculate the balance.
[08:33:43] <kurushiyama> Trinity What you _could_ do is for example a daily preaggreation to get the balance of the account as of today, 00:00:00.000, and when doing a balance calculation for now, you'd only need to add and deduct today's transactions.
[08:34:25] <kurushiyama> Trinity But since you would do the aggregation of transactions within a set time, there again is no need for transactions – the operation is idempotent.
[08:36:38] <kurushiyama> Trinity From first look, I'd say you overnormalized. That happens. It depends on the use case how small the path between overnormalizing and overembedding is.
[08:39:45] <kurushiyama> Trinity It would be helpful if you described what you want to _do_ so that we can have a look wether your data model fits your use case.
[08:50:29] <Trinity> kurushiyama, sorry, I got called in for something could you give me a little time?
[09:20:37] <Trinity> kurushiyama, sorry for the delay. The gist of it is... I have a document named physicalServer that can have virtualServers. Each virtualServer will belong to a user. A vitualServer can only be created if a CPU is available on the physicalServer. A physicalServer only contains 8 available CPUs when it is first instantiated. So the general process looks similar to this, application needs to check for documents in the physicalServer collection to see
[09:20:38] <Trinity> which ones have available CPUs. It will select at random a physicalServer to use. It will then need to update the corresponding physicalServer's document to reflect the newly used CPUs and the new virtualServers that has been included so that if another virtualServer is added it will not select CPUs that have already been reserved. This virtualServer's meta data will then need to be added to the virtualServer collection and also be added to the u
[09:21:22] <Trinity> My currently implementation goes as following: Add all information regarding this transaction to a transaction document and then proceed with the updates while updating the transaction as necessary for each step this way I can rollback if anything occurrs
[09:22:02] <Trinity> what I am confused about is whether or not this method would work for a replicated or sharded db although I have no immediate plans for sharding but I may consider replication.
[09:24:30] <Trinity> kurushiyama, but they wouldn't be related and virtualServers may include other meta data that would be completely unrelated to physicalServer. Perhaps i'm thinking too much in the traditional RDBMS way
[09:25:54] <Trinity> the only relation between virtualServers and physicalServers are that they are a part of physicalServers. And most data writes/reads stem from virtualServers rather than physicalServers
[09:27:47] <Trinity> kurushiyama, furthermore I will still need to update the 'user' collection so i'll still need to utilize some sort of two-phase commit
[09:30:46] <kurushiyama> Trinity Well, that is only partially true: The data of the virtual servers is related to the physical server, as it is related to a machine running on that server.
[09:31:50] <Trinity> I guess it would be more accurate to say virtualProcess as that is what the user actually gets
[09:34:00] <Trinity> hmm interesting. I was going to argue that you'll then need more reads but it would end up being the same I suppose
[09:34:54] <Trinity> kurushiyama, so what would you define as reasons to use separate collections vs embedded documents?
[09:35:23] <kurushiyama> Trinity http://blog.mahlberg.io/blog/2015/11/05/data-modelling-for-mongodb/
[09:38:28] <Trinity> kurushiyama, so from that article, if you expect an embedded document to blow up in size put it in its own collection?
[09:39:10] <kurushiyama> Trinity Not quite. if you have a 1:n, for large to potentially unlimited sizes off n, I'd use a collection
[09:48:38] <kurushiyama> Trinity Or in case the subdocs for small sizes of N might grow indefinetly.
[11:06:21] <ArGGu^^> Hello, what is the best way to calculate size of specifig documents in all collections?
[11:06:57] <kurushiyama> ren0v0 Use facilities for dealing with logs which are built for that purpose.
[11:06:58] <ren0v0> kurushiyama, sure, but why include it at all if its not automatic or configurable
[11:09:21] <ArGGu^^> Derick project that I am working currently, is multisite. And I would like to know how much each site consumes storage.
[11:11:09] <ArGGu^^> constantly, I would store the information somewhere and update always when new data is inserted in database.
[11:11:58] <Derick> ArGGu^^: it would be a very slow operation - I would recommend just using multiple databases
[11:14:58] <ArGGu^^> Derick but when I update the values, I only need to update the changed. So would that be slow? I mean I could insert new document and then just check it size and add it into storage counter.
[11:15:55] <Derick> ArGGu^^: it's really figetty to get right, where the data size is just a property to read for a database (or rather, the collections that make them up)
[11:17:52] <ArGGu^^> Derick you mean the document size that example Object.bsonsize returns, would not be equal to the bytes on disk?
[11:20:08] <ArGGu^^> Derick because if it would be then I just update single value per site when I insert/update/replace/remove in database.
[12:17:43] <atbe> Does anyone else have experience with creating a MongoClient using an ipv6 address?
[13:29:00] <atbe> cheeser: http://stackoverflow.com/questions/38375308/mongoclient-instantiation-using-ipv6-address-throws-mongoexception-host-and-po
[13:30:05] <cheeser> yeah. for some odd reason we're reparsing those addresses. I'm asking the other guys why that it is.
[15:03:22] <benjwadams> Newish to mongo. You can index a field on an embedded array of subdocuments, correct?
[15:04:16] <benjwadams> Thanks. I'm weighing the old pros/cons of embedded subdocs vs separate collections
[16:13:11] <cr0mulent> Hello, I am new to mongodb. I am looking to query an existing database which has keys where the value references the objectid in another collection. For instance, language : {ObjectId: …} is this considered a foreign key in mongodb parlance? Could someone point in the right direction for joining values from two collections?
[16:14:42] <cheeser> you can query by that key value but joins aren't entirely a mongodb thing. there is $lookup, though, https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/
[16:19:05] <havardge> Hi, I am seeing slow lookups on id in my application. I use the current release of Spring Data in a Spring Application.
[16:21:10] <havardge> Can someone point me in the right direction of figuring this out, perhaps it is even as expected?
[16:23:23] <havardge> It seems strange that 150 findOne(...) queries on an indexed field should take 1000ms
[16:28:00] <atbe> cheeser: I understand that this fix will only make it's way in after the next release, but is there any way I can get a build with the fix integrated earlier than that?
[16:37:59] <cheeser> you can use a SNAPSHOT build if you're comfortable with that. or build a local copy.
[16:39:10] <xQuasar> Hi, apologies if this isn't the right place, but I'm working with MongoDB 3.2 in Java and I wanted to only select a few fields from my "clusters" collection, so I used projections as follows:
[16:39:19] <xQuasar> FindIterable<Document> clusters = clustersCollection.find(currentTradesFilter).projection(Projections.include("trades.$", CLUSTER_STRING, CLUSTER_NUMBER));
[16:40:00] <xQuasar> However, the projection appears to limit the number of results in my trades array ("trades.$") to 1
[16:40:25] <xQuasar> when there should be a few more array elements in trades[] that I want to be returned
[16:40:31] <atbe> I'm comfortable with a snapshot build, if all it includes is the fix on top of 3.2.2. Has the fix been confirmed working? cheeser
[16:51:48] <atbe> cheeser: no worries, thank you for following up and getting it checked out. I will wait for an official release just to be safe and so my qa guys can test it thoroughly.
[17:32:26] <oded> I'm having a replicaset, but the client using mongoid doesn't seem to care that the primary is gone and gets execution expired errors instead of switching to the new primary
[17:37:52] <kali> oded: most of the time, the driver does not proactively reconfigure itself. it just let the failing connection go and open new ones with the reconfigured replicat.
[17:38:54] <kali> oded: so you should get a few errors, probably as many as the number of open connection the driver is keeping in its pool, but then you should be fine
[17:39:41] <oded> kali: ok so I actually removed the downed server from the client configuration and restarted it, and I'm still getting "MONGODB | execution expired"
[17:40:07] <oded> I'm seeing at the backup server (this is what I'm testing now) that the client has connected though
[19:31:27] <ceecko> hi, I'm looking for help with a strange issue. I have 1 member replica set configured and can connect via mongo shell, however via the node driver I always get error "MongoError: no primary found in replicaset". Everything worked correctly up until now and we only ran db.repairDatabase() in the meantime. Any ideas?
[19:48:55] <checklist> Hi -- I have a quick question about schema configuration, and I was wondering if anyone had any suggestions?
[19:50:08] <ceecko> cheeser: yeah, I'm just looking at mongodb-core if there wasn't any changes for one member replicaSets...
[19:50:26] <checklist> Free SO upvotes for anyone that wants it http://stackoverflow.com/questions/38381034/checkbox-option-type-monodb-schema-design
[20:04:39] <ceecko> cheeser: I just discovered this is an issue of the latest Mongodb node driver. The issue was not present in older version and I've already opened a ticket in Jira NODE-758
[20:12:00] <Karbowiak> hey guys, i'm rather new to this whole mongodb ordeal, and i'm trying to update a field (a timestamp) that is mistakenly set as a string, to a Date field, and i'm not 100% sure i'm doing this correctly, but so far this is the query i've put together: db.getCollection('killmails').updateMany({"killTime": {$type: "string"}}, {$set: {"killTime": new D
[20:24:53] <Karbowiak> was actually easier than i expected, still not sql easy, but easier than i would've figured :D
[21:04:23] <hdon> hi all :) i'm reading http://blog.mongodb.org/post/88473035333/6-rules-of-thumb-for-mongodb-schema-design-part-3 and i want to know if there any frameworks out there to help abstract the application away from the exact structure of your schema. if i change my mind about denormalizing some particular relationship, how can i avoid rewriting my application code?
[23:42:57] <MacWinner> i think i have a racecondition where 2 updates with upserts come in at the same time.. the second one creates a new document instead of updating the first document
[23:43:50] <MacWinner> ahh.. so the second query could perform the search before the first query actually inserts the data?
[23:44:29] <MacWinner> is there a simple solution the ensure the second update always modifies the first document?
[23:46:35] <MacWinner> got it, thanks! i should read teh docs.. it's clearly stated there: "To avoid inserting the same document more than once, only use upsert: true if the query field is uniquely indexed."