Log file Viewer
#mongodb logs for Thursday the 18th of February, 2016
[00:04:45] <asteele> hi i did a mongodump (No params) on my old db, and now i got the files over scp and did mongorestore, and it seems to have created all of the collections but has no data inside, any clue why? i am pretty sure the data is in the dump, because some fof the files were hundreds of mb
[01:52:26] <chrisbolton> I’m trying to atomically update a document, find an embedded document by id and then update that document as well, one call. Any ideas?
[01:52:51] <chrisbolton> I need the document and the embedded document to be updated at the same time.
[02:01:07] <chrisbolton> nyexpress: thanks, I appreciate the help. I’ve been googling and testing for a while now.
[02:08:30] <chrisbolton> http://stackoverflow.com/questions/35471408/mongoid-atomic-inc-and-push-on-nested-document-in-array
[03:00:02] <rickyw59> Is it possible to have a compound shard key and tag aware sharding on one of the fields of the compound shard key?
[03:30:43] <cookiepus> Quick q: I am looking to allow item re-ordering in my app (each item is a document in mongo) - what's a good way to represent the ordering in the DB? it's all about relative ordering.
[05:06:35] <saira_123> hi professionals , i want to run mongodb 2.6.11 and 3 on Solaris 10, but it seems the binaries available on website are for solaris 11, can someone please guide me
[05:12:15] <kurushiyama> saira_123: You might be able to build from source. But I am not too confident.
[05:21:38] <saira_123> khurushiyama bundle of thanks for reply , i am trying to build for past few days but everytime it breaks somewhere, wish i could find already build packages from somewhere
[05:24:43] <kurushiyama> saira_123: The main problem will be glibc. I am not overconfident that a matching package is available for the one required by mongodb.
[05:26:17] <kurushiyama> BUT: If you plan on using MongoDB in production, it is a bad idea to have a shared machine, anyway
[05:27:00] <kurushiyama> saira_123: So as long as running Solaris is not a hard requirement, I'd use a dedicated Linux machine.
[05:33:17] <saira_123> kurushiyama Bundle of thanks , i am going to wipe away solaris and install RHEL
[05:33:58] <kurushiyama> saira_123: That would have been my suggestion for a Linux distribution. However, you can try with CentOS, first, if tight on license.
[05:34:46] <saira_123> kurushiyama actually i want it for my student research project, i am comparing pefromance of mongo versions. i hope it does count as a research :)
[05:37:14] <saira_123> kurushiyama: i am Masters student and my thesis is to publish a research in journal, having a little experience on mongo i want to do something on it that counts as research and can be published
[05:37:29] <saira_123> kurushiyama i am Masters student and my thesis is to publish a research in journal, having a little experience on mongo i want to do something on it that counts as research and can be published
[05:38:21] <saira_123> kurushiyama do you think extending this article research on sharded cluster will count as research? mongodb.com/blog/post/performance-testing-mongodb-30-part-1-throughput-improvements-measured-ycsb
[05:49:27] <kryo_> hi, i have a collection with an array field "items" which contains Objects with an ID. given an array of IDs i want to find a document where the items match exactly
[05:50:33] <kryo_> on the mongodb site it says that the order of the array matters, anything i could do to get around that?
[06:10:04] <Boomtime> i just quickly tested this in 3.2 and it worked fine - if i specify an array in the find (with no operators) it must exactly match
[06:10:31] <Boomtime> can you provide a sample document / find and output from such, in a gist showing the "didn't work" part?
[06:27:43] <kurushiyama> Well, it stems from there. But I havent needed a single line of JS to deal with it, so far ;)
[06:29:05] <kurushiyama> It sure is. However, the use case is a bit opaque. Why not call that API directly? If it is volatile data, maybe redis is more appropriate?
[06:30:02] <kurushiyama> I do not have a clue on what item is supposed to mean. What are you storing there and why? What do you want to extract from that data?
[06:37:53] <kryo_> also if i extracted them to their own collection i would have to write a mongoose schema for it
[06:40:12] <kryo_> kurushiyama: here is an example of one of them https://gist.github.com/kryogenic/65f5d7eed4a9f717c5d6
[06:43:35] <kurushiyama> Well, here is whatI think. When _queries_ get complicated beyond being simple comparisons, they tend to get slow(ish). So I prevent that at almost all cost. Embedding more often than not is just convenient.
[06:48:34] <kryo_> i updated the gist with a full schema example, with only one item in the array.. still doesn't work :S https://gist.github.com/kryogenic/d9b2c47b4099bfaf2605
[06:49:34] <kryo_> kurushiyama: i'm trying to do a simple query on the ID, which is an indexed value
[06:53:04] <kurushiyama> Well, it sure is. But remember what you asked? You need to compare multiple values against the index. ;)
[06:57:37] <Boomtime> kryo_: your query does not match - the document you quote does not have a field 'id' inside a subdocument of 'item' that _exactly_ matches ["1260932633"]
[06:58:43] <kryo_> Boomtime: this gist right? https://gist.github.com/kryogenic/d9b2c47b4099bfaf2605
[07:00:28] <Boomtime> ok, following the gist you just posted your query is: db.inventory.findOne({'items.id': ["1260932633"]})
[07:01:37] <Boomtime> you are specifying an exact match - the only reason it works without the array specifiers in that one case is due to the fact that the first field in that specific document ('item') is an array which changes how the server permits a match
[07:02:41] <kryo_> i need something like this db.inventory.find({'items.id': {$only: ["1260932633"]}})
[07:04:20] <kryo_> yeah i think the aggregation framework could do it https://docs.mongodb.org/manual/reference/operator/aggregation/setEquals/#exp._S_setEquals
[07:05:42] <kryo_> i'm thinking maybe i should just store an extra field called "item_ids" which is just a sorted string representation of the ids
[07:07:42] <Boomtime> yeah, can't update with aggregation - but if you can locate it, you can still issue the update with predicates that exactly match to ensure it has not changed since you got the answer
[09:26:06] <vak> my collection records have a structure { tiny_field1, tiny_field2, tiny_field3, huge_field}. It looks like, indexing doesn't help to retrieve fast the tiny_fields from HDDs, because the huge_field makes records to be spread over separate disk sectors... how to deal with this case without splitting the collection in two?
[09:28:48] <vak> "touch" command seem to pre-cache all fields and therefore will touch all the huge_field data that only needed on request
[10:32:14] <Folkol> About WiredTiger: "When the storage engine detects conflicts between two operations, one will incur a write conflict causing MongoDB to transparently retry that operation." Who is doing this retry? Can we configure Mongo to not retry? (Or can we add custom logic to handle the conflict resolution?) It seems like a weird use case to just retry, it will either be conflict again -- or overwrite the other writers' document.
[10:34:13] <vak> kurushiyama: i list my 3 tiny_fields explicitly, so i expect it is already a projection, it is just defined in different form... false?
[10:35:04] <kurushiyama> vak: Nope, should work, too, If it is still slow, you might want to see the output of .explain()
[10:39:03] <kurushiyama> vak: I dont see in which way this should help. Review the output of .explain(). If your query is properly indexed (that is, al queried field are contained within the index, there is no need to read the large document. And even if, the large field should not be read. Furthermore, documents are guaranteed not to be fragmented.
[10:54:29] <vak> kurushiyama: it looks like a corresponding index was not taken, albeit the projection is exactly the one that is indexed as a compound index...
[10:55:21] <kurushiyama> vak: Put it to pastebin or somewhere (db.getIndices() and output of .explain() ) and I'll have a look.
[12:08:25] <CustosLimen> so when I run: mongo localhost:27017 --eval "db_admin = db.getSiblingDB( \"admin\" ); result = db_admin.runCommand( { logRotate : 1 } ); print( tojson( result ) );" the file just dissapears
[12:27:28] <pac1> on mongo there's a command that's related to the system logrotate. It is different. Also this was changed in 3.0.0.
[12:28:51] <pac1> mongo's log rotate can, depending on settings, simply close and reopen the log file, resulting in an empty log file, with no rotation.
[12:32:32] <kurushiyama> pac1: It does not really rotate logs, not in the sense of logrotate, the Linux command. You can either use syslog or write to a logfile and configure logrotate accordingly, using SIGUSR1 to instruct mongod to create a new logfile, renaming the old, which you have to deal with, then,
[12:57:00] <pac1> so nothing in mongo's "logrotate" clears the old logs. They'll just pile up. maybe better to configure a real logrotate with it's many options.
[13:25:48] <Haakon_> hello :) anybody that can help me combine these into one query: db.documents.find({month:{$gt:0 , $lt:5},year:2015}) and db.documents.find({month:{$gt:9 , $lt:12},year:2014})
[13:44:36] <pokEarl> http://stackoverflow.com/questions/20821721/multi-range-query-in-mongodb think that might help you Haakon_ ?
[13:45:37] <pokEarl> Anyway hi friends, have a problem a bit out of my depth but I'll describe my understanding of the situation and maybe someone can help out. We are having issues with Mongo Connections not closing in a Java application, the Java application uses Guice for dependency injection. The Guice looks something like this: http://pastebin.com/gZLhth2z
[13:45:53] <pokEarl> From my understanding if this worked properly, it would make sure that there only ever exists 1 instance of MongoClient. Which controls all different collections/database, and if you do mongoClient.close() on that client, it should close all current connections. Right?
[14:04:25] <Ben_1> (the insertion itself needs much memory but after the insertion it should be freed)
[14:04:48] <Ben_1> so I closed all my applications, also my java application that sent the data to mongoDB
[14:07:59] <StephenLynx> the _id will take a much greater space proportionally than if you had just the _id and a big unindexed field
[14:08:18] <Ben_1> so I have physical devices and every device have several sensors, so I inserted a collection for every device and inserted a document for every sensor
[14:09:40] <Ben_1> yes every document have one field named _id and is filled with a string like component:sensorname:timestamp
[14:10:24] <Haakon_> pokEarl: hmmm am i reading it wrong or do that example check on different values of only one thing?
[14:11:19] <StephenLynx> since your documents are small, I think they will use more RAM for the same size than if you had larger documents.
[14:11:46] <Ben_1> StephenLynx: collections: 4273 avgObjSize: 124.011816838996 dataSize:3587439880 storageSize: 962543616
[14:13:33] <StephenLynx> that's dynamic collection creation and I am pretty sure most of those collection have the same structure.
[14:14:36] <Ben_1> no, one collection could have the same structure as another but it's not guaranteed
[14:16:19] <StephenLynx> if your system had a couple of decades and were badly maintained then maybe it could have achieved dozens or hundreds of collections. you don't start right away with thousands by design.
[14:17:38] <StephenLynx> try with a single collection with a field for the device type instead and check how much RAM is consumed
[14:17:51] <Ben_1> I use the serialnumber of that device to name the collections. So it is better to have one collections and user _id: device:component:sensor:timestamp?
[14:20:28] <StephenLynx> when you throw everything on that string you can't create indexes nor query properly, Ben_1
[14:20:35] <Haakon_> If someone can explain or give me the query to combine the 2 i will be very happy :) db.documents.find({month:{$gt:9 , $lt:12},year:2014}) and db.documents.find({month:{$gt:0 , $lt:5},year:2015})
[14:20:40] <Ben_1> StephenLynx: I read in the mongo documentation such a single _id field is very good for search operations because I can use regex for queries
[14:26:42] <Ben_1> StephenLynx: If an index exists for the field, then MongoDB matches the regular expression against the values in the index, which can be faster than a collection scan.
[14:29:01] <geloescht> Hi, I am trying to use the projection operator $elemMatch in an exclusive projection like so: { componentRegistered: 0, localization: { '$elemMatch': { language: 'en' } } }. However I do not get any fields except for localization and _id, as if it was an inclusive projection. Is this the expected behaviour?
[14:31:17] <StephenLynx> and this is not just a subjective opinion because of standards like comprehensibility, you are losing performance here.
[14:31:38] <Ben_1> ok you say it is faster to store that data in another field inside my document and if I'm searching for sensorx and june 2005 it is faster than the regex in my _id?
[14:32:57] <Ben_1> that makes no sense because mongoDB searches inside every document for that values
[14:35:15] <Ben_1> I believe what you say, but I have to read the documentation to learn how it works in detail
[14:35:24] <GothAlice> geloescht: That is intended behaviour; explicit projection = "give me just these fields".
[14:36:53] <geloescht> GothAlice: OK, but shouldn't MongoDB return an error in this case, because exclusive projections have no effect when combined with $elemMatch?
[14:37:32] <StephenLynx> he is thinking of storing several fields on a single string and search using regex instead of having multiple fields.
[14:39:59] <GothAlice> Ben_1: String indexes _can_ use regexen in certain situations, such as prefix searching, but finding deep values can't really use the index in any meaningful way. Using such a large string as _id, which is automatically indexed, is a _massive_ waste of resources.
[14:40:17] <GothAlice> (Explains memory usage, too, as for performance reasons MongoDB tries to keep indexes in RAM.)
[14:41:31] <geloescht> GothAlice: does your document have other keys apart from extra and name? Like maybe 'party'. I'd expect that field to be returned when doing an exclusive projection.
[14:42:23] <GothAlice> geloescht: Ref: https://docs.mongodb.org/manual/core/read-operations-introduction/#projections
[14:43:35] <GothAlice> "…may either specify a list of fields to return or list fields to exclude in the result documents." If you have any "1" values in your projection, it excludes all fields except the ones you mark with "1" values. If you only have "0" values, it does the reverse. $elemMatch is an effectively "1" value.
[14:44:58] <geloescht> GothAlice: huh, it seems to work in the mongo shell, but not from my node app. Maybe mongoose is doing funny stuff.
[14:45:31] <DrPeeper> referring to GeoJSON, would it be best to create a separate collection in the database to house the _id and the {type: "Polygon", coordinates: [[[]]] } data? or just index them in the original collection
[14:46:04] <Ben_1> StephenLynx: I'm using a paper as basis for my work and I thought he was experienced in that field but he missunderstood the usage of compound indexes I think
[14:46:18] <GothAlice> geloescht: If you need schema validation, use MongoDB's schema validation: https://docs.mongodb.org/manual/core/document-validation/
[14:46:27] <geloescht> GothAlice: I am gradually figuring that out now. Unfortunately I do not have that liberty...
[14:48:49] <GothAlice> DrPeeper: If you separate them, you'll need additional queries to fake join. Why _not_ embed?
[14:48:50] <geloescht> Speaking of... Is there a good way to log raw server communication in the node driver in a human-readable format? I only get stuff like "[DEBUG-Server:26553] 1455803967843 message [2400... snip...0000] received from localhost:27017"
[14:49:30] <Haakon_> can someone help me please :) just message me. i have 2 queries that i have to combine into one :)
[14:49:33] <DrPeeper> GothAlice: I don't have a reason not to, that is why I was asking if it would be better to have a separate collection or just leave as is :)
[14:50:08] <GothAlice> DrPeeper: http://www.javaworld.com/article/2088406/enterprise-java/how-to-screw-up-your-mongodb-schema-design.html is the general summary I often link describing when split out might be a good idea. Generally its not. :)
[14:50:14] <geloescht> Tried to decode the hex part in brackets as BSON, but that seems to give me mostly empty objects.
[14:52:47] <GothAlice> DrPeeper: Generally, MongoDB encourages you to structure your data in a way which optimizes use, rather than preferring some "perfect" structure. This can lead to what might look like data duplication, that is in fact optimization to save on making recursive queries application-side. I.e. thread = {_id: …, title: "Awesome post.", forum: {_id: …, title: "Epic Fourms"}, …}
[14:53:05] <GothAlice> The title could be looked up, but it's infinitely better to get it "for free" with the thread record.
[14:56:48] <DrPeeper> figure if I can pick up language semantics quickly, it shouldn't be too much of a learning curve :)
[15:01:27] <StephenLynx> its easy to fall into forcing the tool into the wrong behavior because of how you think things should be.
[15:17:19] <JustMozzy> hello everyone. I am wondering. The eval function has been deprecated in version 3.0. is there an alternative to it?
[15:26:45] <tambu> Question on replica sets, if you are setting them up in two data centers, the docs say to make sure a majority is in your primary data center. Assuming that.. if you first datacenter goes offline (power outage). Your second data center cannot become primary because it doesn't have a majority is there a way to force one to become primary until you can get the other datacenter back on?
[15:27:47] <tambu> I was thinking perhaps if you logged into the datacenter2 mongo and increased the votes it has perhaps? then once the others come back online decrease the votes and step it down to force a reelection?
[16:06:58] <geloescht> GothAlice: I got some insight now, and no, it is not Mongooses fault! It seems to depend on the _order_ of the projection keys! Those two queries give different results: db.apps.find({componentName: 'default'}, {dummy:0, localization: { $elemMatch: { language: 'en' } } }), db.apps.find({componentName: 'default'}, {localization: { $elemMatch: { language: 'en' } }, dummy:0 })
[16:07:28] <tambu> DrPeeper sorry I stepped away yes I was planning to reconfigure the datacenter 2 manually to switch them to have a primary.. otherwise who do you have dr failover?
[16:08:13] <GothAlice> (The first one in the projection, that is. I.e. if dummy:0 is first, it's an exclusion projection, if $elemMatch is first it's an inclusion projection.)
[16:08:48] <tambu> deathanchor: I'm not sure I follow how is consistency broken? if you force a new primary it's now primary if your datacenter one comes back online wouldn't they all sync to the datacenter2 primary?
[16:09:22] <GothAlice> geloescht: As I mentioned earlier, $elemMatch is treated as a "1" in projection.
[16:10:02] <GothAlice> Potentially not; sometimes you might want to find an element that matches, but don't actually care about the matched value, if you follow.
[16:15:02] <vak> guys, I have removed a directory with an unneeded database (perDB structure) while the mongodb service was stopped. Now it won't start saying: "couldn't get file length when opening mapping ..."
[16:16:13] <geloescht> I still think there is something fishy about the way MongoDB treats $elemMatch. But at least I get the expected behaviour now by putting my "dummy: 0" projection at the end...
[16:19:52] <Ben_1> everytime before inserting bulk my counter increases by 1, so at the end my output says 5000 bulk insertions.
[16:20:17] <geloescht> GothAlice: weirdest thing is, properties in an object don't even have a guaranteed order!
[16:20:46] <Ben_1> in my callback method I have a counter too, so everytime I get the response "inserted" from mongoDB that counter increases by one. But this counter says 4993 at the end of the insertion process
[16:20:47] <GothAlice> Now, on the other hand, the language you're using might not preserve that order.
[16:21:17] <GothAlice> On the gripping hand, languages that don't naturally use ordered mappings do also provide ordered BSON structures you can use for those times when order matters.
[16:21:41] <geloescht> GothAlice: my node version might just happen to preserve order... but what if that changes some day?
[16:22:14] <GothAlice> geloescht: On that day we'll all start using Clueless as our primary programming language.
[16:22:18] <cheeser> though you might be hitting a duplicate key exception which the java driver should report in the WriteResult
[16:22:19] <DrPeeper> this is annoying.. xmltodict; need to rename ['a']['b']['c']['polygon'] to ['a']['b']['c']['type: "Polygon, coordinates"']
[16:22:27] <GothAlice> geloescht: I.e. worrying about such fundamental change hypotheticals is not worthy of time.
[16:23:23] <geloescht> GothAlice: http://stackoverflow.com/questions/9179680/is-it-acceptable-style-for-node-js-libraries-to-rely-on-object-key-order
[16:24:48] <GothAlice> geloescht: https://github.com/mongodb/node-mongodb-native/blob/2.1/lib/utils.js#L41 < as mentioned, utilities are usually provided to preserve order where it matters.
[16:28:52] <tambu> I see each mongo instance in a replica set can have a priority, if I wanted to have better control over who became primary it would make sense to assign each member a different priority like 4,3,2,1 so that they would switch in a predetermined order. I realize this isn't 100% sometimes a lower might become primary for a short period but they would balance out from what I've ready. Are there any downside to assigning priority
[16:29:24] <geloescht> GothAlice: interesting! If I try that syntax like that for projections it results in documents that only have an _id field...
[16:31:09] <geloescht> just thought that syntax might be accepted everywhere where an ordered map is expected
[16:31:31] <GothAlice> However, I don't JS, and the separation of mongodb-core vs. node-mongodb-native and other packages is making it difficult for me to track down the equivalent of https://github.com/mongodb/mongo-python-driver/tree/master/bson
[16:32:08] <GothAlice> I.e. https://github.com/mongodb/mongo-python-driver/blob/master/bson/son.py#L33-L40
[16:44:09] <Ben_1> cheeser: sorry for my late answer, at the moment I don't write any index so mongoDB generates the documents index.
[16:44:59] <Ben_1> just using Document doc = new Document(); and append data with keys like "device", "component", "sensor"
[16:47:39] <cheeser> then you're not hitting any key issues. what makes you think you're losing records?
[16:48:33] <GothAlice> Ben_1: As a note, if you're using sharding, count() totals might not be entirely accurate.
[16:52:41] <Ben_1> cheeser: mh good question, in the callback i'm using t.printStackTrace only it it's not null (this time *gg*) and the counter increase before the printStackTrace is called. One line before insertMany the output is 5000, in my callback the counter says 4985 this time (so everytime the callback counts a different result)
[16:55:55] <GothAlice> … I'm very curious why people keep coming in with "problems" (that are rarely actually problems) discovered through testing of driver internals. :|
[16:56:26] <Lachezar> Please help me with modeling the following: A "User" is assigned in multiple "Divisions". There is a "Vehicle" that can be assigned to a specific "Division", or not. One of the cases that I need is to show (page of) all vehicles that a User can access (assigned to any of his divisions, or to no division).
[17:04:46] <GothAlice> Ben_1: In the last week we've had users come in with "issues" around the wire protocol different versions of the driver uses (one using the more modern command form), amongst other oddities.
[17:05:29] <GothAlice> (I mention this because using stack traces to examine internal state… seems like an interesting diagnostic choice.)
[17:18:25] <Ben_1> on tuesday I had a problem with t.printstacktrace because it was null and I was calling printstacktrace. The driver was catching the nullpointer so there was no stacktrace at all and I was thinking the callback is not called.
[17:18:46] <Ben_1> But for my current problem, I disabled all output and leave just t.printStackTrace enabled, but no error is thrown.
[17:38:40] <tubbo> was just wondering if anyone had any good resources, besides the MongoDB documentation and university.mongodb.org, on doing aggregation and map/reduce queries?
[17:39:03] <tubbo> looking at the docs now, but wanted to see if there were any "best practices" or tips i should be reading, as this whole world of querying is totally new to me.
[18:00:21] <Ben_1> is it better to use more documents or more attributes? I could make a document for every device (about 5000) or a document for every sensor entry (about 33850000)
[18:01:32] <cheeser> the likelihood of pushing the document size over 16MB would be pretty high if you embedded the entries in a sensor's document
[18:02:01] <Ben_1> yes that's true, a way could be nested attributes but I think this is not so performant like a document for every sensor
[18:03:22] <GothAlice> Ben_1: http://www.devsmash.com/blog/mongodb-ad-hoc-analytics-aggregation-framework
[18:04:29] <GothAlice> That link goes into detail on the impact of different methods of storing sensor event-based data, the impact on query performance, as well as data storage and index sizes.
[18:46:46] <Haakon_> Hello there :) Im kinda new to mongo and i have a problem with combining 2 queries to one anyone that can help me a bit? :D
[18:46:48] <DrPeeper> i'm having to convert a bunch of xml documents to BSON that have a <polygon> buried deep in there.. what is the most efficient way to do this in pythong?
[18:47:35] <Ben_1> cheeser: I reproduced the problem, it seems if my application is too fast, mongoDB will skip several entries, on a specific rate It will not skip entries but callbacks. Here my testcode: http://pastebin.com/JytdwTih
[18:51:00] <Ben_1> if it is too fast I get an exception that the queue is full (although I have 8GB of free memory at that moment), but just if mongodb skips entries. On the special case when I get no error and few callbacks are skippt, no error is thrown.
[18:51:34] <cheeser> Ben_1: there's no need to chunk like that. stick all your docs into a list and then insertMany()
[18:53:09] <Ben_1> cheeser: in my case that's not possible because I get every List of Documents asynchronously from every device
[18:54:31] <Ben_1> so 5000 async Document lists and waiting for all 5000 devices is not performant :P
[18:58:19] <Ben_1> that's just an example, in my application I send them to mongoDB once I have them. This would be no problem in production I think because no Device will send its data so fast to my application. But I think it's odd that if I let the loop sleep for 60ms, all docs are written but few callbacks are not called.
[19:01:42] <GothAlice> Ben_1: In my own testing, with appropriate use of confirmed vs. unconfirmed writes I was able to squeeze just shy of two million round-trip (thus actually ~4 million) event records per second on a single node with four event producers and two event consumers. In production, since I checked recently, we actually processed around a million records in January.
[19:04:56] <GothAlice> Ben_1: https://gist.github.com/amcgregor/4207375 < we use a hybrid approach involving capped collections for maximal performance on the ephemeral (information that can be rebuilt) data, a concrete collection for non-ephemeral data, with pre-aggregation (based on the research in the link I previously gave) to give O(1) report performance.
[19:05:31] <GothAlice> It's Python, though, so there may also be language differences at play. For example, Python can intern (store once, in the compiled byte code) most simple structures and constants, including mappings.
[19:05:33] <Ben_1> GothAlice: my post was not about Performance issues with my model but the fact this problem occurs.
[19:06:03] <DrPeeper> GothAlice: do you have any advice on xmltodict and renaming the key of a buried key/value pair?
[19:06:43] <GothAlice> DrPeeper: If you're bulk processing large amounts of data, xmltodict will not be your friend as it unpacks and thus needs the whole XML document at once. :/
[19:08:22] <GothAlice> My own XML → MongoDB processing code is… not simple. https://github.com/marrow/contentment/tree/develop/web/component/asset < the XML sub-package here, with hooks throughout most of the componentized models.
[19:08:58] <GothAlice> https://gist.github.com/amcgregor/c6ae92325a941ba70568?ts=4 < this is the XML format parsed and generated with this code.
[19:09:34] <GothAlice> (Things with AlertCamelCase represent documents/sub-documents, lower-case tags represent fields.)
[19:14:24] <GothAlice> DrPeeper: As a note, I'm looking to extract generic MongoDB ←→ XML interchange from Contentment into marrow.mongo, as it does seem generally useful. https://github.com/marrow/contentment/wiki/XML-Interchange-Format < this wiki page needs a little updating, but the premise is there. Feedback would be greatly appreciated, esp. any notes on improving its generic-ness.
[19:44:59] <GothAlice> https://github.com/marrow/contentment/blob/develop/web/component/asset/xml/importers.py#L20 is the entry point for that.
[19:45:25] <GothAlice> Currently this is using ElementTree, but when I refactor it into m.mongo, I'll be switching to SAX, as cheeser suggested.
[19:45:55] <GothAlice> But generally, renaming a dictionary key can be accomplished via: somedictionary['new_key'] = somedictionary.pop('old_key')
[19:47:27] <DrPeeper> but does it have to be the full path of the key? e.g. somedictionary['root']['next']['new_key'] = somedictionary.pop['root']['next']['old_key']
[19:48:11] <GothAlice> somedictionary['root']['next']['new_key'] = somedictionary['root']['next'].pop('old_key') - and watch which parenthesis / brackets you're using. .pop() is a method call.
[19:49:10] <GothAlice> No worries. Solving the problem quickly is important, but XML interchange is a common enough problem I see people tackle that, really, we ought to come up with a generic solution that'll work for most people instead of NIH'ing all over the place. ;)
[19:51:28] <DrPeeper> and the next craptastic part is converting the way the polygon is handled in XML vs how mongodb does it!
[19:52:32] <GothAlice> Eew, regex. I always try whatever possible to avoid regexen for data conversion like this. (If your coords are a lat,long comma-separated string, for example, a data.partition(',') call will be infinitely faster than a regex to split.)
[19:53:04] <GothAlice> (Fun fact, partition is also faster than split, and split(',', 1), specifying the maximum number of splits, is slower than not. ;)
[19:56:55] <GothAlice> coords = [(float(i.partition(',')[0]), float(i.partition(',')[2])) for i in somedict['polygon'].split()]
[19:57:17] <GothAlice> (It'd be prettier unrolled into a real loop instead of all compressed into a comprehension like that, but you get the idea. ;)
[20:07:14] <beryl_> so when i want to add a document to the collection i am doing a count() and then if the number is smaller than the max, i add the document
[20:08:45] <Caullyn> Hey guys. I'm trying to print all of the uuid field of a collection to a file. When I try it this way, it is switching 'db.my_coll..' to the actual database name: http://fpaste.org/325077/45582501/
[20:08:54] <Caullyn> the query works when I'm in mongo console, so the fact that it's getting changed is making it return 0 results.
[20:08:54] <beryl_> how can i be sure with mongodb that i didn't exceed the max number of document per foreign key without application side mutexes?
[20:09:27] <cheeser> beryl_: personally, i'd have a janitor thread that culls the older items that over over the limit
[20:14:27] <beryl_> so as long as the counter is smaller than the max value i can add an element to the collection
[20:48:49] <Caullyn> I figured it out. mongo db_name -u administrator -p --authenticationDatabase admin --eval "rs.slaveOk(); db.coll.find({}, {'uuid':1}).forEach(printjson)" > output.json
[21:02:03] <Doyle> Hey. It's mentioned frequently to set the RA to 32, or the avg size of your documents, but is there any suggestion on block size?
[21:12:11] <cheeser> http://i83.photobucket.com/albums/j316/liz_marcs/Reaction%20Gifs/X8Ufi_zps62cda975.gif
[21:15:03] <GothAlice> DrPeeper: The eternal question isn't "does this key exist", but rather "what is the value associated with that key?"
[21:15:22] <GothAlice> The error is telling you explicitly that whatever you _think_ the value is, it _isn't_ a dictionary.
[21:29:37] <GothAlice> Ask, don't ask to ask. :) (Or all I can offer is a not-so-helpful "go faster". ;)
[21:32:34] <beryl> the thing is if i count and then insert if the number is smaller than the max, i have a race condition
[21:33:36] <cheeser> you need a transactional view of that document and you can't have it because mongo doesn't support it.
[21:34:17] <GothAlice> https://docs.mongodb.org/manual/tutorial/model-data-for-atomic-operations/ — you could, in theory, use an atomic counter and an update-if-not-modified query to increment the count only if it's below the limit, and _only if that query succeeds, add the record_.
[21:34:37] <GothAlice> Aye, multiple writes, with a bailout edge case where the counter is incremented but the insert doesn't complete, but it's a edge-case trade-off. ;)
[21:35:14] <cheeser> depending on how you want to handle the overflow, you could use findAndModify() with $inc and $push
[21:35:35] <cheeser> but that would just prevent writes on overflow. if you want to eject the older records, that'd be multiple writes.
[21:36:49] <GothAlice> An atomic counter field, with a update-if-not-modified update over plain update() would work.
[21:37:10] <DrPeeper> If a document doesn't contain a key, is it still possible to do an 'ensureIndex' of that key?
[21:38:05] <GothAlice> cheeser: I find findAndModify to be extremely counter-intuitive (i.e. returning the record prior to modification by default, sometimes) and over-used in situations where you already have a copy of the document. :(
[21:38:15] <beryl> the problem is that it is not a document with an array of items, but a collections with multiple documents that belongs to a specific user, each user having maximum 1000 documents
[21:38:59] <GothAlice> beryl: Sorta. You're back to needing two writes as I initially described: update the user's counter, and only if that succeeds actually try to insert the document.
[21:39:59] <GothAlice> cheeser: You're still misunderstanding. user = {name: "GothAlice", documentCount: 12}, otherCollectionDocument = {owner: "GothAlice", …}
[21:41:14] <GothAlice> beryl: In the example I provided just then, you'd db.user.update({name: "GothAlice", documentCount: {$lt: 1000}}, {$inc: {documentCount: 1}}), check the result, and only if the record was found and updated do your db.otherCollection.insert().
[21:45:43] <DrPeeper> oh.. it looks like there really must be data in the indexed field for a geospatial index of type "2Dsphere"
[21:47:12] <cheeser> i just created a 2dsphere index on a collection that didn't even exist prior to creating the index.
[22:29:13] <BenLubar> it's not for a production thing, it's just that my development server is an old machine that originally had Windows XP
[22:31:03] <StephenLynx> BenLubar, kurushiyama WT IS production ready and the default engine for 3.2
[22:31:43] <xissburg_> do I have to enable client access control to allow applications to use my mongod instance?
[22:32:12] <xissburg_> Well.. I cannot tell exactly but it seems that my app can only read from the database
[22:33:12] <kurushiyama> StephenLynx: Now thats interesting. I have seen weird things after migration to WT, not the last of it being a drastically increased RAM need.
[22:38:55] <StephenLynx> any web framework is cancerous trash by design that offer zero anything, hog your performance, makes debugging harder and introduces more bugs.
[22:39:45] <StephenLynx> I have been working with node for two years now and I got previous experience with web back ends. from all this experience I concluded that web frameworks have no purpose to exist.
[22:42:34] <StephenLynx> of course that when you are clueless is hard to figure what the problem is, so most people in this situation assume everything is bad because they don`t know any better.
[22:43:10] <xissburg> nah, once you see what's going on you realize.. yeah, this shit is really fucked
[22:44:37] <cheeser> if you want to have a geek off, feel free to PM each other. this really isn't the place.
[23:01:25] <xissburg> Now on the app too.. hmm I think I am finding my way around it.. gotta investigate other things
[23:01:55] <xissburg> basically I am doing this https://github.com/ParsePlatform/parse-server/wiki/Migrating-an-Existing-Parse-App
[23:02:36] <xissburg> I have a whole application which uses Parse, but now they're shutting down their hosting service and we have to deal with it
[23:03:10] <xissburg> you do this https://github.com/ParsePlatform/parse-server/wiki/Migrating-an-Existing-Parse-App
[23:22:28] <xmlnewbi> so if I have an extreemly large dataset a few gigs. all stored with timestamps. I want to do a limit 100 or so on the query. but I need to cherry pick so I get intervials not just the latest 100 entries. what is that even called? having hard time searching it
[23:29:54] <xmlnewbi> thinking im going to something ugly like making 100 queries limit 1 at some rounded version of the timestamp
[23:33:33] <GothAlice> xmlnewbi: https://jira.mongodb.org/browse/SERVER-22225 https://jira.mongodb.org/browse/SERVER-22068 https://jira.mongodb.org/browse/SERVER-22069 — careful of the RNG, there, esp. if you're using WiredTiger as your back-end engine.
[23:34:13] <GothAlice> cheeser: And yup, even after the segfaults we suffered from were fixed, incorrect RNG seeding is… an interesting new problem.