Log file Viewer
#mongodb logs for Thursday the 23rd of July, 2015
[09:31:34] <aps> I did db.fsyncLock( ) at secondary member (version 2.6), took EBS snapshot of data and journal drive, used this snapshot to create new volume mounted at a new mongo (3.0.4) and added this new mongo to the replica set.
[09:32:36] <aps> It started replication from the beginning. I see data and journal file are present on the new disk. What am I missing here?
[10:21:22] <crashev> hello, does mongodb has sill problems with lossing randomly data on writes or is it just a sound of past ?
[10:21:57] <Zelest> used mongodb in production for over 3 years now.. never happened a single time so far..
[10:28:56] <pamp> Someone knows why I'm getting this exception :exception: socket exception [CONNECT_ERROR] for shard0000:27017
[10:30:15] <pamp> the connection between shard is ok, I can connect to second shard from the first, contrariwise too
[10:32:54] <crashev> Zelest: wonder when all those articles and issues come from then, about inconsistencies and losing data
[10:33:32] <crashev> Zelest: just read this one http://cryto.net/~joepie91/blog/2015/07/19/why-you-should-never-ever-ever-use-mongodb/
[11:49:50] <crised> if the data from my db is public to read... When building a website... Is it possible/recommended to go directly from front end / browser to the database?
[12:00:08] <lmatteis> and i want to display all the logs separated by domain (domain per row) along with the percentage uptime i recorded (which is essentially just calculated for each request for that row)
[12:00:27] <lmatteis> would an aggregate be performant to generate such thing dynamically (each time a user accesses the page)? or do i need to run a pre-aggregate offline instead?
[12:01:17] <coudenysj> lmatteis: the query will be the same, some you could try to do it on the fly, if it is too slow, offload it
[12:02:03] <lmatteis> coudenysj: by offload it you mean that i could run the query, say with a cron job, and then update a separate index with the results and serve that on the site?
[12:02:34] <coudenysj> something like that, but the aggregation query will be the same on both cases
[12:03:04] <lmatteis> coudenysj: plus aren't indexes on the collection already building such view themeselves ?
[12:38:11] <StephenLynx> https://www.google.com.br/search?client=ubuntu&channel=fs&q=mongodb+sort+case+insensitive&ie=utf-8&oe=utf-8&gfe_rd=cr&ei=ld-wVav7N4aC8QfYnrgY
[12:57:54] <Siyfion> I have a query "{ 'subRecipes.recipe._id': doc._id }" that finds the documents that need updating
[12:58:11] <Siyfion> I thought this would work: Recipes.update({ },{ $pull: { 'subRecipes': { 'recipe._id': doc._id } } },{ multi: true });
[12:58:18] <Siyfion> But it doesn't seem too... I thought I might be able to use the "$" operator to pull the index found, but I'm not sure how.
[13:00:18] <jacksnipe> I'm having a problem with the mongodb php driver: Queries constructed using the MongoDate class wind up coming out as "$gt" : {"sec" : ..., "usec" : ...} instead of anything using any actual date types
[13:04:06] <deathanchor> jacksnipe: this is what I have in my code: https://gist.github.com/deathanchor/2b2887520ff4d9e25dfa
[13:05:07] <jacksnipe> will try this different construction, doesn't look like it's different than what I'm doing tho :(
[13:10:37] <[diecast]> jacksnipe: say I have some collections with documents that I want to insert templated information with
[13:17:39] <[diecast]> it is a document with information like endpoint connections, oid, application associations, etc
[13:22:03] <jacksnipe> that's certainly provided by a lot of mongodb ORMs, tho I don't think it's provided at the db level
[13:28:32] <jacksnipe> honestly it sounds like you could implement that on your own in any OO language pretty trivially
[15:36:00] <nalum> hello all, I'm trying to get the name of a collection from a collection object in python (pymongo 2.8) but it's just printing <common.classes.Datastore.Executable object at 0x7f0e8eb20190> when I try to print it as follows: print self.collection.name
[16:04:30] <_ritchie_> and the project is just getting started so it’s not too late to port to another lib
[16:05:13] <_ritchie_> one thing I know I’d like to do is take a dict and save it as a Mongo Document
[16:05:44] <GothAlice> Thus, your expectations going in are what's causing you to interpret MongoEngine as being sub-par.
[16:06:08] <saml> I got {_id:1, v: [1,2,3]}, {_id:2, v: [2, 3, 4, 4]}, ... how can I aggregate them so that {_id:1, v: sum of $v} ?
[16:06:14] <_ritchie_> ok that’s fair, MongoEngine seems like it’s purpose is to provide some validation of the Schema?
[16:06:47] <GothAlice> Literally mapping fields in documents (the "dictionaries", which MongoDB calls "documents") to attributes of objects in Python-land.
[16:07:23] <GothAlice> Identical to the difference between raw SQL and a layer like SQLAlchemy (which is an "active record" Object Relational Mapper.)
[16:08:42] <GothAlice> _ritchie_: https://github.com/marrow/cache/blob/develop/marrow/cache/model.py#L86-L197 is an example simple model of mine. Note that this performs field renaming (i.e. the attribute "value" is stored as "v" in the underlying document).
[16:08:59] <saml> actually, i got {_id:1,v:[{name:'Foo',score:2}, {name:'Foo', score:3}, {name:'Bar', ..}...]} and want {_id:1, v:[{name:'Foo', score: 5}, ...]}
[16:09:39] <_ritchie_> GothAlice: there’s like 3 other applications in production that read these configs as Mongodocuments
[16:10:16] <_ritchie_> and it’s a large-ish company so I don’t have the opportunity to make a change like that easily
[16:11:27] <_ritchie_> I’m just making changes to one web application which saves the configs to mongo
[16:13:00] <GothAlice> There are innumerable additional benefits to using an abstraction layer, whatever that layer might be. But I doubt any will convince you. (Signals, integrity, containerization, modularity, testability, …)
[16:13:14] <StephenLynx> think about the dependency in the scenario before thinking they need the dependency.
[16:15:03] <GothAlice> Because I'm using an abstraction layer in marrow.cache, I can literally, with a minimum of work, drop MongoEngine for raw PyMongo, or drop MongoDB itself for any other database layer, without impacting users of my library beyond needing to update some configuration directives. (And potentially deploy a different database, but you get the idea.)
[16:19:12] <_ritchie_> GothAlice: is there a way to try to cast a Dict as a MongoEngine Document? something like for all keys of the dict if the key is a field on the document keep its value otherwise discard
[16:20:30] <GothAlice> _ritchie_: Basic Python syntax says MyDocument(**somedict) will work, if the dictionary keys match field names on the document class.
[16:21:03] <GothAlice> However, at what point do you have a raw dictionary? If using MongoEngine, that shouldn't happen.
[16:22:46] <_ritchie_> also thanks for your help with the Python syntax, typically I write Javascript or Ruby
[16:25:26] <StephenLynx> I assume your software handles this data one way or the other before doing that.
[16:36:31] <_ritchie_> StephenLynx: so would a more typical use case for Mongoengine be when you have a Document that once instantiated as a Python object also has business logic associated with it?
[16:37:31] <StephenLynx> but from what I know these things do, using just schema validation is way, way, waaaay too small to justify their usage.
[16:56:05] <GothAlice> _ritchie_: However, if light-weight data validation is all you really need, you can glue https://github.com/marrow/schema#4-validation in there.
[16:56:24] <GothAlice> (It's a very tiny library for doing data schemas, transformation, and validation, with more tests than lines of code. ;)
[17:00:09] <agend> hi - what happens when i insert document - and it's _id is already taken - does it give error?
[17:04:13] <agend> is it possible to make insert as update - some operators magic maybe - or the other way - update as insert?
[17:23:59] <blizzow> Can someone here explain what the logging situation is with mongo 3.x? Manually adding a kill -SIGUSR1 $mongoPID and doing gross regex to search for and manage timestamped mongo log files seems hackish. What is the proper way to rotate logs with mongo 3.x?
[17:31:37] <aps> WriteConcernException: { "serverUsed" : "ec2-x-x-x-x.compute-1.amazonaws.com:xxx" , "ok" : 1 , "n" : 1 , "updatedExisting" : true , "err" : "waiting for replication timed out at shard-a" , "code" : 64}
[17:48:40] <ciwolsey> if im search date ranges does mongo automatically deal with time zone differences?
[17:51:35] <ciwolsey> "MongoDB stores all dates and times in UTC. This allows the timezone support and translation to be done application side instead of server side."
[17:54:47] <ciwolsey> does that mean i have to convert all my dates from the local time zone to utc before performing the search?
[18:31:22] <StephenLynx> since mongo converts everything before storing, I guess it would also convert to UTC when using a supplied date.
[18:38:30] <kali> i'm not sure if that will clarify it in any way, but let met try: mongodb date is necessarily UTC. there is no provision to store a separate timezone in the database.
[18:40:25] <kali> the database ignore the timzeone, but the driver and the language will make sure they provide utc to the database
[18:56:39] <iSpoof> how can I add a new field to all existing documents in the collection, but not appended at the end of the documents but in some specific position? I've been checking $position and $push but this is for array values. Is that doable after all? I'm thinking on the functionality that RDBS implement through ALTER TABLE .. AFTER .. but I don't know if MongoDB db.collection.update() provides some way to do that
[19:04:31] <deathanchor> also you don't get much choice in the ordering of the data within a document in mongo.
[19:06:51] <iSpoof> deathanchor: I see... so I'd need Mongoose or something to emulate schemas and put the field in arbitrary positions?
[19:07:48] <iSpoof> I understand that for a machine, the position of the key in a json object is utterly irrelevant, i just wanted to have the new field close to the top so its easier to read in the MongoLab ui : )
[19:09:16] <iSpoof> anyways, i got the answer i needed, and makes plenty of sense. thanks deathanchor
[19:09:34] <deathanchor> yeah all schema/ordering is done on the application endpoint, mongodb doesn't provide any methods for what you are asking.
[19:10:48] <deathanchor> yeah, I just get what I need when I query: .find({ search : "forthis"}, { show : 1, me: 1, this: 1} )
[20:33:39] <pokEarl> Uhm so some questions around resource use and optimal performance. Like lets say you have a document with 3 values of equal length/type in it. Is getting all of them 3 times as resource intensive as just getting one? So you should always just get out the values you need? Does nested or non nested have any effect?
[20:35:30] <StephenLynx> after that, how much load it takes to handle the found document depends on other operations (aggregation stages) and how much RAM it needs to hold the found documents.
[20:36:31] <mike_edmr> requesting specific fields has an infinitesimal cpu penalty compared to just serving up the doc as-is
[20:38:56] <StephenLynx> in general, it is more efficient to project only the information you want to use.
[20:42:40] <pokEarl> Yeah =p I am making an API as a layer over mongoDBs (with Java/jackson returning Json) and I have classes representing the different documents and was considering if I should be making tons of different methods to get out all the different combinations of values or something, sacrificing a lot of potential flexibility
[20:45:39] <mike_edmr> I wouldn't think you'd want your object methods to be making individual calls for each value when you might be accessing multiple methods that could have been supplied by a single mongo call
[20:47:18] <pokEarl> I am not/what do you mean? obviously it makes sense but not sure what I gave that impression by :P Will still use projections where it makes sense obviously but not going to create a million different methods with different projection variations for every getById type method or whatever =p
[20:50:39] <mike_edmr> if you had a customer document and a Customer class that wraps it with getName and getAddress, you wouldn't want a getName and a getAddress to make seperate calls to mongo
[20:51:00] <mike_edmr> so it'd be best to provide methods that do all the things you need to do in a single call for a particular use
[20:51:42] <mike_edmr> like if you have a function that needs access to name and address, make a getNameAndAddress
[20:56:32] <pokEarl> nono I don't, i just make 1 call for the document and it gets mapped to the class by Jackson/Mongojack
[21:02:41] <pokEarl> No, but I was not sure how suboptimal it is which is why I asked ;P And the conclussion I reached is that obviously there is some performance to be gained. But it is not worth creating a semi-infinite amount of different verisons of every single call you might want just to apply different projections depending on the API call. Like its fine just having 1 getDocument/Id method which returns the document by the id, instead of tons of different methods
[21:02:41] <pokEarl> for every getDocument/Id combination of Values. Like with 15 different values in a document I am sure that represents a pretty large amount of different ways the fields can be combined =p
[21:53:20] <pokEarl> is there a way to get a collection to show the "maxed out" documents if that question even makes sense? I guess maybe that is not possible because of schemalessness or something?
[21:55:05] <pokEarl> Like if theres document 1 through 1 million with value1 through 10, but then the next document has a value 11, is there a way to find all such values without knowing beforehand what they are, and not going through all the documents? :P
[21:57:46] <mike_edmr> you would either need to go through the documents with a tablescan or create an index to optimize that query
[22:55:32] <joannac> vzp_: define "block". mongorestore is continuously sending writes, so your reads will probably end up waiting