Log file Viewer
#mongodb logs for Monday the 6th of June, 2016
[08:43:39] <ShekharReddy> hey guys how can i start the mongodb on ubuntu 14.04, i have started the service using sudo service mongod start
[09:09:36] <chris|> the documentation about backup up a secondary using filesystem snapshot says I "may skip" calling fsyncLock on it if journaling is enabled. Should I read that as "you may skip, but it's recommended not to" or is it safe to do so? Also, if I do not lock the secondary, should I freeze it ?
[09:26:17] <chris|> quick followup, why is this important? When calling db.fsyncLock(), ensure that the connection is kept open to allow a subsequent call.
[09:30:21] <chris|> that would be strange, as the docs also say: Closing the connection may make it difficult to release the lock. ;)
[09:45:53] <chris|> sure: https://docs.mongodb.com/manual/reference/command/fsync/#impact-on-read-operations
[10:44:07] <joannac> fsyncLock(), queued writes, reads (i.e. auth checks) will queue behind the queued writes
[12:32:44] <miracee> Hello, we organise a database dev room on FrOSCon (Germany) and I am looking for speakers ... we want to provide multiple systems.
[12:41:33] <zzookk> hello guys. tell me pls. "find().sort({"thisfieldname":-1}).limit(1)" and find.max - same constructions if i want to get highest value?
[14:56:14] <cppking> hello guys, I got a question; if a replSet's master (A) down, then one secondary(B) will become master, if node A has a priority larger than any other node. what happens when A start after 10 hours and join the replSet, does data of 10 hours will lost ??
[15:07:18] <KostyaSha> cppking, i may mistaken, but firstly it do election based on votes and only then priority influence
[15:38:00] <cppking> KostyaSha: Do you understand me? My replSet has 3 node (A B C), C is arbiter, A has a priority larger than B, if Master A down , B will become master, 10 hours later, If I start A, will data produced in 10 hours be lost ??
[15:40:08] <cppking> because when A start, A will become master, then data is inconsist between A and B, so data will be lost;
[16:47:00] <troll_king> my code is mongojs('irclogs').collection('logs') but when i console.log it show only 1 function
[17:20:45] <renlo> This might be a stupid question: at my work we have a Python web application. This web application fetches data from Mongo, then it serializes the data into a JSON format. Thing is, the number of documents that are serialized can be ~ 20k documents. When Python serializes the data, the server goes to 100% CPU usage, and it takes a number of seconds (5-10). Is there a way to get Mongo to be output as JSON?
[17:21:21] <renlo> or is that just from using a Python client which is automagically turning JSON documents into some Python dict, and then its getting turned into JSON documents again?
[17:29:45] <StephenLynx> if you want, you could try with a more CPU-efficient runtime environment, like node.
[17:30:07] <cheeser> by the time you see the structs in python, they're already python dicts and not bson/json
[17:30:22] <cheeser> so if you're converting those dicts to something and *then* to a json string, you're doing too much.
[17:31:32] <renlo> cheeser: so its safe to say I am probably first converting BSON to a Python object / dict, then converting the Python object / dict to JSON?
[17:31:39] <cheeser> now, those are probably not *large* docs but large and small are fairly subjective
[17:32:04] <cheeser> renlo: yes. the driver is already doing the BSON document -> python dict for you
[17:32:24] <renlo> okay thanks guys, I'll see if I can a) convert the BSON straight to JSON with the Python driver, and if that doesnt work b) look into using node
[17:33:26] <renlo> it's a Django web app, how would you do that? Spawn a subprocess which runs the mongoexport command?
[17:33:59] <renlo> cheeser: is that a 'yeesh' as in 'my god dont do that' or a 'yeesh' as in 'yes' lol
[17:34:45] <cheeser> exec()ing out like that could potentially eat up any speed gains of using mongoexport
[17:35:42] <cheeser> fwiw, the java driver is much faster than mongodump so it's not necessarily safe to assume the tools are faster than the drivers.
[17:36:21] <cheeser> this particular isse was with mongodump specifically. we didn't profile mongoexport
[17:42:20] <cheeser> e.g., the java driver exposes a type called BsonDocument which still has the documents in their BSON form and not, say, a Map
[17:42:51] <cheeser> so there'd be no conversion of types from their BSON types to native Java types. Given this, you can encode them JSON without double decoding.
[17:43:54] <renlo> I found this: https://api.mongodb.com/python/current/api/bson/json_util.html , I think if I pass it the pymongo query (cursor?) then it will do it without converting it to a dict first, though I am not 100% sure aboout that
[17:54:09] <cheeser> he said they're considering a "BsonJsonCodec" in the next quarter or two but they're still in the discussion phase.
[17:54:44] <cheeser> he said to email him at jesse@mongodb.com and introduce yourself. he said they're definitely eyeballing this use case and this could help them move forward.
[17:56:31] <cheeser> if that returns false, the C extensions aren't being used and you're running the pure python bson code which is much slower than the C libs
[17:58:53] <cheeser> oh, that's kind of a disappointment because you could've had a huge win by recompiling with the C extensions. :)
[18:02:33] <Forbidd3n> Hey everyone. Quick question. Is it best to store all elements in one data set or have multiple datasets linked by referential ID?
[18:03:46] <cheeser> you should avoid using references as the whole point of mongodb is to store self-contained documents.
[18:04:06] <cheeser> each external reference is another round trip to the database to fetch that document.
[18:10:04] <Forbidd3n> cheeser: for example- {"schools":{"Bears":{"classes":{"Math":{"days":["M","T","W"]},"Science":{"days":["M","W"]},"English":{"days":["T","W"]}}
[18:10:41] <Forbidd3n> I know that isn't correct format, but my question is really the depth of a collection, is it relevant?
[18:12:13] <cheeser> exactly how to model your documents is hard outside the context what's in them and how they're going to be used and queried.
[18:13:49] <Forbidd3n> cheeser: I understand. I will be querying different levels using the Slim framework API and updating multiple levels
[18:18:32] <Forbidd3n> cheeser: http://blog.mongodb.org/post/87200945828/6-rules-of-thumb-for-mongodb-schema-design-part-1 : here it says if you are going to have One-to-Squillions then you should reference by ID from another collection
[18:19:36] <Forbidd3n> so static data I think I can put in one collection, and consistent changing data in it's own collection so it can be updated by ID, remove or new ones added easier than filter through other data.
[18:39:15] <Forbidd3n> StephenLynx: if a sub item in the collection needs to be the reference to the other collection, then I would need the sub item to be its own collection as well, correct?
[18:40:44] <Forbidd3n> StephenLynx: correct. I would have to store the id of the referenced collections
[18:41:38] <Forbidd3n> StephenLynx: the project I am working on is Cruise Lines, Cruise Ships, Itineraries.
[18:41:58] <StephenLynx> sometimes you don't want to use _id because you want something readable for example.
[18:44:43] <StephenLynx> because mongo doesn't implement relations, you can do anything you wish for that.
[18:45:31] <StephenLynx> if you reference it by the _id you will have to get the user's login from that id to do so
[18:46:14] <StephenLynx> since the login is unique, its as good as _id or any other unique index when referencing users.
[18:46:38] <Forbidd3n> gotcha, so in this case you can request a user by 'loginname' and then you have the _id to get related records if needed
[18:48:18] <StephenLynx> that way you don't have to instance and ObjectId from the string when querying
[18:49:28] <Forbidd3n> so in the CruiseLine collection I would store Ships which would be 'ID' from the ship collection and in the ship collection I would store Schedule which would be each schedule for the ship - am I on the right track here?
[18:59:29] <Forbidd3n> the first set would be the line collection and the second the ship collection
[19:02:14] <Forbidd3n> StephenLynx: because I didn't want to lay it all out if the concept was incorrect
[19:03:13] <Forbidd3n> basically I would be storing the lines in a collection and referencing the ships from it's own collection
[19:04:27] <renlo> what are the fastest Mongo drivers? I think cheeser mentioned that the Java driver was faster than mongoexport or something? Anyone have any experience with the different drivers?
[19:07:06] <cheeser> there's one benchmark that showed php outperforming all others but given php's ... nature I don't think anyone besides Derick and Jeremy trust that one
[19:08:11] <Forbidd3n> StephenLynx: what section? I understand what you mean by laying it out, which I can do, but trying to figure out how to link the tables and wanted to see how you are doing it
[19:16:34] <Forbidd3n> there will be more elements for each collection, but just put that for starters
[19:20:57] <Forbidd3n> the api will allow the user to search a schedule by port and get all the ships that go there
[19:27:28] <Forbidd3n> I will be creating my json code from php arrays so I have to figure out the format for that to store in mongo
[19:27:57] <StephenLynx> yeah, consult the driver and then check on the terminal client for the types being used.
[19:28:15] <StephenLynx> if I am not mistaken, PHP driver has its own types for some stuff, like dates.
[19:32:10] <jokke> i'm trying to collect some metrics from my mongodb instance. For this i run collStats on every collection in every database
[19:40:13] <StephenLynx> pay attention to those, so the type used on the db is the type you set out to use on the model.
[20:05:32] <zylo4747> is there any way to create a free tier cloud manager deployment or is that no longer possible?
[20:22:36] <kurushiyama> zylo4747: I strongly advice against setting up your cluster with CM if you do not plan to continue to use it.
[20:52:09] <kurushiyama> StephenLynx: Sorry, "fix the connectivity issue" was context-free for me and did not get it, hence I asked.
[21:18:14] <ebarault> well, i'm having random "MongoError: not authorized on XXX to execute command {YYY}" with mongo 3.2, since very recently
[21:19:51] <gaboesquivel> I only need to filter the result set based on the values contained in that field.
[21:20:53] <gaboesquivel> for instance all documents where cell_options.environment contains either {value: 'Water' } or {value: 'Gym'}
[22:07:13] <poz2k4444> Hi guys, can somebody can help me with a problem I have with mongo-connector and elastic search?
[22:32:12] <hyperboreean> hey guys, any patterns on how to iterate really big collections ? I have one which has almost 900M documents and is close to imposible to go over it, even without any filtering
[22:46:51] <gaboesquivel> thanks in advanced if anyone can help me out with that query, http://stackoverflow.com/questions/37667901/how-to-search-for-documents-containing-objects-with-specific-values-in-a-nested