Log file Viewer
#mongodb logs for Thursday the 1st of October, 2015
[03:29:38] <sorabji> howdy. i'm working on an application to tail the oplog in a write intensive mongo cluster. i have a hidden secondary that i'm connecting to and creating a tailable cursor to its local oplog
[03:31:04] <sorabji> this issue i'm having is keeping up with the rate that documents are being inserted into the oplog. i'm wondering if there's any pointers you can offer to speed the process up
[07:01:16] <Boomtime> but what you're asking for is going to produce a very long list - that list is just a selected set of novel deployments
[07:15:02] <Boomtime> @plodder: dunno any specific, but you'll find plenty of projects on github using mongodb in some form
[07:15:29] <Boomtime> @psuriset__: http://docs.mongodb.org/manual/administration/production-notes/#mongodb-and-numa-hardware
[07:16:13] <Boomtime> @psuriset__: yeah, mileage varies, push it hard and you'll probably see differences
[07:17:15] <psuriset__> Boomtime, For centos, i dont find init scripts. i suspect it to be init issue.
[07:24:40] <psuriset__> Boomtime, Two things. 1) Thu Oct 1 03:19:03.084 [initandlisten] exception in initAndListen: 10310 Unable to lock file: /var/lib/mongodb/mongod.lock. Is a mongod instance already running?, terminating
[07:25:05] <psuriset__> 2) Already interleave done as you see in logs. still it does mention to interleave
[07:25:29] <Boomtime> -> exception in initAndListen: 10310 Unable to lock file: /var/lib/mongodb/mongod.lock. Is a mongod instance already running?, terminating
[07:25:57] <psuriset__> Boomtime, yes. same thing. i killed mongod and started with numactl interleave
[07:27:38] <psuriset__> Boomtime, Without interleave : https://gist.github.com/psuriset/b465710fdb4560322a25
[07:29:47] <Boomtime> and whenever you specify interleaving (which doesn't "take" btw since the process reports none in both cases), the lockfile error occurs instead?
[07:30:15] <psuriset__> Boomtime, After little google, i realised init script need to be added. Added that too.
[07:31:07] <psuriset__> Boomtime, interleaving is happening and listing on port. i feel interleave is working.
[07:33:12] <psuriset__> Boomtime, looks like similar issue: https://jira.mongodb.org/browse/SERVER-6008
[07:34:09] <Boomtime> centos isn't deb though right? and fixed before your version... otherwise, yeah, looks similar
[07:37:31] <Boomtime> ok, can you test this without the init.d? like, try using a command-line and don't even bother with --fork or log, just try launching a server direct in a shell
[07:38:17] <Boomtime> when possible, you should at least get the latest 2.4 too, or better upgrade to latest 3.0
[07:42:01] <psuriset__> Boomtime, my idea it to get perf numbers with/without interleave. Ofcourse interleave would be good.
[07:42:25] <psuriset__> Boomtime, Just want to make sure, how other components will perform in such scenario
[12:12:49] <terabyte> in my mongodb directory I have a number of files. myschema.0,.1,.2,.3,.4,.5 they double in size each time. I understand each file is 'preallocated' so given the latest is 2gb i haven't actually used the 2gb yet. just wondering, is the database data all contained in the latest 2gb file, or is it spread across all 5 files?
[12:30:28] <Grummfy> hello, a question about how to do thing. I need to check on a single collection all record that match an element 'a' with another element 'b' but 'b' is in an array. So elementmatch + where is not possible, so i try with aggregate + unwind but where is not possible inside. So except with a map reduce I don't see easy way to do it
[12:33:36] <sorabji> howdy. i'm working on an application to tails the oplog in a write intensive mongo cluster. i have a hidden secondary that i'm connecting to and creating a tailable cursor to its local oplog
[12:33:51] <sorabji> the issue i'm having is keeping up with the rate that documents are being inserted into the oplog. i'm wondering if there's any pointers you can offer to speed the process up
[12:43:22] <Bookwormser> Good morning. Did the method for importing a schema change between 2.6.9 and 3.0.6? I keep getting a bad JSON array format - found no opening bracket '[' in input source error, though the schemas have never contained brackets.
[12:56:10] <jamiel> Hi all, does WiredTiger do any prefixing of field names for collection storage like it does on indexes. ie. Does it make using short field names redundant as a space saving technique? Or will that still help..? Thanks.
[13:16:11] <NoReflex> hello! I have a collection af about 121 million rows with an average size of 168 bytes; I'm running a query which uses an index that should retrieve around 1.4 million rows
[13:16:50] <NoReflex> this query runs in about 5000 seconds on my machine while with postgres with the same database and index it runs in about 200 seconds (that is 25x faster in PG)
[13:18:59] <NoReflex> the collection holds data sent by many devices; ti represents the device, mo is the moment, rt is the record type while the other data is not useful for indexing
[13:19:46] <NoReflex> here you can see the definition in PG and the queries used: http://pastebin.com/92yyQU3w
[13:20:32] <NoReflex> the only difference is that some key -values are transformed in columns in postgres
[13:26:10] <jamiel> Extra 10 years of development on PG? :) j/k ... you probably want to send us the explain() results for that query and the index definition
[13:29:07] <Ulrar> Hi, I installed a mongodb for a client and I have some problems with access rights. I have a user with a userAdmin role on a databse, but that user still gets "not authorized" errors when trying to insert data in that database
[13:32:28] <NoReflex> jamiel, I pasted the useful parts of the explain output here: http://pastebin.com/pTmmFN1j
[13:35:21] <Ulrar> Okay my bad, I added the readWrite role and it works. feels a little strange that userAdmin can't write, but okay
[13:40:32] <jamiel> @NoReflex you should try creating the index with the field which has the highest cardinality first or which would be the most restrictive of the dates ... "rt" should definitely be the last field in the index as it provides no benefit really, it really depends on your data but you should experiment with {ti: 1, mo: 1, rt: 1 } and { mo: 1, ti: 1, rt: 1 } and
[13:41:01] <jamiel> Also do a db.coll.stats(1024 * 1024) and check the size of the index whether it fits in memory or if the query is going to disk
[13:48:18] <NoReflex> jamiel, there are around 2200 distinct ti values in the collection of which I'm selecting around 500, 4 rt values (0,1,2,3) of which I'm selecting only one
[13:48:46] <NoReflex> regarding the moment (mo) the data spans around 10 months fo which I'm selecting 6 months
[13:55:47] <MadWasp> Hi, I need to change approximately 20 million documents in my mongo db. They currently reference another collection in which documents have a params field. This params field should become part of the 20 million documents. What would be the fastest way to achieve this?
[14:18:42] <MadWasp> For now I’ve written a PHP Script to do it but it takes like a couple of days to run.
[16:23:55] <bmarick> Hey, I have come across an SSL host validation issue. Is there anyone here that could point me to a location in the code base so I can try to track down the error?
[16:33:01] <brianboyko> Hello. I'm building out the schemas now using Mongoose. How do you create an array of objects? Specifically, each test has questions, each question has a correct response, a student response, and a point value.
[18:19:26] <steeze> but like $only. i want users who ONLY have 'user' in their roles array. this matches any user with 'user' + other roles
[19:31:47] <blizzow> I'm running a query via pymongo: db.mycollection.find( { "_id": "mydocumentname", $and: [ { "timestamp": { $exists: true } } ] } )
[19:48:27] <teknopaul> Hi I'm finding that update() or updateOne() wipes all the data in an object, my update has no need for "dot notatioin" AFACS its a flat object, so I'm just trying to add some fields { '$set': { desc: 'baa', lastUpdate: 1443728439630 } }
[19:57:58] <StephenLynx> about to say that and didn't even had to check your code. that happened to me sometimes when I started with mongo
[20:01:06] <StephenLynx> you might want to make an update that does other stuff without a $set though
[20:39:08] <pagios> pagios: Don't use MongoDB - it has many issues, and is essentially never the right solution to your problem. Here's a list of issues: http://cryto.net/~joepie91/blog/2015/07/19/why-you-should-never-ever-ever-use-mongodb/ and a more technical article at http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/ CAN SOMEONE please explain this to me once and for all coz i am tired of ppl's comments and noise
[20:39:58] <pagios> featurs of my app: authentication, checking credits, credit manipulation, backend user statistics
[20:45:09] <teknopaul> For prototyping with nodejs I'd say mongodb has to be a pretty good option. Using a document DB is very handy for this.
[20:45:43] <StephenLynx> "hurr I used the database wrong, I am not retarded, is the database that is bad derpderpderp"
[20:47:08] <pagios> so when should i use a relationsal db and when should i use a document based db?
[20:48:25] <StephenLynx> mongo's limitations: counts are slow, if I am not mistaken. you have a 16mb limit on documents
[20:48:28] <teknopaul> Read this "For most cases, what you want is actually a relational database. " and you realise the guy is on a rant.
[20:50:07] <StephenLynx> that your data shouldn't be too relational and involve too many collections in the first place.
[20:50:22] <StephenLynx> if your operation involves a dozen collections, you really should use a relational database.
[20:51:18] <StephenLynx> if you need to keep track of a considerable amount of relations, thats where relational dbs comes in.
[20:51:30] <pagios> whenyou say relational it means my documents are so interelated, tables are so interconnected?
[20:51:47] <idd2d> I have Users, each has_many Addresses (via parent referencing). What's the best way to find which User has the most Addresses?
[20:52:40] <pagios> if i have say a user profile document, a user credit document, and a user token document , that not very tightly relational
[20:53:41] <pagios> but if i have a query that reads document1 , compares with doc 2 and modified docu3 thats too relationsal righht?
[20:55:18] <StephenLynx> when you have to make an update that keeps relational integrity across 4 collections or so.