Log file Viewer

[06:00:14] <tubbo> hey i'm getting a strange error from mongodb. i'm using mongoid and i have been trying to figure out what this means

[06:00:16] <tubbo> error 16837: "cannot use the part (0 of images.0._id) to traverse the element ({0: null})"

[06:28:37] <rafibd01717> why we use customdata: key?

[06:28:56] <rafibd01717> why we use customdata: key?

[06:28:56] <rafibd01717> why we use customdata: key?

[06:41:15] <rafibd01717> what is writeConcern?

[06:41:34] <joannac> rafibd01717: https://docs.mongodb.org/manual/core/write-concern/

[06:44:02] <rafibd01717> @joannac: I didnt understand it clearly as my english is not good

[06:44:12] <rafibd01717> can anyone please explain me?

[08:13:18] <pyios> htop shows that mongodb use virtual memory hit 200G , horrible of that

[08:29:40] <kali> pyios: that's the way mongodb works: it maps in virtual memory all the database files. it's less horrible than it sounds

[08:30:03] <kali> pyios: https://docs.mongodb.org/manual/faq/storage/

[08:30:58] <pyios> with my application runs , the vm is uses raise to be more and more

[08:31:36] <pyios> and now it consume 98% memory ,include swap mem

[08:32:02] <pyios> And machine is stuck with long time delay for operations

[08:32:26] <kali> how big are your database files ?

[08:32:42] <pyios> how to detect that?

[08:33:07] <kali> just check the files size

[08:33:23] <pyios> there is several files

[08:34:57] <kali> what about the total ? :P

[08:35:07] <pyios> 114G

[08:35:25] <pyios> total is 114G

[08:35:43] <kali> ok. the files are mapped twice, so 200GB sounds right

[08:36:15] <kali> moving on. do you lines for show operations in the dtabase logs ?

[08:36:49] <pyios> can not understand what you mean ,pardon?

[08:37:30] <kali> sorry, i need more coffee, and my contact lenses.

[08:37:47] <kali> do you see logs lines for slow operations ? in the logs ?

[08:38:19] <pyios> find() operation

[08:38:36] <someone235> How can I import data from mongodump?

[08:38:46] <pyios> now shell command is stuck too

[08:39:52] <someone235> I need to restore a whole database with mongorestore. How can I do it?

[08:41:23] <kali> pyios: can you paste one typical line somewhere ?

[08:42:20] <pyios> now machine is stuck ,and I can not use shell command to cat log file :(

[08:51:07] <pyios> kali:Mon Nov 16 16:39:07.959 [DataFileSync] flushing mmaps took 39871ms for 63 files

[08:51:59] <pyios> the log has not mentioned memory issues

[08:53:10] <kali> mmm that's a lot

[08:59:50] <kali> pyios: could you paste the output of db.serverStatus() somewhere ?

[09:00:02] <kali> pyios: how many database/collection are ther e?

[09:00:20] <pyios> 3 database ,8 collections

[09:02:07] <pyios> kali:http://paste.ubuntu.com/13297553/

[09:04:01] <pyios> kali:several minutes earlier ,I type a find() operation on one collection ,then it stuck the database for almost 10 minutes

[09:04:36] <pyios> After I close that operation ,it resumes work

[09:10:17] <kali> pyios: check out for long running ops in db.currentOp()

[09:12:33] <pyios> if my application’s io to mongdb is blocked so long time ,I think it failed

[09:13:14] <pyios> I mean that is equal with none working at all

[12:17:39] <vsv> Hello guys. I would like to ask if somebody has experience with MongdoDB auditing. I am using Spring and the @Version annotation. I would like to overwrite the default initial value (currently zero) to one. Is it possible to provide a parameter to the @Version annotation?

[12:31:18] <cheeser> why would the version number matter?

[12:34:14] <Ben_1> hey

[12:34:49] <Ben_1> is it possible to run just one config server for sharding and make a backup to use it, if the config server crashes?

[13:00:18] <Ben_1> is it possible to run just one config server for sharding and make a backup to use it, if the config server crashes?

[13:06:24] <yopp> cheeser, is it possible that when storage under heavy load, and there a significant delays in io operations, WT can go nuts? Is there any open issues about that?

[13:08:58] <yopp> We're found that when storage under heavy load (esxi vdp stuff is happening), mongo start to eat all the CPU, and performance degrades over the time after that

[13:45:48] <cheeser> yopp: doesn't sound familiar but you should probably talk to the kernel guys. the mongodb-users mailing list would be a good place.

[15:22:44] <tombert1> hello! if anyone here has experience using the mongoose interface for node.js, I'm trying to do an update on a date field, but it throws an error unless I do a `.toISOString()` at the end. Is this the only way to update a date field?

[15:22:51] <gabrielsch> can I aggregate (group, sort, sum) on collection and return values from other? (based from the aggregate)

[15:23:37] <StephenLynx> I suggest you don't use mongoose, tombert1

[15:23:38] <gabrielsch> example: get most sold products (Product collection) based on orders (Order collection)

[15:24:31] <tombert1> StephenLynx: I don't have a choice in the matter. The project has already been written with Mongoose

[15:58:37] <gabrielsch> Hi! How can I project a string of arrays on aggregation? ['a', 'b', 'c']

[17:03:28] <Ben_1> is it possible to run just one config server for sharding and make a backup to use it, if the config server crashes?

[17:03:42] <Derick> not in production

[17:26:58] <Ben_1> (17:59:01) Derick: not in production <<<< not possible or not recommended?

[17:29:32] <Derick> not recommended

[17:36:15] <Ben_1> (18:24:51) Derick: not recommended <<< But if I backup the config servers data once a day, why is it dangerous?

[17:36:36] <Derick> because you might have backedup a corrupted state, and once a day is likely not going to be enough

[17:36:47] <Derick> there is a reason why in 3.2 config servers are becoming a replicaset

[17:48:20] <cheeser> +1

[18:44:41] <dropshot> I have messages from an exchange, which each contain a ticker. The problem is some messages involve trades in two or three tickers. Is there any way to deal with documents with repeated categories?

[19:40:46] <Doyle> Any thoughts on swap memory size for mongodb systems? I'm seeing 100% swap space utilization, with a lot of bi/bo

[19:42:20] <dddh> what is your vm.swappiness ?

[19:42:59] <amcsi> hi

[19:43:06] <Doyle> 30

[19:43:11] <Doyle> looks like default

[19:43:15] <amcsi> does mongod output all commands run by default?

[19:43:26] <Doyle> amcsi, no, the log would be insane

[19:43:42] <Doyle> you can turn on profiling, but be sure you ahve some kind of log rotate setup or your disk could explode

[19:44:28] <dddh> Doyle: 60 is default

[19:45:56] <dddh> Doyle: mongodb 3.0 with default storage engine mmapv1 ?

[19:46:33] <Doyle> dddh, 2.6.7 mmapv1

[19:50:53] <dddh> Doyle: a lot of bi/bo or si/so ?

[19:51:41] <Doyle> r b swpd free buff cache si so bi bo in cs us sy id wa st

[19:51:41] <Doyle> 1 0 1048572 184776 25700 11202116 0 0 4520 973 0 0 4 3 63 30 0

[19:51:41] <Doyle> 1 0 1048511206748 0 0 11544 10184 13135 6966 4 2 78 15 0

[19:51:56] <Doyle> that didnt' work, but ti's all bi/bo

[19:52:00] <Doyle> si/so is 0

[19:52:11] <cambrian> How would I get a list of all potential fields that exist in a collection?

[19:53:09] <cambrian> Essentially derive a universal schema from a collection of documents.

[19:53:12] <Doyle> cambrian, db.collectionName.findOne().pretty() ?

[19:53:49] <dddh> Doyle: you could try to set vm.swappiness=1

[19:55:57] <dddh> cambrian: you can try https://github.com/variety/variety

[19:56:04] <cambrian> @Doyle: That works for 1 document, but some documents might have some extra fields. How could I get the "master schema" describing all possible fields.

[19:56:05] <Doyle> dddh, how much swap do you give a mongodb instance?

[19:56:31] <Doyle> Oh, Variety seems useful

[19:57:14] <cambrian> Yeah, that looks like it does exactly what I am looking for. Thanks!

[19:57:37] <dddh> Doyle: I use 16gb of swap on 64gb sever

[19:57:48] <dddh> but I use zswap

[19:58:41] <Doyle> If you don't use zswap, do you still think it's right to allocate 1.5x the sysmem to the swap?

[19:58:58] <Doyle> I read somewhere that 1GB swap was sufficient for mongodb...

[19:59:25] <dddh> mongodb mmapv1 shouldn't use swap if your server is dedicated

[20:00:07] <Doyle> it's dedicated, and using mmapv1, which is why I'm surprized at the bi/bo counts

[20:00:16] <Doyle> si/so I guess would be worse

[20:00:56] <Doyle> if those climb there's an issue. I'll throw some more GBs at the swap on the next instances that go up and see what they look like

[20:00:59] <Doyle> thanks dddh

[20:07:03] <repxxl> will mongo query be slow when i also use date which is not indexed?

[20:11:53] <dddh> repxxl: check explain plan

[20:15:38] <dddh> repxxl: db.coll.explain().find({doc: 'abc', date: ISODate('2015-11-11')})

[20:16:22] <dddh> query planner could use another index in your query

[20:17:39] <repxxl> dddh a query allways use just one index ?

[20:18:15] <repxxl> dddh so when i have multiple indexes in query i should just make sure it uses the right index first? to avoid heavy load

[20:22:48] <dddh> it can use multiple indexes, but it could scan only one index and if you have an index on 'doc' and db.collection.find({doc: 100}, {_id: 0, doc})

[20:24:32] <amcsi> help, even with this, mongo is still outputtin "I" level info on COMMANDs: mongo_1 | 2015-11-16T20:19:20.385+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", storage: { mmapv1: { smallFiles: true } }, systemLog: { component: { command: { verbosity: -1 } } } }

[20:27:19] <amcsi> mongod*

[20:30:04] <repxxl> guys i wanna send the submited data to all followers do u think 1milion inserts at one time will take long ? assuming a user has 1m followers.

[20:31:22] <cheeser> it will take a while, yes. use bulk operations to help

[20:32:35] <repxxl> cheeser well i have different ideas one is to send all the followers the data like inserts, second is to fetch the data with every user seperately and another idea is to let hadle the inserts a cron job which would not affect the load time when a user submits something for example

[20:43:12] <dddh> I wish mongodb had transactions ..

[20:45:12] <amcsi> how do I set the verbosity level so Informational would NOT be included? 0 seems to be the minimum verbosity and it seems to include Informational

[20:45:23] <amcsi> https://docs.mongodb.org/v3.0/reference/log-messages/#severity-levels

[23:34:43] <guxi> I'm having an difficulty understanding the way to reference documents properly. Say i have following collections: schools, classes and students. I need Students to be referenced to a class, and a class to be referenced to a shool. This would be very simple to store and query in SQL. What would be the best approach for MongoDB, in order to effeciently query referenced data in both directions?