Log file Viewer
#mongodb logs for Monday the 9th of May, 2016
[02:52:30] <ycon_> Hi all, I'm trying to run mongoDB for the F8 app. Not having success- ECONNREFUSED keeps happening on the port http://dpaste.com/0DXPRSE
[03:11:07] <ycon_> joannac: 501 13271 12745 4006 0 31 0 2445080 812 - S+ 0 ttys000 0:00.00 grep mongo
[03:12:46] <joannac> https://docs.mongodb.com/getting-started/shell/tutorial/install-mongodb-on-os-x/#run-mongodb
[03:16:58] <joannac> 2016-05-09T13:08:01.320+1000 I STORAGE [initandlisten] exception in initAndListen: 29 Data directory /data/db not found., terminating
[03:17:55] <ycon_> the instructions here really dont make much sense then hey https://github.com/fbsamples/f8app
[03:19:49] <ycon_> It still doesn't find /data/db when I mkdir -p data/db. It says permission denied if I mkdir -p /data/db (first slash). That the issue? http://dpaste.com/2WTGMAM
[03:28:19] <ycon_> joannac: Any ideas on getting it running? the first slash causes an issue http://dpaste.com/2WTGMAM
[03:35:52] <ycon_> joannac: Now that I've installed to the root- it tells me file: /data/db/mongod.lock errno:13 Permission denied Is a mongod instance already running?, terminating2016-05-09T13:27:27.035+1000 I CONTROL [initandlisten] dbexit: rc: 100
[07:02:51] <Adam0410> Hey guys, I am trying to store a javascript function in a mongo database, for me to retrieve and run (from within node.js not the mongo shell) at a later date, is there a way to do this?
[08:53:25] <jokke> if i add a shard i get this error message: "Can't use localhost as a shard since all shards need to communicate. Either use all shards and configdbs in localhost or all in actual IPs. host: ip-172-31-29-6.us-west-2.compute.internal:27018 isLocalHost:0"
[08:58:16] <gain_> I've a collection like this { username: "me", access: [ { token: "asdf", datetime: "1234" }, { token: "qwer", datetime: "5678" } ] }, { username: "you", access: [ { token: "zxcv", datetime: "9012" }, { token: "poiu", datetime: "3456" } ] }
[09:00:01] <gain_> and I need to have a result like this [ { username: "me", token: "asdf", datetime: "1234" }, { username: "me", token: "qwer", datetime: "5678" }, { username: "you", token: "zxcv", datetime: "9012" }, { username: "you", token: "poiu", datetime: "3456" } ] }
[09:02:47] <kurushiyama> gain_: The first question arising to me is why you do an array of embedded documents the first place.
[09:02:55] <kurushiyama> okke: As for your first problem: either all elements of a sharded cluster are on localhost or none.
[09:04:15] <kurushiyama> jokke: You _configure_ your sharded setup. Now, how should shard a reach a host which is called "localhost", denoting a totally different machine?
[09:14:10] <gain_> kurushiyama: I'm a noob, now I'll dig the aggregation and let you know... thanks for the hint ;)
[09:33:31] <kurushiyama> gain_: Your modelling looks wrong to me. You are logging user logins with access tokens, I assume?
[09:35:30] <kurushiyama> jokke: Well, please add check the configs of the other servers. Please add the sh.addShard() commands you used. The shards and the config server should all listen to something != 127.0.0.1
[09:36:27] <gain_> kurushiyama: yes, I use that "pattern" (having a child array of objects) with different collections...
[09:37:57] <jokke> if i run mongo ip-172-31-29-6.us-west-2.compute.internal:27018 i land on the shard
[09:38:03] <kurushiyama> jokke: make sure DNS is configured properly and that the shard really listens on 27018
[09:38:16] <gain_> kurushiyama: I'm a noob, so if you say to me that is wrong I trust you... where can I find some doc about nosql data modeling and best practises?
[09:39:11] <kurushiyama> gain_: The thing is this: You impose a theoretical limit on the numbe of logins with that model, since BSON docs are limited to 16MB
[09:40:33] <jokke> i assume if `mongo ip-172-31-29-6.us-west-2.compute.internal:27018` works, DNS and port configs are configured properly
[09:41:26] <kurushiyama> jokke: each single node is required to be able to access the host. So that would be config and mongos, iir your setup correctly.
[09:44:02] <gain_> kurushiyama: thanks for the hint! I'll take that into account during db modeling... ^^
[10:16:48] <kurushiyama> jokke: Rule of thumb: each member of a cluster needs to be able to resolve every other member + most likely acess the according ports.
[10:18:14] <jokke> then i don't quite understand why a mongos should run on the application server. I don't see why the application server should be accessible by the db host
[10:23:08] <kurushiyama> jokke: It does not necessarily have to access it: but it needs to be able to resolve it.
[10:23:43] <kurushiyama> jokke: Since you are experiencing problems, you should make sure that everything works as expected first before gettign smart on it ;P
[11:04:01] <chinychinchin> { $or: [ { campaignId: { $in: [ 16861 ] }, organizationId: 11288, dateLog: { $gte: new Date(1461106800000), $lte: new Date(1461365999000) } }, { $and: [ { originId: null }, { request: { $in: [ "MO", "MT" ] } }, { campaignId: null }, { organizationId: 11288 }, { dateLog: { $gte: new Date(1461106800000), $lte: new Date(1461365999000) } } ] } ] }
[12:07:07] <hemangpatel> Is there any alternative for MMS for enterprise ? I already checked compose.io
[12:28:03] <zerOnepal> Hi there, what is the best recommendation on deleting the huge mongo records, I need to delete about 800GB of data on a production server
[12:28:54] <kurushiyama> zerOnepal: documents in collection, collection, whole databases or complete instances?
[12:30:54] <zerOnepal> currently I am using sth like this: to fetch a record I want to delete, and delete it on loop, db.collection.findOne(created_at: {$lt: new Date('2015/11/30 11:59:59');}})
[12:31:42] <zerOnepal> but I wish there exists some batch deletion things, which supports query and doesn't hamper the production traffic
[12:32:06] <kurushiyama> zerOnepal: it is called bulk operations. And every operation affects the other.
[12:32:23] <zerOnepal> I can't drop the whole collections, I need to keep last 8 months records intact
[12:36:37] <kurushiyama> zerOnepal: Depends on wether you want to delete the old documents once or wether you want to always keep data for 8 months only
[12:37:59] <kurushiyama> zerOnepal: As for the removal: db.yourcoll.drop({created_at:{$lt: ISODateEigthMonthAgo }})
[12:41:09] <zerOnepal> isn't this db.yourcoll.drop({created_at:{$lt: ISODateEigthMonthAgo }}) expensive for 800 GB weigh collection ?
[12:42:10] <kurushiyama> zerOnepal: Again, you can not have the cake and eat it. If your hardware dimensions do not fit your use cases, there is something seriously wrong with your setup.
[12:44:52] <zerOnepal> yes, kurushiyama data seems to be accumulated since 5years back, and never been archived...
[12:46:11] <kurushiyama> zerOnepal: There are 3 ways to reduce the impact of such an operation: Add a random wait while iterating over the docs, use bulk operations and execute them after some waiting or do it during low load times.
[12:46:50] <kurushiyama> zerOnepal: If you only ever want to keep the last 8 months, I'd guess that using a TTL index is the best option.
[12:47:39] <zerOnepal> but would that be applicable for very old data, like I have data of 5years horizon
[12:47:41] <kurushiyama> zerOnepal: With using a TTL index, you basically make sure that your collection will never accumulate more than 8month + 1 minute of data
[12:54:49] <kurushiyama> cheeser: Which reminds me of a question I have: Would it make sense to have a "TTL deletion window" similar to the balancing window to reduce the impact of TTL deletions in case there are a lot of deletions at a single point in time? Not that I personally had any noticeable impact of TTL deletions, just curious. I guess basically the question is "How does the TTL removal work internally?"
[13:06:09] <kurushiyama> cheeser: Maybe I should write a feature request and see wether people like it.
[14:07:07] <hemangpatel> kurushiyama, I have stack trace : http://pastie.org/private/vuqrsy94zexupf1efv56rg
[14:10:28] <kurushiyama> hemangpatel: I am pretty sure that I told you where to look for the problem, and even gave you a hint what to look for...
[15:48:58] <bros> I know Mongoose sucks, but can anybody here answer a question related to its Schema class?
[15:49:53] <kurushiyama> bros: Aside from "Do not use it"? I am afaraid I can not. If it is about modelling in general, I may.
[16:25:05] <emacsnw> suppose I have a collection of users (email, status, ...). with index {email: 1, status: 1}, query db.users.find({email: "foo@bar.org", status: {$in: [0, 1]}}) scanns the same amount of documents as db.users.find({email: "foo@bar.org"}) does.
[16:31:50] <landonn> cheeser: did you just kick me fron ##java ? i said it was the wrong channel, no need to do that
[16:33:19] <landonn> i don't like how a mongodb employee kicks people who critize mongodb in a Java community channel anyway. what an asshole.
[16:36:57] <StephenLynx> I also don't think what people do on other channels is pertinent to this channel.
[16:38:17] <kurushiyama> landonn: You make a lot of assumptions and conclusions based on them. And then, calling somebody an asshole based on mere assumptions is not exactly what I would call polite, reasonable or justified.
[16:40:37] <StephenLynx> so you would have to take it to somewhere where the company actually handles.
[16:42:34] <kurushiyama> I'd rather guess somebody took the supersonic shortcut through his upbringing.
[17:06:11] <BrianBoyko> Hello. I'm getting a very weird error from Mongo. It works perfectly from one view of my front-end, but not another, despite the code being almost identical. :( I get AssertionError: { [MongoError: key $end must not start with '$'] -- problem is, I'm not using $end anywhere that I can see.
[17:08:35] <xialvjun> I have a foreignkey bid ref to another collection's _id...I don't know should I save the bid as a normal string or an ObjectID
[17:25:09] <silviolucenajuni> xialvjun ObjectID is smallest that String. Have a question about this on StackOverflow http://stackoverflow.com/questions/18680462/mongodb-benefit-of-using-objectid-vs-a-string-containing-an-id
[17:33:50] <kurushiyama> xialvjun: Another thing is that you are not forced to use ObjectId as _id. You can use an int or even objects...
[17:37:03] <kurushiyama> xialvjun: Just to let you know that you do not _have_ to use ObjectId. For a lot of applications, a more natural _id might well be useful.
[17:44:41] <kurushiyama> cheeser: For the record: Even if you did, considering his behavior here, it would have presumably been for a good reason.
[18:29:40] <skullcrasher> would you recommend mongodb also for tasks that could be "easier" solved by using a rdbms like user management? Reason I'm asking is I can think of this user managment in mysql and the rest of the data in mongodb, but all in on db would be nice maybe
[18:30:51] <kurushiyama> skullcrasher: a) User management is at least as easy in MongoDB as it is in MySQl b) even if it wasn't dealing with two datasources creates an overhead negating any, even just perceived advantage.
[18:32:35] <skullcrasher> so basically going "the hard way", and looking how to transfer such stuff to mongodb
[18:32:58] <kurushiyama> skullcrasher: If user management is hard for you in MongoDB, I would not suggest to implement something security related until it is easy for you.
[18:33:35] <kurushiyama> skullcrasher: No offense, but it is _extremely_ easy in MongoDB, and if it is hard, you probably do not know enough, yet.
[18:33:59] <skullcrasher> kurushiyama, well I know about embedding etc, just not sure on how to setup this all properly.
[18:34:42] <kurushiyama> skullcrasher: For starters: Embedding should only be used for limited sized subdocs or arrays of subdocs.
[18:35:04] <kurushiyama> skullcrasher: That is one thing you should get rid of asap: The fear of redundancy
[18:35:29] <cheeser> i would argue that user management is easier in mongodb given the flexible nature of metadata
[18:36:17] <skullcrasher> kurushiyama, so redundancy is ok in many parts I think, just curious about updating this then.
[18:36:39] <skullcrasher> if e.g. user adress is saved in 4 different locations and he updates this.
[18:37:03] <kurushiyama> skullcrasher: Here is what I usually suggest: First, et your use cases /user stories right. Next, derive the questions you have on your data from them. Then, model accordingly, so that your questions get _answered_ in the most efficient way.
[18:37:13] <skullcrasher> although a heavier update wouldn't be that problem then, as it isn't changed that often, right?
[18:38:38] <skullcrasher> kurushiyama, well I did the mongodb java course (was quite good, but missed many things already :P)
[18:38:38] <kurushiyama> skullcrasher: because you correctly stated that a user changes his username probably... what... every half year?
[18:39:27] <kurushiyama> skullcrasher: Well, Here is what I suggest: Forget JPA. Forget JDO, and for god's sake, forget Spring Data.
[18:39:28] <skullcrasher> the "technical" stuff like sharding, queries etc on an existing data structure is ok to do
[18:40:38] <kurushiyama> skullcrasher: Presumably, you are accustomed to "Identify entity, identify properties, identify relations"
[18:41:09] <kurushiyama> skullcrasher: With NoSQL in general, and MongoDB in particular, you do it the other way around.
[18:43:10] <skullcrasher> kurushiyama, ok. I think I should find more to read about data modelling until asking new questions ;P
[18:43:26] <jeroentrappers> I have a small question about the mongodb journal prealloc files. We have a deployment on AWS using 3 volumes, one for the rootFS, one for the data and one for the logs. It seems that the journal prealloc files /var/lib/mongodb/journal are allocated on the rootFS volume, while the actual data and journal reside on /data (data volume)
[18:46:34] <jeroentrappers> Would it not be more logical for mongodb to prealloc the journal files under the database directory as well?
[18:47:11] <jeroentrappers> we just didn't take the prealloc files into consideration when choosing mount points for our data volume. Obviously
[18:47:25] <hdkiller> i am facing some scaling issues and i would like to ask what route i should choose to be able to handle more connections/requests per secs on a 'relative small' dataset
[18:47:58] <hdkiller> i need to set up replicasets? or is there a decent up to date guide what points me toward the proper way of doing this?
[18:50:26] <kurushiyama> hdkiller: However, the drivers are replica set aware, so failover can be done fairly transparent
[18:50:56] <kurushiyama> jeroentrappers: but that is no real load distribution, and furthermore we still have a tradeoff in terms of consitency
[18:51:22] <jeroentrappers> yes, sure... But I would advice to setup a replicaset, and then apply sharding as well
[18:52:10] <kurushiyama> I am with jeroentrappers here. If you have enough data for a sharded cluster, it is probably worth thinking about availability.
[18:52:53] <jeroentrappers> hdkiller -> read up on replica sets for availability, and apply sharding for more performance, better distribution of load
[18:55:57] <kurushiyama> Derick: Well, I was "invited" to have me listed there, so I happen to know ;)
[18:55:59] <jeroentrappers> I was still thinking about my peralloc files, and instead of changing the mount points, I could just symlink the /var/lib/mongodb/journal onto the /data volume
[18:56:42] <jeroentrappers> that should also fix my volumes / data distribution problem one per node.
[18:58:20] <uatec> my document has a bunch of children fields, and one of them is an array. When I do my upsert, it merges the old and new arrays.
[18:58:33] <uatec> However, the old and new are both the same, so i'm getting loads of duplicates. :| What Can I do about this?
[19:07:03] <jeroentrappers> uatec: in your merge logic, treat the array as a set https://lodash.com/docs#uniq
[19:24:52] <uatec> although, jeroentrappers, i'm not sure i can see how to specify my own merge logic
[21:10:30] <Gloomy> Hi :) Have a small question, when I do find({field: 'string'}), it finds also all documents where 'string' is only a part of the field (ex field: 'stringy', field:'superstring'). How do I avoid this?
[21:11:18] <Gloomy> hmm my bad, I went to fast with my writing :D It is a regexp match. Why does it do it?
[21:13:37] <Gloomy> Hmm I'm not super good with regex but it's /string/, not /.?string.?/ ? shouldn't it only return exact matches?