pmxbot IRC Log Viewer

[00:27:44] <pokEarl> Hmm, any suggestions on best way to write tests in Java when mongoDB is involved? an Embedded mongoDB (flapdoodle?) Fakemongo/Fongo, DBUnit, or using mockito is what seems to be the alternatives, but I don't really know what I am doing :(

[00:34:38] <cheeser> i just test with a dummy mongo. i'm writing this to make that easier: https://github.com/evanchooly/bottlerocket

[00:42:37] <pokEarl> Hmm ok, thanks for the suggestion cheeser, will look into it more tomorrow.

[04:13:11] <daidoji> hello, did mongoexport use to report records/s or am I misremembering?

[08:01:16] <nixstt> what can I do about a replicaset member stuck in rollback?

[08:03:17] <nixstt> is there anyway to bring a member stuck in rollback back online or do I have to do a full resync

[08:07:49] <kali> nixstt: it depends. if it is stuck before the first record of your oplog, then you need a full resync

[09:14:17] <_blizzy_> could anyone helpe me figure out why when I run this, on line 24, I get the error 'TypeError: undefined is not a function.' https://gist.github.com/NotBlizzard/60360e599ab7b995a33d

[09:23:49] <kas84> Hi

[09:24:22] <kas84> is there a way to do a mainteinance operation like the drop of a collection without affecting so much to the whole mongodb engine?

[09:45:41] <kali> kas84: mmmm collection drop has an impact on your system ? it should be relatively painless

[09:45:56] <kas84> yes

[09:46:00] <kas84> 100% mongodb cpu

[09:46:19] <kali> you really drop the collection ? not remove the documents from it ?

[09:46:29] <kas84> I’m using mongo 3 so I guess it has to delete all documents in the collection

[09:46:39] <kas84> in this case, I didn’t mind

[09:47:05] <kas84> but it’s pretty much the same, if any, drop should have better perf, right?

[09:48:05] <kali> yeah, drop on the collection should be fast. remove() with no filtering, on the other hand will be linear time

[10:25:31] <aps> Why is WiredTiger not the default storage engine yet? Is it not production ready?

[10:27:45] <coudenysj> aps: it will be the default in a bit, if I'm correct

[10:29:51] <aps> coudenysj: So, what should I use if I'm creating a new db right now?

[10:35:06] <_blizzy_> could anyone help me figure out why at line 78, it's not saving the changes? https://gist.github.com/NotBlizzard/000ca4b492ac8cdce2d6

[10:40:06] <aps> I want to use WiredTiger as the storage engine but my team-mates are skeptical since it's not the default yet. May I know the reason why it's still not default? Is it not production ready yet?

[10:49:15] <_blizzy_> hello?

[10:49:52] <lmatteis> guys

[10:49:58] <lmatteis> how do i restore from json.gz files in a folder?

[10:53:30] <kali> lmatteis: chances are high you can't. mongodb bson format is richer than json, so json is not a suitable format for db backup and restore. what you can do is "import" these json files

[10:53:42] <_blizzy_> could anyone help me figure out why at line 78, it's not saving the changes? https://gist.github.com/NotBlizzard/000ca4b492ac8cdce2d6

[11:07:57] <pokEarl> cheeser (or whoever else might know) asked you yesterday about testing in Java for mongoDB and you suggested a dummy mongo, I am told we do have a test database set up as a replicate of our actual Database that I can use for testing, but I have gotten the impression that relying on an external Database while testing is not good?

[11:09:32] <cheeser> pokEarl: i don't mind it, personally. mocking a database doesn't actually test much of your code

[11:10:49] <pokEarl> But is there any major downside to just connecting to an external DB that I am not seeing vs setting up an in memory/embedded DB for the test?

[11:11:13] <cheeser> major? no.

[11:11:46] <cheeser> but there is no embedded form of mongodb so you'll need something external to your tests anyway.

[11:14:03] <pokEarl> Just that tests should optimally always be self sustaining etc I guess, but can probably do that at some later point if I/We find it wortwhile =p There seems to be something called flapdoodle which helps you with have an embedded mongo for testing but I might be lying here

[11:50:42] <Lope> Is there anything bad about storing HTML files in mongoDB for log purposes?

[11:55:33] <Lope> I need to keep zip files that are about 70K in size for about a week (11MB/user) or a month (44mb/user). I'm not at all worried about storage requirements in the near future given the number of users.

[12:02:32] <cheeser> you could use gridfs for that.

[12:02:48] <cheeser> or just store the html pages directly. you could even use TTL indexes to age them out of the system.

[13:04:16] <bogn> here is what pokEarl seems to be referring to: https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo

[14:00:26] <_blizzy_> could anyone help me figure out why my data isn't saving? https://gist.github.com/NotBlizzard/000ca4b492ac8cdce2d6 line 78

[14:01:20] <ito> hi. when i'm doing a mongodump/mongorestore on a replica set these messages appear in the verbose output:

[14:01:28] <ito> [ReplicaSetMonitorWatcher] dbclient_rs replSetGetStatus failed{ ok: 0.0, errmsg: "unauthorized" }

[14:01:37] <ito> could someone be so kind and shed some light on the reason/meaning of these messages?!

[14:08:39] <deathanchor> ito: using authentication on your dbs?

[14:08:50] <ito> deathanchor: yes

[14:09:02] <deathanchor> sounds like that's the problem

[14:09:26] <deathanchor> sorry I don't have much experience with authentication

[14:12:23] <ito> deathanchor: thanks anyway :-) i'll hang out here a little more maybe someone has a hunch what's causing the problem.

[14:12:39] <_blizzy_> sorry, I disconnected.

[14:12:45] <_blizzy_> anyone think they know the problem?

[14:21:49] <_blizzy_> sorry for double posting, but I disconnected earlier, so

[14:21:52] <_blizzy_> could anyone help me figure out why my data isn't saving? https://gist.github.com/NotBlizzard/000ca4b492ac8cdce2d6 line 78

[14:23:20] <StephenLynx> try without mongoose.

[14:23:31] <StephenLynx> it does nothing but get in the way and be slow.

[14:23:47] <_blizzy_> oh.

[14:27:17] <Lope> cheesr: that's amazing. But I don't want to JUST save files, I want to save them as part of a document with their meta data.

[14:27:42] <StephenLynx> Lope

[14:27:44] <StephenLynx> what you need?

[14:27:52] <StephenLynx> you can add metadata to files on gridfs.

[14:29:49] <Lope> StephenLynx: wow, amazing.

[14:30:02] <Lope> Okay I need to store some binary data (compressed HTML)

[14:30:22] <StephenLynx> sure

[14:30:24] <Lope> along with some meta data that describes it

[14:30:27] <StephenLynx> no problem

[14:30:31] <StephenLynx> that is what I do on lynxchan

[14:30:44] <StephenLynx> when you write the data, you just provide the metadata object.

[14:30:46] <Lope> And I want to get rid of it after a week or a month. (I'll see based on storage requirements)

[14:31:03] <StephenLynx> you can store a field on the file with its expiration

[14:31:13] <StephenLynx> and run on a schedule a task to delete all expired documents.

[14:31:16] <Lope> So the metadata object encapsulates the binary data with the metadata?

[14:31:19] <StephenLynx> you can't use tll expiration on gridfs.

[14:31:27] <StephenLynx> no, the metadata is stored separately.

[14:31:36] <StephenLynx> gridfs uses two collections

[14:31:38] <StephenLynx> chunks

[14:31:40] <StephenLynx> and files

[14:31:46] <StephenLynx> chunks store the data itself

[14:32:02] <Lope> I see

[14:32:10] <StephenLynx> files stores all information regarding the files, including a field that contains an object that you can put arbitrary data.

[14:32:23] <StephenLynx> this field is called metadata

[14:32:26] <Lope> wasn't GridFS a 3rd party hack? has it been merged into the official Mongo code now?

[14:32:41] <StephenLynx> dunno, started using mongo a year ago.

[14:32:53] <Lope> I recall 2-3 years ago GridFS was regarded as experimental.

[14:33:12] <StephenLynx> my current project revolves around gridfs, no issues so far.

[14:33:30] <Lope> Yeah, well I see they recommend GridFS on the mongoDB website.

[14:33:33] <StephenLynx> aside from the node driver storing strings as something else other than utf-8.

[14:33:37] <Lope> Well I'd just like to do whatever is simplest.

[14:33:53] <Lope> I think just storing my compressed HTML inside normal documents will be the simplest, surely?

[14:33:58] <StephenLynx> but that was easily fixable, I just convert strings to utf-8 buffers.

[14:34:04] <StephenLynx> no way.

[14:34:09] <Lope> The average file size is only 70KB

[14:34:23] <StephenLynx> with grifs you can stream the files and handle them as actual files.

[14:34:33] <StephenLynx> besides, it will require less work.

[14:34:41] <Lope> Yeah, but I'm only keeping the files to investigate if something goes wrong.

[14:34:51] <Lope> I'm not serving them from a webserver or anything.

[14:34:55] <Lope> They're log'ish.

[14:35:03] <StephenLynx> still easier.

[14:35:15] <StephenLynx> with gridfs you can just get them from the server.

[14:35:22] <Lope> Okay, well i've never stored binary data inside a document before.

[14:35:42] <Lope> So I don't know what that involves. I don't know anything about GridFS either.

[14:35:54] <StephenLynx> try it, its pretty simple.

[14:35:56] <Lope> Can you recommend a "hello world" gridfs example? (read and write with meta data)

[14:36:06] <StephenLynx> which driver?

[14:36:30] <Lope> I'm using Node.JS

[14:36:40] <Lope> Using mongojs which is a wrapper for the official mongo driver.

[14:37:07] <StephenLynx> I can point you either to the official documentation which contains examples, let me get it

[14:37:10] <Lope> It provides an interface which is as closely identical to the mongo terminal application as possible.

[14:37:16] <StephenLynx> http://mongodb.github.io/node-mongodb-native/2.0/api/GridStore.html#write

[14:37:24] <StephenLynx> or to how I use it on my project

[14:37:32] <Lope> Thanks bro, very kind of you :)

[14:37:49] <StephenLynx> https://gitlab.com/mrseth/LynxChan/blob/master/src/be/engine/gridFsHandler.js#L24

[14:38:04] <StephenLynx> I both write data and transfer files from the filesystem to it.

[14:38:07] <Lope> I'll BRB

[15:02:37] <bogn> last time I had a look at GridFS gridfs-fuse was the way to mount the files as a file system that you can browse. gridfs-fuse was alpha software, has that changed?

[15:04:27] <bogn> well the project hasn't been updated since then: https://github.com/mikejs/gridfs-fuse

[15:07:00] <bogn> hm, but it has a lot of forks: https://github.com/mikejs/gridfs-fuse/network

[15:19:46] <nixstt> is it safe to kil a running remove() query that’s been running for a long time in the mongo shell?

[15:30:10] <skiold> i'm looking into stats extraction for members of a replica set using the v2 ruby driver. If I'm not explicit about the driver .new() options the code always connects to the primary. What makes more sense in the options hash: `:connect => :direct` or `:read => {:mode => :nearest}`

[15:30:25] <skiold> i'm always connecting to the mongod on localhost

[16:06:47] <spokrandt> hello, trying to get mongo up for my 2nd yime. built 3 systems got them up and running and had to change there names. I reconfigured the replica set and the master sees the replicas as DOWN. can someone point me to a doc that explains how to drop everything and start over.. and yes I've looked on google

[16:31:46] <kali> spokrandt: just strop them and delete everything in the dbpath directory

[19:32:10] <plastikman> I have a question about Mongos

[19:32:35] <plastikman> If i only have a 3 node cluster, one master, one slave, and one arbiter...Do i actually need mongos?

[19:32:44] <kali> no

[19:32:48] <kali> mongos is for sharding

[19:33:32] <plastikman> That is what I thought. I was led down the wrong path by the consultant....shocker.

[19:33:49] <plastikman> So basically there is no need for mongos until I need to shard.

[19:33:59] <kali> yep

[19:34:01] <plastikman> thanks

[19:34:20] <kali> it would only make sense if you're pretty sure you'll need sharding and want to avoid the migration later

[19:34:32] <kali> but... that migration is not very to do anyway

[19:34:37] <kali> +hard

[19:35:09] <plastikman> well, right now i will not be crushing these machines anyway.

[19:36:48] <plastikman> so if I want to split up my read and writes, would I just talk to the master for writes then talk to the slave for reads? does the arbiter do anythign other than vote in the event of a failure?

[19:37:18] <plastikman> I am a complete mongoDB newbie

[19:37:26] <plastikman> I am sorry if these questions are simple

[19:38:33] <kali> the arbiter only votes, it does not hold any "real" data

[19:39:09] <plastikman> Understood. It is not a router or anything special like that. Just there to protect against split brain

[19:39:33] <plastikman> basically

[19:39:35] <plastikman> ?

[19:39:36] <kali> plastikman: the client will let specify at various levels (connection, db, collection, query) if you want the reads to occur preferably on a slave or the master

[19:40:07] <kali> by the way, the right terminology is "primary" and "secondary"

[19:40:11] <plastikman> thanks

[19:40:12] <plastikman> :)

[19:40:51] <plastikman> So in a nutshell: The clients need to be individually configured to speak to the "Primary" for writes and the "Secondary" for reads.

[19:41:22] <plastikman> If I end up splitting up the jobs a node failure is not transparent, correct?

[19:41:29] <kali> it's even simpler than that. you provides them the address of one (or preferably several) of the nodes, and they find out what's what

[19:41:43] <plastikman> s/jobs/read\/write/

[19:41:51] <plastikman> ah

[19:42:20] <plastikman> So If I attempt to write to the secondary, I will be sent to the primary by mongo?

[19:43:06] <kali> if you connect a client to a secondary, it will find out the topology and open a connection to the primary too

[19:43:32] <kali> when you perform a write, it will be sent to the primary. for a secondary, it depends on the preference you specified

[19:43:41] <kali> s/for a secondary/for a read/

[19:44:42] <plastikman> excellent, that helps

[19:45:23] <kali> and it was free

[19:45:31] <plastikman> :D

[19:45:37] <plastikman> you just samed me 20K in support costs

[19:45:38] <plastikman> hahah

[19:45:44] <plastikman> *saved

[20:04:20] <Doyle> Hi guys. Is there a preferred linux distro for mongodb? Centos vs Debian?

[20:23:46] <symbol> Curious if any of you actually have documents within a collection with differing structure. While a lot of the intros to Mongo like to show that you can store varying structure...I haven't seen it done in the wild.

[20:24:46] <StephenLynx> hm

[20:24:48] <StephenLynx> I do that.

[20:25:08] <symbol> I'd love an example :)

[20:25:08] <StephenLynx> I store bans and range bans on the same collection

[20:25:18] <StephenLynx> the difference is that bans have an ip and range bans a range.

[20:25:27] <StephenLynx> also range bans don't have an expiration

[20:25:37] <StephenLynx> I think their differences are those.

[20:25:47] <StephenLynx> most of their stuff is common though

[20:25:59] <symbol> not familiar with bans, what's the collection called?

[20:26:03] <StephenLynx> bans

[20:26:05] <StephenLynx> :v

[20:26:14] <StephenLynx> let me get the doc

[20:26:31] <symbol> Ohhhh, as in banning an ip or range.

[20:26:34] <louie_louiie> @Doyle - i am not sure about the others on here, but i use "yum" and its supported pretty well so i am guess Centos works

[20:26:49] <StephenLynx> I use with centOS too

[20:26:52] <StephenLynx> works perfectly

[20:26:58] <StephenLynx> I tried debian once for a server and hated it.

[20:27:01] <louie_louiie> Debian uses apt-get if i am not mistaken. i am not sure how supported it is

[20:27:06] <StephenLynx> https://gitlab.com/mrseth/LynxChan/blob/master/doc/Model.txt symbol

[20:27:31] <symbol> StephenLynx: This is for your fourum, yeah?/

[20:29:18] <symbol> So, would a document with range ban have a null ip or would it be missing the ip field entirely?

[20:29:26] <Doyle> Haha, thanks StephenLynx

[20:29:41] <Doyle> I prefer Centos when I can't shell out for RHEL

[20:29:54] <Doyle> or there's no need for support

[20:30:32] <StephenLynx> it would be missing, I guess.

[20:30:48] <StephenLynx> since mongo treats null and undefined the same, I don't think it makes much of a difference.

[20:31:08] <StephenLynx> and this is for a new project

[20:31:21] <StephenLynx> but it is pretty much a forum of sorts too

[20:31:49] <symbol> That makes sense - I haven't started working with driver schemas just yet...been mostly in the shell.

[20:32:26] <StephenLynx> I don't use a schema.

[20:32:35] <StephenLynx> I just have this documentation that I follow in my application.

[20:32:49] <StephenLynx> if I mess, the data gets messed.

[20:33:00] <StephenLynx> which used to happen often :^)

[20:33:03] <StephenLynx> like update without set

[20:33:05] <symbol> Gotcha - what driver do you use?

[20:33:11] <StephenLynx> regular node.js one.

[20:33:17] <symbol> Mongoose?

[20:33:22] <StephenLynx> nope

[20:33:27] <StephenLynx> mongoose is cancer coded.

[20:33:34] <StephenLynx> I wouldn't ever use it.

[20:33:39] <symbol> I hate their documentation.

[20:33:42] <symbol> _hate_

[20:33:48] <StephenLynx> I hate their existence.

[20:33:52] <symbol> hehe :D

[20:33:56] <symbol> So you use the native one then?

[20:34:00] <StephenLynx> yes

[20:34:17] <symbol> Oh sweet baby jesus, their docs just made me smile.

[20:34:26] <StephenLynx> yeah, its great.

[21:31:29] <duallain> I'm sure this is a silly question, but I've been looking for a bit and couldn't find an answer. I've tried to using WiredTiger, how can I verify that it was successful?

[23:01:07] <meonkeys> anyone using the slack integration? I'm wondering if channel name has to somehow include team name, or if it is really just "#channel"

[23:01:21] <meonkeys> (also, it would be really nice if there were some way to test alerts)

[23:01:31] <StephenLynx> what is slack

[23:02:19] <meonkeys> StephenLynx: https://slack.com - a chat app

[23:02:45] <meonkeys> Like IRC but not as good, and not FLOSS. :)

[23:03:10] <StephenLynx> let me guess

[23:03:15] <StephenLynx> boss told to use it

[23:03:49] <meonkeys> uh, anyway, here's the (resolved) ticket I'm referring to: https://jira.mongodb.org/browse/MMS-1685 . Can't find docs for same.

[23:12:47] <diegoaguilar> Hello I'm tyring this update db.groupg18.find({_id: new ObjectId("55a5502fef67112c685cf9a0"), "admins.id": "55a54ce5ef67112c685ce8ab"},

[23:12:48] <diegoaguilar> {$set: {"admins.$.name": "JORGE MIGUEL CECIN SAID"}

[23:12:48] <diegoaguilar> })

Log file Viewer

Help | Karma | Search:

#mongodb logs for Wednesday the 15th of July, 2015