pmxbot IRC Log Viewer

[00:12:54] <synthmeat> can i somehow "flush" working set? at app start i need to run through all the documents, but not during later operation of the app

[00:13:00] <synthmeat> (2.6.7)

[00:18:18] <joannac> synthmeat: for what purpose

[00:18:38] <joannac> put it this way, whatever is in memory is kind of useful

[00:18:51] <joannac> flushing guarantees nothing in is memory

[00:19:33] <synthmeat> joannac: because i don't have memory and i will most likely not touch 99% of those documents from initial startup ever again

[00:20:03] <synthmeat> joannac: disclaimer here that i don't know nothing of mongo at scale

[00:20:11] <synthmeat> i run 1GB instance ^_^

[00:20:12] <joannac> the os is smart enough to page out things it doesn't need

[00:20:28] <synthmeat> that's good enough answer for me :)

[00:20:40] <joannac> you can, but you're trading 1% possible useful stuff for flush and guaranteed 0% useful stuff in memory

[00:21:21] <synthmeat> but, this way i have to guarantee that ALL docs from my collection can fit in memory

[00:21:30] <joannac> ?

[00:21:55] <joannac> yes, if you're accessing all your docs, and they won't all fit in memory, your performance will suck

[00:21:58] <synthmeat> well, if my app at startup run through all the docs in the collection, won't they all go to memory?

[00:22:17] <joannac> yes

[00:22:24] <joannac> unless they don't all fit

[00:22:31] <synthmeat> but, if i'm not accessing them again for the most part, the memory consumption should go down and stabilize?

[00:22:32] <joannac> at which point some will then be paged out

[00:22:41] <joannac> nope

[00:22:54] <joannac> put it this way, your memory should always be at 100%

[00:23:01] <joannac> if it's not, you wasted money on RAM

[00:23:18] <synthmeat> hm, ok, that makes sense too

[00:23:30] <synthmeat> will ssd and swap help?

[00:23:35] <joannac> yes

[00:25:04] <synthmeat> thank you, joannac

[00:27:06] <pamp_> hi

[00:27:52] <pamp_> I need to query data where document have two fields with the same vale

[00:28:02] <pamp_> how can I do it?

[00:28:18] <pamp_> {a :1,b:1}

[00:28:25] <joannac> aggregation

[00:28:56] <pamp_> Yes I know, aggregation is the solution, but how

[00:29:06] <pamp_> can U show me an example?

[00:29:59] <joannac> $project stage with $eq

[00:30:25] <joannac> $project: {a: $a, b: $b, c: {$eq: [$a, $b]}}

[00:30:52] <pamp_> thanks @joannac

[00:30:59] <pamp_> I will try now

[00:34:54] <pamp_> hey

[00:35:33] <pamp_> In that way we get all documents, isn't possible get only the values with the equality

[00:37:04] <pamp_> db.testes.aggregate([ {$project: {a:"$a",b:"$b", c:{$eq:["$a","$b"]}}}, {$match:{c:true}} ])

[00:37:59] <pamp_> I done like this, and it work, but doesn t exist a simple way to do that

[03:04:55] <GitGud> up

[06:32:05] <fishstiicks> howdy. why might i be getting incorrect documents back from a simple find()? my collection and schema are both very straightforward. i'm using mongolab and have double checked my documents in the browser multiple times. i'm only "selecting" on one string value.

[07:02:32] <joannac> fishstiicks: pastebin please

[07:17:17] <fishstiicks> joannac: i've never pastebin'd this sort of problem. what would you like me to send, specifically?

[07:18:14] <joannac> fishstiicks: the query and the result?

[07:25:12] <fishstiicks> joannac: sorry, pastebin was having a moment

[07:25:18] <fishstiicks> joannac: http://pastebin.com/JGVzQ9hR

[07:27:00] <fishstiicks> joannac: there's another 'vendor' string that i can use, and interestingly enough, it returns all matches PLUS some of the 'gafy' matches (but not all of them)

[08:44:28] <Freman> hello

[08:44:30] <Freman> this is slow db.log_info.find({preview: /upstream/i, date : {$gte: ISODate("2010-10-15T15:25:00.000+10"), $lt : ISODate("2010-10-15T15:35:00.000+10")}})

[08:44:32] <Freman> can I do it in a way that reduces the range of data to seach to the date range first, before running regex over it?

[09:01:38] <vfuse> is it normal behavious for mongodb when writing to hourly documents to build up CPU as well as disk I/O till the end of the hour like this http://imgur.com/ngevNs5

[09:04:10] <bendem> vfuse, what do yo mean by "writing to hourly documents"?

[09:04:47] <vfuse> upserting data every minute for each hour to a document

[09:06:30] <bendem> I doubt it's normal/intended

[09:07:05] <bogn> depends on the storage engine

[09:07:51] <vfuse> I’m using wiredtiger, it just seems odd and i’m afraid it won’t scale well when traffic goes up

[09:11:04] <bogn> you shouldn't be doing in-place updates with WiredTiger

[09:11:16] <bogn> in high frequency that is

[09:13:58] <bogn> and this whole preallocation thing that you need for time series using MMAP should no longer be used with WiredTiger

[09:14:21] <bogn> preallocation is actually a joy to get rid of

[09:19:43] <vfuse> bogn: I see it has to do wiht getting rid of padding factor and reusing space so it will have to rewrite the whole document every time

[09:22:52] <bogn> as far as I recall using in-place updates with WiredTiger is close to writing the whole document every time, yes

[09:23:55] <bogn> array appending is the way to go with metrics stored in WiredTiger. So far for my theoretical knowledge gathered from the docs and MongoDB Days events

[09:27:40] <vfuse> I’m asuming you mean using $push when talking about array appening instead of using $set?

[09:28:56] <bogn> yes

[09:29:28] <vfuse> even using $push it will still rewrite the whole document every time?

[09:29:35] <bogn> but don't grow arrays unbounded, use hourly (or otherwise grouped) docs still

[09:29:40] <bogn> no

[09:29:45] <bogn> it shouldn't

[10:13:03] <m3t4lukas> hey guys

[10:13:59] <m3t4lukas> with a TTL index, does the record get excluded from the index after expiration or does the document get deleted after expiry of the TTL?

[10:15:20] <bogn> vfuse do you have any indications that mongod writes whole documents every time even though you are using $push?

[10:15:56] <vfuse> bogn: I’m in the process of switching my upserts from $set to $push i’ll let you know when it’s done

[10:16:32] <bogn> when you say upsert you actually mean update do you?

[10:16:46] <bogn> ah, nevermind

[10:18:35] <vfuse> update with upsert flag

[10:20:02] <bogn> the fact that WiredTiger doesn't like in-place updates stems from it using a MVCC approach

[10:23:42] <bogn> or rather: the fact that when using $set with WT is not using in in-place updates but full updates stems from it using a MVCC approach

[10:52:27] <gulzar> !ping

[11:20:43] <pamp> Hi

[11:21:09] <pamp> its possible using aggregation framework with Linq in c#

[11:21:10] <pamp> ??

[11:21:13] <pamp> driver 1.1

[11:24:40] <gulzar> can we create and import a csv file in a single step, on mongodb?

[11:24:57] <gulzar> 1st step --create a database

[11:25:08] <gulzar> 2nd import a csv file to it

[11:27:41] <pamp> parse the file.. I think

[12:37:55] <MadWasp> Hey guys, I’m using the MongoDB Hadoop Plugin, what would my mongo.input.uri look like if I read from multiple collections?

[12:38:11] <MadWasp> It’s stated in the docs that it’s possible but it doesn’t say how

[12:43:30] <MadWasp> found it :)

[13:05:01] <vfuse> bogn: so far using $push instead of $set hasn’t changed anything I/O wise

[13:24:23] <bogn> do you still preallocate

[13:28:57] <vfuse> no not preallocating

[13:30:23] <livcd> doh .explain() does not return number of scanned objects

[13:30:51] <livcd> do i need to enable something ?

[14:43:38] <mrmccrac> anyone know if there are issues with the redhat repo?

[14:43:40] <mrmccrac> http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/RPMS/mongodb-org-3.0.7-1.el6.x86_64.rpm: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"

[17:15:17] <pehlert> Hey folks. I'm wondering whether there is a method to remove multiple documents and return them? Like findAndModify, just for more than one document

[17:37:17] <JKON> Hi..! I'm using running mongodb mapreduce calculations from node/mongoose env and I was wondering if there is some way I can define the map and reduce on client side and for finalize call function stored on server?

[20:26:01] <user1> Hi. Newbie question, I'm just getting started with mongo. My app consists of status data (flat key/value list) for different devices. What would be preferable to deal with this: each device gets its own flat collection (collectionX pymongoclient.db.deviceX), or a single collection where key = devicename, and value = list of status k/v pairs? (collection is pymongoclient.db.all, filter for name=deviceX when writing data). The for

[20:26:01] <user1> mer seems better but if it doesn't matter then I'd rather do the 2nd one, to simplify things for the reader of the data.

[20:48:32] <Joker_> hello all

[20:49:21] <Joker_> any php heads in here willing to help troubleshoot a mailing script?

[20:49:58] <idd2d> #php?

[21:33:18] <shlant> hi all. Anyone know why I would get a SELF_SIGNED_CERT_IN_CHAIN error when trying to connect to mongo with mongoose? I know my key, cert and ca are correct as I can connect outside of mongoose (with shell)

[21:45:16] <NotBobDole> Okay all, i'm running mongodb in a centos docker container. it times out when setting up the prealloc files on a fresh install

[21:45:16] <NotBobDole> https://docs.mongodb.org/manual/core/journaling/

[21:45:34] <NotBobDole> https://bugs.launchpad.net/tripleo/+bug/1468246

[21:45:45] <NotBobDole> is a bug related to my issue, but I can't find a resolution to what I'm doing.

[21:46:51] <NotBobDole> is a bug related to my issue, but I can't find a resolution to what I'm doing.

[21:46:57] <NotBobDole> woops.

[21:46:59] <StephenLynx> is it exclusive to the container?

[21:48:47] <NotBobDole> What do you mean exclusive?

[21:49:29] <StephenLynx> does the bug happen when you run it outside the container?

[21:51:03] <NotBobDole> StephenLynx: Seems to. Low powered laptop, and I'm controlling this with systemd. Might be a systemd timeout issue.

[21:57:01] <NotBobDole> https://bugzilla.redhat.com/show_bug.cgi?id=1040573

[21:57:11] <NotBobDole> I'm installing mongodb 3, by the way

[22:04:54] <NotBobDole> Not a mongo problem I guess. Mongod doesn't provide a mongod.service file. i added in a longer timeout to my custom mongod.service file. Trying it now

[22:05:33] <StephenLynx> are you using centOS 7?

[22:05:39] <StephenLynx> I can run mongo on it just fine

[22:05:40] <NotBobDole> StephenLynx: Yes.

[22:05:47] <StephenLynx> something with your setup.

[22:05:50] <NotBobDole> Using systemctl without daemon installed?

[22:06:01] <StephenLynx> where did you installed it from?

[22:06:15] <NotBobDole> https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.0/x86_64/

[22:06:58] <NotBobDole> Then yum install mongodb-org and mongodb-org-server

[22:06:58] <StephenLynx> yup, it matches.

[22:07:17] <StephenLynx> just org would be fine, org-server is included in org, afaik

[22:07:31] <NotBobDole> I think so, too.

[22:07:33] <StephenLynx> yes, it installs with a sysvinit file.

[22:07:42] <StephenLynx> that you can use systemctl to interact with.

[22:08:04] <NotBobDole> that file requires libdaemon, which we are trying to avoid using.

[22:08:15] <StephenLynx> no idea what that means.

[22:08:49] <NotBobDole> If you install mongodb on a blank centos 7 machine, you can't use mongodb.

[22:08:56] <StephenLynx> eh?

[22:08:58] <NotBobDole> Without also installing libdaemon.

[22:09:01] <StephenLynx> ah

[22:09:06] <StephenLynx> whats wrong with that?

[22:09:09] <NotBobDole> The mongod.service file that is provided requires that to run.

[22:09:21] <NotBobDole> mongod will not be manged directly by systemctl

[22:09:25] <NotBobDole> and make log management a pain.

[22:09:30] <StephenLynx> welp

[22:09:33] <NotBobDole> :D

[22:10:06] <NotBobDole> Yeah. I can't believe that mongodb sitll isn't writing a proper systemd file.

[22:10:40] <NotBobDole> https://www.google.com/search?q=mongod.service+file+centos+7&ie=utf-8&oe=utf-8

[22:10:41] <StephenLynx> yeah. having an actual systemd init file would be good.

[22:10:55] <NotBobDole> I've got my custom one which I ripped off some post online

[22:11:21] <NotBobDole> seems to work, but I had that timeout issue. I'm standing up the enviornment again, it'l be a few more minutes till I find out if it worked.

[23:39:57] <Bioblaze> Can you store a Array of Objects inside of another Array of Objects?

[23:40:03] <Bioblaze> Within Monodb?

[23:40:24] <Bioblaze> I have a Field called "Data" it has 5 Fields in it, the last Field is called "Meta-Data"

[23:40:35] <Bioblaze> Can i have that as a Array as well?

[23:42:50] <StephenLynx> you can.

[23:42:55] <StephenLynx> but you should ask yourself if you should.

[23:47:27] <Bioblaze> Well its to store the social data

[23:47:30] <Bioblaze> links to peoples websites

[23:47:32] <Bioblaze> is that a bad thing?

[23:47:33] <Bioblaze> o.o

[23:48:34] <StephenLynx> after a certain degree of complexity, yes.

[23:48:43] <StephenLynx> it is a very bad idea to embed data.

[23:48:44] <Bioblaze> kk then I wont do it.

[23:48:46] <Bioblaze> thank you

[23:48:46] <Bioblaze> :)

Log file Viewer

Help | Karma | Search:

#mongodb logs for Friday the 30th of October, 2015