pmxbot IRC Log Viewer

[01:38:37] <rusty78> Quick question if anyone is here

[01:38:54] <rusty78> I am querying sPost.find({

[01:38:54] <rusty78> room: data.room

[01:38:55] <rusty78> })

[01:38:55] <rusty78> .sort("rank")

[01:38:55] <rusty78> .limit(10)

[01:38:55] <rusty78> .exec(callback);

[01:39:16] <rusty78> This is however returning the last 10 sPost by date and then sorting by rank

[01:39:23] <rusty78> is there anyway I can get the top 10 rank objects?

[03:01:01] <partycoder> rusty78 you are doing it wrong

[03:02:56] <partycoder> rusty78, that does not scale

[03:03:25] <partycoder> you can implement functionality like that, but that does not scale

[03:04:06] <partycoder> unless you want to build a functional prototype (as opposed to an actual serious application) i would refrain from doing that

[03:06:22] <rusty78> Yeah I found out

[03:06:22] <rusty78> thank you for looking at it though :p

[03:06:23] <rusty78> Why is it so bad?

[03:06:24] <rusty78> It's written in mongoose btw, should have included that before

[03:06:24] <rusty78> sorry for my novice-ness btw :/

[03:06:24] <rusty78> I limit my results to only 10 and sort by rank

[03:06:24] <rusty78> Any recommendations for alternative implementation?

[03:06:41] <partycoder> ok...

[03:07:00] <partycoder> how many users will your application have

[03:07:16] <rusty78> lets say 10,000 is what I want to aim for supporting

[03:07:30] <partycoder> concurrent or daily

[03:07:33] <partycoder> or registered users

[03:08:06] <rusty78> For the sake of argument, let's say 10,000 at once

[03:08:06] <rusty78> or if it'd be easier, 1000

[03:08:27] <partycoder> ok

[03:08:31] <partycoder> do you ever expect that number to grow

[03:08:37] <rusty78> yes

[03:08:46] <partycoder> then

[03:08:51] <partycoder> let's say, you need to grow.

[03:08:57] <rusty78> alright I'm all ears :)

[03:08:59] <partycoder> how do you scale a computing system

[03:09:11] <rusty78> Either vertically or horizontally

[03:09:21] <partycoder> ok, and you have a vertical limit

[03:09:27] <partycoder> so, let's say, horizontally

[03:09:36] <partycoder> how do you plan to scale horizontally

[03:10:00] <rusty78> Hmm not sure.. I figured my current implementation wasn't so awful as is

[03:10:14] <partycoder> your implementation is still not in the scope of this

[03:10:22] <partycoder> we are a step back still

[03:10:29] <partycoder> how do you plan to horizontally scale your application

[03:11:03] <partycoder> at the database level

[03:11:09] <partycoder> ?

[03:11:35] <partycoder> "i don't know" is a valid answer

[03:13:39] <partycoder> so

[03:13:46] <partycoder> any clue so far?

[03:15:33] <partycoder> the truth is

[03:15:39] <rusty78> Not sure

[03:15:39] <rusty78> Best guess would be to use 2+ databases for different kinds of data

[03:15:39] <rusty78> but as for optimizing mongo, I am rather novice for where to start with that :(

[03:15:40] <partycoder> you can avoid me, and avoid this question

[03:15:40] <rusty78> I don't know

[03:15:40] <rusty78> is my answer ^

[03:15:40] <rusty78> Kinda a scary question

[03:15:40] <rusty78> I figured the current syntax I had limited the db calls to only 10 objects

[03:15:41] <rusty78> and that alone saved time/resources

[03:15:41] <rusty78> I'm not exactly sure what you are pointing out but I am definitely all ears for how to improve

[03:15:41] <rusty78> Oh I had mixed up my terms

[03:15:41] <rusty78> I assume you meant sharding for horizontal?

[03:15:46] <partycoder> ok

[03:15:51] <partycoder> i had some lag it seems

[03:16:02] <partycoder> ok the thing is...

[03:16:04] <rusty78> Yeah just saw these messages

[03:16:08] <partycoder> let's say you have shards

[03:16:13] <rusty78> alrighty

[03:16:25] <partycoder> imagine how would your query execute

[03:16:32] <partycoder> if data is in different shards

[03:16:41] <rusty78> Not sure unfortunately, I have only done sharding very small-scale testing

[03:16:53] <rusty78> as for production level systems, I'm not as experienced at all

[03:17:01] <partycoder> well it's expensive

[03:17:11] <partycoder> it needs to go to multiple shards and ask them for stuff

[03:17:30] <partycoder> then, i really hope

[03:17:30] <rusty78> yea

[03:17:41] <partycoder> that the room field has an index

[03:18:37] <partycoder> the other thing is

[03:18:40] <rusty78> It does not currently (..don't hate me too much)

[03:18:46] <partycoder> document oriented databases

[03:18:56] <partycoder> are better used when you merge on write not on read

[03:19:46] <rusty78> I am constantly doing both of the above so mongodb is versatile for my needs atm

[03:19:58] <rusty78> if I need to scale with better consideration to my db, what would you recommend on read?

[03:20:16] <partycoder> there's a short book called scaling mongodb

[03:20:22] <partycoder> it's really short

[03:20:40] <partycoder> it's mostly about understanding sharding

[03:20:48] <partycoder> if you know the basics you can probably skip it

[03:20:52] <partycoder> but you can also

[03:20:59] <partycoder> try to obtain execution plans for your queries

[03:21:01] <rusty78> http://www.amazon.com/Scaling-MongoDB-Kristina-Chodorow/dp/1449303218

[03:21:10] <partycoder> load test on a populated database and see what happens

[03:21:14] <rusty78> is that it?

[03:21:21] <partycoder> (load tests on empty databases do not count)

[03:21:26] <partycoder> yes

[03:21:27] <dimon222> hm, is it from mongo dev?

[03:22:10] <dimon222> > is a core contributor to the MongoDB project - seems so :O

[03:22:34] <rusty78> So partycoder

[03:22:37] <rusty78> instead of how I am doing it

[03:22:45] <partycoder> just load test and see for yourself

[03:22:55] <partycoder> i might be wrong

[03:23:01] <partycoder> you should still load test your app

[03:23:05] <rusty78> What would you recommend for a better way of structuring my queries

[03:23:18] <rusty78> Yeah, I will have to test it tomorrow since I am rather about to hit the hay

[03:23:25] <partycoder> try to imagine how can you distribute load

[03:23:31] <partycoder> across different computers

[03:23:40] <partycoder> the premise of a distributed system is that a single computer can't do all the work

[03:23:53] <partycoder> think how can you distribute load evenly

[03:24:09] <partycoder> and keep the ability to keep adding computing infrastructure as you need it

[03:24:32] <partycoder> if you lose this ability you will have a huge bottleneck

[03:24:59] <rusty78> Alright I will read the book to understand more on the aspects of scaling

[03:25:04] <rusty78> thank you for recommending the resource

[03:25:10] <partycoder> it's not rocket science

[03:25:26] <partycoder> it's like, you order lots of food, and you need people to eat it

[03:25:27] <rusty78> I have a quick question though - is mongoose considered a poor implementation for scaling with mongo?

[03:25:43] <rusty78> Well the theory is easy to grasp without material, however the implementation is not :p

[03:25:44] <partycoder> mongoose runs on the app server

[03:25:51] <partycoder> the app server is usually a stateless layer

[03:26:02] <partycoder> that can be scaled horizontally

[03:26:13] <partycoder> if you have stateful services you need to be more careful about those

[03:26:36] <partycoder> mongoose itself is not slow just be mindful of the kind of queries it produces

[03:26:47] <partycoder> mongodb has a log, you can see the log and inspect what is going on.

[03:27:39] <partycoder> you can also increase the verbosity of the mongo server

[03:27:45] <partycoder> to add more details to the logs

[03:28:00] <rusty78> Noted and noted

[03:28:08] <partycoder> and again

[03:28:14] <partycoder> try to take a test database

[03:28:21] <partycoder> populate it with lots of records

[03:28:41] <partycoder> and see what happens with the queries

[03:29:26] <partycoder> while it's true that you should not optimize prematurely

[03:29:51] <partycoder> horizontal scalability is not a "feature" that you can build in the end

[03:30:10] <partycoder> it's a non functional requirement that has to be considered through the entire development lifecycle

[03:30:34] <rusty78> Wise words - I was told something about mongodb auto sharding

[03:30:42] <rusty78> for initial deployment for a novice like myself

[03:30:48] <rusty78> stay away from it or initially use it?

[03:31:06] <partycoder> well if you stay away from it it is unlikely that you will come to understand it

[03:31:22] <partycoder> just try to test your assumptions

[03:31:41] <rusty78> ok

[03:31:45] <rusty78> got it - thank you

[03:31:57] <partycoder> good luck

[03:34:21] <rusty78> :)

[05:57:31] <lxsameer> guys, i have to process 5Gb of data in a database is there anything faster than native javascript ?

[06:13:47] <rusty78> LISP

[06:13:48] <rusty78> maybe

[14:21:11] <Terabyte> does mongodb make a distinction between true&false, 1&0?

[14:21:31] <StephenLynx> yes.

[14:21:35] <StephenLynx> aah

[14:21:38] <StephenLynx> wait

[14:21:46] <StephenLynx> you talking about the bit operand?

[14:21:55] <Terabyte> so if something is inserted as true, i can't search for it as '1'

[14:22:01] <Terabyte> well, the number 1

[14:22:11] <StephenLynx> ah

[14:22:13] <StephenLynx> yes.

[14:22:18] <StephenLynx> it has a boolean type.

[14:22:23] <StephenLynx> true != 1

[14:22:27] <Terabyte> mmm ok

[14:22:33] <StephenLynx> that is

[14:22:37] <StephenLynx> when you are comparing values.

[14:22:47] <Terabyte> searching for documents

[14:23:04] <StephenLynx> if you just want to indicate a condition, true work the same as

[14:23:05] <StephenLynx> 1

[14:23:06] <Terabyte> find a document where blah=1, or blah=true, if I didn't know how the documented had been inserted, would it matter?

[14:23:24] <StephenLynx> yeah, it would.

[14:23:32] <Terabyte> ok, so my search criteria has to match the insert type

[14:23:34] <StephenLynx> you would have to perform and $or

[14:23:42] <StephenLynx> or maybe an $in

[14:23:50] <Terabyte> nah, I'll just get the datatype right

[14:23:51] <StephenLynx> $in:[1,true]

[14:23:51] <Terabyte> :)

[14:24:01] <StephenLynx> yeah, that would be the optimal :v

[14:59:38] <triplefoxxy> I use Meteor.js (and therefore MongoDB) as part of a stage show. My MongoDB databases only exist for 5 minutes and have < 10MB for data. I need the data to survive a crash (so no in-memory) and I need low latency. What storage engine do I use?

[15:14:26] <StephenLynx> if your databases are vanishing on crashes, it has nothing to do with mongo,afaik.

[15:15:00] <triplefoxxy> StephenLynx: They're not vanishing. Just saying I cannot use in-memory.

[15:15:09] <StephenLynx> ah

[15:15:23] <StephenLynx> have you tested the default engine?

[15:15:46] <StephenLynx> wt will become default on 3.1, so you might want to test it too.

[15:16:23] <StephenLynx> by the way, if you use mongo just for a cache, redis might be what you need.

[15:16:56] <triplefoxxy> Yep, mmapv1 causes my machien to stutter as the kernel moves the large db files into cache memory. I'm just about to test with smallFiles: true.

[15:17:18] <StephenLynx> from what I heard, redis is faster than mongo IF all your data fits on RAM.

[15:17:47] <triplefoxxy> StephenLynx: I have to use MongoDB for Meteor.js.

[15:17:59] <StephenLynx> why do you need to use meteor?

[15:18:46] <triplefoxxy> Because the development benfit outweights the issue causes by MongoDB.

[15:18:51] <StephenLynx> hm, hum.

[15:19:30] <StephenLynx> it probably shoots in curves in fries eggs if its so good you are willing to use the wrong database.

[15:19:33] <StephenLynx> and fried*

[15:20:20] <triplefoxxy> Since posting my initial question, I found the smallFiles options. I looks like what I need and what Meteor.js uses for it's dev MongoDB instance. I'll report back.

[19:17:46] <ollien> Hey there. I have a collection with an array of objects. I want to be able to use a find query to just get the matching objects from those arrays. For example, say a document in the collection is {array:[{a:1,b:4},{a:1,b:4},{a:2,b:4}]}, I want to get all objects that have {a:1} in them. My attempt was to use db.collection.find({array:{$elemMatch:{a:4}}) but that just returns the entire document, rather than the specific object. Could someone point me in

[19:17:46] <ollien> the right diretion?

[19:21:42] <partycoder> well

[19:21:56] <partycoder> when you use that find query, it will return you the documents that match your query

[19:22:09] <partycoder> then you need to use a projection to return only the part of the document you are interested in

[19:22:16] <partycoder> does that answer your question ollien

[19:22:41] <ollien> partycoder can I use a projection to return specific items within arrays?

[19:23:17] <ollien> I know I can do it with fields of the document, but I don't know how I would go about it with specifics items within an array

[19:23:59] <ollien> I don't want to return the whole array, but rather just the matching items in the array

[19:25:18] <partycoder> have you tried what i suggested?

[19:25:50] <partycoder> collection.find(query, projection)

[19:25:52] <ollien> partycoder I'm not sure how I would go about it to be quite honest. I know I could use a projection to say, only return the array from the document, but not how to just return the objects within the array

[19:25:54] <ollien> right, I know'

[19:26:01] <partycoder> where projection would be something like

[19:26:21] <partycoder> {"arrayfield.$": 1}

[19:26:31] <partycoder> try it please, it's faster than pondering if it works or not

[19:27:48] <ollien> partycoder Ah I see, didn't know about that .$ operator

[19:27:55] <ollien> where in the docs could I find some information about that?

[19:29:08] <partycoder> this is you now: https://www.youtube.com/watch?v=SSKatciX3hA

[19:29:31] <ollien> partycoder I did try it, and it worked, but I want to read more about it in the docs

[19:29:38] <ollien> just so I understand what I did

[19:30:07] <partycoder> just google "mongo $ positional projection operator"

[19:30:23] <ollien> thank you. I didn't know it was called a positional projection operator

[19:30:31] <partycoder> or $ operator

[19:30:41] <partycoder> might also get you there

[19:30:44] <ollien> thanks so much!

[19:30:49] <partycoder> you're welcome

[19:31:16] <partycoder> however though, i have to say, ollien

[19:31:38] <partycoder> if you do this for multiple documents in a collection

[19:31:57] <partycoder> this might not be very performant

[19:32:22] <partycoder> you might want to run .explain() in your query to see the cost of it, before you plug it into your application

[19:32:29] <ollien> partycoder is there a more performant way of doing it?

[19:32:38] <partycoder> what i'm saying is

[19:32:47] <partycoder> try to not do this for multiple documents at the same time

[19:33:04] <partycoder> like: give me all the documents that have an array field containing this value

[19:33:13] <partycoder> that will murder your database

[19:33:20] <ollien> that was the plan to do...well shit.

[19:33:48] <partycoder> so, the presence of those values

[19:33:52] <partycoder> what do they represent

[19:34:04] <ollien> partycoder what would they actually do in my database?

[19:34:24] <partycoder> i think if you are looking for those values

[19:34:40] <partycoder> is because they represent some kind of category

[19:34:46] <ollien> partycoder well, sorta

[19:34:55] <ollien> the plan for a schema was to have something like this

[19:35:23] <partycoder> well the problem with that query is that it will do a full collection scan each time you run it

[19:35:29] <ollien> {username:sessions:[sessionKey:key,ipAddress:ipAddress,expires:expires]}

[19:35:37] <ollien> Then one would search by sessionKey and by ipAddress

[19:35:47] <ollien> to find the appropriate session

[19:36:06] <partycoder> you mean

[19:36:40] <partycoder> {_id: username, sessions: [ { sessionKey: key, ipAddress: ip, expires: expires }, ... ] ?

[19:37:10] <ollien> partycoder the username wouldn't be in the id field, but rather it's own field, but yes.

[19:37:26] <partycoder> so you would need to scan the entire collection

[19:37:33] <ollien> right

[19:37:35] <partycoder> to find the sessions for a user?

[19:37:40] <ollien> yeah

[19:37:43] <partycoder> that doesn't scale :)

[19:37:47] <ollien> that's true :)

[19:37:52] <ollien> I was thinking of it a bit naivley

[19:37:57] <partycoder> well

[19:38:07] <partycoder> is this for an actual consumer product?

[19:38:11] <ollien> nah

[19:38:13] <ollien> just a blogging platform

[19:38:17] <ollien> like a wordpress remake

[19:38:34] <partycoder> but this is for user sessions

[19:38:35] <partycoder> well

[19:38:38] <partycoder> whatever

[19:38:44] <ollien> I do now see a better way I could do it

[19:38:46] <partycoder> just try to not hit multiple records at once

[19:38:53] <partycoder> that's never a good idea

[19:38:53] <ollien> partycoder right.

[19:39:01] <ollien> I might do something like this instead

[19:39:03] <partycoder> the idea is that, you have an id, and you hit 1 record at a time

[19:39:18] <ollien> fair point

[19:39:25] <ollien> I could do something like this

[19:39:28] <partycoder> i mean, 1 "document"

[19:39:38] <partycoder> like what i just said probably

[19:39:41] <ollien> {_id:id,username:username,ip:ip,expires:expires}

[19:39:46] <ollien> would prolly make more sense

[19:39:50] <partycoder> _id: id?

[19:39:53] <ollien> err

[19:39:54] <partycoder> what is that id

[19:39:55] <ollien> _id:sessionKey

[19:40:00] <ollien> wasn't thinking

[19:40:00] <partycoder> why session key?

[19:40:17] <ollien> That needs to be distributed to the user as a cookie, so storing it as the id makes sense, no?

[19:40:19] <ollien> it would be a uuid

[19:40:26] <StephenLynx> ollien

[19:40:29] <ollien> yo

[19:40:32] <StephenLynx> I just store the current session at the user document.

[19:40:44] <partycoder> that was what i was suggesting

[19:41:02] <ollien> StephenLynx right, but the user would login with just an ip and a session token

[19:41:11] <StephenLynx> yes.

[19:41:15] <ollien> no username would be submitted to the server

[19:41:18] <StephenLynx> yes.

[19:41:21] <partycoder> that's entirely up to you

[19:41:24] <ollien> Wouldn't that require going througth every elemtn?

[19:41:27] <StephenLynx> no.

[19:41:37] <ollien> StephenLynx I'm curious how you would do it then

[19:41:39] <StephenLynx> but

[19:41:43] <ollien> A user can, theoretically, have multiple sessions

[19:41:46] <StephenLynx> aah

[19:41:50] <partycoder> yes.

[19:41:58] <StephenLynx> I wouldn't allow it.

[19:42:02] <partycoder> you need to think

[19:42:02] <StephenLynx> or

[19:42:02] <ollien> I would

[19:42:04] <ollien> it makes sense

[19:42:07] <StephenLynx> don't use the ip.

[19:42:13] <StephenLynx> just send the login and token.

[19:42:14] <ollien> User has 2 computers, each computer gets its own session token

[19:42:19] <StephenLynx> and renew token on special occasions.

[19:42:28] <partycoder> not a problem with the schema i proposed

[19:42:36] <partycoder> {_id: username, sessions: [ { sessionKey: key, ipAddress: ip, expires: expires }, ... ]

[19:42:39] <StephenLynx> why does it need a session token for each computer?

[19:42:52] <partycoder> however array fields usually suck

[19:42:52] <ollien> StephenLynx so the user can revoke it

[19:43:00] <partycoder> you can use a subdocument instead of an array

[19:43:03] <StephenLynx> can't he just logout?

[19:43:06] <StephenLynx> and delete the cookie?

[19:43:13] <partycoder> stop ignoring

[19:43:16] <ollien> partycoder right. That's essentially what I'm using now, but you said that would have the problem of searching through every document, no?

[19:43:37] <ollien> StephenLynx what if an attacker manages to login, how would he remove it then?

[19:43:43] <partycoder> {_id: username: sessions: { key1: {...}, key2: {...}, key3: {...} } }

[19:43:54] <StephenLynx> use changes his password, token gets changed too and logs out every other computer.

[19:43:58] <StephenLynx> that is what I do.

[19:44:00] <ollien> huh didn't think of that

[19:44:07] <ollien> the subdocuments that is partycoder

[19:44:09] <StephenLynx> I also change token when the password is reset using e-mail

[19:44:31] <partycoder> daily active users, not registered users

[19:44:41] <ollien> partycoder is there a nice way to use .find on subdocuments?

[19:45:53] <partycoder> .find({ _id: id }, {sessions.key: 1})

[19:46:11] <partycoder> no arrays

[19:46:20] <StephenLynx> 'sessions.key'

[19:46:29] <partycoder> right

[19:46:37] <partycoder> arrays in my experience are a problem

[19:46:50] <partycoder> if you are using array positions to add/remove values

[19:47:00] <partycoder> and you try to do it concurrently

[19:47:18] <partycoder> you usually end doing unintended things

[19:47:21] <StephenLynx> not always. but if they can expand dynamically without bounds, they are often a problem.

[19:47:39] <StephenLynx> or if you have to constantly write to them.

[19:47:49] <partycoder> let's say you say: remove the 1st element, and there's another request asking the same

[19:48:01] <partycoder> that "1st element" might not be the element you wanted to remove

[19:48:08] <StephenLynx> but honestly, that object with dynamic keys you proposed is no better.

[19:48:11] <partycoder> it's better to say

[19:48:23] <partycoder> "delete subdocument 'bob'"

[19:48:50] <partycoder> no better?

[19:48:56] <StephenLynx> yeah.

[19:49:12] <StephenLynx> usually they cause even more problems than arrays.

[19:49:15] <preaction> at least the subdocument would be idempotent

[19:49:24] <ollien> partycoder maybe I'm missing something, but where in this schema do you define session.key?

[19:49:34] <preaction> "delete array index 0" will do something different the second time you ask it

[19:49:34] <partycoder> "they cause more problem than arrays". that's an opinion not a fact

[19:49:35] <StephenLynx> its the key of the sub documents.

[19:49:47] <partycoder> correct preaction

[19:49:53] <preaction> which is what partycoder exactly said

[19:50:11] <partycoder> as a person that had to solve multiple production issues related to the incorrect usage of mongo arrays

[19:50:22] <partycoder> i can say StephenLynx that i don't share your opinion

[19:50:27] <StephenLynx> I am not saying they are good.

[19:50:37] <StephenLynx> I avoid them on a number of cases.

[19:50:44] <StephenLynx> but having dynamic schemas is even worse.

[19:50:53] <partycoder> "worse"

[19:51:12] <partycoder> that's an opinion not a fact

[19:51:16] <partycoder> can you provide a fact?

[19:51:22] <partycoder> you are being a bit dogmatic here

[19:51:55] <partycoder> i am talking from my own experience of using both on consumer products of high traffic volume

[19:52:21] <partycoder> and i'm giving you factual evidence

[19:52:27] <StephenLynx> http://stackoverflow.com/questions/17877619/json-schema-with-dynamic-key-field-in-mongodb

[19:52:34] <partycoder> you are just saying "i don't like this"

[19:52:38] <StephenLynx> I was not defending arrays.

[19:53:11] <StephenLynx> "At the moment (and probably in the future, too) it's not possible to query MongoDB collections with wildcards in fieldnames (thanks to @gWiz). "

[19:53:32] <partycoder> well, there you have an argument

[19:53:39] <partycoder> schema validation is more difficult

[19:53:48] <partycoder> correct, it is more difficult but not impossible

[19:53:55] <StephenLynx> dynamic schemas really, really narrow your use case.

[19:53:59] <partycoder> at least you don't run into concurrency problems

[19:54:03] <StephenLynx> again

[19:54:07] <StephenLynx> I was not defending arrays.

[19:54:15] <partycoder> another way you can prevent that from happening

[19:54:17] <StephenLynx> I was just saying dynamic keys are even worse.

[19:54:26] <partycoder> is using a version number in your query

[19:54:41] <partycoder> find({_id: id, version: version}, {...})

[19:54:53] <partycoder> i mean, findAndModify..

[19:54:56] <StephenLynx> now you have an inconsistent database.

[19:55:17] <StephenLynx> that will require checks on runtime.

[19:55:31] <partycoder> in that case, if you are modifying an old version it will fail

[19:55:42] <partycoder> "inconsistent database"

[19:55:50] <partycoder> ok enough

[19:55:58] <partycoder> i'm done

[19:56:22] <StephenLynx> ¯\_(ツ)_/¯

[19:57:16] <ollien> partycoder did some playing around in the console and I think I got it

[19:57:17] <ollien> thanks so much!

[20:00:18] <partycoder> ollien, just...

[20:00:22] <partycoder> be careful with arrays

[20:00:27] <ollien> partycoder will do

[20:00:32] <partycoder> let's say i hit refresh

[20:00:37] <partycoder> and i expire a session

[20:00:47] <partycoder> simultaneously, by removing it from the array

[20:00:47] <StephenLynx> also

[20:00:51] <partycoder> and you remove it by position

[20:01:03] <partycoder> you will run into issues

[20:01:03] <StephenLynx> if you use independent documents for sessions, you can use documentation expiration

[20:01:15] <StephenLynx> and let mongo deal with cleaning them for you.

[20:01:43] <ollien> StephenLynx neat!

[20:02:05] <partycoder> at a cost

[20:02:18] <ollien> partycoder what's the cost?

[20:02:31] <StephenLynx> fake relations in general can be dangerous too.

[20:02:36] <StephenLynx> if you abuse them.

[20:02:52] <StephenLynx> because mongo doesn't have joins or relational integrity checks.

[20:02:53] <ollien> StephenLynx what do you mean fake relations?

[20:03:15] <StephenLynx> if you use separate documents on their own collection, you will have to relate a field of the user in the session.

[20:03:36] <StephenLynx> like its _id or login.

[20:03:47] <StephenLynx> plus it will use more space, if I am not mistaken.

[20:03:54] <ollien> StephenLynx got it

[20:03:55] <ollien> thanks

[20:04:02] <StephenLynx> but this case seems to be sane to use fake relations.

[20:04:24] <partycoder> it depends

[20:04:44] <partycoder> every 1 minute the process that removes expires documents runs

[20:04:57] <StephenLynx> the process?

[20:05:01] <StephenLynx> its mongo that does it.

[20:05:12] <partycoder> i mean, the "task"

[20:05:19] <StephenLynx> he just have to remember to query for only sessions that haven't expired yet.

[20:05:23] <partycoder> it is mongo that does it yes

[20:05:31] <StephenLynx> because mongo doesn't remove them right away.

[20:05:43] <partycoder> yes

[20:05:53] <ollien> StephenLynx when does mongo remove them?

[20:05:59] <partycoder> every 1 minute!

[20:06:04] <ollien> ah!

[20:06:06] <ollien> that would make sense

[20:06:08] <StephenLynx> hm, not.

[20:06:11] <partycoder> http://docs.mongodb.org/manual/core/index-ttl/

[20:06:18] <StephenLynx> let me see

[20:07:01] <partycoder> might be configurable, but at least it's not in the docs

[20:07:05] <StephenLynx> hm.

[20:07:36] <StephenLynx> I remember performing some tests and It would take a few seconds, but I don't remember them staying up for up to a whole minute before being removed.

[20:07:37] <partycoder> well, should not be a really high cost

[20:07:58] <StephenLynx> but I wasn't very thorough.

[20:08:02] <partycoder> but that would lock the collection

[20:08:05] <partycoder> until it's finished

[20:08:29] <partycoder> if the collection is really large (unlikely for a blogging platform) should not be a problem

[20:08:36] <partycoder> is not really large*

[20:08:52] <partycoder> and, well... i messed that statement up

[20:09:27] <partycoder> if the collection is not really large it should not be a problem. in blogs most users are unregistered users so you don't need sessions for them

[20:10:03] <partycoder> you might need sessions only for access control purposes

[20:10:27] <partycoder> and if that is the case you can make it as inefficient as you want

[20:10:50] <ollien> partycoder what if, theoretically, it was large

[20:10:53] <ollien> how would TTL be innefficient?

[20:10:58] <ollien> because of the amount of time it takes to expire?

[20:11:05] <partycoder> let's say it runs every 60 seconds

[20:11:15] <partycoder> and 1000 documents have to be removed

[20:11:33] <partycoder> then 1000 documents will be removed and during that time the db will lock

[20:11:41] <StephenLynx> the collection will lock.

[20:12:14] <ollien> partycoder so would it be better to implement my own TTL process?

[20:12:16] <partycoder> that depends on your mongo version

[20:12:20] <StephenLynx> hm

[20:12:41] <StephenLynx> I would decide that after testing TTL performance.

[20:12:53] <partycoder> and... you might want to take a look at the mongo lock behavior

[20:12:57] <StephenLynx> that.

[20:12:59] <ollien> StephenLynx I didn't test performance but I did a quick test

[20:13:10] <StephenLynx> reinventing the wheel is not trivial.

[20:13:10] <ollien> it does in fact take about 60 seconds for the TTL to actually occur

[20:13:14] <partycoder> it's not collection based it's db based

[20:13:32] <StephenLynx> yeah, but the issue is not that, is the time it will lock during the removal of expired documents.

[20:13:43] <ollien> StephenLynx right, of course

[20:13:56] <partycoder> so, in the end, it mostly depends

[20:14:07] <partycoder> 1. size of the db, 2. amount of documents expiring at the same time

[20:14:48] <partycoder> and you can monitor the lock status of your db

[20:15:08] <StephenLynx> plus I only found people with issues with it in very old versions.

[20:15:24] <StephenLynx> I wouldn't really avoid using it because of these concerns.

[20:15:44] <partycoder> it's only a problem if you have huge collections

[20:15:58] <ollien> partycoder if one did have a huge collection, what would you do for TTL?

[20:16:14] <partycoder> do it on demand

[20:16:22] <partycoder> as users connect

[20:16:30] <ollien> partycoder as in when a user connects with an expired session?

[20:16:44] <partycoder> your app can establish if it's expired

[20:16:50] <partycoder> add a timestamp field or something

[20:16:50] <StephenLynx> run your own timer to remove the session as soon as it expires would be an option too.

[20:16:58] <ollien> right of course

[20:17:01] <ollien> awesome

[20:17:02] <ollien> Thanks you guys!

[20:17:10] <partycoder> however you might run into another problem then

[20:17:15] <partycoder> collection size

[20:17:28] <partycoder> you might end with lots of expired data in your db

[20:18:01] <partycoder> well

[20:18:06] <partycoder> if you ever get to that problem

[20:18:19] <partycoder> you will have enough money to hire someone to do it for you.

[20:18:33] <partycoder> until then you might be fine just using ttl

[20:21:53] <StephenLynx> yeah. reinventing the wheel is for when you both know what and why you are doing it.

[20:29:55] <bonhoeffer> hey -- what are positional arguments?

[20:30:10] <bonhoeffer> for example, this doesn't work: mongodump -db meteor --host 127.0.0.1:3001

[20:31:16] <partycoder> how does it fail bonhoeffer

[20:31:30] <partycoder> what's the error you are getting

[20:31:30] <bonhoeffer> positional arguments not allowed

[20:31:42] <partycoder> isn't it --db?

[20:31:44] <bonhoeffer> positional argument is a name that is not followed by an equal sign (=) and default value

[20:32:12] <bonhoeffer> ah -- yes -- or -d

[20:32:41] <partycoder> so

[20:32:44] <partycoder> have you tried that

[20:36:21] <partycoder> bonhoeffer...

[20:38:51] <bonhoeffer> partycoder: sorry -- works great

[20:38:59] <bonhoeffer> thanks -- was coding in the zone in my uber

[20:39:12] <bonhoeffer> in a terrible traffic jam

[20:40:53] <bonhoeffer> btw, how can i find out what port my mongodb is using? (say, via netstat)

[20:41:39] <bonhoeffer> netstat -ln | grep -E '27017|28017' <--works well enough

[20:42:07] <partycoder> what os are you using

[20:42:17] <partycoder> netstat doesn't behave the same across multiple os

[20:43:02] <partycoder> at least linux you can do -tlpn

[20:43:10] <partycoder> -p gives you the process name and you can filter by mongo

[20:43:35] <partycoder> if mongo is configured to use another port it won't show up

[20:44:16] <bonhoeffer> ubuntu

[20:44:28] <partycoder> try then

[20:44:36] <partycoder> netstat -tlpn | grep mongod

[20:44:37] <bonhoeffer> -tlpn <-- applies to mongo binary?

[20:44:39] <partycoder> or something like that

[20:44:40] <bonhoeffer> ah

[20:45:22] <partycoder> the mongo log will also tell you

[20:45:28] <partycoder> what is going on

[20:46:21] <bonhoeffer> ahh . . . good idea

[21:04:38] <bonhoeffer> hey -- i'm trying to use https://github.com/sheharyarn/mongo-sync to sync my db -- but the db is not accessible through port 80

[21:04:57] <bonhoeffer> for example, my url: '127.0.0.1' port: 27017

[21:05:28] <bonhoeffer> stands -- any ideas? what really needs to happen is for mongo sync to log on locally

[21:05:42] <bonhoeffer> or for me to use a mongo cloud server or web-enabled database

[21:08:06] <bonhoeffer> any thoughts, partycoder ?

[21:10:51] <partycoder> sync your db with what

[21:11:17] <partycoder> that's a terrible idea

[21:11:29] <bonhoeffer> i have a local db, i need it on the server

[21:11:35] <bonhoeffer> the data on the server

[21:11:49] <partycoder> why would you export development database data into production

[21:12:15] <bonhoeffer> maybe connect via ssh tunnel

[21:12:20] <partycoder> wait

[21:12:23] <partycoder> a step back

[21:12:27] <partycoder> what are you trying to achieve

[21:12:40] <partycoder> i've used mongo for years and i've never had to use something like this

[21:12:59] <partycoder> you need to import configuration data into your production server?

[21:13:12] <bonhoeffer> ?

[21:13:20] <partycoder> why are you doing this "sync"

[21:13:27] <partycoder> what data are you syncing

[21:13:29] <partycoder> and why

[21:13:33] <StephenLynx> maybe he wants what replication is for?

[21:13:47] <partycoder> why would you ever want to replicate a development database into production

[21:14:14] <partycoder> also replication is 2 ways

[21:14:18] <bonhoeffer> it is more the data i need

[21:14:31] <bonhoeffer> i need data locally to be on the server

[21:14:36] <partycoder> why

[21:14:46] <bonhoeffer> umm . . . why not?

[21:14:46] <partycoder> what data is this

[21:14:52] <bonhoeffer> just a series of records

[21:14:57] <bonhoeffer> 2 mb

[21:14:59] <partycoder> for what

[21:15:13] <partycoder> is it configuration data or what

[21:15:20] <bonhoeffer> no

[21:15:27] <bonhoeffer> user inputted data

[21:15:31] <partycoder> is it data that never changes?

[21:15:33] <bonhoeffer> say a list of books

[21:15:42] <bonhoeffer> no -- just a one time event

[21:15:56] <bonhoeffer> this is a journal -- it is the historical journal entries

[21:16:13] <bonhoeffer> i've built the app -- deployed it -- now i want to populate it with my historical data

[21:16:21] <bonhoeffer> one time -- then the data are on the server

[21:16:33] <partycoder> then use mongodump/mongorestore once

[21:16:38] <partycoder> and then continue to use your deployed app

[21:16:40] <partycoder> no need for syncing

[21:17:43] <partycoder> then if you can't modify your data with your deployed app

[21:17:55] <partycoder> then you might be good not using a database at all

[21:17:58] <partycoder> put it in a file

[21:22:09] <StephenLynx> that.

[21:22:15] <StephenLynx> simple tools for simple tasks.

Log file Viewer

Help | Karma | Search:

#mongodb logs for Sunday the 28th of June, 2015