Log file Viewer

[18:23:37] <aclark> j00bar: redis q!

[18:23:48] <j00bar> yo

[18:24:50] <aclark> j00bar: i've got some code in pythonpackages.com that does some going through of lists then gets the score of the things in a list from a sorted set, both redis data types i mean

[18:25:02] <aclark> the first time it runs it takes like 5 years

[18:25:16] <aclark> then with no caching, the 2nd time it takes only seconds

[18:25:39] <aclark> so i'm trying to figure out how to optimize and wondering if redis is "helping me" somehow

[18:26:10] <aclark> e.g. http://pythonpackages.com/users

[18:26:15] <aclark> if i were to restart the site now

[18:26:24] <aclark> that page would take > 30 seconds to load

[18:26:40] <aclark> so, i know i can cache the results

[18:26:50] <aclark> but i'm more curious about why it seems redis is caching for me

[18:27:02] <aclark> (and if no one clicks on the page for a while the same thing happens)

[18:27:22] <j00bar> is your redis data set size smaller than available memory?

[18:28:07] <aclark> either that, or maybe i'm doing something in my code that i'm forgetting or missing… but i don't think so. i would say my redis data sets are relatively small, so yes

[18:28:45] <aclark> keys * takes only a few seconds to run

[18:29:04] <j00bar> do you want to share code?

[18:29:21] <aclark> j00bar: sure let me paste something

[18:31:24] <aclark> j00bar: this is the "gist" of it: https://gist.github.com/1838016 (get it)

[18:32:38] <aclark> kind of nasty, but organic… meaning i have refactored it down to that but that still may not be the best way

[18:34:01] <j00bar> aclark: what's the point of the list - for ordering?

[18:34:54] <aclark> so if someone searches for Plone X times the list has ['Plone', 'Plone', 'Plone,… and for each Plone i look up how many times it's been "featured"

[18:35:08] <aclark> j00bar: right, to present the results the way you see on the front page and /users

[18:35:29] <aclark> i return a dict so there are no dups

[18:35:45] <j00bar> er...

[18:37:32] <aclark> actually wait let me explain more

[18:37:58] <aclark> here is get_stats: https://gist.github.com/1838046

[18:38:15] <aclark> get_stats nested does a call to get_stats for each item

[18:38:44] <aclark> ProbablyNotTheBestApproach™

[18:40:27] <j00bar> agreed. sub optimal.

[18:40:29] <aclark> i know

[18:40:36] <j00bar> this is where you want sort.

[18:40:39] <aclark> this feels like it should be a redis dict

[18:40:40] <j00bar> fuck your scored set

[18:40:44] <aclark> hah

[18:41:28] <aclark> i need to rewrite this shit when i move to heroku…

[18:41:30] <j00bar> store integers in, say, feature_count::Plone

[18:41:46] <kennethreitz> aclark: my irc client just pinged :)

[18:41:56] <j00bar> then say SORT list_key BY nosort GET # GET feature_count::*

[18:42:09] <aclark> kennethreitz: heroku!

[18:42:14] <kennethreitz> lol

[18:42:22] <aclark> j00bar: oh yeah that's good stuff, thanks

[18:42:29] <j00bar> i feel like if i'm not asian, i should call it helloku

[18:42:36] <aclark> j00bar: i haven't tapped that stuff in redis yet

[18:42:44] <kennethreitz> it's heroku + haiku

[18:42:46] <kennethreitz> damnit

[18:42:49] <kennethreitz> hero + haiku

[18:43:19] <aclark> it's salesforce + ku = salesku :-)

[18:45:18] <aclark> kennethreitz: if i want to move my redis data to heroku i probably need to script it yes?

[18:45:24] <aclark> (vs. moving dump.db)

[18:45:33] <kennethreitz> no idea :)

[18:45:42] <aclark> i need to "migrate" fields anyway

[18:45:44] <kennethreitz> i'm not familar with the redis to go offering

[18:45:50] <kennethreitz> i mean i use it

[18:45:50] <aclark> kennethreitz: k

[18:45:53] <kennethreitz> you can connect to it locally

[19:57:02] <hazmat> hmm.. python and openstack meetup on the same night

[20:02:54] <MattBowen> that is hard

[20:10:36] <benji> I wish I wasn't just an honorary member of dcpython and could actually get to meetups without having to drive for an hour and a half.

[20:49:40] <aclark> yikes… move to DC? :-)

[21:37:28] <hazmat> train ride

[21:37:36] <hazmat> and local pickup

[21:37:48] <hazmat> benji, where do you live?

[21:38:22] <benji> hazmat: outside of Fredericksburg, VA

[21:39:45] <hazmat> yeah.. that's pretty time consuming for a trip, but with a hotspot the train isn't so bad.

[21:40:03] <aclark> hazmat: thinking out loud, why is openstack moving at a snail's pace? jacobian wrote python-cloudservers which i use then forked it to openstack and it was never heard from again. meanwhile the rackspace org has some python cloud dns lib that recommends "look at the tests" to see how it works… meanwhile, i just want to do everything rackspace VM related from the command line (i.e. including dns)

[21:56:52] <hazmat> aclark, openstack is moving pretty fast

[21:57:14] <hazmat> aclark, it doesn't appear your asking about openstack

[21:57:16] <hazmat> but about rackspace

[21:58:44] <hazmat> aclark, there is python-novaclient but i dont know that rackspace has actually deployed openstack yet

[21:58:54] <hazmat> hp public cloud is openstack as is ATT's

[22:34:41] <aclark> hazmat: the only reason it's an openstack q is because of python-cloudservers renaming itself to openstack.compute: https://github.com/jacobian/python-cloudservers

[22:35:10] <aclark> hazmat: but yeah i don't really care how it works