Log file Viewer
#pypa-dev logs for Saturday the 2nd of July, 2016
[18:29:41] <nlh> one thing I wanted to ask - now we have it hosted at pypi.io, can we please update the link at the top of the github page
[18:30:14] <nlh> I think it would be motivating for new contributors to see the new design, without having to find the link via the docs :)
[18:34:59] <JosephBywater> I'm trying to setup 'warehouse' however, when I run 'make initdb' I get an error https://dpaste.de/arZT I have installed xz-utils from http://tukaani.org/xz/
[18:37:03] <JosephBywater> Errrm, it says it's not installed. Is the link I provided where I should download it from? I ran the configure file.
[18:39:49] <dstufft> JosephBywater: if you loaded any pages on the site in your browser Warehouse keeps a persistent pool of connections
[18:42:34] <dstufft> JosephBywater: it shouldn't take that long typically-- when it detects you need to rebuild it has to do a bunch of work, but normally it can just reuse the re-built containers and just start them back up
[18:43:09] <dstufft> but the first time you run it and when it thinks it needs to rebuild it'll take longer
[18:43:48] <dstufft> (that's basically if you update one of Dockerfile package.json requirements/main.txt requirements/deploy.txt)
[18:48:01] <JosephBywater> dstufft: Well, the website loads, but when I click browse projects or do a search I get errors
[18:48:34] <JosephBywater> `elasticsearch.exceptions.NotFoundError: TransportError(404, 'index_not_found_exception', 'no such index')`
[18:55:38] <JosephBywater> dstufft: Does the command make initdb 'finish' or is it meant to do https://dpaste.de/ZQ7J this
[18:56:17] <JosephBywater> Oh, it works... turns out it takes a few minutes. Last time there were errors, this time there were not.
[18:58:17] <dstufft> And not really, if you ``make purge`` you'll delete the data and you'll need to run it again then, but rebuilding containers doesn't lose data so unless you do something to delete your data (like delete your docker-machine or something) you shouldn't need to run it again generally
[18:59:12] <dstufft> sometimes we'll do a data migration that won't be represented in the database migrations and will require a new initdb
[19:08:52] <dstufft> JosephBywater: https://github.com/pypa/warehouse/issues?q=is%3Aopen+is%3Aissue+label%3A%22Great+for+new+contributors%22
[19:24:39] <JosephBywater> dstufft: Do I need to make build and make serve after making changes or just make serve?
[19:34:21] <controversial> So why do we actually need Docker? It seems like I have to get into complicated VM stuff...
[19:43:24] <dstufft> controversial: We use docker so folks don't have to worry about setting up PostgreSQL, Elasticsearch, Camo, Redis, RabbitMQ, Celery, Gunicorn, and a process for keeping static files up to date
[19:46:41] <dstufft> controversial: yea that's harder :( I don't have a great answer for that-- as far as an alternative, you can just run those things yourself of course, but part of why we have Docker is because running all those things is a _pain_. You might be able to run docker remotely but I'm not sure how well that works with mounting a local FS on a remote host
[19:47:18] <apollo13> dstufft: which of those is actually a pain? I do not know camo, but the rest in general runs on my box
[19:48:24] <dstufft> apollo13: indivdually they're not, it's just getting them all downloaded and installed and running
[19:49:59] <dstufft> Yea, also version differences-- here we have pretty tight controls over the versions of each of those things, but that gets harder when you are installing from a bunch of different sources
[19:50:16] <berker> I had to install warehouse manually because docker asked me to update my kernel version :P
[19:50:24] <controversial> https://docs.docker.com/v1.8/installation/mac/ looks like it supports older hardware than the newer docker for mac
[19:50:49] <controversial> Also: are people _opposed_ to having a Slack channel? It's frankly a lot easier than IRC
[19:52:42] <controversial> but Slack will give me native notifications, it's far easier on multiple devices, you can have multiple channels in one team, you can mention users, you can see 10,000 messages of history, the list goes on
[19:53:43] <dstufft> controversial: re: Slack, I know that jaraco (who does setuptools stuff) wanted something other than IRC. I think the discussion kind of stalled around that though. IRC exists and had a bunch of inertia already and anytime you change medium's like that you end up losing people (and ideally you end up gaining people too!)
[19:54:27] <apollo13> the biggest thing for me probably is that my irc client actually does not drain the battery as fast as the slack webapp
[19:54:53] <controversial> Slack _does_ have an IRC integration that will automatically handle cross-posting of messages between platforms
[20:00:14] <nlh> generally excellent - I'm sorry I didn't really get to catch up with you at django con :(
[20:00:49] <nlh> apollo13: so given that you're hanging out on this chan, should we expect some PRs on warehouse from you? :P
[20:01:13] <apollo13> nlh: that is the plan in the longrun, currently getting familiar with the server infrastructure behind though
[20:02:11] <apollo13> well, my plan was to have dstufft work a little bit less on pypi systemadministration so he can focus on warehouse :p
[20:04:09] <nlh> JosephBywater: unless you have several commits doing the same thing, its not needed
[20:04:30] <controversial> They're hard to get rid of afaik, once you've pushed. Normally I would git reset --hard HEAD^ && git pull --rebase, but that won't work if you've already pushed the merge commit
[20:06:07] <dstufft> controversial: a squash merge won't retain all the individual commits on a PR, it'll take the entire diff, compress it down to a single commit, and allow the merger to generate a commit message for the entire PR (prefilled using the commit messages of all of the commits though)
[20:06:14] <JosephBywater> Anyway, I've got a pull request here. https://github.com/pypa/warehouse/pull/1346
[20:06:32] <dstufft> so other than the person doing the merge needing to clean up the commit message, adding random "make X work" commits won't matter
[20:06:47] <controversial> I installed Docker Toolbox on my mac, which seems to be supported on my ancient 2010 machine, but it's been on "creating SSH key" for a few minutes now...
[20:09:25] <dstufft> controversial: everyone has different opinions on the right way to merge :) nice thing about squash merges is that you don't have to worry too much about what folks are doing in their commit history, and each commit on ``master`` represents a reviewed state of the repository. You lose some history, but that is still available in the review tool so it's not too bad
[20:09:29] <apollo13> controversial: mhm, I am pretty sure you could write a broken slake client too, not sure how some stupid webirc client would be part of any serious argument
[20:13:03] <dstufft> nlh: oh yea, since I set it up to require squash merges now, when you press the merge button you'll need to review the commit title/body that Github generates to make sure it sounds reasonable
[20:20:35] <controversial> Any specific reason for Node.js v5.x ? It throws massive deprecation warnings...
[20:22:29] <controversial> dstufft: By "massive" I just meant that it takes up the whole terminal window and pauses for 10 seconds
[20:23:35] <controversial> Oh, I don't know. It seems like 4.x is stable and 6.x is the current version, so I'm guessing 5 got dropped in between
[20:24:06] <controversial> I was kind of assuming it was a known thing, maybe it only comes into being through multiple levels of dependencies, though
[20:25:51] <dstufft> controversial: if it's the first time you ran it, it has to build a bunch of stuff
[20:25:55] <controversial> dstufft: ok. Seems like there are a lot of deprecations throughout the stuff it installs
[20:27:00] <dstufft> controversial: yea, I just assumed that was node.js's ecosystem moving fast :D I'm really really not involved in node.js in any major capcity so I just kinda chuck together whatever I can make work and then try not to think about it past that :)
[20:27:57] <controversial> dstufft: What's your role at PyPA? Are you just a community contributor or an employee?
[20:29:55] <dstufft> controversial: so PyPA is not really a company, it's just sort of an informal organization, the PSF is the legal entity that "owns" PyPI but it's a community project really so they're really just there to be the person on the line when legal things need decided. As far as who I am, I am employed by Hewlet Packard Enterprise to work on making Python's packaging better, which I'm currently focusing on trying to get Warehouse ready and
[20:29:55] <dstufft> launched to replace the dying old code base. So I'm an employee of somewhere, but I also sort of am a community contributor
[20:31:49] <controversial> dstufft: How do Docker containers work as far as saving the state? I'm assuming the container will be restarted regularly between when I'm working on it, will the long build process happen every time or does it save its state somehow?
[20:32:46] <dstufft> controversial: the long build process was building the container image, it'll take that image and reuse it in the future. For most of the code changes you don't have to do anything, we mount your host OS inside of the running container and have it configured to auto restart on code changes
[20:33:05] <dstufft> controversial: if you change dependencies you'll need tod o the long build process again, or you change something that we don't mount inside of the container
[20:33:22] <dstufft> https://github.com/pypa/warehouse/blob/master/docker-compose.yml#L31-L34 is a list of the things we mount inside the container
[20:34:14] <dstufft> controversial: yea, the intial build takes a bit :/ after that ``make serve`` takes ~30s or so to start up as long as you don't change one of {Dockerfile package.json requirements/main.txt requirements/deploy.txt} that makes it think it needs to rebuild the container images
[20:34:52] <dstufft> it'll also download some images from the docker registry, those you won't have to redownload unless you delete them
[20:35:34] <dstufft> ``make serve`` is all you need to do, and you'll have everything going then, the web server, the background worker, database server, etc
[20:35:48] <dstufft> the only thing you'll need to do is once everytihng is running, you'll need to bootstrap the database data
[20:37:48] <controversial> The project i've done that's closest to this is probably https://github.com/controversial/wikipedia-map. For that, the build process is awful. Basically, if you clone it, you can tweak the front-end, but it will still be calling the API running on my server. So if you want to change the backend, you have to manually change a variable to point to localhost:5000 and then run the Flask server
[20:38:26] <controversial> That's the only thing I've done that involves both a front-end and a backend supporting it
[20:38:37] <JosephBywater> dstufft: How often does the pre-production deployment of Warehouse get rebuilt/updated from the git repository?
[20:39:42] <dstufft> controversial: yea, the one nice hting about all this docker thing is once you get docker itself setup is everything else is handled for you :)
[20:40:25] <dstufft> JosephBywater: we auto deploy every commit that lands on ``master`` automatically once the tests pass on master. Tests and build process take maybe 5-10 minutes
[20:40:39] <dstufft> we also aggressively cache stuff though, so even if it's deployed you might see cached data for awhile
[20:41:59] <dstufft> controversial: It's cached by our CDN so it makes things faster, but also it lets us serve ~350TB/month and 3 billion requests/month with 3 backend servers :)
[20:42:32] <controversial> FYI, Docker build has now resorted to spitting out massive ASCII art drawings in my terminal
[20:42:37] <JosephBywater> dstufft: So how does the server run the website? Surely it has to make build and make serve everytime?
[20:44:09] <dstufft> JosephBywater: the server itself doesn't use docker, Redis, PostgresSQL, etc all of that persist between deploys, so it only has to compile the web app portion and that only takes a few minutes, plus it's got more build caching in play (but it's more complicated to setup)
[20:48:10] <JosephBywater> nlh: Just realised that you can still get to the dependencies tab if you add /#dependencies to the end of the URL.
[20:50:49] <controversial> My `make serve` command (which has now been running for 50 minutes) has been frozen on "[20:25:33] Finished 'watch' after 25 ms" for a very long time
[20:52:55] <controversial> JosephBywater: It still shows it's actively running `make serve`, and it hasn't given me control of my terminal back. So clearly, it's not done
[20:55:00] <controversial> (I have a script that lets me upload my latest screenshot to Imgur, I just run `uploadimage screenshot | pbcopy`. It's one of the more useful things I've written :D )
[21:00:43] <controversial> JosephBywater: Connecting says that the warehouse database doesn't exist, but I can't run `make initdb` because no other tabs will see the docker container
[21:04:07] <JosephBywater> You haven't run make initdb yet, however, you cannot run it at the moment because requested a page and it would say another service is using it, so you would have to run make serve again and then make initdb before visiting localhost:80
[21:04:22] <dstufft> ``make serve`` never returns, it keeps running in the terminal you had opens o you can see logs
[21:04:38] <dstufft> I actually forgot, you odn't need to run ``make serve`` again, you can also just edit a file like warehouse/config.py
[21:05:05] <dstufft> that'll make the server restart and cause it to let go of the persistent database connection
[21:05:58] <controversial> dstufft: Edited the file, but `make initdb` still can't find the Docker container
[21:06:05] <JosephBywater> dstufft: I made a mistake but I don't know why it's happening. (You can see the closing comment tag '#}') https://github.com/pypa/warehouse/issues/1309
[21:06:18] <dstufft> controversial: so in the ``make serve`` terminal, you'll see stdout/stderr of all of the little containers, for each process we're running, then ``make initdb`` in another terminal (which _will_ return) will create the DB and such
[21:09:41] <dstufft> JosephBywater: https://github.com/pypa/warehouse/commit/b38a458dbe2767cfb74da43873e6481796c81964#diff-2300880d801924899c92e292b10886ccL179
[21:14:03] <dstufft> if so, eval "$(docker-machine env)" in a terminal should setup the correct environment variables so the docker client can connect to it
[21:16:38] <JosephBywater> Mac mini (Mid 2011); 2.3 GHz Intel Core i5; 16 GB 1600 MHz DDR3; Intel HD Graphics 3000 512 MB;
[21:18:40] <controversial> dstufft: Can I have a quick rundown of how the stuff Warehouse is using works (I think Jinja)? For most changes, will I need to change both HTML and Python, or just HTML? Does Jinja make it so that the front-end and the backend are no longer really separate?
[21:21:13] <dstufft> controversial: the new stuff might have automatically setup his profile to call docker-machine env at start
[21:25:10] <dstufft> controversial: running back and forth a bit to do grilling so forgive my bursts of comments :) but basically it's just a Pyramid web app that uses Jinja2 for the templating engine. It's not got a seperate front end/backend as far as daemon processes go, it's jsut a single app and it renders the HTML directly (after querying the DB in the view). IT's got some light JS for handling various things but it's not a super heavy JS app
[21:25:39] <dstufft> It uses Elasticsearch for it's search engine, PostgreSQL for the DB, Redis for transiet data storage
[21:27:23] <controversial> In Jinja2, if I see something that looks like a function call, is a `grep` the only way to intelligently find its definition?
[21:30:05] <dstufft> if it's like {{ functioncall() }} it'll most likely be listed here https://github.com/pypa/warehouse/blob/master/warehouse/config.py#L222-L227
[21:30:43] <dstufft> then it'll most likely be listed here https://github.com/pypa/warehouse/blob/master/warehouse/config.py#L210-L220
[21:37:46] <dstufft> controversial: there are a handful listed elsewhere, like https://github.com/pypa/warehouse/blob/master/warehouse/i18n/__init__.py#L27-L37 but they'll all either be in the first two links, or they'll be in an includeme() function somewhere (includeme being called automatically whenever you see osomething like: https://github.com/pypa/warehouse/blob/master/warehouse/config.py#L271)
[21:40:00] <controversial> When you have a 6 year old laptop with a 10 year old battery, sometimes it dies at 75% battery.
[21:52:32] <JosephBywater> dstufft: For some reason, on some pages, the Dependencies tab still shows... is this a result of the caching?
[22:40:51] <sigmavirus24> controversial: I think Jinja2 has had non-experimental python 3 support for a long time. apollo13 I suspect controversial has found a section of jinja2's docs that are wrong or an old copy of the docs
[22:41:33] <controversial> sigmavirus24: http://jinja.pocoo.org/docs/dev/intro/#experimental-python-3-support
[22:42:32] <sigmavirus24> Right. Jinja2 is on 2.8 and has had several releases since 2.7 was released in 2013.
[22:47:54] <dstufft> it may work with other implementations/versions but if it does, it's an accident :)