Log file Viewer
#pypa-dev logs for Thursday the 17th of April, 2014
[17:08:40] <agronholm> I'll get that rebase conflict resolved as soon as my brain starts working again (which will require some sleep)
[18:45:29] <yusuket> dstufft: I’m making the changes to https://github.com/pypa/warehouse/pull/299 that you suggested, and I had a couple questions
[18:45:53] <yusuket> 1: I’m wondering what kind of normalization method you were looking for to normalize package naming
[18:46:42] <yusuket> I copied the implementation directly from pypi, so if we change the behaviour in warehouse it’s going to be act differently (just something to be aware of)
[18:47:13] <yusuket> and if we do change it, it doesn’t look like there’s much more required than the method “normalize” you wrote
[18:47:24] <yusuket> Looking at pep426 I found this: http://legacy.python.org/dev/peps/pep-0426/#name
[18:48:14] <yusuket> which makes me think it a package name validator would be the only additional piece of work we need
[18:48:47] <yusuket> mainly because right now, pypi just normalizes out characters that are invalid in the new spec
[18:49:12] <yusuket> thoughts? Also my second question had to do with the merging of noramlize and normalize_package_name
[18:49:29] <yusuket> I noticed FastlyFormatter normalizes all keys, just not those specific to package names
[18:49:47] <yusuket> so I’m not sure if it should share the same normalizer as the package name normalizer
[19:23:47] <yusuket> ok, as long as it’s ok to use the same scheme for package names and also for fastly keys
[19:25:33] <dstufft> The Fastly keys are not the cache keys, but they are just additional metadata so that you can purge multiple things at once
[19:25:58] <dstufft> e.g. all pages that have content dependent on the "warehouse" database entires, are tagged with the key "project/warehouse"
[19:27:07] <dstufft> But that's what it does yea, basically just an one to many mapping that lets you purge a whole bunch of things
[19:27:59] <dstufft> (like for Warehouse, /simple/foo/ is case insensitive on the foo part, so if you update foo, you also want to purge /simple/Foo/ and /simple/FOO/ and /simple/FoO/ etc
[19:28:49] <yusuket> although couldn’t you also have those three routes map to the same cache object? or is that tricky to do?
[19:30:31] <dstufft> So Varnish by default uses the url of the request, and eventually yes we'll use some custom varnish tricks so that it recognizes that the <foo> part is case insensitive so that it only caches one object for all the various spellings
[19:31:40] <dstufft> [15:30:10] <dstufft> So Varnish by default uses the url of the request, and eventually yes we'll use some custom varnish tricks so that it recognizes that the <foo> part is case insensitive so that it only caches one object for all the various spellings
[19:32:28] <yusuket> so here’s my other question: what did you want to specifically see different about the normalization method?
[19:32:54] <yusuket> I read thread the PEP and it looks like the normalization already does what it needs to (lowercase, ‘_’ -> ‘-‘)
[19:33:20] <yusuket> the only thing that I see we don’t have now is validation that the name doesn’t have illegal characters
[19:33:22] <dstufft> Hmm, the PEP also talks about confusable characters, and I thought I implemented that on PyPI already
[19:33:58] <dstufft> Index servers MAY consider "confusable" characters (as defined by the Unicode Consortium in TR39: Unicode Security Mechanisms) to be equivalent.
[19:35:17] <dstufft> the one trick for confusables, it needs to take into account that we're case insensitive too
[19:36:34] <dstufft> I did implement this though -> ALTER TABLE packages ADD CONSTRAINT packages_valid_name CHECK (name ~* '^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$'::text);
[19:37:36] <yusuket> do you think it’s worth it to re-implement that on the server side? My reasoning is it makes it clearer for someone reading the code to understand what the whole spec compliance looks like
[19:38:47] <dstufft> I generally look at database constraints as "make sure that even if the app is buggy we don' get bad data"
[19:39:14] <yusuket> Ok, i’ll make sure the value errors actually correlate to statements in the PEP
[19:39:53] <yusuket> how about something like: “Distribution names MUST start and end with an ASCII letter or digit. (see PEP-426)”
[19:43:07] <dstufft> It's been mostly me up till now :] Richard has done some but he's real busy with his job, and my job gives me ~1/2 my time to work on packaging stuff
[19:43:22] <dstufft> If there's anything that confuses you or anything like that, make sure to open up issues to document stuff
[19:44:29] <yusuket> I actually haven’t encountered anything difficult to understand yet, but I’ve only touched the db code so far
[19:44:53] <yusuket> I’d like to take a crack at implementing the legacy pypi register/package submission apis after this though
[19:46:30] <dstufft> I'm changing the DB stuff around a little bit, the actual classes don't much, but instead of app.db.packaging.get_project(name) it'll be app.query(Packaging).get_project(name)
[19:48:30] <yusuket_> I see, less magic since you’re expicitly passing in the class instead of instantiating it beforehand or instantiating via reflection somewhere?
[19:50:53] <dstufft> basically app is an instance of Warehouse, and it just has a list of classes that it instiates and sticks on app.db in it's __init__
[19:51:26] <dstufft> but you just kind of have to know it does that, and you have to kind of know that app.db.packaging is an instance of warehouse.packaging.db.Database
[19:52:17] <yusuket> yeah, and you want to make it clear that query method are obtained via a class right? and make it clear as to where you can find that class?
[19:52:48] <yusuket> because right now you just know from heuristics that packaging = packaging/db.py
[19:54:44] <dstufft> https://gist.github.com/dstufft/8562f47203349856f480 vs https://github.com/pypa/warehouse/blob/master/tests/accounts/test_views.py#L45-L85
[20:05:43] <ErikRose> jezdez: dstufft: Doing some reqs.txt 2.0 sketches. Findings so far: YAML is the only off-the-shelf format I like, and it's super complicated. I don't think I'll end up recommending it, but research continues.
[20:06:52] <ErikRose> My current germ of a sketch looks like "somepackage>=3.0" for simple cases but then has indented key-value pairs underneath it for more complicated ones.
[20:06:59] <dstufft> I was thinking of a Python file for requirements 2.0 which "compiles" down to a json lock file. which made some stuff real nice, like implementing groups and such
[20:07:30] <ErikRose> I read all your comments on https://github.com/pypa/pip/issues/1175, if that's what you're repeating.
[20:10:26] <ErikRose> You were talking about a separate reqsfile (supporting > and <) and lockfile (supporting only ==) in issue 1175. I would very much like one language that handles both, for mental and codebase simplicity.
[20:12:32] <ErikRose> Also, what do you mean by "groups"? Like "stuff for prod" and "stuff for dev"?
[20:13:27] <ErikRose> You must also have some ideas, then, about additional pip commands and such to create such files. I would be eager to hear them. :-)
[20:16:10] <ErikRose> dstufft: I don't see anything in that gist that appies; is that just background?
[20:16:35] <Ivo> ErikRose: its still pretty powerful, but you don't need a freaking huge c-compiled library to parse it ({lib,py}yaml)
[20:18:38] <dstufft> ErikRose: so the thought about groups, is they are thing likes "production" and "development", you can opt to install them (or not) but they always affective the dependency resolution
[20:19:23] <dstufft> it's on my todo list, and there are a few libraries that implement something, though I'm not sure I like those libraries or not
[20:20:17] <Ivo> I've narrowed down that the best implementation would be a pure python boolean sat solver
[20:20:41] <dstufft> here's another thing with some comments on it https://gist.github.com/dstufft/e61c97ee30192e575140
[20:21:54] <dstufft> Ivo: they were my ideas 2 years ago before I was a pip developer when I was going to make my own installer
[20:22:06] <Ivo> there is actually one complete example of it, 0install implemented one in pure python
[20:23:25] <ErikRose> Well, jacobian's format doesn't support nesting >1 level: https://gist.github.com/dstufft/e61c97ee30192e575140/#comment-266748
[20:23:46] <ErikRose> That's why I ruled out ini. Also, it has a Windows flavor, but that's a secondary repulsiveness.
[20:25:00] <ErikRose> dstufft: Attributes on things, like peep-style hashes, the -e flag, or use cases like https://github.com/pypa/pip/issues/1433
[20:25:24] <ErikRose> Ivo: The motivation is to be extensible so we don't have to do reqs.txt 3.0 for awhile. :-)
[20:26:13] <Ivo> But, you just need to get the overall idea from it, it is designed for its own packaging system
[20:27:36] <ErikRose> I'd have to work through the implications of execing arbitrary Python at install time (again).
[20:28:31] <dstufft> if a lockfile doesn't exist, it does the dependency resolution using the python file, and then generates a .lock file
[20:28:35] <Ivo> IMO you should only need to execute code for building shit, not ever just specifying requirements
[20:29:11] <dstufft> you could implement it using an AST parser if you really wanted, but I don't think it's a big deal
[20:29:34] <dstufft> this isn't seomthing you're going to execute random untrusted requirements files like you do with setup.py
[20:30:27] <ErikRose> This is a somewhat deeper rabbit hole than I expected when I sat down to merge in peep-like functionality. :-)
[20:32:19] <ErikRose> Not to mention it has nothing to do with Python except that it happens to be in the stdlib
[20:32:48] <dstufft> ErikRose: quick overview of Gemfile // Gemfile.lock, you typically specify ranges in a Gemfile, like "ok this library uses semver and the current version is 1.4.2, so >=1.4.2,<1.5" whereas a Gemfile.lock says "1.4.6 exactly",
[20:33:17] <dstufft> "update my lockfile" and it'll update it to the latest version of whatever you specify in your Gemfile
[20:33:32] <dstufft> so if you are currently on 1.4.6, and 1.4.7 is released, but also 1.5, you'll only update to 1.4.7
[20:34:01] <dstufft> it makes your deploys 100% reproduceable via the lockfile, without having to manually manage the lockfile ala requirements.txt
[20:34:29] <dstufft> plus Gemfile you only specify your top level stuff, but lockfiles list the entire dependency tree
[20:35:28] <ErikRose> At the risk of replaying the whole conversation from https://github.com/pypa/pip/issues/1175 again, why not dispense with the non-pinning reqs file and favor setup.py or the upcoming declarative equivalent?
[20:35:32] <dstufft> I see two ways to do it, one is "add it onto the existing stuff" in which case i'd just shoehorn it into the existing #md5=<foo>&sha256=<bar>" stuff that we already support (we support it for #egg and #subdirectory)
[20:35:59] <ErikRose> Granted, it would require a real dependency solver to obsolete the use case of "I'm smarter than the resolver, so I did it by hand" use case, but I'm not in a hurry.
[20:36:17] <dstufft> peep isn't really related to the req 2.0 format other than it'll provide a nicer way of specifying that info
[20:37:34] <dstufft> ErikRose: setup.py includes metadata, rquirements files do not, setup.py are "abstract" dependencies (I depend on requests, but I don't know where it's coming from because this is a package not an environment file) whereas requirements are concrete (I depend on requests, and I give an explicit source url of PyPI)
[20:37:35] <ErikRose> Very seldom do I yank something out of version control in a reqs file. Very often do I want to pin the hashes nonetheless.
[20:37:58] <dstufft> ErikRose: I'm pretty sure we don't support that at the moment, but I don't think there's any reason we couldnt is what I mean
[20:38:22] <Ivo> current requirements is just too simple, and bolting on things makes it unwieldly; already a lot of people complain and get confused by needing various --option's for different things in req files, and its not even handled properly everywhere https://github.com/pypa/pip/pull/1720
[20:39:26] <ErikRose> It's too bad we have to repeat info from setup.py, though, like package names.
[20:40:28] <dstufft> now for peep like functionality you have to do that because pip lacks a lockfile :)
[20:40:59] <dstufft> ah my ticket was specifically about supporting hashes on urls in a reuqirements.txt or on the cli
[20:41:42] <ErikRose> Right. And, for the record, I would never propose the syntax Django==1.4#sha256=.... ;-)
[20:42:45] <ErikRose> It's quasi-consistent with the URL format. It's got that going for it. But weird? Oh boy.
[20:43:40] <ErikRose> Alright, this is overwhelming, and I am tired. I think I've got to sleep on this and let it all settle.
[20:43:42] <dstufft> so I guess it depends on how much work you want to do and/or how long you want tto wait
[20:45:11] <qwcode> dstufft, I actually did a PR for your #468 long ago https://github.com/pypa/pip/pull/735
[20:46:55] <Ivo> what is the workflow of taking things from develop to master? what's master's purpose atm? :S:S
[20:49:02] <qwcode> I occasionly cherry-pick docs stuff to master, only if it's critical, but usually after merging to develop first
[20:54:49] <qwcode> it would be harmless to merge this really, but for me, I would close it, and just say it's already fixed in develop
[20:56:15] <Ivo> soo... anyone wanna review my PRs for pip? There are actually some that should make testing a lot faster :D
[20:58:20] <qwcode> Ivo, for testing (or any "small" changes), feel free to merge yourself if you feel comfortable/confident with it. (my opinion at least)
[21:04:13] <qwcode> we had conversations in the past about trying to enforce a 2 vote rule or something... but the informal rule now seems to be solo merges are ok for "small" things, and it's a judgement call to bring in votes when you're not sure, or when it's obvious it's a major decision or controversial
[21:05:28] <Ivo> I think a good middleground would be to just look for one other dev's LGTM (i.e code review) before merging
[21:09:45] <dstufft> I think it's hard to get reviews here partially because it's really hard to actually grok what any one change does without a bunch of untangling
[21:11:42] <dstufft> I think if we actually manage to refactor pip so that it makes sense, reviews would be a lot easier
[21:12:24] <dstufft> but part of the problem there is refactoring is hard, our test suite isn't the greatest, and I think I'm the only one who gets time in their day job to do this, so time is short :/
[21:13:08] <Ivo> this is also most definitely the case for virtualenv as well, except 1000 times worse
[21:13:10] <dstufft> I think this is also why we are really bad at merging PRs from non contributors too
[21:14:00] <pf_moore> yeah, I try to be a bit more willing to merge PRs from non-contributors than I would be for my own ones, just because it's good to give encouragement
[21:14:55] <qwcode> back when I was young... https://groups.google.com/forum/#!topic/python-virtualenv/dPDkQWaBXts
[21:15:05] <Ivo> could we merge a whole lot of stuff for virtualenv, and release a beta for a month? I know you were sad that not very many people have tried your pip RCs dstufft but I can't see any other way and there are some fixes that would really help some ppl atm
[21:16:51] <Ivo> if pip comes out there's no reason not to release a patch version for virtualenv as well
[21:17:55] <pf_moore> what stuff for virtualenv? A lot of the recent churn seems to be around the activate scripts. I'd be tempted to say they are OK to merge just because they are (somewhat) optional.
[21:18:03] <dstufft> Ivo: I'm just saying we've historically had the virtualenv and pip release cycles aligned
[21:18:48] <Ivo> pf_moore: just *fixing* activate scripts for shells, and also there's one that stops virtualenv working on arm atm; off the top of my head
[21:19:20] <pf_moore> dstufft: I think the main thing is not to release virtualenv with pip included then release a new pip a couple of weeks later. Otherwise, it's not a big deal
[21:19:56] <pf_moore> Ivo: yeah, I generally don't comment on those because as a non-Unix guy I don't really have a view
[21:21:02] <pf_moore> I *can* run Unix, it's just that when I do I spend more time getting annoyed at how hard it is to set up Ubuntu or whatever how I like it than actually testing anything ;-)
[21:21:22] <dstufft> (and then pip will never work on Windows ever again because i'm pretty sure the only reason it does is pf_moore
[21:22:14] <dstufft> like when I broke the entire test suite on windows and nobody noticed until pf_moore tried to run the tests :V
[21:22:58] <pf_moore> what I like about all this is that all I actually do is moan, then other people do the work and I get the credit. Total win :-)
[21:23:45] <dstufft> like when people thank me for wheel support and i'm all like, "well actually I had nothing to do with that :V"
[21:24:18] <pf_moore> yeah, dholth must be getting pretty fed up with everyone else getting the credit...
[21:25:05] <pf_moore> I even tried to create a bdist_simple format. Got nowhere, then Daniel came up with wheels and saved the world.
[21:25:23] <Ivo> i was gonna say, you must sometimes feel like a mother for 100 kids to tell their problems to sometimes dstufft
[21:28:09] <Ivo> heh, they seemed to have a "kill it with fire" attitude the first whiff they got of ensurepip :(
[21:28:38] <dstufft> they were _really_ nasty on their mailing lists when PEP453 was being discussed
[21:30:24] <Ivo> pf_moore: it all comes from the fact they provide all the software for the people that use their OS (where windows gives you a browser and tells you good luck)
[21:31:59] <Ivo> so when pip comes along and also purports to give people software, and (OMGOSH) from an uncurated index, tempers get frayed
[21:32:25] <Ivo> I think they'll like us a lot more if wwe manage to get --user by default working sometime
[21:33:11] <pf_moore> yeah, but I sort of lose the thread when they complain about /usr/local - I thought that was specifically *for* the local sysadmin to use...
[21:33:51] <dstufft> it's on Debian deriviates that they install to /usr/local by default because they patched their python
[21:34:14] <Ivo> (when run under sudo, and it doesnt help it just outright fails when run without sudo, by default)
[21:34:14] <dstufft> but yea I think if we make --user the default, the distros will hate us far less
[21:35:45] <pf_moore> What I don't get is understanding where, on my Linux PC, I'm allowed to install "global" stuff that isn't OS-owned
[21:36:11] <pf_moore> I thought that was /usr/local, but the distros seem to expect to own that (or is that just debian/ubuntu?)
[21:36:20] <Ivo> you install global stuff that isnt distro owned, by running `sudo make install` with the source of some software
[21:36:48] <dstufft> but people install stuff to /usr/local, break their OS, and then complain to Debian about it
[21:37:39] <pf_moore> well, I'd say that if /usr/local is for me, and installing stuff there breaks the OS, then that's an OS bug (just not the one you'd first think it is)
[21:49:10] <Ivo> whoever this guy is he does interesting reporting https://www.youtube.com/watch?v=CUNVwbp_JSA