[07:26:46] <dstufft> tdsmith: it's wrong, there's some bug that only sometimes happens
[07:26:56] <dstufft> if you do pip --version it'll be 6.1.0
[07:27:52] <tdsmith> well, then i ran pip install -U pip again: https://www.dropbox.com/s/qdazinz8ddkzbqx/Screenshot%202015-04-07%2000.26.39.png?dl=0 -- it says "uninstalling pip-6.0.8" ?
[07:27:57] <tdsmith> god knows i'll never be able to repro this
[07:29:15] <apollo13> well you did not think that you could just skip 6.0.8, did you?
[08:22:59] <dstufft> ronny: oh, the concept in general? I don't think it's likely that it's going to be an official thing, I don't personally like it much - but the way I want to go with things is that the build tool is completely pluggable so it's entirely up to developers what they want the "UX" to be for their build tools
[08:23:05] <mgedmin> ha ha https://pip.pypa.io/en/stable/news.html claims 6.1.0 is "unreleased"
[08:25:02] <dstufft> mgedmin: yea, I forgot to bump that when I cut the release
[08:25:13] <dstufft> I'm not gonna issue a new release just for that though, I already fixed it in the repo
[08:25:42] <mgedmin> I did that once or twice myself
[08:26:21] <mgedmin> (this is why my 'make release' generally checks my changelog with horrible Makefile magic: https://github.com/mgedmin/objgraph/blob/master/Makefile#L106-L108)
[08:26:50] <ronny> reminds me that i need something like that in gumby elf ^^
[08:27:12] <ronny> dstufft: i suppose you'd prefer a setup.cfg?
[08:27:52] <dstufft> ronny: myself personally? probably a yaml file. I don't find json files very friendly for a human to edit and ini files are kinda crummy.
[08:28:13] <dstufft> internally no matter what it is, it'll get compiled down to JSON inside of the sdist or wheel (long term)
[08:28:19] <ronny> dstufft: i see, yaml is tricky, because its so magical, i get really strange issues sometimes
[08:29:44] <dstufft> ronny: yea, I'm not entirely happy with YAML, but I think it's the least worst option for a human editable file
[08:30:20] <dstufft> an executable file isn't that big of a deal in the future (tm) when it's only used on the developer machine to generate a sdist
[08:31:00] <ronny> apollo13: my take at developer facing packaging tools
[08:31:00] <doismellburning> I think it's worth distinguishing between "YAML as a serialization format" and "YAML as a human-writable structured data format"
[08:31:12] <doismellburning> and for the latter, I really like it
[08:32:09] <ronny> apollo13: https://bitbucket.org/RonnyPfannschmidt/gumby_elf , currently only has a develop command (however one that supports pip uninstall)
[08:36:19] <xafer> hello, https://github.com/pypa/pip/pull/2629 should be ok to review/merge :)
[08:40:01] <xafer> will rebase it to deal with conflicts
[08:42:07] <ronny> dstufft: btw, do you have any writeups on the yaml format you'd like?
[08:42:33] <dstufft> ronny: nope, haven't thought much on it because I've been working backwards from the other side
[08:42:48] <dstufft> I just know I don't like json or ini as a human writable file
[08:43:00] <dstufft> no comments in JSON and things like no trailing commas and stuff is user hostile
[08:53:48] <ionelmc> the question is about boostrapping
[08:53:59] <ronny> and the develop install installs a python file in site-packages that will exec the development root module and is uninstallable via pip
[08:54:07] <ionelmc> let me rephrase: about making the package importable
[08:55:30] <ronny> ionelmc: the __init__.py thats written out sets the __path__ and __file__ of the module, then execfiles the __file__ thats in the real worktree
[08:55:47] <ronny> from that one it can import subpackes/submodules because of __path__
[08:56:03] <ionelmc> ronny: so that's enough to trick the import system?
[09:02:27] <ionelmc> but that also means you support dists with multiple root-module/packages
[09:02:29] <ronny> ionelmc: i had that initially, but when experimenting with layouts, and import issues, i concluded it would be best to make it deliberate in that way
[09:04:07] <ionelmc> ronny: how come you picked that bootstraper file over a pth file?
[09:05:43] <ronny> ionelmc: pth files cant sanely rename modules, and always add to sys.path, bootstrapers keep sys.path lean and support a more free layout wrt the src folder
[09:06:39] <ionelmc> ronny: ok yeah, but that's a flaw of your file layout, not pth per se
[09:07:14] <ronny> ionelmc: the flaw is deliberate, to prevernt pushing a sorce tree into sys.path directly (had too many issues with it)
[09:07:39] <ionelmc> so my problem with your current layout is that it's hard to port projects to it, or move back to setuptools if needs be
[09:08:04] <ionelmc> i'd rather see a ./src/gumby_elf/__init__.py
[09:09:08] <ionelmc> if you have that you only need to push ./src on sys.path, not the working tree (.)
[09:09:29] <ionelmc> and you don't need to have that weird bootstrap module anymore
[09:10:58] <ronny> ionelmc: i'd still need the bootstrap module to run the develop command for the first time
[09:13:06] <ronny> ionelmc: yup, im convinced now that i should put the package name into the src folder and use a recorded pth file instead of a bootstrap python file
[09:15:59] <haypo> hi. is pip 6.1 released or not? https://pypi.python.org/pypi/pip points to 6.1, but https://pip.pypa.io/en/stable/news.html says "6.1.0 (unreleased)"
[09:17:07] <ronny> haypo: news was accidentially not updated. it will be fixed later
[09:30:00] <haypo> pip doesn't provide a stable API or something like that?
[09:30:09] <haypo> for openstack, there is an open issue: https://bugs.launchpad.net/tempest/+bug/1440984
[09:31:27] <ionelmc> haypo: why do you guys even use that? isn't it some internal thing in pip ?
[09:31:41] <dstufft> haypo: No, pip has no API other than the CLI. There is an open ticket or two about providing one, but it requires someone to enumerate what they actually need in an API
[09:32:19] <haypo> ionelmc, the script is here, https://github.com/openstack/requirements/blob/master/update.py
[09:33:14] <haypo> ionelmc, i don't know this script. it looks to be used to update requirements.txt in other projects
[09:33:38] <haypo> ionelmc, when a dependency is upgrade in openstack, it's first upgraded in openstack/requirements which contains all dependencies with all versions
[09:33:54] <haypo> then, other projects are updated from openstack/requirements with update.py
[09:34:17] <ionelmc> haypo: so there's a single requirement set for *all projects* ?
[09:34:25] <haypo> ionelmc, openstack is very complex, it has more than +230 dependencies: https://github.com/openstack/requirements/blob/master/global-requirements.txt
[09:35:06] <dstufft> ionelmc: because pip doesn't properly handle multiple things depending on the same thing, openstack has a work around where they list the exact specifier tou're allowed to use for each dependency
[09:35:06] <haypo> ionelmc, global-requirements.txt is a global policy for versions. it's common to require a minimum version and sometimes to exclude some versions
[09:35:09] <ionelmc> i have 200 deps in my project
[09:36:05] <ionelmc> tho it's split in just 2 monoliths
[09:37:42] <ionelmc> haypo: how does openstack know what projects use what deps?
[09:38:14] <ionelmc> eg: if I maintain the global-req.txt, how do i know what projects use what?
[09:38:23] <haypo> ionelmc, update.py parses requirements.txt of a project and then updates versions of each line (i'm not sure, but i guess that it does something like that)
[09:38:30] <haypo> ionelmc, each project has its own requirements.txt
[09:38:37] <ionelmc> because i can't be looking at all the req.txt from all the many small projects
[09:38:46] <haypo> example for nova: https://github.com/openstack/nova/blob/master/requirements.txt
[09:39:12] <ionelmc> haypo: i got that, i'm asking about the global req.txt
[09:39:57] <haypo> ionelmc, sorry, i don't understand your question
[09:40:43] <haypo> dstufft, "No, pip has no API other than the CLI" too bad :-(
[09:40:44] <ionelmc> haypo: as a maintainer of https://github.com/openstack/requirements/blob/master/global-requirements.txt how can i know what projects are going to be affected if i change versions for a dependency?
[09:41:13] <dstufft> ionelmc: openstack has a massive CI system, I don't think any change to that file can happen without running the tests across a bunch of things
[09:43:52] <haypo> ionelmc, requirements.txt of each component is not updated immediatly. a bot proposes to update requirements.txt, and we have extensive tests for each pull request
[09:44:02] <ionelmc> dstufft: i'm only asking about the process, i don't doubt their tests :)
[09:44:14] <haypo> ionelmc, if we notify a failure, it's then fixed in the project, or the global requirements is modified
[09:44:44] <haypo> as any other pull requests, it requires two +2 votes to be approved
[09:44:52] <haypo> (only core developers can vote +2)
[09:46:04] <ionelmc> haypo: so how does the maintainer of https://github.com/openstack/requirements/blob/master/global-requirements.txt decide to update a dependency there?
[10:13:32] <jhermann> isn't it nice when you are about to ask for a feature, and it's already in there? :)
[10:13:47] <jhermann> dstufft, :thumbsup: for -r taking URLs
[10:15:22] <dstufft> haypo: it was released a few hours ago
[10:15:53] <haypo> dstufft, oh, it already bites me :)
[10:16:08] <haypo> FYI i opened https://bugs.launchpad.net/keystone/+bug/1441083 for the argparse issue
[10:16:34] <dstufft> haypo: open an issue on pip please
[10:16:50] <dstufft> that's an unexpected side effect of a change I was on the fence on anyways
[10:16:59] <haypo> dstufft, maybe later, right now i just cannot work, so i have to fix devstack :-p
[10:17:01] <dstufft> I'll just revert it and cut a 6.1.1
[10:18:19] <haypo> dstufft, i don't know anything about pkg_resources. can you explain me how it checks dependencies at runtime? where are these dependencies stored? could you please see the traceback at https://bugs.launchpad.net/keystone/+bug/1441083 ?
[10:22:01] <haypo> oh, i found /usr/lib/python2.7/site-packages/pysaml2-2.4.0-py2.7.egg-info/requires.txt:argparse using strace+grep :)
[10:23:04] <haypo> too bad, pysaml2 is not maintained by openstack, and its setup.py contains install_requirements=[..., 'argparse', ...]
[10:24:23] <dstufft> haypo: https://github.com/pypa/pip/pull/2650 should fix it
[10:24:45] <haypo> dstufft, is it possible to use environment markers in the install_requirements parameter of setup()?
[10:26:43] <haypo> dstufft, thanks! i completed https://bugs.launchpad.net/keystone/+bug/1441083 with pysaml & your pull request
[10:32:52] <dstufft> haypo: I added a (temporary) shim for he update.py problem too
[10:33:01] <dstufft> haypo: that attribute will be gone in the _next_ release
[10:38:34] <jhermann> haypo, it's Python, you can do anything you want, as long as it works ;)
[11:21:35] <haypo> dstufft, "btw I released a 6.1.1 to deal with unexpected breakage" thank you so much!
[11:21:59] <haypo> dstufft, any idea how we can avoid such issues in the future? it's not the first time that a pip release "broke" openstack
[11:22:45] <dstufft> haypo: little bit of openstack not touching internals so much, a little bit of getting better tests in pip itself. PAckaging as a bit of a postel's tarpit effect going on so it's hard to make any changes without breaking _someone_
[11:22:56] <dstufft> and openstack does crazier things than most
[11:24:14] <haypo> dstufft, maybe openstack should provide you a CI to always test openstack with the development version of pip?
[11:25:47] <haypo> dstufft, using pip 6.1.1, i get -> AttributeError: 'NoneType' object has no attribute 'url'
[11:26:06] <nanonyme> Would anyone happen to know whether you can instantiate classes and call their methods as entry_point?
[11:26:14] <haypo> dstufft, oh, wait, the full traceback: http://paste.openstack.org/show/199343/
[11:26:54] <haypo> dstufft, your workaround property doesn't work if link is None. you need something like return getattr(self.link, 'url', None)
[11:27:13] <ionelmc> dstufft: haypo: pre-releases could help, if openstack would test them first
[11:27:37] <haypo> ionelmc, openstack has a team of people *paid* to ensure that openstack tests are running well
[11:27:50] <haypo> ionelmc, you should try to contact them, for example, you can join #openstack-infra
[11:29:04] <ionelmc> haypo: well they could test the git branch then, if they are really paid, dstufft has announced 6.1.0 a month ago or something
[11:30:13] <haypo> ionelmc, (well, i'm not 100% sure that there is a team paid to run the CI)
[11:30:22] <ionelmc> just saying, openstack's fault is they did not test the upcoming release code
[11:30:33] <haypo> ionelmc, it's a communication issue ;)
[11:32:41] <haypo> dstufft, i checked getattr(self.link, 'url', None), it worked for me
[11:33:09] <linovia> dstufft: are there already written plans / discussion about the pip test suite improvement ?
[11:33:26] <haypo> ionelmc, i don't want to say that the fault comes from pip or openstack, i don't care
[11:33:29] <ionelmc> haypo: iow, if only openstack has issues with packaging tools then maybe openstack is doing something wroooong
[11:33:38] <haypo> ionelmc, i'm trying to find a solution to avoid regressions in the future
[11:34:59] <haypo> ionelmc, "if only openstack has issues with packaging tools" the argparse issues comes from pysaml2. with pip 6.0.8, it was possible to install & use pysaml2. with pip 6.1.0, it's no more possible
[11:35:01] <dstufft> I hang out in #openstack-infra so I tend to try and sort out where things need fixed there
[11:35:53] <ionelmc> ronny: neeeever seen that before! ;-)
[11:35:54] <haypo> IMO the real issue is that PyPI has almost 57.700 packages, and that pip has too few tests
[11:36:26] <dstufft> linovia: there's no written plans beyond "we need better tests"
[11:37:33] <ionelmc> "better tests" as in "high threshold for pain"?
[11:52:31] <linovia> can't this be done by instantiating the the class in the module and calling the instance in the setup ?
[11:52:40] <ronny> nanonyme: an entrypoint shows an object that is obrainable via import + recursive getattr
[11:53:02] <ronny> nanonyme: calls are not part of that
[11:53:05] <nanonyme> linovia, well, the "right" way would be to turn the method into a classmethod
[11:53:13] <ronny> nanonyme: whats the final use-case?
[11:53:28] <ronny> nanonyme: in ase of doub, just make a _foo function and use that in the ep
[11:53:40] <nanonyme> But this isn't my project and changing the method to classmethod would be a pretty drastic change
[11:54:15] <linovia> nanonyme: you class method could also do as ronny said and instantiate the class + call your function
[11:54:40] <linovia> nanonyme: and I'm not convinced a class method is the right way to go
[11:55:07] <nanonyme> ronny, basically there's a AssistantClass which has a method main(). This used to be done by having a separate script that instantiated it and called main
[11:55:31] <haypo> dstufft, a workaround for the .url attribute has been commited. update.py now works with pip 6.0.8, pip 6.1 and 6.1.1
[11:55:38] <ronny> nanonyme: just make a function that does that?
[11:55:39] <nanonyme> And scripts=[] then included this helper script
[11:55:54] <haypo> dstufft, it looks like all pip 6.1 issues in openstack have been fixed (quickly), cool!
[11:57:11] <ronny> *insert a all the things meme picture here*
[11:57:39] <ionelmc> nanonyme: so it's either me getting stabbed cause you use a metaclass, or you just fix your methods
[11:58:04] <nanonyme> ronny, ends up messy. Most likely cleanest would be to make main() a classmethod and call everything inside main() on the created instance instead of self
[11:58:21] <ronny> nanonyme: you can do that as well
[11:58:35] <ionelmc> damnit now I have this urge to describe the metaclass solution
[11:58:39] <nanonyme> I'm pretty sure it's the best way to do it but it's also the most invasive way to do it :)
[12:39:54] <ionelmc> nanonyme: now say you regret asking that question :-p
[12:52:53] <haypo> ionelmc, dstufft : oh, i patch to "attempt to stop using pip internals" https://review.openstack.org/#/c/171157/
[13:41:52] <pf_moore> Is it weird that using a pip session to get https://pypi.python.org/simple takes longer than just using requests.get()?
[13:44:18] <pf_moore> I was trying to use pip's cache to avoid hitting the net, and I built a session by nicking the code from pip.basecommand (_build_session)
[13:45:03] <pf_moore> But doing len(session.get().text) takes about 2 secs with that session, whereas len(requests.get().text) takes about 1 sec...
[13:57:04] <ionelmc> pf_moore: maybe you could profile it, or trace it?
[13:57:36] <ionelmc> pf_moore: give https://pypi.python.org/pypi/hunter a shot
[14:01:51] <pf_moore> ionelmc: possibly, but I don't really want to go digging into the pip internals here. This isn't a huge "my code is too slow" issue, I just thought that seeing as I have this bunch of downloaded stuff pip maintains, maybe it'd be good to use it.
[14:02:23] <pf_moore> I was just surprised to be that wrong - "not noticeably faster" I could accept, but slower seems odd.
[14:03:19] <pf_moore> Given that a pip session isn't really intended for use outside of pip's code, I'm not expecting much. I wonder if the cache is keyed off the program name somehow?
[14:05:08] <pf_moore> Looks like my code isn't even reading the pip cache files. (The last access time didn't change).
[14:05:37] <pf_moore> So the problem is that I'm not building the session correctly. Hardly a surprise.
[14:06:52] <pf_moore> It would be nice if pip had a supported "build a session" API to let 3rd party code make use of the pip cache.
[14:30:45] <pf_moore> yep, it was the default values for the various session options. I wasn't picking up the right defaults to match pip.
[14:31:24] <pf_moore> 0.05 sec vs 2 sec is a much better ratio (cache: non-cache).
[23:11:54] <thorie> hello.. anyone seen this pip error before? Exception __init__.py", line 236 ... http://pastebin.com/kKi5YEHX
[23:16:03] <thorie> it only started happening after i upgraded my robotframework pip package