Log file Viewer
#pypa-dev logs for Friday the 16th of January, 2015
[02:12:11] <pmxbot> jaraco pushed 5 commits to setuptools (https://bitbucket.org/pypa/setuptools/) :
[12:59:47] <ionelmc> dstufft: can you explain why you did that module reloading? (the del from sys.modules)
[13:00:39] <dstufft> ionelmc: because I want an import in the "real" code (not in the site shim) to import from the global site packages, not our copied copy.
[13:03:50] <dstufft> ionelmc: well for instance if debian updates a patch in one of those files and it gets updated in the global site packages then the virtualenv will need recreated to get that bug fixed otherwise. With this patch it only needs updated if the bug effects virtualenv itself. In addition it also means that people using the import finder to locate those files will get the same file as import will. Finally it matches the behavior of the venv module
[13:04:10] <ionelmc> dstufft: i have found that builtins (and prolly C ext module too) have a reloading issues, they can't be reloaded
[15:30:12] <ionelmc> i think i'm gonna abduct dstufft and keep him in a basement till he closes all the pull requests :-)
[15:30:43] <pf_moore> lol - I see no reason not to merge that one, let me do a quick check then I'll do so
[16:24:49] <pf_moore> Ah, I read the bit about screwy travis errors and misunderstood what you were saying. My bad, sorry.
[16:26:04] <dstufft> pf_moore: I assume since you were raising issues with the progress bar code, it means you've ran the develop branch on windows since we merged the prettier progress bar code? everything work ok?
[16:32:48] <ionelmc> dstufft: you were very ambiguous, you didn't actually made any request to make it for develop
[16:50:12] <pf_moore> dstufft: No, this was just from the released version. But the bit of testing I did with the develop version seemed fine...
[16:51:58] <pf_moore> You'd have never seen the issue anyway, it needs the buggy code plus an enhanced console that handles ANSI escapes including show/hide cursor
[16:52:43] <pf_moore> Between all the stuff that hacks ANSI escapes for Windows, I'm amazed they interact with each other as cleanly as they do...
[16:52:53] <dstufft> pf_moore: is there osme consoles we want to show/hide the cursor? is there some way to detect it?
[16:54:11] <pf_moore> Well, you could check for an environment variable ConEmuANSI = "ON". But if that's present you should probably just turn colorama off altogether.
[16:54:34] <pf_moore> Might be a source of more bugs we can't reproduce, though, rather than being a help...
[16:55:31] <dstufft> did pip's progress bar work previous to 6.0 on Windows? I didn't think it would work at all on Windows... I was surprised when someone said it did work
[16:57:06] <pf_moore> Big problem these days is, the bar isn't shown if you get a download from cache. So I had to keep finding new things to download, or clear the cache...
[16:58:03] <dstufft> easier then trying to find new things and better then blowing away your cache dir :D
[16:59:09] <dstufft> pf_moore: the hide_cursor fix, is that a big deal? should I do a backport and release 6.0.7?
[17:09:53] <pf_moore> I assume that if you do a 6.0.7 for any *other* reason it'll get added in, or are you not releasing from develop these days
[17:17:53] <pf_moore> dstufft, ionelmc: if we're looking at extra ci fo virtualenv, would it be worth adding appveyor for Windows stuff?
[17:20:02] <ionelmc> pf_moore: this is the test matrix for travis https://travis-ci.org/ionelmc/virtualenv/builds/47264714
[17:21:22] <dstufft> ionelmc: you're going to need to be more specific about why we'd want circle ci. I don't think layering on extra CI services is in general a good idea unless the additional services add something, but I'm not sure what Circle CI is adding
[17:23:18] <dstufft> adding more CI services means mroe places to look, more things to remember, more ways to restart the builds if one fails, etc. I added appveyor but I don't think that anyone but me can actually restart or cancel a build for instance. It complicates working with the project and sometimes complication is OK because it gets us something (like appveyor gets us Windows support), what does Circle CI get us?
[17:24:02] <dstufft> Honestly I don't even like the fact we have _two_ services now, and I'm talking to the Travis guys about if there's any way we get can windows into Travis so I can drop appveyor
[17:27:43] <ionelmc> eg: https://circleci.com/gh/ionelmc/virtualenv/16 and https://travis-ci.org/ionelmc/virtualenv/builds/46978125
[17:28:17] <ionelmc> there are a couple more cases like that but i don't really wanna manually search all the builds
[17:32:27] <dstufft> I'm looking at the test output now, it looks like circle CI isn't configured correctly or something, but more importantly, what were the bugs? Were they something that a generic test could be written that exposes them?
[17:35:09] <dstufft> Ok, so given that it couldn't find something and you indicated something about the sims, I'm going to guess this is becuse the shims aren't directly executable and you need bash to execute them?
[17:46:15] <dstufft> Honestly you're not really giving a clear picture of what's going on here and I don't have time today to decipher what the issue was and whether it's some one off things that it would be better to write tests for and run in the normal CI or if whatever setup Circle CI itself uses is one that is likely to need watched for regressions specific to it (such as Windows). Given that I don't think adding a second pyenv installed Python on an Ubuntu
[17:46:15] <dstufft> 12.04 box is likely a going end up showing unique regressions that aren't just bugs that can be fixed and tested for without more information to the contrary I'd rather not add more complexity to our CI
[17:51:26] <ionelmc> it's not like you're going to personally pay for circleci (or maintaining the configuration)
[17:58:06] <dstufft> I assume anything that gets added to a repo will eventually end up being maintained by the maintainers of that repository. I assume at some point our support pythons will change, or the way we invoke our tests, or something which will cause the CI systems to need to be adjusted to deal with the change. Anytime a PR gets accepted I assume myself and the other maintainers are going to end up responsible for it. and that's OK, but it means that
[17:58:06] <dstufft> I want to understand what we're adding and why and I want to agree that the benefit outweighs
[17:59:19] <dstufft> I don't understand what kinda of problems were exposed on circleci (I don't even know if it was a single problem or a bunch of problems) and whether they were the kinds of things that you just write tests for or if they were the kinds of things you add more environments to your cI system for
[18:04:16] <dstufft> to be clear, I don't know if there's a good reason to add circle ci or not, there may very well be I want to understand that reason before we do that though
[18:06:10] <sigmavirus24> ==dstufft. Having more CI systems with basically identical environments run the tests doesn't seem like an overall win when people have to main those CI systems' configurations
[18:09:15] <ionelmc> sigmavirus24: they aren't identically technically, travis doesn't have the shims
[18:10:23] <sigmavirus24> Can those shims be implemented in new travis tests instead of adding a whole new service + configuration?
[18:11:06] <ionelmc> prolly not, it would be anywhere close to what you'd get from vanilla pyenv (what circleci has)
[18:13:07] <sigmavirus24> ionelmc: more work upfront, less work down the road or less work up front, more work down the road is how I see this
[18:17:03] <pf_moore> ionelmc: from my POV, it's another test environment I'll end up needing to restart builds on
[18:17:44] <dstufft> I have a migraine, so I'm going to go lay down and pop some pain killers. I suggest the best path forward if you feel like a change to the CI is warranted is to post on pypa-dev so that everyone can comment on it. Or post an issue and highlight everyone. Either way I think it should probably be discussed in a wider audience than just whoever happened to be on IRC right now
[18:23:36] <pf_moore> Agreed. We're not going to get CI for every quirky setup that people can think of (sorry if a Windows user calling pyenv "quirky" seems offensive ;-)) so we should only add them when they hit a big currently-uncovered chunk of our userbase. And maybe not even then.
[19:23:31] <herrwolfe> dstufft: do you have a moment to talk about PR 2138 (warning about missing extras)?
[21:40:56] <pmxbot> jaraco pushed 3 commits to setuptools (https://bitbucket.org/pypa/setuptools/) :
[21:48:14] <dstufft> jaraco: incase you didn't see, there's a COPR repo (Fedora 19+, RHEL/CentOS 6+) for PyPA now, if you want to manage uploading new setuptools releases to it I can give you perms, if you want me to do it I can do that too :)
[21:50:12] <dstufft> jaraco: it's just a thing that makes it possible to easily have project specific yum repos
[21:50:45] <herrwolfe> dstufft: I remember you telling me that you aren't super familiar with that section of the codebase. Currently, if a user wants to installs several extras, like twisted[a, b, c] and a doesn't exist, it looks like b and c would never be tried
[21:51:13] <dstufft> essentially just these steps need completed -> https://github.com/pypa/copr/blob/master/UPDATING.md (although maybe they aren't complete, the guy who wrote them packages rpm so he might be forgetting something that's just "automatic")
[21:51:51] <dstufft> (I don't have one of those, I'm just going to spin up a throw away cloud fedora box)
[21:51:56] <herrwolfe> dstufft: would you prefer if I wrote the warning to be shown for each missing option?
[21:53:28] <dstufft> herrwolfe: you know I saw somewhere that we might be flat out erroring on an invallid extra in certain circmumstances, so we might be able to just do it as a flat out error, but if it stays a warning it should probably log a warning for all the misssing ones, whether that's one warning that lists all of the missing ones or a warning per I don't have a strong preference
[21:53:37] <jaraco> dstufft: I'd like to keep the setuptools release process portable. Perhaps the release script could trigger a post-release hook in another system that would do the COPR build.
[21:54:48] <dstufft> jaraco: eventually I'm going to have some automation for all of this, my goal is to make it so doing a release is basically ``git tag -s 1.0.0 && git push --tags`` so that's certainly a possible thing
[21:55:18] <dstufft> I'm focusing on pip originally for that automation but once I have it I want to apply it to virtualenv, and if you want it, setuptools too
[21:55:26] <herrwolfe> dstufft: ok I'll try to write it so that it logs a warning for all missing extras, and still proceed with the install for those that it can. Thanks :)
[21:56:38] <dstufft> pf_moore: is there something like SSH for Windows that I can execute commands on a Windows box from a Python script? (ideally a python script running on Linux)
[22:00:00] <jaraco> dstufft, I believe ansible uses Powershell remoting to mechanically invoke behavior.
[22:03:50] <pf_moore> Needs some setup on the target box, it's not enabled by default, but easy enough
[22:13:25] <dstufft> jaraco: http://logs.openstack.org/89/142689/8/gate/gate-tempest-dsvm-large-ops/c437c9d//logs/devstacklog.txt.gz#_2015-01-16_22_05_01_778
[22:16:41] <dstufft> jaraco: [17:14:30] <fungi> dstufft: having it back for long enough to figure out what we should call instead of easy_install.sys_executable() would be helpful, yes
[22:18:29] <jaraco> That is, if it's easier to fix pbr/openstack and saves an iteration on setuptools, that'd be better.
[22:22:27] <dstufft> yea so is a few other folks, lifeless and mordred and jeblair and a few others heh, there's some cross over ;)
[22:22:49] <dstufft> openstack integrates closer to setuptools and does more things than most, so they got broken easier
[22:26:46] <dhellmann> jaraco: it's hard to tell what's part of the official API for setuptools and what's not :-/
[22:27:13] <dhellmann> jaraco: it might be quicker to fix this in pbr, but we may not be the only project that was using that value
[22:28:39] <sigmavirus24> I didn't think /any/ of setuptools' command submodules were part of the API
[22:28:57] <sigmavirus24> Or rather, I never expect any command a library provides to be part of the PAI
[22:29:40] <sigmavirus24> OpenStack also imports things from pip which just seems like a bad idea but I don't know a better way to accomplish what they're goal is
[22:29:50] <jaraco> dhellmann, You're right. Much of the API is implicit... and I'm willing to accept breakage, compensating with a rapid response to emergent issues.
[22:30:21] <dhellmann> jaraco: the problem is, every time the openstack CI breaks 500-800 developers can't do anything :-/
[22:30:58] <dhellmann> I should resurrect that patch to add some integration tests to setuptools. I saw you commented on it recently.
[22:31:29] <dhellmann> sigmavirus24: oh, well, that's not production stuff so if it breaks I think the hit isn't too bad
[22:32:05] <dhellmann> jaraco: it looks like this failure was with the fixtures package, though I don't know if that does anything "special"
[22:34:38] <dstufft> dhellmann: on the other hand, if those 500-800 developers wanted to make sure they could do things more often their companies could commit resources to helping us out :D
[22:38:51] <dstufft> (I say that not to be a jerk, just recogizing the reality that we're severely undermanned for how many critical pieces we work on, and I'm more or less the only person who gets paid to work on these tools which means that we have to make choices that optimize our available time, some of that is moving faster and trying to respond to breakage rather than trying to outright prevent it in all cases)
[22:39:57] <pmxbot> jaraco pushed 3 commits to setuptools (https://bitbucket.org/pypa/setuptools/) :