[06:22:45] <pradyunsg> Any help figuring out why this is failing would be welcome: https://github.com/pradyunsg/pip/pull/4/checks?check_run_id=573039525#step:6:2679
[06:24:51] <toad_polo> pradyunsg: I'm going to go with "lack of moral fortitude"
[07:46:56] <pradyunsg> Not on my system, nor any other CI provider's environments.
[07:47:26] <pradyunsg> Other than that, yes, it is a Python 2 specific failure from what I can tell. I'm inclined to add a @pradyun_does_not_care_about_python_2_anymore decorator now. ๐คท๐ปโโ๏ธ
[07:47:34] <devesh> how did you check other CI providers? Do you have them set up locally?
[07:48:14] <pradyunsg> devesh: the test runs on Azure Pipelines and Travis CI and passes, as part of master. ๐
[07:48:38] <devesh> Ohh, so it's failing on github ci?
[09:13:20] <pradyunsg> That decorator marks that the test is incompatible with "venv" -- which is a specific tool for generating a specific kind of virtual environment. "venv" is Python 3 only, and ships as part of the standard library.
[09:13:55] <pradyunsg> devesh: you seem to be able to reproduce the issue -- since you're also on a Mac, could you tell me the steps to reproduce?
[09:14:18] <pradyunsg> "the issue" == the test failing
[09:14:21] <devesh> Okay, I ran `tox -e py3 -- tests/unit/test_locations.py` from system python, and it tells me `SKIP [1] tests/unit/test_locations.py:94: Incompatible with venv`
[09:36:12] <pradyunsg> tests passed for me in a virtualenv created using `/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python`. :(
[09:44:16] <devesh> The syntax used seems fine: monkeypatch.setattr(obj, name, value, raising=True)
[09:44:37] <pradyunsg> devesh: yea, the test works on other systems. :P
[09:44:53] <pradyunsg> there's going to be something subtle that's causing the issue.
[09:46:02] <pradyunsg> I suggest working backward from the assert, printing-where-reasonable to see where the value originates from. The code is going to hop around between pip / distutils in all likelyhood.
[09:46:18] <devesh> can you outline what the test does in short
[09:47:10] <devesh> I can follow along the code accordingly :)
[09:48:02] <pradyunsg> devesh: it checks that pip's "distutils_scheme" function respects distutils configuration, done via configuration files that distutils supports.
[09:49:05] <devesh> okay, so you set the scripts to somewhere/else and patch distutils to find them
[09:49:23] <devesh> and assert whether distutils_scheme actually did that
[09:49:35] <pradyunsg> devesh: in some sense; yea.
[09:51:16] <pradyunsg> devesh: the test writes a config file saying "install-scripts=somewhere/else"; does a monkey-patch to make distutils read that file; and asserts that the same value comes out of "distutils_scheme" gives us `"script": "somewhere/else"` (for our purposes, distutils_scheme does a `install-xyz` -> `xyz`).
[09:51:41] <pradyunsg> s/gives us/which should give us/
[09:52:09] <devesh> got it, let me see what I can do here
[10:06:33] <devesh> So if I write a explicit function like def mock_find_config_files(self): return [f]
[10:07:34] <devesh> and use it instead of lambda, and print d.find_config files within distutils_scheme
[10:07:55] <devesh> I do get <bound method Distribution.mock_find_config_files of <setuptools.dist.Distribution instance at 0x11141aa70>> which means it's getting mocked pradyunsg
[10:08:24] <pradyunsg> it's getting or not getting?
[10:08:46] <pradyunsg> I'm sorry; I don't follow what you're saying.
[10:10:11] <devesh> So the find_config_files atrribute of the object d where `d = Distribution(dist_args)` within distutils_scheme
[10:10:45] <devesh> points to the mocked function name I defined explicitly, I think it means that it's getting mocked
[10:13:28] <pradyunsg> devesh: I see what you mean. I don't think I was concerned that monkeypatch isn't working -- more like the intended effect of the moneypatch is not happening and idk why that is. :)
[10:27:09] <devesh> Actually 2.7 doesn't have anything called sys.base_prefix
[10:27:13] <pradyunsg> https://github.com/pradyunsg/pip/pull/4/checks?check_run_id=573591303#step:6:59 <- did I mention I genuinely hate working w/ Python 2?
[10:27:59] <devesh> print is not a function in python2
[10:29:12] <devesh> Aah, I was checking the source code for python3.8 where sys.base_prefix was present
[10:32:08] <pradyunsg> techalchemy: today was my day for fighting w/ Azure Pipelines and tests failing for unknown reasons. :) https://usercontent.irccloud-cdn.com/file/pAlkGnmj/Screenshot%202020-04-09%20at%203.42.30%20PM.png
[10:34:08] <devesh> You haven't started naming your commits "Why"
[10:34:22] <devesh> You are still far away from the real pain :)
[10:34:48] <pradyunsg> devesh: I have a high pain tolerance level. :P
[10:42:00] <devesh> So the ideal behaviour would have been that after calling `d.parse_config_files()`, `i.install_scripts` would be non-None
[14:17:47] <sumanah> hi techalchemy - hope you are doing well
[14:18:29] <sumanah> I'm working today on a few things including updating https://wiki.python.org/psf/Fundable Packaging Improvements -- would love suggestions on what specific Pipenv features or activities perhaps should go there
[14:19:08] <sumanah> jaraco: toad_polo: I'm working today on updating https://wiki.python.org/psf/Fundable Packaging Improvements and would appreciate improvements on features, activities, etc. you think could go in there, espec. re: setuptools
[14:19:36] <sumanah> I may be able to have a conversation with a funder tomorrow so I'm hoping to get at least some more idea-seeds in there by the time I head to bed tonight
[14:20:15] <toad_polo> If you want to give more details about the funder (we can do this privately), I could probably come up with some ideas targeted to the things they are interested in and the size of the project they may want to fund.
[14:20:33] <toad_polo> I think the biggest thing we could use would be documentation.
[14:22:25] <toad_polo> If the turnaround is reasonably quick then some funding to get someone allocated for a distutils/setuptools-based standardized editable installs proof of concept would also probably be fruitful.
[14:22:57] <toad_polo> That's probably a reasonably small project, but you may be able to get companies to donate an engineer for it.
[14:23:48] <toad_polo> Something small that is maybe GSOC-sized would be adding support for pyproject.toml as a way to configure setuptools: https://github.com/pypa/setuptools/issues/1688
[14:24:15] <pradyunsg> toad_polo: moving distutils into setuptools might be a good project for this?
[14:24:25] <toad_polo> Yeah, moving distutils into setuptools would also be good.
[14:24:31] <sumanah> Joshua Lock from VMWare has been talking with us in Zulip https://python.zulipchat.com/ about some in-kind donation of engineer services which may be a good avenue for some of this
[14:24:53] <sumanah> toad_polo: I think I need more details to put into https://wiki.python.org/psf/Fundable%20Packaging%20Improvements#Make_setuptools_the_reference_implementation_of_the_distutils_API
[14:25:33] <sumanah> toad_polo: the third CZI cycle will open in June for Chan Zuckerberg Initiative funding https://chanzuckerberg.com/rfa/essential-open-source-software-for-science/
[14:26:43] <sumanah> MOSS takes applications on a rolling basis https://www.mozilla.org/en-US/moss/ and, I hope, we'll be done with our current MOSS project in a few months, so it would be good to prep applications now to submit
[14:26:57] <devesh> Wow, so there are companies which fund open source projects as well. Nice to know
[14:27:12] <sumanah> devesh: please see my blog post https://www.harihareswara.net/sumana/2019/12/04/0 for more
[14:27:26] <sumanah> I am glad that some companies and nonprofit foundations fund open source work, yes
[14:27:37] <cnx> hi everyone, has been a while since the last time I'm here
[14:27:38] <devesh> Wow, you have a blog post on every thing OSS :)
[14:30:34] <cnx> BTW about the wheel/sdist decision thing, can we improve the logging
[14:30:51] <cnx> or more generically, can we have more logging levels
[14:31:00] <pradyunsg> sumanah: Designing an improved Warehouse API could be a good project to add. It's tangled up in with upload API discussions, and the metadata problem for dependency resolution.
[14:32:15] <pradyunsg> Though @di, dstufft and EWDurbin likely have inputs on that. :)
[14:33:12] <pradyunsg> cnx is refering to this issue: https://github.com/pypa/pip/issues/8003
[14:34:32] <pradyunsg> cnx: I'll respond to you in a moment.
[14:35:48] <cnx> pradyunsg, okay; more context: in the last week (I think) there are a few issues relating to pip choosing sdist over wheel (or not find wheel at all as pfmoore said) and users has no clue why that's happening
[14:36:42] <cnx> simply adding the logic to the log would be fine but I wonder where, since
[14:37:14] <cnx> (1) debug is just too noisy at the moment
[14:37:36] <cnx> (2) the decision making is the part that make the verbose log noisy
[14:38:29] <pradyunsg> sumanah: IMO "Add pipfile support to pip" should likely get updated, since Brett Cannon and Tzu-Ping made some progress toward standardizing an interoperable lockfile format IIUC. Personally, I'd suggest we phrase it more in terms of "Standardize a lockfile format and add support to pip" but there's folks here who are better at handling communication / branding / position than me. :P
[14:54:07] <devesh> pradyunsg: In order to add docs on what different keys in say a json output look like, will the reference guide be the right place?
[14:54:39] <devesh> this is in reference to adding what different keys in the output of say pip show --format=json would signify
[14:56:16] <cnx> devesh: I did not collect enough understanding on this, but why JSON instead of plain text as CLI output?
[14:57:11] <pradyunsg> cnx: yea, we really need to take a deep look at pip's output. That's on my pipeline, and there's a big "re-visiting pip's output" issue that basically is the broader issue for discussing this.
[14:57:42] <devesh> plain text needs parsing logic, which might break for any change made by the library, JSON being a standard format might be more useful IMO
[14:58:06] <pradyunsg> JSON is a good data interchange / transport format.
[14:59:28] <devesh> As per https://github.com/pypa/pip/pull/7967#pullrequestreview-390303366 , I think we need to show what each key in the output signify
[15:42:09] <sumanah> woodruffw: EWDurbin: I was just updating some notes on the wiki and saw that https://wiki.python.org/psf/PackagingWG#Warehouse:_Facebook_gift could use an update, in case we know more about the TUF rollout timeline
[15:44:55] <woodruffw> sumanah: hey! i don't have any timeline updates on my side -- i've been parceling work on the Warehouse components of TUF as the TUF team + contributors have been making needed changes, so the timeline there is mostly contingent on how fast the upstream completes the abstract file storage mechanism that Warehouse requires
[15:46:14] <woodruffw> the other major component is the key generation and signing ceremonies -- i've been working on those in parallel since they don't require (much, any) changes on the TUF side. the first stage in the TUF rollout would be performing the key generation ceremony, which we could do as soon as the runbook is completed and every keyholder is in possession of their HSM + secure offline computer
[15:46:58] <sumanah> woodruffw: Thanks for summarizing. (HSM?)
[15:47:20] <woodruffw> whoops! hardware security module, the yubikey-like things that we're using to store keypairs
[15:48:44] <sangy> fun fact: I do have an HSM's for dummies book sitting on my desk at NYU
[15:48:56] <sangy> it's amazing what that editorial covers sometimes
[15:49:15] <woodruffw> getting these HSMs to play nicely has been an adventure :-)
[15:49:37] <sumanah> woodruffw: "every keyholder" -- so this is the people who will be administering PyPI itself, correct? So, on the order of 5 people rather than hundreds or thousands. Correct?
[15:50:31] <woodruffw> sumanah: correct -- the keyholders are going to be a body of N people (currently planned for N=6), of whom 50% are required to sign the chain of trust
[15:50:45] <sumanah> sangy: :-) I miss a few things from my office; there are a few books I didn't grab on that last-day-for-a-while
[15:51:20] <woodruffw> the plan that Trail of Bits proposed is to have 3 PyPI admins and 3 non-PyPI admins from the PSF, one of whom is a legal counsel
[15:52:08] <woodruffw> the idea being that the PyPI admins perform signing in the normal case, and we have 3 opportunities for recovery (including a legal principal) should any of the HSMs become lost or compromised
[15:52:14] <sangy> woodruffw: yeah it's funny how many building blogs in the security world are under-specified like that. I was just recently dealing with the fact that as of tpm 2.0 their sysfs file node is just gone, so the best way to see if your TPM complies with 2.x is just to see if that node is missing but you get tpm-stuff back...
[15:52:18] <sumanah> woodruffw: Thank you for this! I'm just a curious bystander here as far as the TUF timeline goes and I have zero project management responsibility on it, which is novel :-)
[15:52:44] <woodruffw> sumanah: no problem! happy to give an update :)
[15:53:12] <woodruffw> sangy: yeah, there's definitely an element of truth to cryptographers not being the best at UX...
[15:53:16] <sumanah> I mean, I want to know more so I can plan other stuff, grant applications, possible pip implementation, etc. etc., but it is refreshing to be asking as somewhat more of a community member
[15:53:36] <sumanah> woodruffw: I'm so glad we have Simply Secure helping us with this pip UX work!
[15:54:14] <woodruffw> sumanah: yeah, that was great news to hear :-)
[15:54:53] <sumanah> If we have some funding for future TUF/in-toto rollout work all the way through the toolchain, it would be cool to be able to bring in them or someone like them
[15:55:11] <sumanah> ok, back to a call. thanks woodruffw
[17:51:28] <pradyunsg> BTW, @techalchemy did you manage to get Windows downloaded eventually?
[17:52:03] <pradyunsg> https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/ is apparently the link for the 20GB download for a free evaluation VM.
[17:52:14] <techalchemy> pradyunsg, yeah i did that days ago
[17:52:41] <pradyunsg> techalchemy: awesome; I was fairly certain but figured it wouldn't hurt to confirm. :)
[17:53:10] <techalchemy> that wasn't the issue, something about secure boot restrictions / linux security modules not letting me run windows, i didn't manage to sort it out
[17:53:47] <techalchemy> I did spin up a vm on GCE and tests passed there, story of my life tbh
[17:54:10] <techalchemy> just like the random python3.6 failure that continues to happen on linux that i can't reproduce anywhere but fails every single time i run it in ci
[17:55:07] <pradyunsg> techalchemy: I spent a lotta time w/ AP today -- and their "complete" but not-containing-enough-info-to-work-with documentation -- and I can try taking a stab at pipenv's CI situation over the next little bit, if that'd be useful.
[17:55:38] <techalchemy> pradyunsg, sure, if you have _any_ thoughts about why i can't get it to report anything about windows i'd be really interested
[17:57:52] <techalchemy> i've downloaded the raw logs even, i.e. select 'failed' 'abandoned' and 'other' from https://dev.azure.com/pypa/pipenv/_build/results?buildId=21643&view=ms.vss-test-web.build-test-results-tab&runId=1338206&paneView=attachments & sort by duration, click on the longest one (e.g. python 3.8 in this case) and go to 'attachments'
[17:58:59] <techalchemy> but for some reason it exits with code 1
[17:59:02] <pradyunsg> techalchemy: I'll experiment on a PR on my own fork. You mentioned that you tried running tests on a subset and you still had these issues; could you tell me where that run-with-subsets is?
[18:00:54] <techalchemy> pradyunsg: gone now, i split them in half basically and ran by markers.
[18:01:12] <techalchemy> the test command is what you need i guess
[18:01:52] <pradyunsg> techalchemy: it's GitHub -- the code is still there: https://github.com/pypa/pipenv/commit/d3f0aba3f633b4042fbc849e7439b15c8ef88c4f :P
[18:02:06] <techalchemy> i wasn't sure if i force-pushed over that
[18:02:24] <pradyunsg> techalchemy: GitHub still preserves the reference.
[18:02:24] <techalchemy> i dont think it preserves reflogs
[18:15:08] <sumanah> And then he has to head to bed. So you probably won't get an answer from him till tomorrow.
[18:15:26] <devesh> Ohh no worries, I can ask him at a later time
[18:15:40] <sumanah> devesh: well, you could also just leave a comment on GitHub so this can happen asynchronously
[18:15:55] <sumanah> devesh: I mentioned the other day that it would probably make sense for you to find some projects where their review capacity and speed matches your work capacity. Have you started looking into that yet?
[18:17:16] <devesh> Hi sumanah, I did leave one, a few days back, didn't want to leave too many :)
[18:17:37] <devesh> Yes, I left a comment on one of the issues I wanted to work for in twine, I am waiting for the response from the other end
[18:18:07] <sumanah> devesh: ok. What is your expectation about how fast you should be getting responses on things?
[18:18:31] <devesh> Nothing as such actually. I don't have any expectations
[18:18:40] <devesh> I always wait for atleast 3-4 days before asking again
[18:18:55] <sumanah> Well, I would say you do have some expectations, then, of 3-4 days, because you are escalating once you wait that long
[18:19:11] <sumanah> And what are your criteria for what deserves escalating or re-pinging, and what can wait possibly for weeks or months?
[18:19:46] <devesh> Umm, I haven't thought about it that a lot, guess whatever is the general etiquette towards these things
[18:20:05] <devesh> I understand this is not a corporate setup, and all are volunteers here
[18:20:05] <sumanah> devesh: ok, then let's talk about etiquette in this situation
[18:20:19] <devesh> So any opinions and education I can get in this regard is welcome
[18:20:36] <sumanah> devesh: Pradyun, Tzu-Ping, Paul, and I are volunteering some of our time, but we are also being paid by PSF to work on the pip resolver right now, and we have deadlines
[18:20:57] <sumanah> https://wiki.python.org/psf/Pip2020DonorFundedRoadmap and https://pyfound.blogspot.com/2019/12/moss-czi-support-pip.html have more info on that
[18:21:17] <devesh> Ohh nice, didn't know about it. Thanks for sharing :) Looks great
[18:21:25] <sumanah> ei8fdb, a user experience researcher in this channel, is also one of my colleagues on this paid project
[18:22:02] <sumanah> The vast majority of the time, no one working on pip, or nearly no one working on pip, is being paid to do so. Right now is an exception
[18:22:24] <sumanah> There are several pip maintainers who are not contracted to work on this paid project
[18:22:27] <toad_polo> Anyone know if there's a canonical bug for making pip install stuff that fails `python_requires`?
[18:23:06] <toad_polo> It causes a lot of problems when you are trying to test outside the specified support matrix.
[18:23:10] <sumanah> some people maintaining various https://packaging.python.org/key_projects/ packaging projects are paid to do so. Spack, for instance, has several paid people
[18:24:05] <sumanah> devesh: I suggest you start looking at Spack https://packaging.python.org/key_projects/#spack as a project to contribute to, since it also deals with Python and packaging, and I think the speed of their reviews is likely to be a little faster
[18:25:59] <sumanah> devesh: but in terms of etiquette: if you need to get a response to an issue or pull request because it is stopping you from getting/doing something you need to do, then re-pinging on the issue is good. If you need that response urgently because you have a deadline coming up, saying so in IRC is fine. But in general, please try to use asynchronous media, such as GitHub and the mailing lists, when possible
[18:27:16] <devesh> So generally I do almost always ping on Github, and wait for a few days before asking here
[18:27:31] <devesh> But I get your point, I should give it more time.
[18:27:36] <sumanah> devesh: I think I haven't seen you post yet on https://mail.python.org/archives/list/distutils-sig@python.org/ -- I think it would be good practice for you if, the next time you have a question about "how does this work?" or "where does this come from?" or some general learning question, I would like for you to mail the list
[18:27:48] <sumanah> devesh: I actually do not think you have understood me
[18:29:18] <sumanah> we want your contributions and we want to ensure you have the information you need, but right now, you are defaulting to "I haven't gotten a GitHub response in n days, I will follow up in IRC" and I am asking you to add some additional filtering criteria there.
[18:30:31] <sumanah> Filtering criteria like "do I need to get a response on this in order to meet a deadline someone else is expecting of me?" or "is the lack of a response on this stopping me from getting or doing something I need" (note: NEED, not want)
[18:31:13] <sumanah> Does that make sense devesh ?
[18:31:28] <devesh> Okay, then may I ask what is the primary purpose of this channel, I will try to stick to those topics in the future
[18:31:39] <devesh> And use more of async media like you said for other things
[18:32:24] <devesh> As I said, I am also learning things here, so pardon my ignorance of such things :)
[18:32:28] <sumanah> devesh: The primary purpose is: live, informational conversation about the development of Python packaging tools, including talking through bugs, teaching each other, etc.
[18:32:48] <sumanah> devesh: I think you're still missing what I am saying about -- not the purpose, but the SHAPE of the different conversational mediums
[18:34:18] <sumanah> IRC, code comments, GitHub, the mailing lists, live conferences are all ways to communicate with each other and with our future selves about the tools we are making
[18:35:46] <sumanah> they have different shapes -- different rhythms
[18:36:00] <devesh> That's true, some are for more pressing and live convos, and some can be used for things which can happen in the background
[18:37:04] <sumanah> So -- just for the next week devesh could you try to use GitHub and the mailing lists, async media, for nearly everything, and then ask yourself "does this really need to be a live conversation?" before asking for something in IRC?
[18:37:20] <sumanah> and then let's talk again and see how that went?
[18:38:11] <sumanah> and I do suggest you look at Spack as a project that has more maintainers currently working on it for more hours per week. I think they will be able to respond to issues and PRs faster.
[18:39:27] <devesh> Or slowing down the pace a bit, and contribute to pip. See anyways I can't be contributing at this pace for long anyways
[18:40:37] <devesh> I guess it's also about what expectations a contributor is setting for a repo, and he has to be able to change it according to the maintainers pace
[18:42:13] <devesh> And I think you have been trying to set that expectation since you are driving the project as well
[18:43:43] <devesh> I am sure you are not looking to drive any contributor away from contributing further, but want to set expectations so that you can have that harmony
[18:43:57] <sumanah> Thank you for understanding that
[18:44:23] <devesh> No I understood that, it's hard to judge tone through textual medium, but I think I get your point
[18:44:43] <sumanah> I do think that it would be good for you to try to take a week where you default to NOT escalating any pip questions in IRC
[18:44:54] <sumanah> and instead get used to using the mailing lists
[18:45:11] <sumanah> you can learn what the different communication tools are better at
[18:45:44] <sumanah> if you want to someday be a maintainer or co-maintainer of your own project, you need to have some fluency in those different tools, and be able to help _others_ figure out when to move a conversation from one to another
[18:46:33] <sumanah> ok. So it's now been about half an hour since I started talking with you about this, and I'm going to go do some of my billable work now. This is not any kind of rebuke or criticism, just being clear
[18:46:37] <devesh> haha yes, everything synchronous would be chaos
[18:46:44] <sumanah> thanks for understanding and for being willing to try something new, devesh
[18:47:16] <devesh> No I understand all your points being a project manager. Every project manager has to manage resources well
[18:47:45] <devesh> and help others out on it, which you are doing well.
[19:39:19] <sumanah> toad_polo: "making pip install stuff that fails `python_requires`?" - you mean an issue regarding adding the ability to do that to pip?
[19:40:48] <toad_polo> sumanah: Yeah, there's been some discussion here and there about why pip always hard-fails when you try to install something with the wrong version of Python.
[19:47:15] <devesh> Hi sumanah: I saw that their is also a google groups mailing list https://groups.google.com/forum/#!forum/pypa-dev , Is this also used for the same purpose as the one you shared before?