Log file Viewer
#pypa-dev logs for Tuesday the 24th of March, 2020
[09:44:13] <pradyunsg> EWDurbin: di_codes: ^ do we want the big ML/AI packages to ship their models separately? Like, would doing that affect PyPI?
[10:51:35] <travis-ci> Change view : https://github.com/pypa/pip/compare/d43699b11149...8c95de11ccb6
[13:54:11] <devesh> Can this channel be used for asking questions about PR's, issues etc as well, given that those questions are already asked on github?
[13:54:55] <pradyunsg> devesh: GitHub is likely best, since it allows for longer form discussions and does not need everyone to be online at the same time.
[13:55:51] <devesh> Sure I will do that, so what else is the motive of this channel, what else can be discussed here?
[13:58:19] <pradyunsg> For other folks wondering, devesh recently started contributing to pip, during a sprint conducted by BangPypers (https://www.meetup.com/BangPypers/).
[13:59:34] <devesh> Thanks for the intro @pradyunsg, the contribution was just a doc, but I learned the ways of contributing to open-source :)
[14:01:55] <devesh> I created another PR: https://github.com/pypa/pip/pull/7891 which is a bit better contrib than the previous one
[14:26:57] <McSinyx[m]> I'm writing my proposal about it according to the discussion here yesterday, but I have yet to find the part of code indicating the (assumed) issue
[14:35:06] <pradyunsg> you can see the "flow" of pip install in the pip/_internal/commands/install.py file.
[14:36:36] <McSinyx[m]> I was asking for confirmation that if the download-install stages need decoupled
[14:37:46] <McSinyx[m]> following the flow is like falling down the rabbit hole, I've been through 10 wrapper functions and still haven't find what actually do the downloading
[14:38:18] <pradyunsg> All the downloading happens as part of https://github.com/pypa/pip/blob/master/src/pip/_internal/commands/install.py#L332.
[14:38:18] <pradyunsg> The actual installation happens after https://github.com/pypa/pip/blob/master/src/pip/_internal/commands/install.py#L380.
[14:38:58] <pradyunsg> https://github.com/pypa/pip/blob/master/src/pip/_internal/network/download.py <- code for downloading a file.
[14:44:17] <McSinyx[m]> TBH I'm feeling a bit lost now: there are 2 refactoring direction I can think of
[14:44:30] <pradyunsg> If you 'pip install flask', you’ll notice that there’s a clear stage where all the downloads happen and then a separate step where all the installs happen.
[14:44:46] <McSinyx[m]> techalchemy: that's not blaming, I have too read it now or later; one is to mess with the way resolver
[14:44:55] <pradyunsg> They’re decoupled and the only thing that passes between those two is… basically… a list of what to install. :)
[14:45:48] <McSinyx[m]> pradyunsg: silly me yea the functional try would be much easier then read every thing (-:
[14:46:35] <pradyunsg> techalchemy: you led McSinyx[m] down a path that made them think InstallRequirement was the answer.
[14:47:42] <techalchemy> McSinyx[m], pradyun should just guide you where he thinks your time is best spent anyhow, i am only giving half attention to this which is going to lead you back down another weird road no doubt :p
[14:48:32] <McSinyx[m]> my current concern is here: https://github.com/pypa/pip/issues/825#issuecomment-302020952, which leads to the next approach, to provide a fake file system that let caller waits on the same download
[14:49:21] <McSinyx[m]> techalchemy: I'm overwhelm by the amount of infomation either way no worries
[14:50:46] <McSinyx[m]> the second would be basically sth we can plug into the place of Downloader, but I've noidea if it's possible (for me to do it)
[14:51:58] <McSinyx[m]> techalchemy: can alchemy help in my case, I mean the burn out probably a few days from now d-:
[15:30:46] <McSinyx[m]> I think it'd be better for me to draft a patch to see the posibility of each approach
[15:31:31] <McSinyx[m]> from what I've found, seemingly I'd need to start working from operations.prepare.unpack_url
[16:30:09] <sumanah> hi techalchemy - may I ask you to please merge https://github.com/pypa/pipenv/pull/4145 ?
[16:31:16] <techalchemy> hey sumanah yeah i'll be back on pipenv stuff again this week, had a detour last week without much open source work
[16:47:03] <sumanah> am spreading the word in various places about the pip resolver work and how people can help now https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html
[16:55:20] <sumanah> Great! next techalchemy I see a few people asking about whether particular issues and whether they block things
[16:56:38] <sumanah> techalchemy: a few comments starting at https://github.com/pypa/pipenv/issues/3369#issuecomment-599118982
[16:57:51] <techalchemy> i still need to actually release requirementslib, something was stopping me from doing that i felt
[16:59:00] <techalchemy> I also am pretty sure i made that change somewhere and it was like 4 lines of code
[17:01:44] <sumanah> techalchemy: this is also an issue you added to the release milestone in March 2019
[17:08:57] <sumanah> techalchemy: can you check the settings on https://readthedocs.org/projects/pipenv/ to ensure that it is building from pypa's master?
[17:09:13] <sumanah> because I'm looking at https://pipenv.pypa.io/en/latest/ and it doesn't include the link fixes that you just merged
[17:12:06] <sumanah> techalchemy: while we wait for that - how about you go ahead and make a requirementslib release?
[17:14:48] <sumanah> techalchemy: ok that's weird, if I open in a fresh browser then the links are fixed
[17:16:54] <sumanah> techalchemy: ok! I think you may also need to kick off a build for "stable" as well as "latest"
[17:20:47] <sumanah> techalchemy: and you can merge https://github.com/pypa/pipenv/pull/4160 to fix the GitHub logo in the upper right and update it to go to the right repo. I had ignored the hacks.html file earlier; should not have.
[17:26:45] <devesh> Hi guys, I had a question about the way unit tests work in pip, specifically test_uninstall.py
[17:30:33] <devesh> I was trying to run "pip install -e ." in test_uninstall.py, using the code provided at https://github.com/pypa/pip/blob/master/tests/functional/test_uninstall.py#L528
[17:31:41] <devesh> Specifically `script.pip('install', '-e', '.', expect_stderr=True, cwd=pkg_path)`
[17:32:56] <devesh> When you run "pip install -e ." generally, you get both an *.egg-link and easy-install.pth file
[17:34:29] <devesh> To give more background, this question is in suggestion by a committer for me to create a test case here: https://github.com/pypa/pip/pull/7891#issuecomment-603308975
[17:35:57] <sumanah> hey devesh - that's useful background but I am looking forward to hearing your question :-)
[17:36:35] <devesh> When I run "pip install -e ." generally, I get both an *.egg-link and easy-install.pth file but when I run script.pip('install', '-e', '.', expect_stderr=True, cwd=pkg_path), I just get *.egg-link
[17:38:10] <sumanah> devesh: still not sure what your question is. :-) maybe you want to ask something like "is this a bug?" or "why is this happening?" or "should I try to fix this?" :-)
[17:39:23] <devesh> Different behaviour for the same command, one run on command prompt, and one run via tox and pytest in unit testing
[17:39:36] <sumanah> devesh: thanks for looking into this. remind me - have you ever poked around the pip codebase before? I don't know all the contributors' names.....
[17:39:51] <pradyunsg> I'd suggest checking which version of setuptools you're on currently, and trying the version that was release around the time the issue was filed. It might be a case of different behaviours across different setuptools versions.
[17:42:44] <pradyunsg> devesh: is there a difference in behaviour in the test run, vs running the commands on your terminal/command prompt?
[17:45:28] <sumanah> devesh: welcome to open source :-) I wrote something you might find useful: https://www.harihareswara.net/sumana/2016/10/12/0 on the work of making hypotheses in situations like this. I think you're doing well overall but figure you might find a few tips helpful to improve further!
[17:55:06] <devesh> (venv) $ echo "from setuptools import setup; setup(name='repro', version='0.1dev')" > setup.py
[17:56:11] <devesh> This is the behaviour in terminal @pradyunsg, as you can see, after creating a dummy package and running "pip install -e .", both repro.egg-link and easy-install.pth are present
[17:56:17] <sumanah> btw devesh - the next time you need to paste a bunch of stuff into an IRC conversation you can use a service like https://hastebin.com/ to make a short ephemeral "paste" and then give us the link
[18:18:44] <sumanah> while we wait for pradyunsg -- devesh do you have any ideas about why this is happening?
[18:19:13] <sumanah> and hey techalchemy -- how is the requirementslib stuff going? do you think we should pin to an old version of hypothesis or fix the underlying behavior?
[18:19:53] <devesh> Not really @sumanah , I just recently have started dabbling with pytest, and the unit tests here are at a different level
[18:20:13] <devesh> I am trying out different things and figuring out from print statements and debug logs
[18:22:21] <sumanah> devesh: so pradyunsg mentioned " suggest checking which version of setuptools you're on currently, and trying the version that was release around the time the issue was filed. It might be a case of different behaviours across different setuptools versions." let's start there
[18:25:39] <devesh> And I see that the latest release is 46.1.1 3 days ago as per https://github.com/pypa/setuptools/releases, so I think I should try a newer version
[18:31:13] <devesh> The code is taken from the first few lines of https://github.com/pypa/pip/blob/master/tests/functional/test_uninstall.py#L528
[18:32:04] <devesh> And I used "tox -e py tests/functional/test_uninstall.py::test_uninstall_editable_and_pip_install_and_easy_install_remove -s" to run this particular function
[18:38:42] <devesh> One thing I just realised that I should be using "script.site_packages_path" which points to site-packages inside virtualenv of pytest
[18:39:10] <devesh> Instead of "script.site_packages", which points to site-packages inside virtualenv where pip is being used, is that correct?
[18:41:20] <devesh> script.site_packages_path is /private/var/folders/xg/blp845_s0xn093dyrtgy936h0000gp/T/pytest-of-devesh/pytest-1/test_uninstall_editable_and_pi0/workspace/venv/lib/python3.7/site-packages
[18:43:16] <pradyunsg> Hmm... Looks like site_packages_path is the one for the virtualenv that you want to use.
[18:44:19] <devesh> Yes, so I am now able to see the same behavior in terminal as well as through pytest
[18:44:49] <pradyunsg> ssurbhi, gutsytechster: ^ this could be another thing worth considering to include in your GSoC proposal - working on cleaning up PipTestEnvironment.
[18:46:19] <pradyunsg> ssurbhi, gutsytechster: actually, cleaning up the situation with scripttest + PipTestEnvironment would be *awesome*! About as awesome as getting coverage reports working. 🙃
[18:48:10] <pradyunsg> devesh: yea. Improving it has been one of the things we've been chipping away at but it's tough work.
[18:48:37] <pradyunsg> Especially when the entire codebase has a lot of burden from technical debt in general. 🤷🏻♂️
[18:48:49] <devesh> Making the actual code change was easy, now writing the unit test for it is where I am facing challenges
[18:49:26] <pradyunsg> Okay, so this is good to know. I wasn't sure about the nuance of site_packages vs site_packages_path, but that's nice to know.
[18:50:27] <devesh> Like how the two names are similar, but point to two entirely different things, and can be better named?
[18:51:35] <pradyunsg> That way we don't lose the knowledge we just gained in the past few minutes. :)
[18:57:26] <devesh> I have a dilemma, the fix I have written makes sure that the uninstall of a package goes through even if easy-install.pth doesn't exist,
[18:58:02] <devesh> But if I remove this file while writing a unit test, I also lose the path of pip, since easy-install.pth is being used for it by pytest to run the uninstall command
[19:02:09] <pradyunsg> devesh: I'm not sure what the fix would be for that, and it's been a pretty long day for me so I'm going to head to bed now.
[19:48:38] <devesh> @pradyunsg: https://github.com/pypa/pip/issues/7893 , the issue w.r.t what we discovered in test_uninstall.py
[20:00:16] <sumanah> techalchemy -- how is the requirementslib stuff going? do you think we should pin to an old version of hypothesis or fix the underlying behavior?
[21:26:26] <sumanah> techalchemy: should I be looking at https://github.com/sarugaku/requirementslib/branches ?
[21:27:05] <sumanah> techalchemy: just in case you get hit by a bus or a virus, could I ask you to push a WIP branch too?
[21:30:03] <sumanah> and then Bruce Willis leaps out just in time to jump into your window before it's destroyed
[21:31:26] <sumanah> maybe just before he grabs your hard drive and runs away with it, he sees you and quips, "Door-to-door service."
[22:05:50] <sumanah> techalchemy: ok! so it's past 6pm ET - do you have time to talk about any of the other items on the March release TODO list? or do you need to go?
[22:08:08] <techalchemy> i've been kind of all over the place today as i'm sure you can tell, i keep getting pulled in 20 different directions
[22:18:09] <techalchemy> everything else is good on requirementslib afaik, i think i can release there
[22:18:24] <sumanah> so the "Review requirements-lib issues for other pipenv release blockers" item - you've done that? in that case I will check it off
[22:18:45] <sumanah> the issue you filed, "We shouldn't install upgraded versions of test dependencies in CI", is the only blocker to a requirementslib release
[22:19:14] <sumanah> there are a few more items: "Merge https://github.com/pypa/pipenv/tree/feature/vendor-update to address missing dependencies/imports" and "make new releases" of pythonfinder & vistir
[22:22:36] <sumanah> techalchemy: and/or a release of requirementslib since I see you have now merged in #215
[22:23:38] <travis-ci> Change view : https://github.com/pypa/twine/compare/5482d6877df0cd90545c4708e5abe13b359f8a08...f7402e0d2e1c1e6ecde61dc8bcba515b807a48a0
[22:24:56] <techalchemy> sumanah, sorry was just doing a dry run with the release generation script and the news file generator syntax changed a tiny bit
[22:25:26] <techalchemy> i just pushed the fix to master so i'll release that shortly, release just requires pushing a tag
[22:25:37] <travis-ci> Change view : https://github.com/pypa/pip/compare/145c189a4998b52121c09ea669f7463dad03e154...8c95de11ccb6ce75f67835766007d36f4ce6c941
[22:43:27] <sumanah> techalchemy: I hope we are able to get a prerelease out by the end of the month