pmxbot IRC Log Viewer

[00:23:01] <njs> di_codes: "type to narrow the list" is also useful for huge lists like classifiers

[02:41:43] <techalchemy> altendky: i've been pushing some stuff to your branch

[02:41:45] <techalchemy> hope you don't mind

[02:42:00] <techalchemy> just to get your test passing

[02:43:12] <altendky> techalchemy: sure, thanks. I got off on other stuff this afternoon

[02:44:06] <techalchemy> no worries the failures are silly

[02:44:49] <techalchemy> i think the test was failing on windows due to the pipfile only containing a newline in it, or something, hard to say

[02:49:05] <altendky> techalchemy: a couple changes don't totally make sense but I'm on my phone and it's late. I'll take a look tomorrow in more detail

[02:49:47] <techalchemy> I'm kind of taking random guesses myself

[02:50:25] <techalchemy> I just want to get this merged asap, if I can't get it sorted with the last commmits I'll just branch from your stuff and see

[02:52:00] <altendky> The mtime checks were intentional just to make sure we really did actually write something, otherwise the test is pointless. Sadly I had that issue in the beginning

[02:54:57] <vivsoni> i tried to update one python package in pypi

[02:55:35] <vivsoni> from https://pypi.org/project/python-3parclient/4.2.3/ to https://pypi.org/project/python-3parclient/4.2.4/

[02:55:46] <vivsoni> from python-3parclient 4.2.3 to 4.2.4

[02:55:55] <vivsoni> the project description was missing in the later uploaded package

[02:57:24] <vivsoni> can someone please let me know what was missed

[03:04:10] <altendky> techalchemy: https://github.com/pypa/pipenv/pull/2069/files#diff-9a741909c8cb9fa73daacdae0ffe1aacR95 It's not clear why this needed to be changed. The idea there was simply to set the neediness in the file. If the initial creation was wrong it should be changed I think, not the rewriting of it

[03:04:29] <altendky> *Set the newlines in the file

[03:06:36] <altendky> vivsoni: got a link to the source?

[03:08:29] <altendky> vivsoni: at least now you don't have a long description specified https://github.com/hpe-storage/python-3parclient/blob/master/setup.py

[03:09:32] <altendky> vivsoni: I'm no expert at this but here's my example https://github.com/altendky/altendpy/blob/develop/setup.py#L14

[03:09:56] <altendky> vivsoni: do not the new option after that (new to me anyways) to specify the format of the data

[03:10:04] <altendky> Do note...

[03:10:48] <altendky> Anyways, off to bed here. Back around 9 us Eastern (if the kids sleep in)

[03:10:58] <vivsoni> altendky: ok.. Thanks :)

[03:22:44] <techalchemy> altendky: the error makes no sense to me

[03:23:41] <techalchemy> that was just a change I attempted in order to make it write the lockfile differently

[03:23:49] <techalchemy> \n' == '\n'<- that's the error

[03:24:29] <techalchemy> i have basically no clue how to interpret that, the first value is 'written newlines' aka 'f.newlines' and it only happens on windows

[05:20:43] <vivsoni> parameter to set proxy in pypirc ? pls

[12:05:27] <mgedmin> techalchemy: that looks like somebody printed a literal \r and it moved the cursor to column 1, overwriting the beginning of the error message

[12:05:38] <mgedmin> \r\n and windows, *sigh*

[12:05:47] <mgedmin> but how come \n is escaped and \r isn't?

[13:20:07] <altendky> mgedmin: sounds like a good guess. Might be something about the AppVeyor web ui if it hasn't been tried locally. I'll get to it in a bit

[14:53:50] <techalchemy> altendky: I've spent a chunk of time on it and I'm still not sure what's up, I think it relates to how python 2 handles the newline argument in the io library on windows

[14:54:11] <altendky> techalchemy: i was just getting setup again to look at this

[14:54:26] <techalchemy> you were right to question my change about the project instantiation though

[14:54:31] <techalchemy> completely unnecessary

[14:55:05] <altendky> techalchemy: first i'm going to check if that error message shows up right locally :]

[14:55:31] <altendky> techalchemy: it's interesting it only fails for one of the py2 jobs though

[14:57:57] <techalchemy> altendky: it will only show up if you run in py27

[14:58:04] <techalchemy> it was hard to make it happen

[14:58:26] <altendky> techalchemy: and only in the TEST_SUITE=install job https://ci.appveyor.com/project/pypa/pipenv/build/1.0.843

[14:58:40] <techalchemy> right that's the only place where it gets run

[14:59:10] <altendky> oh, ok. i hadn't dug into the meaning of that setting

[14:59:46] <techalchemy> I am under the impression that py27 ignores the 'newline' argument to io.open on windows

[15:00:05] <techalchemy> and that you have to write in binary mode

[15:00:17] <techalchemy> which i messed with but caused even more problems

[15:00:27] <techalchemy> so I'm considering just adding a skip for windows/py27

[15:04:34] <techalchemy> heading to the office brb

[15:10:02] <mgedmin> maybe make the test assertion easier to read? e.g. assert written_newlines == u'\n', "expected '\\n', got %r" % written_newlines

[15:10:17] <mgedmin> (and also file a pytest bug about messing up assertion messages when one of the string contains a \r)

[15:10:39] <altendky> mgedmin: i was first going to see what came out locally.

[15:11:02] <mgedmin> hey I can repro the pytest bug on linux

[15:11:13] <altendky> mgedmin: alrighty then :]

[15:11:23] <mgedmin> https://hastebin.com/uhuqaregas.py

[15:15:25] <altendky> mgedmin: thanks

[15:50:46] <altendky> mgedmin: added the message to the assert to compensate. thanks

[17:08:32] <altendky> mgedmin: pytest also fails to handle the newline strings properly in the parametrization info https://ci.appveyor.com/project/pypa/pipenv/build/1.0.863/job/ypvgpjv7f64p6si6#L176

[17:10:20] <altendky> techalchemy: something is totally messed up on my system. i can't get consistent results from shell and pycharm and i seem to have to reinstall to get changes and... now it's working for the pipfile but not the lockfile. *sigh*

[17:10:57] <techalchemy> altendky: I'm having similar experiences, i made a ramdisk and switched to cmd.exe and can replicate the failures consistently as long as I install into a ramdisk

[17:11:47] <techalchemy> I think we may be using 'newline' incorrectly

[17:11:50] <altendky> techalchemy: i'm not a huge fan of this running pipenv tests via pipenv thing (not that i've isolated that as the cause or anything). or even just installing pipenv test dependencies via the pipenv under test

[17:12:05] <altendky> techalchemy: you saw my example in the issue? it seemed to work fine

[17:12:22] <altendky> hmm, lemme try it in cmd.exe

[17:12:31] <techalchemy> I don't think i could replicate it in powershell

[17:12:38] <techalchemy> because reasons

[17:13:23] <altendky> techalchemy: i get the same output from blah.py in cmd.exe so i suspect the underlying functionality is fine.

[17:14:09] <altendky> techalchemy: is pipenv install --dev going out and downloading pipenv because the other bug hasn't been merged relating to the -e . stuff still getting a lockfile entry?

[17:14:09] <techalchemy> feel free to revert any changes I made, I pushed a branch to the main repo with them

[17:14:22] <techalchemy> ...

[17:14:24] <altendky> techalchemy: already did a couple of them :]

[17:14:33] <techalchemy> uh

[17:14:52] <altendky> techalchemy: something like that would explain the goofy nature of our attempts to recreate

[17:15:10] <techalchemy> 'that is definitely impossible'

[17:15:27] <techalchemy> aka I will check because it sounds likely

[17:15:28] <altendky> i'm adding `raise Exception('hah')` before we write the file (in pipenv proper, not the tests) and... it still succeeds at running!

[17:15:49] <altendky> and then i mess around and finally it starts failing

[17:17:00] <techalchemy> that's likely just because you need to either set PYTHONDONTWRITEBYTECODE

[17:17:17] <techalchemy> or or you need to reinstall whenever you make updates

[17:17:19] <techalchemy> idk :p

[17:17:23] <techalchemy> On input, if newline is None, universal newlines mode is enabled. Lines in the input can end in '\n', '\r', or '\r\n', and these are translated into '\n' before being returned to the caller.

[17:17:33] <altendky> shouldn't need either. i never do either for my normal dev and basically never have issues

[17:18:30] <altendky> techalchemy: sure, i'm not actually checking the newline characters myself. and my test runs fine in windows py2.7 cmd.exe. unless you see an issue with my 10 line test script for that?

[17:19:38] <techalchemy> altendky: because you are running the tests out of the project directory

[17:20:01] <techalchemy> (which is the reason people started making src/ base folders)

[17:20:44] <altendky> techalchemy: i use src/ myself. but i don't follow. i'm modifying pipenv, not it's tests, and not seeing the proper effects of those changes (like it crashing). at least not seeing them consistently

[17:21:30] <altendky> like, i added `raise Exception('write_toml')` as the very first line in write_toml and the tests didn't fail on the assertion about the return value being 0

[17:21:40] <altendky> and then eventually it did

[17:21:47] <techalchemy> that's cached python

[17:22:05] <altendky> i'm setting the PYTEST_ADDOPTS: '--cache-clear'

[17:22:27] <techalchemy> that's only the test cache

[17:22:27] <techalchemy> del /S *.pyc

[17:22:32] <altendky> i just don't get why it happens here and not in my other two years of python dev

[17:23:31] <techalchemy> this is an edge case related to the backported version of NamedTemporaryFile (which I backported, likely somewhat sloppily, possibly why it wasn't fully backported to begin with)

[17:23:48] <techalchemy> how are you running the tests?

[17:24:06] <techalchemy> it's possible there's some secret step nobody ever mentioned

[17:24:07] <techalchemy> lol

[17:24:21] <techalchemy> both me and uranusjr can replicate the issue consistently

[17:24:38] <altendky> at this point i'm `.venv\scripts\pipenv run pytest tests/integration/test_project.py::test_maintain_file_line_endings` right now in the shell

[17:24:48] <altendky> similar in pycharm except specifying the pipenv-script.py

[17:24:59] <altendky> i've done various things trying to get it to respond correctly

[17:25:22] <techalchemy> idk anything about pycharm really but here's essentially what I'm doing, I'll ask him what he's doing

[17:25:43] <altendky> techalchemy: i do see the issue.

[17:25:54] <altendky> but i can't consistently modify code and have those changes show up

[17:26:02] <techalchemy> just pipenv sync

[17:27:08] <altendky> i'm really not a fan of needing pipenv to install itself properly to run tests. i mean sure, for tests to pass it should be able to install itself properly, but that just mixes too much together i think

[17:27:56] <techalchemy> i mean how else would you propose that pipenv be tested?

[17:29:04] <techalchemy> also I'm not sure how that relates to the issue anyway

[17:29:20] <altendky> i get the desire to not use 'classic' mechanisms and i have not considered the details of an alternative. so, this isn't a complaint. just an observation that it's a bad situation. if the .+remote-url in pipfile.lock bug is the cause, then it's just wasted several hours of both our time.

[17:29:47] <altendky> well, even if just the cause of confusion, even if unrelated to the underlying issue

[17:30:15] <techalchemy> what remote url bug? are you talking about the one that double-locks editable installations?

[17:30:27] <altendky> yeah

[17:30:38] <altendky> i'm going to checkout a fresh copy

[17:30:51] <techalchemy> did you re-lock or something?

[17:30:53] <altendky> pipenv sync didn't make a difference, still not running my added code

[17:31:06] <techalchemy> just checkout the lockfile from master, you shouldn't ever have to relock

[17:31:09] <altendky> my issues with that didn't ever involve having run lock at all

[17:31:26] <techalchemy> the lockfile doesn't have 2 copies of pipenv in it

[17:31:32] <techalchemy> because we call the local version pipenv in the pipfile

[17:31:43] <altendky> the CONTRIBUTING.rst says to `pipenv install --dev`

[17:31:49] <altendky> perhaps it should say sync?

[17:32:02] <techalchemy> it shouldn't.

[17:32:17] <techalchemy> it says what it should say, I am merely giving you an alternative version because you have already installed it.

[17:32:35] <altendky> starting fresh here

[17:34:00] <altendky> if i follow CONTRIBUTING.rst and then modify a source file (not a test), what should i do to run the tests against that modification?

[17:35:11] <techalchemy> depends if you have PYTHONDONTWRITEBYTECODE set

[17:35:23] <techalchemy> usually nothing since it's installed in editable mode

[17:35:37] <altendky> i do not have that set

[17:35:45] <techalchemy> for testing and since it's a virtualenv i often just rerun `pipenv install --dev` anyway

[17:36:13] <techalchemy> then you have cached bytecode after you run things and you need to delete it with `del /S *.pyc`

[17:37:21] <altendky> so i edit, `pipenv install --dev`, `del /S *.pyc`, and then `pipenv run pytest tests` to test my edits

[17:37:45] <techalchemy> i'd delete cached stuff first, but sure

[17:38:13] <techalchemy> i'm running this >pipenv run pytest -n auto -v --ignore="./pipenv/patched" --ignore="./pipenv/vendor" -k test_maintain_file_line_endings

[17:39:05] <altendky> i edit, `del /S *.pyc`, `pipenv install --dev`, and then `pipenv run pytest -n auto -v --ignore="./pipenv/patched" --ignore="./pipenv/vendor" -k test_maintain_file_line_endings` to test my edits

[17:39:19] <techalchemy> sure, if that works, it works for me

[17:40:26] <techalchemy> if you see an issue in the contributing guideline tell me

[17:40:32] <altendky> no tests ran, i'll specify the full path as before

[17:41:39] <techalchemy> oh... that's a different problem

[17:42:30] <techalchemy> are you in the pipenv project dir / do you have it installed?

[17:43:44] <altendky> i thought i followed the instructions. i am in the pipenv project dir, i `python setup.py develop`ed, i `pipenv install --dev`ed

[17:44:41] <techalchemy> hm, me and uranusjr both use custom activated virtualenvs

[17:44:45] <techalchemy> lol

[17:46:55] <altendky> ugh, cloned and didn't checkout the branch. *sigh*

[17:47:07] <altendky> fresh again

[17:47:17] <techalchemy> I basically do this:

[17:49:20] <techalchemy> imdisk -a -s 4G -m R: -p "/FS:NTFS /Y" && set RAM_DISK=R:\\ && set PYPI_VENDOR_DIR=".\tests\pypi\" && C:\Python27\Scripts\virtualenv.exe R:\.venv && R:\.venv\Scripts\activate.bat

[17:50:05] <techalchemy> then> pip install -e . && pipenv install --dev && pipenv run pytest -n auto -v --ignore="./pipenv/patched" --ignore="./pipenv/vendor" tests/

[17:50:08] <altendky> well, maybe i'll switch to a ramdisk. is that normal dev for you? or just the thing that happened to show the issue here?

[17:50:19] <techalchemy> that's what I always do

[17:50:23] <techalchemy> both on windows and linux

[17:50:29] <techalchemy> tests run way faster

[17:50:49] <altendky> so you do use pip, not `python setup.py develop`

[17:50:59] <techalchemy> yeah

[17:51:10] <techalchemy> to be fair i've never read our contributing docs so they are quite possibly wrong

[17:51:39] <altendky> at the very least they don't consider windows a first class citizen (i think that's a proper doc reference) :]

[17:52:32] <techalchemy> lol

[17:52:52] <techalchemy> that's unfortunate, me and uranusjr both do aa significant amount of dev work on windows

[17:52:58] <techalchemy> I'm like 50/50 and he's like 75% at least

[18:09:56] <altendky> techalchemy: permission denied with the ramdisk stuff when i `pipenv install --dev`... anyways. it's just the lockfile failing, not the pipfile. so it doesn't seem like an issue with the concept of using newlines and io.open. also, that is the result i get without the ramdisk so i'll just deal with slow for now i guess.

[18:10:13] <techalchemy> altendky: it's nto a conceptual issue

[18:10:41] <techalchemy> I think we have it fixed

[18:10:46] <techalchemy> er crap

[18:32:13] <sumanah> https://blog.github.com/2018-05-02-issue-template-improvements/ might help us

[18:35:57] <sumanah> separately: https://github.com/qutebrowser/qutebrowser/pull/3884/commits "Flask 1.0 is out, pip made breaking changes, warehouse is a thing, new requests soon. So much fun in python world lately." can read this a few ways, but I read it as positive

[18:39:53] <altendky> techalchemy: if i adjust my example to use os.open() and pass the file descriptor to io.open() rather than the path, it works differently. https://gist.github.com/altendky/11e7fe3c662ad2fdf482eb326f11428e (in case that's news... it sounds like you may be way ahead of me)

[18:40:05] <altendky> presumably i've got some flags issues

[18:40:14] <techalchemy> lol

[18:40:17] <techalchemy> tha tis news altendky

[18:40:29] <techalchemy> I actually have flag issues related to how we handle opening files on python 2

[18:41:51] <techalchemy> OH!

[18:42:03] <techalchemy> this made me realize what it is

[18:49:22] <altendky> techalchemy: https://gist.github.com/altendky/11e7fe3c662ad2fdf482eb326f11428e all my little demo needed was os.O_BINARY. it looks like we don't have that in our case.

[18:49:24] <altendky> i think

[18:49:39] <altendky> afaik python always uses binary at the os level and implements text stuff itself

[18:50:49] <techalchemy> yes

[18:50:52] <techalchemy> only for python 2

[18:51:04] <techalchemy> and only for when your newlines dont match the system defaults and only when writing

[18:51:14] <techalchemy> uranusjr put up a PR with a fix possibly

[18:55:04] <sumanah> toad_polo: ^ hey, in case you want to edit your proposal a bit, and if you have any suggested time periods for those stages, add them?

[18:55:08] <sumanah> unsolicited advice

[18:56:37] <toad_polo> sumanah: I left out time periods because I'm not sure how aggresive the warehouse devs want to be. I'm not even sure the order needs to be what I laid out there (2 & 3 can be swapped pretty easily)

[19:01:37] <sumanah> toad_polo: got it!

[19:02:13] <toad_polo> sumanah: Updated my proposal with a general time frame.

[19:02:50] <sumanah> rockin'.

[19:12:06] <techalchemy> altendky: https://ci.appveyor.com/project/pypa/pipenv/build/1.0.875

[19:12:09] <techalchemy> gotta run for a bit bbs

[20:58:51] <sumanah> np

[21:46:49] <techalchemy> altendky: that pytest bug...

[22:04:41] <altendky> techalchemy: at least we got there

[22:04:55] <techalchemy> yeah looking it all over now

[22:05:10] <techalchemy> I'm just skeptical of removing the ability to toggle off binary flags for python 2

[22:05:17] <techalchemy> its' fine for python 3 because that's how it's handled anyway

[22:06:30] <techalchemy> but I was experimenting earlier and any way of toggling binary writes besides this approach requires passing +b as a mode

[22:06:40] <techalchemy> so I think this is the way

[22:18:40] <techalchemy> ok altendky I looked really hard and I could only find 2 incredibly nitpicky comments to make about this one, reviewed and then I think we're good

[22:19:57] <techalchemy> thanks again for all your hard work on this, it kind of inspired a cascade of other changes that snowballed this one into a more complicated situation

[22:31:01] <altendky> techalchemy: we're using the pathlib backport already?

[22:33:45] <techalchemy> are we not?

[22:34:05] <techalchemy> I mean it's already installed, in setup.py, etc

[22:34:15] <techalchemy> and imported in project.py

[22:34:54] <techalchemy> unless past-me stealth imported it in my fork of your pr

[22:39:15] <altendky> I'll get this later this evening

[22:39:54] <altendky> Or you are welcome to, I don't think I have an issue with the changes requested

[22:40:28] <techalchemy> awesome, I'm gonna merge it for now just so we can start merging other stuff

[22:40:30] <techalchemy> thanks again

[22:56:49] <altendky> techalchemy: thank you

[22:57:14] <techalchemy> this was one of the most bizarre and annoying issues

[22:57:15] <techalchemy> lol

[22:58:34] <techalchemy> but your work + the simultaneous work we had going on with regard to the timing of lockfile deletion uncovered the underlying broader issue that was preventing us from accurately implementing the selective upgrade strategy

[22:59:31] <techalchemy> which led to the atomic_open_for_write implementation which couldn'

[23:00:03] <techalchemy> t accommodate your PR since it relied mkstemp instead of io.open, so that inspired me to backport the new features from NamedTemporaryFile

[23:00:12] <techalchemy> so that's how we wound up here, in case you didn't follow the entire train

[23:02:50] <altendky> I didn't, but I know this stuff often blows up... :]

[23:04:11] <altendky> And regardless you get credit for responsiveness. I've got a twisted pr been open and untouched for a couple weeks.

[23:09:01] <runciter> altendky, sorry, i've been busy :(

[23:09:32] <runciter> hope to get to it this weekend

[23:10:12] <altendky> runciter: it's a core thing and it's been waiting 7 years already... :] I didn't expect it to be quick

[23:12:24] <techalchemy> 7 years?

[23:12:25] <techalchemy> lol

[23:16:13] <altendky> techalchemy: yeah, started way back then. A branch was made... Then a couple weeks ago I ran into the issue and got the branch caught up.

[23:16:32] <techalchemy> Tests green, btw. Merged.

[23:17:09] <altendky> Maybe I take on that pypiwin32 thing next

[23:17:26] <techalchemy> I have been bad about responding to their questions

[23:17:41] <techalchemy> I think they asked me what they are doing wrong and I told them to stop importing windows things in their setup.py

[23:17:52] <techalchemy> without at least checking if the user is actually running windows

[23:18:14] <techalchemy> but their response was kind of confused, like where else would we do something like this

[23:20:10] <altendky> I went and removed my dependency on it in favor of 'twisted; sys_platform != "Windows"', 'twisted[windows_platform]; sys_platform == "Windows"', (iirc) but no help. I presumably already forgot the details of your explanation :[

Log file Viewer

Help | Karma | Search:

#pypa-dev logs for Thursday the 3rd of May, 2018