pmxbot IRC Log Viewer

[00:26:52] <kober> ionelmc: ok, before it seemed like it was an issue with me generating the file too late. Now I see it in my package: pollweb-0.0.1.dev0/assets/manifest.json but when I `pip install` that package in site-packages/pollweb/ there is no assets folder

[00:26:58] <kober> and no manifest.json file in there anywhere

[00:28:58] <kober> in the sdist its at the same location as the setup.py

[00:31:56] <ionelmc> kober: include_package_data=True

[00:32:12] <ionelmc> MANIFEST.in controls sdist, not bdist or installs

[00:32:22] <ionelmc> by default that is

[00:32:44] <kober> I will try that, thank you!

[00:35:47] <kober> Didn't seem to help any, the manifest.json doesn't end up in the package anywhere

[00:36:33] <kober> I'm installing via `pip install dist/pollweb-0.0.1.dev0.tar.gz`

[00:37:17] <kober> Interesting. Doesn't look like it made it into the tar with that flag either like it was before

[00:37:49] <ionelmc> kober: remove all egg-info/build dirst

[00:38:21] <ionelmc> don't try packaging changes in a dirty place

[00:41:12] <kober> ok, so I see it when I generate the sdist: hard linking assets/manifest.json -> pollweb-0.0.1.dev0/assets and in the tar ball: pollweb-0.0.1.dev0/assets/manifest.json

[00:41:45] <kober> but a pip install doesn't get it there

[00:42:32] <ionelmc> kober: show your setup.py and file layout

[00:58:18] <kober> ionelmc: https://paste.fedoraproject.org/314307/45368313/ the setup.py

[01:00:58] <kober> files: https://paste.fedoraproject.org/314308/68327614/

[01:02:06] <kober> I basically need the `smassets` folder to show up as package data for `pollweb` so that I can read it from python

[01:03:15] <kober> or at least `manifest.json` from the smassets folder

[01:03:27] <kober> Its generated from an external command though

[01:04:09] <ionelmc> kober: your build_js command seems wrong

[01:04:13] <ionelmc> it wrong with wrong cwd

[01:04:35] <ionelmc> it need to be in the build dir (you should have some attribute for it on the command instance)

[01:04:59] <ionelmc> s/wrong/runs/

[01:06:12] <ionelmc> kober: here's a correct example: https://github.com/pytest-dev/pytest-cov/blob/master/setup.py#L30-L35

[01:06:13] <kober> It seems to work because manfiest.json ends up in the tar ball but I'm willing to change it

[01:07:02] <ionelmc> i think you have some sideeffects there

[01:07:11] <ionelmc> eg, second run it will work

[01:07:40] <ionelmc> on the other hand, pip don't run `setup.py install` two times

[01:07:42] <ionelmc> so there's that

[01:10:01] <kober> Oh, so what you are saying is this should be build command not a sdist command?

[01:11:05] <ionelmc> i dunno

[01:11:18] <ionelmc> i can never remember how the whole thing works :)

[01:11:43] <ionelmc> every time i have to trace distutils/setuptools to figure out how the damn thing works

[01:12:00] <ionelmc> kober: eg: https://pypi.python.org/pypi/hunter

[01:12:22] <ionelmc> actually that's how i figured out how to do the pth install trick

[01:13:25] <ionelmc> kober: i'm not sure sdist supports with subcommand thing

[01:13:31] <kober> Does look like this only works the second time because I cleaned everything and I see its generated the dynamic MANIFEST.in file without my stuff: https://paste.fedoraproject.org/314310/14536840/

[01:13:32] <ionelmc> i might be wrong tho

[01:13:45] <kober> The command definitely gets triggered though

[01:13:53] <ionelmc> hm

[01:14:03] <kober> I see it generate all the files

[01:14:22] <ionelmc> kober: it is run too late, after files have been collected

[01:14:36] <kober> What I need is a command that will run during sdist so that my package has the manfiest.json shipped with it

[01:14:36] <ionelmc> i suspect

[01:14:54] <ionelmc> kober: why don't you override run in the sdist command

[01:15:15] <ionelmc> or try having build_js before all the other sdist subcommands

[01:18:06] <kober> Whatever reads manifest.in happens too soon for this command at all

[01:18:10] <kober> reading manifest template 'MANIFEST.in' warning: no files found matching 'smassets/pollweb/manifest.json'

[01:18:30] <kober> it reads that before triggering, even if build_js is the first subcommand

[01:18:36] <kober> let me try overriding run

[01:19:58] <kober> That seems to be triggering early enough

[01:20:15] <kober> So now I'll be back to the original issue which is why the sdist with manifest.json added doesn't work

[01:26:29] <ionelmc> kober: elaborate, what doesnt work

[01:26:44] <kober> ionelmc: yeah, so now it happens every time (generates the file before sdist), but still doesn't get installed

[01:26:57] <kober> ls ~/venvs/tmp-a671c8fc0dfdaa39/lib/python3.5/site-packages/pollweb/

[01:26:58] <kober> __init__.py __pycache__ templates

[01:27:25] <ionelmc> show your setup.py again

[01:27:59] <ionelmc> and a `unzip -l thesdist.zip`

[01:28:12] <kober> In the tar ball generated there is the folder/file I want: pollweb-0.0.2.dev0/smassets/pollweb/manifest.json

[01:29:30] <kober> ionelmc: here is the setup.py https://paste.fedoraproject.org/314313/68500414/

[01:30:01] <ionelmc> and you don't have it?

[01:30:07] <ionelmc> i don't understand the prbolem

[01:30:35] <kober> ionelmc: here is the file list: https://paste.fedoraproject.org/314314/14536850/

[01:31:27] <ionelmc> kober: create smassets/__init__.py

[01:31:59] <ionelmc> include_package_data=True only works with packages

[01:32:20] <kober> ahh, that'll be a problem since smassets isn't generated by me. I guess I could just copy the json file into my package though

[01:32:24] <kober> and then not have to worry about it not being there

[01:32:30] <ionelmc> do it

[01:33:01] <ionelmc> one has to wonder why you're using npm when you have python tho :)

[01:34:16] <ionelmc> it's like shopping for pliers in the grocery store

[01:36:14] <kober> front-end guy has this whole workflow for generating his assets and everything and I don't want to break him. I just need the manifest file in python so I can decache

[09:01:26] <domenkozar> is Jason R Coombs around here?

[10:38:16] <ionelmc> /msg pmxbot !help

[10:38:36] <ionelmc> !seen jaraco

[10:38:37] <pmxbot> I last saw jaraco speak at 2016-01-18 21:59:17+00:00 in channel #dcpython

[10:39:46] <ionelmc> domenkozar: well, that would be him ^

[10:40:14] <domenkozar> ionelmc: thanks

[13:01:58] <jml> does current pip work w/ 3.2? https://travis-ci.org/testing-cabal/testtools/jobs/104124560 seems to indicate otherwise

[13:02:03] <jml> although it might be that I'm doing something wrong

[13:02:46] <mgedmin> if by "current pip" you mean 8.0, then no, it does not work with 3.2

[13:03:05] <jml> otoh, that's using pip 6.0.7

[13:03:27] <mgedmin> oh, the "UserWarning: Support for Python 3.0-3.2 has been dropped. Future versions will fail here." comes from the vendored pkg_resources

[13:03:47] <mgedmin> jml, look closer, that build job uses pip 6.0.7 to pip -U pip

[13:03:56] <mgedmin> and it gets 8.0.2

[13:04:02] <jml> ah, I see.

[13:06:13] <ravenx> anyone have success moving from virtualenv to docker?

[13:06:49] <ravenx> right now, what I don't like is that, everytime I run a build/test on Jenkins, pip does the whole song and dance of rebuilding the virtualenv, building/downloading/compiling dependencies

[13:06:58] <ravenx> I would like something that only does it when it detects a change

[13:07:05] <mgedmin> wheel caching could help

[13:07:30] <mgedmin> actually, if you don't ask jenkins to wipe the workspace, it'll reuse the existing virtualenv and won't have to download/compile/install anything

[13:08:49] <ravenx> mgedmin: really? even if, in my Jenkins buildscript (shell option) i have a "virtualenv test; source test/bin/activate", Jenkins will be smart enough to skip the virtualenv step, and go right to source?

[13:09:33] <mgedmin> it won't skip steps

[13:09:56] <mgedmin> virtualenv test will maybe overwrite the test/bin/python with another copy of itself, but it won't clear the site-packages

[13:10:20] <mgedmin> any subsequent pip install will find existing installed packages and won't have to reinstall them

[13:10:39] <mgedmin> the risk here is when you remove a dependency from your setup.py/requirements.txt but of course it remains installed in the existing virtualenv

[13:10:52] <mgedmin> and maybe your code still relies on imports from that dep, but you won't notice it's missing because tests won't fail

[13:10:53] <ravenx> mgedmin: true. but that hoepfully shouuuuuudln't cause much of a problem.

[13:11:05] <mgedmin> there are solutions for this (e.g. pip-sync from pip-tools)

[13:11:09] <jml> or nix ;)

[13:11:42] <jml> or possibly bazel.io

[13:14:19] <ravenx> cuz right now i have a feeling that the reason my jenkins is crashing every other day is that it is getting pounded by doing pip work and not the unittests itself lol

[13:16:35] <mgedmin> crashing? how crashing?

[13:17:28] <ravenx> sometimes when something is building, the entire website just halts and refuses to go back up

[13:18:20] <mgedmin> can you ssh in during that time? do you have any kind of cpu/memory/swap/disk io monitoring graphs for the time (e.g. collectd)?

[13:18:44] <ravenx> i don't have collectd, most i have is a nagios thing for that server.

[13:18:53] <ravenx> though i do have a feeling that it could be a lack of ram.

[13:19:05] <ravenx> I read somewhere that Jenkins is ram intensive.

[13:19:21] <mgedmin> are you the one who complained a few days ago about jenkins java processes eating all possible CPU cores while running jobs, to the point where it wouldn't respond to http requests?

[13:21:15] <ravenx> that was not me

[13:21:19] <ravenx> but now that you re-word it that way

[13:21:23] <ravenx> that sounds like the problem i am having right now

[14:42:56] <dstufft> jml: right, no 3.2 in pip 8+, but you can either ``pip install pip<8`` or ``curl https://bootstrap.pypa.io/3.2/get-pip.py | python``

[14:57:07] <jml> dstufft: thanks

[15:20:18] <jml> TIL: pip uses the 'PIP_VERSION' environment variable

[15:31:12] <dstufft> jml: that's a bit of a bug

[15:31:15] <dstufft> sort of

[15:31:37] <dstufft> we have a system that automatically translates any CLI flags to environment variables and to config files

[15:31:45] <dstufft> and --version is a cli flag and gets translated

[15:32:55] <jml> dstufft: Ah, I see. I was just confused for a while as I got "ValueError: invalid truth value '<8'" errors when trying to restrict my pip version

[15:33:14] <dstufft> yea, you're not the first to run into it

[15:33:16] <jml> e.g. "maybe it's some weird shell escaping thing, trying to read from fd 8?!"

[15:33:29] <dstufft> it doesn't really make sense for --version to be in the config or env variables

[15:33:35] <dstufft> it just is because it's automatic

[15:40:08] <jml> now that I've restricted pip, I'm getting setuptools errors: https://travis-ci.org/jml/testtools/jobs/104659475 AttributeError: 'module' object has no attribute 'SourceFileLoader'

[15:40:20] <jml> I've tried restricting to <19.5, but that doesn't seem to help

[15:40:46] <jml> hmm. maybe I want <19.4.1

[17:00:11] <nedbat> what's the new way to do dependency_links? I want my setup.py to be generic ("I need foobar"), but my actual installation to know that foobar is at a github url. is there a way?

[17:06:49] <dstufft> nedbat: add it to your requirements.txt that you pass to pip via -r

[17:07:01] <dstufft> https://caremad.io/2013/07/setup-vs-requirement/ has more info

[17:07:22] <nedbat> dstufft: i see, so install the prereq from github first, then the mention in setup.py will be satisfied?

[17:08:27] <dstufft> nedbat: yea, lnoger term we'll have "direct dependencies" which will let you depend on stuff like that directly... but you won't be able to upload anything with a direct dependency to PyPI

[17:11:07] <nedbat> dstufft: any ideas for something like a private file that says, "Anytime pip tries to install foobar, it should get it from this github url"? Then I could just "pip install thing", and if thing needed foobar, it would come from github?

[17:12:34] <dstufft> there's nothing that explicit, you can add a find-links = https://github.com/blah blah blah#egg=thing-1.0+nedbat though I believe

[18:05:15] <wsanchez> Any reason why "python setup.py sdist upload" would sometimes not include header files?

[18:05:43] <dstufft> wsanchez: is "computer" a good answer?

[18:05:58] <wsanchez> On PyPI, kerberos-1.2.3 (C extension) is missing headers. I just uploaded 1.2.4 and they are there; no changes except the version.

[18:06:17] <wsanchez> dstufft: Yeah that's my best theory. Was hoping you had a better one. :-)

[18:06:30] <ngoldbaum> dstufft: http://whatever.computer/

[18:06:32] <wsanchez> Oh well, I'll just pay attention next time.

[18:06:38] <dstufft> is it using anything fancy like svn magic support

[18:08:41] <dstufft> wsanchez: 1.2.3 doesn't seem to have a MANIFEST.in file in the package.

[18:09:39] <wsanchez> Both were svn checkout, and sdist upload from within the directory

[18:10:05] <wsanchez> I did 1.2.3 last week, so further details are fuzzy. Meh.

[18:10:25] <dstufft> 1.2.3 is also missing the LICENSE file

[18:11:44] <dstufft> wsanchez: sorry I don't have a better answer, I don't see anything that stands out except the reason it's not included is probably realted to the MANIFEST.in file not existing

[18:11:48] <dstufft> why that doesn't exist, I dunno

Log file Viewer

Help | Karma | Search:

#pypa logs for Monday the 25th of January, 2016