PMXBOT Log file Viewer

Help | Karma | Search:

#pypa logs for Thursday the 12th of November, 2015

(Back to #pypa overview) (Back to channel listing) (Animate logs)
[15:59:52] <mitsuhiko> dstufft: what's the best way to pin packages to a semver like thing
[15:59:55] <mitsuhiko> and then to make specific pins on top
[16:00:17] <mitsuhiko> eg: we want to use this specific version of the package, but want to consider upgrading to something api compatible
[16:00:30] <mitsuhiko> but not further
[16:06:52] <xafer> I think you are looking for https://www.python.org/dev/peps/pep-0440/#compatible-release i.e. `~=2.0` for the 2.0 series
[17:51:42] <nedbat> does pypi have a cdn fronting it now?
[17:52:49] <tos9> yes
[17:52:51] <tos9> fastly
[17:52:56] <tos9> (for like ~a year I think)
[18:01:06] <dstufft> Since May of 2013
[18:01:29] <tos9> oh boy time flies
[18:01:39] <dstufft> does it ever
[18:01:59] <dstufft> remember when you couldn't make a validated TLS connection to PyPI and neither pip nor easy_install validated TLS anyways evne if you tried
[18:02:10] <tos9> yes :/
[18:02:10] <dstufft> that was pretty cool how we all used to just randomly execute crap downloaded via http
[18:02:33] <tos9> dstufft: apparently CPAN still does
[18:02:39] <tos9> I learned this 2 weeks ago
[18:03:10] <tdsmith> it's amazing how un-burninated the universe is vs how burninated it totally could be
[18:03:55] <dstufft> tos9: W E L P
[18:04:05] <tos9> yup.
[18:04:23] <dstufft> tdsmith: It terrifies me that I know of several multi million dollar companies that ``pip install`` directly from PyPI in their production infra
[18:04:57] <tos9> dstufft: hooray! we're not one of them anymore. Although we might become one again, I will probably fling devpi out a window very soon.
[18:06:37] <dstufft> tos9: pip 8 has peep functionality built in thanks to the excellent ErikRose
[18:06:45] <dstufft> so you can at least do it without trusting me now
[18:06:57] <dstufft> well, other than the fact that I also have some level of control over pip
[18:07:03] <dstufft> so I guess you can't stop trusting me, sorry!
[18:07:53] <tos9> !!
[18:08:02] <dstufft> mitsuhiko: xafer is right about ~=, though that's only pip 6+ and setuptools 8+ so just ftr
[18:08:06] <tos9> this appears to partially solve the non-addressable-per-package-index thing I've complained about.
[18:08:21] <dstufft> which is that
[18:08:22] <tos9> (you trust me I'm sure to complain)
[18:08:33] <tos9> dstufft: not being able to say "foo comes from bar index"
[18:08:38] <dstufft> oh right
[18:08:52] <dstufft> you still can't do that, but you can make it an error if it doesn't match the hash at least
[18:08:52] <tos9> it solves the "swoop-in and steal the name" part of it at least
[18:08:56] <tos9> yeah
[18:09:36] <dstufft> whenever I get around to reworking our multiple repo support the rest of that will get taken care of
[18:25:34] <kober> What does this mean setuptools must be installed to install from a source distribution
[18:26:51] <jaraco> kober: It sounds like you have a pure-pip installation (without setuptools), and you’re trying to install a package that’s not a wheel, so pip requires setuptools to be installed to build the wheel for installation.
[19:35:46] <mitsuhiko> does anyone know how to inject files into an sdist build?
[19:35:49] <mitsuhiko> that do not originate in the source folder
[20:17:29] <jaraco> mitsuhiko: two ways I use. (1) Use setuptools_scm and commit the files to the SCM repository or (2) use MANIFEST.in.
[20:17:59] <mitsuhiko> that's not really what i'm after but i found a solution for my issue anyways at this point
[20:19:07] <mitsuhiko> it involves fetching a finalized sdist command and extending the file list after copying files manually over to the build folder
[20:19:51] <jaraco> I missed the distinction, but now I understand. Glad you found something that worked.
[20:30:54] <mitsuhiko> i'm gonna rage
[20:31:09] <mitsuhiko> dstufft: how do you guys not get crazy with this
[20:32:22] <dstufft> mitsuhiko: I like how you assume it hasn't made us crazy :D
[20:43:14] <mitsuhiko> dstufft: can you just give me one pointer?
[20:44:16] <dstufft> um
[20:44:17] <dstufft> it should
[20:44:29] <dstufft> https://github.com/jaraco/setuptools/blob/master/setuptools/command/develop.py#L116-L117
[20:44:33] <dstufft> it's not conditional on anything
[20:45:50] <mitsuhiko> great.
[20:45:51] <mitsuhiko> :(
[20:45:57] <dstufft> mitsuhiko: is it actually not running build_ext, or is build_ext not doing anything?
[20:46:03] <mitsuhiko> it's not invoking build_ext
[20:46:04] <mitsuhiko> i will investigate
[20:46:10] <mitsuhiko> the thing is, i have no ext modules
[20:46:20] <mitsuhiko> so i already had to overwrite has_ext_modules on the dist to get it to run
[20:46:25] <mitsuhiko> i'm attaching a subcommand to build_ext
[20:46:32] <mitsuhiko> as this is the only path that seems to be invoked inplace and not for all builds
[20:46:35] <dstufft> ohh
[20:46:38] <mitsuhiko> i'm that far down the rabbithole
[20:47:09] <dstufft> is this OSS or otherwiose publically available?
[20:47:18] <mitsuhiko> all of this started because we want to bundle node modules
[20:47:24] <mitsuhiko> dstufft: https://github.com/getsentry/sentry/pull/2301
[20:47:45] <dstufft> mitsuhiko: you might be interested in something I did just recently
[20:47:46] <dstufft> sec
[20:47:58] <dstufft> https://github.com/pypa/pip/pull/3234
[20:48:15] <dstufft> I didn't test it with pip install or -e or anything though :/
[20:49:07] <dstufft> mitsuhiko: I don't see you overriding build_ext at all here, it looks like you've overridden just plain old build?
[20:49:43] <dstufft> in your cmdclass={} I mean
[20:59:49] <dstufft> gotta go, hopefully see ya
[20:59:53] <dstufft> er
[20:59:57] <dstufft> meant to say, hopefully that helped
[21:01:05] <mitsuhiko> dstufft: sorry, got dragged away
[21:01:07] <mitsuhiko> will look
[21:30:05] <dstufft> mitsuhiko: Get it to work?
[21:30:13] <mitsuhiko> dstufft: i figured it out
[21:30:21] <dstufft> awesome, glad to hear!
[21:30:28] <mitsuhiko> basically there are two bdist_ext
[21:30:34] <mitsuhiko> one is from distutils the other is from setuptools
[21:30:37] <mitsuhiko> i subclassed the wrong thing
[21:30:58] <mitsuhiko> also, the one from setuptools does not invoke subcommands. because reasons
[21:32:00] <dstufft> mitsuhiko: for whatever it's worth, it's very likely that in the nearish future pip will support pluggable build systems
[21:32:46] <dstufft> so in the nearish future it's very likely you'll be able to not use setuptools/distutils at all :D (of course with the caveat that you'll need a setup.py shim if you want to support older pips)
[21:34:11] <tos9> what are the chances in the nearish future that there will be a non-pip wheel installer that I should / can use
[21:34:25] <tos9> so that I can just not install pip on servers
[21:34:55] <dstufft> uh
[21:34:58] <apollo13> tos9: what is the difference between installing pip and $other_tool?
[21:35:16] <dstufft> probably not likely, but I'm confused why you don't want pip but you're OK with some other thing being installed
[21:35:18] <mitsuhiko> apollo13: why do i need a tool at all?
[21:35:24] <tos9> apollo13: the difference would be $other_tool would be small and focused, but $other_tool doesn't even *need* to be installed on the server
[21:35:32] <tos9> I can sit on $workstation and run $other_tool $all_my_servers
[21:35:40] <mitsuhiko> tos9: i install wheels with unzip
[21:35:41] <apollo13> mitsuhiko: well feel free to unzip and rsync if you feel like it
[21:35:41] <mitsuhiko> works
[21:35:46] <tos9> mitsuhiko: I'm considering that
[21:35:51] <mitsuhiko> apollo13: i'm doing that
[21:35:52] <mitsuhiko> sans rsync
[21:36:14] <apollo13> so why another tool for it :D
[21:37:00] <dstufft> mitsuhiko: tos9 Note: That won't do everything. Right now it'll get you most of the way there but you won't get script wrappers. In the future it's likely we'll add post-install hooks as well which you won't get with that either
[21:37:07] <dstufft> but if you just need a way to get the core files it'll work
[21:37:25] <tos9> we *do* use script wrappers
[21:37:39] <dstufft> tos9: fwiw pip works zipped up if you have a OS bundle for SSL certiificates or you aren't using ssl
[21:38:12] <dstufft> if you don't want to install pip, you just want to have a executable file you can scp around execute, and delete
[21:38:41] <tos9> dstufft: I think my overall problem with pip right now is I don't know yet how to treat it
[21:38:48] <tos9> maybe ultimately I *do* want pex or something
[21:39:08] <tos9> but yesterday $coworker asked me "what's our protocol for managing pip versions in our infra, is that via config management or manual"
[21:39:15] <dstufft> pip is a sort of weirdly scoped tool
[21:39:17] <tos9> and my ansewr was "right now I don't have a good answer, so it's manual"
[21:39:38] <dstufft> it *should* be focused on developers and we *should* be using pex or some other thing to make single file deployments
[21:39:50] <tos9> because ultimately I *do* want to pin a minimum version, but pip is an oddball -- most of the time I want latest, because older pips stop to work occasionally, but latest is obviously dangerous for other reasons
[21:39:53] <dstufft> but none of those stories are as fully baked as pip is, so pip gets all over the place
[21:39:58] <tos9> dstufft: Yeah.
[21:40:00] <tos9> Agreed.
[21:40:51] <dstufft> I haven't actually used pex, but I hear it works pretty OK if you have a fairly homogenious environment and can ensure Python is installed
[21:41:27] <tos9> I haven't either, and I'm sure I'd be into some fun if I tried it, not sure how many people out there are using pex with pypy.
[21:42:20] <dstufft> I think the things that pex does should be mostly drop in for PyPy... but PyPy might not be used by anyone else who is using pex so there's a good chance for weirdo edge cases
[21:42:29] <dstufft> I'm not sure what Twitter was running, but I assume CPython
[21:44:24] <dstufft> at the end of the day, pex is really just a zip file with a __main__ that has some special handling for C extensions and other things that aren't zipimport safe
[21:47:17] <dstufft> my strategy is mostly just run with pip's latest release
[21:47:30] <dstufft> but I'm probably one of the folks best positioned to deal with breakages
[21:54:25] <mitsuhiko> tos9: i tried to actually make a desktop distributable python a week ago
[21:54:31] <mitsuhiko> it's ... horrible
[21:54:39] <tos9> dstufft: for running latest I'm less worried about it being pip's fault and more that when something is broken it will have to be a thing I remember to check.
[21:54:55] <mitsuhiko> i only got osx running so far
[21:55:05] <tos9> mitsuhiko: I think I saw you trying to do that
[21:55:12] <tos9> I'm sure you sympathize with glyph's article
[21:55:15] <mitsuhiko> and that one was the only one that actually worked because you can fuck around with binaries so that they locate resources relatively
[21:55:22] <mitsuhiko> on linux i have no clue
[21:55:25] <tos9> did you find ned_deily?
[21:55:43] <mitsuhiko> tos9: didn't bother. i managed to just rewrite all binaries post compilation
[21:55:49] <tos9> ah
[21:55:54] <mitsuhiko> https://github.com/mitsuhiko/lektor/blob/master/gui/bin/make-python-framework-relocatable <- install_name_tool ftw
[21:55:58] <tos9> cool
[21:56:04] <mitsuhiko> i think i also found out why it's not done by default
[21:56:08] <mitsuhiko> virtualenv breaks in hilarous ways
[21:56:27] <mitsuhiko> so i am unpacking wheels into it
[21:56:33] <mitsuhiko> instead of running pip. which fails
[21:56:39] <mitsuhiko> but the end result is a working interpreter
[21:56:41] <dstufft> another tool that is a bundle of mandatory sadness
[21:57:07] <mitsuhiko> it's frustrating because all the tools we have just make it worse rather than easier
[21:57:13] <mitsuhiko> this is one we should learn from the javascript community
[21:57:24] <mitsuhiko> they look at each platform individually
[21:57:28] <mitsuhiko> and make their shit working like the platform wants
[21:57:41] <mitsuhiko> we just try to force everything into some bizarro unix world that exists nowhere
[21:57:50] <mitsuhiko> and then we make windows special
[21:57:55] <mitsuhiko> but also in a way that makes no sense on windows
[22:00:57] <dstufft> I'm not sure what that means
[22:01:07] <dstufft> Do you mean Python does this, or Pytohn packaging does
[22:05:59] <jaraco> I remember my first PyCon (2008), I was surprised to see so few Windows users. My sense is that because Python is dominated by Unix users, it tends to be Unix first and Windows as an afterthought and only by volunteers. The Windows support then becomes marginalized, creating an unfortunate feedback loop.
[22:06:24] <dstufft> pip still doesn't have Windows CI
[22:06:34] <jaraco> Some champions in the Windows world have gone a long way to alleviating that pain, but it’s always there.
[22:06:37] <dstufft> pretty sure it only works on Windows because pfmoore is awesome
[22:06:45] <jaraco> !m pfmoore
[22:06:46] <pmxbot> you're doing good work, pfmoore!
[22:06:53] <utek> hmm
[22:16:10] <jaraco> But for me, as a primarily Windows user, Python was a very attractive language for how it bridges the platforms without hiding them (like the JVM does).