PMXBOT Log file Viewer

Help | Karma | Search:

#pypa logs for Monday the 2nd of March, 2015

(Back to #pypa overview) (Back to channel listing) (Animate logs)
[00:03:37] <dstufft> I like git way better than Hg
[00:03:43] <dstufft> hg is confusing as hell :[
[00:04:01] <dstufft> there are like, 3 different mechancisms for creating a branch
[00:04:30] <dstufft> and the most obvious one tells you that you probably don't want it when you try to use it
[00:04:39] <dstufft> then why the hell is it the one named branch >:[
[00:05:16] <agronholm> you mean anonymous branches, named branches and bookmarks?
[00:05:56] <dstufft> oh right, anonymous branches
[00:05:59] <dstufft> 4 things
[00:06:01] <dstufft> and patch queues
[00:06:11] <agronholm> those are not really branches or anything comparable
[00:06:41] <mathieui> hg and git are not that much different, except hg needs plugins and custom configuration for many git features, and makes it less tempting to rewrite history
[00:07:06] <agronholm> yeh, hg doesn't work to my satisfaction by default
[00:07:12] <agronholm> have to enable a lot of extensions first
[00:07:12] <mathieui> same here
[00:07:31] <dstufft> agronholm: it's branch-ish, what you'd use patch queues for in mercurial I'd use a branch for in git
[00:07:37] <agronholm> I didn't even know about the history rewriting extension before someone educated me at last pycon
[00:07:47] <dstufft> there's like 3 different history rewriting extensions
[00:07:50] <dstufft> and I think 2 of them ship in core
[00:07:53] <mathieui> it’s a very decent DVCS (especially when you take bzr into account), but I still prefer git because it’s “batteries included”
[00:07:59] <agronholm> yeh
[00:08:04] <agronholm> and git is faster
[00:08:34] <mathieui> I haven’t ever seen a benchmark on that, but that’s my feeling too
[00:08:51] <agronholm> just my personal experience
[00:08:54] <dstufft> histedit, transplant, rebase all come with Hg
[00:09:33] <agronholm> but transplant is deprecated in favor of the graft command, yes?
[00:10:41] <dstufft> is it deprecated? this page doens't say it is -> http://mercurial.selenic.com/wiki/TransplantExtension just that in newer Hg's you can use graft
[00:12:51] <dstufft> anyways, my point is that my experience with using Hg, is that when I want to do somehting, someone tells me to use an extension, and then I google it and there are like 2-4 options for what extension I want and I typically get annoyed at trying to evaluate them and just give up and use hg diff --git and patch files in my current directory to do things
[00:13:21] <doismellburning> dstufft: that sounds familiar
[00:20:28] <dstufft> I think the primary reason why the meme that hg is easier to use but git is more powerful came from the early days before git had much porclean commands so it was all plumbing and because hg commands match existing VCS commands functionality better, whereas git commands match the git object model better
[08:17:32] <ronny> dstufft: these days a lot more has been moved into core and core extensions
[12:05:50] <ronny> dstufft: im wondering, would it be acceptable to bundle setup_requires wheels inside of a sdist (i.e. setuptools, setuptools_scm and so on)
[17:00:59] <Ivoz> ronny: you still want to check they're up to date
[17:01:06] <Ivoz> which may make them useless
[17:26:30] <tdsmith> if a package wants to build python bindings for a thin x86_64 library on OS X on a universal (i386+x86_64) python, what should it do?
[17:27:22] <tdsmith> is it more correct to die or to drop -arch i386?
[17:27:39] <apollo13> depends, how much do you value your life?
[17:28:20] <tdsmith> probably more than i value having a universal extension module
[17:30:24] <ionelmc> tdsmith: is it a swift death?
[17:58:40] <Ivoz> tdsmith: will you find any systems not running x86_64 ?
[17:59:28] <tdsmith> i think it's unlikely
[18:00:40] <tdsmith> someone's asking homebrew if a library can always be a universal build so that their python module builds with system python (which is universal) and i'm wondering if they should just change how their python module is built
[18:00:52] <tdsmith> i'm surprised we don't encounter this more often
[18:07:52] <Ivoz> why the heck are they asking homebrew to not deal with homebrew's python
[18:08:47] <Ivoz> having to deal with os x' system python is always ugly
[18:10:10] <tdsmith> homebrew considers the request reasonable, for better or worse :) we try not to force users to depend on our versions of things to avoid the macportsian dilemma of having to rebuild perl every time you want to upgrade something
[22:16:32] <ronny> Ivo: why would it make them useless, you'd be shipping known good versions, no updates needed
[22:16:52] <ronny> ionelmc: it might actually prevent breakage
[22:17:20] <Ivo> you think they're good at the time
[22:17:50] <Ivo> until theres a bug, at which point they need an update
[22:18:03] <Ivo> at which point, you're instead shipping known bad versions.
[22:18:48] <Ivo> which is why in most cases vendoring is frowned upon
[22:20:37] <nedbat> I am using "pip install -q --exists-action w -r requirements.txt", and a git link in the requirements.txt file will not update unless I uninstall the package first. I've tried bumping the version number in the referenced repo, doesn't help.
[22:21:12] <nedbat> the line in the requirements.txt file is: git+https://github.com/edx/edx-lint.git@c592557ecee8f2f1ef14ad984e1353f41aba0b47#egg=edx_lint
[22:21:28] <tos9> --exists-action isn't for whether the package is installed
[22:21:33] <tos9> it's for build directory nonsense
[22:21:49] <tos9> if you want to update you need -U
[22:21:50] <Ivo> you want an --update
[22:21:53] <ronny> Ivo: updates can break things just as well ^^
[22:22:11] <tos9> but yes, updates are like any other updates in pip, they don't do what you want / mean
[22:22:35] <ronny> i prefer makign a new release to fix things over having a existing release that worked as expected break automatically
[22:23:03] <nedbat> Ivo: are you talking to me or ronny?
[22:23:16] <Ivo> nedbat: you want --update in your command
[22:23:18] <Ivo> or -U
[22:24:09] <Ivo> ronny: hopefully people making use of semantic versioning and proper use of ~= will fix most of that as well
[22:24:13] <ronny> nedbat: i thinkhes talking to you, doesnt make sense at me
[22:24:48] <ronny> Ivo: i tend to do that ^^ but i wouldnt trust on it
[22:25:05] <nedbat> Ivo: ok, but if the requirements.txt had "some_package=1.2.3", and I had 1.1 installed, it would install 1.2.3 even without the --update, right?
[22:25:19] <ronny> nedbat: yup
[22:25:42] <nedbat> is there a syntax I can use on a git url that would have the same effect?
[22:25:46] <Ivo> for example httplib2 doesn't really use semantic version, means we have to make sure it works right every new version we use with pip
[22:25:50] <ronny> nedbat: scm links are flaky - in particular with dvcs you have next to no way to figure ordering
[22:26:18] <tos9> nedbat: they would, but if you were in that situation, you should use -U anyhow
[22:26:18] <ronny> hmm, i consider httplib/httplib2 and requests total messes ^^
[22:26:23] <nedbat> ronny: I can't append "==0.2" to get the version number in there?
[22:26:29] <tos9> nedbat: basically, use no -U for fresh installs, and -U for updates
[22:26:30] <Ivo> nedbat: you can also refer to a branch to install, or tag, instead of a particular commit, if you like
[22:26:44] <nedbat> tos9: well, -U is a big hammer: it applies to all the packages, and all their dependencies
[22:26:52] <tos9> nedbat: yes
[22:26:57] <tos9> nedbat: and there isn't really a solution to that problem anyhow
[22:27:00] <tos9> nedbat: other than recursive pinning
[22:27:08] <tos9> so unless you're doing that, you're going to break anyhow
[22:27:13] <tos9> and if you are doing that, then -U is fine
[22:27:16] <Ivo> nedbat: no, the thing (and its version) you're installing is decided by its exact state at checkout, not what you want to specify
[22:27:36] <nedbat> Ivo: except pip won't look at the version number in the checkout it seems.
[22:27:58] <nedbat> Ivo: I have 0.1, I specified a hash which is 0.2, and pip didn't install it.
[22:28:37] <Ivo> nedbat: would suggesting tagging commits
[22:29:03] <nedbat> Ivo: will that do what I want? Does pip assume the tag is a version number, and use it to decide to install?
[22:29:04] <tos9> that won't help
[22:29:09] <tos9> commits are not versions
[22:29:38] <Ivo> but anyway, if you don't want projects updating themselves willy-nilly, then restrict their versions
[22:30:14] <nedbat> Ivo: I'm familiar with the promise and challenges of pinning.
[22:30:33] <ronny> hmm
[22:30:34] <nedbat> Ivo: I'm trying to understand the logic pip uses to decide what a git url means, and when to install it.
[22:30:43] <Ivo> https://www.python.org/dev/peps/pep-0440/#version-specifiers
[22:31:02] <ronny> nedbat: afair all it does is clone, then install from the checkout
[22:31:04] <tos9> nedbat: they're used for the case where pip clones, they just specify what to checkout.
[22:31:09] <tos9> nedbat: the information isn't persisted
[22:31:13] <tos9> so pip doesn't know when it changes
[22:31:23] <tos9> or that what you have is different, or what version is there, or anything.
[22:31:33] <Ivo> getting pip to do an editable install might help, -e
[22:31:37] <nedbat> tos9: but pip freeze has the version number.
[22:31:45] <nedbat> Ivo: I can try that, thanks.
[22:31:52] <ronny> nedbat: my personal oppinion is to never use git urls, instead using devpi and something like setuptools_scm in comibination
[22:32:19] <nedbat> ronny: that's something I'll look into. It won't get me past this bump right now though.
[22:32:20] <tos9> nedbat: persisted in the version information for comparison, it may be stored elsewhere
[22:32:28] <ronny> nedbat: pip takes as version whatever setup.py egg_info will put in
[22:32:45] <tos9> nedbat: I forget if pip does that (shows you commits) when you don't use -e -- does it, or are you using -e?
[22:33:01] <nedbat> tos9: i'm not using -e.
[22:33:04] <ronny> nedbat: so if the project in question doesnt have a scm aware setup.py, you get whatever was in there statially
[22:33:29] <nedbat> ronny: what do you mean by a scm aware setup.py?
[22:33:58] <ronny> nedbat: a setup.py that will ask the underlying scm for a version if possible and fallback to known metadata otherwise
[22:33:58] <nedbat> tos9: with -e, pip freeze does have the entire git url, including hash and #egg
[22:34:01] <tos9> nedbat: it doesn't do that for me.
[22:34:06] <ronny> (see setuptools_scm)
[22:34:09] <nedbat> tos9: what doesn't do what?
[22:34:10] <tos9> nedbat: Yeah. And without, I don't have any VCS info in there. do you?
[22:34:38] <nedbat> tos9: right, git without -e is just "edx_lint==0.1".
[22:34:38] <tos9> -e persists the checkout, so git can know what commit you have.
[22:34:50] <tos9> nedbat: Right -- so like I said, pip has no way of comparing versions
[22:35:14] <tos9> if you have -e, you still keep the checkout, so theoretically pip could be more intelligent if you have -e yourrequirement in your requirements.txt
[22:35:55] <ronny> nanonyme: hmm, could you take a look and comment on setuptools_scm then maybe?
[22:35:55] <nanonyme> Eventually someone anyway forgets to bump the version number and then it's just a lie
[22:35:56] <tos9> but not "really" since you could have changed the repo yourself, so that seems kind of... hairy to me, I dunno.
[22:36:22] <tos9> nanonyme: At $WORK, we don't use version numbers.
[22:36:25] <nanonyme> ronny, well, everything that changes is built in separate Visual Studio projects and added to the Python packages as includes
[22:36:31] <nedbat> tos9: without -e, pip has the version (freeze shows it), and the git clone it makes from the url will have a version in the setup.py. There is enough information to install the right version.
[22:36:31] <tos9> We never want to be on anything but latest.
[22:37:06] <nanonyme> tos9, you never end up having to backport bug fixes to customers for old versions?
[22:37:08] <tos9> nedbat: without -e I don't see the version in freeze
[22:37:17] <tos9> nanonyme: I mean for internal stuff
[22:37:21] <nanonyme> Aha, right
[22:37:26] <tos9> nanonyme: so luckily, the answer is "no"
[22:37:36] <ronny> nanonyme: so all your stuff is non-python?
[22:37:48] <nedbat> tos9: hmm, don't know why, I get the version, just as if I had installed from PyPI.
[22:38:09] <nanonyme> ronny, we use Python mostly as a tool to bind against C/C++ projects and writing test automation against them
[22:38:11] <tos9> nedbat: sorry, you said version
[22:38:26] <tos9> nedbat: I thought you meant commit (git version, whatever you're using)
[22:38:29] <nedbat> right.
[22:38:31] <ronny> nanonyme: i see, then hgdistver aint that usefull for you
[22:38:33] <tos9> nedbat: So yes, I have version in pip freeze, it's just a regular install
[22:38:33] <nanonyme> ronny, Python has reasonable FFI so it works nicely for that
[22:38:35] <nedbat> tos9: I have the version, not the commit.
[22:38:47] <tos9> nedbat: but pip can't know what version your commit is?
[22:38:58] <nedbat> tos9: once it clones it it can.
[22:39:03] <tos9> nedbat: it doesn't know if it needs to clone
[22:39:12] <tos9> it might not
[22:39:19] <nedbat> tos9: sure, so clone it and find out.
[22:39:27] <ronny> nedbat: does your setup.py determine the version from static commited file or does your setup.py ask the scm?
[22:39:28] <nedbat> tos9: the proposed solution (-U) would clone all the time anyway.
[22:39:38] <tos9> nedbat: why :P? I'd prefer not doing network IO, and if I want network IO, I tell pip to upgrade
[22:39:45] <nedbat> ronny: in this case, the version is a simple string in setup.py
[22:40:07] <nedbat> tos9: ok, I'm looking for a middle ground between "maintain radio silence" and "upgrade everything everywhere"
[22:40:17] <tos9> anyways, I'm goign to shut up, because I think we all know what the parameters are at this point
[22:40:22] <ronny> nedbat: then that is what pip will see as version for all checkouts until you bump
[22:40:25] <nedbat> tos9: I think we do :)
[22:40:39] <tos9> nedbat: Yup. I don't htink there is a good solution for that other than recursive pinning, but maybe someone else does.
[22:40:44] <nedbat> ronny: well, except that using a git url, it doesn't seem to clone the repo to find the version string.
[22:41:01] <tos9> The real solution is for pip to grow actual upgrade, but that is limited by either dstufft's time or someone else doing it it sounds like
[22:41:12] <ronny> hmm, now im starting to thtink im missremembering some details
[22:41:16] <nedbat> tos9: what would "actual upgrade" do?
[22:41:22] <tos9> nedbat: minimal upgrade
[22:41:34] <nedbat> tos9: shallow upgrade.
[22:41:36] <tos9> nedbat: yes
[22:42:04] <tos9> upgrade just the packages that changed versions, although if they specify new versions of deps, you have to do taht too, but not anything whose current version satisfies the new version's dep
[22:42:59] <tos9> nedbat: I say "actual upgrade" because to me this is the correct default, and not the crazy behavior we currently have
[22:43:06] <tos9> (which is just the simpler thing to implement udnerstandably)
[22:43:40] <Ivo> nedbat: so had you tried https://github.com/edx/edx-lint.git@c592557ecee8f2f1ef14ad984e1353f41aba0b47#egg=edx_lint==0.2
[22:44:24] <nedbat> Ivo: no, I've not seen that done before. Is that mentioned somewhere?
[22:48:16] <Ivo> sorta, under https://pip.pypa.io/en/latest/reference/pip_install.html#vcs-support
[22:48:45] <Ivo> but basically, you saying pip install git+https://<giturl>#egg=edx_lint
[22:48:52] <Ivo> is just saying you want edx_lint installed
[22:49:00] <Ivo> pip sees that edx_lint is installed
[22:49:14] <Ivo> so doesn't do anything; unless you also pass the --upgrade flag
[22:50:54] <Ivo> works exactly the same as a normal requirement from pypi, wont do upgrades unless you tell it to install a specific version not compatible with that that's already installed
[22:53:18] <dstufft> minimal upgrade isn't exactly right either
[22:54:02] <dstufft> what we really need is for pip to understand the fact that there are more version constraints than exist in a current invocation
[22:54:42] <tomprince> It is sometimes nice that it doesn't.
[22:55:28] <nedbat> Ivo: thanks! That syntax is just what I needed.
[22:55:44] <tos9> dstufft: aren't the sum total of those additional complaints the definition of "minimal install"?
[22:55:58] <tos9> s/complaints/constraints
[22:56:28] <Ivo> I think people wanting minimal installs haven't realised how much ruby & js managed to get right with semantic versioning
[22:56:43] <Ivo> when the majority of projects actually start using
[22:57:00] <Ivo> doing smart & appropriate pinning actually just works
[22:57:11] <Ivo> and a 'minimal install' isn't needed any more
[22:57:21] <tos9> that isn't true, because semantic versioning is a lie.
[22:57:44] <Ivo> apparently lies work
[22:57:46] <tos9> and ruby and JS don't get it right by a long shot.
[22:57:47] <nedbat> Ivo: interestingly, that syntax doesn't seem to be mentioned on that page either...
[22:58:27] <dstufft> sem ver is approximately fine
[22:58:29] <tos9> Ivo: not in the least -- just find one of 3 separate jashkenas bug tickets where a version of underscore or another jashkenas project saw major breaking changes in minor releases.
[22:58:33] <Ivo> apparently it should be edx_lint-0.2
[22:58:41] <tos9> sem ver is fine, but it is not a guarantee, and you still need to pin.
[22:58:52] <Ivo> tos9: lol. underscore isn't semantically versioned.
[22:58:55] <tos9> it's a loose commitment that everyone should be making anyhow regardless of versioning scheme
[22:59:04] <dstufft> wait
[22:59:07] <Ivo> so you're proving me right
[22:59:08] <dstufft> who said anything about not pinning
[22:59:13] <Ivo> lodash is semantically versioned
[22:59:17] <Ivo> use that instead
[22:59:32] <Ivo> nedbat had reservations about it
[22:59:53] <tos9> Ivo: it isn't but packaging tools would treat it as such -- well unless they specifically don't, I don't know about what various tools do
[23:00:02] <tos9> the point is sem ver doesn't solve thi sproblem.
[23:00:33] <Ivo> tos9: just last week there were minor scuffles in whether projects like backbone should switch to lodash because underscore was breaking stuff with minor versions bumps
[23:00:36] <nedbat> Ivo: my problem isn't so much with pinning, as not wanting to use the big -U hammer to get my latest git repo installed. But you gave me the right solution: a version number on the git url
[23:00:46] <tos9> Ivo: yes? maybe we're not talking about the same thing I guess
[23:00:57] <tos9> I'm saying sem ver doesn't at all have anything to do with needing minimal install
[23:01:36] <Ivo> well I was talking about things going well when semantic versioning gets done right, you told me it was a lie and pointed out a project that happened to not follow it at all, so I'm pretty confused, that's a non-sequiter
[23:01:54] <tos9> Ivo: I was saying that in this context -- sem ver doesn't solve pinning problems.
[23:02:11] <tos9> specifically, if you pin with ~= or whatever on a sem ver project, you *still* need minimal install.
[23:02:13] <dstufft> tos9: the problem with a naive minimal upgrade is, what do you do when you need to upgrade your versions, but there are competing constraints, e.g. pip install --minimal-upgrade foo, where foo depends on bar>=1.0, but you have bar 0.8 installed, what most people want is that you upgrade bar to the latest version that matches the >=1.0 specifier, but what if you also have osmething else installed that also depends on bar, but it depends on bar <3
[23:02:25] <Ivo> tos9: why do you still need a minimal install
[23:02:43] <tos9> Ivo: Because sem ver has nothing to do with this. Minimal install is how you do "least risk" upgrades
[23:02:48] <tos9> sem ver is a lie --
[23:02:52] <dstufft> I think that bundler gets this correct FWIW
[23:02:55] <tos9> it means "I Promise not to change things in minor versions"
[23:03:01] <tos9> this is a lie, because it relies on some unchecked promis
[23:03:13] <tos9> so it's a good promise to make, but it's worthless for you as a user
[23:03:38] <Ivo> when you follow sem ver, and the projects you rely on do, you SPECIFY the risk in the version. So you specify your risk yourself, while doing a full upgrade
[23:04:01] <tos9> dstufft: personally I'd expect behavior that tries to satisfy all the constraints I guess
[23:04:04] <Ivo> e:g you say I only want patch releases - low risk. Specified right in the version
[23:04:08] <Ivo> thats why its semantic
[23:04:10] <tos9> dstufft: and to error otherwise
[23:04:27] <tos9> dstufft: which probably makes implementation a PITA, but yeah.
[23:05:13] <dstufft> you have Gemfile, which you record broad rangers (such as with ~>), when you do the deploy you generate a Gemfile.lock which pins you to ==, and then you can upgrade specific dependencies in the Gemfile.lock (but within the constraints of the Gemfile)
[23:05:23] <tos9> Ivo: "This is the level of risk" doesn't correspond with "and by the way, I want to incur that risk every time I want to upgrade some other thing"
[23:05:51] <tos9> Ivo: if someone releases another minor version, and you want to upgrade some other package (a major version say, or to a specific minor version), it is unacceptable to upgrade the first package, just because of semver
[23:05:52] <Ivo> it quite perfectly can
[23:06:11] <tos9> unless you specifically indicate that that's the behavior you want (and I know that's a thing, I think it's a bad one)
[23:06:46] <Ivo> you rely on your dependent packages to get things right at the same time
[23:07:05] <Ivo> which ofc used to be absolutely and unequivocally impossible, because everyone used version to mean different things
[23:07:17] <Ivo> but thats just the status quo we've been used to
[23:07:20] <Ivo> doesn't have to stay that way
[23:07:38] <dstufft> ehh
[23:07:42] <dstufft> an upgrade is a risk
[23:07:47] <dstufft> it doesn't matter what version number it is
[23:07:55] <tos9> right
[23:07:57] <dstufft> <insert obligatory xkcd>
[23:08:05] <tos9> and not minimizing that risk every single time is basically a bad idea IMO.
[23:08:08] <dstufft> the idea behind semver is that you communicate the expected level of risk
[23:08:48] <Ivo> and if you're complex enough, confliciting sub-dependency conflicts can hit you no matter which policy you like, they'll just hit you in a different manner
[23:08:53] <dstufft> that communication is really great for packaging libraries
[23:09:05] <dstufft> e.g. you can do something like ~=1.0 in your install_requires
[23:09:57] <Ivo> tos9: upgrading the least things also means you get the least possible security updates. How are you specifying that risk?
[23:10:29] <tos9> I don't think that's how that works.
[23:10:38] <tos9> I don't upgrade to the latest version because maybe it has security upgrades
[23:10:42] <dstufft> that communication is less useful for things like where you're doing something like deploying webapps, where you want to minimize the risk of a deploy breaking because of an unrelated thing updating
[23:11:10] <dstufft> specific deployments should more or less _always_ pin (IMO)
[23:11:17] <dstufft> that's best practice as far as I'm concerned
[23:11:17] <tos9> dstufft: yeah, agreed, that was my point about still needing minimal install
[23:11:26] <tos9> cool, sounds like we agree on that much then
[23:11:49] <Ivo> but then, when you're doing the deployment, you're doing static pinning anyway
[23:12:00] <Ivo> it's going to be a minimal upgrade by your definition no matter
[23:12:12] <dstufft> (my point isn't that we don't need a better upgrade mechanism, just that inverting it from greedy to lazy won't actually solve the problem, it'll just shuffle deck chairs around until we get other pieces in that will make it work)
[23:12:35] <tos9> dstufft: it sounded like you're saying "differentt upgrade mechanism"
[23:12:46] <tos9> personally I don't care if it's different and the current one goes or different and the current one stays :P
[23:12:58] <dstufft> tos9: I think pip freeze for deployment isn't that great
[23:13:27] <dstufft> and typically the workflow people want is <do minimal upgrade>, pip freeze to update the requirements.txt
[23:13:48] <dstufft> I think to really make a good deployment story we need to rework the way we handle requirements.txt files
[23:13:56] <tos9> yeah, except requirements.txt conflates what I need with what this deployment has right now
[23:13:59] <tos9> (so I don't do the latter part)
[23:14:01] <Ivo> holy shit awesome ue4 just went free
[23:14:53] <dstufft> (I think I'm explaining this really poorly TBH. It's a sort of subtle issue and I have a hard time putting what I mean into words about it)
[23:15:30] <Ivo> I don't get what all the fuss is about