Log file Viewer

[19:08:33] <dunk> Hi. I've got an issue and I am not sure if it's a problem with the dependency resolver or just my conception of what I should do. I've got a main.in, test.in, and dev.in. I am trying to generate test.txt using test.in and the output of `pip-compile` run against main.in.

[19:09:38] <dunk> The problem is that the output of main.in (I've used main.txt) has pinnings for package dependencies, and the second step is treating those as fixed.

[19:13:19] <dunk> What I mean is that, while it would make sense for the downstream steps to fix versions completely on things I specified in the main.in, it doesn't make sense that it fixes versions of dependencies in the same way, when some package downstream needs a higher version?

[19:14:06] <dunk> I seem to have to do a manual package upgrade, but shouldn't pip-compile be able to know that this isn't a fixed package version specified upstream - only a dependency - and so it could suggest / do the upgrade automatically?

[19:20:58] <tos9> dunk: I can't parse the question, can you give an example

[19:21:16] <tos9> The point of pip-compile is to pin versions of all dependencies, including transitive ones, and yeah you should do that (if of course you should have a requirements.txt in the first place)

[19:21:22] <dunk> tos9: sorry, yeah, it's not quite clear is it

[19:22:32] <tos9> no worries yeah maybe rephrasing or an example

[19:22:38] <dunk> tos9: ok, so, I have flask and pip-tools both needing Click, and that's in main.in. test.in has black in it, and it's barfing because it requires Click 7.1.2, but Click 7.0 was previously pinned in main.txt.

[19:23:08] <dunk> (black is what requires Click 7.1.2)

[19:23:14] <dunk> The new version, anyway

[19:23:25] <tos9> fun, it literally pins that version from below?

[19:23:51] <tos9> yup it does

[19:24:07] <tos9> so probably you can do `-r *main.in*` rather than `main.txt`?

[19:24:15] <tos9> And let pip-compile get you versions that are compatible with your testing deps too

[19:39:54] <dunk> tos9: ah, so that works?

[19:40:52] <dunk> tos9: two things: (1) do you mean to put that at the top of the test.in or on the `pip-compile` command line? and (2) why the wildcards around main.in?

[19:41:24] <tos9> dunk: test.in, and sorry I was just emphasizing

[19:41:45] <dunk> gotya

[19:42:08] <tos9> it should work yes, neither pip nor pip-compile treat any file extension specially, it's just a file, so you can use it with `-r` same as any other

[19:42:22] <tos9> (I have definitely done what you're doing before, but it's yeah trial and error)

[19:47:22] <dunk> tos9: That works (yay), though a small part of me wonders whether this is actually safe. I can (and do!) now have different versions of dependencies in my output main.txt and test.txt files...

[19:47:46] <tos9> right

[19:47:59] <tos9> (sorry, 25 years later we're still all paying for Python's bad import system :/)

[19:48:19] <tos9> it's certainly possible something goes wrong there

[19:48:22] <dunk> 😨

[19:48:59] <tos9> in theory the way this should work is you constrain your production deps *forward* onto the versions required by the testing deps and then you're all good, your testing deps will be a compatible superset of your prod ones

[19:49:02] <dunk> I can hand-hack the target file, and it'll work, but it's a bit of a crap experience for the next developer who won't know

[19:49:19] <tos9> in practice that's not possible in general, since a testing dep may depend on an incompatible version

[19:49:31] <dunk> Hmm, yeah

[19:49:35] <tos9> or well S/{in theory, in practice}/{in practice, in theory}/

[19:50:17] <dunk> I don't suppose the new pip resolver helps here does it?

[19:50:24] <tos9> Not in a way I can think of.

[19:50:42] <tos9> Let me see if I can find what I did the last time I solved this.

[19:51:30] <dunk> thanks :-)

[19:52:11] <tos9> to be honest I think your best practical option I can think of may be a constraints file, even though you'll have some manual step for now but it'll be an explicit one

[19:52:24] <tos9> by which I mean, go back to having your test-requirements.txt file do -r requirements.txt

[19:52:38] <tos9> then add a constraints.txt file, add -c constraints.txt to your requirements.*in* file

[19:52:48] <tos9> then in that constraints file, add constraints to get the test-requirements.txt file to compile

[19:52:56] <tos9> (so add click>=7.1.2 to it)

[19:53:39] <tos9> then your prod versions will compile to the test-requirements.txt ones, and you have a mechanism to signal to your prod requirements that there's particular versions you want to use since they'll be the ones your tests run under

[19:53:52] <tos9> the only fun may be that pip-tools has some funny bugs around constraints IIRC (heck, so does pip)

[19:54:40] <dunk> Oh, perfect

[19:55:05] <dunk> I don't really mind if it's manual. The most important thing is that it's explicit

[19:55:48] <tos9> right

[19:55:53] <tos9> maybe give that a shot

[21:40:23] <dunk> tos9: I went with force-upgrading click in the previous step (main.[in|txt])

[21:41:57] <dunk> pip-compile is of course eating its own output, and so it's the previous invocation (some time in the distant past) that's to blame. I was a bit confused as it's not an explicitly named package, so I thought that it would be auto-managed, but as you outlined it's not really possible.

[21:42:20] <dunk> I didn't realise that you can just force-update the dependencies as well as the main named packages

[21:42:34] <dunk> But thanks, I was struggling there and you made it a lot clearer!