Log file Viewer
#pypa-dev logs for Thursday the 9th of January, 2020
[22:01:46] <cooperlees> Peoples - TemptorSent is making a plugin for bandersnatch thet allows people to use metadata from the JSON API arbitrarily filter what bandersnatch pulls down
[22:04:40] <techalchemy> TemptorSent, for specifiersets, if you only want to talk about them I'm happy to chat ;) if you want to change how they work you'll need pradyun or donald (so probably pradyun)
[22:16:32] <TemptorSent> Hi techalchemy, what I'm specifically trying to accomplish is calculate the intersection of two SpecifierSets
[22:22:13] <techalchemy> TemptorSent, when you say 'intersection' do you mean the overlapping versions in two different ranges?
[22:23:09] <techalchemy> or do you actually mean the intersection of the specifiersets i.e. the specifiers that are the same between the two sets
[22:23:51] <TemptorSent> techalchemy: Not just ranges, full SpecifierSets, including the annoying ~= and ===, along with == and !=, all anded.
[22:25:36] <techalchemy> 'not just ranges' as in the first of the two things I described, and additionally you would like to do more?
[22:25:48] <TemptorSent> So given two SpecifierSets, return a SpecifierSet that represents the logical intersection of those two sets.
[22:26:31] <techalchemy> if I have a specifierset: ">2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*", and another specifierset: ">2.7", the intersection of these is ">2.7"
[22:28:14] <TemptorSent> Consider [">=2.7", "!=3.5.*", "<3.8"] intersect [ "<=3.7" ] for a simple case, the intersection would be [ ">=2.7", "!=3.5.*", "<=3.7" ]
[22:28:22] <techalchemy> if I have a specifierset: ">=2.7,<3.8" and another specifierset ">=2.7" the intersection is ">=2.7". calculating that is pretty easy with specifiersets
[22:29:09] <techalchemy> > Consider [">=2.7", "!=3.5.*", "<3.8"] intersect [ "<=3.7" ] -- have you done this in python? I think you are using the wrong terminology here so I dont want to dwell on it too long
[22:30:34] <techalchemy> TemptorSent, in set math, an intersection identifies elements that are the same in two different sets
[22:30:44] <TemptorSent> I'm talking about matematically creating intervals from SpecifierSets, determining the overlapping intervals, including whether the endpoints are open or closed, and representing the result as a SpecifierSet.
[22:31:53] <TemptorSent> Yes, if we treat specifiers as strings, that's true -- I'm trying to treat them as tuples representing bounding conditions and discontinuities.
[22:32:14] <techalchemy> well an intersection is an intersection no matter what, but specifiers _are_ just strings
[22:33:01] <techalchemy> like, they have some notion of whether something is contained within their bounds, but not a notion of every member of their range
[22:34:11] <techalchemy> that's why I asked the initial clarifying question, "do you mean the overlapping versions in two different ranges?"
[22:34:41] <TemptorSent> It's easy enough to handle the simple cases by sorting on version, but it gets much more "entertaining" when you have something like ["!=1.2.*" ] intersect ["==1"]
[22:35:37] <TemptorSent> Not necessarily, it could be that it requires version 1.x, but 1.2.x had a known bug tha was fixed later and didn't exist before.
[22:36:03] <techalchemy> well you're now talking about a different issue, which is how required python versions are specified in metadata
[22:37:10] <TemptorSent> And what I meant by not simply two overlapping ranges is the fact that SpecifierSets can represent multiple ranges interrupted by discontinuities, which tend to cause most of the problems in implementation.
[22:38:56] <TemptorSent> So I was trying to avoid using ugly brute force methods, but I couldn't find any means of sorting *Specifiers* with respect to each other, which leads to a lot of code to do something that should be trivial :)
[22:39:43] <techalchemy> TemptorSent, https://github.com/sarugaku/requirementslib/blob/master/src/requirementslib/models/markers.py#L479
[22:40:32] <techalchemy> That's the chain of calls I use starting from markers which might contain `python_version` keys
[22:41:21] <techalchemy> TemptorSent, possibly you can start at https://github.com/sarugaku/requirementslib/blob/master/src/requirementslib/models/markers.py#L306
[22:46:16] <techalchemy> oh I had the added concern of the fact that I generate these during dependency resolution and people use the result in their lockfiles, so I have to reduce needless churn
[22:48:36] <TemptorSent> I chose to use integers and assing positive/negative ints to the greater/less side of the ball.
[22:48:45] <techalchemy> I think <= operators always become < operators and > becomes >= unless it's a python version and it's out of bounds
[22:49:08] <techalchemy> i had no real reason for my choices besides aesthetic ones as far as I can remember
[22:50:31] <techalchemy> oh you had the same translation as what I did, yeah I suspect I gave it some thought at the time
[22:51:15] <TemptorSent> but that last one has some irritating corner cases in terms of evaulating the relative versions
[22:51:51] <techalchemy> I think i strip off ".*" from versions which I suspect might cause some problems
[22:53:14] <TemptorSent> ...its a general way of sorting those versions that I'm getting really lost on :)
[22:53:57] <techalchemy> you don't have to sort them, you just have to express them all in the same specifierset
[22:55:46] <TemptorSent> Basically, I just want to know if the python_version matches some SpecifierSet provided by the user.
[22:57:09] <TemptorSent> Dealing with orphaned code written for specilized applications for the indefinite future.
[22:57:38] <TemptorSent> ...meaning I need to be able to mirror all the related packages and keep track of them.
[22:59:41] <TemptorSent> ...given what I'm using it for, I'm tempted to just let the user provide a list of one or two digit python versions to check against in reverse
[23:02:33] <techalchemy> you can always just find out what specific things they need and assume you can just grab whatevers out there
[23:04:08] <TemptorSent> I don't have a list of what I need to support up front, it's entirely possible something comes up in 5 years that needs me do dig up some 10 year old thesis to process.
[23:04:27] <TemptorSent> techalchemy: Are you sure about that? I get a lot of stuff coming back with 404s.
[23:12:05] <toad_polo> People could be removing it actively, though that seems unlikely to be a significant fraction of packages doing that.
[23:14:35] <techalchemy> toad_polo, did everyone agree on a date/time for packaging related discussions / hatchery event and/or should i comment on one of the threads on the subject, I struggle very hard to keep up with discourse
[23:25:02] <techalchemy> toad_polo, ok, likely the day before the conference then? I may need to update my travel request