Log file Viewer

[14:03:46] <Crys> Hi, the data_files option isn't working for me any more with setuptools 17. It works with plain distutils.

[14:06:15] <Crys> As soon as I import setuptools the data_files options gets completely ignored. sdist doesn't include the files.

[14:19:54] <nedbat> is there a way to invoke pip to uninstall a package that won't fail if the package isn't installed?

[14:35:18] <xafer> `pip uninstall package_name || true` ? :-P

[14:36:03] <ronny> nedbat: in what context?

[14:36:18] <nedbat> xafer: hmm, that could work.

[14:36:37] <ronny> (an ansible playbook with pip: name=package state=absent might be a nice overkil for example)

[14:36:44] <nedbat> ronny: I'm migrating an app from django 1.4 to 1.8. That means that South should not be installed. I'm trying to figure out the smoothest path for the script to do that.

[14:37:07] <ronny> nedbat: dont you do immutable deployments?

[14:37:20] <ronny> nedbat: pip-tools is a package that has the tools to handle it

[14:37:21] <nedbat> ronny: this is for all the developer machines.

[14:37:54] <ronny> i see, i tend to remake my virtualenv every time a updat coemes in

[14:38:01] <nedbat> ronny: i'm looking forward to using pip-tools, but we are pinned on an earlier version of pip at the moment.

[14:38:11] <nedbat> ronny: you wouldn't if it involved scipy, numpy and nltk :)

[14:38:31] <ronny> are those still unable to build wheels?

[14:38:57] <nedbat> ronny: not sure. i know we aren't yet configured so that all the developers have access to wheels.

[14:39:00] <ronny> i have a plan for a pep that would make repeated isntalls of all those a matter of seconds

[14:39:22] <ronny> but i dont have the time to write and protytype the tooling

[14:50:02] <xafer> with wheel caching, it only takes a few seconds already

[15:00:21] <ronny> xafer: stil keeping duplicate files around

[15:28:12] <nedbat> ronny: what does that mean?

[15:29:08] <ronny> nedbat: the basic idea is to provide something like a global preinstalled location for wheels that all virtualenvs can use to symlink/link instead of unpacking/installing a wheel

[15:29:29] <nedbat> ronny: I see.

[15:29:40] <ronny> so each package installation would copy no data and only create 2- links and and the scripts for the local env

[19:36:55] <theuni2> dstufft: hey

[19:37:08] <dstufft> theuni2: hi!

[19:37:54] <theuni2> dstufft: a bit busy (my son-to-be started shuffling my schedule around). how can i help?

[19:38:53] <dstufft> theuni2: I have a question, I'm considering moving around URLs on PyPI so that packages get a more unique path so we don't run into problems where a file URL ever is stale. I was wondering if bandersnatch would have any problems if PyPI started using something like /packages/<hash>/<filename> (all of the URLs in /simple/ and stuff would point to the new, correct URLs)

[19:39:30] <theuni2> Hmm. Not sure about that. My first memory is that I have assumptions about paths in there.

[19:39:49] <theuni2> I'm pretty sure I'll follow redirects when downloading

[19:39:58] <theuni2> bandersnatch would then expose the old addresses

[19:40:03] <theuni2> I think it wouldn't break.

[19:40:12] <theuni2> But it wouldn't follow the new scheme automatically either.

[19:40:20] <theuni2> You _could_ try against a test PyPI

[19:40:51] <dstufft> right now we have a bit of a race condition in PyPI around uploading or deleting files, where if someone uploads a new file but the transaction fails (or if the delete fails on a delete) we end up with a bad state which I think using hsahes would fix

[19:41:04] <dstufft> I'll test it of course, i was wondering more if you had any objection to something like that

[19:41:14] <theuni2> well

[19:41:31] <theuni2> well, i think you can solve that internally but still keep the old adresses intact … no/

[19:41:32] <theuni2> ?

[19:42:14] <theuni2> why would those inconsistent states be visible to the outside _except_ for a broken transaction? i.e. why does that state have to live longer than a single request?

[19:42:44] <theuni2> uploads are done to a temporary location and then renamed transactionally?

[19:43:03] <theuni2> just guessing here :)

[19:43:18] <dstufft> Not sure what you mean by solve that internally, fundamentally the filesystem isn't transactional so we either make changes *before* the transaction commits (and risk uploading/deleting a file and having the transaction ultimately fail) or we make changes after the transaction commits (and risk having a file that's supposed to be there but isn't, or a deletion that was supposed to happen but didn't)

[19:43:31] <theuni2> well

[19:43:37] <theuni2> rename is transactional

[19:43:41] <theuni2> s/transactional/atomic

[19:44:03] <theuni2> there are patterns depending on your transaction manager to make this transactional. but if the overall environment isn't prepared for this kind of stuff then it gets hairy.

[19:44:24] <dstufft> We don't use the local FS, we use S3, but even if so, the rename can fail and the DB transaction has already completed

[19:44:28] <theuni2> .oO(stuff that was solved in a system whose name shall not be spoken)

[19:44:52] <theuni2> ah and s3 ultimately becomes directly accessed from the outside?

[19:44:59] <dstufft> from Fastly yea

[19:45:08] <theuni2> i think redirects are the way to go then

[19:45:09] <dstufft> Fastly hit's S3 directly

[19:45:45] <dstufft> Ok

[19:45:46] <dstufft> thanks!

[19:45:49] <dstufft> I'm going to experiment a bit

[19:45:49] <theuni2> so the regular URL should exhibit 404 or 302 (not 301 as it may change back to 404 in the future)

[19:46:21] <theuni2> it's basically about abstracting the physical storage addresses away

[19:46:31] <theuni2> let me know how it works out

[19:46:49] <theuni2> in the meantime, i'll try to _not_ get a baby in the next 7 days while travelling … :)

[19:47:00] <dstufft> heh

[19:47:05] <dstufft> good luck with that :D

[19:47:37] <theuni2> well, we stick to the scouts rule: be prepared

[19:47:40] <theuni2> so we are.

[19:47:48] <theuni2> everything else is thermodynamics, I guess

[19:47:59] <theuni2> good luck with pypi :)

[19:48:03] <dstufft> see ya :)