Log file Viewer

[22:41:35] <techalchemy> chrahunt, I was looking over your ramdisk additions to pip -- did that actually speed up test runs?

[22:42:04] <chrahunt> techalchemy, yes, by about 10 minutes

[22:42:11] <techalchemy> sick

[22:42:18] <techalchemy> chrahunt, where did you get the script for that?

[22:43:13] <chrahunt> I pieced it together from advice online and Powershell best practices lists - I don't think it exists anywhere in one piece like that

[22:44:36] <techalchemy> nicely done then! I've wanted that for ages but wasn't sure it was even possible on azure, did you experiment with different memory allocations?

[22:45:33] <chrahunt> I didn't - just confirmed that our tests didn't use /that/ much memory or /that/ much space in TEMP and then picked a reasonable round value

[22:45:56] <techalchemy> hah ok

[22:46:16] <techalchemy> you're doing some great work

[22:46:40] <chrahunt> thanks

[22:46:56] <chrahunt> The biggest killer feature, if you're interested, is profiling on Azure Pipelines: https://github.com/chrahunt/pip/pull/1/files

[22:47:20] <techalchemy> I had to reread the entire codebase for pip-shims and I felt like even though everything changed so much it was substantially easier to reason about

[22:47:59] <techalchemy> hrm is that new? I'm not at all familiar with that

[22:48:24] <techalchemy> oh you wrote it, i see :p

[22:52:36] <chrahunt> yup, the generated files can be loaded up into Windows Performance Analyzer or exported to log files. In pip's case it showed some of our biggest wait periods had NTFS journal lock acquire and general filesystem operations on the stack, implying faster disks might help

[22:59:54] <techalchemy> chrahunt, that's super helpful actually. I've never used that tool as I don't have tons of windows profiling knowledge but the issue I usually run up against is that we spin off subprocesses in massive numbers to pip which uses a custom temporary directory implementation and due to the process hierarchy if we do this with vcs dependencies a lot of times git never relinquishes locks

[23:01:09] <techalchemy> might be able to do some better digging on that this way

[23:01:46] <techalchemy> currently i have a bunch of workarounds to try and reclaim ownership of .git directories if cleanup fails

[23:05:28] <chrahunt> techalchemy, have you used Process Monitor? That's the first thing I would try since it gives much more straightforward information

[23:05:52] <techalchemy> chrahunt, is that the gui thing? if so yeah

[23:06:09] <chrahunt> https://docs.microsoft.com/en-us/sysinternals/downloads/procmon

[23:06:26] <techalchemy> i mean i have a custom ctypes based implementation that does system calls to find all of the children of the caller process and forcibly terminate them

[23:08:14] <techalchemy> but there's a weird race condition i think where windows will allow the process to complete before locks are released and it causes cleanup to fail, and since pip's tempdirs don't use weakrefs the failure isn't handled at interpreter exit

[23:08:36] <techalchemy> I think my current fix involves a patched version of pip that we vendor which just uses weakrefs to clean up after themselves