28 December 2020

Rust build hangs

About a month ago, I had a weird problem with the Rust compiler. I recompiled my project a lot to test some minor changes I had done to the code. About every other time, the compiler would hang for more than a minute before finishing up. Of course, this was very disruptive to incremental development.

Naturally, this is the perfect opportunity to make dank memes and jokes about slow Rust compilation times, but I'll spare you with that here. Frustrated but intrigued (a very productive state of mind), I started to analyse this performance problem. I only changed one letter in main.rs and triggered an incremental compilation. It took 10 seconds (this is already very long in my opinion, but that's another story entirely). Then I repeated this process, only changing a single character back and forth. Seemingly randomly, it either took 10 seconds or about 80 seconds. How weird. At this point, I started to think that there might be a bug in the Rust compiler. Maybe a race condition that causes it to become stuck for a while, given the right circumstances.

Unfortunately, speculation doesn't make problems go away. I wanted to understand what's going on and possibly file a detailed bug report. What is the compiler doing during these 70 additional seconds it takes? The task manager promptly revealed the answer: Nothing.

That's weird. So the compiler is blocked on something, either internally or because of system resources. In the Performance tab of the Task Manager, you can access the Resource Monitor to show you a more detailed view of what system resources an application uses. This is interesting: rustc.exe is only actively accessing one file while it's hanging: C:\$LogFile (NTFS Volume Log).

A quick google search lead me to the advice that I should try to optimise my drive. The utility to do this used to be called Disk Defragmenter, but since the advent of SSDs, which shouldn't be defragmented, Microsoft renamed it to Optimize Drives. For SSDs, this utility issues the TRIM command which informs the SSD about which blocks of data are no longer considered in use and can be wiped internally. My disk needed optimisation, so I optimised it. It took less than a second.

After that, the problem with the stuck compilation could no longer be reproduced. I'm still puzzled as to why this issue caused the Rust compiler to stall for so long. During that time, the disk could have been optimised dozens of times. I also wondered about disk health. Professionally, I troubleshoot broken flash storage a lot, and one of the telltale signs of corrupted internal data structures is when it takes minutes to access a block. However, a month after optimisation, my disk still seems to be perfectly fine.

Once again, the compiler was not at fault. Note that the disk I had been using was almost full (500GB capacity, 50GB free). Rust compiler artefacts are quite large. Even for smaller projects with just a handful of dependencies, they reach tens of Gigabytes. I know that debug data can be large, but I find the size of these artefacts to be inappropriately large. When I download and build a bunch of toy Rust projects, suddenly my entire disk space is gone. I guess a toolchain cannot be perfect. Either way, my initial problem was solved.