Rollup merge of #152057 - haampie:hs/fix/bootstrap-respect-jobserver-protocol, r=clubby789 bootstrap: respect POSIX jobserver When bootstrapping Rust, the `-j N` flag was passed to CMake, which was then forwarded to Ninja. This prevents the jobserver from being used, and as a result leads to oversubscription when Rust is just one of the many packages built as part of a larger software stack. Since Cargo and the Rust compiler have long supported the jobserver, it would be good if also bootstrapping Rust itself would participate in the protocol, leading to composable parallelism. This change allows bootstrapping to respect an existing FIFO based jobserver. Old pipe based jobservers are not supported, because they are brittle: currently the Python scripts in bootstrap do not inherit the file descriptors, but do pass on `MAKEFLAGS`, which has lead to errors like "invalid file descriptor" in the past. Because Ninja only supports FIFO based jobservers, it's better to focus on new jobservers only, which shouldn't suffer from the "invalid file descriptor" issue. In summary: * Bootstrap Cargo passes `MAKEFLAGS` verbatim to subprocesses if it advertises a FIFO style jobserver, otherwise it unsets it. This ensures subprocesses respect the jobserver during bootstrap. * `llvm.rs` does not pass `-j` to `cmake` when a FIFO style jobserver is set in `MAKEFLAGS`. This ensures Ninja respects the jobserver. * Bootstrap Cargo no longer unsets `MKFLAGS`: from git blame, GNU Make considered it a historical artifact back in 1992, and it is never read by GNU Make, it's only set for backwards compatibility in case sub-Makefiles read it. --- I've tested this with the [Spack package manager](https://github.com/spack/spack) starting the POSIX jobserver, building node.js and rust in parallel with `-j16`, which looks like this: ```console $ pstree 382710 python3─┬─python3 └─python3─┬─python3─┬─make───make───6*[ccache───g++───cc1plus] │ └─{python3} └─python3─┬─python3.11───bootstrap───cmake───ninja-build───10*[sh───ccache───g++───cc1plus] └─{python3} ``` As you can see there are 10 `g++` processes running for rust, and `6` for node.js, and with a mix of `make` and `ninja` as build tools :). (The only violation I see now is `rust-lld`, but I think that'll be fixed with the LLVM 23 release)
Website | Getting started | Learn | Documentation | Contributing
This is the main source code repository for Rust. It contains the compiler, standard library, and documentation.
Performance: Fast and memory-efficient, suitable for critical services, embedded devices, and easily integrated with other languages.
Reliability: Our rich type system and ownership model ensure memory and thread safety, reducing bugs at compile-time.
Productivity: Comprehensive documentation, a compiler committed to providing great diagnostics, and advanced tooling including package manager and build tool (Cargo), auto-formatter (rustfmt), linter (Clippy) and editor support (rust-analyzer).
Read “Installation” from The Book.
If you really want to install from source (though this is not recommended), see INSTALL.md.
See https://www.rust-lang.org/community for a list of chat platforms and forums.
See CONTRIBUTING.md.
Rust is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0), with portions covered by various BSD-like licenses.
See LICENSE-APACHE, LICENSE-MIT, and COPYRIGHT for details.
The Rust Foundation owns and protects the Rust and Cargo trademarks and logos (the “Rust Trademarks”).
If you want to use these names or brands, please read the Rust language trademark policy.
Third-party logos may be subject to third-party copyrights and trademarks. See Licenses for details.