Rollup merge of #140763 - sayantn:test-amx, r=dianqk Change codegen of LLVM intrinsics to be name-based, and add llvm linkage support for `bf16(xN)` and `i1xN` *[View all comments](https://triagebot.infra.rust-lang.org/gh-comments/rust-lang/rust/pull/140763)* This PR changes how LLVM intrinsics are codegen # Explanation of the changes ## Current procedure This is the same for all functions, LLVM intrinsics are _not_ treated specially - We get the LLVM Type of a function simply using the argument types. For example, the following function ```rust #[link_name = "llvm.sqrt.f32"] fn sqrtf32(a: f32) -> f32; ``` will have LLVM type simply `f32 (f32)` due to the Rust signature ### Pros - Simpler to implement, no extra complexity involved due to LLVM intrinsics ### Cons - LLVM intrinsics have a well-defined signature, completely defined by their name (and if it is overloaded, the type parameters). So, this process of converting Rust signatures to LLVM signatures may not work, for example the following code generates LLVM IR without any problem ```rust #[link_name = "llvm.sqrt.f32"] fn sqrtf32(a: i32) -> f32; ``` but the generated LLVM IR is invalid, because it has wrong signature for the intrinsic ([Godbolt](https://godbolt.org/z/6ff9hrcd5), adding `-Zverify-llvm-ir` to it will fail compilation). I would expect this code to not compile at all instead of generating invalid IR. - LLVM intrinsics that have types in their signature that can't be accessed from Rust (notable examples are the AMX intrinsics that have the `x86amx` type, and (almost) all intrinsics that have vectors of `i1` types) can't be linked to at all. This is a (major?) roadblock in the AMX and AVX512 support in stdarch. - If code uses an non-existing LLVM intrinsic, even `-Zverify-llvm-ir` won't complain. Eventually it will error out due to the non-existing function (courtesy of the linker). I don't think this is a behavior we want. ## What this PR does - When linking to **non-overloaded** intrinsics, we use the function `LLVMIntrinsicGetType` to directly get the function type of the intrinsic from LLVM. - We then use this LLVM definition to _verify_ the Rust signature, and emit a proper error if it doesn't match, instead of silently emitting invalid IR. - Lint if linking to deprecated or invalid LLVM intrinsics > [!NOTE] > This PR only focuses on non-overloaded intrinsics, overloaded can be done in a future PR Regardless, the undermentioned functionalities work for **all** intrinsics - If we can't find the intrinsic, we check if it has been `AutoUpgrade`d by LLVM. If not, that means it is an invalid intrinsic, and we error out. - Don't allow intrinsics from other archs to be declared, e.g. error out if an AArch64 intrinsic is declared when we are compiling for x86 ### Pros - It is now not possible (or at least, it would require _significantly_ more leaps and bounds) to introduce invalid IR using **non-overloaded** LLVM intrinsics. - As we are now doing the matching of Rust signatures to LLVM intrinsics ourselves, we can now add bypasses to enable linking to such non-Rust types (e.g. matching 8192-bit vectors to `x86amx` and injecting `llvm.x86.cast.vector.to.tile` and `llvm.x86.cast.tile.to.vector`s in callsite) > [!NOTE] > I don't intend for these bypasses to be permanent. A better approach will be introducing a `bf16` type in Rust, and allowing `repr(simd)` with `bool`s to get Rust-native `i1xN`s. These are meant to be short-time, as I mentioned, "bypass"es. They shouldn't cause any major breakage even if removed, as `link_llvm_intrinsics` is perma-unstable. This PR adds bypasses for `bf16` (via `i16`), `bf16xN` (via `i16xN`) and `i1xN` (via `iM`, where `M` is the smallest power of 2 s.t. `M >= N`, unless `N <= 4`, where we use `M = 8`). This will unblock AVX512-VP2INTERSECT and a lot of bf16 intrinsics in stdarch. This PR also automatically destructures structs if the types don't exactly match (this is required for us to start emitting hard errors on mismmatches). ### Cons - This only works for non-overloaded intrinsics (at least for now). Improving this to work with overloaded intrinsics too will involve significantly more work. # Possible ways to extend this to overloaded intrinsics (future) ## Parse the mangled intrinsic name to get the type parameters LLVM has a stable mangling of intrinsic names with type parameters (in `LLVMIntrinsicCopyOverloadedName2`), so we can parse the name to get the type parameters, and then just do the same thing. ### Pros - For _most_ intrinsics, this will work perfectly, and is a easy way to do this. ### Cons - The LLVM mangling is not perfectly reversible. When we have `TargetExt` types or identified structs, their name is a part of the mangling, making it impossible to reverse. Even more complexities arise when there are unnamed identified structs, as LLVM adds more mangling to the names. - @nikic's work on LLVM intrinsics will remove the name mangling, making this approach impossible ## Use the `IITDescriptor` table and the Rust function signature We can use the base name to get the `IITDescriptor`s of the corresponding intrinsic, and then manually implement the _matching_ logic based on the Rust signature. ### Pros - Doesn't have the above mentioned limitation of the parsing approach, has correct behavior even when there are identified structs and `TargetExt` types. Also, fun fact, Rust exports all struct types as literal structs (unless it is emitting LLVM IR, then it always uses named identified structs, with mangled names) ### Cons - **Doesn't** actually use the type parameters in the name, only uses the base name and the Rust signature to get the llvm signature (although we _can_ check that it is the correct name). It means there would be no way to (for example) link against `llvm.sqrt.bf16` until we have `bf16` types in Rust. Because if we are using `u16`s (or any other type) as `bf16`s, then the matcher will deduce that the signature is `u16 (u16)` not `bf16 (bf16)` (which would lead to an error because `u16` is not a valid type parameter for `llvm.sqrt`), even though the intended type parameter is specified in the name. - Much more complex, and hard to maintain as LLVM gets new `IITDescriptorKind`s These 2 approaches might give different results for same function. Let's take ```rust #[link_name = "llvm.is.constant.bf16"] fn foo(a: u16) -> bool ``` The name-based approach will decide that the type parameter is `bf16`, and the LLVM signature is `i1 (bf16)` and will inject some bitcasts at callsite. The `IITDescriptor`-based approach will decide that the LLVM signature is `i1 (u16)`, and will see that the name given doesn't match the expected name (`llvm.is.constant.u16`), and will error out. Reviews are welcome, as this is my first time _actually_ contributing to `rustc` @rustbot label T-compiler A-codegen A-LLVM r? codegen
Website | Getting started | Learn | Documentation | Contributing
This is the main source code repository for Rust. It contains the compiler, standard library, and documentation.
Performance: Fast and memory-efficient, suitable for critical services, embedded devices, and easily integrated with other languages.
Reliability: Our rich type system and ownership model ensure memory and thread safety, reducing bugs at compile-time.
Productivity: Comprehensive documentation, a compiler committed to providing great diagnostics, and advanced tooling including package manager and build tool (Cargo), auto-formatter (rustfmt), linter (Clippy) and editor support (rust-analyzer).
Read “Installation” from The Book.
If you really want to install from source (though this is not recommended), see INSTALL.md.
See https://www.rust-lang.org/community for a list of chat platforms and forums.
See CONTRIBUTING.md.
For a detailed explanation of the compiler's architecture and how to begin contributing, see the rustc-dev-guide.
Rust is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0), with portions covered by various BSD-like licenses.
See LICENSE-APACHE, LICENSE-MIT, and COPYRIGHT for details.
The Rust Foundation owns and protects the Rust and Cargo trademarks and logos (the “Rust Trademarks”).
If you want to use these names or brands, please read the Rust language trademark policy.
Third-party logos may be subject to third-party copyrights and trademarks. See Licenses for details.