Merge pull request #2546 from lolbinarycat/patch-7
Update bootstrapping docs to account for new stage meanings
diff --git a/rust-version b/rust-version
index 6ec700b..a399b5c 100644
--- a/rust-version
+++ b/rust-version
@@ -1 +1 @@
-6bcdcc73bd11568fd85f5a38b58e1eda054ad1cd
+425a9c0a0e365c0b8c6cfd00c2ded83a73bed9a0
diff --git a/src/SUMMARY.md b/src/SUMMARY.md
index 9ded467..a161273 100644
--- a/src/SUMMARY.md
+++ b/src/SUMMARY.md
@@ -103,11 +103,11 @@
- [The `rustdoc-json` test suite](./rustdoc-internals/rustdoc-json-test-suite.md)
- [GPU offload internals](./offload/internals.md)
- [Installation](./offload/installation.md)
+ - [Usage](./offload/usage.md)
- [Autodiff internals](./autodiff/internals.md)
- [Installation](./autodiff/installation.md)
- [How to debug](./autodiff/debugging.md)
- [Autodiff flags](./autodiff/flags.md)
- - [Current limitations](./autodiff/limitations.md)
# Source Code Representation
diff --git a/src/autodiff/limitations.md b/src/autodiff/limitations.md
deleted file mode 100644
index 90afbd5..0000000
--- a/src/autodiff/limitations.md
+++ /dev/null
@@ -1,27 +0,0 @@
-# Current limitations
-
-## Safety and Soundness
-
-Enzyme currently assumes that the user passes shadow arguments (`dx`, `dy`, ...) of appropriate size. Under Reverse Mode, we additionally assume that shadow arguments are mutable. In Reverse Mode we adjust the outermost pointer or reference to be mutable. Therefore `&f32` will receive the shadow type `&mut f32`. However, we do not check length for other types than slices (e.g. enums, Vec). We also do not enforce mutability of inner references, but will warn if we recognize them. We do intend to add additional checks over time.
-
-## ABI adjustments
-
-In some cases, a function parameter might get lowered in a way that we currently don't handle correctly, leading to a compile time type mismatch in the `rustc_codegen_llvm` backend. Here are some [examples](https://github.com/EnzymeAD/rust/issues/105).
-
-## Compile Times
-
-Enzyme will often achieve excellent runtime performance, but might increase your compile time by a large factor. For Rust, we already have made significant improvements and have a list of further improvements planed - please reach out if you have time to help here.
-
-### Type Analysis
-
-Most of the times, Type Analysis (TA) is the reason of large (>5x) compile time increases when using Enzyme. This poster explains why we need to run Type Analysis in the bottom left part: [Poster Link](https://c.wsmoses.com/posters/Enzyme-llvmdev.pdf).
-
-We intend to increase the number of locations where we pass down Type information based on Rust types, which in turn will reduce the number of locations where Enzyme has to run Type Analysis, which will help compile times.
-
-### Duplicated Optimizations
-
-The key reason for Enzyme offering often excellent performance is that Enzyme differentiates already optimized LLVM-IR. However, we also (have to) run LLVM's optimization pipeline after differentiating, to make sure that the code which Enzyme generates is optimized properly. As a result you should have excellent runtime performance (please fill an issue if not), but at a compile time cost for running optimizations twice.
-
-### Fat-LTO
-
-The usage of `#[autodiff(...)]` currently requires compiling your project with Fat-LTO. We technically only need LTO if the function being differentiated calls functions in other compilation units. Therefore, other solutions are possible, but this is the most simple one to get started.
diff --git a/src/building/bootstrapping/debugging-bootstrap.md b/src/building/bootstrapping/debugging-bootstrap.md
index 9c5ebbd..fb90c0f 100644
--- a/src/building/bootstrapping/debugging-bootstrap.md
+++ b/src/building/bootstrapping/debugging-bootstrap.md
@@ -1,120 +1,100 @@
# Debugging bootstrap
-There are two main ways to debug bootstrap itself. The first is through println logging, and the second is through the `tracing` feature.
-
-> FIXME: this section should be expanded
+There are two main ways of debugging (and profiling bootstrap). The first is through println logging, and the second is through the `tracing` feature.
## `println` logging
Bootstrap has extensive unstructured logging. Most of it is gated behind the `--verbose` flag (pass `-vv` for even more detail).
-If you want to know which `Step` ran a command, you could invoke bootstrap like so:
+If you want to see verbose output of executed Cargo commands and other kinds of detailed logs, pass `-v` or `-vv` when invoking bootstrap. Note that the logs are unstructured and may be overwhelming.
```
$ ./x dist rustc --dry-run -vv
learning about cargo
running: RUSTC_BOOTSTRAP="1" "/home/jyn/src/rust2/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "metadata" "--format-version" "1" "--no-deps" "--manifest-path" "/home/jyn/src/rust2/Cargo.toml" (failure_mode=Exit) (created at src/bootstrap/src/core/metadata.rs:81:25, executed at src/bootstrap/src/core/metadata.rs:92:50)
running: RUSTC_BOOTSTRAP="1" "/home/jyn/src/rust2/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "metadata" "--format-version" "1" "--no-deps" "--manifest-path" "/home/jyn/src/rust2/library/Cargo.toml" (failure_mode=Exit) (created at src/bootstrap/src/core/metadata.rs:81:25, executed at src/bootstrap/src/core/metadata.rs:92:50)
-> Assemble { target_compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu } }
- > Libdir { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu }, target: x86_64-unknown-linux-gnu }
- > Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu }, force_recompile: false }
-Removing sysroot /home/jyn/src/rust2/build/tmp-dry-run/x86_64-unknown-linux-gnu/stage1 to avoid caching bugs
- < Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu }, force_recompile: false }
- < Libdir { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu }, target: x86_64-unknown-linux-gnu }
...
```
-This will go through all the recursive dependency calculations, where `Step`s internally call `builder.ensure()`, without actually running cargo or the compiler.
-
-In some cases, even this may not be enough logging (if so, please add more!). In that case, you can omit `--dry-run`, which will show the normal output inline with the debug logging:
-
-```
- c Sysroot { compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu }, force_recompile: false }
-using sysroot /home/jyn/src/rust2/build/x86_64-unknown-linux-gnu/stage0-sysroot
-Building stage0 library artifacts (x86_64-unknown-linux-gnu)
-running: cd "/home/jyn/src/rust2" && env ... RUSTC_VERBOSE="2" RUSTC_WRAPPER="/home/jyn/src/rust2/build/bootstrap/debug/rustc" "/home/jyn/src/rust2/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-Zbinary-dep-depinfo" "-Zroot-dir=/home/jyn/src/rust2" "-v" "-v" "--manifest-path" "/home/jyn/src/rust2/library/sysroot/Cargo.toml" "--message-format" "json-render-diagnostics"
- 0.293440230s INFO prepare_target{force=false package_id=sysroot v0.0.0 (/home/jyn/src/rust2/library/sysroot) target="sysroot"}: cargo::core::compiler::fingerprint: fingerprint error for sysroot v0.0.0 (/home/jyn/src/rust2/library/sysroot)/Build/TargetInner { name_inferred: true, ..: lib_target("sysroot", ["lib"], "/home/jyn/src/rust2/library/sysroot/src/lib.rs", Edition2021) }
-...
-```
-
-In most cases this should not be necessary.
-
-TODO: we should convert all this to structured logging so it's easier to control precisely.
-
## `tracing` in bootstrap
-Bootstrap has conditional [`tracing`][tracing] setup to provide structured logging.
+Bootstrap has a conditional `tracing` feature, which provides the following features:
+- It enables structured logging using [`tracing`][tracing] events and spans.
+- It generates a [Chrome trace file] that can be used to visualize the hierarchy and durations of executed steps and commands.
+ - You can open the generated `chrome-trace.json` file using Chrome, on the `chrome://tracing` tab, or e.g. using [Perfetto].
+- It generates [GraphViz] graphs that visualize the dependencies between executed steps.
+ - You can open the generated `step-graph-*.dot` file using e.g. [xdot] to visualize the step graph, or use e.g. `dot -Tsvg` to convert the GraphViz file to an SVG file.
+- It generates a command execution summary, which shows which commands were executed, how many of their executions were cached, and what commands were the slowest to run.
+ - The generated `command-stats.txt` file is in a simple human-readable format.
+
+The structured logs will be written to standard error output (`stderr`), while the other outputs will be stored in files in the `<build-dir>/bootstrap-trace/<pid>` directory. For convenience, bootstrap will also create a symlink to the latest generated trace output directory at `<build-dir>/bootstrap-trace/latest`.
+
+> Note that if you execute bootstrap with `--dry-run`, the tracing output directory might change. Bootstrap will always print a path where the tracing output files were stored at the end of its execution.
[tracing]: https://docs.rs/tracing/0.1.41/tracing/index.html
+[Chrome trace file]: https://www.chromium.org/developers/how-tos/trace-event-profiling-tool/
+[Perfetto]: https://ui.perfetto.dev/
+[GraphViz]: https://graphviz.org/doc/info/lang.html
+[xdot]: https://github.com/jrfonseca/xdot.py
### Enabling `tracing` output
-Bootstrap will conditionally build `tracing` support and enable `tracing` output if the `BOOTSTRAP_TRACING` env var is set.
+To enable the conditional `tracing` feature, run bootstrap with the `BOOTSTRAP_TRACING` environment variable.
-#### Basic usage
-
-Example basic usage[^just-trace]:
-
-[^just-trace]: It is not recommended to use *just* `BOOTSTRAP_TRACING=TRACE` because that will dump *everything* at `TRACE` level, including logs intentionally gated behind custom targets as they are too verbose even for `TRACE` level by default.
+[tracing_subscriber filter]: https://docs.rs/tracing-subscriber/latest/tracing_subscriber/filter/struct.EnvFilter.html
```bash
-$ BOOTSTRAP_TRACING=bootstrap=TRACE ./x build library --stage 1
+$ BOOTSTRAP_TRACING=trace ./x build library --stage 1
```
Example output[^unstable]:
```
-$ BOOTSTRAP_TRACING=bootstrap=TRACE ./x check src/bootstrap/
+$ BOOTSTRAP_TRACING=trace ./x build library --stage 1 --dry-run
Building bootstrap
- Compiling bootstrap v0.0.0 (/home/joe/repos/rust/src/bootstrap)
- Finished `dev` profile [unoptimized] target(s) in 2.74s
- DEBUG bootstrap parsing flags
- bootstrap::core::config::flags::Flags::parse args=["check", "src/bootstrap/"]
- DEBUG bootstrap parsing config based on flags
- DEBUG bootstrap creating new build based on config
- bootstrap::Build::build
- TRACE bootstrap setting up job management
- TRACE bootstrap downloading rustfmt early
- bootstrap::handling hardcoded subcommands (Format, Suggest, Perf)
- DEBUG bootstrap not a hardcoded subcommand; returning to normal handling, cmd=Check { all_targets: false }
- DEBUG bootstrap handling subcommand normally
- bootstrap::executing real run
- bootstrap::(1) executing dry-run sanity-check
- bootstrap::(2) executing actual run
-Checking stage0 library artifacts (x86_64-unknown-linux-gnu)
- Finished `release` profile [optimized + debuginfo] target(s) in 0.04s
-Checking stage0 compiler artifacts {rustc-main, rustc_abi, rustc_arena, rustc_ast, rustc_ast_ir, rustc_ast_lowering, rustc_ast_passes, rustc_ast_pretty, rustc_attr_data_structures, rustc_attr_parsing, rustc_baked_icu_data, rustc_borrowck, rustc_builtin_macros, rustc_codegen_llvm, rustc_codegen_ssa, rustc_const_eval, rustc_data_structures, rustc_driver, rustc_driver_impl, rustc_error_codes, rustc_error_messages, rustc_errors, rustc_expand, rustc_feature, rustc_fluent_macro, rustc_fs_util, rustc_graphviz, rustc_hir, rustc_hir_analysis, rustc_hir_pretty, rustc_hir_typeck, rustc_incremental, rustc_index, rustc_index_macros, rustc_infer, rustc_interface, rustc_lexer, rustc_lint, rustc_lint_defs, rustc_llvm, rustc_log, rustc_macros, rustc_metadata, rustc_middle, rustc_mir_build, rustc_mir_dataflow, rustc_mir_transform, rustc_monomorphize, rustc_next_trait_solver, rustc_parse, rustc_parse_format, rustc_passes, rustc_pattern_analysis, rustc_privacy, rustc_query_impl, rustc_query_system, rustc_resolve, rustc_sanitizers, rustc_serialize, rustc_session, rustc_smir, rustc_span, rustc_symbol_mangling, rustc_target, rustc_trait_selection, rustc_traits, rustc_transmute, rustc_ty_utils, rustc_type_ir, rustc_type_ir_macros, stable_mir} (x86_64-unknown-linux-gnu)
- Finished `release` profile [optimized + debuginfo] target(s) in 0.23s
-Checking stage0 bootstrap artifacts (x86_64-unknown-linux-gnu)
- Checking bootstrap v0.0.0 (/home/joe/repos/rust/src/bootstrap)
- Finished `release` profile [optimized + debuginfo] target(s) in 0.64s
- DEBUG bootstrap checking for postponed test failures from `test --no-fail-fast`
-Build completed successfully in 0:00:08
-```
-
-#### Controlling tracing output
-
-The env var `BOOTSTRAP_TRACING` accepts a [`tracing` env-filter][tracing-env-filter].
-
-There are two orthogonal ways to control which kind of tracing logs you want:
-
-1. You can specify the log **level**, e.g. `DEBUG` or `TRACE`.
-2. You can also control the log **target**, e.g. `bootstrap` or `bootstrap::core::config` vs custom targets like `CONFIG_HANDLING`.
- - Custom targets are used to limit what is output when `BOOTSTRAP_TRACING=bootstrap=TRACE` is used, as they can be too verbose even for `TRACE` level by default. Currently used custom targets:
- - `CONFIG_HANDLING`
-
-The `TRACE` filter will enable *all* `trace` level or less verbose level tracing output.
-
-You can of course combine them (custom target logs are typically gated behind `TRACE` log level additionally):
-
-```bash
-$ BOOTSTRAP_TRACING=CONFIG_HANDLING=TRACE ./x build library --stage 1
+ Finished `dev` profile [unoptimized] target(s) in 0.05s
+15:56:52.477 INFO > tool::LibcxxVersionTool {target: x86_64-unknown-linux-gnu} (builder/mod.rs:1715)
+15:56:52.575 INFO > compile::Assemble {target_compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }} (builder/mod.rs:1715)
+15:56:52.575 INFO > tool::Compiletest {compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, target: x86_64-unknown-linux-gnu} (builder/mod.rs:1715)
+15:56:52.576 INFO > tool::ToolBuild {build_compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, target: x86_64-unknown-linux-gnu, tool: "compiletest", path: "src/tools/compiletest", mode: ToolBootstrap, source_type: InTree, extra_features: [], allow_features: "internal_output_capture", cargo_args: [], artifact_kind: Binary} (builder/mod.rs:1715)
+15:56:52.576 INFO > builder::Libdir {compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, target: x86_64-unknown-linux-gnu} (builder/mod.rs:1715)
+15:56:52.576 INFO > compile::Sysroot {compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, force_recompile: false} (builder/mod.rs:1715)
+15:56:52.578 INFO > compile::Assemble {target_compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }} (builder/mod.rs:1715)
+15:56:52.578 INFO > tool::Compiletest {compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, target: x86_64-unknown-linux-gnu} (builder/mod.rs:1715)
+15:56:52.578 INFO > tool::ToolBuild {build_compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, target: x86_64-unknown-linux-gnu, tool: "compiletest", path: "src/tools/compiletest", mode: ToolBootstrap, source_type: InTree, extra_features: [], allow_features: "internal_output_capture", cargo_args: [], artifact_kind: Binary} (builder/mod.rs:1715)
+15:56:52.578 INFO > builder::Libdir {compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, target: x86_64-unknown-linux-gnu} (builder/mod.rs:1715)
+15:56:52.578 INFO > compile::Sysroot {compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, force_recompile: false} (builder/mod.rs:1715)
+ Finished `release` profile [optimized] target(s) in 0.11s
+Tracing/profiling output has been written to <src-root>/build/bootstrap-trace/latest
+Build completed successfully in 0:00:00
```
[^unstable]: This output is always subject to further changes.
+#### Controlling tracing output
+
+The environment variable `BOOTSTRAP_TRACING` accepts a [`tracing_subscriber` filter][tracing-env-filter]. If you set `BOOTSTRAP_TRACING=trace`, you will enable all logs, but that can be overwhelming. You can thus use the filter to reduce the amount of data logged.
+
+There are two orthogonal ways to control which kind of tracing logs you want:
+
+1. You can specify the log **level**, e.g. `debug` or `trace`.
+ - If you select a level, all events/spans with an equal or higher priority level will be shown.
+2. You can also control the log **target**, e.g. `bootstrap` or `bootstrap::core::config` or a custom target like `CONFIG_HANDLING` or `STEP`.
+ - Custom targets are used to limit what kinds of spans you are interested in, as the `BOOTSTRAP_TRACING=trace` output can be quite verbose. Currently, you can use the following custom targets:
+ - `CONFIG_HANDLING`: show spans related to config handling
+ - `STEP`: show all executed steps. Note that executed commands have `info` event level.
+ - `COMMAND`: show all executed commands. Note that executed commands have `trace` event level.
+
+You can of course combine them (custom target logs are typically gated behind `TRACE` log level additionally):
+
+```bash
+$ BOOTSTRAP_TRACING=CONFIG_HANDLING=trace,STEP=info,COMMAND=trace ./x build library --stage 1
+```
+
[tracing-env-filter]: https://docs.rs/tracing-subscriber/0.3.19/tracing_subscriber/filter/struct.EnvFilter.html
+Note that the level that you specify using `BOOTSTRAP_TRACING` also has an effect on the spans that will be recorded in the Chrome trace file.
+
##### FIXME(#96176): specific tracing for `compiler()` vs `compiler_for()`
The additional targets `COMPILER` and `COMPILER_FOR` are used to help trace what
@@ -123,12 +103,6 @@
[cleanup-compiler-for]: https://github.com/rust-lang/rust/issues/96176
-### Rendering step graph
-
-When you run bootstrap with the `BOOTSTRAP_TRACING` environment variable configured, bootstrap will automatically output a DOT file that shows all executed steps and their dependencies. The files will have a prefix `bootstrap-steps`. You can use e.g. `xdot` to visualize the file or e.g. `dot -Tsvg` to convert the DOT file to a SVG file.
-
-A separate DOT file will be outputted for dry-run and non-dry-run execution.
-
### Using `tracing` in bootstrap
Both `tracing::*` macros and the `tracing::instrument` proc-macro attribute need to be gated behind `tracing` feature. Examples:
@@ -149,15 +123,6 @@
todo!()
}
- #[cfg_attr(
- feature = "tracing",
- instrument(
- level = "trace",
- name = "Foo::run",
- skip_all,
- fields(compiler = ?builder.compiler),
- ),
- )]
fn run(self, builder: &Builder<'_>) -> Self::Output {
trace!(?run, "entered Foo::run");
@@ -172,21 +137,6 @@
- Explicitly pick an instrumentation name via `name = ".."` to distinguish between e.g. `run` of different steps.
- Take care to not cause diverging behavior via tracing, e.g. building extra things only when tracing infra is enabled.
-### Profiling bootstrap
-
-You can set the `BOOTSTRAP_PROFILE=1` environment variable to enable command execution profiling during bootstrap. This generates:
-
-* A Chrome trace file (for visualization in `chrome://tracing` or [Perfetto](https://ui.perfetto.dev)) if tracing is enabled via `BOOTSTRAP_TRACING=COMMAND=trace`
-* A plain-text summary file, `bootstrap-profile-{pid}.txt`, listing all commands sorted by execution time (slowest first), along with cache hits and working directories
-
-Note: the `.txt` report is always generated when `BOOTSTRAP_PROFILE=1` is set — tracing is not required.
-
-Example usage:
-
-```bash
-$ BOOTSTRAP_PROFILE=1 BOOTSTRAP_TRACING=COMMAND=trace ./x build library
-```
-
### rust-analyzer integration?
Unfortunately, because bootstrap is a `rust-analyzer.linkedProjects`, you can't ask r-a to check/build bootstrap itself with `tracing` feature enabled to get relevant completions, due to lack of support as described in <https://github.com/rust-lang/rust-analyzer/issues/8521>.
diff --git a/src/building/optimized-build.md b/src/building/optimized-build.md
index 863ed97..46def66 100644
--- a/src/building/optimized-build.md
+++ b/src/building/optimized-build.md
@@ -118,7 +118,7 @@
```
3. Run the tool with the `local` mode and provide necessary parameters:
```bash
- ./build/host/stage0-tools-bin/opt-dist local \
+ ./build/host/stage1-tools-bin/opt-dist local \
--target-triple <target> \ # select target, e.g. "x86_64-unknown-linux-gnu"
--checkout-dir <path> \ # path to rust checkout, e.g. "."
--llvm-dir <path> \ # path to built LLVM toolchain, e.g. "/foo/bar/llvm/install"
diff --git a/src/img/coverage-branch-counting-01.png b/src/img/coverage-branch-counting-01.png
index c445f35..7c6c845 100644
--- a/src/img/coverage-branch-counting-01.png
+++ b/src/img/coverage-branch-counting-01.png
Binary files differ
diff --git a/src/img/dataflow-graphviz-example.png b/src/img/dataflow-graphviz-example.png
index 718411a..7baa37e 100644
--- a/src/img/dataflow-graphviz-example.png
+++ b/src/img/dataflow-graphviz-example.png
Binary files differ
diff --git a/src/img/github-cli.png b/src/img/github-cli.png
index c3b0e77..88ba95f 100644
--- a/src/img/github-cli.png
+++ b/src/img/github-cli.png
Binary files differ
diff --git a/src/img/github-whitespace-changes.png b/src/img/github-whitespace-changes.png
index 9a19a10..e235a30 100644
--- a/src/img/github-whitespace-changes.png
+++ b/src/img/github-whitespace-changes.png
Binary files differ
diff --git a/src/img/llvm-cov-show-01.png b/src/img/llvm-cov-show-01.png
index 35f0459..ce4dec1 100644
--- a/src/img/llvm-cov-show-01.png
+++ b/src/img/llvm-cov-show-01.png
Binary files differ
diff --git a/src/img/other-peoples-commits.png b/src/img/other-peoples-commits.png
index e4fc2c7..0c949d8 100644
--- a/src/img/other-peoples-commits.png
+++ b/src/img/other-peoples-commits.png
Binary files differ
diff --git a/src/img/rustbot-submodules.png b/src/img/rustbot-submodules.png
index c2e6937..c099fdf 100644
--- a/src/img/rustbot-submodules.png
+++ b/src/img/rustbot-submodules.png
Binary files differ
diff --git a/src/img/submodule-conflicts.png b/src/img/submodule-conflicts.png
index e90a6bb..5d4caf0 100644
--- a/src/img/submodule-conflicts.png
+++ b/src/img/submodule-conflicts.png
Binary files differ
diff --git a/src/img/wpa-initial-memory.png b/src/img/wpa-initial-memory.png
index b602066..177d92c 100644
--- a/src/img/wpa-initial-memory.png
+++ b/src/img/wpa-initial-memory.png
Binary files differ
diff --git a/src/img/wpa-stack.png b/src/img/wpa-stack.png
index 29eb5a5..a4a7135 100644
--- a/src/img/wpa-stack.png
+++ b/src/img/wpa-stack.png
Binary files differ
diff --git a/src/offload/installation.md b/src/offload/installation.md
index b376e96..d1ebf33 100644
--- a/src/offload/installation.md
+++ b/src/offload/installation.md
@@ -1,6 +1,6 @@
# Installation
-In the future, `std::offload` should become available in nightly builds for users. For now, everyone still needs to build rustc from source.
+`std::offload` is partly available in nightly builds for users. For now, everyone however still needs to build rustc from source to use all features of it.
## Build instructions
@@ -42,30 +42,3 @@
```
./x test --stage 1 tests/codegen-llvm/gpu_offload
```
-
-## Usage
-It is important to use a clang compiler build on the same llvm as rustc. Just calling clang without the full path will likely use your system clang, which probably will be incompatible.
-```
-/absolute/path/to/rust/build/x86_64-unknown-linux-gnu/stage1/bin/rustc --edition=2024 --crate-type cdylib src/main.rs --emit=llvm-ir -O -C lto=fat -Cpanic=abort -Zoffload=Enable
-/absolute/path/to/rust/build/x86_64-unknown-linux-gnu/llvm/bin/clang++ -fopenmp --offload-arch=native -g -O3 main.ll -o main -save-temps
-LIBOMPTARGET_INFO=-1 ./main
-```
-The first step will generate a `main.ll` file, which has enough instructions to cause the offload runtime to move data to and from a gpu.
-The second step will use clang as the compilation driver to compile our IR file down to a working binary. Only a very small Rust subset will work out of the box here, unless
-you use features like build-std, which are not covered by this guide. Look at the codegen test to get a feeling for how to write a working example.
-In the last step you can run your binary, if all went well you will see a data transfer being reported:
-```
-omptarget device 0 info: Entering OpenMP data region with being_mapper at unknown:0:0 with 1 arguments:
-omptarget device 0 info: tofrom(unknown)[1024]
-omptarget device 0 info: Creating new map entry with HstPtrBase=0x00007fffffff9540, HstPtrBegin=0x00007fffffff9540, TgtAllocBegin=0x0000155547200000, TgtPtrBegin=0x0000155547200000, Size=1024, DynRefCount=1, HoldRefCount=0, Name=unknown
-omptarget device 0 info: Copying data from host to device, HstPtr=0x00007fffffff9540, TgtPtr=0x0000155547200000, Size=1024, Name=unknown
-omptarget device 0 info: OpenMP Host-Device pointer mappings after block at unknown:0:0:
-omptarget device 0 info: Host Ptr Target Ptr Size (B) DynRefCount HoldRefCount Declaration
-omptarget device 0 info: 0x00007fffffff9540 0x0000155547200000 1024 1 0 unknown at unknown:0:0
-// some other output
-omptarget device 0 info: Exiting OpenMP data region with end_mapper at unknown:0:0 with 1 arguments:
-omptarget device 0 info: tofrom(unknown)[1024]
-omptarget device 0 info: Mapping exists with HstPtrBegin=0x00007fffffff9540, TgtPtrBegin=0x0000155547200000, Size=1024, DynRefCount=0 (decremented, delayed deletion), HoldRefCount=0
-omptarget device 0 info: Copying data from device to host, TgtPtr=0x0000155547200000, HstPtr=0x00007fffffff9540, Size=1024, Name=unknown
-omptarget device 0 info: Removing map entry with HstPtrBegin=0x00007fffffff9540, TgtPtrBegin=0x0000155547200000, Size=1024, Name=unknown
-```
diff --git a/src/offload/usage.md b/src/offload/usage.md
new file mode 100644
index 0000000..9f51998
--- /dev/null
+++ b/src/offload/usage.md
@@ -0,0 +1,112 @@
+# Usage
+
+This feature is work-in-progress, and not ready for usage. The instructions here are for contributors, or people interested in following the latest progress.
+We currently work on launching the following Rust kernel on the GPU. To follow along, copy it to a `src/lib.rs` file.
+
+```rust
+#![feature(abi_gpu_kernel)]
+#![no_std]
+
+#[cfg(target_os = "linux")]
+extern crate libc;
+#[cfg(target_os = "linux")]
+use libc::c_char;
+
+use core::mem;
+
+#[panic_handler]
+fn panic(_: &core::panic::PanicInfo) -> ! {
+ loop {}
+}
+
+#[cfg(target_os = "linux")]
+#[unsafe(no_mangle)]
+#[inline(never)]
+fn main() {
+ let array_c: *mut [f64; 256] =
+ unsafe { libc::calloc(256, (mem::size_of::<f64>()) as libc::size_t) as *mut [f64; 256] };
+ let output = c"The first element is zero %f\n";
+ let output2 = c"The first element is NOT zero %f\n";
+ let output3 = c"The second element is %f\n";
+ unsafe {
+ let val: *const c_char = if (*array_c)[0] < 0.1 {
+ output.as_ptr()
+ } else {
+ output2.as_ptr()
+ };
+ libc::printf(val, (*array_c)[0]);
+ }
+
+ unsafe {
+ kernel_1(array_c);
+ }
+ core::hint::black_box(&array_c);
+ unsafe {
+ let val: *const c_char = if (*array_c)[0] < 0.1 {
+ output.as_ptr()
+ } else {
+ output2.as_ptr()
+ };
+ libc::printf(val, (*array_c)[0]);
+ libc::printf(output3.as_ptr(), (*array_c)[1]);
+ }
+}
+
+#[cfg(target_os = "linux")]
+unsafe extern "C" {
+ pub fn kernel_1(array_b: *mut [f64; 256]);
+}
+```
+
+## Compile instructions
+It is important to use a clang compiler build on the same llvm as rustc. Just calling clang without the full path will likely use your system clang, which probably will be incompatible. So either substitute clang/lld invocations below with absolute path, or set your `PATH` accordingly.
+
+First we generate the host (cpu) code. The first build is just to compile libc, take note of the hashed path. Then we call rustc directly to build our host code, while providing the libc artifact to rustc.
+```
+cargo +offload build -r -v
+rustc +offload --edition 2024 src/lib.rs -g --crate-type cdylib -C opt-level=3 -C panic=abort -C lto=fat -L dependency=/absolute_path_to/target/release/deps --extern libc=/absolute_path_to/target/release/deps/liblibc-<HASH>.rlib --emit=llvm-bc,llvm-ir -Zoffload=Enable -Zunstable-options
+```
+
+Now we generate the device code. Replace the target-cpu with the right code for your gpu.
+```
+RUSTFLAGS="-Ctarget-cpu=gfx90a --emit=llvm-bc,llvm-ir" cargo +offload build -Zunstable-options -r -v --target amdgcn-amd-amdhsa -Zbuild-std=core
+```
+
+Now find the <libname>.ll under target/amdgcn-amd-amdhsa folder and copy it to a device.ll file (or adjust the file names below).
+If you work on an NVIDIA or Intel gpu, please adjust the names acordingly and open an issue to share your results (either if you succeed or fail).
+First we compile our .ll files (good for manual inspections) to .bc files and clean up leftover artifacts. The cleanup is important, otherwise caching might interfere on following runs.
+```
+opt lib.ll -o lib.bc
+opt device.ll -o device.bc
+rm *.o
+rm bare.amdgcn.gfx90a.img*
+```
+
+```
+clang-offload-packager" "-o" "host.out" "--image=file=device.bc,triple=amdgcn-amd-amdhsa,arch=gfx90a,kind=openmp"
+
+clang-21" "-cc1" "-triple" "x86_64-unknown-linux-gnu" "-S" "-save-temps=cwd" "-disable-free" "-clear-ast-before-backend" "-main-file-name" "lib.rs" "-mrelocation-model" "pic" "-pic-level" "2" "-pic-is-pie" "-mframe-pointer=all" "-fmath-errno" "-ffp-contract=on" "-fno-rounding-math" "-mconstructor-aliases" "-funwind-tables=2" "-target-cpu" "x86-64" "-tune-cpu" "generic" "-resource-dir" "/<ABSOLUTE_PATH_TO>/rust/build/x86_64-unknown-linux-gnu/llvm/lib/clang/21" "-ferror-limit" "19" "-fopenmp" "-fopenmp-offload-mandatory" "-fgnuc-version=4.2.1" "-fskip-odr-check-in-gmf" "-fembed-offload-object=host.out" "-fopenmp-targets=amdgcn-amd-amdhsa" "-faddrsig" "-D__GCC_HAVE_DWARF2_CFI_ASM=1" "-o" "host.s" "-x" "ir" "lib.bc"
+
+clang-21" "-cc1as" "-triple" "x86_64-unknown-linux-gnu" "-filetype" "obj" "-main-file-name" "lib.rs" "-target-cpu" "x86-64" "-mrelocation-model" "pic" "-o" "host.o" "host.s"
+
+clang-linker-wrapper" "--should-extract=gfx90a" "--device-compiler=amdgcn-amd-amdhsa=-g" "--device-compiler=amdgcn-amd-amdhsa=-save-temps=cwd" "--device-linker=amdgcn-amd-amdhsa=-lompdevice" "--host-triple=x86_64-unknown-linux-gnu" "--save-temps" "--linker-path=/ABSOlUTE_PATH_TO/rust/build/x86_64-unknown-linux-gnu/lld/bin/ld.lld" "--hash-style=gnu" "--eh-frame-hdr" "-m" "elf_x86_64" "-pie" "-dynamic-linker" "/lib64/ld-linux-x86-64.so.2" "-o" "bare" "/lib/../lib64/Scrt1.o" "/lib/../lib64/crti.o" "/ABSOLUTE_PATH_TO/crtbeginS.o" "-L/ABSOLUTE_PATH_TO/rust/build/x86_64-unknown-linux-gnu/llvm/bin/../lib/x86_64-unknown-linux-gnu" "-L/ABSOLUTE_PATH_TO/rust/build/x86_64-unknown-linux-gnu/llvm/lib/clang/21/lib/x86_64-unknown-linux-gnu" "-L/lib/../lib64" "-L/usr/lib64" "-L/lib" "-L/usr/lib" "host.o" "-lstdc++" "-lm" "-lomp" "-lomptarget" "-L/ABSOLUTE_PATH_TO/rust/build/x86_64-unknown-linux-gnu/llvm/lib" "-lgcc_s" "-lgcc" "-lpthread" "-lc" "-lgcc_s" "-lgcc" "/ABSOLUTE_PATH_TO/crtendS.o" "/lib/../lib64/crtn.o"
+```
+
+Especially for the last command I recommend to not fix the paths, but rather just re-generate them by copying a bare-mode openmp example and compiling it with your clang. By adding `-###` to your clang invocation, you can see the invidual steps.
+```
+myclang++ -fuse-ld=lld -O3 -fopenmp -fopenmp-offload-mandatory --offload-arch=gfx90a omp_bare.cpp -o main -###
+```
+
+In the final step, you can now run your binary
+
+```
+./main
+The first element is zero 0.000000
+The first element is NOT zero 21.000000
+The second element is 0.000000
+```
+
+To receive more information about the memory transfer, you can enable info printing with
+```
+LIBOMPTARGET_INFO=-1 ./main
+```
diff --git a/src/queries/example-0.png b/src/queries/example-0.png
index 14b46c4..dd67d5f 100644
--- a/src/queries/example-0.png
+++ b/src/queries/example-0.png
Binary files differ
diff --git a/src/stability.md b/src/stability.md
index c26d342..3c4c65f 100644
--- a/src/stability.md
+++ b/src/stability.md
@@ -182,6 +182,11 @@
`#![warn(deprecated_in_future)]`.
## unstable_feature_bound
-The `#[unstable_feature_bound(foo)]` attribute can be used together with `#[unstable]` attribute to mark an `impl` of stable type and stable trait as unstable. In std/core, an item annotated with `#[unstable_feature_bound(foo)]` can only be used by another item that is also annotated with `#[unstable_feature_bound(foo)]`. Outside of std/core, using an item with `#[unstable_feature_bound(foo)]` requires the feature to be enabled with `#![feature(foo)]` attribute on the crate. Currently, only `impl`s and free functions can be annotated with `#[unstable_feature_bound]`.
+The `#[unstable_feature_bound(foo)]` attribute can be used together with `#[unstable]` attribute to mark an `impl` of stable type and stable trait as unstable. In std/core, an item annotated with `#[unstable_feature_bound(foo)]` can only be used by another item that is also annotated with `#[unstable_feature_bound(foo)]`. Outside of std/core, using an item with `#[unstable_feature_bound(foo)]` requires the feature to be enabled with `#![feature(foo)]` attribute on the crate.
+
+Currently, the items that can be annotated with `#[unstable_feature_bound]` are:
+- `impl`
+- free function
+- trait
[blog]: https://www.ralfj.de/blog/2018/07/19/const.html
