src/serialization.md

# Serialization in Rustc

Rustc has to [serialize] and deserialize various data during compilation.
Specifically:

- "Crate metadata", mainly query outputs, are serialized in a binary
  format into `rlib` and `rmeta` files that are output when compiling a library
  crate, these are then deserialized by crates that depend on that library.
- Certain query outputs are serialized in a binary format to
  [persist incremental compilation results].
- The `-Z ast-json` and `-Z ast-json-noexpand` flags serialize the [AST] to json
  and output the result to stdout.
- [`CrateInfo`] is serialized to json when the `-Z no-link` flag is used, and
  deserialized from json when the `-Z link-only` flag is used.

## The `Encodable` and `Decodable` traits

The [`rustc_serialize`] crate defines two traits for types which can be serialized:

```rust,ignore
pub trait Encodable<S: Encoder> {
    fn encode(&self, s: &mut S) -> Result<(), S::Error>;
}

pub trait Decodable<D: Decoder>: Sized {
    fn decode(d: &mut D) -> Result<Self, D::Error>;
}
```

It also defines implementations of these for integer types, floating point
types, `bool`, `char`, `str` and various common standard library types.

For types that are constructed from those types, `Encodable` and `Decodable` are
usually implemented by [derives]. These generate implementations that forward
deserialization to the fields of the struct or enum. For a struct those impls
look something like this:

```rust,ignore
#![feature(rustc_private)]
extern crate rustc_serialize;
use rustc_serialize::{Decodable, Decoder, Encodable, Encoder};

struct MyStruct {
    int: u32,
    float: f32,
}

impl<E: Encoder> Encodable<E> for MyStruct {
    fn encode(&self, s: &mut E) -> Result<(), E::Error> {
        s.emit_struct("MyStruct", 2, |s| {
            s.emit_struct_field("int", 0, |s| self.int.encode(s))?;
            s.emit_struct_field("float", 1, |s| self.float.encode(s))
        })
    }
}
impl<D: Decoder> Decodable<D> for MyStruct {
    fn decode(s: &mut D) -> Result<MyStruct, D::Error> {
        s.read_struct("MyStruct", 2, |d| {
            let int = d.read_struct_field("int", 0, Decodable::decode)?;
            let float = d.read_struct_field("float", 1, Decodable::decode)?;

            Ok(MyStruct { int, float })
        })
    }
}
```

## Encoding and Decoding arena allocated types

Rustc has a lot of [arena allocated types]. Deserializing these types isn't
possible without access to the arena that they need to be allocated on. The
[`TyDecoder`] and [`TyEncoder`] traits are supertraits of `Decoder` and
`Encoder` that allow access to a `TyCtxt`.

Types which contain arena allocated types can then bound the type parameter of
their `Encodable` and `Decodable` implementations with these traits. For
example

```rust,ignore
impl<'tcx, D: TyDecoder<'tcx>> Decodable<D> for MyStruct<'tcx> {
    /* ... */
}
```

The `TyEncodable` and `TyDecodable` [derive macros][derives] will expand to such
an implementation.

Decoding the actual arena allocated type is harder, because some of the
implementations can't be written due to the orphan rules. To work around this,
the [`RefDecodable`] trait is defined in `rustc_middle`. This can then be
implemented for any type. The `TyDecodable` macro will call `RefDecodable` to
decode references, but various generic code needs types to actually be
`Decodable` with a specific decoder.

For interned types instead of manually implementing `RefDecodable`, using a new
type wrapper, like `ty::Predicate` and manually implementing `Encodable` and
`Decodable` may be simpler.

## Derive macros

The `rustc_macros` crate defines various derives to help implement `Decodable`
and `Encodable`.

- The `Encodable` and `Decodable` macros generate implementations that apply to
  all `Encoders` and `Decoders`. These should be used in crates that don't
  depend on `rustc_middle`, or that have to be serialized by a type that does
  not implement `TyEncoder`.
- `MetadataEncodable` and `MetadataDecodable` generate implementations that
  only allow decoding by [`rustc_metadata::rmeta::encoder::EncodeContext`] and
  [`rustc_metadata::rmeta::decoder::DecodeContext`]. These are used for types
  that contain `rustc_metadata::rmeta::Lazy`.
- `TyEncodable` and `TyDecoder` generate implementation that apply to any
  `TyEncoder` or `TyDecoder`. These should be used for types that are only
  serialized in crate metadata and/or the incremental cache, which is most
  serializable types in `rustc_middle`.

## Shorthands

`Ty` can be deeply recursive, if each `Ty` was encoded naively then crate
metadata would be very large. To handle this, each `TyEncoder` has a cache of
locations in its output where it has serialized types. If a type being encoded
is in the cache, then instead of serializing the type as usual, the byte offset
within the file being written is encoded instead. A similar scheme is used for
`ty::Predicate`.

## `Lazy<T>`

Crate metadata is initially loaded before the `TyCtxt<'tcx>` is created, so
some deserialization needs to be deferred from the initial loading of metadata.
The [`Lazy<T>`] type wraps the (relative) offset in the crate metadata where a
`T` has been serialized.

The `Lazy<[T]>` and `Lazy<Table<I, T>>` type provide some functionality over
`Lazy<Vec<T>>` and `Lazy<HashMap<I, T>>`:

- It's possible to encode a `Lazy<[T]>` directly from an iterator, without
  first collecting into a `Vec<T>`.
- Indexing into a `Lazy<Table<I, T>>` does not require decoding entries other
  than the one being read.

**note**: `Lazy<T>` does not cache its value after being deserialized the first
time. Instead the query system is the main way of caching these results.

## Specialization

A few types, most notably `DefId`, need to have different implementations for
different `Encoder`s. This is currently handled by ad-hoc specializations:
`DefId` has a `default` implementation of `Encodable<E>` and a specialized one
for `Encodable<CacheEncoder>`.

[arena allocated types]: memory.md
[AST]: the-parser.md
[derives]: #derive-macros
[persist incremental compilation results]: queries/incremental-compilation-in-detail.md#the-real-world-how-persistence-makes-everything-complicated
[serialize]: https://en.wikipedia.org/wiki/Serialization

[`CrateInfo`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_codegen_ssa/struct.CrateInfo.html
[`Lazy<T>`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_metadata/rmeta/struct.Lazy.html
[`RefDecodable`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/ty/codec/trait.RefDecodable.html
[`rustc_metadata::rmeta::decoder::DecodeContext`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_metadata/rmeta/decoder/struct.DecodeContext.html
[`rustc_metadata::rmeta::encoder::EncodeContext`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_metadata/rmeta/encoder/struct.EncodeContext.html
[`rustc_serialize`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_serialize/index.html
[`TyDecoder`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/ty/codec/trait.TyEncoder.html
[`TyEncoder`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/ty/codec/trait.TyDecoder.html