| r[input] |
| # Input format |
| |
| r[input.syntax] |
| ```grammar,lexer |
| CHAR -> [U+0000-U+D7FF U+E000-U+10FFFF] // a Unicode scalar value |
| |
| ASCII -> [U+0000-U+007F] |
| |
| NUL -> U+0000 |
| |
| EOF -> !CHAR // End of file or input |
| ``` |
| |
| r[input.intro] |
| This chapter describes how a source file is interpreted as a sequence of tokens. |
| |
| See [Crates and source files] for a description of how programs are organised into files. |
| |
| r[input.encoding] |
| ## Source encoding |
| |
| r[input.encoding.utf8] |
| Each source file is interpreted as a sequence of Unicode characters encoded in UTF-8. |
| |
| r[input.encoding.invalid] |
| It is an error if the file is not valid UTF-8. |
| |
| r[input.byte-order-mark] |
| ## Byte order mark removal |
| |
| If the first character in the sequence is `U+FEFF` ([BYTE ORDER MARK]), it is removed. |
| |
| r[input.crlf] |
| ## CRLF normalization |
| |
| Each pair of characters `U+000D` (CR) immediately followed by `U+000A` (LF) is replaced by a single `U+000A` (LF). This happens once, not repeatedly, so after the normalization, there can still exist `U+000D` (CR) immediately followed by `U+000A` (LF) in the input (e.g. if the raw input contained "CR CR LF LF"). |
| |
| Other occurrences of the character `U+000D` (CR) are left in place (they are treated as [whitespace]). |
| |
| r[input.shebang] |
| ## Shebang removal |
| |
| r[input.shebang.removal] |
| If a [shebang] is present, it is removed from the input sequence (and is therefore ignored). |
| |
| r[input.tokenization] |
| ## Tokenization |
| |
| The resulting sequence of characters is then converted into tokens as described in the remainder of this chapter. |
| |
| > [!NOTE] |
| > The standard library [`include!`] macro applies the following transformations to the file it reads: |
| > |
| > - Byte order mark removal. |
| > - CRLF normalization. |
| > - Shebang removal when invoked in an item context (as opposed to expression or statement contexts). |
| > |
| > The [`include_str!`] and [`include_bytes!`] macros do not apply these transformations. |
| |
| [BYTE ORDER MARK]: https://en.wikipedia.org/wiki/Byte_order_mark#UTF-8 |
| [Crates and source files]: crates-and-source-files.md |
| [shebang]: shebang.md |
| [whitespace]: whitespace.md |
