Create Whitespace grammar productions
This does not create any new productions, instead preferring comments.
rust-lang/reference#1974 will involve pulling out the horizontal
whitespace into a separate production.
Comment wording (and casing) is modeled off of
https://www.unicode.org/reports/tr31/#R3a.
I left off a "unicode" prefix for ASCII items as they are likely common
enough in that context that specifying them as "unicode" could cause
more confusion.
diff --git a/src/input-format.md b/src/input-format.md
index 5d2a692..cf35b29 100644
--- a/src/input-format.md
+++ b/src/input-format.md
@@ -6,12 +6,6 @@
@root CHAR -> <a Unicode scalar value>
NUL -> U+0000
-
-TAB -> U+0009
-
-LF -> U+000A
-
-CR -> U+000D
```
r[input.intro]
diff --git a/src/whitespace.md b/src/whitespace.md
index 45d58b3..761a142 100644
--- a/src/whitespace.md
+++ b/src/whitespace.md
@@ -1,21 +1,34 @@
r[lex.whitespace]
# Whitespace
+r[whitespace.syntax]
+```grammar,lexer
+@root WHITESPACE ->
+ // end of line
+ LF
+ | U+000B // vertical tabulation
+ | U+000C // form feed
+ | CR
+ | U+0085 // Unicode next line
+ | U+2028 // Unicode LINE SEPARATOR
+ | U+2029 // Unicode PARAGRAPH SEPARATOR
+ // Ignorable Code Point
+ | U+200E // Unicode LEFT-TO-RIGHT MARK
+ | U+200F // Unicode RIGHT-TO-LEFT MARK
+ // horizontal whitespace
+ | TAB
+ | U+0020 // space ' '
+
+TAB -> U+0009 // horizontal tab ('\t')
+
+LF -> U+000A // line feed ('\n')
+
+CR -> U+000D // carriage return ('\r')
+```
+
r[lex.whitespace.intro]
Whitespace is any non-empty string containing only characters that have the
-[`Pattern_White_Space`] Unicode property, namely:
-
-- `U+0009` (horizontal tab, `'\t'`)
-- `U+000A` (line feed, `'\n'`)
-- `U+000B` (vertical tab)
-- `U+000C` (form feed)
-- `U+000D` (carriage return, `'\r'`)
-- `U+0020` (space, `' '`)
-- `U+0085` (next line)
-- `U+200E` (left-to-right mark)
-- `U+200F` (right-to-left mark)
-- `U+2028` (line separator)
-- `U+2029` (paragraph separator)
+[`Pattern_White_Space`] Unicode property.
r[lex.whitespace.token-sep]
Rust is a "free-form" language, meaning that all forms of whitespace serve only
