rustdoc: Represent `--output-format=json` coverage and ir differently

`--output-format=json` does a few different things:

- By itself, it emits a JSON representation of a crate's API ("IR
  JSON").
- When used with `--show-coverage`, it prints the doc coverage as JSON,
  instead of an ASCII table ("Coverage JSON").

These two cases need to be handled entirely differently. There's no
overlapping code, they just are called the same way.

By making these separate variants, we don't need to check the
`show_coverage` variable each time we check `is_json`, which makes it
harder to write a bug that checks for IR JSON, and accidentally happens in
coverage JSON too.

Also, as a driveby, improves some instability error messages, and cleans
up their tests, and adds other related tests.