Add snapshot testing (#1690)

blog/2023-02-20-snapshot-testing.md

@@ -0,0 +1,59 @@
+---
+author: Franck Nijhof
+authorURL: https://twitter.com/frenck
+authorImageURL: /img/profile/frenck.png
+authorTwitter: frenck
+title: Added support for snapshot testing
+---
+
+Home Assistant [now supports snapshot testing](https://github.com/home-assistant/core/pull/88323)
+for our Python codebase.
+
+Snapshot testing (also known as approval tests) are tests that assert values
+against a stored reference value (the snapshot); ensuring the output of code
+remains the same over time.
+
+Snapshot tests are different from regular (functional) tests and do not replace
+functional tests, but they can be very useful for testing larger test outputs.
+Within Home Assistant they could, for example, be used to test entity states, device
+or entity registry items, or the output of a diagnostic dump.
+
+Take, for example, this diagnostic test, which uses a snapshot to assert the
+output of a diagnostic dump:
+
+```python
+# tests/components/example/test_diagnostics.py
+async def test_diagnostics(
+    hass: HomeAssistant,
+    hass_client: ClientSessionGenerator,
+    init_integration: MockConfigEntry,
+    snapshot: SnapshotAssertion,
+) -> None:
+    """Test diagnostics."""
+    assert (
+        await get_diagnostics_for_config_entry(hass, hass_client, init_integration)
+        == snapshot
+    )
+```
+
+When this test is run for the first time, it will fail, as no snapshot exists.
+To create (or update) a snapshot, run pytest with the `--snapshot-update` flag,
+which will create a snapshot file in the `snapshots` directory of this component.
+
+The snapshot file is named after the test file, in this case:
+`tests/components/example/snapshots/test_diagnostics.ambr`. The snapshot files
+are human-readable and must be committed to the repository.
+
+Any sequential runs of the tests will then compare the results against the
+snapshot. If the results are different, the test will fail.
+
+Snapshots are an easy way to ensure the output of code remains the same over
+time and can greatly reduce the amount of testing code needed (while providing)
+a full assert against a complete output.
+
+Snapshot testing in Home Assistant is build on top of [Syrupy](https://github.com/tophat/syrupy),
+which has been extended to handle Home Assistant-specific data structures.
+
+More information on testing integrations,
+can be found [in our documentation](/docs/development_testing).
+

docs/development_testing.md

@@ -105,3 +105,69 @@ If you can't avoid a PyLint warning, add a comment to disable the PyLint check f
   - Modify a `ConfigEntry` via the config entries interface [`hass.config_entries`](https://github.com/home-assistant/core/blob/4cce724473233d4fb32c08bd251940b1ce2ba570/homeassistant/config_entries.py#L570).
   - Assert the state of a config entry via the [`ConfigEntry.state`](https://github.com/home-assistant/core/blob/4cce724473233d4fb32c08bd251940b1ce2ba570/homeassistant/config_entries.py#L169) attribute.
   - Mock a config entry via the `MockConfigEntry` class in [`tests/common.py`](https://github.com/home-assistant/core/blob/4cce724473233d4fb32c08bd251940b1ce2ba570/tests/common.py#L658)
+
+### Snapshot testing
+
+Home Assistant supports a testing concept called snapshot testing (also known
+as approval tests), which are tests that assert values against a stored
+reference value (the snapshot).
+
+Snapshot tests are different from regular (functional) tests and do not replace
+functional tests, but they can be very useful for testing larger test outputs.
+Within Home Assistant they could, for example, be used to:
+
+- Ensure the output of an entity state is and remains as expected.
+- Ensure an area, config, device, entity, or issue entry in the registry is and
+  remains as expected.
+- Ensure the output of a diagnostic dump is and remains as expected.
+- Ensure a FlowResult is and remains as expected.
+
+And many more cases that have large output, like JSON, YAML, or XML results.
+
+The big difference between snapshot tests and regular tests is that the results
+are captured by running the tests in a special mode that creates the snapshots.
+Any sequential runs of the tests will then compare the results against the
+snapshot. If the results are different, the test will fail.
+
+Snapshot testing in Home Assistant is built on top of [Syrupy](https://github.com/tophat/syrupy),
+their documentation can thus be applied when writing Home Assistant tests.
+This is a snapshot test that asserts the output of an entity state:
+
+```python
+# tests/components/example/test_sensor.py
+async def test_sensor(
+    hass: HomeAssistant,
+    snapshot: SnapshotAssertion,
+) -> None:
+    """Test the sensor state."""
+    state = hass.states.get("sensor.whatever")
+    assert state == snapshot
+```
+
+When this test is run for the first time, it will fail, as no snapshot exists.
+To create (or update) a snapshot, run the test with
+the `--snapshot-update` flag:
+
+```shell
+pytest tests/components/example/test_sensor.py --snapshot-update
+```
+
+This will create a snapshot file in the `tests/components/example/snapshots`.
+The snapshot file is named after the test file, in this case `test_sensor.ambr`,
+and is human-readable. The snapshot files must be committed to the repository.
+
+When the test is run again (without the update flag), it will compare the
+results against the stored snapshot and everything should pass.
+
+When the test results change, the test will fail and the snapshot needs to be
+updated again.
+
+Use snapshot testing with care! As it is very easy to create a snapshot,
+it can be tempting to assert everything against a snapshot. However, remember,
+it is not a replacement for functional tests.
+
+As an example, when testing if an entity would go unavailable when the device
+returns an error, it is better to assert the specific change you expected: 
+Assert the state of the entity became `unavailable`. This functional test is a
+better approach than asserting the full state of such an entity using a
+snapshot, as it assumes it worked as expected (when taking the snapshot).