From 234fd01d65bb98cf8c4e2b166a8b92e89dff5858 Mon Sep 17 00:00:00 2001 From: Franck Nijhof Date: Mon, 20 Feb 2023 19:13:55 +0100 Subject: [PATCH] Add snapshot testing (#1690) --- blog/2023-02-20-snapshot-testing.md | 59 ++++++++++++++++++++++++++ docs/development_testing.md | 66 +++++++++++++++++++++++++++++ 2 files changed, 125 insertions(+) create mode 100644 blog/2023-02-20-snapshot-testing.md diff --git a/blog/2023-02-20-snapshot-testing.md b/blog/2023-02-20-snapshot-testing.md new file mode 100644 index 00000000..1d76bf1e --- /dev/null +++ b/blog/2023-02-20-snapshot-testing.md @@ -0,0 +1,59 @@ +--- +author: Franck Nijhof +authorURL: https://twitter.com/frenck +authorImageURL: /img/profile/frenck.png +authorTwitter: frenck +title: Added support for snapshot testing +--- + +Home Assistant [now supports snapshot testing](https://github.com/home-assistant/core/pull/88323) +for our Python codebase. + +Snapshot testing (also known as approval tests) are tests that assert values +against a stored reference value (the snapshot); ensuring the output of code +remains the same over time. + +Snapshot tests are different from regular (functional) tests and do not replace +functional tests, but they can be very useful for testing larger test outputs. +Within Home Assistant they could, for example, be used to test entity states, device +or entity registry items, or the output of a diagnostic dump. + +Take, for example, this diagnostic test, which uses a snapshot to assert the +output of a diagnostic dump: + +```python +# tests/components/example/test_diagnostics.py +async def test_diagnostics( + hass: HomeAssistant, + hass_client: ClientSessionGenerator, + init_integration: MockConfigEntry, + snapshot: SnapshotAssertion, +) -> None: + """Test diagnostics.""" + assert ( + await get_diagnostics_for_config_entry(hass, hass_client, init_integration) + == snapshot + ) +``` + +When this test is run for the first time, it will fail, as no snapshot exists. +To create (or update) a snapshot, run pytest with the `--snapshot-update` flag, +which will create a snapshot file in the `snapshots` directory of this component. + +The snapshot file is named after the test file, in this case: +`tests/components/example/snapshots/test_diagnostics.ambr`. The snapshot files +are human-readable and must be committed to the repository. + +Any sequential runs of the tests will then compare the results against the +snapshot. If the results are different, the test will fail. + +Snapshots are an easy way to ensure the output of code remains the same over +time and can greatly reduce the amount of testing code needed (while providing) +a full assert against a complete output. + +Snapshot testing in Home Assistant is build on top of [Syrupy](https://github.com/tophat/syrupy), +which has been extended to handle Home Assistant-specific data structures. + +More information on testing integrations, +can be found [in our documentation](/docs/development_testing). + diff --git a/docs/development_testing.md b/docs/development_testing.md index e8038963..a8154434 100644 --- a/docs/development_testing.md +++ b/docs/development_testing.md @@ -105,3 +105,69 @@ If you can't avoid a PyLint warning, add a comment to disable the PyLint check f - Modify a `ConfigEntry` via the config entries interface [`hass.config_entries`](https://github.com/home-assistant/core/blob/4cce724473233d4fb32c08bd251940b1ce2ba570/homeassistant/config_entries.py#L570). - Assert the state of a config entry via the [`ConfigEntry.state`](https://github.com/home-assistant/core/blob/4cce724473233d4fb32c08bd251940b1ce2ba570/homeassistant/config_entries.py#L169) attribute. - Mock a config entry via the `MockConfigEntry` class in [`tests/common.py`](https://github.com/home-assistant/core/blob/4cce724473233d4fb32c08bd251940b1ce2ba570/tests/common.py#L658) + +### Snapshot testing + +Home Assistant supports a testing concept called snapshot testing (also known +as approval tests), which are tests that assert values against a stored +reference value (the snapshot). + +Snapshot tests are different from regular (functional) tests and do not replace +functional tests, but they can be very useful for testing larger test outputs. +Within Home Assistant they could, for example, be used to: + +- Ensure the output of an entity state is and remains as expected. +- Ensure an area, config, device, entity, or issue entry in the registry is and + remains as expected. +- Ensure the output of a diagnostic dump is and remains as expected. +- Ensure a FlowResult is and remains as expected. + +And many more cases that have large output, like JSON, YAML, or XML results. + +The big difference between snapshot tests and regular tests is that the results +are captured by running the tests in a special mode that creates the snapshots. +Any sequential runs of the tests will then compare the results against the +snapshot. If the results are different, the test will fail. + +Snapshot testing in Home Assistant is built on top of [Syrupy](https://github.com/tophat/syrupy), +their documentation can thus be applied when writing Home Assistant tests. +This is a snapshot test that asserts the output of an entity state: + +```python +# tests/components/example/test_sensor.py +async def test_sensor( + hass: HomeAssistant, + snapshot: SnapshotAssertion, +) -> None: + """Test the sensor state.""" + state = hass.states.get("sensor.whatever") + assert state == snapshot +``` + +When this test is run for the first time, it will fail, as no snapshot exists. +To create (or update) a snapshot, run the test with +the `--snapshot-update` flag: + +```shell +pytest tests/components/example/test_sensor.py --snapshot-update +``` + +This will create a snapshot file in the `tests/components/example/snapshots`. +The snapshot file is named after the test file, in this case `test_sensor.ambr`, +and is human-readable. The snapshot files must be committed to the repository. + +When the test is run again (without the update flag), it will compare the +results against the stored snapshot and everything should pass. + +When the test results change, the test will fail and the snapshot needs to be +updated again. + +Use snapshot testing with care! As it is very easy to create a snapshot, +it can be tempting to assert everything against a snapshot. However, remember, +it is not a replacement for functional tests. + +As an example, when testing if an entity would go unavailable when the device +returns an error, it is better to assert the specific change you expected: +Assert the state of the entity became `unavailable`. This functional test is a +better approach than asserting the full state of such an entity using a +snapshot, as it assumes it worked as expected (when taking the snapshot).