186 Commits

Author SHA1 Message Date
David Rapan
d5b5a328d7
feat: Add opt-in IPv6 for containers (#5879)
Configurable and w/ migrations between IPv4-Only and Dual-Stack

Signed-off-by: David Rapan <david@rapan.cz>
Co-authored-by: Stefan Agner <stefan@agner.ch>
2025-06-12 11:32:24 +02:00
David Rapan
3b575eedba
Add IPv6 address generation mode & privacy extensions (#5892)
* feat: Add IPv6 address generation mode & privacy extensions

Signed-off-by: David Rapan <david@rapan.cz>

* Use NetworkManager fixture for settings init tests

This fixes the test by since the extended implementation now can read
the version of NetworkManager.

* Add pytest for addr_gen_mode

---------

Signed-off-by: David Rapan <david@rapan.cz>
Co-authored-by: Stefan Agner <stefan@agner.ch>
2025-05-20 17:03:08 +02:00
Stefan Agner
b5a7e521ae
Copy additional backup locations in jobs (#5890)
Instead of copying the backup in the main job, lets copy them in
separate job per location. This allows to use the same backup error
handling mechanism as for add-ons and folders.

This makes the stage introduced in #5784 somewhat redundant, but
before removing it, let's see if this approach works out.
2025-05-20 15:18:23 +02:00
Stefan Agner
d0d11db7b1
Harmonize folder and add-on backup error handling (#5885)
* Harmonize folder and add-on backup error handling

Align add-on and folder backup error handling in that in both cases
errors are recorded on the respective backup Jobs, but not raised to
the caller. This allows the backup to complete successfully even if
some add-ons or folders fail to back up.

Along with this, also record errors in the per-add-on and per-folder
backup jobs, as well as the add-on and folder root job.

And finally, align the exception handling to only catch expected
exceptions for add-ons too.

* Fix pytest
2025-05-15 10:14:35 +02:00
Stefan Agner
85f8107b60
Recreate aiohttp ClientSession after DNS plug-in load (#5862)
* Recreate aiohttp ClientSession after DNS plug-in load

Create a temporary ClientSession early in case we need to load version
information from the internet. This doesn't use the final DNS setup
and hence might fail to load in certain situations since we don't have
the fallback mechanims in place yet. But if the DNS container image
is present, we'll continue the setup and load the DNS plug-in. We then
can recreate the ClientSession such that it uses the DNS plug-in.

This works around an issue with aiodns, which today doesn't reload
`resolv.conf` automatically when it changes. This lead to Supervisor
using the initial `resolv.conf` as created by Docker. It meant that
we did not use the DNS plug-in (and its fallback capabilities) in
Supervisor. Also it meant that changes to the DNS setup at runtime
did not propagate to the aiohttp ClientSession (as observed in #5332).

* Mock aiohttp.ClientSession for all tests

Currently in several places pytest actually uses the aiohttp
ClientSession and reaches out to the internet. This is not ideal
for unit tests and should be avoided.

This creates several new fixtures to aid this effort: The `websession`
fixture simply returns a mocked aiohttp.ClientSession, which can be
used whenever a function is tested which needs the global websession.

A separate new fixture to mock the connectivity check named
`supervisor_internet` since this is often used through the Job
decorator which require INTERNET_SYSTEM.

And the `mock_update_data` uses the already existing update json
test data from the fixture directory instead of loading the data
from the internet.

* Log ClientSession nameserver information

When recreating the aiohttp ClientSession, log information what
nameservers exactly are going to be used.

* Refuse ClientSession initialization when API is available

Previous attempts to reinitialize the ClientSession have shown
use of the ClientSession after it was closed due to API requets
being handled in parallel to the reinitialization (see #5851).
Make sure this is not possible by refusing to reinitialize the
ClientSession when the API is available.

* Fix pytests

Also sure we don't create aiohttp ClientSession objects unnecessarily.

* Apply suggestions from code review

Co-authored-by: Jan Čermák <sairon@users.noreply.github.com>

---------

Co-authored-by: Jan Čermák <sairon@users.noreply.github.com>
2025-05-06 16:23:40 +02:00
Stefan Agner
9470f44840
Improve /auth API request sanitation (#5843)
* Add basic test coverage for /auth API

* Check /auth API is called from an add-on

Currently the /auth API is only available for add-ons. Return 403
for calls not originating from an add-on.

* Handle bad json in auth API

Use the API specific JSON load helper which raises an APIError. This
causes the API to return a 400 error instead of a 500 error when the
JSON is invalid.

* Avoid redefining name 'mock_check_login'

* Update tests/api/test_auth.py
2025-04-25 15:17:25 +02:00
Stefan Agner
de497cdc19
Add dedicated version update refresh for main components (#5833)
* Add dedicated update information reload

Currently we have the /refresh_updates endpoint which updates the main
component versions (Core, OS, Supervisor, Plug-ins) and the add-on
store at the same time. This combined update causes more update
information reloads than necessary.

To allow fine grained update refresh control introduce a new endpoint
/reload_updates which asks Supervisor to only update main component
versions (learned through the version json files).

The /store/reload endpoint already allows to update the add-on store
separately.

* Add pytest

* Update supervisor/api/__init__.py
2025-04-24 15:46:18 +02:00
Stefan Agner
5d07dd2c42
Add country to Supervisor info (#5826)
Similar to timezone also add country information to the Supervisor
info. This is useful to set country specific configurations such as
Wireless radio regulatory setting. This is also useful for add-ons
which need country information but only have hassio API access.
2025-04-22 16:18:23 +02:00
Jan Čermák
adfb433f57
Intercept host logs Range header for Systemd v256+ compatibility (#5827)
Since Systemd v256 the Range header must not end with a trailing colon.
We relied on this undocumented feature when following logs, and the
frontend or CLI may still use it in requests. To fix the requests
failing with new Systemd version, intercept the header and fill in the
num_entries to maximum possible value, which avoids the journal-gatewayd
returning the response prematurely and also works on older Systemd
versions.

The journal-gatewayd would still return response if follow flag is used
along with num_entries, but this behavior is unchanged and would be
better fixed in the backend.

Link: https://github.com/systemd/systemd/issues/37172
2025-04-22 09:05:49 +02:00
Stefan Agner
6fad7d14e1
Avoid using host system socket for logs tests (#5825)
Make sure we mock the systemd journal gateway socket for tests. This
makes the test work on systems which have systemd-journal-gatewayd
installed.
2025-04-17 16:23:34 +02:00
Stefan Agner
59a7e9519d
Fix root path requests (#5815)
* Fix root path requests

Since #5759 we've tried to access the path explicitly. However, this
raises KeyError exception when trying to access the proxied root path
(e.g. http://supervisor/core/api/). Before #5759 get was used, which
lead to no exception, but instead inserted a `None` into the path.

It seems aiohttp doesn't provide a path when the root is accessed. So
simply convert this to no path as well by setting path to an empty
string.

* Add rudimentary pytest for regular proxy requets
2025-04-07 11:09:45 +02:00
Mike Degatano
01a682cfaa
Fix mypy issues in backups and dbus (#5792)
* Fix mypy issues in backups module

* Fix mypy issues in dbus module

* Fix mypy issues in api after rebase

* TypedDict to dataclass and other small fixes

* Finish fixing mypy errors in dbus

* local_where must exist

* Fix references to name in tests
2025-03-31 17:03:54 -04:00
Stefan Agner
8fe17d9270
Improve Home Assistant Core WebSocket proxy implementation (#5790)
* Improve Home Assistant Core WebSocket proxy implementation

This change removes unnecessary task creation for every WebSocket
message and instead creates just two tasks, one for each direction.
This improves performance by about factor of 3 when measuring 1000
WebSocket requests to Core (from ~530ms to ~160ms).

While at it, also handle all WebSocket message related to closing the
WebSocket and report all other errors as warnings instead of just info.

* Improve logging and error handling

* Add WS client error test case

* Use asyncio.gather directly

* Use asyncio.wait to handle exceptions gracefully

* Drop cancellation handling and correctly wait for the other proxy task
2025-03-28 10:35:49 +01:00
Jan Čermák
0a684bdb12
Add API for swap configuration (#5770)
* Add API for swap configuration

Add HTTP API for swap size and swappiness to /os/config/swap. Individual
options can be set in JSON and are calling the DBus API added in OS
Agent 1.7.x, available since OS 15.0. Check for presence of OS of the
required version and return 404 if the criteria are not met.

* Fix type hints and reboot_required logic

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Fix formatting after adding suggestions from GH

* Address @mdegat01 review comments

- Improve swap options validation
- Add swap to the 'all' property of dbus agent
- Use APINotFound with reason instead of HTTPNotFound
- Reorder API routes

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-03-27 17:53:46 +01:00
Mike Degatano
9222a3c9c0
Report stage with error in jobs (#5784)
* Report stage with error in jobs

* Copy doesn't lose track of the successful copies

* Add stage to errors in api output test

* revert unneessary change to import

* Add tests for a bit more coverage of copy_additional_locations
2025-03-27 10:07:06 -04:00
Jan Čermák
92cadb4c55
Fix /supervisor/reload after refactoring (#5791)
As discussed in [1], refactoring in #5759 changed signature of the
reload method and CLI now gets unexpected schema when `ha su reload` is
called. Change the method to return None as before and add a test for a
proper body content.

[1] https://github.com/home-assistant/supervisor/pull/5759/files#diff-1b4ed26f31e52ff5fe53efdc695eebacb1e46411f23cce58295591b2b20cd3faR238
2025-03-27 10:03:57 -04:00
Stefan Agner
81fc15d6ac
Handle unexpected WebSocket messages during auth (#5788)
* Handle unexpected WebSocket messages during auth

When an add-on does not respond or closes the WebSocket connection
during the authentication phase Supervisor does not handle errors
gracefully. Simply log such unexpected authentication to avoid
unnecessary stack traces in the log and make such cases no longer
appear on Sentry.

* Add pytest

* Introduce a timeout of 10s
2025-03-26 22:13:59 +01:00
Mike Degatano
0636e49fe2
Enable mypy part 1 (addons and api) (#5759)
* Fix mypy issues in addons

* Fix mypy issues in api

* fix docstring

* Brackets instead of get with default
2025-03-25 15:06:35 -04:00
Mike Degatano
80f7f07341
Add blockbuster option to API (#5746)
* Add blockbuster option to API

* cache not lru_cache
2025-03-25 09:40:43 +01:00
Mike Degatano
6ef4f3cc67
Add blockbuster library and find I/O from unit tests (#5731)
* Add blockbuster library and find I/O from unit tests

* Fix lint and test issue

* Fixes from feedback

* Avoid modifying webapp object in executor

* Split su options validation and only validate timezone on change
2025-03-06 16:40:13 -05:00
Mike Degatano
324b059970
Move write of core state to executor (#5720) 2025-03-04 17:49:53 +01:00
Stefan Agner
f8bab20728
Replace non-unicode characters for add-on static files (#5712)
* Replace non-unicode characters for add-on static files

Add-on documentation and changelog get read and returned as text file.
However, in case the original author used non-unicode characters, or
the file corrupted, loading currently fails with an UnicodeDecodeError.

Let's just use the built-in replace error handling of Python, so they
appear for the user as  non-unicode characters by replacing them with
the official unicode replacement character "�".

* Remove superflous parameter for binary files

* ruff format

* Add pytests
2025-03-03 20:14:39 +01:00
Mike Degatano
86133f8ecd
Move read_text to executor (#5688)
* Move read_text to executor

* Fix issues found by coderabbit

* formated to formatted

* switch to async_capture_exception

* Find and replace got one too many

* Update patch mock to async_capture_exception

* Drop Sentry capture from format_message

The error handling got introduced in #2052, however, #2100 essentially
makes sure there will never be a byte object passed to this function.
And even if, the Sentry aiohttp plug-in will properly catch such an
exception.

---------

Co-authored-by: Stefan Agner <stefan@agner.ch>
2025-03-01 16:02:43 +01:00
Stefan Agner
696dcf6149
Initialize Supervisor Core state in constructor (#5686)
* Initialize Supervisor Core state in constructor

Make sure the Supervisor Core state is set to a value early on. This
makes sure that the state is always of type CoreState, and makes sure
that any use of the state can rely on it being an actual value from the
CoreState enum.

This fixes Sentry filter during early startup, where the state
previously was None. Because of that, the Sentry filter tried to
collect more Context, which lead to an exception and not reporting
errors.

* Fix pytest

It seems that with initializing the state early, the pytest actually
runs a system evaluation with:
Starting system evaluation with state initialize

Before it did that with:
Starting system evaluation with state None

It detects that the container runs as privileged, and declares the
system as unhealthy.

It is unclear to me why coresys.core.healthy was checked in this
context, it doesn't seem useful. Just remove the check, and validate
the state through the getter instead.

* Update supervisor/core.py

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Make sure Supervisor container is privileged in pytest

With the Supervisor Core state being valid now, some evaluations
now actually run when loading the resolution center. This leads to
Supervisor getting declared unhealthy due to not running in a privileged
container under pytest.

Fake the host container to be privileged to make evaluations not
causing the system to be declared unhealthy under pytest.

* Avoid writing actual Supervisor run state file

With the Supervisor Core state being valid from the very start, we end
up writing a state everytime.

Instead of actually writing a state file, simply validate the the
necessary calls are being made. This is more conform to typical unit
tests and avoids writing a file for every test.

* Extend WebSocket client fixture and use it consistently

Extend the ha_ws_client WebSocket client fixture to set Supervisor Core
into run state and clear all pending messages.

Currently only some tests use the ha_ws_client WebSocket client fixture.
Use it consistently for all tests.

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-02-28 18:01:55 +01:00
Stefan Agner
f4d69f1811
Make advanced logs error test work in all test environments (#5692)
When developing/testing in a Supervised environment, the
systemd-journal-gatewayd socket is actually available. Mock the
socket Path file to make the test independent of the pytest
environment.
2025-02-28 12:59:20 +01:00
Mike Degatano
31193abb7b
FileConfiguration uses executor for I/O (#5652)
* FileConfiguration uses executor for I/O

* Fix credentials tests

* Remove migrate_system_env as its very deprecated
2025-02-26 19:11:11 +01:00
Stefan Agner
34939cfe52
Remove I/O in event loop for backup load, import and remove (#5647)
* Avoid IO in event loop when removing backup

* Refactor backup size calculation

Currently size is lazy loaded when required via properties. This
however is blocking the async event loop.

Backup sizes don't change. Instead of lazy loading the size of a backup
simply determine it on loading/after creation.

* Fix tests for backup size change

* Avoid IO in event loop when loading backups

* Avoid IO in event loop when importing a backup
2025-02-19 16:00:17 +01:00
Stefan Agner
4c108eea64
Always validate Backup before restoring (#5632)
* Validate Backup always before restoring

Since #5519 we check the encryption password early in restore case.
This has the side effect that we check the file existance early too.
However, in the non-encryption case, the file is not checked early.

This PR changes the behavior to always validate the backup file before
restoring, ensuring both encryption and non-encryption cases are
handled consistently.

In particular, the last case of test_restore_immediate_errors actually
validates that behavior. That test should actually have failed so far.
But it seems that because we validate the backup shortly after freeze
anyways, the exception still got raised early enough.

A simply `await asyncio.sleep(10)` right after the freeze makes the
test case fail. With this change, the test works consistently.

* Address pylint

* Fix backup_manager tests

* Drop warning message
2025-02-14 18:19:35 +01:00
Mike Degatano
129a37a1f4
Prevent race condition with location reload and backups list (#5602) 2025-02-05 14:24:37 +01:00
Stefan Agner
9164d35615
Fix restoring unencrypted backup in corner case (#5600)
* Fix restoring unencrypted backup in corner case

If a backup has a encrypted and unencrypted location, and the encrypted
location is beeing restored first, the encryption key is still cached.
When the user restores the unencrypted backup next, it will fail because
the Supervisor tries to use encryption key still.

* Add integration test for restoring backups with and without encryption

* Rename _validate_location_password to _set_location_password

* Reload backup metadata from restore location

* Revert "Reload backup metadata from restore location"

This reverts commit 9b47a1cfe9a2682a0908e08cd143373744084fb7.

* Make pytest work/punt the ball on docker config restore issue

* Address pylint error
2025-02-04 17:53:22 +01:00
Stefan Agner
58df65541c
Handle non-existing file in Backup password check too (#5599)
* Handle non-existing file in Backup password check too

Make sure we handle a non-existing backup file also when validating
the password.

* Update supervisor/backups/manager.py

Co-authored-by: Mike Degatano <michael.degatano@gmail.com>

* Add test case and fix password check when multiple locations

* Mock default backup unprotected by default

Instead of setting the protected property which we might not use
everywhere, simply mock the default backup to be unprotected.

* Fix mock of protected backup

* Introduce test for validate_password

Testing showed that validate_password doesn't return anything. Extend
tests to cover this case and fix the actual code.

---------

Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
2025-02-04 11:23:05 +01:00
Mike Degatano
7f39538231
Update cache if a backup file is missing (#5596)
* Update cache if a backup file is missing

* Remove references to single file reload
2025-02-03 13:46:57 +01:00
Stefan Agner
30cbb039d0
Handle non-existing backup file (#5590)
* Make the API return 404 for non-existing backup files

* Introduce BackupFileNotFoundError exception

* Return 404 on full restore as well

* Fix remaining API tests

* Improve error handling in delete

* Fix pytest

* Fix tests and change error handling to agreed logic

---------

Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
2025-01-31 14:27:24 +01:00
Stefan Agner
28a87db515
Avoid test failure by not checking exact size of backup (#5594)
* Avoid test failure by not checking exact size of backup

This is a workaround for the fact that the backup size is not exactly
the same every time. This is due to the fact that the inner gziped tar
file can vary in size due to difference in json file (key order) and
potentially also different field values (UUID, backup slug).

It seems that sorting the keys makes the actual difference today, but
this has runtime overhead and might not catch all cases.

Simply check if size property is there and a number bigger than 0
instead.

* Fix pytest
2025-01-31 11:30:43 +01:00
Mike Degatano
8b897ba537
Fix bug when uploading backup to a mount (#5585) 2025-01-28 18:30:37 +01:00
Mike Degatano
c8f1b222c0
Add sizes per location and support .local (#5581) 2025-01-28 11:41:51 +01:00
Stefan Agner
1b0aa30881
Extend backup upload API with file name parameter (#5568)
* Extend backup upload API with file name parameter

Add a query parameter which allows to specify the file name on upload.
All locations will store the backup with the same file name.

* ruff format

* Update tests to cover bad filename

* Fix ruff check error

* Drop unnecessary logging
2025-01-27 10:01:29 +01:00
Stefan Agner
690f1c07a7
Use version which is treated CalVer by AwesomeVersion (#5572)
* Use version which is treated CalVer by AwesomeVersion

The current dev version `99.9.9dev` is treated as unkown version type
by AwesomeVersion. This prevents the version from comparing with
actual Supervisor versions, e.g. from an exsiting backup file.

Make the development version a valid CalVer version so development
versions can handle non-development backups.

* Bump to year 9999
2025-01-24 09:59:50 +01:00
Mike Degatano
61a2101d8a
Backup protected status can vary per location (#5569)
* Backup protected status can vary per location

* Fix test_backup_remove_error test

* Update supervisor/backups/backup.py

* Add Docker registry configuration to backup metadata

* Make use of backup location fixture

* Address pylint

---------

Co-authored-by: Stefan Agner <stefan@agner.ch>
2025-01-23 15:05:35 -05:00
Stefan Agner
b7412b0679
Update Python to 3.13 (#5564)
* Bump Supervisor to Python 3.13

* Update ruff configuration to 0.9.1

Adjust pyproject.toml for ruff 0.9.1. Also make sure that latest version
of ruff is used in pre-commit.

* Set default configuration for pytest-asyncio

* Run ruff check

* Drop deprecated decorator no_type_check_decorator

The upstream PR (https://github.com/python/cpython/issues/106309) says
this never got really implemented by type checkers.

* Bump devcontainer to latest release
2025-01-21 11:57:30 +01:00
Mike Degatano
0073227785
Add env on core restart due to restore (#5548)
* Add env on core restart due to restore

* Move is_restore to backup manager
2025-01-16 18:15:06 +01:00
Mike Degatano
600bf91c4f
Sort jobs by creation in API (#5545)
* Sort jobs by creation in API

* Fix tests missing new field

* Fix sorting logic around child jobs
2025-01-16 09:51:44 +01:00
Stefan Agner
c2f6e319f2
Check password early on backup restore (#5519)
Introduce a validate password method which only peaks into the archive
to validate the password before starting the actual restore process.
This makes sure that a wrong password returns an error even when
restoring the backup in background.
2024-12-31 13:58:12 +01:00
Mike Degatano
d8101ddba8
Use status 404 in more places when appropriate (#5480) 2024-12-17 11:18:32 +01:00
Mike Degatano
de68868788
Restore backup from specific location (#5491) 2024-12-17 11:09:32 +01:00
Mike Degatano
90590ae2de
Add all addons flag to partial backups (#5490) 2024-12-16 18:25:58 +01:00
Mike Degatano
02ceb713ea
Add location to backup download and remove APIs (#5482) 2024-12-12 19:44:40 +01:00
Mike Degatano
774aef74e8
Backup not found returns 404 instead of 400 (#5479) 2024-12-10 22:30:07 +01:00
Mike Degatano
1f893117cc
Fix backup consolidate and upload duplicate (#5472) 2024-12-09 10:03:49 +01:00
Mike Degatano
d44e995aed
Add size in bytes to backups (#5473) 2024-12-07 10:27:23 +01:00