* Avoid aiodns resolver memory leak
In certain cases, the aiodns resolver can leak memory. This also
leads to Fatal `Python error… ffi.from_handle()`. This addresses
the issue by ensuring that the resolver is properly closed
when it is no longer needed.
* Address coderabbitai feedback
* Fix pytest
* Fix pytest
Configurable and w/ migrations between IPv4-Only and Dual-Stack
Signed-off-by: David Rapan <david@rapan.cz>
Co-authored-by: Stefan Agner <stefan@agner.ch>
This reverts commit 63fde3b4109310e95ebdcc8e3c23a04ff96ba592.
This change introduced another more severe regression, causing all
add-ons that haven't been started since Supervisor startup to cause
errors during their backup. More sophisticated check would have to be
implemented to address edge cases during backups for non-existing
add-ons (or their config actually).
Fixes#5924
* Avoid early DNS plug-in start
A connectivity check can potentially be triggered before the DNS
plug-in is loaded. Avoid calling restart on the DNS plug-in before
it got initially loaded. This prevents starting before attaching.
The attaching makes sure that the DNS plug-in container is recreated
before the DNS plug-in is initially started, which is e.g. needed
by a potentially hassio network configuration change (e.g. the
migration required to enable/disable IPv6 on the hassio network,
see #5879).
* Mock DNS plug-in running
* Use journal-gatewayd's new /boots endpoint to list boots
Current method we use for getting boots has several known downsides, for
example it can miss some incomplete boots and the performance might be
worse than what we could get by using Systemd directly. Systemd was
missing a method to get list boots through the journal-gatewayd but that
should be addressed by the new /boots endpoint added in [1] which
returns application/json-seq response containing all boots as reported
in `journalctl --list-boots`.
Implement Supervisor methods to parse this format and use the endpoint
at first, falling back to the old method if it fails.
[1] https://github.com/systemd/systemd/pull/37574
* Log info instead of warning when /boots is not present
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Split records only by RS instead of LF in journal_boots_reader
* Strip only RS, json.loads is fine with whitespace
---------
Co-authored-by: Stefan Agner <stefan@agner.ch>
Process NetworkManager interface updates in case PrimaryConnection
changes. This makes sure that the /network/interface/default/info
endpoint can be used to get the IP address of the primary interface.
* Use add-on config timestamp to determine add-on update age
Instead of using the current timestamp when loading the add-on config,
simply use the add-on config modification timestamp. This way, we can
get a timetsamp even when Supervisor got restarted. It also simplifies
the code a bit.
* Fix pytest
* Patch stat() instead of modifing fixture files
* feat: Add IPv6 address generation mode & privacy extensions
Signed-off-by: David Rapan <david@rapan.cz>
* Use NetworkManager fixture for settings init tests
This fixes the test by since the extended implementation now can read
the version of NetworkManager.
* Add pytest for addr_gen_mode
---------
Signed-off-by: David Rapan <david@rapan.cz>
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Trigger auto-update through Core WebSocket call
Instead of auto-updating add-ons on Supervisor side trigger an update
through Core via a WebSocket command. This makes sure that the backup
is categorized correctly and all backup features like retention are
applied.
* Add pytest
* Fix pytest
* Fix pytest
* Fix pytest
* Fix pytest
* Fix pytest cleaner
* Set timestamp of add-on far into the past
Instead of copying the backup in the main job, lets copy them in
separate job per location. This allows to use the same backup error
handling mechanism as for add-ons and folders.
This makes the stage introduced in #5784 somewhat redundant, but
before removing it, let's see if this approach works out.
* Harmonize folder and add-on backup error handling
Align add-on and folder backup error handling in that in both cases
errors are recorded on the respective backup Jobs, but not raised to
the caller. This allows the backup to complete successfully even if
some add-ons or folders fail to back up.
Along with this, also record errors in the per-add-on and per-folder
backup jobs, as well as the add-on and folder root job.
And finally, align the exception handling to only catch expected
exceptions for add-ons too.
* Fix pytest
* Handle non-existing addon config dir
Since users have access to the root of all add-on config directories,
they can delete the directory of an add-ons at any time. Hence we need
to handle gracefully if it doesn't exist anymore.
* Add pytest
* Recreate aiohttp ClientSession after DNS plug-in load
Create a temporary ClientSession early in case we need to load version
information from the internet. This doesn't use the final DNS setup
and hence might fail to load in certain situations since we don't have
the fallback mechanims in place yet. But if the DNS container image
is present, we'll continue the setup and load the DNS plug-in. We then
can recreate the ClientSession such that it uses the DNS plug-in.
This works around an issue with aiodns, which today doesn't reload
`resolv.conf` automatically when it changes. This lead to Supervisor
using the initial `resolv.conf` as created by Docker. It meant that
we did not use the DNS plug-in (and its fallback capabilities) in
Supervisor. Also it meant that changes to the DNS setup at runtime
did not propagate to the aiohttp ClientSession (as observed in #5332).
* Mock aiohttp.ClientSession for all tests
Currently in several places pytest actually uses the aiohttp
ClientSession and reaches out to the internet. This is not ideal
for unit tests and should be avoided.
This creates several new fixtures to aid this effort: The `websession`
fixture simply returns a mocked aiohttp.ClientSession, which can be
used whenever a function is tested which needs the global websession.
A separate new fixture to mock the connectivity check named
`supervisor_internet` since this is often used through the Job
decorator which require INTERNET_SYSTEM.
And the `mock_update_data` uses the already existing update json
test data from the fixture directory instead of loading the data
from the internet.
* Log ClientSession nameserver information
When recreating the aiohttp ClientSession, log information what
nameservers exactly are going to be used.
* Refuse ClientSession initialization when API is available
Previous attempts to reinitialize the ClientSession have shown
use of the ClientSession after it was closed due to API requets
being handled in parallel to the reinitialization (see #5851).
Make sure this is not possible by refusing to reinitialize the
ClientSession when the API is available.
* Fix pytests
Also sure we don't create aiohttp ClientSession objects unnecessarily.
* Apply suggestions from code review
Co-authored-by: Jan Čermák <sairon@users.noreply.github.com>
---------
Co-authored-by: Jan Čermák <sairon@users.noreply.github.com>
Similar to #5825, make sure we mock the systemd journal gateway socket
for tests. This makes the test work on systems which have
systemd-journal-gatewayd installed.
* Check local store repository for changes
Instead of simply assume that the local store repository got changed,
use mtime to check if there have been any changes to the local store.
This mimics a similar behavior to the git repository store updates.
Before this change, we end up in the updated repo code path, which
caused a re-read of all add-ons on every store reload, even though
nothing changed at all. Store reloads are triggered by Home Assistant
Core every 5 minutes.
* Fix pytest failure
Now that we actually only reload metadata if the local store changed
we have to fake the change as well to fix the store manager tests.
* Fix path cache update test for local store repository
* Take root directory into account/add pytest
* Rename utils/__init__.py tests to test_utils_init.py
* Improve backup upload location determination
For local backup upload locations, check if the location is on the same
file system an thuse allows to move the backup file after upload. This
allows custom backup mounts. Currently there is no documented,
persistent way to create such mounts in with Home Assistant OS
installations, but since we might add local mounts in the future this
seems a worthwhile addition.
Fixes: #5837
* Fix pytests
* Add basic test coverage for /auth API
* Check /auth API is called from an add-on
Currently the /auth API is only available for add-ons. Return 403
for calls not originating from an add-on.
* Handle bad json in auth API
Use the API specific JSON load helper which raises an APIError. This
causes the API to return a 400 error instead of a 500 error when the
JSON is invalid.
* Avoid redefining name 'mock_check_login'
* Update tests/api/test_auth.py
* Add dedicated update information reload
Currently we have the /refresh_updates endpoint which updates the main
component versions (Core, OS, Supervisor, Plug-ins) and the add-on
store at the same time. This combined update causes more update
information reloads than necessary.
To allow fine grained update refresh control introduce a new endpoint
/reload_updates which asks Supervisor to only update main component
versions (learned through the version json files).
The /store/reload endpoint already allows to update the add-on store
separately.
* Add pytest
* Update supervisor/api/__init__.py
* Unify Supervisor event message functions
Unify functions which send WebSocket messages of type
"supervisor/event". This deduplicates code and hopefully avoids further
diversication in the future.
While at it, remove unused HomeAssistantWSNotSupported exception. It
seems the only place this exception is used got removed in #3317.
* Test message delivery during shutdown states
Similar to timezone also add country information to the Supervisor
info. This is useful to set country specific configurations such as
Wireless radio regulatory setting. This is also useful for add-ons
which need country information but only have hassio API access.
Since Systemd v256 the Range header must not end with a trailing colon.
We relied on this undocumented feature when following logs, and the
frontend or CLI may still use it in requests. To fix the requests
failing with new Systemd version, intercept the header and fill in the
num_entries to maximum possible value, which avoids the journal-gatewayd
returning the response prematurely and also works on older Systemd
versions.
The journal-gatewayd would still return response if follow flag is used
along with num_entries, but this behavior is unchanged and would be
better fixed in the backend.
Link: https://github.com/systemd/systemd/issues/37172
* Fix root path requests
Since #5759 we've tried to access the path explicitly. However, this
raises KeyError exception when trying to access the proxied root path
(e.g. http://supervisor/core/api/). Before #5759 get was used, which
lead to no exception, but instead inserted a `None` into the path.
It seems aiohttp doesn't provide a path when the root is accessed. So
simply convert this to no path as well by setting path to an empty
string.
* Add rudimentary pytest for regular proxy requets
* Fix mypy issues in backups module
* Fix mypy issues in dbus module
* Fix mypy issues in api after rebase
* TypedDict to dataclass and other small fixes
* Finish fixing mypy errors in dbus
* local_where must exist
* Fix references to name in tests
* Improve Home Assistant Core WebSocket proxy implementation
This change removes unnecessary task creation for every WebSocket
message and instead creates just two tasks, one for each direction.
This improves performance by about factor of 3 when measuring 1000
WebSocket requests to Core (from ~530ms to ~160ms).
While at it, also handle all WebSocket message related to closing the
WebSocket and report all other errors as warnings instead of just info.
* Improve logging and error handling
* Add WS client error test case
* Use asyncio.gather directly
* Use asyncio.wait to handle exceptions gracefully
* Drop cancellation handling and correctly wait for the other proxy task
* Add API for swap configuration
Add HTTP API for swap size and swappiness to /os/config/swap. Individual
options can be set in JSON and are calling the DBus API added in OS
Agent 1.7.x, available since OS 15.0. Check for presence of OS of the
required version and return 404 if the criteria are not met.
* Fix type hints and reboot_required logic
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Fix formatting after adding suggestions from GH
* Address @mdegat01 review comments
- Improve swap options validation
- Add swap to the 'all' property of dbus agent
- Use APINotFound with reason instead of HTTPNotFound
- Reorder API routes
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Report stage with error in jobs
* Copy doesn't lose track of the successful copies
* Add stage to errors in api output test
* revert unneessary change to import
* Add tests for a bit more coverage of copy_additional_locations
* Handle unexpected WebSocket messages during auth
When an add-on does not respond or closes the WebSocket connection
during the authentication phase Supervisor does not handle errors
gracefully. Simply log such unexpected authentication to avoid
unnecessary stack traces in the log and make such cases no longer
appear on Sentry.
* Add pytest
* Introduce a timeout of 10s
WipeDevice method was dropped from OS Agent code in [1]. Remove it from
the mock class to sync with the current API. There is no usage of
WipeDevice in the Supervisor codebase, only ScheduleWipeDevice is
called.
[1] https://github.com/home-assistant/os-agent/pull/225
* Finish out effort of adding and enabling blockbuster
* Skip getting addon file size until securetar fixed
* Fix test for devcontainer and blocking I/O
* Fix docker fixture and load_config to post_init
* Use Sentry helper function to report warnings
Don't use Sentry directly but the existing helper function.
* Add pytest that Sentry is by default off
* Address ruff
* Address ruff
* Add blockbuster library and find I/O from unit tests
* Fix lint and test issue
* Fixes from feedback
* Avoid modifying webapp object in executor
* Split su options validation and only validate timezone on change
* Move read_text to executor
* switch to async_capture_exception
* Finish moving read_text to executor
* Cover read_bytes and some write_text calls as well
* Fix await issues
* Fix format_message
* Replace non-unicode characters for add-on static files
Add-on documentation and changelog get read and returned as text file.
However, in case the original author used non-unicode characters, or
the file corrupted, loading currently fails with an UnicodeDecodeError.
Let's just use the built-in replace error handling of Python, so they
appear for the user as non-unicode characters by replacing them with
the official unicode replacement character "�".
* Remove superflous parameter for binary files
* ruff format
* Add pytests
* Fix cloning of add-on store repository
Since #5669, the add-on store reset no longer deletes the root
directory. However, if the root directory is not present, the current
code no longer invokes cloning, instead tries to load the git
repository directly.
With this change, the code clones whenever there is no .git directory,
which works for both cases.
* Fix pytest
* Move read_text to executor
* Fix issues found by coderabbit
* formated to formatted
* switch to async_capture_exception
* Find and replace got one too many
* Update patch mock to async_capture_exception
* Drop Sentry capture from format_message
The error handling got introduced in #2052, however, #2100 essentially
makes sure there will never be a byte object passed to this function.
And even if, the Sentry aiohttp plug-in will properly catch such an
exception.
---------
Co-authored-by: Stefan Agner <stefan@agner.ch>
Since #5696 we don't need to load the resolution center early. In fact,
with #5686 this is even problematic for pytests in devcontainer, since
the Supervisor Core state is valid and this causes AppArmor evaluations
to run (and fail).
Actually, #5696 removed the resolution center. #5686 brought it
accidentally back. This was seemingly a merge error.
* Initialize Supervisor Core state in constructor
Make sure the Supervisor Core state is set to a value early on. This
makes sure that the state is always of type CoreState, and makes sure
that any use of the state can rely on it being an actual value from the
CoreState enum.
This fixes Sentry filter during early startup, where the state
previously was None. Because of that, the Sentry filter tried to
collect more Context, which lead to an exception and not reporting
errors.
* Fix pytest
It seems that with initializing the state early, the pytest actually
runs a system evaluation with:
Starting system evaluation with state initialize
Before it did that with:
Starting system evaluation with state None
It detects that the container runs as privileged, and declares the
system as unhealthy.
It is unclear to me why coresys.core.healthy was checked in this
context, it doesn't seem useful. Just remove the check, and validate
the state through the getter instead.
* Update supervisor/core.py
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Make sure Supervisor container is privileged in pytest
With the Supervisor Core state being valid now, some evaluations
now actually run when loading the resolution center. This leads to
Supervisor getting declared unhealthy due to not running in a privileged
container under pytest.
Fake the host container to be privileged to make evaluations not
causing the system to be declared unhealthy under pytest.
* Avoid writing actual Supervisor run state file
With the Supervisor Core state being valid from the very start, we end
up writing a state everytime.
Instead of actually writing a state file, simply validate the the
necessary calls are being made. This is more conform to typical unit
tests and avoids writing a file for every test.
* Extend WebSocket client fixture and use it consistently
Extend the ha_ws_client WebSocket client fixture to set Supervisor Core
into run state and clear all pending messages.
Currently only some tests use the ha_ws_client WebSocket client fixture.
Use it consistently for all tests.
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
