* Return cursor of the first host logs entry via headers
Return first entry's cursor via custom `X-First-Cursor` header that can
be consumed by the client and used for continual requesting of the
historic logs. Once the first fetch returns data, the cursor can be
supplied as the first argument to the Range header in another call,
fetching accurate slice of the journal with the previous log entries
using the `Range: entries=cursor[[:num_skip]:num_entries]` syntax.
Let's say we fetch logs with the Range header `entries=:-19:20` (to
fetch very last 20 lines of the logs, see below why not
`entries:-20:20`) and we get `cursor50` as the reply (the actual value
will be much more complex and with no guaranteed format). To fetch
previous slice of the logs, we use `entries=cursor50:-20:20`, which
would return 20 lines previous to `cursor50` and `cursor30` in the
cursor header. This way we can go all the way back to the history.
One problem with the cursor is that it's not possible to determine when
the negative num_skip points beyond the first log entry. In that case
the client either needs to know what the first entry is (via
`entries=:0:1`) or can iterate naively and stop once two subsequent
requests return the same first cursor.
Another caveat, even though it's unlikely it will be hit in real usage,
is that it's not possible to fetch the last line only - if no cursor is
provided, negative num_skip argument is needed, and in that case we're
pointing one record back from the current cursor, which is the previous
record. The least we can return without knowing any cursor is thus
`entries=👎2` (where the `2` can be omitted, however with
`entries=👎1` we would lose the last line). This also explains why
different `num_skip` and `num_entries` must be used for the first fetch.
* Fix typo (fallback->callback)
* Refactor journal_logs_reader to always return the cursor
* Update tests for new cursor handling
If no cursor is specified and negative num_skip is used, we're pointing
one record back from the last one, so host logs always returned 101
lines as the default. This was also the case of the lines query argument
that used the number directly as num_skip. Instead of doing that, point
N-1 records to the back and then get N records. Handle 1 record and
invalid numbers silently to avoid the need for error handling in
unpractical edge cases.
* Update DNS plug-in on network change
Restart the DNS plug-in when the primary network changes network
changes. This makes sure any potential host OS DNS configuration
changes get picked up by the DNS plug-in as well.
* Add a test case
---------
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Improve WiFi settings error handling
Currently, the frontend potentially provides no WiFi settings dictionary
but still tries to update other (IP address) settings on the interface.
This leads to a stack trace since network manager is not able to fetch
the WiFi settings from the settings dictionary. Simply fill out what
we can and let NetworkManager provide an error.
Also allow to disable a network interface which has no configuration.
This avoids an error when switching to auto and back to disabled then
press save on a new wireless network interface.
* Add debug message when already disabled
* Add pytest for incomplete WiFi settings as psoted by frontend
Simulate the frontend posting no WiFi settings. Make sure the Supervisor
handles this gracefully.
* Allow to set user DNS through API with auto mode
Currently it is only possible to set DNS servers when in static mode.
However, there are use cases to set DNS servers when in auto mode as
well, e.g. if no local DNS server is provided by the DHCP, or the provided
DNS turns out to be non-working.
* Fix use separate data structure for IP configuration fallout
Make sure gateway is correctly converted to the internal IP
representation. Fix type info.
* Overwrite WiFi settings completely too
* Add test for DNS configuration
* Run ruff format
* ruff format
* Use schema validation as source for API defaults
Instead of using replace() simply set the API defaults in the API
schema.
* Revert "Use schema validation as source for API defaults"
This reverts commit 885506fd37395eb6cea9c787ee23349dac780b75.
* Use explicit dataclass initialization
This avoid the unnecessary replaces from before. It also makes it more
obvious that this part of the API doesn't patch existing settings.
Since headers are clumsy considering the Core proxy between the frontend
and Supervisor, add a way to adjust number of lines and verbose log
format using query parameters as well. If both query parameters and
headers are supplied, prefer the former, as it's more prominent when
reading through the request logs.
* Make IPv4 and IPv6 parse errors raise an API error
Currently, IP address parsing errors lead to an execption which is not
handled by the `api_validate()` call. By using concrete IPv4 and IPv6
types and `vol.Coerce()` parsing errors are properly handled.
* ruff format
* ruff check
* Improve connection settings fixture
Make the connection settings fixture behave more closely to the actual
NetworkManager. The behavior has been tested with NetworkManager 1.42.4
(Debian 12) and 1.44.2 (HAOS 13.1). This likely behaves similar in older
versions too.
* Introduce separate skeleton and settings for wireless
Instead of having a combined network settings object which has
Ethernet and Wirless settings, create a separate settings object for
wireless.
* Handle addresses/address-data property like NetworkManager
* Address ruff check
* Improve network API test
Add a test which changes from "static" to "auto". Validate that settings
are updated accordingly. Specifically, today this does clear the DNS
setting (by not providing the property).
* ruff format
* ruff check
* Complete TEST_INTERFACE rename
* Add partial network update as test case
* Test stub for keeping shared images after update
* Keep shared images on addon update
* ImageNotFound should only skip the one image not all
* Fix tests and nonetype error
* Normalize logic between two cleanup methods
* Add manual_forced option to addon boot config
* Include client library in pull request template
* Add boot_config to api output so frontend can use it
* `manual_forced` to `manual_only`
This PR minimizes the D-Bus requirements for tests. It does this by
using dbus-daemon directly instead of dbus-launch. The latter is meant
for graphical applications and therefor has X11 dependencies. It also
leaves the D-Bus daemon running after the tests are done. This will
accumulate dbus-daemon processes over time which is not ideal.
I've also considered using dbus-run-session since it is meant to launch
processes with a private D-Bus session. For Python tests one could
launch it like so:
dbus-run-session -- python3 -m pytest ...
Then `DBUS_SESSION_BUS_ADDRESS` would be used automatically by the
`MessageBus` class. However, to keep the current behavior of the tests,
launching the D-Bus daemon manually is the better option.
* Use separate data structure for IP configuration
So far we use the same IpConfig data structure to represent the users
IP setting and the currently applied IP configuration.
This commit separates the two in IpConfig (for the currently applied
IP configuration) and IpSetting (representing the user provided IP
setting).
* Use custom string constants for connection settings
Use separate string constants for all connection settings. This makes
it easier to search where a particular NetworkManager connection
setting is used.
* Use Python typing for IpAddress in IpProperties
* Address pytest issue
* Improve and extend error handling on D-Bus connect
Avoid initializing the Supervisor board since it does not support the
Properties interface (see https://github.com/home-assistant/os-agent/issues/206).
This prevents the following somewhat confusing warning:
No OS-Agent support on the host. Some Host functions have been disabled.
The OS Agent is actually installed on the host, it is just a single
object which caused issues. No functionalty was actually lost, as the
Supervisor board object has no features currently, and all other
interfaces got properly initialized still (thanks to gather()).
Print warnings more fine graned so it is clear which object exactly
causes an issue. Also print a log message on the lowest layer when an
error occures on calling D-Bus. This allows to easier track the actual
D-Bus error source.
Fixes: #5241
* Fix tests
* Use local variable
* Avoid stack trace when board support fails to load
* Fix tests
* Use override mechanism to disable Properties support
Instead of disable loading of Supervised entirly override initialization
to prevent loading the Properties interface.
* Revert "Fix tests"
This reverts commit 1e3c491ace2ebef60945c45573acd8d4464ea83e.
* Stop backup if pre backup failed in Core
* Fix API tests
* Partial backup in ci since there's no Home assistant
* Add ssl folder to partial backup
* Allow backups when Home Assistant is not running
* Undo change to skip db test
* Home Assistant watchdog attempts safe mode after max fails
* Remove duplicate line
* Refactor and logging change from feedback
* Update supervisor/misc/tasks.py
* Fix log text check in test
---------
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Max retries for auto applying addon image fixup
* Update supervisor/resolution/fixups/addon_execute_repair.py
Co-authored-by: Stefan Agner <stefan@agner.ch>
---------
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Fix no changelog API response
* Add comment reasoning HTTP 200 for no changelog
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Apply suggestions from code review
Co-authored-by: Jan Čermák <sairon@users.noreply.github.com>
---------
Co-authored-by: Jan Čermák <sairon@users.noreply.github.com>
Co-authored-by: Stefan Agner <stefan@agner.ch>
We do not need to capture HostNotSupported errors to Sentry. The only
possible code path this error might come from is where the Journal
Gateway Daemon socket is unavailable, which is already reported as an
"Unsupported system" repair.
* Use signals to recognize new disks immediately
* Add test for disabled data disk issue
* Add mock of UDisks2 base service to test
* Apply suggestions from code review
* Shutdown manager first to avoid potential race conditions
* Update tests/dbus_service_mocks/udisks2.py
Co-authored-by: Jan Čermák <sairon@users.noreply.github.com>
---------
Co-authored-by: Stefan Agner <stefan@agner.ch>
Co-authored-by: Jan Čermák <sairon@users.noreply.github.com>
Supervisor logs fallback in get_supervisor_logs didn't work properly
because the exception was caught in api_process_raw instead. This was
not discovered in tests because the side effect raised OSError, which
isn't handled there.
To address that, I split the advanced_logs to two functions, one being a
wrapped API handler, one being plain function returning response without
any additional error handling. The tests now check for both cases of
errors (HassioError and random generic Python error).
Refs #5021
* Allow adoption of existing data disk
* Fix existing tests
* Add test cases and fix image issues
* Fix addon build test
* Run checks during setup not startup
* Addon load mimics plugin and HA load for docker part
* Default image accessible in except
* Use Systemd Journal API for all logs endpoints in API
Replace all logs endpoints using container logs with wrapped
advanced_logs function, adding possibility to get logs from previous
boots and following the logs. Supervisor logs are an excetion where
Docker logs are still used - in case an exception is raised while
accessing the Systemd logs, they're used as fallback - otherwise we
wouldn't have an easy way to see what went wrong.
* Refactor testing of advanced logs endpoints to a common method
* Send error while fetching Supervisor logs to Sentry; minor cleanup
* Properly handle errors and use consistent content type in logs endpoints
* Replace api_process_custom with reworked api_process_raw per @mdegat01 suggestion
As described in #4996, Docker 25+ changes made sub-mounts of the /dev
filesystem to be mounted read-only. Revert to the previous behavior by
adjusting the ReadOnlyNonRecursive option. Cleaner way would be to
upstream support for setting this option via Mount class arguments, so
this change is meant to be rather a hotfix for the issue. Even better
approach would be mounting /dev non-recursively, and taking care of
creating all necessary filesystems when creating containers in
Supervisor.
* Unsupported if wrong image used on virtualization
* Add generic-aarch64 as supported image
* Add virtualization field to API
* Change startup to setup in check
* Use Journal Export Format for host (advanced) logs
Add methods for handling Journal Export Format and use it for fetching
of host logs. This is foundation for colored streaming logs for other
endpoints as well.
* Make pylint happier - remove extra pass statement
* Rewrite journal gateway tests to mock ClientResponse's StreamReader
* Handle connection refused error when connecting to journal-gatewayd
* Use SYSTEMD_JOURNAL_GATEWAYD_SOCKET global path also for connection
* Use parsing algorithm suggested by @agners in review
* Fix timestamps in formatting, always use UTC for now
* Add tests for Accept header in host logs
* Apply suggestions from @agners
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Bail out of parsing earlier if field is not in required fields
* Fix parsing issue discovered in the wild and add test case
* Make verbose formatter more tolerant
* Use some bytes' native functions for some minor optimizations
* Move MalformedBinaryEntryError to exceptions module, add test for it
---------
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Cache existence of addon paths
* Always update submodules
* Switch to an always cached model
* Cache on store addon only
* Fix tests
* refresh_cache to refresh_path_cache
* Fix name change in test
* Move logic into StoreManager
* Allow client to change boot slot via API
* Wrap call to rauc in job that checks for OS
* Reboot after changing the active boot slot
* Add test cases and clean up
* BootName to BootSlot
* Fix test
* Rename boot_name to boot_slot
* Fix tests after field change
