* Return cursor of the first host logs entry via headers
Return first entry's cursor via custom `X-First-Cursor` header that can
be consumed by the client and used for continual requesting of the
historic logs. Once the first fetch returns data, the cursor can be
supplied as the first argument to the Range header in another call,
fetching accurate slice of the journal with the previous log entries
using the `Range: entries=cursor[[:num_skip]:num_entries]` syntax.
Let's say we fetch logs with the Range header `entries=:-19:20` (to
fetch very last 20 lines of the logs, see below why not
`entries:-20:20`) and we get `cursor50` as the reply (the actual value
will be much more complex and with no guaranteed format). To fetch
previous slice of the logs, we use `entries=cursor50:-20:20`, which
would return 20 lines previous to `cursor50` and `cursor30` in the
cursor header. This way we can go all the way back to the history.
One problem with the cursor is that it's not possible to determine when
the negative num_skip points beyond the first log entry. In that case
the client either needs to know what the first entry is (via
`entries=:0:1`) or can iterate naively and stop once two subsequent
requests return the same first cursor.
Another caveat, even though it's unlikely it will be hit in real usage,
is that it's not possible to fetch the last line only - if no cursor is
provided, negative num_skip argument is needed, and in that case we're
pointing one record back from the current cursor, which is the previous
record. The least we can return without knowing any cursor is thus
`entries=👎2` (where the `2` can be omitted, however with
`entries=👎1` we would lose the last line). This also explains why
different `num_skip` and `num_entries` must be used for the first fetch.
* Fix typo (fallback->callback)
* Refactor journal_logs_reader to always return the cursor
* Update tests for new cursor handling
* Update DNS plug-in on network change
Restart the DNS plug-in when the primary network changes network
changes. This makes sure any potential host OS DNS configuration
changes get picked up by the DNS plug-in as well.
* Add a test case
---------
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Allow to set user DNS through API with auto mode
Currently it is only possible to set DNS servers when in static mode.
However, there are use cases to set DNS servers when in auto mode as
well, e.g. if no local DNS server is provided by the DHCP, or the provided
DNS turns out to be non-working.
* Fix use separate data structure for IP configuration fallout
Make sure gateway is correctly converted to the internal IP
representation. Fix type info.
* Overwrite WiFi settings completely too
* Add test for DNS configuration
* Run ruff format
* ruff format
* Use schema validation as source for API defaults
Instead of using replace() simply set the API defaults in the API
schema.
* Revert "Use schema validation as source for API defaults"
This reverts commit 885506fd37395eb6cea9c787ee23349dac780b75.
* Use explicit dataclass initialization
This avoid the unnecessary replaces from before. It also makes it more
obvious that this part of the API doesn't patch existing settings.
* Improve connection settings fixture
Make the connection settings fixture behave more closely to the actual
NetworkManager. The behavior has been tested with NetworkManager 1.42.4
(Debian 12) and 1.44.2 (HAOS 13.1). This likely behaves similar in older
versions too.
* Introduce separate skeleton and settings for wireless
Instead of having a combined network settings object which has
Ethernet and Wirless settings, create a separate settings object for
wireless.
* Handle addresses/address-data property like NetworkManager
* Address ruff check
* Improve network API test
Add a test which changes from "static" to "auto". Validate that settings
are updated accordingly. Specifically, today this does clear the DNS
setting (by not providing the property).
* ruff format
* ruff check
* Complete TEST_INTERFACE rename
* Add partial network update as test case
* Use separate data structure for IP configuration
So far we use the same IpConfig data structure to represent the users
IP setting and the currently applied IP configuration.
This commit separates the two in IpConfig (for the currently applied
IP configuration) and IpSetting (representing the user provided IP
setting).
* Use custom string constants for connection settings
Use separate string constants for all connection settings. This makes
it easier to search where a particular NetworkManager connection
setting is used.
* Use Python typing for IpAddress in IpProperties
* Address pytest issue
* Use Journal Export Format for host (advanced) logs
Add methods for handling Journal Export Format and use it for fetching
of host logs. This is foundation for colored streaming logs for other
endpoints as well.
* Make pylint happier - remove extra pass statement
* Rewrite journal gateway tests to mock ClientResponse's StreamReader
* Handle connection refused error when connecting to journal-gatewayd
* Use SYSTEMD_JOURNAL_GATEWAYD_SOCKET global path also for connection
* Use parsing algorithm suggested by @agners in review
* Fix timestamps in formatting, always use UTC for now
* Add tests for Accept header in host logs
* Apply suggestions from @agners
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Bail out of parsing earlier if field is not in required fields
* Fix parsing issue discovered in the wild and add test case
* Make verbose formatter more tolerant
* Use some bytes' native functions for some minor optimizations
* Move MalformedBinaryEntryError to exceptions module, add test for it
---------
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Allow client to change boot slot via API
* Wrap call to rauc in job that checks for OS
* Reboot after changing the active boot slot
* Add test cases and clean up
* BootName to BootSlot
* Fix test
* Rename boot_name to boot_slot
* Fix tests after field change
* Migrate to Ruff for lint and format
* Fix pylint issues
* DBus property sets into normal awaitable methods
* Fix tests relying on separate tasks in connect
* Fixes from feedback
* Bad message error marks system as unhealthy
* Finish adding test cases for changes
* Rename test file for uniqueness
* bad_message to oserror_bad_message
* Omit some checks and check for network mounts
* Backup and restore track progress in job
* Change to stage only updates and fix tests
* Leave HA alone if it wasn't restored
* skip check HA stage message when we don't check
* Change to helper to get current job
* Fix tests
* Mark jobs as internal to skip notifying HA
* Add udisks2 dbus support
* assert mountpoints
* Comment
* Add reference links
* docstring
* fix type
* fix type
* add typing extensions as import
* isort
* additional changes
* Simplify classes and conversions, fix bugs
* More simplification
* Fix imports
* fix pip
* Add additional properties and fix requirements
* fix tests maybe
* Handle optionality of certain configuration details
* black
* connect to devices before returning them
* Refactor for latest dbus work
* Not .items
* fix mountpoints logic
* use variants
* Use variants for options too
* isort
* Switch to dbus fast
* Move import to parent
* Add some fixture data
* Add another fixture and reduce the block devices list
* Implement changes discussed with mike
* Add property fixtures
* update object path
* Fix get_block_devices call
* Tests and refactor to minimize dbus reconnects
* Call super init in DBusInterfaceProxy
* Fix permissions on introspection files
---------
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Add enhanced logging REST endpoints using systemd-journal-gatewayd
Add /host/logs/entries and /host/logs/{identifier}/entries to expose log
entries from systemd-journald running on the host. Use
systemd-journal-gatewayd which exposes the logs to the Supervisor via
Unix socket.
Current two query string parameters are allowed: "boot" and "follow".
The first will only return logs since last boot. The second will keep
the HTTP request open and send new log entries as they get added to the
systemd-journal.
* Allow Range header
Forward the Range header to systemd-journal-gatewayd. This allows to
select only a certain amount of log data. The Range header is a standard
header to select only partial amount of data. However, the "entries="
prefix is custom for systemd-journal-gatewayd, denoting that the numbers
following represent log entries (as opposed to bytes or other metrics).
* Avoid connecting if systemd-journal-gatewayd is not available
* Use path for all options
* Add pytests
* Address pylint issues
* Boot ID offsets and slug to identifier
* Fix tests
* API refactor from feedback
* fix tests and add identifiers
* stop isort and pylint fighting
* fix tests
* Update default log identifiers
* Only modify /host/logs endpoints
* Fix bad import
* Load log caches asynchronously at startup
* Allow task to complete in fixture
* Boot IDs and identifiers loaded on demand
* Add suggested identifiers
* Fix tests around boot ids
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Use the correct interface name to get properties of systemd
It seems that gdbus (or systemd) automatically pick the correct
interface and return the properties. However, dbussy requires the
correct interface name to get all properties.
* Don't expect array from Strength property
The property returns a type "y" which equates to "guchar":
https://developer-old.gnome.org/NetworkManager/stable/gdbus-org.freedesktop.NetworkManager.AccessPoint.html#gdbus-property-org-freedesktop-NetworkManager-AccessPoint.Strength
It seems that the old D-Bus implementation returned an array. With
dbus-next a integer is returned, so no list indexing required.
* Support signals and remove no longer used tests and code
* Pass rauc update file path as string
That is what the interface is expecting, otherwise the new lib chocks on
the Pathlib type.
* Support Network configuration with dbus-next
Assemble Python native objects and pass them to dbus-next. Use dbus-next
specific Variant class where necessary.
* Use org.freedesktop.NetworkManager.Connection.Active.StateChanged
org.freedesktop.NetworkManager.Connection.Active.PropertyChanged is
depricated. Also it seems that StateChanged leads to fewer and more
accurate signals.
* Pass correct data type to RequestScan.
RequestScan expects an option dictionary. Pass an empty option
dictionary to it.
* Update unit tests
Replace gdbus specific fixtures with json files representing the return
values. Those can be easily converted into native Python objects.
* Rename D-Bus utils module gdbus to dbus
* Use bool for host internet
* Ignore host check if no network manager
* Update supervisor/host/network.py
Co-authored-by: Pascal Vizeli <pvizeli@syshack.ch>
* Check dbus connection isntead of supported features
Co-authored-by: Pascal Vizeli <pvizeli@syshack.ch>
* Adds condition decorator to block execution that require internet
* Fix exsisting tests
* Add internet state to network info
* Add healthy condition
* Add tests
* It's all changed
* rename
* Add free disk space info
* Log available space on docker image install error
* Add unit to log message
* Add 404 check and better log message
* Set default path to supervisor data path
* Clean up attribute access
Co-authored-by: Pascal Vizeli <pvizeli@syshack.ch>
* Move free space helper to hardware
* Add hardware test
Co-authored-by: Pascal Vizeli <pvizeli@syshack.ch>
