* Return cursor of the first host logs entry via headers
Return first entry's cursor via custom `X-First-Cursor` header that can
be consumed by the client and used for continual requesting of the
historic logs. Once the first fetch returns data, the cursor can be
supplied as the first argument to the Range header in another call,
fetching accurate slice of the journal with the previous log entries
using the `Range: entries=cursor[[:num_skip]:num_entries]` syntax.
Let's say we fetch logs with the Range header `entries=:-19:20` (to
fetch very last 20 lines of the logs, see below why not
`entries:-20:20`) and we get `cursor50` as the reply (the actual value
will be much more complex and with no guaranteed format). To fetch
previous slice of the logs, we use `entries=cursor50:-20:20`, which
would return 20 lines previous to `cursor50` and `cursor30` in the
cursor header. This way we can go all the way back to the history.
One problem with the cursor is that it's not possible to determine when
the negative num_skip points beyond the first log entry. In that case
the client either needs to know what the first entry is (via
`entries=:0:1`) or can iterate naively and stop once two subsequent
requests return the same first cursor.
Another caveat, even though it's unlikely it will be hit in real usage,
is that it's not possible to fetch the last line only - if no cursor is
provided, negative num_skip argument is needed, and in that case we're
pointing one record back from the current cursor, which is the previous
record. The least we can return without knowing any cursor is thus
`entries=👎2` (where the `2` can be omitted, however with
`entries=👎1` we would lose the last line). This also explains why
different `num_skip` and `num_entries` must be used for the first fetch.
* Fix typo (fallback->callback)
* Refactor journal_logs_reader to always return the cursor
* Update tests for new cursor handling
* Use Journal Export Format for host (advanced) logs
Add methods for handling Journal Export Format and use it for fetching
of host logs. This is foundation for colored streaming logs for other
endpoints as well.
* Make pylint happier - remove extra pass statement
* Rewrite journal gateway tests to mock ClientResponse's StreamReader
* Handle connection refused error when connecting to journal-gatewayd
* Use SYSTEMD_JOURNAL_GATEWAYD_SOCKET global path also for connection
* Use parsing algorithm suggested by @agners in review
* Fix timestamps in formatting, always use UTC for now
* Add tests for Accept header in host logs
* Apply suggestions from @agners
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Bail out of parsing earlier if field is not in required fields
* Fix parsing issue discovered in the wild and add test case
* Make verbose formatter more tolerant
* Use some bytes' native functions for some minor optimizations
* Move MalformedBinaryEntryError to exceptions module, add test for it
---------
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Improve json performance by porting core orjson utils
* port relevant tests
* pylint
* add test for read_json_file
* add test for read_json_file
* remove workaround for core issue we do not have here
---------
Co-authored-by: Pascal Vizeli <pvizeli@syshack.ch>
* Make check_port asyncio
This requires to change the ingress_port property to a async method.
* Avoid using wait_for
* Add missing async
* Really await
* Set dynamic ingress port on add-on installation/update
* Fix pytest issue
* Rename async_check_port back to check_port
* Raise RuntimeError in case port is not set
* Make sure port gets set on add-on restore
* Drop unnecessary async
* Simplify check_port by using asyncio.get_running_loop()
* Improve D-Bus error handling for NetworkManager
There are quite some errors captured which are related by seemingly a
suddenly missing NetworkManager. Errors appear as:
23-11-21 17:42:50 ERROR (MainThread) [supervisor.dbus.network] Error while processing /org/freedesktop/NetworkManager/Devices/10: Remote peer disconnected
...
23-11-21 17:42:50 ERROR (MainThread) [supervisor.dbus.network] Error while processing /org/freedesktop/NetworkManager/Devices/35: The name is not activatable
Both errors seem to already happen at introspection time, however
the current code doesn't converts these errors to Supervisor issues.
This PR uses the already existing `DBus.from_dbus_error()`.
Furthermore this adds a new Exception `DBusNoReplyError` for the
`ErrorType.NO_REPLY` (or `org.freedesktop.DBus.Error.NoReply` in
D-Bus terms, which is the type of the first of the two issues above).
And finally it separates the `ErrorType.SERVICE_UNKNOWN` (or
`org.freedesktop.DBus.Error.ServiceUnknown` in D-Bus terms, which is
the second of the above issue) from `DBusInterfaceError` into a new
`DBusServiceUnkownError`.
This allows to handle errors more specifically.
To avoid too much churn, all instances where `DBusInterfaceError`
got handled, we are now also handling `DBusServiceUnkownError`.
The `DBusNoReplyError` and `DBusServiceUnkownError` appear when
the NetworkManager service stops or crashes. Instead of retrying
every interface we know, just give up if one of these issues appear.
This should significantly lower error messages users are seeing
and Sentry events.
* Remove unnecessary statement
* Fix pytests
* Make sure error strings are compared correctly
* Fix typo/remove unnecessary pylint exception
* Fix DBusError typing
* Add pytest for from_dbus_error
* Revert "Make sure error strings are compared correctly"
This reverts commit 10dc2e4c3887532921414b4291fe3987186db408.
* Add test cases
---------
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Make dbus-fast calls more robust
* Handle all exceptions and add test
* DBus minimal can't return commands list
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Revert "Disable CAS until it work (#3504)"
This reverts commit 435241bccf3f3c678161b50cfefd4b036f1daba9.
* Revert exception that are not forwarded
* enable for add-ons
* Apply suggestions from code review
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* fix black
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Use the correct interface name to get properties of systemd
It seems that gdbus (or systemd) automatically pick the correct
interface and return the properties. However, dbussy requires the
correct interface name to get all properties.
* Don't expect array from Strength property
The property returns a type "y" which equates to "guchar":
https://developer-old.gnome.org/NetworkManager/stable/gdbus-org.freedesktop.NetworkManager.AccessPoint.html#gdbus-property-org-freedesktop-NetworkManager-AccessPoint.Strength
It seems that the old D-Bus implementation returned an array. With
dbus-next a integer is returned, so no list indexing required.
* Support signals and remove no longer used tests and code
* Pass rauc update file path as string
That is what the interface is expecting, otherwise the new lib chocks on
the Pathlib type.
* Support Network configuration with dbus-next
Assemble Python native objects and pass them to dbus-next. Use dbus-next
specific Variant class where necessary.
* Use org.freedesktop.NetworkManager.Connection.Active.StateChanged
org.freedesktop.NetworkManager.Connection.Active.PropertyChanged is
depricated. Also it seems that StateChanged leads to fewer and more
accurate signals.
* Pass correct data type to RequestScan.
RequestScan expects an option dictionary. Pass an empty option
dictionary to it.
* Update unit tests
Replace gdbus specific fixtures with json files representing the return
values. Those can be easily converted into native Python objects.
* Rename D-Bus utils module gdbus to dbus
* Set 600 premissions on json files
* Add test
* Fix local tar tests
* Fix tar test in action
* Use pytest fixture for tmp_path in tests
* remove not needed things
