* Remove unknown errors from addons
* Remove customized unknown error types
* Fix docker ratelimit exception and tests
* Fix stats test and add more for known errors
* Add defined error for when build fails
* Fixes from feedback
* Fix mypy issues
* Fix test failure due to rename
* Change auth reset error message
* Drop ensure_builtin_repositories
With the new Repository classes we have the is_builtin property, so we
can easily make sure that built-ins are not removed. This allows us to
further cleanup the code by removing the ensure_builtin_repositories
function and the ALL_BUILTIN_REPOSITORIES constant.
* Make sure we add built-ins on load
* Reuse default set and avoid unnecessary copy
Reuse default set and avoid unnecessary copying during validation if
the default is not being used.
* Rename repository fixture to test_repository
Also don't remove the built-in repositories. The list was incomplete,
and tests don't seem to require that anymore.
* Get rid of StoreType
The type doesn't have much value, we have constant strings anyways.
* Introduce types.py
* Use slug to determine which repository urls to return
* Simplify BuiltinRepository enum
* Mock GitRepo load
* Improve URL handling and repository creation logic
* Refactor update_repositories
* Get rid of get_from_url
It is no longer used in production code.
* More refactoring
* Address pylint
* Introduce is_git_based property to Repository class
Return all git based URLs, including the Core repository.
* Revert "Introduce is_git_based property to Repository class"
This reverts commit dfd5ad79bf.
* Fold type.py into const.py
Align more with how Supervisor code is typically structured.
* Update supervisor/store/__init__.py
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Apply repository remove suggestion
* Fix tests
---------
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Harmonize folder and add-on backup error handling
Align add-on and folder backup error handling in that in both cases
errors are recorded on the respective backup Jobs, but not raised to
the caller. This allows the backup to complete successfully even if
some add-ons or folders fail to back up.
Along with this, also record errors in the per-add-on and per-folder
backup jobs, as well as the add-on and folder root job.
And finally, align the exception handling to only catch expected
exceptions for add-ons too.
* Fix pytest
* Fix mypy issues in backups module
* Fix mypy issues in dbus module
* Fix mypy issues in api after rebase
* TypedDict to dataclass and other small fixes
* Finish fixing mypy errors in dbus
* local_where must exist
* Fix references to name in tests
* Finish out effort of adding and enabling blockbuster
* Skip getting addon file size until securetar fixed
* Fix test for devcontainer and blocking I/O
* Fix docker fixture and load_config to post_init
* Add blockbuster library and find I/O from unit tests
* Fix lint and test issue
* Fixes from feedback
* Avoid modifying webapp object in executor
* Split su options validation and only validate timezone on change
* Handle permission error on backup create
Make sure we handle (write) permission errors when creating a backup.
* Introduce BackupFileExistError and BackupPermissionError exceptions
* Make error messages a bit more uniform
* Drop use of exclusive mode
SecureTar does not handle exclusive mode nicely. Drop use of it for now.
* Avoid IO in event loop when removing backup
* Refactor backup size calculation
Currently size is lazy loaded when required via properties. This
however is blocking the async event loop.
Backup sizes don't change. Instead of lazy loading the size of a backup
simply determine it on loading/after creation.
* Fix tests for backup size change
* Avoid IO in event loop when loading backups
* Avoid IO in event loop when importing a backup
* Remove I/O from backup create() function
* Move mount check into exectutor thread
* Remove I/O from backup open() function
* Remove I/O from _folder_save()
* Refactor remove_folder and remove_folder_with_excludes
Make remove_folder and remove_folder_with_excludes synchronous
functions which need to be run in an executor thread to be safely used
in asyncio. This makes them better composable with other I/O operations
like checking for file existence etc.
* Fix logger typo
* Use return values for functions running in an exectutor
* Move location check into a separate function
* Fix extract
* Validate Backup always before restoring
Since #5519 we check the encryption password early in restore case.
This has the side effect that we check the file existance early too.
However, in the non-encryption case, the file is not checked early.
This PR changes the behavior to always validate the backup file before
restoring, ensuring both encryption and non-encryption cases are
handled consistently.
In particular, the last case of test_restore_immediate_errors actually
validates that behavior. That test should actually have failed so far.
But it seems that because we validate the backup shortly after freeze
anyways, the exception still got raised early enough.
A simply `await asyncio.sleep(10)` right after the freeze makes the
test case fail. With this change, the test works consistently.
* Address pylint
* Fix backup_manager tests
* Drop warning message
* Drop Docker config from Supervisor backup
The Docker config is part of the main backup metadata. Because we
consolidate encrypted and unencrypted backups today, this leads to
potential bugs when restoring a backup.
* Drop obsolete encrypt/decrypt functions
* Drop unused Backup Job stage
* Handle non-existing file in Backup password check too
Make sure we handle a non-existing backup file also when validating
the password.
* Update supervisor/backups/manager.py
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Add test case and fix password check when multiple locations
* Mock default backup unprotected by default
Instead of setting the protected property which we might not use
everywhere, simply mock the default backup to be unprotected.
* Fix mock of protected backup
* Introduce test for validate_password
Testing showed that validate_password doesn't return anything. Extend
tests to cover this case and fix the actual code.
---------
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Avoid reordering add-on repositories on Backup load
The `ensure_builtin_repositories` function uses a set to deduplicate
items, which sometimes led to a change of order in elements. This is
problematic when deduplicating Backups.
Simply avoid mangling the list of add-on repositories on load. Instead
rely on `update_repositories` which uses the same function to ensure
built-in repositories when loading the store configuration and restoring
a backup file.
* Update tests
* ruff format
* ruff check
* ruff check fixes
* ruff format
* Update tests/store/test_validate.py
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Simplify test
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Make the API return 404 for non-existing backup files
* Introduce BackupFileNotFoundError exception
* Return 404 on full restore as well
* Fix remaining API tests
* Improve error handling in delete
* Fix pytest
* Fix tests and change error handling to agreed logic
---------
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Backup protected status can vary per location
* Fix test_backup_remove_error test
* Update supervisor/backups/backup.py
* Add Docker registry configuration to backup metadata
* Make use of backup location fixture
* Address pylint
---------
Co-authored-by: Stefan Agner <stefan@agner.ch>
Introduce a validate password method which only peaks into the archive
to validate the password before starting the actual restore process.
This makes sure that a wrong password returns an error even when
restoring the backup in background.
* Fix and extend cloud backup support
* Clean up task for cloud backup and remove by location
* Args to kwargs on backup methods
* Fix backup remove error test and typing clean up
* Add background option to backup APIs
* Fix decorator tests
* Working error handling, initial test cases
* Change to schedule_job and always return job id
* Add tests
* Reorder call at/later args
* Validation errors return immediately in background
* None is invalid option for background
* Must pop the background option from body
* Reduce executor code for docker
* Fix pylint errors and move import/export image
* Fix test and a couple other risky executor calls
* Fix dataclass and return
* Fix test case and add one for corrupt docker
* Add some coverage
* Undo changes to docker manager startup
* Addon startup waits for healthy
* fix import for pylint
* wait_for to 5 in tests
* Adjust tests to simplify async tasks
* Remove wait_boot time from addons.boot tests
* Eliminate async task race conditions in tests
* Speed up backups by increasing buffer size
This is the same change as https://github.com/home-assistant/core/pull/90613
but for supervisor
If the backup takes too long, core will release the lock on the database
and the backup will be no good
2fc34e7cce/homeassistant/components/recorder/core.py (L926)
cpython uses copyfileobj under the hood for fast copies but the default buffer size is quite low which increases the amount of time in python code when copying the sqlite database. As this is the usually the bulk of the backup, increasing the buffer can help reduce the backup time quite a bit.
Ideally this would all use sendfile under the hood as it would shift nearly all the burden out of userspace but tarfile doesn't currently try that 4664a7cf68/Lib/shutil.py (L106)
related:
In testing (non encrypted) improvement was at least as good as https://github.com/python/cpython/issues/71386
* add the const
