Commit Graph

56 Commits

Author SHA1 Message Date
Joakim Plate
c576a68d33 Upgrade pytest-aiohttp (#82475)
* Upgrade pytest-aiohttp

* Make sure executors, tasks and timers are closed

Some test will trigger warnings on garbage collect, these warnings
spills over into next test.

Some test trigger tasks that raise errors on shutdown, these spill
over into next test.

This is to mimic older pytest-aiohttp and it's behaviour on test
cleanup.

Discussions on similar changes for pytest-aiohttp are here:
https://github.com/pytest-dev/pytest-asyncio/pull/309

* Replace loop with event_loop

* Make sure time is frozen for tests

* Make sure the ConditionType is not async

  /home-assistant/homeassistant/helpers/template.py:2082: RuntimeWarning: coroutine 'AsyncMockMixin._execute_mock_call' was never awaited
    def wrapper(*args, **kwargs):
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

* Increase litejet press tests with a factor 10

The times are simulated anyway, and we can't stop the normal
event from occuring.

* Use async handlers for aiohttp

tests/components/motioneye/test_camera.py::test_get_still_image_from_camera
tests/components/motioneye/test_camera.py::test_get_still_image_from_camera
tests/components/motioneye/test_camera.py::test_get_stream_from_camera
tests/components/motioneye/test_camera.py::test_get_stream_from_camera
tests/components/motioneye/test_camera.py::test_camera_option_stream_url_template
tests/components/motioneye/test_camera.py::test_camera_option_stream_url_template
  /Users/joakim/src/hass/home-assistant/venv/lib/python3.9/site-packages/aiohttp/web_urldispatcher.py:189: DeprecationWarning: Bare functions are deprecated, use async ones
    warnings.warn(

* Switch to freezegun in modbus tests

The tests allowed clock to tick in between steps

* Make sure skybell object are fully mocked

Old tests would trigger attempts to post to could services:

```
DEBUG:aioskybell:HTTP post https://cloud.myskybell.com/api/v3/login/ Request with headers: {'content-type': 'application/json', 'accept': '*/*', 'x-skybell-app-id': 'd2b542c7-a7e4-4e1e-b77d-2b76911c7c46', 'x-skybell-client-id': '1f36a3c0-6dee-4997-a6db-4e1c67338e57'}
```

* Fix sorting that broke after rebase
2022-11-29 22:36:36 +01:00
Erik Montnemery
2fe8e95309 Add helper to calculate statistic period start and end (#82493)
* Add helper to calculate statistic period start and end

* Don't parse values in resolve_period

* Add specific test for resolve_period

* Improve typing

* Move to recorder/util.py

* Extract period schema
2022-11-26 13:00:40 -05:00
Erik Montnemery
f4951a4f31 Add CI job which runs recorder tests on MariaDB (#80586)
Co-authored-by: Franck Nijhof <git@frenck.dev>
2022-10-19 13:04:28 +02:00
Erik Montnemery
31a787558f Ensure recorder test fixture is setup before hass fixture (#80528)
* Ensure recorder test fixture is setup before hass fixture

* Adjust more tests
2022-10-19 07:58:47 +02:00
epenet
019d297ff0 Apply hass-relative-import to recorder tests (#78734) 2022-09-19 12:35:03 +02:00
Erik Montnemery
606d544157 Use recorder get_instance function to improve typing (#75567) 2022-07-22 11:58:26 +02:00
J. Nick Koston
00a79635c1 Revert "Remove sqlalchemy lambda_stmt usage from history, logbook, and statistics (#73191)" (#73917) 2022-06-23 13:59:55 -05:00
J. Nick Koston
7ae8bd5137 Remove sqlalchemy lambda_stmt usage from history, logbook, and statistics (#73191) 2022-06-07 16:15:50 -07:00
Erik Montnemery
5f2b4001f3 Separate recorder database schema from other classes (#72977)
* Separate recorder database schema from other classes

* fix logbook imports

* migrate new tests

* few more

* last one

* fix merge

Co-authored-by: J. Nick Koston <nick@koston.org>
2022-06-07 14:41:43 +02:00
Erik Montnemery
edd7a3427c Remove support for databases without ROW_NUMBER (#72092) 2022-05-18 21:52:38 -05:00
Erik Montnemery
037f6947d8 Fail recorder setup with unsupported dialect or version (#70888) 2022-05-18 16:52:46 +02:00
J. Nick Koston
a4c1bcefb9 Tune sqlite based on configured settings (#72016) 2022-05-17 18:12:15 -05:00
J. Nick Koston
98809675ff Convert history queries to use lambda_stmt (#71870)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-15 10:47:29 -05:00
J. Nick Koston
a8f1dda004 Use ciso8601 for parsing MySQLdb datetimes (#71818)
* Use ciso8601 for parsing MySQLDB datetimes

The default parser is this:

5340191feb/MySQLdb/times.py (L66)

* tweak

* tweak

* add coverage for building the MySQLdb connect conv param
2022-05-13 19:26:09 -05:00
J. Nick Koston
222baa53dd Make database access in the eventloop raise an exception (#71547) 2022-05-09 15:22:08 -05:00
J. Nick Koston
29bda196b5 Break apart recorder into tasks and core modules (#71222) 2022-05-02 21:53:56 -07:00
J. Nick Koston
f073f17040 Refactor tracking of the recorder run history (#70456)
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-04-26 09:59:43 -10:00
Erik Montnemery
d045e8678d Use recorder test fixtures in tests (#70652) 2022-04-25 14:23:52 +02:00
Erik Montnemery
eb3458a3d2 Add MutexPool for recorder tests (#69410)
* Add MutexPool for recorder tests

* Fix get_schema_version

* Update test test_last_run_was_recently_clean

* Update test test_shutdown_before_startup_finishes

* Revert comments in test_write_lock_db

* Make the MutexPool lock a class variable

* Remove stale comment

* Move MutexPool

* Tweak debug prints
2022-04-12 07:41:46 -10:00
J. Nick Koston
72fffde77a Fix spelling of periodic in recorder (#69658)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-04-07 23:37:02 -10:00
Erik Montnemery
4a7c978f69 Speedup recorder test test_write_lock_db (#69578) 2022-04-07 15:36:58 +02:00
J. Nick Koston
bde3646293 Always use a commit interval of 0 for the in memory db in tests (#69330) 2022-04-05 12:06:06 -10:00
J. Nick Koston
ec131d685e Auto repack the database on the second sunday of the month (#69314) 2022-04-04 21:39:12 -10:00
J. Nick Koston
bc862e97ed Use a dedicated executor pool for database operations (#68105)
Co-authored-by: Erik Montnemery <erik@montnemery.com>
Co-authored-by: Franck Nijhof <git@frenck.dev>
2022-03-17 23:09:01 -10:00
Stefan Agner
0a9927d18e Avoid locking the database for non-SQLite backends (#63847)
* Avoid locking the database for non-SQLite backends

Currently we only have a lock implementation for SQLite. Just return
success for all other databases as they are not expected to store data
in the config directory and the caller can assume that a backup can
be safely taken.

This fixes `RuntimeError: generator didn't yield` errors when creating
a backup with the current Supervisor dev builds.
2022-01-11 16:17:56 +01:00
Stefan Agner
f0006b92be Allow to lock SQLite database during backup (#60874)
* Allow to set CONF_DB_URL

This is useful for test which need a custom DB path.

* Introduce write_lock_db helper to lock SQLite database

* Introduce Websocket API which allows to lock database during backup

* Fix isort

* Avoid mutable default arguments

* Address pylint issues

* Avoid holding executor thread

* Set unlock event in case timeout occures

This makes sure the database is left unlocked even in case of a race
condition.

* Add more unit tests

* Address new pylint errors

* Lower timeout to speedup tests

* Introduce queue overflow test

* Unlock database if necessary

This makes sure that the test runs through in case locking actually
succeeds (and the test fails).

* Make DB_LOCK_TIMEOUT a global

There is no good reason for this to be an argument. The recorder needs
to pick a sensible value.

* Add Websocket Timeout test

* Test lock_database() return

* Update homeassistant/components/recorder/__init__.py

Co-authored-by: Erik Montnemery <erik@montnemery.com>

* Fix format

Co-authored-by: J. Nick Koston <nick@koston.org>
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2021-12-07 13:16:24 +01:00
Erik Montnemery
be4e9f91b6 Change minimum supported SQLite version to 3.31.0 (#59073) 2021-11-04 16:34:35 +01:00
Erik Montnemery
dfa50a842a Simplify recorder PgSQL version checks (#58533) 2021-10-27 16:05:40 +02:00
Erik Montnemery
e9ba5f3b4b Warn when recorder connects to an unsupported database (#58161) 2021-10-26 13:41:59 +02:00
Erik Montnemery
8ef8838801 Correct detection of row_number support for MariaDB (#57663) 2021-10-14 11:19:39 -07:00
Erik Montnemery
0139bfa749 Detect if mysql and sqlite support row_number (#57475) 2021-10-11 21:17:18 -07:00
Erik Montnemery
5976f898da Add WS API for removing statistics for a list of statistic_ids (#55078)
* Add WS API for removing statistics for a list of statistic_ids

* Refactor according to code review, enable foreign keys support for sqlite

* Adjust tests

* Move clear_statistics WS API to recorder

* Adjust tests after rebase

* Update docstring

* Update homeassistant/components/recorder/websocket_api.py

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>

* Adjust tests after rebase

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-09-27 23:30:13 +02:00
Paulus Schoutsen
557cc792e9 Fix SQLAlchemy test warnings (#54116) 2021-08-08 20:33:47 -07:00
Erik Montnemery
e16a8063a5 Compile statistics for energy sensors (#50829)
* Compile statistics for energy sensors

* Update tests

* Rename abs_value to state

* Tweak

* Recreate statistics table

* Pylint

* Try to fix test

* Fix statistics for multiple energy sensors

* Fix energy statistics when last_reset is not set
2021-05-20 13:05:15 +02:00
J. Nick Koston
e7f7e61e88 Ensure a wal checkpoint is scheduled nightly (#50746) 2021-05-17 16:27:51 -05:00
J. Nick Koston
72288710ca Increase the sqlite cache size from ~2MiB to 8MiB (#50747) 2021-05-17 19:42:12 +02:00
J. Nick Koston
c10836fcee Upgrade to sqlalchemy 1.4.11 (#49538) 2021-04-21 20:29:36 -10:00
J. Nick Koston
6d137d2316 Increase recorder test coverage (#49362)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-04-19 08:22:38 -07:00
J. Nick Koston
9368891b1b Live db migrations and recovery (#49036)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-04-11 20:43:54 -10:00
Franck Nijhof
08870690a6 Fix a collection of tests with missing asserts (#48127)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-03-20 17:23:55 +01:00
J. Nick Koston
e2fd255a96 Cleanup recorder tests (#46836) 2021-02-21 09:52:41 +01:00
J. Nick Koston
9b69549f73 Recover and restart the recorder if the sqlite database encounters corruption while running (#46612) 2021-02-19 21:26:24 -10:00
J. Nick Koston
22dbac259b Ensure recorder shuts down cleanly on restart before startup is finished (#46604) 2021-02-19 22:18:21 -08:00
Franck Nijhof
65cf2fcb6f Drop asynctest (#44746) 2021-01-01 22:31:56 +01:00
J. Nick Koston
557684c3ce Add ability to disable the sqlite3 quick_check (#39479) 2020-09-02 10:12:56 +02:00
J. Nick Koston
45401d4308 Fix flapping recorder last run test (#39134) 2020-08-22 09:35:18 +02:00
J. Nick Koston
7c346c2f7c Skip the sqlite quick_check on clean restarts (#38972) 2020-08-21 14:20:46 +02:00
Bas Nijholt
0427d87ba4 Bump codespell from v1.16.0 to v1.17.1 and fix new spelling errors (#38663) 2020-08-08 14:41:02 +02:00
J. Nick Koston
1acdb28cdd Automatically recover when the sqlite3 database is malformed or corrupted (#37949)
* Validate sqlite database on startup and move away if corruption is detected.

* do not switch context in test -- its all sync
2020-07-17 19:07:37 -10:00
J. Nick Koston
edad387b12 Make recorder execute avoid native conversion by default (#36938) 2020-06-21 21:58:57 -07:00