Commit Graph

106 Commits

Author SHA1 Message Date
J. Nick Koston
c9abe76023 Use multiple indexed group-by queries to get start time states for MySQL (#138786)
* tweaks

* mysql

* mysql

* Update homeassistant/components/recorder/history/modern.py

* Update homeassistant/components/recorder/history/modern.py

* Update homeassistant/components/recorder/const.py

* Update homeassistant/components/recorder/statistics.py

* Apply suggestions from code review

* mysql

* mysql

* cover

* make sure db is fully init on old schema

* fixes

* fixes

* coverage

* coverage

* coverage

* s/slow_dependant_subquery/slow_dependent_subquery/g

* reword

* comment that callers are responsible for staying under the limit

* comment that callers are responsible for staying under the limit

* switch to kwargs

* reduce branching complexity

* split stats query

* preen

* split tests

* split tests
2025-03-02 15:13:06 +01:00
Marc Mueller
f822fd82bb Fix recorder fixture typing (#136174) 2025-01-21 18:18:05 -10:00
G Johansson
64752af4c2 Change minimum SQLite version to 3.40.1 (#135042)
Co-authored-by: J. Nick Koston <nick@koston.org>
2025-01-08 16:34:36 -10:00
J. Nick Koston
43fab48d4e Improve purge performance for PostgreSQL with large databases (#133699) 2024-12-21 10:53:15 +01:00
Erik Montnemery
d26d483a2f Improve recorder util resolve_period (#132264) 2024-12-06 21:06:56 +01:00
Erik Montnemery
deab285db8 Improve tests of recorder util resolve_period (#132259) 2024-12-04 14:01:49 +01:00
Pete
a417d3dcf8 Fix recorder "year" period in leap year (#132167)
* FIX: make "year" period work in leap year

* Add test

* Set second and microsecond to non-zero in test start times

* FIX: better fix for leap year problem

* Revert "FIX: better fix for leap year problem"

This reverts commit 06aba46ec6.

---------

Co-authored-by: Erik <erik@montnemery.com>
2024-12-04 13:21:10 +01:00
Erik Montnemery
d8b618f7c3 Remove support for live recorder data migration of context ids (#125309) 2024-10-26 07:19:03 +02:00
Erik Montnemery
161f37bb98 Add tests which directly test the recorder job wrappers (#125338) 2024-09-24 23:00:00 +02:00
epenet
3e967700fd Add missing return type in test __init__ method (part 2) (#123939)
* Add missing return type in test __init__ method (part 2)

* Adjust

* One more

* One more

* More
2024-08-14 17:59:15 +02:00
J. Nick Koston
a35fa0e95a Warn that the minimum SQLite version will change to 3.40.1 as of 2025.2 (#104298)
Co-authored-by: Robert Resch <robert@resch.dev>
2024-07-31 08:13:04 -05:00
Erik Montnemery
d55d02623a Add recorder test fixture to enable persistent SQLite database (#121137)
* Add recorder test fixture to enable persistent SQLite database

* Fix tests directly using async_test_home_assistant context manager
2024-07-04 09:59:37 +02:00
Erik Montnemery
408e524551 Add recorder test fixture for skipping tests by DB engine (#121118)
* Add recorder test fixture for skipping tests by DB engine

* Fix mistake
2024-07-04 00:02:46 +02:00
Erik Montnemery
5029da6919 Make the async_setup_recorder_instance fixture a context manager (#121086) 2024-07-03 14:05:34 -05:00
epenet
2c7022950c Fix import-outside-toplevel pylint warnings in tests (#119389) 2024-06-11 14:57:50 +02:00
epenet
1317837986 Improve type hints in tests (q-z) (#118381) 2024-05-30 08:48:02 +02:00
Jan Bouwhuis
c616fc036e Move recorder chunk utils to shared collection utils (#118065) 2024-05-24 12:49:39 -10:00
epenet
44f715bd02 Use issue_registry fixture in component tests (#118041) 2024-05-24 15:54:20 +02:00
J. Nick Koston
d06932bbc2 Refactor asyncio loop protection to improve performance (#117295) 2024-05-13 07:01:55 +09:00
Erik Montnemery
1cea22b8ba Fix search/replace mistake in recorder tests (#116933) 2024-05-06 09:03:21 -05:00
Erik Montnemery
5150557372 Convert recorder util tests to use async API (#116926) 2024-05-06 08:25:41 -05:00
Sid
895f73d8e4 Enable Ruff A001 (#115654) 2024-04-21 23:25:27 +02:00
Erik Montnemery
a3059fe504 Record state.last_reported (#114201)
* Record state.last_reported

* Include last_reported in parts of the history API

* Use a bulk update

* fix refactoring error

---------

Co-authored-by: J. Nick Koston <nick@koston.org>
2024-03-27 12:48:06 +01:00
Joost Lekkerkerker
6bb4e7d62c Bump ruff to 0.3.4 (#112690)
Co-authored-by: Sid <27780930+autinerd@users.noreply.github.com>
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
Co-authored-by: J. Nick Koston <nick@koston.org>
2024-03-26 00:02:16 +01:00
Sid
82a60fe8ad Enable Ruff RSE (#113695) 2024-03-18 00:40:38 +01:00
Marc Mueller
7d6251ca08 Add empty line after module docstring [tests q-s] (#112711) 2024-03-08 14:47:22 +01:00
Marc Mueller
2ef2172b01 Convert test helpers to get hass instance to contextmanagers (#109990)
* Convert get_test_home_assistant helper to contextmanager

* Convert async_test_home_assistant helper to contextmanager

* Move timezone reset to async_test_home_assistant helper
2024-02-11 21:23:51 +01:00
Marc Mueller
030727b078 Remove deprecated event_loop fixtures in tests (#109048) 2024-01-29 14:24:58 +01:00
Ville Skyttä
24b1e01d71 Update Ruff to 0.1.8, avoid linter/formatter conflicts (#106080)
* Disable Ruff rules that may conflict with the formatter

* Upgrade Ruff to 0.1.8

- https://github.com/astral-sh/ruff/releases/tag/v0.1.7
- https://github.com/astral-sh/ruff/releases/tag/v0.1.8

* Format with Ruff 0.1.8
2023-12-20 23:55:09 +01:00
J. Nick Koston
9ed745638d Chunk purging attributes and data ids for old SQLite versions (#104296) 2023-11-24 11:46:02 +01:00
Ville Skyttä
2399cd283a Python 3.10 support cleanups (#98640) 2023-08-21 19:14:07 +02:00
J. Nick Koston
95fcdc5684 Speed up sql queries where ORM rows are not needed (#91839)
* Speed up logbook and history queries where ORM rows are not needed

This avoids having sqlalchemy wrap Result in ChunkedIteratorResult
which has additional overhead we do not need for these cases

* more places

* anything that uses _sorted_statistics_to_dict does not need orm rows either
2023-04-21 23:28:07 -04:00
J. Nick Koston
4e6937d20f Avoid multiple round trips to the database for history API calls (#91193)
* delete more code

* tweak

* tweak

* wrappers

* restore lost performance

* restore lost performance

* restore lost performance

* compact

* reduce

* fix refactor

* DRY

* tweak

* delete the start time state injector

* move away the legacy code

* tweak

* adjust

* adjust

* tweak

* ignore impossible

* fix a bug where the first start was changed to the start time when there was no previous history recorded before

* avoid the empty scan most cases

* postgresql

* fixes

* workaround for mariadb < 10.4

* remove unused

* remove unused

* adjust

* bail early

* tweak

* tweak

* fix more tests

* fix recorderrun being init in the future in the test

* run history tests on schema 30 as well

* Revert "run history tests on schema 30 as well"

This reverts commit d798b100ac.

* reduce

* cleanup

* tweak

* reduce

* prune

* adjust

* adjust

* adjust

* reverse later is faster because the index is in forward order and the data size we are reversing is much smaller even if we are in python code

* Revert "reverse later is faster because the index is in forward order and the data size we are reversing is much smaller even if we are in python code"

This reverts commit bf974e103e.

* fix test

* Revert "Revert "reverse later is faster because the index is in forward order and the data size we are reversing is much smaller even if we are in python code""

This reverts commit 119354499e.

* more coverage

* adjust

* fix for table order

* impossible for it to be missing

* remove some more legacy from the all states
2023-04-11 22:38:23 -04:00
epenet
2f7c5a56eb Use tmp_path in recorder tests (#91202) 2023-04-11 09:18:16 +02:00
J. Nick Koston
99b58f157e Bump PyJWT to 2.6.0 (#90134)
* Bump PyJWT to 2.6.0

* fix time being frozen too late which makes the access token creation time in the future

* revert zha change

* fix repairs test

* fix ical test
2023-03-22 20:00:47 -04:00
epenet
ab4a726e6c Add tmpdir to known fixtures in pylint (#89844) 2023-03-17 10:22:02 +01:00
J. Nick Koston
a244749712 Make StatesMetaManager thread-safe when an entity_id is fully deleted from the database and than re-added (#89732)
* refactor to make StatesMetaManager threadsafe

* refactor to make StatesMetaManager threadsafe

* refactor to make StatesMetaManager threadsafe

* refactor to make StatesMetaManager threadsafe

* reduce

* comments
2023-03-15 08:54:02 -04:00
J. Nick Koston
c41f91be89 Deduplicate entity_id in the states table (#89557) 2023-03-12 10:01:58 -10:00
J. Nick Koston
50c31a5355 Move legacy database queries and models to prepare for schema v38 (#89532) 2023-03-11 11:26:30 -10:00
J. Nick Koston
3533e77ec9 Fix using MariaDB slow range select workaround with Postgresql (#88459)
* Mark PostgreSQL range select as fast

Currently we were using the slow range select workaround for
PostgreSQL that was original developed for MariaDB but
its actually slower on PostgreSQ

fixes #83253

* Mark PostgreSQL range select as fast

Currently we were using the slow range select workaround for
PostgreSQL that was original developed for MariaDB but
its actually slower on PostgreSQ

fixes #83253
2023-02-19 21:08:43 -05:00
epenet
185cd61cbd Add type hints to integration tests (recorder) (#88313) 2023-02-17 18:50:09 +01:00
J. Nick Koston
b9f29c27ab Cleanup stale recorder code (#88275)
- Removes dead commit function
- Small cleanup to debug handling in execute
2023-02-16 23:12:11 -05:00
Franck Nijhof
ed79265843 Enable Ruff PT006 (#88165)
* Enable Ruff PT006

* Adjust existing cases

* Fix tests

* Remove unneeded parentheses
2023-02-15 14:09:50 +01:00
J. Nick Koston
db503c0daa Always use UTC time for the MariaDB/MySQL session (#87583) 2023-02-09 11:39:45 -06:00
epenet
ba85fdcd61 Add type hints to integration tests (q-s) (#87706) 2023-02-08 08:12:54 -08:00
Erik Montnemery
720f51657d Add CI job which runs recorder tests on PostgreSQL (#80614)
Co-authored-by: Franck Nijhof <git@frenck.dev>
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-07 09:46:14 -06:00
epenet
4142f0d15d Add return type to tests without arguments (#87613)
* Add return type to tests without arguments

* Black

* Cancel fixture amends
2023-02-07 14:20:06 +01:00
J. Nick Koston
c11f3ffa17 Speed up purge time with newer MariaDB versions (#87409)
* Speed up purge time with newer MariaDB versions

* fix

* document

* document

* document

* rename

* self review

* Update homeassistant/components/recorder/util.py

* fixes
2023-02-05 17:58:34 -05:00
J. Nick Koston
5284837c8f Add a repair issue when using MariaDB is affected by MDEV-25020 (#87040)
closes https://github.com/home-assistant/core/issues/83787
2023-01-31 20:42:07 +01:00
Franck Nijhof
a79885ceaf Enable Ruff SIM117 (#86783) 2023-01-27 11:52:49 +01:00