110 Commits

Author SHA1 Message Date
J. Nick Koston
c9abe76023
Use multiple indexed group-by queries to get start time states for MySQL (#138786)
* tweaks

* mysql

* mysql

* Update homeassistant/components/recorder/history/modern.py

* Update homeassistant/components/recorder/history/modern.py

* Update homeassistant/components/recorder/const.py

* Update homeassistant/components/recorder/statistics.py

* Apply suggestions from code review

* mysql

* mysql

* cover

* make sure db is fully init on old schema

* fixes

* fixes

* coverage

* coverage

* coverage

* s/slow_dependant_subquery/slow_dependent_subquery/g

* reword

* comment that callers are responsible for staying under the limit

* comment that callers are responsible for staying under the limit

* switch to kwargs

* reduce branching complexity

* split stats query

* preen

* split tests

* split tests
2025-03-02 15:13:06 +01:00
epenet
0e263aa427
Standardize homeassistant imports in full-CI tests (#136735) 2025-01-28 13:10:20 -05:00
Marc Mueller
f822fd82bb
Fix recorder fixture typing (#136174) 2025-01-21 18:18:05 -10:00
Erik Montnemery
bde92b34dd
Remove recorder history queries for database schemas < 31 (#125652) 2024-09-10 19:26:19 +02:00
epenet
c76d68503a
Add missing hass type hint in history and recorder tests (#124204) 2024-08-19 15:41:09 +02:00
J. Nick Koston
1879db9f8f
Revert to using call_soon for event triggers and state changed event trackers (#122735) 2024-07-29 11:45:39 +02:00
Erik Montnemery
5029da6919
Make the async_setup_recorder_instance fixture a context manager (#121086) 2024-07-03 14:05:34 -05:00
J. Nick Koston
f9205cd88d
Avoid additional timestamp conversion to set state (#118885)
Avoid addtional timestamp conversion to set state

Since we already have the timestamp, we can pass it on to the State
object and avoid the additional timestamp conversion which can be as
much as 30% of the state write runtime.

Since datetime objects are limited to microsecond precision, we need
to adjust some tests to account for the additional precision that we
will now be able to get in the database
2024-06-05 23:43:34 -04:00
J. Nick Koston
5a609c34bb
Fix blocking I/O in the event loop when loading timezones (#117721) 2024-05-20 11:06:03 +02:00
Erik Montnemery
27fcf72275
Convert history tests to use async API (#116447)
* Convert history tests to use async API

* Add new fixture to help patch recorder

* Modify

* Modify

* Update tests/conftest.py

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>

* Rename fixture

---------

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2024-05-03 08:14:46 +02:00
Sid
d5f883fbf0
Unignore Ruff PLR in tests (#114470)
* Unignore Ruff PLR in tests

* Address review comments

* review comments

* fix import

* Update test_api.py

* Update test_api.py

* Update test_api.py
2024-04-01 11:11:59 +02:00
Sid
aec7a67a58
Unignore Ruff PLE, PLW in tests (#114406)
* Unignore Ruff PLE, PLW in tests

* fix tests
2024-03-29 23:58:30 +01:00
Sid
00ec7f11f0
Enable Ruff rule PT007 (#113764)
Co-authored-by: Franck Nijhof <frenck@frenck.nl>
2024-03-19 09:01:07 +01:00
Sid
6113b99ddd
Replace pylint pointless-statement with ruff B018 (#113582)
* Replace pylint pointless-statement with ruff B018

* fix occurrences of B018

* disable pylint expression-not-assigned as well

---------

Co-authored-by: J. Nick Koston <nick@koston.org>
2024-03-17 10:58:14 +01:00
Joost Lekkerkerker
761933acfe
Improve lists in integrations [G-H] (#113168) 2024-03-13 17:25:27 +01:00
Marc Mueller
32f3f46542
Add empty line after module docstring (2) [tests.components] (#112737) 2024-03-08 19:16:21 +01:00
Marc Mueller
c88b337600
Add empty line after module docstring [tests f-k] (#112709) 2024-03-08 14:50:04 +01:00
Ville Skyttä
e7b6037419
Remove unnnecessary pylint configs from components [e-l]* (#99004) 2023-08-25 09:46:34 +02:00
Marc Mueller
a9ade1f84d
Use asyncio.timeout [core] (#98447) 2023-08-15 08:36:05 -05:00
J. Nick Koston
f251c464e2
Fix history start time state returning None with some postgresql versions (#93555)
* Add additional coverage to history websocket api

related issue #93258

* Add additional coverage to history websocket api

related issue #93258

* Fix results when union query ends up at the end instead of front

* Apply suggestions from code review

* resort

* zero instead

* fix exception

* fix tests
2023-05-25 22:18:19 -04:00
J. Nick Koston
40a47c72f4
Update history tests to avoid patching utcnow (#93487) 2023-05-24 16:09:26 -05:00
Franck Nijhof
acc4b001cd
Fix history YAML deprecation (#92238) 2023-04-29 08:02:34 -04:00
Harvey
bf4559719a
History API entity_id validation (#90067)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-04-14 09:41:54 -10:00
J. Nick Koston
4e6937d20f
Avoid multiple round trips to the database for history API calls (#91193)
* delete more code

* tweak

* tweak

* wrappers

* restore lost performance

* restore lost performance

* restore lost performance

* compact

* reduce

* fix refactor

* DRY

* tweak

* delete the start time state injector

* move away the legacy code

* tweak

* adjust

* adjust

* tweak

* ignore impossible

* fix a bug where the first start was changed to the start time when there was no previous history recorded before

* avoid the empty scan most cases

* postgresql

* fixes

* workaround for mariadb < 10.4

* remove unused

* remove unused

* adjust

* bail early

* tweak

* tweak

* fix more tests

* fix recorderrun being init in the future in the test

* run history tests on schema 30 as well

* Revert "run history tests on schema 30 as well"

This reverts commit d798b100ac45c7f8c4cee5d284d94beed5e3d454.

* reduce

* cleanup

* tweak

* reduce

* prune

* adjust

* adjust

* adjust

* reverse later is faster because the index is in forward order and the data size we are reversing is much smaller even if we are in python code

* Revert "reverse later is faster because the index is in forward order and the data size we are reversing is much smaller even if we are in python code"

This reverts commit bf974e103e651a1334493a9594e08d19e51e392b.

* fix test

* Revert "Revert "reverse later is faster because the index is in forward order and the data size we are reversing is much smaller even if we are in python code""

This reverts commit 119354499ecf7c1025ec40350e97e73d62d3fd4b.

* more coverage

* adjust

* fix for table order

* impossible for it to be missing

* remove some more legacy from the all states
2023-04-11 22:38:23 -04:00
J. Nick Koston
d0d4ab6056
Require a list of entity ids when fetching history (#90992) 2023-04-08 16:14:44 -10:00
J. Nick Koston
0550b17d54
Rework recorder filters to avoid caching mistakes (#90419) 2023-03-28 22:51:46 +02:00
J. Nick Koston
c41f91be89
Deduplicate entity_id in the states table (#89557) 2023-03-12 10:01:58 -10:00
J. Nick Koston
8bd43760b6
Deduplicate event_types in the events table (#89465)
* Deduplicate event_types in the events table

* Deduplicate event_types in the events table

* more fixes

* adjust

* adjust

* fix product

* fix tests

* adjust

* migrate

* migrate

* migrate

* more test fixes

* more test fixes

* fix

* migration test

* adjust

* speed up

* fix index

* fix more tests

* handle db failure

* preload

* tweak

* adjust

* fix stale docs strings, remove dead code

* refactor

* fix slow tests

* coverage

* self join to resolve query performance

* fix typo

* no need for quiet

* no need to drop index already dropped

* remove index that will never be used

* drop index sooner as we no longer use it

* Revert "remove index that will never be used"

This reverts commit 461aad2c52d7d6c2c6ca1df4cb30b77c52027d46.

* typo
2023-03-11 14:54:55 -05:00
Erik Montnemery
4ca0a24f87
Increase test scope of MariaDB + PostgreSQL tests (#87019)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-13 15:58:13 -06:00
epenet
03710e58b5
Add type hints to integration tests (part 9) (#87983) 2023-02-13 12:06:51 +01:00
Erik Montnemery
94519de8dd
Upgrade SQLAlchemy to 2.0.2 (#86436)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-08 08:17:32 -06:00
epenet
4142f0d15d
Add return type to tests without arguments (#87613)
* Add return type to tests without arguments

* Black

* Cancel fixture amends
2023-02-07 14:20:06 +01:00
J. Nick Koston
c612a92cfb
Use python defaults for comparing State, LazyState, and Event objects (#86856)
* Speed up comparing State and Event objects

Use default python implementation for State and Event __hash__ and __eq__

The default implementation compared based on the id() of the object
which is effectively what we want here anyways. These overrides are
left over from the days when these used to be attrs objects

By avoiding implementing these ourselves all of the equality checks
can happen in native code

* tweak

* adjust tests

* write out some more

* fix test to not compare objects

* more test fixes

* more test fixes

* correct stats tests

* fix more tests

* fix more tests

* update sensor recorder tests
2023-01-29 13:31:43 -05:00
Franck Nijhof
ef800335fb
Enable Ruff PT022 (#86792)
* Enable Ruff PT022

* Adjust found cases
2023-01-27 13:57:06 +01:00
J. Nick Koston
0b0e977ce9
Remove history use_include_order configuration option (#86365) 2023-01-22 17:43:05 +01:00
epenet
30bf0634fe
Add per-file-ignore to pylint (#86289) 2023-01-22 17:26:24 +01:00
J. Nick Koston
5ff0479c16
Split history websocket API into its own file (#86364) 2023-01-21 18:45:00 -10:00
J. Nick Koston
8227c84e05
Add support for streaming (push) history (#85892)
* Add support for streaming (push) history

Currently we poll for history data, like logbook we can stream
this data to avoid database overhead

* Update homeassistant/components/history/__init__.py

* merge filter

* expose new api

* expose new api

* expose new api

* expose new api

* coverage

* tests

* fixes

* tweak

* tweak

* tweak

* DRY

* leaky

* test for specific entities

* test for specific entities

* test for specific entities

* test for specific entities

* test for specific entities

* cover

* cover

* more cover

* tweak

* make sure it works before history starts

* fix test

* cover

* tweak

* make sure we unsub on overflow

* Update homeassistant/components/history/__init__.py

* Update homeassistant/components/history/__init__.py

* fix race in test

* fix db executor access

* relo above task creation
2023-01-21 22:03:41 -05:00
J. Nick Koston
b8a1537b58
Improve performance of fetching and storing history and events with the database (#84870) 2023-01-02 13:26:08 -10:00
Aarni Koskela
e4c73259f7
Configure autoflake for pre-commit (#82414) 2022-11-22 08:10:36 +01:00
Erik Montnemery
1582d88957
Remove deprecated history WS API (#82136) 2022-11-16 12:54:03 +01:00
Erik Montnemery
31a787558f
Ensure recorder test fixture is setup before hass fixture (#80528)
* Ensure recorder test fixture is setup before hass fixture

* Adjust more tests
2022-10-19 07:58:47 +02:00
Erik Montnemery
7937bfeedb
Deprecate history integration's statistics API (#78056) 2022-09-08 16:03:43 -04:00
Erik Montnemery
008ac8d10d
Improve statistics metadata WS API (#77209) 2022-08-31 11:30:45 +02:00
J. Nick Koston
a8349a4866
Adjust entity filters to make includes stronger than excludes (#74080)
* Adjust entity filters to make includes stronger than excludes

Fixes #59080

* adjust test for stronger entity glob includes

* sync with docs
2022-06-28 11:42:51 -05:00
Erik Montnemery
b145aeaf75
Fix flaky recorder test (#73733) 2022-06-20 12:27:25 +02:00
J. Nick Koston
8e8fa0399e
Fix statistics_during_period being incorrectly cached (#72947) 2022-06-03 13:04:46 -07:00
J. Nick Koston
5b31414225
Fix misalignments between sql based filtering with the entityfilter based filtering (#72936) 2022-06-02 20:52:53 -07:00
J. Nick Koston
a43d47fa0b
Escape % and _ in history/logbook entity_globs, and use ? as _ (#72623)
Co-authored-by: pyos <pyos100500@gmail.com>
2022-05-27 14:38:29 -07:00
J. Nick Koston
1001f9e39a
Fix last state in history minimal respones when all the states at the end are skipped (#72203) 2022-05-19 20:58:32 -07:00