822 Commits

Author SHA1 Message Date
Erik Montnemery
7e7cb15d5b
Revert "Allows defining list of attributes excluded from history in manifest.json" (#99300)
Revert "Allows defining list of attributes excluded from history in manifest.json (#99283)"

This reverts commit 0366e14630676b2d96f529c9113f5fceb86c705b.
2023-08-30 08:26:26 +02:00
Erik Montnemery
0366e14630
Allows defining list of attributes excluded from history in manifest.json (#99283)
* Move list of attributes excluded from history to manifest.json

* Address comments
2023-08-29 20:14:33 +02:00
J. Nick Koston
6992ea9af0
Bump fnv-hash-fast to 0.4.1 (#99135) 2023-08-27 10:08:21 -05:00
Ville Skyttä
c287bd1a3b
Remove pylint configs flagged by useless-suppression (#99081) 2023-08-26 17:46:03 +03:00
Ville Skyttä
34b47a2597
Remove unnnecessary pylint configs from components [m-r]* (#98924) 2023-08-24 00:56:50 +02:00
Ville Skyttä
3094991236
Upgrade ruff to 0.0.285 (#98647) 2023-08-19 08:17:17 -04:00
Marc Mueller
a9ade1f84d
Use asyncio.timeout [core] (#98447) 2023-08-15 08:36:05 -05:00
J. Nick Koston
5cc72814c9
Bump fnv-hash-fast to 0.4.0 (#97160) 2023-07-24 20:34:46 +02:00
Franck Nijhof
30058297cf
Migrate backported StrEnum to built-in StrEnum (#97101) 2023-07-23 23:19:24 +02:00
Franck Nijhof
86708b5590
Update ruff to v0.0.280 (#97102) 2023-07-23 22:00:26 +02:00
Marc Mueller
77f38e33e5
Import names from typing instead of typing_extensions [3.11] (#97065) 2023-07-22 17:03:44 -05:00
Paulus Schoutsen
7859be6481
Add deduplicate translations script (#96384)
* Add deduplicate script

* Fix forecast_solar incorrect key with space

* Fix utf-8

* Do not create references to other arbitrary other integrations

* Add commented code to only allow applying to referencing integrations

* Tweak

* Bug fix

* Add command line arg for limit reference

* never suggest to update common keys

* Output of script

* Apply suggestions from code review

Co-authored-by: Michael <35783820+mib1185@users.noreply.github.com>

---------

Co-authored-by: Michael <35783820+mib1185@users.noreply.github.com>
2023-07-13 11:52:50 -04:00
Joost Lekkerkerker
b367c95c81
Add more common translations (#96429)
* Add common translations

* Add common translations

* Add common translations

* Add common translations

* Add common translations

* Add common translations

* Add common translations

* Add common translations
2023-07-12 22:00:05 -04:00
Franck Nijhof
d3eda12af4
Migrate recorder services to support translations (#96409)
Co-authored-by: c0ffeeca7 <38767475+c0ffeeca7@users.noreply.github.com>
2023-07-12 17:28:05 +02:00
J. Nick Koston
b2e708834f
Add slots to the StateMachine class (#95849) 2023-07-05 14:00:37 +02:00
Franck Nijhof
3e85a29b86
Move overlapping pylint rules to ruff, disable mypy overlap (#94359) 2023-06-27 17:42:46 +02:00
Erik Montnemery
863b948e7c
Correct calls to super class in RecorderPool (#94923) 2023-06-20 23:26:37 +02:00
Franck Nijhof
c3936e6f14
Replace isort with ruff (#94302) 2023-06-08 23:43:56 +02:00
Ville Skyttä
ca936d0b38
Replace bandit with ruff (#93200) 2023-06-08 22:46:04 +02:00
J. Nick Koston
7995d3777a
Fix package names to match pypi index metadata (#93883)
* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* Fix package names to match pypi index metadata

* uses _

* uses -

* fix metadata
2023-05-31 21:12:53 -04:00
J. Nick Koston
493d78f070
Avoid dict comp in statistic_during_period if there is no conversion (#93699) 2023-05-29 20:59:15 -04:00
J. Nick Koston
2f1f32f0bb
Make unit converter use a factory to avoid looking up the ratios each conversion (#93706) 2023-05-29 20:50:40 +02:00
J. Nick Koston
560e744f1e
Reduce overhead to filter entities in the recorder (#93656)
* Reduce overhead to filter entities in the recorder

* fix type
2023-05-27 19:52:42 -04:00
J. Nick Koston
f251c464e2
Fix history start time state returning None with some postgresql versions (#93555)
* Add additional coverage to history websocket api

related issue #93258

* Add additional coverage to history websocket api

related issue #93258

* Fix results when union query ends up at the end instead of front

* Apply suggestions from code review

* resort

* zero instead

* fix exception

* fix tests
2023-05-25 22:18:19 -04:00
J. Nick Koston
d9b43fc43f
Significantly speed up recorder event listener (#93532)
* Significantly speed up recorder event listener

This code is called every time an event happens since it
subscribes to all events. Its our most frequently called
listener out of the box.

It used to have a seperate filter function but it was
later combined after core had some previous refactoring.

It was never optimized after that happened.

This change reduces the run time by ~70%

* decruft
2023-05-25 21:44:12 -04:00
J. Nick Koston
f09abb0f2e
Fix race in tracking pending writes in recorder (#93414) 2023-05-23 21:47:31 +02:00
J. Nick Koston
cbee514c2a
Bump sqlalchemy to 2.0.15 (#93293) 2023-05-20 14:26:33 -05:00
J. Nick Koston
809f394ff5
Bump sqlalchemy to 2.0.13 (#92936) 2023-05-12 04:37:16 +09:00
Erik Montnemery
c1f716487c
Include data for entire periods in response to recorder/statistics_during_period (#92848)
* Include data for entire periods in response to recorder/statistics_during_period

* Break out function for calculating the end of the month
2023-05-11 11:28:33 +02:00
Erik Montnemery
4568207f9b
Support calculating changes between consecutive sum statistics (#92823)
* Support calculating changes between consecutive sum statistics

* Add support for unit conversion when calculating change

* Don't include sum in WS response unless requested

* Improve tests

* Break out calculating change to its own function

* Improve test coverage
2023-05-11 10:05:58 +02:00
J. Nick Koston
ede1f08c51
Reduce size of migration transactions to accommodate slow/busy systems (#92312)
* Reduce size of migration transactions to accommodate slow/busy systems

related issue #91489

* handle overloaded RPIs better
2023-04-30 23:18:00 -04:00
J. Nick Koston
32f6dd4a82
Bump sqlalchemy to 2.0.12 (#92315)
changelog: https://docs.sqlalchemy.org/en/20/changelog/changelog_20.html#change-2.0.12
2023-04-30 23:11:29 -04:00
J. Nick Koston
bb4c03ce3c
Avoid returning statistics columns that the metadata knows are impossible (#92095) 2023-04-30 20:46:26 +02:00
J. Nick Koston
1a82b353e0
Auto repair incorrect collation on MySQL schema (#92270)
* Auto repair incorrect collation on MySQL schema

As we do more union queries in 2023.5.x if there is a mismatch
between collations on tables, they will fail with an error
that is hard for the user to figure out how to fix

`Error executing query: (MySQLdb.OperationalError) (1271, "Illegal mix of collations for operation UNION")`

This was reported in the #beta channel and by PM from others
so the problem is not isolated to a single user

https://discord.com/channels/330944238910963714/427516175237382144/1100908739910963272

* test with ascii since older maraidb versions may not work otherwise

* Revert "test with ascii since older maraidb versions may not work otherwise"

This reverts commit 787fda1aefcd8418a28a8a8f430e7e7232218ef8.t

* older version need to check collation_server because the collation is not reflected if its the default
2023-04-29 21:17:09 -04:00
J. Nick Koston
e136070718
Speed up purging the database (#92247) 2023-04-29 18:45:14 +02:00
J. Nick Koston
e156d3132e
Ensure purge can cleanup old format detached states in the database (#92145) 2023-04-28 21:03:59 +02:00
J. Nick Koston
47b289b632
Bump sqlalchemy to 2.0.11 to fix a critical regression with postgresql (#92126) 2023-04-27 18:52:43 +02:00
J. Nick Koston
e7e7990b60
Bump sqlalchemy to 2.0.10 (#91982) 2023-04-25 09:45:49 -05:00
J. Nick Koston
0b0c94ee52
Fix migration with negative event type cache (#91910)
* Fix migration with negative event type cache

fixes a regression with #91770

* Update homeassistant/components/recorder/table_managers/event_types.py
2023-04-23 20:56:17 -04:00
Marc Mueller
06520217bc
Update pylint to 2.17.2 (#91894) 2023-04-23 14:46:48 +02:00
J. Nick Koston
34b824a27b
Reduce number of columns when selecting attributes for history (#91717) 2023-04-22 14:21:08 +02:00
J. Nick Koston
6e628d2f06
Add a negative cache to the recorder event type manager (#91770) 2023-04-22 14:09:04 +02:00
J. Nick Koston
b5fbbf8410
Speed up processing binary columns on SQLite/MariaDB/MySQL (#91818) 2023-04-22 13:55:11 +02:00
J. Nick Koston
95fcdc5684
Speed up sql queries where ORM rows are not needed (#91839)
* Speed up logbook and history queries where ORM rows are not needed

This avoids having sqlalchemy wrap Result in ChunkedIteratorResult
which has additional overhead we do not need for these cases

* more places

* anything that uses _sorted_statistics_to_dict does not need orm rows either
2023-04-21 23:28:07 -04:00
J. Nick Koston
adc472862b
Fallback to generating a new ULID on migraiton if context is missing or invalid (#91704)
* Fallback to generating a new ULID on migraiton if context is missing or invalid

It was discovered that postgresql will do a full scan if
there is a low cardinality on the index because of missing
context ids. We will now generate a ULID for the timestamp
of the row if the context data is missing or invalid

fixes #91514

* tests

* tweak

* tweak

* preen
2023-04-19 21:56:07 -04:00
J. Nick Koston
408b2171ae
Handle long format context UUIDs during migration (#91657)
In https://github.com/home-assistant/core/issues/91514 is was discovered
these exist in older versions
2023-04-19 08:40:04 -04:00
J. Nick Koston
2ec1359063
Dynamically size recorder max backlog based on available memory (#90894)
Co-authored-by: Franck Nijhof <git@frenck.dev>
2023-04-18 12:35:49 +02:00
J. Nick Koston
2530031454
Use cache to lookup event type ids in logbook (#91576)
noticed we can speed this up while looking at
https://github.com/home-assistant/core/issues/91514

Note: this will not fix that issue as there is more going on there
2023-04-17 23:40:03 -04:00
J. Nick Koston
fdc6cf3472
Continue recording events until final write (#91260) 2023-04-16 09:22:47 -10:00
J. Nick Koston
1379ad60c8
Ensure recorder always attempts clean shutdown if recorder thread raises (#91261)
* Ensure recorder run shutdown if the run loop raises

If anything goes wrong with the recorder we should
still try to shutdown cleanly

* tweak

* tests

* tests

* handle migraiton failure

* tweak comment

* naming

* order

* order

* order

* reword

* adjust test

* fixes

* threading

* failure case

* fix test

* have to wait for stop because the task blocks on thread join
2023-04-14 21:03:24 -04:00