* Add circular mean statistics
* fixes
* Add has_circular_mean and fix tests
* Fix mypy
* Rename to MEASUREMENT_ANGLE
* Fix kitchen_sink tests
* Fix sensor tests
* for testing only
* Revert ws command change
* Apply suggestions
* test only
* add custom handling for postgres
* fix recursion limit
* Check if column is already available
* Set default false and not nullable for has_circular_mean
* Proper fix to be backwards compatible
* Fix value is None
* Align with schema
* Remove has_circular_mean from test schemas as it's not required anymore
* fix wrong column type
* Use correct variable to reduce stats
* Add guard that the uom is matching a valid one from the state class
* Add some tests
* Fix tests again
* Use mean_type in StatisticsMetato difference between different mean type algorithms
* Fix leftovers
* Fix kitchen_sink tests
* Fix postgres
* Add circular mean test
* Add mean_type_changed stats issue
* Align the attributes with unit_changed
* Fix mean_type_change stats issue
* Add missing sensor recorder tests
* Add test_statistic_during_period_circular_mean
* Add mean_weight
* Add test_statistic_during_period_hole_circular_mean
* Use seperate migration step to null has_mean
* Typo ARITHMETIC
* Implement requested changes
* Implement requested changes
* Split into #141444
* Add StatisticMeanType.NONE and forbid that mean_type can be None
* Fix mean_type
* Implement requested changes
* Small leftover of latest StatisticMeanType changes
* tweaks
* mysql
* mysql
* Update homeassistant/components/recorder/history/modern.py
* Update homeassistant/components/recorder/history/modern.py
* Update homeassistant/components/recorder/const.py
* Update homeassistant/components/recorder/statistics.py
* Apply suggestions from code review
* mysql
* mysql
* cover
* make sure db is fully init on old schema
* fixes
* fixes
* coverage
* coverage
* coverage
* s/slow_dependant_subquery/slow_dependent_subquery/g
* reword
* comment that callers are responsible for staying under the limit
* comment that callers are responsible for staying under the limit
* switch to kwargs
* reduce branching complexity
* split stats query
* preen
* split tests
* split tests
* Refactor recorder data migration
* Fix stale docstrings
* Don't store a session object in BaseRunTimeMigration instances
* Simplify logic in EntityIDMigration.migration_done
* Fix tests
* Record state.last_reported
* Include last_reported in parts of the history API
* Use a bulk update
* fix refactoring error
---------
Co-authored-by: J. Nick Koston <nick@koston.org>
Co-authored-by: Sid <27780930+autinerd@users.noreply.github.com>
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
Co-authored-by: J. Nick Koston <nick@koston.org>
* Handle statistics columns being unmigrated from previous downgrades
If the user downgraded HA from 2023.3.x to an older version without
restoring the database and they upgrade again with the same database
they will have unmigrated statistics columns since we only migrate them
once.
As its expensive to check, we do not want to check every time
at startup, so we will only do this one more time since the
risk that someone will downgrade to an older version is very
low at this point.
* add guard to sqlite to prevent re-migrate
* test
* move test to insert with old schema
* use helper
* normalize timestamps
* remove
* add check
* add fallback migration
* add fallback migration
* commit
* remove useless logging
* remove useless logging
* do the other columns at the same time
* coverage
* dry
* comment
* Update tests/components/recorder/test_migration_from_schema_32.py
* Reduce overhead of legacy database columns on new installs
* Reduce overhead of legacy database columns on new installs
* Reduce overhead of legacy database columns on new installs
* Reduce overhead of legacy database columns on new installs
* not working as expected
* override the type compiler
* override the type compiler
* override the type compiler
* override the type compiler
* Apply suggestions from code review
* pgsql char1
* make entity filter test setup with old schema
* fix some more tests that were mutating state
* fix some more tests that were mutating state
* fix some more tests that were mutating state
* fix more dbstate mutations
* add shim for older tests
* split migration tests
* add coverage for purging legacy data
* tweak
* more fixes
* drop some legacy
* fix another test
* fix a few more
* add casts for postgresql in case someone deletes the schema changes table
* dry
* dry
* dry
* Avoid database executor job to fetch statistic metadata on cache hit
Since we will almost always have a cache hit fetching
statistic meta data we can avoid an executor job
* Avoid database executor job to fetch statistic metadata on cache hit
Since we will almost always have a cache hit fetching
statistic meta data we can avoid an executor job
* Avoid database executor job to fetch statistic metadata on cache hit
Since we will almost always have a cache hit fetching
statistic meta data we can avoid an executor job
* remove exception catch since the threading.excepthook will actually catch this in production
* fix a few missed ones
* threadsafe
* Update homeassistant/components/recorder/table_managers/statistics_meta.py
* coverage and optimistic caching
* Speed up comparing State and Event objects
Use default python implementation for State and Event __hash__ and __eq__
The default implementation compared based on the id() of the object
which is effectively what we want here anyways. These overrides are
left over from the days when these used to be attrs objects
By avoiding implementing these ourselves all of the equality checks
can happen in native code
* tweak
* adjust tests
* write out some more
* fix test to not compare objects
* more test fixes
* more test fixes
* correct stats tests
* fix more tests
* fix more tests
* update sensor recorder tests
* Fire events when long term statistics is updated
* Allow the new events to be subscribed to by anyone
* Address review comments
* Finish renaming events
* Finish renaming events
* Fix do_adhoc_statistics
* Adjust tests
* Adjust tests
* Separate recorder database schema from other classes
* fix logbook imports
* migrate new tests
* few more
* last one
* fix merge
Co-authored-by: J. Nick Koston <nick@koston.org>
* Add a commit interval setting to recorder
* Make the default every 1s instead of immediate
* See attached py-spy flamegraphs for why 1s
* This avoids disk thrashing during event storms
* Make Home Assistant significantly more responsive on busy systems
* remove debug
* Add commit forces for tests that expect commits to be immediate
* Add commit forces for tests that expect commits to be immediate
* make sure _trigger_db_commit is in the right place (all effective "wait_recording_done" calls)
* De-duplicate wait_recording_done code
