From 5b6c51c1548d9f5647c0427396a05d05d76c6788 Mon Sep 17 00:00:00 2001 From: Fabian Affolter Date: Wed, 31 Jan 2018 09:43:05 +0100 Subject: [PATCH] Move all scripts to one location as one page per script (#4554) --- source/_components/influxdb.markdown | 116 +------------------ source/_docs/configuration/secrets.markdown | 58 +--------- source/_docs/tools/benchmark.markdown | 20 ++++ source/_docs/tools/check_config.markdown | 18 +++ source/_docs/tools/credstash.markdown | 33 ++++++ source/_docs/tools/db_migrator.markdown | 72 ++++++++++++ source/_docs/tools/ensure_config.markdown | 18 +++ source/_docs/tools/influxdb_import.markdown | 64 ++++++++++ source/_docs/tools/keyring.markdown | 42 +++++++ source/_docs/tools/scripts.markdown | 42 +------ source/_includes/asides/docs_navigation.html | 8 +- 11 files changed, 282 insertions(+), 209 deletions(-) create mode 100644 source/_docs/tools/benchmark.markdown create mode 100644 source/_docs/tools/check_config.markdown create mode 100644 source/_docs/tools/credstash.markdown create mode 100644 source/_docs/tools/db_migrator.markdown create mode 100644 source/_docs/tools/ensure_config.markdown create mode 100644 source/_docs/tools/influxdb_import.markdown create mode 100644 source/_docs/tools/keyring.markdown diff --git a/source/_components/influxdb.markdown b/source/_components/influxdb.markdown index f2291bda29d..4252068c038 100644 --- a/source/_components/influxdb.markdown +++ b/source/_components/influxdb.markdown @@ -49,123 +49,13 @@ Configuration variables: - **tags** (*Optional*): Tags to mark the data. - **tags_attributes** (*Optional*): The list of attribute names which should be reported as tags and not fields to InfluxDB. For example, if set to `friendly_name`, it will be possible to group by entities' friendly names as well, in addition to their ids. -## {% linkable_title Data migration %} +## {% linkable_title Helper scripts %} -Starting with 0.36 the InfluxDB component has a new schema to store values in the InfluxDB databases. - -- There will no longer be any tags/fields named `time`. -- All numeric fields (int/float/bool) will be stored as float inside InfluxDB database. -- All string fields corresponding to state attributes will be renamed as `FIELDNAME_str`, where `FIELDNAME` is the state attribute, to avoid type conflicts. -- All string fields corresponding to a state will be renamed as `state` (former value). -- Fields named `value` will always be stored as float. -- Fields named `state` will always be stored as string. - -### {% linkable_title Migration script %} - -If you need to migrate your database, you may require to run the `influxdb_migrator` script. Run the script after upgrade to 0.36 but before the first regular start of `hass` version 0.36. - -These are the steps the script will perform: -1. Create a new database (called `DBNAME__old`) to store old data. -2. Copy data from `DBNAME` database to `DBNAME__old` database. -3. Empty `DBNAME` database (using `drop` then `create`). `DBNAME` database is now considered as the new database. -4. For each measurement of `DBNAME__old` database: - 1. Read all points from the current measurement (in groups of 1000 points by default) and convert them. - 2. Send group of points to `DBNAME` database. -5. Delete the `DBNAME__old` database if needed. - -Example to run the script: - -```bash -$ hass --script influxdb_migrator \ - -H IP_INFLUXDB_HOST -u INFLUXDB_USERNAME -p INFLUXDB_PASSWORD \ - -d INFLUXDB_DB_NAME -``` -Script arguments: - -``` -required arguments: - -d dbname, --dbname dbname InfluxDB database name - -optional arguments: - -h, --help show this help message and exit - -H host, --host host InfluxDB host address - -P port, --port port InfluxDB host port - -u username, --username username - InfluxDB username - -p password, --password password - InfluxDB password - -s step, --step step How many points to migrate at the same time - -o override_measurement, --override-measurement override_measurement - Store all your points in the same measurement - -D, --delete Delete old database -``` - -- If you run the script with only the `-h` option, you will get a help printout with a short explanation of the different options. -- The host option defaults to `'127.0.0.1'`. -- The port option defaults to `8086`. -- You should be able to omit username and password if InfluxDB authentication is disabled, which it is by default. -- The step option defaults to `1000`. - - -## {% linkable_title Data import script %} - -If you want to import all the recorded data from your recorder database you can use the data import script. -It will read all your state_change events from the database and add them as data-points to the InfluxDB. -You can specify the source database either by pointing the `--config` option to the config directory which includes the default SQLite database or by giving a sqlalchemy connection URI with `--uri`. -The writing to InfluxDB is done in batches that can be changed with `--step`. - -You can control, which data is imported by using the command line options `--exclude_entities` and `--exclude_domains`. -Both get a comma separated list of either entity-ids or domain names that are excluded from the import. - -To test what gets imported you can use the `--simulate` option, which disables the actual write to the InfluxDB instance. -This only writes the statistics how much points would be imported from which entity. - -Example to run the script: - -```bash -$ hass --script influxdb_import --config CONFIG_DIR \ - -H IP_INFLUXDB_HOST -u INFLUXDB_USERNAME -p INFLUXDB_PASSWORD \ - --dbname INFLUXDB_DB_NAME --exclude_domains automation,configurator -``` -Script arguments: - -``` -required arguments: - -d dbname, --dbname dbname - InfluxDB database name - -optional arguments: - -h, --help show this help message and exit - -c path_to_config_dir, --config path_to_config_dir - Directory that contains the Home Assistant - configuration - --uri URI Connect to URI and import (if other than default - sqlite) eg: mysql://localhost/homeassistant - - -H host, --host host InfluxDB host address - -P port, --port port InfluxDB host port - -u username, --username username - InfluxDB username - -p password, --password password - InfluxDB password - -s step, --step step How many points to import at the same time - -t tags, --tags tags Comma separated list of tags (key:value) for all - points - -D default_measurement, --default-measurement default_measurement - Store all your points in the same measurement - -o override_measurement, --override-measurement override_measurement - Store all your points in the same measurement - -e exclude_entities, --exclude_entities exclude_entities - Comma separated list of excluded entities - -E exclude_domains, --exclude_domains exclude_domains - Comma separated list of excluded domains - -S, --simulate Do not write points but simulate preprocessing - and print statistics -``` +- [Helper script `influxdb_import`](/docs/tools/influxdb_import/) +- [Helper script `db_migrator`](/docs/tools/db_migrator/) (only used for [Home Assistant 0.36](/blog/2017/01/14/iss-usps-images-packages/#influxdb-export)) ## {% linkable_title Examples %} - ### {% linkable_title Full configuration %} ```yaml diff --git a/source/_docs/configuration/secrets.markdown b/source/_docs/configuration/secrets.markdown index 523bba9c4c4..906c40f536e 100644 --- a/source/_docs/configuration/secrets.markdown +++ b/source/_docs/configuration/secrets.markdown @@ -53,65 +53,15 @@ logger: debug ``` This will not print the actual secret's value to the log. -*Option 2*: View where secrets are retrieved from and the contents of all `secrets.yaml` files used, you can use the `check_config` script from the command line: +*Option 2*: View where secrets are retrieved from and the contents of all `secrets.yaml` files used, you can use the [`check_config` script](/docs/tools/check_config/) from the command line: ```bash $ hass --script check_config --secrets ``` This will print all your secrets. -### {% linkable_title Storing passwords in a keyring managed by your OS %} +## {% linkable_title Alternatives to `secrets.yaml` %} -Using [Keyring](https://github.com/jaraco/keyring) is an alternative way to `secrets.yaml`. They can be managed from the command line via the `keyring` script. +- [Using a keyring that is managed by your OS to store secrets](/docs/tools/keyring/) +- [Storing passwords securely in AWS](/docs/tools/credstash/) -```bash -$ hass --script keyring --help -``` - -To store a password in keyring, replace your password or API key with `!secret` and an identifier in `configuration.yaml` file. - -```yaml -http: - api_password: !secret http_password -``` - -Create an entry in your keyring. - -```bash -$ hass --script keyring set http_password -``` - -If you launch Home Assistant now, you will be prompted for the keyring password to unlock your keyring. - -```bash -$ hass -Config directory: /home/homeassistant/.homeassistant -Please enter password for encrypted keyring: -``` - -

- If you are using the Python Keyring, [autostarting](/getting-started/autostart/) of Home Assistant will no longer work. -

- -### {% linkable_title Storing passwords securely in AWS %} - -Using [Credstash](https://github.com/fugue/credstash) is an alternative way to `secrets.yaml`. They can be managed from the command line via the credstash script. - -Before using credstash, you need to set up AWS credentials either via the `aws` command line tool or using environment variables as explained in the [AWS CLI docs](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) as well as creating a KMS key named `credstash` as explained in the [credstash Readme](https://github.com/fugue/credstash#setting-up-kms). After that is complete, you can use the provided script to add secrets to your Home Assistant secret store in credstash. - -```bash -$ hass --script credstash --help -``` - -To store a password in credstash, replace your password or API key with `!secret` and an identifier in `configuration.yaml` file. - -```yaml -http: - api_password: !secret http_password -``` - -Create an entry in your credstash store. - -```bash -$ hass --script credstash set http_password -``` diff --git a/source/_docs/tools/benchmark.markdown b/source/_docs/tools/benchmark.markdown new file mode 100644 index 00000000000..c95894489e3 --- /dev/null +++ b/source/_docs/tools/benchmark.markdown @@ -0,0 +1,20 @@ +--- +layout: page +title: "benchmark" +description: "Script to perform benchmarking of Home Assistant" +release_date: 2017-02-23 11:00:00 +sidebar: true +comments: false +sharing: true +footer: true +redirect_from: /docs/tools/scripts/#benchmark +--- + +For testing the performance of Home Assistant the Benchmark script runs until you exit using Control+C. + +Firing and handling of a million events. + +```bash +$ hass --script benchmark async_million_events +``` + diff --git a/source/_docs/tools/check_config.markdown b/source/_docs/tools/check_config.markdown new file mode 100644 index 00000000000..f3a926783e2 --- /dev/null +++ b/source/_docs/tools/check_config.markdown @@ -0,0 +1,18 @@ +--- +layout: page +title: "check_config" +description: "Script to perform a check of the current configuration" +release_date: 2017-02-23 11:00:00 +sidebar: true +comments: false +sharing: true +footer: true +redirect_from: /docs/tools/scripts/#configuration-check +--- + +Test any changes to your `configuration.yaml` file before launching Home Assistant. This script allows you to test changes without the need to restart Home Assistant. + +```bash +$ hass --script check_config +``` + diff --git a/source/_docs/tools/credstash.markdown b/source/_docs/tools/credstash.markdown new file mode 100644 index 00000000000..0d419a104f4 --- /dev/null +++ b/source/_docs/tools/credstash.markdown @@ -0,0 +1,33 @@ +--- +layout: page +title: "credstash" +description: "Script to store credentials securely in AWS" +release_date: 2017-02-23 11:00:00 +sidebar: true +comments: false +sharing: true +footer: true +redirect_from: /docs/configuration/secrets/#storing-passwords-securely-in-aws +--- + +Using [Credstash](https://github.com/fugue/credstash) is an alternative way to `secrets.yaml`. They can be managed from the command line via the credstash script. + +Before using credstash, you need to set up AWS credentials either via the `aws` command line tool or using environment variables as explained in the [AWS CLI docs](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) as well as creating a KMS key named `credstash` as explained in the [credstash Readme](https://github.com/fugue/credstash#setting-up-kms). After that is complete, you can use the provided script to add secrets to your Home Assistant secret store in credstash. + +```bash +$ hass --script credstash --help +``` + +To store a password in credstash, replace your password or API key with `!secret` and an identifier in `configuration.yaml` file. + +```yaml +http: + api_password: !secret http_password +``` + +Create an entry in your credstash store. + +```bash +$ hass --script credstash set http_password +``` + diff --git a/source/_docs/tools/db_migrator.markdown b/source/_docs/tools/db_migrator.markdown new file mode 100644 index 00000000000..bdce8d1ab04 --- /dev/null +++ b/source/_docs/tools/db_migrator.markdown @@ -0,0 +1,72 @@ +--- +layout: page +title: "db_migrator" +description: "Script to migrate data in an InfluxDB database" +release_date: 2017-02-23 11:00:00 +sidebar: true +comments: false +sharing: true +footer: true +redirect_from: /components/influxdb/#data-migration +--- + +

+This script was only use for 0.36 release cycle! +

+ +Starting with 0.36 the [InfluxDB](omponents/influxdb/) component has a new schema to store values in the InfluxDB databases. + +- There will no longer be any tags/fields named `time`. +- All numeric fields (int/float/bool) will be stored as float inside InfluxDB database. +- All string fields corresponding to state attributes will be renamed as `FIELDNAME_str`, where `FIELDNAME` is the state attribute, to avoid type conflicts. +- All string fields corresponding to a state will be renamed as `state` (former value). +- Fields named `value` will always be stored as float. +- Fields named `state` will always be stored as string. + +## {% linkable_title Migration script %} + +If you need to migrate your database, you may require to run the `influxdb_migrator` script. Run the script after upgrade to 0.36 but before the first regular start of `hass` version 0.36. + +These are the steps the script will perform: + +1. Create a new database (called `DBNAME__old`) to store old data. +2. Copy data from `DBNAME` database to `DBNAME__old` database. +3. Empty `DBNAME` database (using `drop` then `create`). `DBNAME` database is now considered as the new database. +4. For each measurement of `DBNAME__old` database: + 1. Read all points from the current measurement (in groups of 1000 points by default) and convert them. + 2. Send group of points to `DBNAME` database. +5. Delete the `DBNAME__old` database if needed. + +Example to run the script: + +```bash +$ hass --script influxdb_migrator \ + -H IP_INFLUXDB_HOST -u INFLUXDB_USERNAME -p INFLUXDB_PASSWORD \ + -d INFLUXDB_DB_NAME +``` +Script arguments: + +``` +required arguments: + -d dbname, --dbname dbname InfluxDB database name + +optional arguments: + -h, --help show this help message and exit + -H host, --host host InfluxDB host address + -P port, --port port InfluxDB host port + -u username, --username username + InfluxDB username + -p password, --password password + InfluxDB password + -s step, --step step How many points to migrate at the same time + -o override_measurement, --override-measurement override_measurement + Store all your points in the same measurement + -D, --delete Delete old database +``` + +- If you run the script with only the `-h` option, you will get a help printout with a short explanation of the different options. +- The host option defaults to `'127.0.0.1'`. +- The port option defaults to `8086`. +- You should be able to omit username and password if InfluxDB authentication is disabled, which it is by default. +- The step option defaults to `1000`. + diff --git a/source/_docs/tools/ensure_config.markdown b/source/_docs/tools/ensure_config.markdown new file mode 100644 index 00000000000..c9539398567 --- /dev/null +++ b/source/_docs/tools/ensure_config.markdown @@ -0,0 +1,18 @@ +--- +layout: page +title: "ensure_config" +description: "Script to perform a check if the configuration file exists" +release_date: 2017-02-23 11:00:00 +sidebar: true +comments: false +sharing: true +footer: true +redirect_from: /docs/tools/scripts/#existence-of-configuration +--- + +This script checks if the `configuration.yaml` file exists. If the file is not available, one is created. + +```bash +$ hass --script ensure_config +``` + diff --git a/source/_docs/tools/influxdb_import.markdown b/source/_docs/tools/influxdb_import.markdown new file mode 100644 index 00000000000..b8bea1046cc --- /dev/null +++ b/source/_docs/tools/influxdb_import.markdown @@ -0,0 +1,64 @@ +--- +layout: page +title: "influxdb_import" +description: "Script to import data into an InfluxDB database" +release_date: 2017-02-23 11:00:00 +sidebar: true +comments: false +sharing: true +footer: true +redirect_from: /components/influxdb/#data-import-script +--- + +If you want to import all the recorded data from your recorder database you can use the data import script. It will read all your state_change events from the database and add them as data-points to the InfluxDB. You can specify the source database either by pointing the `--config` option to the config directory which includes the default SQLite database or by giving a sqlalchemy connection URI with `--uri`. + +The writing to InfluxDB is done in batches that can be changed with `--step`. + +You can control, which data is imported by using the command line options `--exclude_entities` and `--exclude_domains`. Both get a comma separated list of either entity-ids or domain names that are excluded from the import. + +To test what gets imported you can use the `--simulate` option, which disables the actual write to the InfluxDB instance. This only writes the statistics how much points would be imported from which entity. + +Example to run the script: + +```bash +$ hass --script influxdb_import --config CONFIG_DIR \ + -H IP_INFLUXDB_HOST -u INFLUXDB_USERNAME -p INFLUXDB_PASSWORD \ + --dbname INFLUXDB_DB_NAME --exclude_domains automation,configurator +``` +Script arguments: + +``` +required arguments: + -d dbname, --dbname dbname + InfluxDB database name + +optional arguments: + -h, --help show this help message and exit + -c path_to_config_dir, --config path_to_config_dir + Directory that contains the Home Assistant + configuration + --uri URI Connect to URI and import (if other than default + sqlite) eg: mysql://localhost/homeassistant + + -H host, --host host InfluxDB host address + -P port, --port port InfluxDB host port + -u username, --username username + InfluxDB username + -p password, --password password + InfluxDB password + -s step, --step step How many points to import at the same time + -t tags, --tags tags Comma separated list of tags (key:value) for all + points + -D default_measurement, --default-measurement default_measurement + Store all your points in the same measurement + -o override_measurement, --override-measurement override_measurement + Store all your points in the same measurement + -e exclude_entities, --exclude_entities exclude_entities + Comma separated list of excluded entities + -E exclude_domains, --exclude_domains exclude_domains + Comma separated list of excluded domains + -S, --simulate Do not write points but simulate preprocessing + and print statistics +``` + + diff --git a/source/_docs/tools/keyring.markdown b/source/_docs/tools/keyring.markdown new file mode 100644 index 00000000000..6434337b39f --- /dev/null +++ b/source/_docs/tools/keyring.markdown @@ -0,0 +1,42 @@ +--- +layout: page +title: "keyring" +description: "Script to store secrets in a keyring" +release_date: 2017-02-23 11:00:00 +sidebar: true +comments: false +sharing: true +footer: true +redirect_from: /docs/configuration/secrets/#storing-passwords-in-a-keyring-managed-by-your-os +--- + +Using [Keyring](https://github.com/jaraco/keyring) is an alternative way to `secrets.yaml`. The secrets can be managed from the command line via the `keyring` script. + +```bash +$ hass --script keyring --help +``` + +To store a password in keyring, replace your password or API key with `!secret` and an identifier in `configuration.yaml` file. + +```yaml +http: + api_password: !secret http_password +``` + +Create an entry in your keyring. + +```bash +$ hass --script keyring set http_password +``` + +If you launch Home Assistant now, you will be prompted for the keyring password to unlock your keyring. + +```bash +$ hass +Config directory: /home/homeassistant/.homeassistant +Please enter password for encrypted keyring: +``` + +

+ If you are using the Python Keyring, [autostarting](/getting-started/autostart/) of Home Assistant will no longer work. +

diff --git a/source/_docs/tools/scripts.markdown b/source/_docs/tools/scripts.markdown index a0b96505e3f..2b40ba28c5c 100644 --- a/source/_docs/tools/scripts.markdown +++ b/source/_docs/tools/scripts.markdown @@ -7,47 +7,7 @@ sidebar: true comments: false sharing: true footer: true +redirect_from: /docs/tools/ --- -The command-line and the frontend which simplify common tasks, are helping with migrations, and ensure that Home Assistant runs properly. Please do not confuse those with Home Assistant's [script](/docs/scripts/) feature. -### {% linkable_title Configuration check %} - -Test any changes to your `configuration.yaml` file before launching Home Assistant. This script allows you to test changes without the need to restart Home Assistant. - -```bash -$ hass --script check_config -``` - -### {% linkable_title Existence of configuration %} - -This script checks if the `configuration.yaml` file exists. If the file is not available, one is created. - -```bash -$ hass --script ensure_config -``` - -### {% linkable_title Secrets %} - -There is a method to store secrets outside of your `configuration.yaml` file. For further details, please refer to the [Storing Secrets](/docs/configuration/secrets/) documentation. - -```bash -$ hass --script keyring -``` - -### {% linkable_title Benchmark %} - -For testing the performance of Home Assistant the Benchmark script runs until you exit using Control+C. - -Firing and handling of a million events. - -```bash -$ hass --script benchmark async_million_events -``` - -### {% linkable_title Old scripts %} - -Usually those scripts were only used when a massive update happened and was announced in the release notes. - -- `db_migrator`: Migrate an existing SQLite database to the new schema. -- `influxdb_migrator`: Convert an old InfluxDB to the new format. diff --git a/source/_includes/asides/docs_navigation.html b/source/_includes/asides/docs_navigation.html index 463f65313e5..c697024aac3 100644 --- a/source/_includes/asides/docs_navigation.html +++ b/source/_includes/asides/docs_navigation.html @@ -87,7 +87,13 @@