diff --git a/source/_components/image_processing.tensorflow.markdown b/source/_components/image_processing.tensorflow.markdown new file mode 100644 index 00000000000..81c9324007b --- /dev/null +++ b/source/_components/image_processing.tensorflow.markdown @@ -0,0 +1,163 @@ +--- +layout: page +title: "TensorFlow" +description: "Detect and recognize objects with TensorFlow." +date: 2018-10-24 00:00 +sidebar: true +comments: false +sharing: true +footer: true +logo: tensorflow.png +ha_category: Image Processing +ha_iot_class: "Local Polling" +ha_release: 0.82 +--- + +The `tensorflow` image processing platform allows you to detect and recognize objects in a camera image using [TensorFlow](https://www.tensorflow.org/). The state of the entity is the number of objects detected, and recognized objects are listed in the `summary` attribute along with quantity. The `matches` attribute provides the confidence `score` for recognition and the bounding `box` of the object for each detection category. + +

+ The following packages must be installed on Hassbian after following the setup for the component to work: + `$ sudo apt-get install libatlas-base-dev libopenjp2-7 libtiff5` +

+ +## {% linkable_title Setup %} + +You need to install the `tensorflow` Python packages with: `$ pip3 install tensorflow`. The wheel is not available for all platforms. See [the official install guide](https://www.tensorflow.org/install/) for other options. Hass.io has this package pre-installed. + +This component requires files to be downloaded, compiled on your computer, and added to the Home Assistant configuration directory. These steps can be performed using the sample script at [this gist](https://gist.github.com/hunterjm/6f9332f92b60c3d5e448ad936d7353c3). Alternatively, if you wish to perform the process manually, the process is as follows: + +- Clone [tensorflow/models](https://github.com/tensorflow/models/tree/master/research/object_detection) +- Compile protobuf models located in `research/object_detection/protos` with `protoc` +- Create the following directory structure inside your config directory: + +```bash + |- {config_dir} + | - tensorflow/ + |- object_detection/ + |- __init__.py +``` + +- Copy required object_detection dependancies to the `object_detection` folder inside of the `tensorflow` folder: + + - `research/object_detection/data` + - `research/object_detection/utils` + - `research/object_detection/protos` + +## {% linkable_title Model Selection %} + +Lastly, it is time to pick a model. It is recommended to start with one of the COCO models available in the [Model Detection Zoo](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md). + +The trade-off between the different models is accuracy vs speed. Users with a decent CPU should start with the `faster_rcnn_inception_v2_coco` model. If you are running on an ARM device like a Raspberry Pi, start with the `ssd_mobilenet_v2_coco` model. + +Whichever model you choose, download it and place the `frozen_inference_graph.pb` file in the `tensorflow` folder in your configuration directory. + +## {% linkable_title Configuration %} + +To enable this platform in your installation, add the following to your `configuration.yaml` file: + +```yaml +# Example configuration.yaml entry +image_processing: + - platform: tensorflow + source: + - entity_id: camera.local_file + model: + graph: /home/homeassistant/.homeassistant/tensorflow/frozen_inference_graph.pb +``` + +{% configuration %} +source: + description: The list of image sources. + required: true + type: map + keys: + entity_id: + description: A camera entity id to get picture from. + required: true + type: string + name: + description: This parameter allows you to override the name of your `image_processing` entity. + required: false + type: string +file_out: + description: A [template](/docs/configuration/templating/#processing-incoming-data) for the component to save processed images including bounding boxes. `camera_entity` is available as the `entity_id` string of the triggered source camera. + required: false + type: list +model: + description: Information about the TensorFlow model. + required: true + type: map + keys: + graph: + description: Full path to `frozen_inference_graph.pb`. + required: true + type: string + labels: + description: Full path to a `*label_map.pbtext`. + required: false + type: string + default: tensorflow/object_detection/data/mscoco_label_map.pbtxt + model_dir: + description: Full path to tensorflow models directory. + required: false + type: string + default: /tensorflow inside config + area: + description: Custom detection area. Only objects fully in this box will be reported. Top of image is 0, bottom is 1. Same left to right. + required: false + type: map + keys: + top: + description: Top line defined as % from top of image. + required: false + type: float + default: 0 + left: + description: Left line defined as % from left of image. + required: false + type: float + default: 0 + bottom: + description: Bottom line defined as % from top of image. + required: false + type: float + default: 1 + right: + description: Right line defined as % from left of image. + required: false + type: float + default: 1 + categories: + description: List of categories to include in object detection. Can be seen in the file provided to `labels`. + type: list + required: false +{% endconfiguration %} + +`categories` can also be defined as dictionary providing an `area` for each category as seen in the advanced configuration below: + +```yaml +# Example advanced configuration.yaml entry +image_processing: + - platform: tensorflow + source: + - entity_id: camera.driveway + - entity_id: camera.backyard + file_out: + - "/tmp/{% raw %}{{ camera_entity.split('.')[1] }}{% endraw %}_latest.jpg" + - "/tmp/{% raw %}{{ camera_entity.split('.')[1] }}_{{ now().strftime('%Y%m%d_%H%M%S') }}{% endraw %}.jpg" + model: + graph: /home/homeassistant/.homeassistant/tensorflow/frozen_inference_graph.pb + categories: + - category: person + area: + # Exclude top 10% of image + top: 0.1 + # Exclude right 15% of image + right: 0.85 + - car + - truck +``` + +## {% linkable_title Optimising resources %} + +[Image processing components](https://www.home-assistant.io/components/image_processing/) process the image from a camera at a fixed period given by the `scan_interval`. This leads to excessive processing if the image on the camera hasn't changed, as the default `scan_interval` is 10 seconds. You can override this by adding to your config `scan_interval: 10000` (setting the interval to 10,000 seconds), and then call the `image_processing.scan` service when you actually want to perform processing. diff --git a/source/images/supported_brands/tensorflow.png b/source/images/supported_brands/tensorflow.png new file mode 100644 index 00000000000..e7208340a2c Binary files /dev/null and b/source/images/supported_brands/tensorflow.png differ