From 7cad0dd2fa44328ba485b6cfbec33136caae843c Mon Sep 17 00:00:00 2001
From: Paulus Schoutsen <paulus@paulusschoutsen.nl>
Date: Mon, 17 Oct 2016 22:29:06 -0700
Subject: [PATCH] Add async docs

---
 .../asides/developers_navigation.html         |  11 +-
 source/developers/asyncio.markdown            |  27 +++++
 .../asyncio_categorizing_functions.markdown   |  79 ++++++++++++
 source/developers/asyncio_misc.markdown       |  22 ++++
 .../asyncio_working_with_async.markdown       | 113 ++++++++++++++++++
 5 files changed, 250 insertions(+), 2 deletions(-)
 create mode 100644 source/developers/asyncio.markdown
 create mode 100644 source/developers/asyncio_categorizing_functions.markdown
 create mode 100644 source/developers/asyncio_misc.markdown
 create mode 100644 source/developers/asyncio_working_with_async.markdown

diff --git a/source/_includes/asides/developers_navigation.html b/source/_includes/asides/developers_navigation.html
index 240deeafee8..154e405eac2 100644
--- a/source/_includes/asides/developers_navigation.html
+++ b/source/_includes/asides/developers_navigation.html
@@ -43,9 +43,16 @@
         </ul>
       </li>
       <li>
-        Frontend Development
+        {% active_link /developers/asyncio/ Asynchronous Programming %}
+        <ul>
+          <li>{% active_link /developers/asyncio_categorizing_functions/ Categorizing Functions %}</li>
+          <li>{% active_link /developers/asyncio_working_with_async/ Working with Async %}</li>
+          <li>{% active_link /developers/asyncio_misc/ Miscellaneous %}</li>
+        </ul>
+      </li>
+      <li>
+        {% active_link /developers/frontend/ Frontend Development %}
         <ul>
-          <li>{% active_link /developers/frontend/ Setup Frontend Environment %}</li>
           <li>{% active_link /developers/frontend_add_card/ Add State Card %}</li>
           <li>{% active_link /developers/frontend_add_more_info/ Add More Info Dialog %}</li>
           <li>{% active_link /developers/frontend_creating_custom_panels/ Add Custom Panels %}</li>
diff --git a/source/developers/asyncio.markdown b/source/developers/asyncio.markdown
new file mode 100644
index 00000000000..f663b8d36e9
--- /dev/null
+++ b/source/developers/asyncio.markdown
@@ -0,0 +1,27 @@
+---
+layout: page
+title: "Asynchronous Programming"
+description: "Introduction to the asynchronous core of Home Assistant."
+date: 2016-10-17 21:49
+sidebar: true
+comments: false
+sharing: true
+footer: true
+---
+
+On September 29, 2016 we released [Home Assistant 0.29][0.29] as part of our bi-weekly release schedule. This release introduced a complete overhaul of the core spearheaded by [Ben Bangert][ben].
+
+The old core was set up like a “traditional” threaded application. Each resource that was not thread safe (ie. the state of entities) would be protected by a lock. This caused a lot of waiting and potential inconsistency because a task could now end up waiting halfway through it’s job until some resource got freed.
+
+Our new core is based on an Python’s built-in asyncio module. Instead of having all threads have access to the core API objects, access is now limited to a special thread called the event loop. All components will now schedule themselves as a task to be executed by the event loop. This gives us the guarantee that only one task is executed at once, meaning we no longer need any locks.
+
+The only problem with running everything inside the event loop is when a task is doing blocking I/O, what most third-party Python libraries are doing. For example while requesting new information from a device, the core will stop running until we get a response from the device. To handle this, a task is able to suspend itself until the response is available after which it will be enqueued for the event loop to process the result.
+
+For a task to be able to suspend itself, all code that it calls has to have this capability added. This means in practice that each device integration will need a full rewrite of the library that offers the integration! As this is not something that can be achieved, ever, a 100% backwards compatible API has been added so that no platform will require updating.
+
+The backwards compatible API works by scheduling a task from a different thread and blocking that thread until the task has been processed by the event loop.
+
+### [Next step: Categorizing Functions &raquo;](/developers/asyncio_categorizing_functions/)
+
+[0.29]: https://home-assistant.io/blog/2016/09/29/async-sleepiq-emoncms-stocks/
+[ben]: https://github.com/bbangert/
diff --git a/source/developers/asyncio_categorizing_functions.markdown b/source/developers/asyncio_categorizing_functions.markdown
new file mode 100644
index 00000000000..bd3ff859b8b
--- /dev/null
+++ b/source/developers/asyncio_categorizing_functions.markdown
@@ -0,0 +1,79 @@
+---
+layout: page
+title: "Categorizing Functions"
+description: "A categorization of functions to work with the asynchronous core of Home Assistant."
+date: 2016-10-17 21:49
+sidebar: true
+comments: false
+sharing: true
+footer: true
+---
+
+A piece of work within Home Assistant is represented by a function that will be invoked. It will either run inside our event loop or inside our thread pool, depending on if it is async safe.
+
+Home Assistant uses the convention that all functions that must be run from within the event loop are prefixed with `async_`.
+
+## {% linkable_title The coroutine function %}
+
+Coroutines are special functions based on Python’s generators syntax which allows them to suspend execution while waiting on a result.
+
+Invoking a coroutine function will return a Generator object back, but will not actually begin execution. This object will execute the task when it is either yielded from (from within another coroutine) or it is scheduled on the event loop.
+
+To declare a function a coroutine, import the coroutine annotation from the asyncio package and annotate your function.
+
+```python
+import asyncio
+
+@asyncio.coroutine
+def async_look_my_coroutine(target):
+    result = yield from entity.async_turn_on()
+    if result:
+        print("hello {}".format(target))
+
+hass.loop.create_task(async_look_my_coroutine("world")
+```
+
+In this example, we schedule the coroutine by calling `hass.loop.create_task`. This will add the coroutine to the queue of tasks to be run. When the event loop is running `async_look_my_coroutine` it will suspend the task when `yield from entity.async_turn_on()` is called. At that point a new task will be scheduled to execute `entity.async_turn_on()`. When that job has been executed, `async_look_my_coroutine` will resume.
+
+## {% linkable_title The callback function %}
+
+This is a normal function that is considered safe to be run from within the event loop. A callback is unable to suspend itself and thus cannot do any I/O or call a coroutine. A callback is capable of scheduling a new task but it will not be able to wait for the results.
+
+To declare a function as a callback, import the callback annotation from the core package and annotate your function.
+
+A common use case for a callback in Home Assistant is as a listener for an event or a service call. It can process the incoming information and then schedule the right calls to be made. Example from the automation component.
+
+```python
+from homeassistant.core import callback
+
+@callback
+def async_trigger_service_handler(service_call):
+    """Handle automation trigger service calls."""
+    vars = service_call.data.get(ATTR_VARIABLES)
+    for entity in component.async_extract_from_service(service_call):
+        hass.loop.create_task(entity.async_trigger(vars, True))
+```
+
+In this example, `entity.async_trigger` is a coroutine function. Invoking the coroutine function will return a coroutine task. The passed in parameters will be used when the task gets executed.
+
+To execute the task we have to schedule it for execution on the event loop. This is done by calling `hass.loop.create_task`.
+
+### {% linkable_title Why even have callbacks? %}
+
+You might wonder, if a coroutine can do everything a callback can do, why even have a callback. The reason is performance and better state consistency of the core API objects.
+
+When coroutine A waits for coroutine B, it will suspend itself and schedule a new task to run B. This means that the event loop is now running A, B and then A again. If B is a callback, A will never have to suspend itself and thus the event loop is just running A. The consistency implication is that other events queued to run on the event loop continue to wait until callbacks complete, but will be interleaved when yielding to another coroutine.
+
+## {% linkable_title Event loop and thread safe %}
+
+These are functions that are safe to run both in a thread and inside the event loop. These functions are usually performing a computation or transform data in memory. Anything that does I/O does not fall under this category. Many standard library functions fall in this category. For example generating the sum of a set of numbers using sum or merging two dictionaries.
+
+There is no special annotation to mark functions as part of this category and care should be taken when using these functions from inside the event loop. When in doubt, look at their implementation.
+
+## {% linkable_title Other functions %}
+
+These are all the functions that did not fit in the previous categories. These functions are either thread-safe or not considered safe to be run within the event loop. These are functions that use sleep, or perform I/O.
+
+There is no special annotation necessary to be considered part of this category.
+
+### [Next step: Working with Async &raquo;](/developers/asyncio_working_with_async/)
diff --git a/source/developers/asyncio_misc.markdown b/source/developers/asyncio_misc.markdown
new file mode 100644
index 00000000000..695b26e3e0b
--- /dev/null
+++ b/source/developers/asyncio_misc.markdown
@@ -0,0 +1,22 @@
+---
+layout: page
+title: "Miscellaneous Async"
+description: "A collection of miscellaneous topics about async that didn't fit on the other pages."
+date: 2016-10-17 21:49
+sidebar: true
+comments: false
+sharing: true
+footer: true
+---
+
+## {% linkable_title What about ‘async’ and ‘await’ syntax? %}
+Python 3.5 introduced new syntax to formalize the asynchronous pattern. This is however not compatible with Python 3.4. The minimum required Python version for Home Assistant is based on the Python version shipped with Debian stable, which is currently 3.4.2.
+
+For more information, Brett Cannon wrote [an excellent breakdown][brett] on 'async' and 'await' syntax and how asynchronous programming works.
+
+## {% linkable_title Acknowledgements %}
+
+Huge thanks to [Ben Bangert][ben] for starting the conversion of the core to async, guiding other contributors while taking their first steps with async programming and peer reviewing this documentation.
+
+[brett]: http://www.snarky.ca/how-the-heck-does-async-await-work-in-python-3-5
+[ben]: https://github.com/bbangert/
diff --git a/source/developers/asyncio_working_with_async.markdown b/source/developers/asyncio_working_with_async.markdown
new file mode 100644
index 00000000000..48839b4648e
--- /dev/null
+++ b/source/developers/asyncio_working_with_async.markdown
@@ -0,0 +1,113 @@
+---
+layout: page
+title: "Working with Async"
+description: "A breakdown of all the different ways to work with the asynchronous core of Home Assistant."
+date: 2016-10-17 21:49
+sidebar: true
+comments: false
+sharing: true
+footer: true
+---
+
+Although we have a backwards compatible API, using the async core directly will be a lot faster. Most core components have already been rewritten to leverage the async core. This includes the EntityComponent helper (foundation of light, switch, etc), scripts, groups and automation.
+
+## {% linkable_title Interacting with the core %}
+
+[All methods in the Home Assistant core][dev-docs] are implemented in two flavors: an async version and a version to be called from other threads. The versions for other are merely wrappers that call the async version in a threadsafe manner using [the available async utilities][dev-docs-async].
+
+So if you are making calls to the core (the hass object) from within a callback or coroutine, use the methods that start with async_. If you need to call an async_ function that is a coroutine, your task must also be a coroutine.
+
+## {% linkable_title Implementing an async component %}
+
+We currently do not support async setup for components. We do however support using async functions as service handlers. Just define your handlers as a callback or coroutine and register them as usual.
+
+## {% linkable_title Implementing an async platform %}
+
+For platforms we support async setup. Instead of setup_platform you need to have a coroutine async_setup_platform.
+
+```python
+setup_platform(hass, config, add_entities, discovery_info=None):
+    # Setup your platform outside of the event loop.
+```
+
+Will turn into:
+
+```python
+import asyncio
+
+@asyncio.coroutine
+def async_setup_platform(hass, config, async_add_entities,
+                         discovery_info=None):
+    # Setup your platform inside of the event loop
+```
+
+The only difference with the original parameters is that the add_entities function has been replaced by the coroutine `async_add_entities`.
+
+## {% linkable_title Implementing an async entity %}
+
+You can make your entity async friendly by converting your update method to be async. This requires the dependency of your entities to also be async friendly!
+
+```python
+class MyEntity(Entity):
+    def update(self):
+        """Retrieve latest state."""
+        self._state = fetch_state()
+```
+
+Will turn into:
+
+```python
+import asyncio
+
+class MyEntity(Entity):
+    @asyncio.coroutine
+    def async_update(self):
+        """Retrieve latest state."""
+        self._state = yield from async_fetch_state()
+```
+
+Make sure that all properties defined on your entity do not result in I/O being done. All data has to be fetched inside the update method and cached on the entity. This is because these properties are read from within the event loop and thus doing I/O will result in the core of Home Assistant waiting until your I/O is done.
+
+## {% linkable_title Calling async functions from threads %}
+
+Sometimes it will happen that you’re in a thread and you want to call a function that is only available as async. Home Assistant includes a few async helper utilities to help with this.
+
+In the following example, `say_hello` will schedule `async_say_hello` and block till the function has run and get the result back.
+
+```python
+from homeassistant.util.async import run_callback_threadsafe
+
+def say_hello(hass, target):
+    return run_callback_threadsafe(
+        hass.loop, async_say_hello, target).result()
+
+def async_say_hello(hass, target):
+    return "Hello {}!".format(target)
+```
+
+## {% linkable_title Dealing with passed in functions %}
+
+If your code takes in functions from other code, you will not know which category the function belongs to and how they should be invoked. This usually only occurs if your code supplies an event helper like `mqtt.async_subscribe` or `track_state_change_listener`.
+
+To help with this, there are two helper methods on the hass object that you can call from inside the event loop:
+
+#### {% linkable_title hass.async_run_job %}
+
+Use this method if the function should be called as soon as possible. This will call callbacks immediately, schedule coroutines for execution on the event loop and schedule other functions to be run inside the thread pool.
+
+| Callback | Call immediately.
+| Coroutine | Schedule for execution on the event loop.
+| Other functions | Schedule for execution in the thread pool.
+
+#### {% linkable_title hass.async_add_job %}
+
+Use this method if the function should be called but not get priority over already scheduled calls.
+
+| Callback | Schedule for execution on the event loop.
+| Coroutine | Schedule for execution on the event loop.
+| Other functions | Schedule for execution in the thread pool.
+
+### [Next step: Miscellaneous &raquo;](/developers/asyncio_misc/)
+
+[dev-docs]: https://dev-docs.home-assistant.io/en/master/api/core.html
+[dev-docs-async]: https://dev-docs.home-assistant.io/en/dev/api/util.html#module-homeassistant.util.async