Update LLM API docs (#2196)

This commit is contained in:
Paulus Schoutsen 2024-05-28 01:46:27 -04:00 committed by GitHub
parent 8704a08e1b
commit f43f2e4894
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194

View File

@ -3,7 +3,7 @@ title: "Home Assistant API for Large Language Models"
sidebar_label: "LLM API"
---
Home Assistant can interact with large language models (LLMs). By exposing a Home Assistant API to an LLM, the LLM can fetch data or control Home Assistant to better assist the user. Home Assistant comes with a built-in LLM API but custom integrations can register their own to provide more advanced functionality.
Home Assistant can interact with large language models (LLMs). By exposing a Home Assistant API to an LLM, the LLM can fetch data or control Home Assistant to better assist the user. Home Assistant comes with a built-in LLM API, but custom integrations can register their own to provide more advanced functionality.
## Built-in Assist API
@ -17,7 +17,7 @@ The LLM API needs to be integrated in two places in your integration. Users need
### Options flow
The chosen API should be stored in the config entry options. It should hold a string reference to the API id. If no API is selected, the key must be omitted.
The chosen API should be stored in the config entry options. It should hold a string reference to the API ID. If no API is selected, the key must be omitted.
In your options flow, you should offer a selector to the user to pick which API should be used.
@ -96,8 +96,6 @@ class MyConversationEntity(conversation.ConversationEntity):
self, user_input: conversation.ConversationInput
) -> conversation.ConversationResult:
"""Process the user input."""
prompt: str = ...
intent_response = intent.IntentResponse(language=user_input.language)
llm_api: llm.API | None = None
tools: list[dict[str, Any]] | None = None
@ -116,13 +114,30 @@ class MyConversationEntity(conversation.ConversationEntity):
return conversation.ConversationResult(
response=intent_response, conversation_id=user_input.conversation_id
)
prompt += "\n" + llm_api.prompt_template
tools = [
_format_tool(tool) # TODO format the tools as your LLM expects
for tool in llm_api.async_get_tools()
]
if llm_api:
empty_tool_input = llm.ToolInput(
tool_name="",
tool_args={},
platform=DOMAIN,
context=user_input.context,
user_prompt=user_input.text,
language=user_input.language,
assistant=conversation.DOMAIN,
device_id=user_input.device_id,
)
api_prompt = llm_api.async_get_api_prompt()
else:
api_prompt = llm.async_render_no_api_prompt(self.hass)
prompt = "\n".join((user_prompt, api_prompt))
# Interact with LLM and pass tools
for _iteration in range(10):
response = ... # Get response from your LLM. Include available tools and the results of previous tool calls
@ -206,6 +221,7 @@ The `llm.Tool` class has the following attributes:
The `llm.Tool` class has the following methods:
#### `async_call`
Perform the actual operation of the tool when called by the LLM. This must be an async method. Its arguments are `hass` and an instance of `llm.ToolInput`.
Response data must be a dict and serializable in JSON [`homeassistant.util.json.JsonObjectType`](https://github.com/home-assistant/home-assistant/blob/master/homeassistant/util/json.py).
@ -222,7 +238,7 @@ The `ToolInput` has following attributes:
| `context` | Context | The `homeassistant.core.Context` of the conversation |
| `user_prompt` | string | The raw text input that initiated the tool call |
| `language` | string | The language of the conversation agent, or "*" for any language |
| `assistant` | string | The assistant name used to control exposed entities. Currently only `conversation` is supported. |
| `assistant` | string | The assistant name used to control exposed entities. Currently, only `conversation` is supported. |
| `device_id` | string | The device_id of the device the user used to initiate the conversation. |
### API
@ -263,4 +279,3 @@ The `llm.API` class has the following attributes:
| `id` | string | A unique identifier for the API. Required. |
| `name` | string | The name of the API. Required. |
| `prompt_template` | string | A template to render the prompt to describe the tools to the LLM. |