jeans/home-assistant.io
jeans/home-assistant.io
1
0
Fork 0
mirror of https://github.com/home-assistant/home-assistant.io.git synced 2025-07-15 21:36:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

2024.8: Textual tweaks to Ollama section

Browse Source
This commit is contained in:
Franck Nijhof 2024-08-07 17:09:38 +02:00
parent b9d769bce6
commit 9f88581ce3
No known key found for this signature in database
GPG Key ID: D62583BA8AB11CA3
1 changed files with 10 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
source/_posts/2024-08-07-release-20248.markdown
View File

@ -230,15 +230,18 @@ This is achieved thanks to [@Shulyaka] adding support for the brand new tools
API in Ollama. The performance of the local models has been fine tuned by
[@AllenPorter].
Allen created a new [LLM benchmark suite](https://github.com/allenporter/home-assistant-datasets/tree/main/reports#assist-mini) that is more balanced, less focused on edge cases, and uses fewer exposed entities. We scored the different models with this new benchmark and the cloud-based models scored 98%, but local LLMs did not do nearly as well. Through prompt tuning and fixes included in this release,
we have been able to get local LLMs to score a reasonable 83%.
Allen created a new [LLM benchmark suite] that is more balanced, less focused
on edge cases, and uses fewer exposed entities. We scored the different models
with this new benchmark, and the cloud-based models scored 98%, but local LLMs
did not do nearly as well.
Through prompt tuning and fixes included in this release, we have gotten local
LLMs to score a reasonable 83%. We will continue to test new models while
improving our prompts and tools to achieve a higher score.
<img class="no-shadow" src="/images/blog/2024-08/llama-3.1-iteration.png" alt="Graph showing the iteration progress of implementing local Ollama support using the Llama 3.1 8B model.">
We will continue to test new models, while improving our prompts
and tools to achieve a higher score.
If you would like to experiment with local LLMs using Home Assistant, we currently
If you want to experiment with local LLMs using Home Assistant, we currently
recommend using the Llama 3.1 8B model and exposing fewer than 25 entities. Note
that smaller models are more likely to make mistakes.
@ -246,6 +249,7 @@ that smaller models are more likely to make mistakes.
[@Shulyaka]: https://github.com/Shulyaka
[control your home using Large Language Models]: /blog/2024/06/05/release-20246/#dipping-our-toes-in-the-world-of-ai-using-llms
[GoogleAI]: /integrations/google_generative_ai_conversation/
[LLM benchmark suite]: https://github.com/allenporter/home-assistant-datasets/tree/main/reports#assist-mini
[OpenAI]: /integrations/openai_conversation
## Integrations