Home Assistant finally integrates Python 3.11

MichaIng · 18 June 2023 13:13

Okay, it is completely unrelated to the Python version. ha-av compilation fails as well with Python 3.11.3 and 3.10.12. Searched a bit around and indeed it is the FFmpeg version which is too low on Bullseye: avutil: add AV_FRAME_DATA_SEI_UNREGISTERED side data type · FFmpeg/FFmpeg@bc8ab08 · GitHub
This was merged with FFmpeg 4.4 and Bullseye ships 4.3.

There is sadly nothing we can reasonably do about it. Either Home Assistant decides to revert this recent change to keep support for older FFmpeg version, or Home Assistant is not supported on 32-bit ARM Debian Bullseye. This is btw not an issue on x86 or aarch64 as PyPI provides pre-compiled wheels for these architectures. piwheels (for 32-bit ARM) provides them for Python 3.9 only, but I hope they will add Python 3.11 wheels soon now that Debian Bookworm was released.

What is however interesting is that it starts up fine with Python 3.10.12. I can access the web UI just fine. With the Python 3.11 builds I got a HTTP 500 error when trying to access. Not sure if I would just have needed to wait longer. But ha-av compilation is attempted on every service start (taking huge time and CPU recources) + the warning, so not an acceptable state.

whyisthisbroken · 18 June 2023 13:36

So in short, there is no solution at the moment to fix it… I mean only 32bit doesn’t work, 64bit should work right? Maybe I try also the script and install it manually

MichaIng · 18 June 2023 13:41

64-bit works and Debian Bookworm works.

MichaIng · 18 June 2023 13:43

For reference, all info added to the related HA forum topic: Unable to install package ha-av - #39 by MichaIng - Configuration - Home Assistant Community
I do not think they are gonna revert it or make the dependency of this variable optional, but at least we tried and all information is on the table.

EDIT: Btw, CI tests didn’t catch this issue as the service starts up (successfully) and is listening on the web UI port. These post-startup compilation failures are hard to catch .

MichaIng · 18 June 2023 17:24

Okay, never ending story: The reason for the unreachable web interface is unrelated to ha-av, but a different error, which indeed appears with Python 3.11(.4) only, not with Python 3.10.12:

Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]: 2023-06-18 18:52:10.341 ERROR (MainThread) [aiohttp.server] Error handling request
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]: Traceback (most recent call last):
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/home/homeassistant/.pyenv/versions/3.11.4/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     resp = await request_handler(request)
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/home/homeassistant/.pyenv/versions/3.11.4/lib/python3.11/site-packages/aiohttp/web_app.py", line 504, in _handle
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     resp = await handler(request)
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:            ^^^^^^^^^^^^^^^^^^^^^^
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/home/homeassistant/.pyenv/versions/3.11.4/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 117, in impl
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     return await handler(request)
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:            ^^^^^^^^^^^^^^^^^^^^^^
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/home/homeassistant/.pyenv/versions/3.11.4/lib/python3.11/site-packages/homeassistant/components/http/security_filter.py", line 85, in security_filter_middleware
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     return await handler(request)
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:            ^^^^^^^^^^^^^^^^^^^^^^
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/home/homeassistant/.pyenv/versions/3.11.4/lib/python3.11/site-packages/homeassistant/components/http/forwarded.py", line 82, in forwarded_middleware
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     from hass_nabucasa import (  # pylint: disable=import-outside-toplevel
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/mnt/dietpi_userdata/homeassistant/deps/lib/python3.11/site-packages/hass_nabucasa/__init__.py", line 15, in <module>
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     from .auth import CloudError, CognitoAuth
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/mnt/dietpi_userdata/homeassistant/deps/lib/python3.11/site-packages/hass_nabucasa/auth.py", line 11, in <module>
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     import boto3
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/mnt/dietpi_userdata/homeassistant/deps/lib/python3.11/site-packages/boto3/__init__.py", line 17, in <module>
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     from boto3.session import Session
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/mnt/dietpi_userdata/homeassistant/deps/lib/python3.11/site-packages/boto3/session.py", line 17, in <module>
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     import botocore.session
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/mnt/dietpi_userdata/homeassistant/deps/lib/python3.11/site-packages/botocore/session.py", line 26, in <module>
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     import botocore.client
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/mnt/dietpi_userdata/homeassistant/deps/lib/python3.11/site-packages/botocore/client.py", line 15, in <module>
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     from botocore import waiter, xform_name
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/mnt/dietpi_userdata/homeassistant/deps/lib/python3.11/site-packages/botocore/waiter.py", line 18, in <module>
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     from botocore.docs.docstring import WaiterDocstring
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/mnt/dietpi_userdata/homeassistant/deps/lib/python3.11/site-packages/botocore/docs/__init__.py", line 15, in <module>
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     from botocore.docs.service import ServiceDocumenter
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/mnt/dietpi_userdata/homeassistant/deps/lib/python3.11/site-packages/botocore/docs/service.py", line 14, in <module>
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     from botocore.docs.client import ClientDocumenter, ClientExceptionsDocumenter
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/mnt/dietpi_userdata/homeassistant/deps/lib/python3.11/site-packages/botocore/docs/client.py", line 17, in <module>
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     from botocore.docs.example import ResponseExampleDocumenter
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/mnt/dietpi_userdata/homeassistant/deps/lib/python3.11/site-packages/botocore/docs/example.py", line 13, in <module>
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     from botocore.docs.shape import ShapeDocumenter
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/mnt/dietpi_userdata/homeassistant/deps/lib/python3.11/site-packages/botocore/docs/shape.py", line 19, in <module>
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     from botocore.utils import is_json_value_header
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/mnt/dietpi_userdata/homeassistant/deps/lib/python3.11/site-packages/botocore/utils.py", line 37, in <module>
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     import botocore.httpsession
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/mnt/dietpi_userdata/homeassistant/deps/lib/python3.11/site-packages/botocore/httpsession.py", line 45, in <module>
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     from urllib3.contrib.pyopenssl import (
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:   File "/mnt/dietpi_userdata/homeassistant/deps/lib/python3.11/site-packages/urllib3/contrib/pyopenssl.py", line 97, in <module>
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     util.PROTOCOL_TLS: OpenSSL.SSL.SSLv23_METHOD,
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]:     ^^^^^^^^^^^^^^^^^
Jun 18 18:52:10 NanoPiM1 Home Assistant[19381]: AttributeError: module 'urllib3.util' has no attribute 'PROTOCOL_TLS'

Pretty strange is that this constant should exist until urllib3 v2.0.0, while HA/botocore pulls v1.26.x: Fix ssl deprecation warnings in Python 3.10 · urllib3/urllib3@2ec06d1 · GitHub

I couldn’t figure out whether the deprecation warning in Python 3.10, which lead to the urllib3 change, means an actual API removal in Python 3.11, so that urllib3 v1.26.x is simply not compatible with Python 3.11. But in this case, HA would be not compatible with Python 3.11 and the same error should appear on all architectures and distro versions. Confusing …

EDIT: Indeed same problem on x86. Testing on Debian Bookworm …

EDIT2: Same error on Bookworm. Manually upgrading urllib3 to latest version solves the issue and makes the web UI accessible as well on NanoPi M1. But on the paper it introduces a dependency conflict with botocore.

EDIT3: Same issue with Python 3.11.3. I wonder how this ever worked for anyone? Is it probably due to a very recent dependency chance of one of HA’s dependency, so that a somehow indirectly related module has now an incompatible version without any change on HA?

Not sure how to deal with it: With Python 3.10 the big warning appears, with Python 3.11 the web UI fails completely, all on every architecture and all Debian versions. Giving up for today and see whether bug reports pop up at HA until tomorrow, else will do one myself.

MichaIng · 19 June 2023 13:47

Okay, the util:PROTOCOL_TLS error now solves itself after a service restart.

For a more recent FFmpeg, the Deb Multimedia repository could be used: Deb Multimedia Packages::ffmpeg

It provides more recent Kodi versions as well, not Kodi 20 on Bullseye, but at least 19.5 vs 19.1. It is well known, maintained and provides packages for all Debian versions, including backports, e.g. Trixie already as well. It won’t support ARMv6 RPi models, but on RPi it is dangerous anyway: E.g. Kodi 19.5 means that the RPi accelerated Kodi 19.4 from RPi repo won’t be installed, worsening performance. We could try to solve it with APT pinning, but since the repo ships >700 packages with tons of dependencies/libraries, this would be extremely hard to maintain, apocalypse programmed.

Means: We would only add it for everything else but RPi. Then, given that most DietPi systems are RPi systems, I am not sure whether this is worth the hassle. Probably better to just disable HA for Buster and Bullseye and have users upgrading to Bookworm with our script, where needed.

innovodev · 19 June 2023 15:04

Is there a way to build ffmpeg and use the build on a different system? or must it be built on every device? I am not familiar with which files get installed after build or where they go.

MichaIng · 22 June 2023 21:10

Theoretically yes. But it is much easier to build it on the same system:

This assures that library versions the build was linked against, match. However, this alone mean you at least can run builds on any other system if the Debian version matches.
The CPU architecture needs to match: You can run builds done on one ARMv8 system, like an RPi 4, on other ARMv8 system, like a ROCK 5. But if you do builds on an x86 system, it won’t work, unless you install a special cross-compiling build stack, which makes it more complicated.
When it is about hardware acceleration, there are build flags, sometimes selected automatically, to use the GPU of the host system correctly. On RPi, the kernel/firmware package repository even ships own FFmpeg + development header packages which enable hardware acceleration on RPi. So compiling on another system with standard Debian FFmpeg sources and/or without explicitly setting build flags for RPi, it will most likely not utilise the GPU on RPi that well.

innovodev · 23 June 2023 01:20

I was actually referring to building on device and transferring to another device with the same exact hardware and system image (less the new ffmpeg). If I were to do that, which files would I copy and to where?

MichaIng · 24 June 2023 17:03

Yes that should work. Check the build/install logs for related files. Those are usually installed into /usr/local/ bin, lib and share subdirectories for binaries, shared objects/libraries and data/docs.

astutejoe · 25 June 2023 08:43

Got the exact same issue on the PROTOCOL_TLS after updating to 3.11 due to the deprecation warning

astutejoe · 25 June 2023 08:44

But I managed my own installation in terms of dependency management (using pipenv) and Python version, so I issued:
pipenv install pyopenssl
And it works well now

astutejoe · 25 June 2023 08:44

That’s what my Pipfile looks like:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
wheel = "*"
av = "*"
homeassistant = "*"
sqlalchemy = "*"
fnvhash = "*"
pyopenssl = "*"

[dev-packages]

[requires]
python_version = "3.11"

hrfried · 25 June 2023 13:34

So I just wanted to add my experience here. BTW–I’m new to the forums but have been using DietPi for years now. Love it and recommend it to anyone who will listen, and hoping to contribute more in the future.

So I modified the script in /boot/dietpi/dietpi-software to use 11.3.4 and at first kept running into problems. What I ended up figuring out was that my ufw configuration was blocking access to port 8123 natively and allowing it through and restarting home-assistant.service made the broken web server work fine.

sudo ufw allow 8123
sudo systemctl restart home-assistant.service

This may not be relevant to all use cases, but some kind of notification to update ufw values or automatic ufw handling could help others avoid the same problem.

I honestly forget whether or not I installed ufw directly or if it was a dependency of something else…I run a rustdesk signal/relay server on this RPi 4 and also host a Wireguard VPN–with a DuckDNS config pointing to the device. I’ve also played around with and deleted a bunch of crap on this pi, and was in the middle of reconfiguring a bunch of stuff with my home network so was expecting some weirdness. I kinda used this pi as a staging area for testing things like nextcloud/syncthing/node-red over the last year or so, but have deleted most of those since and moved portions to a dedicated server–may have been lingering configs laying around though. I also updated to Bookworm yesterday and moved the device from being on wifi-only to dual wifi/direct router connection. Also run a pihole upstream (my first DietPi install ) and a nextcloud instance on ubuntu server 22.04.

Just throwing all this out there in case it’s useful to anyone!

whyisthisbroken · 26 June 2023 16:54

@MichaIng @Joulinar
The new update to DietPi 8.19 runs perfectly and the Home Assistant Upgrade to pyenv 3.11 also.
No Error - no fails… everything is fine.
After the update to 8.19 i reinstall HA and everything was done after 15minutes…

Thank you

xperiazn · 3 July 2023 12:04

Hi,

How did you reinstall home assistant.
When i try to run reinstall i got following error.

dietpi-software reinstall 157
 
DietPi-Software
─────────────────────────────────────────────────────
 Mode: Automated reinstall

[FAILED] DietPi-Software | Software title (Home Assistant) is not supported on Debian Bullseye.

I installed newest update before.
Hardware is rpi 2
Raspberry Pi 2 Model B Rev 1.1

Kind regards

Jappe · 3 July 2023 12:12

You would need to upgrade to bookworm first:
https://dietpi.com/blog/?p=2809

xperiazn · 3 July 2023 13:15

Thank you,

Upgraded to bookworm.
Tried to reinstall Homeassistant but now i have another error.

 libatlas-base-dev is already the newest version (3.10.3-13).                                                           
 │ pkg-config is already the newest version (1.8.1-1).                                                                    
 │ pkg-config set to manually installed.                                                                                  
 │ g++ is already the newest version (4:12.2.0-3).                                                                        
 │ Some packages could not be installed. This may mean that you have                                                      
 │ requested an impossible situation or if you are using the unstable                                                     
 │ distribution that some required packages have not yet been created                                                     
 │ or been moved out of Incoming.                                                                                         
 │ The following information may help to resolve the situation:                                                           
 │                                                                                                                        
 │ The following packages have unmet dependencies:                                                                        
 │  libavcodec-dev : Depends: libavcodec58 (= 8:4.3.6-0+deb11u1+rpt2)                                                     
 │  libavdevice58 : Depends: libavcodec58 (= 8:4.3.6-0+deb11u1+rpt2)                                                      
 │                  Depends: libavfilter7 (= 8:4.3.6-0+deb11u1+rpt2)                                                      
 │  libavfilter-dev : Depends: libavfilter7 (= 8:4.3.6-0+deb11u1+rpt2)                                                    
 │  libavformat58 : Depends: libavcodec58 (= 8:4.3.6-0+deb11u1+rpt2)                                                      
 │                  Depends: libsrt1.4-gnutls (>= 1.4.2) but it is not installable                                        
 │ E: Unable to correct problems, you have held broken packages.

Joulinar · 3 July 2023 13:34

yes, known for Raspberry Pi device. Same as this one. The following packages have been kept back: ffmpeg - #8 by Joulinar

sew3521 · 6 July 2023 16:11

I was running into the same issues and pulling my hair out. I feel stupid that it took me stumbling upon your post to consider just restarting the service.
For future people running into issues be smarter than I am and try restarting the service using the below command

systemctl restart home-assistant