Orange Pi 5 Plus: NPU Driver Support for Docker Projects (e.g., rkllama)?

General Discussion
1

Hello everyone,

I’m currently trying to get the NPU on my Orange Pi 5 Plus working to run various AI-related Docker projects, with a specific focus on getting the rkllama project up and running. I’m hoping to get some guidance or share experiences with anyone who might have attempted something similar.

My Hardware:

  • Device: Orange Pi 5 Plus (32GB RAM)

  • Storage: NVMe SSD

  • OS: DietPi v9.15.2

My Goal:

My plan is to use DietPi as the base OS, install OpenMediaVault (OMV) on top of it, and then use Docker to run several different containers. A primary goal is to leverage the NPU for performance in projects like rkllama.

What I’ve Tried So Far:

Unfortunately, I’ve already had a few unsuccessful attempts. I’ve tried this with the official Orange Pi image and also with Armbian, but in every case, I’ve failed to get access to the NPU driver from within the operating system or the Docker container.

I’m relatively new to the world of single-board computers with NPUs, so I’m still learning. My current setup is a Raspberry Pi 4 running OMV with a few Docker containers, and I’m looking to migrate and expand on my new Orange Pi.

Has anyone had success with accessing the NPU on the Orange Pi 5 Plus with a recent DietPi build? I would be very grateful for any help, tips, or pointers on how to get the driver recognized and passed through to Docker containers.

Thanks in advance for your help!

1 Like
2

Hi,
Just installed v9.20.1 on OPi 5 Max and can’t get the NPU to work either, no kernel module present and it seems to be as if disabled by firmware… The vendor repo - RKlinux do have the driver, yet it doesn’t seem to be in the kernel (or initrd) image, or maybe disabled somehow?

3

The kernel module should be rknpu and shipped with the kernel.

Mainline Linux 6.18 (edge branch) ships a mainline Rocket NPU driver for the first time.

Both drivers provide an interface on a /dev/dri/render* device node.

2 Likes
4

The 6.18.0-edge-rockchip64 (from mainline image) identifies the NPU - loads the hantro_vpu module, but doesn’t recognize wireless wlan on OPi 5 Max…

5

What’s the chip on this board? If it’s aic8800, you need a dkms kernel module.
Also, what does hantro vpu have to do with the NPU?

6

On the Trixie image with 6.1.115-vendor-rk35xx, the rknpu driver seems to fail initializing:

[   19.651699] RKNPU fdab0000.npu: Adding to iommu group 0
[   19.653969] RKNPU fdab0000.npu: RKNPU: rknpu iommu is enabled, using iommu mode
[   19.656222] RKNPU fdab0000.npu: Looking up rknpu-supply from device tree
[   19.662346] RKNPU fdab0000.npu: Looking up mem-supply from device tree
[   19.667447] RKNPU fdab0000.npu: can’t request region for resource [mem 0xfdab0000-0xfdabffff]
[   19.669631] RKNPU fdab0000.npu: can’t request region for resource [mem 0xfdac0000-0xfdacffff]
[   19.671541] RKNPU fdab0000.npu: can’t request region for resource [mem 0xfdad0000-0xfdadffff]
[   19.672232] [drm] Initialized rknpu 0.9.8 20240828 for fdab0000.npu on minor 1
[   19.678986] RKNPU fdab0000.npu: Looking up rknpu-supply from device tree
[   19.685198] RKNPU fdab0000.npu: RKNPU: bin=0
[   19.686447] RKNPU fdab0000.npu: leakage=9
[   19.687499] RKNPU fdab0000.npu: Looking up rknpu-supply from device tree
[   19.687514] debugfs: Directory ‘fdab0000.npu-rknpu’ with parent ‘vdd_npu_s0’ already present!
[   19.695969] RKNPU fdab0000.npu: pvtm=916
[   19.701432] RKNPU fdab0000.npu: pvtm-volt-sel=5
[   19.702531] RKNPU fdab0000.npu: Looking up rknpu-supply from device tree
[   19.702549] debugfs: Directory ‘fdab0000.npu-rknpu’ with parent ‘vdd_npu_s0’ already present!
[   19.704149] RKNPU fdab0000.npu: Looking up mem-supply from device tree
[   19.705525] RKNPU fdab0000.npu: avs=0
[   19.706653] RKNPU fdab0000.npu: rockchip_pvtpll_set_volt_sel: error cfg clk_id=6 voltsel (-1)
[   19.707822] RKNPU fdab0000.npu: l=15000 h=85000 hyst=5000 l_limit=0 h_limit=800000000 h_table=0

The /dev/render* doesn’t appear…

7

No, we patched this driver into our mainline Linux builds, no DKMS needed: rockchip64: add AIC8800 SDIO driver · MichaIng/build@79457dc · GitHub
But OPi 5 Max/Ultra use a different one anyway.

Yes that is sadly expected. The Max and Ultra use this Rockchip WiFi chip with bcmdhd driver which does not exist in mainline Linux:

TL;DR: Does modprobe brcmfmac actually work instead?
While the driver is located below some Rockchip vendor subdir, and the chip is labelled AP6611, it is a Broadcom chip (indicated by the module name, but also clearly stated in source files). There is some info that the bcmdhd driver is actually an Android Broadcom WiFi driver, while Linux ships its brcmfmac. That wouldn’t be so surprising, since Rockchip had shared Android and Linux sources earlier, one of the issues with earlier Rockchip vendor kernels throwing a bunch of additional errors and warnings since Android-only flags were enabled. I gave them a hint a while ago related to some particular prominent multi-line warning caused by a flag that really made sense on Android only, and they separated sources with 5.10 IIRC. But I would not wonder if some (originally) Android drivers remained in the sources.

Let’s see, the two supplies it looks up are defined here:

This is the same for all RK3588 boards (with vendor kernel). The nodes are defined here:

There we have the regions it can’t request. And then it seems to retry 2 times, causing the “already present” errors from debugfs.

No idea, neither the driver nor those parts in the device trees (which are all the same across all boards) have seen any changed within 2 years. Do you have any overlays active? Also maybe try with an older kernel builds:

apt policy linux-image-vendor-rk35xx

Don’t forget to downgrade the linux-dtb-vendor-rk35xx package as well. Reminds me that I want to merge those two packages.

8

It may be a stupid question but I thought Teflon only works with the mainline driver (rocket) and not rknpu?

9

Didn’t use overlays (except the ones present in the images, if any). The board appears to be of “rev 1.1”, no idea if this may be the reason, but doubtful. The npu is initialized perfectly in the mainline - 6.18.0-edge-rockchip64:
[ 0.997022] platform fdab0000.npu: Adding to iommu group 0
[ 0.998374] platform fdac0000.npu: Adding to iommu group 1
[ 0.999665] platform fdad0000.npu: Adding to iommu group 2
[ 10.949953] rocket fdab0000.npu: Rockchip NPU core 0 version: 1179210309
[ 10.950294] rocket fdac0000.npu: Rockchip NPU core 1 version: 1179210309
[ 10.950584] rocket fdad0000.npu: Rockchip NPU core 2 version: 1179210309

My bad about the vpu and hantro module mixup, it was a bad grep output very similar to npu initialization… Trying the modprobe brcmfmac on the edge has yielded only this:

[ 512.582827] usbcore: registered new interface driver brcmfmac

wlan didn’t turn up:(

Please, I beg of you to merge these two! :wink: Silly me tried to switch kernels using apt install, rendering the setup unbootable. Due to that, I must add that armbian’s Kernel menu is awesome, makes switching pretty automated.

P.S.

After witnessing armbian’s setup, I must extend deepest gratitude and respect to Michalng and the Team for all the efforts! Und a Happy New Year to everyone!

P.P.S

  • Not a bot :blush:
1 Like
10

The TensorFlow Lite delegate Teflon also works as a liteRT backend for etnaviv
(VeriSilicon VIPNano NPU), which has long been upstream, as well as for the upcoming ethosu (Arm Ethos-U65/U85 NPU).

11

OK, I’m talking about RK3588 only here. The vendor driver is not Rocket.

12

Newer sources for the WiFi driver: GitHub - StreamUnlimited/broadcom-bcmdhd-4359: GPL Broadcom driver for bcm4359 / AP6398S

Though it might not contain all needed chip IDs. Always a problem with scattered sources patched independently, everyone adds own chip IDs and firmware paths as needed.

CoreELEC has also sources, but the base is a little older: GitHub - CoreELEC/ap6xxx-aml: WiFi Broadcom drivers ap6xxx

1 Like
13

Hi,

Happy to report that NPU (and Bluetooth) work out of the box on Opi5Max 6.19.3-edge-rockchip64 (mainline) - DietPi 10.1.2 :clap:
Not happy to report that for wifi - building either of those drivers require broadcom-sta-* packages, no longer available for trixie or in backports, for some reason the ‘all’ arch build is not released any more. They’re there for Bookworm, but I prefer to stay on trixie edge since Collabora guys are making progress on the hardware features in rockchip.

There are some hacky packages for that sta driver I’ve seen around for trixie arm64 or ‘all’, but those dive too deep into the un land - unstable, unrepoducible, unupdateable, etc… so, no wifi for now :smiling_face_with_tear:

Thanks for the update and all the work on the releases!

14

I see, they are still there, but x86-only: Debian -- Package Search Results -- broadcom-sta

So this Broadcom STA Wireless driver is the bcmdhd one, or a compatible one?

It contains a ~7 MiB wlc_hybrid.o_<arch> blob. Also the all package on Bookworm contains this blob only for amd64 and i386, so it likely never worked on ARM, even that it was installable. Since it is a blob, Debian cannot just build it for other architectures. So it is on Broadcom.

The wireless-tools package is also required in order to make use of these modules.

Is this meant literally, i.e. they do not support the nl80211 interface? We just got rid of wireless-tools with last release, with NanoPi Fire3 Linux 4.14 being the last SBC I know with onboard WiFi that does not support that standard :roll_eyes:. Anyway, not relevant as it cannot work on ARM.

This is not a contradiction: you can perfectly run edge Linux 6.19-rcX on Debian Bookworm. Would be interesting whether this Broadcom STA driver really works there, without a matching blob. If so, would be trivial to alter the package to make it installable on ARM.

15

Since @makinikm is using mainline kernel, should’t it be possible to use brcmfmac driver? (https://linux-sunxi.org/Wifi#Ampak_Devices)

I have also a question, here is the BCM4359 listed as PCIe device, but the AP6611 in the OPi 5 Max is using SDIO?
So I question myself, what is the AP6611 BT/Wifi module?
What is it’s hw ID? lspci -nn | grep -i broadcom

16

Sadly not. brcmfmac is also present in the vendor kernel, but it does not support a number of these AP6* chips. The bcmdhd driver really is a different one, and further developed + patched to support WiFi chips used in ARM devices with Android OS, which implies what many SBCs were originally shipped with. The Rockchip vendor kernel sources originally were used for both, Android and plain Linux, until 5.15 IIRC. When I recognized a bunch of nonsense Android-only settings in the defconf, I talked to them that this is causing a bunch of errors in the kernel log on (non-Android) Linux, after which they split it (or said so). But a bunch of custom drivers from Android remained, and were never mainlined, or support for the additional chips merged into brcmfmac in this case. It is a mess …

Yes, another thing annoying about this driver: it can be built only with one of either USB, or PCIe, or SDIO support, but not combining them. This is the reason we had a dedicated vendor kernel build only for Orange Pi 5 Max and Ultra, switching from PCIe to SDIO. But recently I reconsidered that it is very unlikely that someone uses an M.2 module with this AP6* chip that would require PCIe support, or a USB dongle. Only Orange Pi seems to use theses chips, like only Orange Pi uses these likewise problematic UNISOC/Speadtrum chips that (their driver) is not in mainline Linux either, and cause so much issues. Hence I switched that to SDIO support for all vendor kernel builds: rk35xx-vendor: enable bcmdhd SDIO support · MichaIng/build@5fda26c · GitHub
This really caused a lot of extra work previously.