Hmm, the pixel clock rates logged by the old kernel do not match 5k@60Hz at all, and it does say don't use dsc mode as well. How is this possible? Is there another another way to force higher resolution and frequency through limited bandwidth?
Please explain 
.
RC or not is not related. Linux does not enable debug flags on master branch or so, and RC’s are in any case potentially ready for release, just going through more testing and bug fix time. So I guess mainline Linux does just not log mode changes, respectively the mainline Rockchip HDMI driver does not. And it seems to be stricter regarding max pixel clock rates, or does not support whichever magic the vendor kernel does, to force higher resolution/frequency through lower bandwidth.
The HDMI audio codec errors are dumb indeed, respectively that it seems to retry reading that info from the monitor over and over. Funnily, the same error was reported for RPi with Kodi in 2022:
Does HDMI audio finally work despite those errors?