Oh, I didn’t know Ethernet shared the same bus 
Is Wifi on a different bus or it’s also shared with the rest?
WiFi is on a different bus. Btw the bus speed is higher than single USB port and Ethernet alone, so they do not fully limit each other, but still worth to give it a try.
But at the end, if I understood correctly, running out of memory is the issue and not the network speed.
That is true.
Thanks for all your help guys, I felt really good using a small chip for my media center and thought it was cool that it could just be attached on the side of the wall but I guess this is never going to work. I was thinking about getting a Banana Pi M5 as it’s stronger than even a RPi4, but then I saw DietPi doesn’t support BPi anymore so I’ll just use a big old bulky PC my father gave me and run the media server there instead.
Cheers all!
While we do not offer Banana Pi images, if there is a Debian image available for it, e.g. by Armbian, you could convert it into DietPi: https://dietpi.com/docs/hardware/#make-your-own-distribution