Does it help if you disable the Ethernet Energy Efficiency mode?
sudo ethtool --set-eee eth0 eee off
Respectively doing this automatically once the interface is brought up:
sudo sed -i '/iface eth0/a\post-up /sbin/ethtool --set-eee eth0 eee off' /etc/network/interfaces
Btw, having WiFi and Ethernet enabled on just the same network can cause issues. Does the issue happen as well when you disable WiFi? The way network is currently configured, it is more an either or, actually like it is common with Laptops etc. If you need WiFi as fallback for Ethernet or the other way round, better to use ifplugd for an automatic switch. We have a guide for this somewhere here on the forum.