You need to disable the serial console on ttyAMA0 to make it usable for RaspBee. A serial console means there is a login getty waiting for interactive user login via attached serial terminal. In your case you want the port but no login console on it. This also explains why it has tty group permissions .
Also assure that Bluetooth is disabled since ttyAMA0 is by default used by Bluetooth on all onboard-WiFi RPi models.