Lighttpd still binding to port 80 despite a custom port being defined

Salty_Wagyu · February 5, 2023, 12:06pm

DietPi version | 8.13.2
Distro version | bullseye
Kernel version | Linux DietPi-proxmox 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 GNU/Linux
SBC model | VM running in Proxmox

Additional Information (if applicable)

Software title | Lighttpd
Was the software title installed freshly or updated/migrated?
Freshly installed to new system

I’ve always edited /etc/lighttpd/lighttpd.conf and changed the port from 80 to 81 (against advice not to), so that Nginx Proxy Manager can use port 80, due to not having much luck in having a custom port defined in external.conf to work for me in the past.

Now I wanted to try again and do it properly on this new installation and define the custom port via the new changes in pi-hole release notes here: Pi-hole FTL v5.20.1, Web v5.18.1 and Core v5.15 released – Pi-hole which tell to use /etc/lighttpd/conf-enabled instead. Also see: Pihole update changes lighttpd.conf · Issue #5127 · pi-hole/pi-hole · GitHub

Steps to reproduce

Changed server.port back to 80 in /etc/lighttpd/lighttpd.conf
Create custom-port.conf in /etc/lighttpd/conf-enabled/

server.port := 81

sudo systemctl restart lighttpd

Expected behaviour

http://pi.hole:81/admin/ expected to work
NPM to bind to port 80

Actual behaviour

After this change, http://pi.hole:81/admin/ loads fine, but Nginx Proxy Manager (docker) fails to start:

 +] Running 0/1
 ⠿ Container nginx-pm  Starting                                                                                                                  0.2s
Error response from daemon: driver failed programming external connectivity on endpoint nginx-pm (b009ee6921be2e12e46a4387a1b46881da9728aa077e038e5a545d578faa545d): Error starting userland proxy: listen tcp6 [::]:80: bind: address already in use

ss -tulpn shows:
tcp   LISTEN 0      1024            [::]:80            [::]:*

This bind disappears when I stop Lighttpd. Lighttpd seems to be using both port 80 and 81 - I’m not sure this is normal behaviour?

Joulinar · February 5, 2023, 12:36pm

Simply adjust /etc/lighttpd/lighttpd.conf and change server.port to whatever you like. DietPi is not using the orriginal PiHole web server configuration as we need to ensure web server flexibility to be able to run other web apps as well. Which is not possible with original PiHole conf. That’s how my system is running for long now. I moved lighttpd to port 8080 and have NPM running on 80, 81 and 443. Yes usually NPM require port 81 as admin interface port.

root@DietPiProd:~# ss -tulpn | grep LISTEN
tcp   LISTEN 0      5          127.0.0.1:4711       0.0.0.0:*    users:(("pihole-FTL",pid=491,fd=14))
tcp   LISTEN 0      1024         0.0.0.0:8080       0.0.0.0:*    users:(("lighttpd",pid=10253,fd=4))
tcp   LISTEN 0      4096         0.0.0.0:80         0.0.0.0:*    users:(("docker-proxy",pid=904,fd=4))
tcp   LISTEN 0      4096         0.0.0.0:81         0.0.0.0:*    users:(("docker-proxy",pid=885,fd=4))
tcp   LISTEN 0      32           0.0.0.0:53         0.0.0.0:*    users:(("pihole-FTL",pid=491,fd=9))
tcp   LISTEN 0      1000         0.0.0.0:22         0.0.0.0:*    users:(("dropbear",pid=327,fd=4))
tcp   LISTEN 0      4096         0.0.0.0:443        0.0.0.0:*    users:(("docker-proxy",pid=865,fd=4))
tcp   LISTEN 0      5              [::1]:4711          [::]:*    users:(("pihole-FTL",pid=491,fd=19))
tcp   LISTEN 0      1024            [::]:8080          [::]:*    users:(("lighttpd",pid=10253,fd=5))
tcp   LISTEN 0      4096            [::]:80            [::]:*    users:(("docker-proxy",pid=911,fd=4))
tcp   LISTEN 0      4096            [::]:81            [::]:*    users:(("docker-proxy",pid=891,fd=4))
tcp   LISTEN 0      32              [::]:53            [::]:*    users:(("pihole-FTL",pid=491,fd=11))
tcp   LISTEN 0      1000            [::]:22            [::]:*    users:(("dropbear",pid=327,fd=5))
tcp   LISTEN 0      4096            [::]:443           [::]:*    users:(("docker-proxy",pid=872,fd=4))
root@DietPiProd:~#

root@DietPiProd:~# docker container ls
CONTAINER ID   IMAGE                             COMMAND   CREATED        STATUS      PORTS                                                                                  NAMES
1983c6e6e54a   jc21/nginx-proxy-manager:latest   "/init"   9 months ago   Up 2 days   0.0.0.0:80-81->80-81/tcp, :::80-81->80-81/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   npm
root@DietPiProd:~#

Creating a custom conf using server.port := 81 will create an additional port only.

Salty_Wagyu · February 5, 2023, 12:59pm

Ah ok, thanks for the explanation. I will undo everything and go back to changing my port in lighttpd.conf directly in that case

holocron · March 7, 2023, 2:24am

I tried this setup on a current install: Base dietpi with pihole/unbound runnging. Added docker and portainter. Though portainer I set up NPM on a macvlan with it’s own local IP address. I forwarded 80/443 to that IP address of the NPM container. I can access it at IP:81, but any attempts to reverse proxy to local assets fail. Usually with a 504 error.

Searching this out, I found these threads. I changed the lighttpd server port to something other than 80 (888, 8080, etc) and restarted. Still not working. The redirects still fail and I cannot access Pi-Hole.

I would have assumed that the macvlan network set-up on the container would have avoided this, but it looks like lighttpd is still hijacking port 80.

Just to be clear:

Pihole/Unbound installed as native apps
NPM installed as container via Docker and on maclvan

Joulinar · March 7, 2023, 7:55am

A port can be used only once on a system. It’s not possible to have to apps binding to same port twice. Usually, if web server is still LISTEN to port 80, you should not be able to start NPM at all. Next, remove HTTPS configuration from web server. This is not needed anymore as it will be done by NPM. Pls can you reboot your system and share following

ss -tulpn | grep LISTEN
journalctl -u lighttpd
journalctl -u docker

holocron · March 7, 2023, 11:46am

Not sure if this was intended for me, but I’ll start with this before I post the full output:

Mar 06 21:09:57 HoloPi3B systemd[1]: Started Docker Application Container Engine.
Mar 06 21:10:03 HoloPi3B dockerd[494]: time="2023-03-06T21:10:03.700986549-05:00" level=warning msg="macvlan driver does not support port mappings"
Mar 06 21:10:03 HoloPi3B dockerd[494]: time="2023-03-06T21:10:03.701171497-05:00" level=warning msg="macvlan driver does not support port exposures"

could this even be the start of the problem?

Here’s the full output: lighttpd-docker output - Pastebin.com

Joulinar · March 7, 2023, 12:34pm

probably. I never used macvlan as it is not needed. My prod system is running fine with default Docker settings having lighttpd as well as NPM running in prallallel.

On your system, I see lighttpd LISTEN to port 8080 as it should be.

tcp   LISTEN 0      1024         0.0.0.0:8080       0.0.0.0:*    users:(("lighttpd",pid=489,fd=4))

But NPM doesn’t seems to be starting correctly. I see quite some messages regarding DNS failure.

Mar 06 21:10:46 HoloPi3B dockerd[494]: time="2023-03-06T21:10:46.758433912-05:00" level=warning msg="[resolver] failed to read from DNS server: 10.10.0.5:53, query: ;ip-ranges.amazonaws.com.\tIN\t A" error="read udp 10.10.1.50:35580->10.10.0.5:53: i/o timeout"
Mar 06 21:10:47 HoloPi3B xxxxxxxxxxxx[494]: [3/7/2023] [2:10:47 AM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v4
Mar 06 21:10:51 HoloPi3B dockerd[494]: time="2023-03-06T21:10:51.084129223-05:00" level=warning msg="[resolver] failed to read from DNS server: 10.10.0.5:53, query: ;www.cloudflare.com.\tIN\t A" error="read udp 10.10.1.50:38016->10.10.0.5:53: i/o timeout"
Mar 06 21:10:51 HoloPi3B xxxxxxxxxxxx[494]: [3/7/2023] [2:10:51 AM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v6
Mar 06 21:10:55 HoloPi3B dockerd[494]: time="2023-03-06T21:10:55.208739274-05:00" level=warning msg="[resolver] failed to read from DNS server: 10.10.0.5:53, query: ;www.cloudflare.com.\tIN\t A" error="read udp 10.10.1.50:34159->10.10.0.5:53: i/o timeout"

Maybe related to your macvlan settings. You should go back to default settings on Docker.

holocron · March 7, 2023, 1:27pm

I can try that. I did see that error as well:
error="read udp 10.10.1.50:35580->10.10.0.5:53

Ironically, that 10.10.0.5:53 is Pihole on another device–which is the Pihole I’m using currently.

Main reason I did the macvlan was to allow my router to port forward 80/443 to the NPM instance. But I imaging that changing the port for lighttpd to 8080 would allow me to forward 80/443 to the pi’s IP and allow NPM to then grab it.

Also, should NPM be running on it’s own bridge or on the host?

Joulinar · March 7, 2023, 2:13pm

personally, I don’t do any magic with it. I simply installed Docker from DietPi software catalogue and used following script to install/maintain NPM DietPi-Software | NGINX Proxy Manager (NPM) · Issue #4417 · MichaIng/DietPi · GitHub

holocron · March 7, 2023, 4:49pm

That’s basically just running the same that would run in a Portainer stack.

So I’m currently failing to see why it isn’t working. I can access it at 81. I can request an ssl cert via cloudflare. But it fails on the actual redirect.

holocron · March 7, 2023, 4:50pm

Alternately, Dietpi could be updated to properly support this functionality of Docker.

Joulinar · March 7, 2023, 5:03pm

It’s fully supported and possible without huge configuration. Personally it’s running fine on my prod system since months. Check that your port forwarding is correct and ports are bind to local system. Personally I would remove your custom mac address configuration.

holocron · March 7, 2023, 6:56pm

Sure, as I stated, I tried that and it didn’t function properly. Hence my question about if it should be it’s own bridge or host.

It’s great that your system is running fine. Mine does not.

If you’re willing and able to help with this, great.

Joulinar · March 7, 2023, 7:19pm

I shared all the steps I did to set it up. I did not configure any network or interface

Install Docker, PiHole and web service using dietpi-software
Move web service to port 8080
Setup NPM container running my script shared on GitHub
Done

There is just 1 single configuration step needed to switch web service port. No further confirmation needed.

holocron · March 7, 2023, 8:17pm

If it must be set up with your custom script, this should be officially documented somewhere.

If it cannot be set up via docker run, docker compose, or portainer, then the implementation of these are flawed.

Joulinar · March 7, 2023, 8:43pm

Did you ever had a look to my script? It does only

docker volume create npm_data
docker run -d -p '80:80' -p '443:443' -p '81:81' --name=npm --restart=always --env 'DB_SQLITE_FILE=/data/database.sqlite' -v 'npm_data:/data' -v 'npm_data:/etc/letsencrypt' 'jc21/nginx-proxy-manager'

Which is following the official guide of NPM. And there is no need to use my script at all. It’s an offer for everybody not that familiar with Docker and how thinks working.

The main part of the script is to offer an easy to execute way to update the image/container by implementing some checks and automation.

holocron · March 7, 2023, 9:14pm

Yes, and it’s virtually identical to the docker-compose/portainer-stack scripting I used–also from the official guide of NPM.

version: "3"
services:
  app:
    image: jc21/nginx-proxy-manager:latest
    restart: unless-stopped
    ports:
      # These ports are in format <host-port>:<container-port>
      - 80:80 # Public HTTP Port
      - 443:443 # Public HTTPS Port
      - 81:81 # Admin Web Port
      # Add any other Stream port you want to expose
      # - '21:21' # FTP
      
    # Uncomment the next line if you uncomment anything in the section
    environment:
      # Uncomment this if you want to change the location of 
      # the SQLite DB file within the container
      # DB_SQLITE_FILE: "/data/database.sqlite"

      # Uncomment this if IPv6 is not enabled on your host
      DISABLE_IPV6: true
    
    volumes:
      - /mnt/dietpi_userdata/docker/npm/data:/data
      - /mnt/dietpi_userdata/docker/npm/letsencrypt:/etc/letsencrypt

Joulinar · March 7, 2023, 10:32pm

Let’s stop the discussion at this point as we are moving away from the initial issue. Your challenge is related to Docker and the use of macvaln feature and not any port binding of Lighttpd. The Docker topic continue at GitHub Docker + Portainer: Macvlan support possibly flawed? · Issue #6223 · MichaIng/DietPi · GitHub

holocron · March 8, 2023, 1:50am

Agreed. To the lighttpd issue…

I have reverted the Docker installation of NPM to the base configuration utilizing the basic docker compose as documented on the NPM website. Here are the current port exposures on the DietPi host:

tcp   LISTEN 0      32           0.0.0.0:53         0.0.0.0:*    users:(("pihole-FTL",pid=395,fd=5))
tcp   LISTEN 0      1000         0.0.0.0:22         0.0.0.0:*    users:(("dropbear",pid=298,fd=4))
tcp   LISTEN 0      4096         0.0.0.0:443        0.0.0.0:*    users:(("docker-proxy",pid=931,fd=4))
tcp   LISTEN 0      5          127.0.0.1:4711       0.0.0.0:*    users:(("pihole-FTL",pid=395,fd=10))
tcp   LISTEN 0      4096         0.0.0.0:9001       0.0.0.0:*    users:(("docker-proxy",pid=977,fd=4))
tcp   LISTEN 0      4096         0.0.0.0:9002       0.0.0.0:*    users:(("docker-proxy",pid=949,fd=4))
tcp   LISTEN 0      4096         0.0.0.0:111        0.0.0.0:*    users:(("rpcbind",pid=283,fd=4),("systemd",pid=1,fd=33))
tcp   LISTEN 0      4096         0.0.0.0:80         0.0.0.0:*    users:(("docker-proxy",pid=984,fd=4))
tcp   LISTEN 0      4096   100.70.142.45:49616      0.0.0.0:*    users:(("tailscaled",pid=415,fd=26))
tcp   LISTEN 0      1024         0.0.0.0:8080       0.0.0.0:*    users:(("lighttpd",pid=478,fd=4))
tcp   LISTEN 0      4096         0.0.0.0:81         0.0.0.0:*    users:(("docker-proxy",pid=959,fd=4))
tcp   LISTEN 0      32              [::]:53            [::]:*    users:(("pihole-FTL",pid=395,fd=7))
tcp   LISTEN 0      1000            [::]:22            [::]:*    users:(("dropbear",pid=298,fd=5))
tcp   LISTEN 0      1024               *:5252             *:*    users:(("dietpi-dashboar",pid=376,fd=9))
tcp   LISTEN 0      4096            [::]:111           [::]:*    users:(("rpcbind",pid=283,fd=6),("systemd",pid=1,fd=35))
tcp   LISTEN 0      1024            [::]:8080          [::]:*    users:(("lighttpd",pid=478,fd=5))

I think this is the issue here, however, and it may not be related to lighttpd specifically.

Mar 07 19:58:53 Pi3B dockerd[483]: time="2023-03-07T19:58:53.825554804-05:00" level=warning msg="[resolver] failed to read from DNS server: 10.10.0.5:53, qu                                        ery: ;www.cloudflare.com.\tIN\t A" error="read udp 172.19.0.2:59389->10.10.0.5:53: i/o timeout"

10.10.0.5:53 is my Pihole instance running on a VM dietpi install. There’s no good reason why it should be accessible.

Regardless of the above, it appears that something during the dietpi-software uninstall process has rendered the particular device kaput. So I will have to reinstall dietpi fresh and start over. Thanks for your assistance and patience.