Synapse sudden error

Troubleshooting
1

So… my Synapse server was going great then suddenly it give me this message

root - 348 - WARNING - main - ***** STARTING SERVER *****
root - 349 - WARNING - main - Server /usr/local/lib/python3.9/dist-packages/synapse/app/homeserver.py version 1.53.0
synapse.app - 49 - WARNING - sentinel - Failed to listen on 0.0.0.0, continuing because listening on [::]

I checked the other ports and everything is fine in theory (I use nginx as a reverse proxy), even after the update the problem persist.

Of course I point out that I didn’t changed a single config line and after a check everything looks ok…

At this state I can access port 8008 (and gives me “Synapse is running” page) but :443 give me “forbidden” error.

Maybe is something whit some python package version? But if is that why the problem came out even before the update? (Of course this last one is more of a rhetorical angry question).

2

I guess the warning should be ok as long as the system is LISTEN on 8008

ss -tulpn | grep LISTEN

the forbidden points more into direction of issues with the nginx proxy.

3

Well, I runned the command and this is the output:

tcp   LISTEN 0      511          0.0.0.0:80        0.0.0.0:*    users:(("nginx",pid=18503,fd=5),("nginx",pid=18502,fd=5),("nginx",pid=18501,fd=5),("nginx",pid=18500,fd=5),("nginx",pid=18499,fd=5))

tcp   LISTEN 0      1000         0.0.0.0:22        0.0.0.0:*    users:(("dropbear",pid=293,fd=4)) 
                                                                                                  
tcp   LISTEN 0      511          0.0.0.0:443       0.0.0.0:*    users:(("nginx",pid=18503,fd=7),("nginx",pid=18502,fd=7),("nginx",pid=18501,fd=7),("nginx",pid=18500,fd=7),("nginx",pid=18499,fd=7))

tcp   LISTEN 0      511          0.0.0.0:8448      0.0.0.0:*    users:(("nginx",pid=18503,fd=9),("nginx",pid=18502,fd=9),("nginx",pid=18501,fd=9),("nginx",pid=18500,fd=9),("nginx",pid=18499,fd=9))

tcp   LISTEN 0      50                 *:8008            *:*    users:(("python3",pid=25283,fd=12))                                                                                                 

tcp   LISTEN 0      511             [::]:80           [::]:*    users:(("nginx",pid=18503,fd=6),("nginx",pid=18502,fd=6),("nginx",pid=18501,fd=6),("nginx",pid=18500,fd=6),("nginx",pid=18499,fd=6))

tcp   LISTEN 0      1000            [::]:22           [::]:*    users:(("dropbear",pid=293,fd=5))                                                                                                   

tcp   LISTEN 0      511             [::]:443          [::]:*    users:(("nginx",pid=18503,fd=8),("nginx",pid=18502,fd=8),("nginx",pid=18501,fd=8),("nginx",pid=18500,fd=8),("nginx",pid=18499,fd=8))

tcp   LISTEN 0      511             [::]:8448         [::]:*    users:(("nginx",pid=18503,fd=10),("nginx",pid=18502,fd=10),("nginx",pid=18501,fd=10),("nginx",pid=18500,fd=10),("nginx",pid=18499,fd=10))

Is there something wrong? I can’t spot the error

4

no, seems to be fine. Still it seems issue is on your Nginx proxy config (probably)

5

Sorry to be so lazy but since the config file (and everything else) isn’t changed I can’t find the issue.

Nginx reverse proxy config file (it worked until a few days ago):

server {

        server_name DOMAIN;

        listen 80;
        listen [::]:80;

        listen 443 ssl http2 default_server;
        listen [::]:443 ssl http2 default_server;
       
        listen 8448 ssl http2 default_server;
        listen [::]:8448 ssl http2 default_server; 

        location ~* ^(\/_matrix|\/_synapse\/client) {
        proxy_pass http://localhost:8008;
        proxy_set_header X-Forwarded-For $remote_addr;
        client_max_body_size 50M;
        }

        location /.well-known/matrix/client {
        return 200 '{"m.homeserver": {"base_url": "https://DOMAIN"}}';
        default_type application/json;
        add_header Access-Control-Allow-Origin *;
        }

        location /.well-known/matrix/server {
        return 200 '{"m.server": "DOMAIN:443"}';
        default_type application/json;
        add_header Access-Control-Allow-Origin *;
        }
    
    #listen [::]:443 ssl ipv6only=on; # managed by Certbot
    #listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/DOMAIN/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/DOMAIN/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

(Of course DOMAIN is replaced with the actual domain in the running config).
Thanks in advice.

6

I tested your Nginx config and for me it’s working fine. But need to say I copied location section only as rest deviate from the one created by dietpi-letsencrypt. But this should not matter that much

What does the following give your https://your.domain.com/_matrix/static/

Does it matter if you connect from inside or outside your local network?

7 
https://your.domain.com/_matrix/static/

give me 404.

Instead

https://your.domain.com/

give me 403.

8

is there anything else running on that Nginx web server? Any other web application? Or is it used as revers proxy only?

Receiving HTTP 403 might be ok if you configured the web server to block access to the root path. Same effect you would have if you remove the index.html from web server root dir. Btw: how did you install Nginx? Because your config file looks different, compare to a default DietPi setup.

https://your.domain.com/ will not forward any request to your Synapse. Usually it should display default web server welcome page.

The HTTP 404 is a little bit strange. It looks like proxy configuration is not taken into account as the request for sup path _matrix doesn’t get forward to Synapse at all. Because if the proxy forwarding would be OK and Synapse would be offline, you should get at least a 502 Bad Gateway.

Still it seems Nginx might be the issue. Btw, maybe a stupid question. The configuration you shared above, is enabled?

9

Yes, the configuration is enabled, yes it’s a custom config file took from a guide that i’ve tweaked a bit to suit my needs and yes, the nginx server is only used as a reverse proxy.

The very strange part of this story is that it worked fine for a week or so and then (without ANY modification of ANY type (and yes, this means no update to the raspberry and no changes even to the LAN he is in)) stopped to listen on 0.0.0.0.

The warning message printed a homeserver.yaml file version in the python directory, maybe it is involved with this issue? (At this point i don’t know where to look)
Can I ask you to control your version of that file? Or even the python version.

Really thanks in advice.

10

can you share your default config file?

cat /etc/nginx/sites-enabled/default

And where did you stored the custom config file? Can you share name and path?

The message for listen on 0.0.0.0 I do have as well. But, still I guess it is not related.

Mar 10 14:16:17 DietPi3 Synapse[26928]: --------------------------------------------------------------------------------
Mar 10 14:16:18 DietPi3 Synapse[26928]: 2022-03-10 14:16:18,077 - root - 352 - WARNING - main - ***** STARTING SERVER *****
Mar 10 14:16:18 DietPi3 Synapse[26928]: 2022-03-10 14:16:18,078 - root - 353 - WARNING - main - Server /usr/local/lib/python3.9/dist-packages/synapse/app/homeserver.py version 1.54.0
Mar 10 14:16:18 DietPi3 Synapse[26928]: 2022-03-10 14:16:18,513 - synapse.app - 49 - WARNING - sentinel - Failed to listen on 0.0.0.0, continuing because listening on [::]
Mar 10 14:16:18 DietPi3 systemd[1]: Started Synapse (DietPi).

However you could get rid of this message by changing Synapse to LISTEN on localhost only.

G_CONFIG_INJECT '    bind_addresses:' '    bind_addresses: ['\''127.0.0.1'\'']' /mnt/dietpi_userdata/synapse/homeserver.yaml
systemctl restart synapse.service
journalctl -u synapse

But this might not change anything for you (probably)

11

Hi, sorry but I wasn’t home those days, for what matter I never touched the default config of nginx and my personal config is another file in /etc/nginx/sites-available/ (same place as default config) and of course is linked with /etc/nginx/sites-enabled/

I hope to be able to provide the other information later today.

12

Take your time. :slight_smile: