Unable to access podman container on port 443 from internal network

Latest response

Howdy y'all:

I'm not really sure how to phrase this questions to Google to get useful information, so I'll start here.

I have a nginx container hosted via podman. I can access the http (port 80) site just fine on my network, but I'm unable to access the https side. Interestingly, you can access the secure site from the Internet, as I have it running via Cloudflare. I just can't get to it from my internal network. I'm not sure why. I've checked all the firewall ports are open, and they are, and I've make sure my Ubiquiti router isn't blocking ports, and it's not.

I can't make tails of this and any help is greatly appreciated.

Regards

Responses

Hey Wesley,

A few things might be happening here. I'll post what I can think of starting from the container on up.

First question is if you're running podman as root. I am assuming you are due to listening as port 80, but thought I'd check

Second, if you have shell access to the host, let's see if the port is being exposed properly. Let's see if podman can tell us what's being exposed:

# podman inspect $CONTAINER -f='{{.NetworkSettings.Ports}}{{.HostConfig.PortBindings}}'

This is almost assuredly working, since you can access it via CloudFlare, unless you've got a proxy in front of your podman container passing traffic to the local 80 port, doing SSL/TLS termination. Assuming that shows that 443 is known to podman as being exposed, let's make sure that the firewall has the right rules in place.

# iptables -t nat -nvL
- - - - 8< - - - -
Chain CNI-DN-0d266b581f0bbc810f6cc (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       10.88.0.2            0.0.0.0/0            tcp dpt:80
    0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       127.0.0.1            0.0.0.0/0            tcp dpt:80
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:10.88.0.2:80
    0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       10.88.0.2            0.0.0.0/0            tcp dpt:443
    0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       127.0.0.1            0.0.0.0/0            tcp dpt:443
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 to:10.88.0.2:443
- - - - 8< - - - -

We're looking for something showing the ports are being forwarded properly via the NAT; the above shows that my container (NAT'd ipv4 address of 10.88.0.2/16) should forward properly on 80 and 443.

Assuming that's fine, we can move on to the host, where we can verify connectivity locally; the conmon process holds the sockets open in the root network namespace, so let's verify they're present and listening with ss:

# ss -ltnp | egrep 'State|conmon'
State    Recv-Q    Send-Q        Local Address:Port        Peer Address:Port                                                                                    
LISTEN   0         128                 0.0.0.0:80               0.0.0.0:*        users:(("conmon",pid=2911,fd=5))                                               
LISTEN   0         128                 0.0.0.0:443              0.0.0.0:*        users:(("conmon",pid=2911,fd=6))

Assuming 443 is present there, we should be able to locally connect to it. If you have telnet or nc installed, we can do so like this:

# telnet localhost 443
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

If you get Connection refused then nginx in the container probably isn't listening properly. The next thing to do is probably to get on another host in the same subnet, and try the telnet from there; see if works on 80 and 443, and if it only works on one but not the other, we'll probably want to look at all your firewall rules on the problematic host.

I think that's a good start to narrowing down the problem; do you mind trying those out? Let me know if any don't look right or you need help with them. Thanks!