VPN-protected services

In a mesh, peers can communicate directly, subject to the access policies. Here's a small example with two user devices and two application servers, each on its own VPS. The lines show direct VPN tunnels between user devices and application servers.

[Phone]  ---+           +--- [Mattermost]
            +--- VPN ---+
[Laptop] ---+           +--- [Nextcloud]

In a hub-and-spoke network, traffic between the spokes passes through a central gateway. In this version, both the user devices and the application servers participate in the VPN.

[Phone]  -- VPN --+           +-- VPN -- [Mattermost]
                  |           |
                  +-[Gateway]-+
                  |           |
[Laptop] -- VPN --+           +-- VPN -- [Nextcloud]

Both designs, as drawn here, involve installing VPN software on every application server as well as on user devices. That started a little train of thought…

• Packaging: the VPN software needs to be available for all my servers' operating systems.
• DNS: I'd like to keep using the services' current domain names. If those names need to resolve to different addresses inside and outside the VPN, I'll need split DNS, or a similar mechanism to steer requests to the VPN addresses.
• Certificates: the servers still need to pass ACME challenges. I could retain a public HTTP endpoint for HTTP-01 or switch to the DNS-01 challenge, but I'm not particularly keen on the extra work needed to set that up.
• Account management: generating and distributing WireGuard keys manually is fine for a few devices, but less appealing when colleagues need to join, replace devices, or leave.

None of these problems is a blocker in itself, but the complexity does add up in a way that I wasn't imagining at the beginning. Some of the tools above solve much of this already, but what I'd imagined as a Saturday-afternoon project was getting out of hand!

Then I had a realisation: for the access control I wanted, the application servers didn't need to be part of the VPN at all. I could route requests through one gateway and have each application server accept HTTPS connections only from that gateway's public addresses.

[Phone]  -- VPN --+                        +-- HTTPS -- [Mattermost]
                  |                        |
                  +-[Gateway + source NAT]-+
                  |                        |
[Laptop] -- VPN --+                        +-- HTTPS -- [Nextcloud]

The gateway and user devices run NetBird (or WireGuard or Tailscale). The application servers don't. The WireGuard tunnel ends at the gateway, while HTTPS remains encrypted between the user device and the application server. The gateway simply forwards the packets.

The crucial ingredient is source NAT. When forwarding a request, the gateway replaces the device's VPN source address with its own public egress address. The application server's firewall recognises that address and lets the connection through. Replies return to the gateway, which reverses the translation and sends them back through the tunnel.

The services still have public IP addresses, but their HTTPS ports reject connections from everyone else. That's what "behind a VPN" means in this setup. The firewall trusts the gateway; NetBird's policies decide which users and devices may send traffic through it.

This simplifies a few things:

• Existing DNS records stay unchanged: the same public addresses are used inside and outside the VPN.
• Let's Encrypt can keep using HTTP-01, provided the web server serves the challenge directly over public TCP port 80 and doesn't expose the application there.
• No VPN software is needed on the application servers.

There are tradeoffs:

• The gateway becomes a shared dependency for availability and bandwidth, and its public egress addresses need to stay stable.
• The application servers see the gateway's address rather than individual devices' addresses. This might be inconvenient for logging and auditing purposes.
• A large part of this setup's security is offloaded onto the firewalls of the individual application servers. They need to be configured properly to only allow traffic from the gateway.

I'm not claiming this is an original design or a particularly creative approach, at all. It's probably relatively standard. I was just relieved when I realised there was a simple way of implementing this, keeping my existing DNS and certificate setup and without installing VPN software on every application server.

The gateway runs on a Hetzner VPS provisioned with guix deploy.

Each application server must accept HTTPS only from the gateway. Here's a complete nftables example for the standalone test server. The addresses are documentation placeholders: replace them with the gateway's actual public egress addresses.

flush ruleset

define gateway_ipv4 = 192.0.2.1
define gateway_ipv6 = 2001:db8::1

table inet firewall {
    chain inbound {
        type filter hook input priority 0; policy drop;
        iif lo accept
        ct state invalid drop
        ct state established,related accept
        ip protocol icmp accept
        meta l4proto ipv6-icmp accept
        tcp dport ssh accept

        # HTTP is public; the web server restricts what it serves.
        tcp dport http accept
        ip saddr $gateway_ipv4 tcp dport https accept
        ip6 saddr $gateway_ipv6 tcp dport https accept
        tcp dport https reject with tcp reset
    }
    chain forward {
        type filter hook forward priority 0; policy drop;
    }
}

Once deployed, the gateway needs to join the NetBird account. Create a one-off setup key with a short expiry, leaving the ephemeral-peer option disabled: this is a persistent gateway, not a disposable client.

On the gateway, as root, save the key in /root/netbird-setup-key with mode 0600, then enrol the machine:

netbird up \
    --setup-key-file /root/netbird-setup-key \
    --disable-dns \
    --disable-client-routes

After successful enrolment, remove the temporary key file and inspect the connection:

rm --force /root/netbird-setup-key
netbird status --detail

The flags preserve the gateway's existing DNS configuration and prevent it from consuming routes advertised by other routing peers. They don't stop it acting as a routing peer itself. User devices need client routes enabled, so don't copy --disable-client-routes to those devices.

In NetBird Cloud's Networks section:

• Create a Network for the protected services and select the enrolled gateway as its routing peer, assigning a routing-peer group as required.
• Add each application server as a resource using its IP address or domain name. Give the resources a group, such as internal-services.
• Keep masquerading enabled. Without source NAT, the application firewalls won't see the gateway's public source address.
• Add a policy from the group containing the authorised users' devices to the resource group, allowing TCP ports 80 and 443.

Use individual server addresses or domain names rather than a default route such as 0.0.0.0/0. Traffic to those resources goes through the gateway, while other internet traffic takes its usual path. Existing public DNS records can stay unchanged.

Finally, invite users and have them enrol their devices. Check group membership and remove any broad policies you don't intend to keep. A policy permitting access to a resource through the gateway is separate from one permitting access to the gateway itself.