Route traffic to a specific host and port through a tunnel with iptables rules

Access from Hashicorp Vault to Kube-API is crucial. It is required to check if the provided JWT is valid.

The CA cert has only these names DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:localhost, IP Address:10.100.0.1, IP Address:127.0.0.1, IP Address:1.1.1.1, so if we are querying 10.0.0.2:6443 we could not trust the connection.
It is not possible to add 1.1.1.1 to AllowedIPs in the Wireguard config, because this is the endpoint of Wireguard itself.
It is also not possible to whitelist the Vault server's IP, because it is behind a dynamic IP.

It could be possible to reissue CA or to reconfigure k3s to listen on a different interface, but IDK what issues I'll face after, and this is for sure an expected downtime.

The first step was the addition of routing on the server with Kube-API

#!/bin/bash

case $1 in
  start)
    iptables -t nat -A PREROUTING -i wg0 -p tcp --dport 6443 -d 10.0.0.2 -j DNAT --to-destination 1.1.1.1:6443
    iptables -t nat -A POSTROUTING -o eth0 -p tcp -d 1.1.1.1 --dport 6443 -j MASQUERADE
  ;;
  stop)
    iptables -t nat -D PREROUTING -i wg0 -p tcp --dport 6443 -d 10.0.0.2 -j DNAT --to-destination 1.1.1.1:6443
    iptables -t nat -D POSTROUTING -o eth0 -p tcp -d 1.1.1.1 --dport 6443 -j MASQUERADE
  ;;
  *)
  echo "Unknown value in first argument. Should be start or stop"
  ;;
esac

and a systemd unit to auto apply/remove these rules (created, enabled, started)

[Unit]
Description=Firewall rules for wg0
After=sys-subsystem-net-devices-wg0.device
BindsTo=sys-subsystem-net-devices-wg0.device

[Service]
Type=oneshot
ExecStart=/opt/wireguard/wg0-routing.sh start
ExecStop=/opt/wireguard/wg0-routing.sh stop
RemainAfterExit=yes

[Install]
WantedBy=sys-subsystem-net-devices-wg0.device

This allowed access Kube-API if the query was sent to https://10.0.0.2:6443. Still, Vault will not trust this connection, because 10.0.0.2 is not in the DNS names of the current CA cert.

curl --connect-timeout 1 -kX GET https://10.0.0.2:6443
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

The next step: Vault needs to send queries to 1.1.1.1:6433, but actually queries should be sent to 10.0.0.2:6433 instead.
To achieve this, I've added the following script:

#!/bin/bash

case $1 in
  start)
    iptables -t nat -A OUTPUT -p tcp -d 1.1.1.1 --dport 6443 -j DNAT --to-destination 10.0.0.2:6443
    iptables -t nat -A POSTROUTING -p tcp -d 10.0.0.2 --dport 6443 -j SNAT --to-source 10.0.0.3
  ;;
  stop)
    iptables -t nat -D OUTPUT -p tcp -d 1.1.1.1 --dport 6443 -j DNAT --to-destination 10.0.0.2:6443
    iptables -t nat -D POSTROUTING -p tcp -d 10.0.0.2 --dport 6443 -j SNAT --to-source 10.0.0.3
  ;;
  *)
    echo "Usage: $0 start|stop"
  ;;
esac

These iptables rules would:

• Forward the packet through Wireguard connection to 1.1.1.1
• Replace the source IP with Wireguard's IP. Without SNAT, I saw only a single SYN packet from 192.168.0.5 and nothing else.

A similar systemd unit was added, enabled, and started.

The result is: authentication works

JWT_TOKEN=$(kubectl -n my-app-ns create token my-app-sa --duration 10m)
curl -sD /dev/stderr -X POST -d "{\"role\": \"my-app\", \"jwt\": \"${JWT_TOKEN}\"}" https://vault.domain.local/v1/auth/kubernetes/login | jq '.'

HTTP/2 200
...

and I see 403 after 10 minutes with the same JWT

curl -sD /dev/stderr -X POST -d "{\"role\": \"my-app\", \"jwt\": \"${JWT_TOKEN}\"}" https://vault.domain.local/v1/auth/kubernetes/login | jq '.'

HTTP/2 403
...

{
  "errors": [
    "invalid expiration time (exp) claim: token is expired"
  ]
}