DebTik/docs/aerotun-v2.md

AeroTUN v2: operation, bandwidth diagnosis, and upgrade

AeroTUN v2 is incompatible with v1. Upgrade both ends together. Existing tunnel addresses, names and generated PSKs can be reused. The implementation does not fall back to the insecure v1 wire format.

Why the veth fallback failed

On the inspected LXC host, /dev/net/tun is absent, so AeroTUN uses veth plus an AF_PACKET socket. The original implementation forwarded packets with unfinished TX checksums and potentially unsegmented GSO payloads. A raw socket does not complete these operations. A TCP SYN can therefore arrive with an invalid checksum and be discarded, making a bandwidth test time out before data starts.

There were two additional bugs: RX and TX used the same Ethernet buffer concurrently, and the socket read its own injected PACKET_OUTGOING frames. Those could corrupt packets and feed received traffic back through the tunnel.

The fixes disable TX checksum/TSO/GSO/GRO offloads on both veth ends, use separate RX/TX buffers, and ignore outgoing frames both at the socket and in userspace. PeerAddress now installs an explicit peer host route and, for veth, a permanent neighbour entry. Required configuration errors are returned, rather than ignored. No existing interface is deleted to make room for a new tunnel.

Reproduction and validation on 2026-10-30

All bandwidth tests used disposable network namespaces and a private veth underlay; they did not change production links, routes, or services.

• Original HEAD with a test harness using connected /24 overlay addresses: TCP connection timed out, both with and without compression.
• Original protocol with only the veth offloads disabled: TCP connected and transferred data (roughly 72–120 Mbit/s in this diagnostic run). Other veth defects remained, so this was a diagnostic comparison, not a deployable fix.
• The corrected v2 implementation passed forward TCP with two streams, reverse TCP, and UDP with a 1472-byte payload, including inner IPv4 fragmentation.
• A run after the MTU fix delivered about 603–666 Mbit/s TCP without compression, and 788–790 Mbit/s with explicitly repeating/compressible iperf3 payloads. UDP delivered the requested 20 Mbit/s in each case.
• Authentication, forged counters, duplicate/old packets, reflection, both peer restarts, handshake retries, persistence failures and route rollback have regression tests. The integration suite also checks that the original default survives and that the outer endpoint bypasses the overlay.

These are local host measurements, not an Internet benchmark or a throughput promise. Compressible iperf payloads are useful for exercising LZ4 but do not represent already compressed or encrypted application traffic. The initial local tests exercised veth because this LXC has no /dev/net/tun. The subsequent deployment also exercised native TUN on the ARM64 peer; see the live deployment report.

Run checks from the repository root:

go test ./...
go test -race ./internal/aerotun
go vet ./...
node --check internal/web/static/app.js
# Requires root, ip, nsenter, iperf3, network namespace privileges, and veth fallback:
AEROTUN_INTEGRATION=1 go test -run TestVethBandwidthIntegration -v ./internal/aerotun

Protocol and connection status

Each runner creates a random 128-bit boot identity. Control messages carry a two-byte effective MTU advertisement. Both peers reduce to the lower advertised MTU after session authentication, including when their underlay MTUs differ. HELLO, ACK, and CONFIRM bind sessions to both current boot identities. HMAC-SHA256 derives distinct keys for each direction and session pair. XChaCha20-Poly1305 uses the sender identity and a monotonic 64-bit counter as its 24-byte nonce. Counters never wrap. Data from a previous receiver boot is rejected; retired sender sessions cannot replace a newer session. The replay window is updated only after AEAD authentication. This is a custom PSK protocol, without forward secrecy or an independent audit. Use a generated random PSK; validation requires at least 32 characters.

A client sends HELLO immediately and retries every two seconds while disconnected. Keepalives run every 20 seconds. A server learns/roams its endpoint only from authenticated session traffic. A client pins its configured endpoint.

running means the local runner exists. connected additionally requires an authenticated peer packet within 65 seconds. The dashboard uses connected for its connection flag and active count. last_handshake is retained for API compatibility and reports the last authenticated session packet. last_error, dropped_packets, auth_failures, and replay_drops expose failures. Traffic counters advance only after successful socket/device writes; wire counters cover UDP payloads, including control messages, and exclude outer IP/UDP headers.

The server supports one peer per tunnel instance. Use different instances/ports and PSKs for independent peer connections. A peer restart may take up to the 65-second liveness timeout to trigger a new handshake on the remaining peer.

MTU and routing

V2 adds 60 bytes of authenticated header/tag, plus 28 bytes for IPv4/UDP or 48 bytes for IPv6/UDP. The requested MTU is capped by the local route/link MTU minus that overhead, at startup or authenticated peer discovery, and to the peer advertisement. Both veth ends use the same effective MTU. Smaller MTUs elsewhere on the path still require an explicit lower setting; this is not full path MTU discovery.

On the inspected host, the remote endpoint uses a WireGuard interface with MTU

1. The v2 IPv4 outer path therefore caps the AeroTUN MTU at 1332. Keeping 1420 as the overlay MTU would require outer fragmentation. The dashboard/API reports the effective MTU. During live deployment the remote cached path MTU was 1380, which further reduced the negotiated overlay MTU to 1292.

PeerAddress adds a /32 or /128 host route. It does not automatically advertise LAN subnets or enable forwarding/NAT. Add those routes and firewall rules as needed.

DefaultRoute is supported for clients with a fixed IPv4 endpoint. It pins the endpoint to its existing underlay route, then adds 0.0.0.0/1 and 128.0.0.0/1 via the overlay. The original default remains intact. Existing host routes are preserved; conflicting routes cause an error and rollback. Stop removes only routes added by this runner. The server shuts runners down on normal exit. Uncatchable process/host failures can leave veth devices behind; an existing interface is deliberately reported as an error rather than deleted blindly.

Coordinated installation

The prepared binary is at: /root/.local/share/debtik-builds/aerotun-v2/debtik. The ARM64 build is alongside it as debtik-linux-arm64. Both hosts were subsequently upgraded; see the deployment report. For future upgrades, ensure both ends have v2 and management access independent of this AeroTUN connection. Preserve the old binary and /etc/debtik/vpn/aerotun_instances.json on each host.

On each end, after coordinating the restart:

cp /usr/local/bin/debtik /usr/local/bin/debtik.pre-aerotun-v2
install -m 0755 /path/to/new/debtik /usr/local/bin/debtik.next
systemctl stop debtik
mv /usr/local/bin/debtik.next /usr/local/bin/debtik
systemctl start debtik
journalctl -u debtik -n 50 --no-pager

If an old veth remains after stopping the OLD service, verify its ownership and that no runner uses it before removing that specific old AeroTUN interface. V1 shutdown did not reliably clean up its veths. Do not delete unrelated interfaces.

Then test the peer overlay address before testing LAN routing or a default route. For iperf3, start the server on the opposite overlay address and run the client against that address; testing the underlay address does not measure AeroTUN. Start UDP tests with a 1000-byte payload and increase only after confirming MTU.

For the subsequent default-UDP congestion and RX fix, see the Bandwidth Test follow-up.

The deployed pair has since moved to AeroTUN v3. V2 remains an explicit legacy option for coordinated rollback.