Building own VPN protocol

In Part 1 of this series, I demonstrated how to build a functional VPN protocol from scratch. I successfully encapsulated IP packets into UDP datagrams, handled NAT traversal, and deployed the solution across Docker, AWS, and even an OpenWRT home router.

While the protocol worked, it lacked the two most critical features of any production-grade VPN: Peer Authentication and Data Encryption. Without these, anyone on the wire could sniff or modify the traffic. In this post, I’m going to fix that.

For this enhancement, I chose the ChaCha20 stream cipher. It has quickly become the modern industry standard, utilized by heavyweights like WireGuard and QUIC (HTTP/3). Unlike legacy ciphers like AES, which process data in fixed blocks, ChaCha20 generates a continuous stream of pseudo-random bits that are mixed with your data. There are several reasons why it’s perfect for a custom VPN project:

Software Performance: ChaCha20 is incredibly fast in pure C and doesn't require specialized hardware instructions (like AES-NI) to perform well.
Security: It offers constant-time encryption and decryption by design, making it naturally resistant to side-channel timing attacks.
Kernel Support: It is natively included in modern OS kernels, which is where high-performance networking lives.

To implement this, I’ll use libsodium - a lightweight, "opinionated" C library that is easy to cross-compile and already available in the OpenWRT ecosystem. In addition to ChaCha20 encryption, libsodium includes the Poly1305 authenticator to handle data integrity, ensuring that the encrypted data remains untampered with - a feature we can leverage for even more robust protocol enhancements in the future.

Before we can exchange encrypted data, the client and server must prove their identity and agree on a secret Session Key. For this implementation, we utilize a Pre-Shared Key (PSK) and a three-step handshake:

Client generates 32 random bytes (Rc) and sends them to the server.
Server responds with its own 32 random bytes (Rs).
Client sends a Hash-based Message Authentication Code (HMAC). This combines a cryptographic hash function with our Pre-Shared Key to produce a fixed-length string. The server recalculates the HMAC using the PSK and the random values (Rc and Rs). If the results match, the client is authenticated, and the process continues.

Once verified, both sides derive a symmetric Session Key from the exchanged values. This key is used for all subsequent data encryption and decryption, ensuring that our traffic is protected by a fresh, session-specific secret. With the handshake complete, the "Control Plane" hands off to the "Data Plane." Every IP packet passing through our UDP tunnel is now encrypted using ChaCha20. Because the encryption is symmetric and the cipher is optimized for software, the overhead is minimal, and the performance remains high even on low-power hardware like home routers.

With the updated code compiled and deployed within my Docker environment, it was time to verify the security layer in action. I used packet capture tools to inspect the traffic and ensure the handshake and encryption were performing as expected.

First, let's look at the control plane. Here is the capture of the 3-way handshake between the client and server:

Once the handshake is finalized and the session key is established, all subsequent traffic is encrypted. Below is a capture of an encrypted ICMP (ping) packet sent through the tunnel:

When building a custom protocol, understanding the packet overhead is crucial. By comparing the size of the original IP packets to our encrypted UDP datagrams, I found that this protocol adds 56 bytes of overhead.

This overhead is a vital metric to keep in mind when configuring the MTU on the tunnel interface. Failing to account for these 56 bytes could lead to packet fragmentation, which significantly degrades performance.

While our VPN now provides solid encryption and authentication, it is still a "minimal viable product." To reach production-grade reliability, there are several advanced features I plan to explore in future articles:

Packet Replay Protection: Preventing attackers from "replaying" captured valid packets.
Data Integrity Checks: Fully implementing Poly1305 for AEAD (Authenticated Encryption).
Data Framing & MTU Handling: Better management of fragmentation and variable packet sizes.
Multi-client Support: Moving from a point-to-point tunnel to a server that handles multiple peers.
Session Management: Implementing robust handling for disconnections, timeouts, and key rotation.

The updated C code for both the client and server, including the “libsodium” integration, is provided below:

Server (tun_udp_server_crypto.c):

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <signal.h>
#include <time.h>

#include <sys/types.h>
#include <sys/ioctl.h>
#include <sys/socket.h>
#include <sys/select.h>

#include <net/if.h>
#include <linux/if_tun.h>

#include <arpa/inet.h>
#include <netinet/in.h>

#include <sodium.h>

#define BUFSIZE 2000
#define KEYLEN 32

#define FLAG_ENCRYPT  0x01
#define MSG_DATA      0x00
#define MSG_KEEPALIVE 0x01

#define LOG(fmt, ...) fprintf(stderr,"[SERVER] " fmt "\n",##__VA_ARGS__)

static volatile sig_atomic_t running = 1;
void handle_signal(int sig){ (void)sig; running = 0; }

struct __attribute__((packed)) client_hello {
    uint8_t flags;
    uint8_t Rc[32];
};

int tun_alloc(char *dev){
    struct ifreq ifr;
    int fd = open("/dev/net/tun", O_RDWR);
    if(fd < 0){ perror("tun"); exit(1); }

    memset(&ifr, 0, sizeof(ifr));
    ifr.ifr_flags = IFF_TUN | IFF_NO_PI;
    strncpy(ifr.ifr_name, dev, IFNAMSIZ);

    if(ioctl(fd, TUNSETIFF, &ifr) < 0){
        perror("TUNSETIFF"); exit(1);
    }

    strcpy(dev, ifr.ifr_name);
    return fd;
}

int main(int argc, char *argv[]){
    if(argc != 4){
        fprintf(stderr,"Usage: %s <tun> <listen_port> <psk>\n", argv[0]);
        return 1;
    }

    signal(SIGINT, handle_signal);
    signal(SIGTERM, handle_signal);

    if(sodium_init() < 0){
        fprintf(stderr,"libsodium init failed\n");
        return 1;
    }

    unsigned char *psk = (unsigned char*)argv[3];

    char tun_name[IFNAMSIZ];
    strncpy(tun_name, argv[1], IFNAMSIZ);
    int tun_fd = tun_alloc(tun_name);

    int sock = socket(AF_INET, SOCK_DGRAM, 0);
    struct sockaddr_in local = {0};
    local.sin_family = AF_INET;
    local.sin_port = htons(atoi(argv[2]));
    bind(sock, (struct sockaddr*)&local, sizeof(local));

    struct sockaddr_in client;
    socklen_t clen = sizeof(client);

    struct client_hello hello;
    unsigned char Rs[32], session_key[KEYLEN], auth[crypto_auth_BYTES];
    unsigned char material[64];

    recvfrom(sock, &hello, sizeof(hello), 0,
             (struct sockaddr*)&client, &clen);

    int encrypt_enabled = (hello.flags & FLAG_ENCRYPT) != 0;
    LOG("Encryption %s", encrypt_enabled ? "ENABLED" : "DISABLED");

    randombytes_buf(Rs, sizeof(Rs));
    sendto(sock, Rs, sizeof(Rs), 0,
           (struct sockaddr*)&client, clen);

    if(encrypt_enabled){
        recvfrom(sock, auth, sizeof(auth), 0,
                 (struct sockaddr*)&client, &clen);

        memcpy(material, hello.Rc, 32);
        memcpy(material + 32, Rs, 32);

        crypto_generichash(session_key, sizeof(session_key),
                           material, sizeof(material),
                           psk, strlen((char*)psk));

        if(crypto_auth_verify(auth, material, sizeof(material),
                              session_key) != 0){
            LOG("AUTH failed");
            return 1;
        }
    }

    unsigned char buf[BUFSIZE];
    unsigned char outbuf[BUFSIZE + 64];
    uint64_t nonce_counter = 0;

    while(running){
        fd_set fds;
        FD_ZERO(&fds);
        FD_SET(tun_fd, &fds);
        FD_SET(sock, &fds);

        struct timeval tv = {1, 0};
        select((tun_fd > sock ? tun_fd : sock) + 1,
               &fds, NULL, NULL, &tv);

        if(FD_ISSET(sock, &fds)){
            int len = recv(sock, outbuf, sizeof(outbuf), 0);
            if(len <= 0) continue;

            if(outbuf[0] == MSG_KEEPALIVE)
                continue;

            if(encrypt_enabled){
                unsigned char nonce[12];
                memcpy(nonce, outbuf + 1, 12);

                unsigned long long n;
                if(crypto_aead_chacha20poly1305_ietf_decrypt(
                    buf, &n, NULL,
                    outbuf + 13, len - 13,
                    NULL, 0,
                    nonce, session_key) == 0){
                    write(tun_fd, buf, n);
                }
            }else{
                write(tun_fd, outbuf + 1, len - 1);
            }
        }

        if(FD_ISSET(tun_fd, &fds)){
            int n = read(tun_fd, buf, BUFSIZE);
            if(n <= 0) continue;

            if(encrypt_enabled){
                unsigned char nonce[12] = {0};
                memcpy(nonce + 4, &nonce_counter, 8);
                nonce_counter++;

                unsigned long long clen;
                outbuf[0] = MSG_DATA;
                crypto_aead_chacha20poly1305_ietf_encrypt(
                    outbuf + 13, &clen,
                    buf, n,
                    NULL, 0, NULL,
                    nonce, session_key);

                memcpy(outbuf + 1, nonce, 12);
                sendto(sock, outbuf, clen + 13, 0,
                       (struct sockaddr*)&client, clen);
            }else{
                outbuf[0] = MSG_DATA;
                memcpy(outbuf + 1, buf, n);
                sendto(sock, outbuf, n + 1, 0,
                       (struct sockaddr*)&client, clen);
            }
        }
    }

    close(sock);
    close(tun_fd);
    return 0;
}

Client (tun_udp_client_crypto.c):

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <signal.h>
#include <time.h>

#include <sys/types.h>
#include <sys/ioctl.h>
#include <sys/socket.h>
#include <sys/select.h>

#include <net/if.h>
#include <linux/if_tun.h>

#include <arpa/inet.h>
#include <netinet/in.h>

#include <sodium.h>

#define BUFSIZE 2000
#define KEYLEN 32

#define FLAG_ENCRYPT  0x01
#define MSG_DATA      0x00
#define MSG_KEEPALIVE 0x01

#define KEEPALIVE_INTERVAL 10

#define LOG(fmt, ...) fprintf(stderr,"[CLIENT] " fmt "\n",##__VA_ARGS__)

static volatile sig_atomic_t running = 1;
void handle_signal(int sig){ (void)sig; running = 0; }

struct __attribute__((packed)) client_hello {
    uint8_t flags;
    uint8_t Rc[32];
};

int tun_alloc(char *dev){
    struct ifreq ifr;
    int fd = open("/dev/net/tun", O_RDWR);
    if(fd < 0){ perror("tun"); exit(1); }

    memset(&ifr, 0, sizeof(ifr));
    ifr.ifr_flags = IFF_TUN | IFF_NO_PI;
    strncpy(ifr.ifr_name, dev, IFNAMSIZ);

    if(ioctl(fd, TUNSETIFF, &ifr) < 0){
        perror("TUNSETIFF"); exit(1);
    }

    strcpy(dev, ifr.ifr_name);
    return fd;
}

int main(int argc, char *argv[]){
    if(argc != 7){
        fprintf(stderr,
            "Usage: %s <tun> <server_ip> <server_port> <local_port> <psk> <enc>\n",
            argv[0]);
        return 1;
    }

    signal(SIGINT, handle_signal);
    signal(SIGTERM, handle_signal);

    if(sodium_init() < 0){
        fprintf(stderr,"libsodium init failed\n");
        return 1;
    }

    int encrypt_enabled = atoi(argv[6]) ? 1 : 0;
    LOG("Encryption %s", encrypt_enabled ? "ENABLED" : "DISABLED");

    unsigned char *psk = (unsigned char*)argv[5];

    char tun_name[IFNAMSIZ];
    strncpy(tun_name, argv[1], IFNAMSIZ);
    int tun_fd = tun_alloc(tun_name);
    LOG("TUN=%s", tun_name);

    int sock = socket(AF_INET, SOCK_DGRAM, 0);

    struct sockaddr_in local = {0};
    local.sin_family = AF_INET;
    local.sin_port = htons(atoi(argv[4]));
    bind(sock, (struct sockaddr*)&local, sizeof(local));

    struct sockaddr_in server = {0};
    server.sin_family = AF_INET;
    server.sin_port = htons(atoi(argv[3]));
    inet_aton(argv[2], &server.sin_addr);

    /* -------- HANDSHAKE -------- */

    struct client_hello hello;
    unsigned char Rs[32], session_key[KEYLEN], auth[crypto_auth_BYTES];
    unsigned char material[64];

    hello.flags = encrypt_enabled ? FLAG_ENCRYPT : 0;
    randombytes_buf(hello.Rc, sizeof(hello.Rc));

    sendto(sock, &hello, sizeof(hello), 0,
           (struct sockaddr*)&server, sizeof(server));

    recv(sock, Rs, sizeof(Rs), 0);

    if(encrypt_enabled){
        memcpy(material, hello.Rc, 32);
        memcpy(material + 32, Rs, 32);

        crypto_generichash(session_key, sizeof(session_key),
                           material, sizeof(material),
                           psk, strlen((char*)psk));

        crypto_auth(auth, material, sizeof(material), session_key);
        sendto(sock, auth, sizeof(auth), 0,
               (struct sockaddr*)&server, sizeof(server));
    }

    /* -------- DATA LOOP -------- */

    unsigned char buf[BUFSIZE];
    unsigned char outbuf[BUFSIZE + 64];
    uint64_t nonce_counter = 0;
    time_t last_tx = time(NULL);

    while(running){
        fd_set fds;
        FD_ZERO(&fds);
        FD_SET(tun_fd, &fds);
        FD_SET(sock, &fds);

        struct timeval tv = {1, 0};
        select((tun_fd > sock ? tun_fd : sock) + 1,
               &fds, NULL, NULL, &tv);

        time_t now = time(NULL);
        if(now - last_tx >= KEEPALIVE_INTERVAL){
            unsigned char ka = MSG_KEEPALIVE;
            sendto(sock, &ka, 1, 0,
                   (struct sockaddr*)&server, sizeof(server));
            last_tx = now;
        }

        if(FD_ISSET(tun_fd, &fds)){
            int n = read(tun_fd, buf, BUFSIZE);
            if(n <= 0) continue;

            if(encrypt_enabled){
                unsigned char nonce[12] = {0};
                memcpy(nonce + 4, &nonce_counter, 8);
                nonce_counter++;

                unsigned long long clen;
                outbuf[0] = MSG_DATA;
                crypto_aead_chacha20poly1305_ietf_encrypt(
                    outbuf + 13, &clen,
                    buf, n,
                    NULL, 0, NULL,
                    nonce, session_key);

                memcpy(outbuf + 1, nonce, 12);
                sendto(sock, outbuf, clen + 13, 0,
                       (struct sockaddr*)&server, sizeof(server));
            }else{
                outbuf[0] = MSG_DATA;
                memcpy(outbuf + 1, buf, n);
                sendto(sock, outbuf, n + 1, 0,
                       (struct sockaddr*)&server, sizeof(server));
            }
            last_tx = now;
        }

        if(FD_ISSET(sock, &fds)){
            int len = recv(sock, outbuf, sizeof(outbuf), 0);
            if(len <= 0) continue;

            if(outbuf[0] == MSG_KEEPALIVE)
                continue;

            if(encrypt_enabled){
                unsigned char nonce[12];
                memcpy(nonce, outbuf + 1, 12);

                unsigned long long n;
                if(crypto_aead_chacha20poly1305_ietf_decrypt(
                    buf, &n, NULL,
                    outbuf + 13, len - 13,
                    NULL, 0,
                    nonce, session_key) == 0){
                    write(tun_fd, buf, n);
                }
            }else{
                write(tun_fd, outbuf + 1, len - 1);
            }
        }
    }

    close(sock);
    close(tun_fd);
    return 0;
}

Updated script “run.sh“ to create tunnel interface, define pre-shared key and run either client or server:

#!/bin/sh
set -e

if [ "$#" -lt 1 ]; then
    echo "Usage: $0 <server|client> [args...]"
    exit 1
fi

MODE=$1
shift

# Default TUN interface
TUN_IF=tun0

# Create TUN interface if it doesn't exist
ip tuntap add dev $TUN_IF mode tun || true
ip link set $TUN_IF up

# IP address must be passed as argument
if [ -z "$1" ]; then
    echo "You must provide TUN IP address as first argument"
    exit 1
fi
TUN_IP=$1
shift

ip addr add $TUN_IP/30 dev $TUN_IF || true

#Preshared key
PSK="ThisIsMyPreSharedKey123!"

# Run server or client
if [ "$MODE" = "server" ]; then
    # Args: <listen_port>
    if [ -z "$1" ]; then
        echo "Server mode requires listen port"
        exit 1
    fi
    LISTEN_PORT=$1
    shift
    tcpdump -i eth0 -w ./server.pcap &
    TCPDUMP_PID=$!
    echo "tcpdump started with pid $TCPDUMP_PID"
    echo "Starting server on port $LISTEN_PORT, TUN IP $TUN_IP"
    exec ./tun_udp_server $TUN_IF $LISTEN_PORT $PSK
elif [ "$MODE" = "client" ]; then
    if [ "$#" -lt 3 ]; then
        echo "Client mode requires <server_ip> <server_port> <local_port>"
        exit 1
    fi
    SERVER_IP=$1
    SERVER_PORT=$2
    LOCAL_PORT=$3
    echo "Starting client to $SERVER_IP:$SERVER_PORT from local port $LOCAL_PORT, TUN IP $TUN_IP"
    exec ./tun_udp_client $TUN_IF $SERVER_IP $SERVER_PORT $LOCAL_PORT $PSK
else
    echo "Unknown mode: $MODE"
    exit 1
fi

Updated “Dockerfile” to include “libsodium“

FROM debian:bookworm

RUN apt-get update && \
    apt-get install -y gcc iproute2 iputils-ping net-tools tcpdump libsodium-dev && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY tun_udp_server_crypto.c .
COPY tun_udp_client_crypto.c .
COPY run.sh .

# Make sure script is executable and remove CRLF
RUN chmod +x /app/run.sh && sed -i 's/\r$//' /app/run.sh

# Compile programs
RUN gcc tun_udp_client_crypto.c -lsodium -o tun_udp_client
RUN gcc tun_udp_server_crypto.c -lsodium -o tun_udp_server

# Use run.sh as ENTRYPOINT, so arguments are passed correctly
ENTRYPOINT ["/app/run.sh"]

Building own VPN protocol (Part 2)

Comments

More from this blog

Building own traffic load balancer

Building own VPN protocol (Part 1)

Command Palette

Comments

More from this blog