Setting up Netbird for secure networking with SWAG and Keycloak with CRDB as backend
Practical guide to Netbird deployment and integration with Keycloak and CockroachDB.
I have been using Tailscale and Wireguard (Manual) for a while and I think it is time to move to something that is selfhosted and easy to configure. I have tried several solutions in the past such as ZeroTier, Netmaker, Headscale and Innernet. Those were never up to what I wanted, either they missed proper administrative UI (Headscale), Were hard to deploy (ZeroTier), Had too many bugs (Netmaker) or did not support all the devices I need (Innernet).
After watching Netbird for a while, it seems to have matured quite well as compared to where it was last year, that is when I tested it last. Seems that a lot of work also went into the quick deploy scripts (not that we are going to use it) and the inner-working of nodes. This post will utilise several components new components and others that I have already written posts about, more specifically - Keycloak and Cockroach DB HA.
- Keycloak is the user management component and authentication proxy.
- CockroachDB (CRDB) is the underlying database supporting Keycloak.
- Technitium is a DNS server that will help help geographically balance requests and provide HA capabilities for apps.
There is a complete docker compose file at the end for bringing up netbird.
Some links may be affiliate links that keep this site running.
Setting up a database
First, for setting up CRDB check out this writeup for HA. Once you have a running CRDB instances, we need to move on to setting up Keycloak as our authentication provider.
Setting up SWAG
Basic configuration
Lets create our folder structure, and edit the compose file:
1
2
mkdir /docker/swag
nano /docker/swag/docker-compose.yaml
I am using RFC2136 to verify certificates by updating ACME records on Technitium DNS, however you can check the SWAG documentation for other validation methods.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
---
version: "2.1"
services:
swag:
image: lscr.io/linuxserver/swag
container_name: swag
cap_add:
- NET_ADMIN
environment:
- PUID=1000
- PGID=1000
- TZ=Europe/London
- URL=yourdomain.url
- SUBDOMAINS=www,
- VALIDATION=dns
- DNSPLUGIN=rfc2136
- ONLY_SUBDOMAINS=false #optional
- STAGING=false #optional
volumes:
- ./config:/config
ports:
- 443:443
restart: unless-stopped
networks:
- lsio
networks:
lsio:
external: true
Make sure to create your external lsio network using
docker network create lsio
You can now bring up the container using docker compose up and once you see the configuration of the variables you can shut the container down or exit it.
Certificates using DNS and RFC2136
Now we will edit our dns validation using nano config/dns-conf/rfc2136.ini and update the values of your TSIG keys as explained in this part of the Technitium post.
Keycloak/SWAG configuration
The last part that is left for us here is to configure the forwarding to Keycloak for all requests to authenticate.
Lets create the configuration file:
1
sudo nano /docker/swag/config/nginx/proxy-confs/keycloak.subdomain.conf
We will paste inside the following:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
## Version 15/02/2024
# make sure that your keycloak container is named keycloak
# make sure that your dns has a cname set for auth.*
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name auth.*;
include /config/nginx/ssl.conf;
client_max_body_size 0;
location / {
include /config/nginx/proxy.conf;
include /config/nginx/resolver.conf;
set $upstream_app keycloak;
set $upstream_port 8443;
set $upstream_proto https;
proxy_pass $upstream_proto://$upstream_app:$upstream_port;
}
}
Setting up Keycloak
Create the directory structure:
1
2
sudo mkdir -p /docker/keycloack/certs
sudo chown $USER:$USER -R /docker
Before we continue with creating the compose configuration, copy over the client and ca certificates for CRDB over to the certs folder.
If you followed the previous guide on setting up HA crdb, then you will need to run the following command
docker run --rm --name=roach-temp -v "ckdb-temp:/cockroach/cockroach-data" -v "/docker/cockroachdb/certs:/certs" -v "/docker/cockroachdb/safe-dir:/safe-dir" cockroachdb/cockroach:v23.1.13 cert create-client auth --certs-dir=/certs --ca-key=/safe-dir/ca.keyIf however you have your own crdb instance running, then you can drop the docker part and run
cockroach cert create-client auth --certs-dir=/certs --ca-key=/safe-dir/ca.key
authis the username that we will be using to login from CRDB.
Create database users
Run the commands below on the CRDB instance to create your database and user with access to the database.
1
2
CREATE DATABASE keycloak;
CREATE USER auth WITH PASSWORD 'auth';
We do not need to assign a password as the authentication is done using certificates.
1
GRANT ALL PRIVILEGES ON DATABASE keycloak TO auth;
Configure Keycloak
We are now going to set up SWAG as a reverse proxy before we bring up Keycloak, this will let us put letsencrypt certs in front of Keycloak.
Lets now create the docker-compose file for keycloak with our custom CRDB configuration, feel free to use a .env file for the configuration.
1
nano /docker/keycloak/docker-compose.yaml
Paste the following, change the network to one that you use with a swag container (configured later on).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
version: '3.8'
services:
keycloak:
image: quay.io/phasetwo/keycloak-crdb:latest
hostname: keycloak
container_name: keycloak
restart: unless-stopped
volumes:
- ./certs:/certs
- /docker/swag/config/etc/letsencrypt/live/selfhosted.club:/webcerts:ro
command: start --proxy edge --db-username ${KC_DB_USER} --db-password ${KC_DB_PASSWORD}
networks:
- lsio
env_file:
- .env
networks:
lsio:
external: true
You’ll notice that we are using a different image than keycloak/keycloak, and that i because the main development does not have a crdb driver to connect with. Thanks to xgp, who incorporated and patched the original image to work with crdb, it is possible now; more info can be found here.
Keycloak .env file configuration:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# DB Configuration
KC_DB=cockroach
KC_DB_URL_HOST=
KC_DB_URL_PORT=26257
KC_DB_URL_DATABASE=
KC_DB_USER=
KC_DB_PASSWORD=
JDBC_PARAMS=?ssl=true&sslmode=require&sslrootcert=/certs/ca.crt&sslcert=/certs/client.auth.crt&sslkey=/certs/client.auth.key
# HTTPS configuration
KC_HOSTNAME=fqdn.of.your.domain
KC_HTTPS_KEY_STORE_FILE=/webcerts/KCcert.pfx
KC_HTTPS_KEY_STORE_PASSWORD=strongpassword
# Additional DB options for crdb
KC_TRANSACTION_XA_ENABLED=false
KC_TRANSACTION_JTA_ENABLED=false
# Keycloack admin user
KEYCLOAK_ADMIN=admin
KEYCLOAK_ADMIN_PASSWORD=password
Convert certificates for keycloak use
We will create a script to convert the web certificates for use in keycloak, this will be run after every renewal of certbot.
1
nano /docker/swag/config/etc/letsencrypt/renewal-hooks/post/pkcs12convert.sh
Paste in the script, remember to replace yourdomain with your keycloak domain/subdomain.
1
2
3
4
#!/bin/sh
openssl pkcs12 -export -out /docker/swag/config/etc/letsencrypt/live/yourdomain/KCcert.pfx -inkey /docker/swag/config/etc/letsencrypt/live/yourdomain/privkey.pem -in /docker/swag/config/etc/letsencrypt/live/yourdomain/cert.pem -certfile /docker/swag/config/etc/letsencrypt/live/yourdomain/chain.pem -passout pass:strongpassword
echo "pkcs#12 generated!"
We need to make the file exetubale, and lets run it to convert the certificates SWAG already requested.
1
2
chmod +x /docker/swag/config/etc/letsencrypt/renewal-hooks/post/pkcs12convert
sh /docker/swag/config/etc/letsencrypt/renewal-hooks/post/pkcs12convert.sh
Bringing up keycloak
Now that everything is done, lets bring up keycloak so it can populate all of the tables in the database.
1
2
cd /docker/keycloak
docker compose up -d && docker compose logs -f
You’ll see in the logs population of the tables, if you head to your CRDB console, you will find the database is replicated across all 3 nodes that we configured earlier:
Keycloak on CRDB
Netbird Setup
The official guide can be found here, you should follow the guide until you reach the “Set properties in the setup.env file” portion. Create the folder structure and clone the netbird repo:
1
2
3
4
5
6
7
8
9
10
cd /docker
#!/bin/bash
REPO="https://github.com/netbirdio/netbird/"
# this command will fetch the latest release e.g. v0.8.7
LATEST_TAG=$(basename $(curl -fs -o/dev/null -w %{redirect_url} ${REPO}releases/latest))
echo $LATEST_TAG
# this command will clone the latest tag
git clone --depth 1 --branch $LATEST_TAG $REPO
Let’s create our setup.env file with the configuration.
1
nano /docker/netbird/infrastructure_files/setup.env
Paste the following into the file, changing the values that you need:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
NETBIRD_AUTH_OIDC_CONFIGURATION_ENDPOINT=`https://<YOUR_KEYCLOAK_HOST_AND_PORT>/realms/netbird/.well-known/openid-configuration`
NETBIRD_USE_AUTH0=false
NETBIRD_AUTH_CLIENT_ID=`netbird-client`
NETBIRD_AUTH_SUPPORTED_SCOPES="openid profile email offline_access api"
NETBIRD_AUTH_AUDIENCE=`netbird-client`
NETBIRD_AUTH_DEVICE_AUTH_CLIENT_ID=`netbird-client`
NETBIRD_MGMT_IDP="keycloak"
NETBIRD_IDP_MGMT_CLIENT_ID="netbird-backend"
NETBIRD_IDP_MGMT_CLIENT_SECRET=<NETBIRD_BACKEND_CLIENT_SECRET>
NETBIRD_IDP_MGMT_EXTRA_ADMIN_ENDPOINT="https://<YOUR_KEYCLOAK_HOST_AND_PORT>/admin/realms/netbird"
NETBIRD_LETSENCRYPT_DOMAIN=<your high level domain>
NETBIRD_DOMAIN=<your high level domain>
Lets generate the compose file using the following commands:
1
2
3
sudo apt install jq
cd /docker/netbird/infrastructure_files
./configure.sh
Our docker compose for Netbird will be available at the artifacts folder inside.
Netbird/SWAG Configuration
We need to create a configuration file for nginx:
1
sudo nano /docker/swag/config/nginx/proxy-confs/netbird.subdomain.conf
Paste the following into the configuration file”
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
## Version 15/02/2024
# make sure that your dns has a cname set for netbird.*
server {
# HTTP server config
listen 80;
server_name netbird.*;
# 301 redirect to HTTPS
location / {
return 301 https://$host$request_uri;
}
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name netbird.*;
include /config/nginx/ssl.conf;
# client_max_body_size 0;
client_header_timeout 1d;
client_body_timeout 1d;
proxy_connect_timeout 30s;
proxy_send_timeout 180s;
proxy_read_timeout 180s;
proxy_buffer_size 12288;
proxy_buffers 4 8192;
grpc_read_timeout 3600s;
grpc_send_timeout 3600s;
grpc_socket_keepalive on;
location / {
include /config/nginx/proxy.conf;
include /config/nginx/resolver.conf;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Forwarded-Proto https;
set $upstream_app artifacts-dashboard-1;
set $upstream_port 80;
set $upstream_proto http;
proxy_pass http://artifacts-dashboard-1:80;
}
location /signalexchange.SignalExchange/ {
include /config/nginx/proxy.conf;
include /config/nginx/resolver.conf;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Forwarded-Proto https;
set $upstream_app artifacts-signal-1;
set $upstream_port 80;
set $upstream_proto grpc;
grpc_pass grpc://artifacts-signal-1:80;
}
location /api {
include /config/nginx/proxy.conf;
include /config/nginx/resolver.conf;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Forwarded-Proto https;
set $upstream_app artifacts-management-1;
set $upstream_port 443;
set $upstream_proto http;
proxy_pass https://artifacts-management-1:443;
}
location /management.ManagementService/ {
include /config/nginx/proxy.conf;
include /config/nginx/resolver.conf;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Forwarded-Proto https;
set $upstream_app artifacts-management-1;
set $upstream_port 443;
set $upstream_proto grpc;
grpc_pass grpcs://artifacts-management-1:443;
}
}
Adjustments to Netbird configuration
We need to do a few small changes, so follow up on the sections that you need.
Some of the steps below ONLY apply if you use a wildcard certificate for your domain.
Adjust port of Signal URI
1
nano /docker/netbird/infrastructure_files/artifacts/management.json
For Signal configuration to pass, we need to remove the port number from the file above, find the values and replace the 10000 with 443.
1
2
3
4
5
...
"Signal": {
"Proto": "https",
"URI": "netbird.selfhosted.club:443",
...
Leverage SWAG certs
As we are already running SWAG as a reverse proxy and it requests certificates for us, we are going to tap into those certificates and modify a few other settings we need.
1
nano /docker/netbird/infrastructure_files/artifacts/management.json
Find the lines with CertFile and CertKey, remove the subdomain from the entry and save the file, it will look like this:
1
2
3
4
5
"HttpConfig": {
....
"CertFile": "/etc/letsencrypt/live/selfhosted.club/fullchain.pem",
"CertKey": "/etc/letsencrypt/live/selfhosted.club/privkey.pem",
...
Now modify the docker compose file:
1
nano /docker/netbird/infrastructure_files/artifacts/docker-compose.yaml
You will need to find and replace the line below across the docker compose, from:
1
2
volumes:
- netbird-letsencrypt:/etc/letsencrypt/
To:
1
2
volumes:
- /docker/swag/config/etc/letsencrypt:/etc/letsencrypt/:ro
Remove the volume mapping for
netbird-letsencryptat the end of the file as the volumes are no longer in use.
SWAG Network
If you are using the lsio network for SWAG, make sure to add that network to the dashboard, management and signal containers.
Open up firewall ports for coturn:
1
2
sudo ufw allow 3478/udp # Listening port for STUN/TURN
sudo ufw allow 49152:65535/udp # Range for dynamic relay connections
Bring up Netbird
1
2
cd /docker/netbird/infrastructure_files/artifacts
docker compose up -d && docker compose logs -f
You will get LetsEncrypt error about the certificate issuance, that’s ok, we are leveraging the SWAG ones.
Netbird Dashboard with connected peers
Post-up
One last thing we need to do is to restart Netbird when the certificates are refreshed, this is done with a simple script.
Lets create the script:
1
nano /docker/swag/config/etc/letsencrypt/renewal-hooks/post/30-restart_containers.sh
Paste in:
1
2
3
4
5
6
7
8
#!/bin/bash
echo "restarting keycloak"
docker restart keycloak
sleep 10 # Pauses the script for 10 seconds
echo "restarting netbird"
docker restart artifacts-dashboard-1 artifacts-management-1
And we need to make it an executable:
1
chmod +x /docker/swag/config/etc/letsencrypt/renewal-hooks/post/30-restart_containers.sh
To be able to restart the containers from SWAG, you will need to give access to the docker socket.
Issues
One major issue I stumbled across was some ports not being opened and nodes failing to connect to the signal container. After troubleshooting for several hours, it was all solved by a restart… 😅
High-Availability
Currently Netbird does not support HA for the management container, however it is in the roadmap. You can follow this guide to create a S3 docker volume and map the sqlite/json database to it to achieve HA - It is probably not a good idea to have it active-active as a deployment. For Keycloak, for HA just recreate the docker container on the other side as keycloak is stateless, it only needs a connection to the database.
Full docker compose file
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
version: "3"
services:
#UI dashboard
dashboard:
image: netbirdio/dashboard:latest
restart: unless-stopped
environment:
# Endpoints
- NETBIRD_MGMT_API_ENDPOINT=https://netbird.selfhosted.club:443
- NETBIRD_MGMT_GRPC_API_ENDPOINT=https://netbird.selfhosted.club:443
# OIDC
- AUTH_AUDIENCE=netbird-client
- AUTH_CLIENT_ID=netbird-client
- AUTH_CLIENT_SECRET=
- AUTH_AUTHORITY=https://keycloak.selfhosted.club/realms/netbird
- USE_AUTH0=false
- AUTH_SUPPORTED_SCOPES=openid profile email offline_access api
- AUTH_REDIRECT_URI=
- AUTH_SILENT_REDIRECT_URI=
- NETBIRD_TOKEN_SOURCE=accessToken
# SSL
- NGINX_SSL_PORT=443
# Letsencrypt
- LETSENCRYPT_DOMAIN=netbird.selfhosted.club
- LETSENCRYPT_EMAIL=
volumes:
- /docker/swag/config/etc/letsencrypt:/etc/letsencrypt/:ro
networks:
- lsio
# Signal
signal:
image: netbirdio/signal:latest
restart: unless-stopped
volumes:
- netbird-signal:/var/lib/netbird
networks:
- lsio
# Management
management:
image: netbirdio/management:latest
restart: unless-stopped
depends_on:
- dashboard
volumes:
- netbird-mgmt:/var/lib/netbird
- /docker/swag/config/etc/letsencrypt:/etc/letsencrypt/:ro
- ./management.json:/etc/netbird/management.json
command: [
"--port", "443",
"--log-file", "console",
"--disable-anonymous-metrics=false",
"--single-account-mode-domain=netbird.selfhosted.club",
"--dns-domain=netbird.selfhosted"
]
networks:
- lsio
# Coturn
coturn:
image: coturn/coturn:latest
restart: unless-stopped
domainname: netbird.selfhosted.club
volumes:
- ./turnserver.conf:/etc/turnserver.conf:ro
network_mode: host
command:
- -c /etc/turnserver.conf
networks:
lsio:
external: true
volumes:
netbird-mgmt:
netbird-signal: