HA and failover for your site using Technitium DNS

I always like it when my services are always up, but that usually entails having multiple proxies that route and check that the backends are up - this would mean that all the traffic HAS to go through your proxies first, and then rerouted from the proxy to the service and back - this is ideal when everything sits in the same environment and connectivity is cheap; not so much when all the services are decentralised, across continents and all bandwidth is charged.

This is where Technitium comes in, while cloudflare has this capability, it is going to cost you around $20 a month to set up a DNS balancer. For $20 a month I can maintain 3-4 DNS servers in different regions - or 2 DNS and 2 Web instances, the savings are decent over time, and I get to mess a bit more with DNS and other selfhosted solutions - win/win for me 🤣.

Some links may be affiliate links that keep this site running.

Let’s get started

If you have more than one DNS server, you will need to have the apps installed on ALL DNS servers, and the config replicated across the apps.

Failover

You’re first going to need to install the Failover App on Technitium, the app contains various ways to check if the host or service is online (These checks are done from the DNS server).

Technitium Failover App

I found that the defaults are quite well set in the app and show what you need to do, there are a few other things to explore here and that will come up in future posts, for now we will just configure an additional monitor for MySQL/Mariadb and use the default one for HTTPS. This will be for the ghost instance I wrote in this post.

I host majority of my cloud instances on HostHatch VPS (Virtual Private Server) Instance (In Asia) for a steal. Some of the other hosts I use are RackNerd (US) and WebHorizon (Asia+Europe) VPS, and decided that it is time to move away from Linode - which is a Great service, but I am looking to reduce the billing on instances. For comparison, I save more than 50% on HostHatch compared to Linode ($3.33 compared to $8) - Don't get me wrong, if this was an extremely (like REALLY) critical application, I would keep it on Linode.

MySQL check

Configure mysql check by going into the Failover App config, add the following:

{
      "name": "tcp3306",
      "type": "tcp",
      "interval": 60,
      "retries": 3,
      "timeout": 10,
      "port": 3306,
      "emailAlert": "default",
      "webHook": "default"
    },

The code above will check that the machine is listening on port 3306 - that means that the MySQL/Mariadb server is up.

We will now configure an app record for the databases, do note that this does not take into consideration your geo-location, I will show an example for that with the ghost (web app) instance:

Technitium Failover Example

Above, you’ll see the definition of the Failover app for addresses - you can either do a failover between addresses or CNAMES.

• Primary - The primary address that will be checked and traffic routed to.
• Secondary - If the primary is down, this is where traffic will go now.
• serverDown - as per the documentation, this should be a status page rather than a third server.
• healthCheck - this is the type of health check conducted - based on the config file definition.
• healthCheckUrl - this is relevant pretty much only for http/https types of checks, it will connect to the IP addresses you’ve configured using that URL.
• allowTxtStatus - This is an interesting one, if it is set to true you can query your DNS server for a TXT record of the same name as the APP, this will tell you the status of the failover app and at what it is pointing.

WebApp check

This is how it would look like for a web server:

Technitium failover configuration for webapp

Here you can see the use of the healthCheckUrl setting for this specific website. This means that if the primary location does not respond to HTTPS on that IP to the URL defined, then it will failover to the secondary defined IP.

TXT DNS Record

You can query your DNS server for a TXT record of the same name as the APP record to see the status of the server, you should see something as below if your "allowTxtStatus": true configuration is set.

      "Name": "la-primary.selfhosted.club",
      "Type": "TXT",
      "Class": "IN",
      "TTL": "30 (30 sec)",
      "RDLENGTH": "144 bytes",
      "RDATA": {
        "Text": "app=failover; addressType=Primary; address=31.220.30.180; healthCheck=https; healthCheckUrl=https://www.selfhosted.club/; healthStatus=Healthy;"

Chaining Failover with Geo-location

As what I also like doing besides Failover is to have a Geo location based access for latency sake, it is possible to chain your records on to another. In the example below, you can see the la-primary and the sg-primary records, which are Failover App records being chained under the Geo location app to first redirect the the traffic based on geolocation and then have the failover kick in.

Technitium geo distance app

Once that is done, register for a free account at MaxMind, and download the city database.

While the geodistance app does infact comes with a database, you might want to keep it updated and once a while download the new datasets.

MaxMind Geo Database download

After downloading, what I do is extract it, and zip the mmdb file, then upload it back into the app. Yes I know, I am working on an automation here….

Chaining of records

Now that we have the database updated, and the app installed we can now configure the geolocation routing, and that will also leverage the failover for us. I think that here the config is quite self explanatory…

Chaining of failover and geo distance records

How do we test this?

Well, the easiest is going to keycdn and doing a traceroute from different locations to see which one gets routed where - you should see different IP addresses depending on the distance of the testing server from your server. As you can see the different IPv6 addresses from the testing below, these get routed to different servers, that means that your visitors or workloads are redirected to the closest server over to them.

Both servers are up, traceroute from different locations

Now I will test the failover by bringing down the LA server, and run the keycdn tool once again, this time we should be redirected only to Singapore as the LA location should be down.

Same test as above but one server is down, see the redirection?

Voila, we are redirected only to the Singapore server, we can also confirm this by the TXT record showing the la-primary being down. Let’s test the DNS record:

      "Name": "la-primary.selfhosted.club",
      "Type": "TXT",
      "Class": "IN",
      "TTL": "30 (30 sec)",
      "RDLENGTH": "180 bytes",
      "RDATA": {
        "Text": "app=failover; addressType=Primary; address=31.220.30.180; healthCheck=https; healthCheckUrl=https://www.selfhosted.club/; healthStatus=Failed; failureReason=Connection timed out.;"

Yup, TXT record shows the connection is timed out.

Done!

Congrats! You now have a redundant Database/Webapp where visitors get routed to based on their geolocation without a loadbalancer.