I went on to setup my own public DNS node built on top of PiHole using Cloudflared and DNSCrypt and private VPN node support with Wireguard in cloud with log monitoring using Grafana/Loki and container monitoring/management using WeaveWork Scope and Portainer all automated with Ansible, Packer and Terraform 😎. This is similar to the setup I run at home of an Ad blocking VPN with DNS over HTTPS on a Raspberry Pi4 , so why do it again ? 😅

This is automated so the stack can be created in under 5mins from scratch in the cloud 😉
It's more complex when all your routing is behind a home router, this is a cloud instance can kill and provision again.
It was OK when I was running the VPN when away from home locally connecting to my home network blocking all those pesky ads. Being abroad meant taking speed hit due to traffic routing out and coming back in.
Now I am able to use my own public DNS node and block all ads while keeping traffic on local connection utilising all available speed 💪 when I don't need to keep data encrypted over a public network.
Also it exposes only my IP to my DNS server sending cloud IP to the DNS services adding another layer of privacy, poor man's DNS privacy 😬

So cutting the tirade short by the end you should have an idea of how to have a setup of your own beyond just those tutorials that teach you how to install PiHole with a VPN that you can't even upgrade without destroying your setup. What you should have would be something like below which will be deployed using container images fetched from a container registry.

My personal node is sitting in Hetzner using a CX11 cloud instance type. It is based on 1 vCPU and 2GB RAM and comes with 20GB SSD and 20TB Bandwidth. All for 2.66 Euro(excl VAT), unbelievable ini't ?

I am going to assume some basic understanding of the tools used for this setup. So first of what we need to do is build a base image snapshot upon which we will build our application server image which I call JARVIS. To do so we will use Packer. I am not going to go in to Ansible part as it depends on what you want in your snapshot you can just point the playbook_file to your playbook.

Packer

Base image

To build the base image make sure you generate the Hetzner API TOKEN and set it as an environment variable. This recipe will build an image on Debian 10 and save it as debian-base-snapshot. Two things to note are that you should use the location and server_type where you will run the final server. The things ansible playbook installs in this step for me are copying ssh keys, installing docker and other dev tools required as per my needs.

To build run following commandpacker build packer.json

{ "variables": { "hcloud_token": "{{env `HCLOUD_TOKEN`}}" }, "builders": [ { "token": "{{ user `hcloud_token` }}", "server_name": "base-packer", "snapshot_name": "debian-base-snapshot", "snapshot_labels": { "name": "debian-base-snapshot" }, "type": "hcloud", "image": "debian-10", "location": "nbg1", "server_type": "cx11", "ssh_username": "root" } ], "provisioners": [ { "type": "shell", "inline": [ "sleep 30", "apt-get update", "apt-get -y upgrade", "apt-get update && apt-get install -y wget curl gcc make python python-dev python-setuptools python-pip libffi-dev libssl-dev libyaml-dev" ] }, { "type": "ansible", "extra_arguments": ["--vault-password-file=~/.helsing_ansible_vault_pass"], "playbook_file": "../../../../ansible/base.yml" } ] }

Jarvis image

Once you have the base snapshot we will use that in place of Hetzner provided snapshot to build our server snapshot used to run it. You can use the same token used in previous step for this as well. The recipe is very similar to the one used for base snapshot. It uses its own playbook installing what is needed in this image, One of the things I do is git clone the source repo in this stage.

packer build packer.json

{ "variables": { "hcloud_token": "{{env `JARVIS_HCLOUD_TOKEN`}}" }, "builders": [ { "token": "{{ user `hcloud_token` }}", "server_name": "jarvis-packer", "snapshot_name": "debian-jarvis-snapshot-base", "type": "hcloud", "image_filter": { "with_selector": [ "name==debian-base-snapshot" ], "most_recent": true }, "snapshot_labels": { "name": "debian-jarvis-snapshot-base" }, "location": "nbg1", "server_type": "cx11", "ssh_username": "root" } ], "provisioners": [ { "type": "shell", "inline": [ "sleep 30", "apt-get update", "apt-get -y upgrade", "apt-get update && apt-get install -y wget curl gcc make python python-dev python-setuptools python-pip libffi-dev libssl-dev libyaml-dev" ] }, { "type": "ansible", "extra_arguments": ["--vault-password-file=~/.helsing_ansible_vault_pass"], "playbook_file": "../../../../ansible/jarvis.yml" } ] }

Terraform

Last I worked with Terraform was while setting up my stack and the latest version at that time was 0.12 so working on this I upgraded to the latest 0.14.8 and it requires that you setup version.tf

version.tf

terraform { required_providers { hcloud = { source = "hetznercloud/hcloud" version = "~> 1.25.2" } } required_version = "~> 0.14"}

Here is the recipe that creates the infrastructure it uses the snapshot created earlier. Hetzner has recently launched their Firewall simplifying managing ports. This will create 1 server instance of cx11 and a firewall with the defined rules attached to it for traffic.

provider.tf

tarraform plan -out=jarvis.out

tarraform apply "jarvis.out"

variable "JARVIS_HCLOUD_TOKEN" {}provider "hcloud" { token = var.JARVIS_HCLOUD_TOKEN}data "hcloud_image" "jarvis_image" { with_selector = "name=debian-jarvis-snapshot-base"}data "hcloud_ssh_keys" "all_keys" {}resource "hcloud_firewall" "jarvis_firewall" { name = "jarvis-firewall" rule { // SSH direction = "in" protocol = "tcp" port = "22" source_ips = [ "0.0.0.0/0", "::/0" ] } rule { // HTTP direction = "in" protocol = "tcp" port = "80" source_ips = [ "0.0.0.0/0", "::/0" ] } rule { // DNS direction = "in" protocol = "udp" port = "53" source_ips = [ "0.0.0.0/0", "::/0" ] } rule { // DNS direction = "in" protocol = "tcp" port = "53" source_ips = [ "0.0.0.0/0", "::/0" ] } rule { // HTTPS direction = "in" protocol = "tcp" port = "443" source_ips = [ "0.0.0.0/0", "::/0" ] } rule { // WIREGUARD direction = "in" protocol = "udp" port = "52828" source_ips = [ "0.0.0.0/0", "::/0" ] } rule { // GRAFANA direction = "in" protocol = "tcp" port = "13443" source_ips = [ "0.0.0.0/0", "::/0" ] } rule { // SCOPE direction = "in" protocol = "tcp" port = "15443" source_ips = [ "0.0.0.0/0", "::/0" ] } rule { // PORTAINER direction = "in" protocol = "tcp" port = "19443" source_ips = [ "0.0.0.0/0", "::/0" ] }}resource "hcloud_server" "jarvis_server" { name = "jarvis-hetzner" image = data.hcloud_image.jarvis_image.id server_type = "cx11" labels = { "name" = "jarvis-hetzner" } location = "nbg1" ssh_keys = data.hcloud_ssh_keys.all_keys.ssh_keys.*.name firewall_ids = [hcloud_firewall.jarvis_firewall.id]}

That sorts out our infrastructure, now comes the important part what are we going to run and how. It is going to run 12 services using Docker Compose.

OpenResty
HAProxy
PiHole
Grafana
Loki
Promtail
Prometheus
Scope
Portainer
Cloudflared
DNSCrypt
WireGuard

All Dockerfiles setup in folder as such for convenience to be used in docker-compose file

Docker Compose

First of we need to create docker-compose.yml file that we will use to build container images that can then be tagged and pushed to a registry later to be pulled for running.

This file is used to create images with all the configs and settings required by each image that you can run as first step before tagging and pushing to registry. Lets call this file docker-compose-ci.yml and we build images as following

docker-compose -f docker-compose-ci.yml build

version: "3.7"services: #OPENRESTY 10.0.3.2 openresty: build: context: ./openresty dockerfile: ./Dockerfile image: jarvis/openresty #HAPROXY 10.0.3.6 haproxy: build: context: ./haproxy dockerfile: ./Dockerfile args: BASIC_AUTH_USERNAME: ${BASIC_AUTH_USERNAME} BASIC_AUTH_PASSWORD: ${BASIC_AUTH_PASSWORD} BASIC_AUTH_REALM: ${BASIC_AUTH_REALM} HAPROXY_HTTP_SCHEME: ${HAPROXY_HTTP_SCHEME} HAPROXY_STATS_URI: ${HAPROXY_STATS_URI} HAPROXY_STATS_REFRESH: ${HAPROXY_STATS_REFRESH} image: jarvis/haproxy #PIHOLE 10.0.3.3 pihole: build: context: ./pihole dockerfile: ./Dockerfile args: TZ: ${TZ} WEBPASSWORD: ${WEBPASSWORD} DNS1: 10.0.3.4#5053 # cloudflared IP Address DNS2: 10.0.3.5#5053 # DNSCrypt IP Address image: jarvis/pihole #CLOUDFLARED 10.0.3.4 cloudflared: build: context: ./cloudflared dockerfile: ./Dockerfile args: TZ: ${TZ} TUNNEL_DNS_UPSTREAM: ${TUNNEL_DNS_UPSTREAM} image: jarvis/cloudflared #DNSCRYPT 10.0.3.5 dnscrypt: build: context: ./dnscrypt dockerfile: ./Dockerfile image: jarvis/dnscrypt #WIREGUARD 10.0.3.7 wireguard: build: context: ./wireguard dockerfile: ./Dockerfile args: PUID: ${PUID} PGID: ${PGID} TZ: ${TZ} PEERS: ${PEERS} PEERDNS: 10.0.3.3#53 #pi-hole IP Address image: jarvis/wireguard #GRAFANA 10.0.3.8 grafana: build: context: ./grafana dockerfile: ./Dockerfile args: GF_SECURITY_ADMIN_PASSWORD: ${GF_SECURITY_ADMIN_PASSWORD} image: jarvis/grafana #LOKI 10.0.3.9 loki: build: context: ./loki dockerfile: ./Dockerfile image: jarvis/loki #PROMTAIL 10.0.3.10 promtail: build: context: ./promtail dockerfile: ./Dockerfile image: jarvis/promtail #SCOPE 10.0.3.11 scope: build: context: ./scope dockerfile: ./Dockerfile args: ENABLE_BASIC_AUTH: ${ENABLE_BASIC_AUTH} BASIC_AUTH_USERNAME: ${BASIC_AUTH_USERNAME} BASIC_AUTH_PASSWORD: ${BASIC_AUTH_PASSWORD} image: jarvis/scope #PROMETHEUS 10.0.3.12 prometheus: build: context: ./prometheus dockerfile: ./Dockerfile image: jarvis/prometheus#PORTAINER 10.0.3.13 portainer: build: context: ./portainer dockerfile: ./Dockerfile image: jarvis/portainer

Once we have all the images built, tagged and pushed (we will get to the tagging and pushing part when we see deployment, I am using Gitlab repos so they come with private registry). We use docker-compose with right set of port, volume and command mapping to bring all the services up. Read through the file to make yourself comfortable

docker-compose up

version: "3.7"services: #OPENRESTY 10.0.3.2 openresty: image: registry.gitlab.com/jarvis/openresty:amd64 container_name: jarvis_openresty networks: network: ipv4_address: 10.0.3.2 aliases: - jarvis_openresty depends_on: - haproxy expose: - "80" - "443" ports: - "80:80" - "443:443" - "13443:13443" - "15443:15443" - "19443:19443" volumes: - ./openresty/ssl:/usr/local/openresty/nginx/conf/config/ssl #HAPROXY 10.0.3.6 haproxy: image: registry.gitlab.com/jarvis/haproxy:amd64 container_name: jarvis_haproxy networks: network: ipv4_address: 10.0.3.6 aliases: - jarvis_haproxy depends_on: - pihole - grafana - prometheus - scope expose: - "80" - "18081" - "18443" - "13000" - "15000" - "19000" #PIHOLE 10.0.3.3 pihole: image: registry.gitlab.com/jarvis/pihole:amd64 container_name: jarvis_pihole volumes: - "pihole_data:/etc/pihole" - "pihole_dnsmasq_data:/etc/dnsmasq.d" - "/dev/null:/var/log/pihole.log:ro" depends_on: - cloudflared - dnscrypt expose: - "80/tcp" - "67/udp" ports: - "53:53/tcp" - "53:53/udp" environment: - DNSMASQ_LISTENING=all - IPv6=false - PIHOLELOG=/dev/null networks: network: ipv4_address: 10.0.3.3 aliases: - jarvis_pihole dns: - 127.0.0.1 - 1.1.1.1 cap_add: - NET_ADMIN #CLOUDFLARED 10.0.3.4 cloudflared: image: registry.gitlab.com/jarvis/cloudflared:amd64 container_name: jarvis_cloudflared expose: - "49312/tcp" - "5053/udp" networks: network: ipv4_address: 10.0.3.4 aliases: - jarvis_cloudflared #DNSCRYPT 10.0.3.5 dnscrypt: image: registry.gitlab.com/jarvis/dnscrypt:amd64 container_name: jarvis_dnscrypt expose: - "5053/tcp" - "5053/udp" volumes: - "dnscrypt_data:/config" networks: network: ipv4_address: 10.0.3.5 aliases: - jarvis_dnscrypt #WIREGUARD 10.0.3.7 wireguard: image: registry.gitlab.com/jarvis/wireguard:amd64 container_name: jarvis_wireguard volumes: - "wireguard_data:/config" - "/lib/modules:/lib/modules" depends_on: - pihole ports: - "52828:51820/udp" cap_add: - NET_ADMIN - SYS_MODULE networks: network: ipv4_address: 10.0.3.7 aliases: - jarvis_wireguard depends_on: - pihole #GRAFANA 10.0.3.8 grafana: image: registry.gitlab.com/jarvis/grafana:amd64 container_name: jarvis_grafana volumes: - "grafana_data:/var/lib/grafana:rw" - ./grafana/config/datasource.yml:/etc/grafana/provisioning/datasources/datasource.yml expose: - "3000" networks: network: ipv4_address: 10.0.3.8 aliases: - jarvis_grafana #LOKI 10.0.3.9 loki: image: registry.gitlab.com/jarvis/loki:amd64 container_name: jarvis_loki expose: - "3100" command: -config.file=/etc/loki/local-config.yaml networks: network: ipv4_address: 10.0.3.9 aliases: - jarvis_loki #PROMTAIL 10.0.3.10 promtail: image: registry.gitlab.com/jarvis/promtail:amd64 container_name: jarvis_promtail volumes: - /var/log:/var/log command: -config.file=/etc/promtail/config.yml networks: network: ipv4_address: 10.0.3.10 aliases: - jarvis_promtail # SCOPE 10.0.3.11 scope: image: registry.gitlab.com/jarvis/scope:amd64 container_name: jarvis_scope networks: network: ipv4_address: 10.0.3.11 aliases: - jarvis_scope expose: - "4040" pid: "host" privileged: true labels: - "works.weave.role=system" volumes: - "/var/run/docker.sock:/var/run/docker.sock:rw" command: - "--probe.docker=true" - "--weave=false" #PROMETHEUS 10.0.3.12 prometheus: image: registry.gitlab.com/jarvis/prometheus:amd64 container_name: jarvis_prometheus volumes: - "prometheus_data:/var/lib/prometheus:rw" expose: - "9090" networks: network: ipv4_address: 10.0.3.12 aliases: - jarvis_prometheus #PORTAINER 10.0.3.13 portainer: image: registry.gitlab.com/jarvis/portainer:amd64 container_name: jarvis_portainer restart: always networks: network: ipv4_address: 10.0.3.13 aliases: - jarvis_portainer volumes: - /var/run/docker.sock:/var/run/docker.sock:ro - portainer_data:/data expose: - "9000"networks: network: driver: bridge ipam: config: - subnet: 10.0.3.0/24volumes: wireguard_data: {} pihole_data: {} pihole_dnsmasq_data: {} cloudflared_data: {} dnscrypt_data: {} grafana_data: {} prometheus_data: {} portainer_data: {}

For automating the deployment phase so that you don't need to go in to your server to run all the docker commands I found a very nice OSS not very widely know called Stack Up

pressly/sup

It makes things quite convenient, it is similar to ansible in a sense that you just have to define instructions in YAML and it executes it over SSH. Always remember no matter what new kid on the block software there might be BASH is always the King of the hood. In below file you can see different stages defined that can be grouped together in a target to create a pipeline. Its a quick and simple CI/CD pipeline that you can control from your command line.

version: 0.5env: ENV: <set var> PWD: <set var> CR_USER: <set var> CR_PAT: <set var> CONTAINER_REGISTRY: <set var> PROJ_ID: <set var>networks: production: hosts: - root@servercommands: connect: desc: Check host connectivity run: uname -a; date; hostname once: true build: desc: Build Docker image run: cd $PWD/$PROJ_ID && git pull && source ~/.bashrc && docker-compose -f docker-compose-ci.yml build once: true docker_login: desc: Login to Gitlab container registry run: docker login $CONTAINER_REGISTRY -u $CR_USER -p $CR_PAT once: true docker_tag: desc: Tag images for CI registry run: >- docker images | grep "^${PROJ_ID}_" | awk '{print $1}' | xargs -I {} echo {} | xargs -I {} docker image tag {} $CONTAINER_REGISTRY/{}:latest once: true docker_push: desc: Push images to CI registry run: >- docker images | grep "^${CONTAINER_REGISTRY}" | awk '{print $1}' | xargs -I {} echo {} | xargs -I {} docker image push {}:latest once: true restart: desc: Restart docker containers run: systemctl restart docker-compose@$PROJ_ID once: true docker_ps: desc: List docker process run: sleep 30; docker ps once: true test: desc: test run: uname -a; date; hostname once: truetargets: deploy: - connect - build - docker_login - docker_tag - docker_push - restart - docker_ps

sup production deploy

Now a little bit over logging and monitoring that you can run from anywhere on the globe. Grafana is pretty amazing even though it might seems daunting to use for a personal project it sure makes your life comfortable. Loki along with promtail is super efficient for streaming your logs while using prometheus to capture metrics from HAProxy and OpenResty gives some nice insights beyond logs. Also if you want you can setup Dashboards over prometheus metrics or loki logs.

haproxy

nginx

loki

For container monitoring Scope and Portainer both are an overkill :P but I like Scope's UI and metric presentation while Portainer's management of Docker stack is unbeatable, this allows me to not go to the server for any reason and allows me to even debug my containers from the browser as they both allow you to exec in to your running containers from the browser.

scope

Some times we just underestimate how low resources we need to run so many services and how we can learn so many new things because of the amazing OSS community and their contributions.

I hope you find this useful and pushes you to increase your privacy, block those pesky ads and tracking or learn to play around and setup your own cloud infrastructure.