aucubin.de

Making my services available from the outside

Sun, 18 Jun 2023 00:00:00 GMT

I've been selfhosting my services for a couple of years with dedicated servers from different hosting providers, but lately I decided to move all my services on a machine inside my server rack at home. As I didn't want to make my IP publicly known I decided against opening ports on my router and instead used a setup consisting of a VPS and an VPN.

I also didn't want to use Cloudflare as I wanted to be independent of the provider that is used to make everything available outside. With the setup I have now I can basically switch the VPS and only need to do some small setup to get it working again.

What is needed?

A VM or physical machine where your services will be hosted
A VPS from an hosting provider. I use a CAX11 Cloud Server from Hetzner.
Wireguard as VPN to connect the VM and the VPS together.

Setting up the VPS

I bought the CAX11 and selected Debian as operation system. Afterwards I did my normal base setup by installing fail2ban, switching iptables with nftables and creating a dedicated user instead of using root to log into ssh.

Setting up the VPN

As VPN I'm using Wireguard as it is very easy to install and configure.

I installed wg-quick by running sudo apt install wireguard-tools on the VPS. In order to connect two machines with wireguard you need to create three things on each machine:

public key
private key
pre-shared key

The keys will be used to authenticate the wireguard clients with each other and encrypt the traffic between them. They will need to be created on both clients, which in my case is the VPS and the service machine.

Run wg genkey | (umask 0077 && tee peer_VPS.key) | wg pubkey > peer_VPS.pub to generate both public (peer_A.pub) and private (peer_A.key) key.

Then run wg genpsk > peer_VPS-peer_service.psk to generate the pre-shared key. After creating the keys on both machines you should have the following files:

peer_VPS.key
peer_VPS.pub
peer_service.key
peer_service.pub
peer_VPS-peer_service.psk

The next step is to create an wireguard config file for the VPS under /etc/wireguard/wg0.conf. Mine looks like this:

[Interface]
Address=192.168.10.1/24
ListenPort=51820
PrivateKey=PRIVATE_KEY <-- copy the content of peer_VPS.key here
PreUp=sysctl -w net.ipv4.ip_forward=1
PreUp=sysctl -w net.ipv6.conf.all.forwarding=1
PostDown=sysctl -w net.ipv4.ip_forward=0
PostDown=sysctl -w net.ipv6.conf.all.forwarding=0
MTU=1280

# service
[Peer]
PublicKey=PUBLIC_KEY <-- copy the content of peer_service.pub here
PresharedKey=PRESHARED_KEY <-- copy the content of peer_VPS-peer_service.psk here
AllowedIPs=192.168.10.2/32

You should then be able to start the VPN by running systemctl enable --now wg-quick@wg0. After running wg in the command line you should seen an output like this:

interface: wg0
  public key: PUBLIC_KEY <-- should be the value of peer_VPS.pub
  private key: (hidden)
  listening port: 51820

peer: PUBLIC_KEY <-- should be the value of peer_service.pub
  preshared key: (hidden)
  endpoint: your-ip
  allowed ips: 192.168.10.2/32

As the other side of the VPN is not yet running you will not be able to ping the other side, so we still need to set it up.

Create the Wireguard configuration in /etc/wireguard/wg0.conf on the other client (in my case the service machine) like this:

[Interface]
ListenPort = 51821
PrivateKey = PRIVATE_KEY <-- should be the value of peer_service.key
Address = 192.168.10.2/24

[Peer]
PublicKey = PUBLIC_KEY <-- should be the value of peer_VPS.pub
PresharedKey = PRESHARED_KEY <-- copy the content of peer_VPS-peer_service.psk here
AllowedIPs = 192.168.10.1/32
Endpoint = your-ip:51820 <-- your-ip should be the ip address of the VPS
PersistentKeepalive = 25

You can see that this configuration file has an endpoint and PersistentKeepalive configured. In order for wireguard to work the machines need to exchange traffic with each other, which normally happens something gets routed over the wg0 interface.

As the service machine is not reachable from the outside, it needs to connect to the VPS. For that reason we use endpoint to specify to which ip and port the service machine needs to connect and PersistentKeepalive=25 to have the machine send a keep-alive message every 25s to the vps, in order to keep the connection running.

Now you can start wireguard on the service machine by running systemctl enable --now wg-quick@wg0 and you should be able to ping the machines after some seconds. Your wg output should also now look like this:

interface: wg0
  public key: PUBLIC_KEY <-- should be the value of peer_VPS.pub
  private key: (hidden)
  listening port: 51820

peer: PUBLIC_KEY <-- should be the value of peer_service.pub
  preshared key: (hidden)
  endpoint: your-ip
  allowed ips: 192.168.10.2/32
  latest handshake: 20 seconds ago
  transfer: 900.22 MiB recived, 211.6 MiB sent

If you can't ping the machines check if the ip range used (here 192.168.10.0/24) is in use anywhere or a firewall is enabled on the machines. To debug you can ping on one machine and run tcpdump -i wg0 on the other machine to check if traffic even arrives on the other machine.

If it doesn't it's also possible that your keys don't match - try to regenerate them in this case.

Making the services available from the outside

In my case I run my services in docker compose on the service machine and use traefik, so my docker-compose.yml of traefik looks like this:

version: "3.3"

services:
  traefik:
    image: "traefik:latest"
    container_name: "traefik"
    restart: always
    network_mode: "host"
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.le.acme.tlschallenge=true"
      - "--certificatesresolvers.le.acme.storage=/acme.json"
    volumes:
      - "$PWD/acme.json:/acme.json"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

An simple docker-compose.yml of string-is looks like this:

version: '3'

services:
  string-is:
    image: daveperrett/string-is
    restart: always
    container_name: string-is
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.string-is.rule=Host(`string-is.aucubin.de`)"
      - "traefik.http.routers.string-is.entrypoints=websecure"
      - "traefik.http.routers.string-is.tls.certresolver=le"

Traefik handles TLS and certificates, so the only thing open is to connect the 443/tcp port of the VPS with the 443/tcp port of the service machine. You could forward the ports directly with nftables (or iptables), but I decided against it and use HAProxy in order to have better logging and be able use load-balancing if needed.

The first thing is to install haproxy on the VPS by running sudo apt install haproxy. There will be a default haproxy.cfg shipped under /etc/haproxy/haproxy.cfg to which I added the following block:

frontend main_https_listen
	bind	:443 v4v6
	bind	:::443 v6only
         mode	tcp
	option	tcplog
         default_backend service

backend service
	mode	tcp
	balance	source
	option	tcp-check
	server	service 192.168.10.2:443 check

This will basically do the following:

The frontend block tells HAProxy to listen on 443/tcp (both IPv4 and IPv6) in TCP Proxy mode. This means HAProxy will just forward TCP traffic to the backend, other than the HTTPS mode where HAProxy would run HTTP and you can adjust headers or enable TLS on it.
The backend block tells HAProxy that the backend service has 192.168.10.2:443 as endpoint to forward to and to check whether its available by trying to open a TCP connection on the backend.

After adjusting the configuration file you can restart haproxy with systemctl restart haproxy and after some seconds you should see that the service is up when running journalctl -b -u haproxy. You then just need to wait until Traefik could fetch TLS Certificates from Let's Encrypt and you are ready to go. You should now be able to connect to your services without being in your internal network.

Building a weather station with an ESP8266

Sat, 28 May 2022 00:00:00 GMT

Some time ago I decided that I wanted to track some weather data like temperature, humidity and CO2 in my apartment. The data should be imported into a InfluxDB so that I can use Grafana in order to create diagrams based on this data. After looking over the pre-built options I decided to just built the sensor from scratch with a ESP8266 and a couple of sensors.

What is needed?

ESP8266
I decided to use a NodeMCU as it comes with a USB port that has a serial programmer directly embedded.
DHT21 (AM2301) Temperature + Humidity Sensor
This sensor is used to track the temperature and the humidity in the room.
SGP30 CO2 Sensor
This sensor is used to calculate the CO2 and TVOC level in the room.

Preparation

SGP 30

The first thing to do is to solder the header pins onto the SGP30. Once that is done the first thing to do is to connect the SGP30 to the ESP8266 and use the test programm from Adafruit in order to calibrate the co2 sensor by calculating the baseline. I used the following pins for the connection:

SGP 30 Pin	ESP8266 Pin
VIN	3V
GND	GND
SCL	D1
SDA	D2

There are two options to do the calibration:

Use the Arduino IDE to install the sgp30test library, run it and then use the serial monitor to get the right data out. This is described in the article from Adafruit.
Download the example from the SGP30 Github and use your preferred tooling to upload it to the ESP8266.
As I will use PlatformIO for the code anyways I will show the second option.

PlatformIO

An installation guide for PlatformIO can be found on their page. I will use visual studio code so it basically comes down to installing the PlatformIO extension from the visual studio code store.

The first thing is to create a new Project in PlatformIO with the following settings:

The next step is to add the Adafruit SGP30 library from the Library Page:

You will directly find the sgp30test example in the library page so you can copy it from there (if not get it from the github link above) and replace the src/main.cpp with the sgp30test example. Then you can connect your NodeMCU to your computer and flash the software by using the "PlatformIO: Upload icon" in the bottom bar of visual studio code .
Before you can then use the Serial Monitor you need to adjust the Serial Monitor Speed to 115200 baud. That can be done by adding monitor_speed = 115200 to the platformio.ini inside the project. Afterwards you can click on the "PlatformIO: Serial Monitor" icon and should be able to see data flooding in:

Now you just need to open a window and wait for 10-15 minutes while the baseline gets calibrates. After this time note down the baseline values that get printed (in the screenshot they are 0x8E42 and 0x9265).

DHT21

The DHT21 does not need any solder and can directly be connected to the NodeMCU. I used the following pins:

DHT21 Pin	ESP8266 Pin
VDD	3V
Data	D6
GND	GND

MQTT

To push the data from the NodeMCU into InfluxDB I decided to use MQTT as I can attach Telegraf to MQTT and also use MQTT to display the data on other devices. In order to publish data to MQTT a MQTT broker like Eclipse Mosquitto is needed. As I use Docker for all my self-hosting needs I used the following docker-compose.yml to create a instance of Eclipse Mosquitto on my server:

version: "3"

services:
  mosquitto:
    image: eclipse-mosquitto:latest
    container_name: mosquitto
    volumes:
      - $PWD/config:/mosquitto/config
      - mosquitto-data:/mosquitto/data
    ports:
      - 1883:1883
    restart: always

volumes:
    mosquitto-config:
    mosquitto-data:

By default mosquitto does not use authentication. To fix this add password_file /mosquitto/config/password_file to config/mosquitto.conf inside the folder where the docker file is defined. You can then add users with docker exec -it mosquitto mosquitto_passwd /mosquitto/config/password_file <username>.

Flashing

Now that everything is set up you can get the code from the repository. A couple of things need to be changed.

There is a cred.h.template in the include folder.
This file contains secrets like the wifi password for the ESP8266 and the credentials for the MQTT broker.
Rename the file to cred.h and enter the values for your wifi and MQTT broker.
The setup method of the src/main.cpp contains a call to sgp.setIAQBaseline. The arguments are the baseline values that you gathered from the sgp30test util in the preparation.
If you want to check the topic names for the data in MQTT you need to change the second argument of the Adafruit_MQTT_Publish objects on top of src/main.cpp.

Afterwards you can flash the code on the ESP8266. You can use the serial monitor to check if the ESP can correctly connect to your wifi and mqtt broker.

InfluxDB

The MQTT data can easily be imported into InfluxDB by using Telegraf and using the mqtt_consumer input plugin.

[[inputs.mqtt_consumer]]
  servers = ["tcp://broker_url:1883"]

  topics = [
    "weather/aucubin/temp",
    "weather/aucubin/humidity",
    "weather/aucubin/tvoc",
    "weather/aucubin/eco2"
  ]

  client_id = "client_name"

  username = "username"
  password = "password"

  data_format = "value"
  data_type = "float"