Hetzner GPU server setup: firewall, hardening and roll-out

On this page

• Hetzner Robot firewall: stateless, so the replies need rules
• Server hardening with one script, and why it refuses to lock you out
• Scripted build of a GPU server: what stopped the first run
• Roll-out and release check: the first findings were the checker's own
• What a customer can forget when setting up a GPU server
• Sources

A rented GPU server arrives with a login and nothing else. Between that login and an OCR service that can take documents sit a firewall that has to be written by hand, a hardening step, a build, a roll-out and a check. Each of the five has a way to fail that shows up late.

Hetzner Robot firewall: stateless, so the replies need rules

The firewall of the provider sits in front of the server and is set in the provider's web console. It is stateless: it does not know that a packet is the answer to a request the server sent. With the default action set to discard, every reply has to be allowed by a rule of its own.

The rule set for this server has seven rules, each one written once for IPv4 and once for IPv6:

• SSH, HTTP and HTTPS in (22, 80, 443)
• ping
• TCP replies, identified by the ACK flag
• UDP replies

Without the reply rules, package downloads, name resolution and the certificate request would send packets out and never see the answer. Outgoing traffic keeps the provider's default, which blocks mail. ICMPv6 stays open, because IPv6 does not work without it.

After the rules were set, the checks from outside were: SSH with a key works, name resolution works, HTTPS to the outside works over IPv4 and over IPv6, ping answers, and the only port listening is 22.

Server hardening with one script, and why it refuses to lock you out

deploy/haerten.sh hardens a fresh Ubuntu server in one run. Before it changes anything, it checks that the operator account has a valid SSH key, because switching off passwords without one would lock the operator out. If the check fails, the script stops with exit code 3 and touches nothing.

What it sets:

• An SSH drop-in file: no passwords, root only with a key, three login attempts.
• The configuration is tested with sshd -t; if the test fails, the script rolls back (exit code 4).
• The host firewall (ufw) denies incoming traffic except 22, 80 and 443; the SSH rule goes in before the firewall is switched on.
• fail2ban bans an address for one hour after five failed attempts.
• Security updates install on their own, without an automatic reboot.

A second run changes nothing and says so. --dry-run prints what would change. The script is covered by a test with faked commands (tools/test_haerten.py), which exercises the lock-out guard and the rollback.

The host firewall and the provider's firewall do different jobs. A port closed in one stays closed, whatever the other says.

Scripted build of a GPU server: what stopped the first run

The build is one script, gpu/rebuild.sh, with every version pinned. A check before it, gpu/preflight.sh, reads driver, card, memory, disk, clock, network and package sources and lists all problems at once, with exit code 3 and no stack trace.

The first rebuild on this server stopped at the Python wheel of the character engine. The wheel was missing from PyPI at the pinned version, the package index it had been pointed at returned an error, and pip skipped it without a word. The fix is the wheel as a direct URL with a hash. Later runs found a neighbour of each kind:

• Scripts lost their executable bit on the way through git archive.
• A development package for a library was not pinned and could not be resolved.
• The language-model service did not start because the Python development headers were not installed.
• A fresh bare-metal server has no NVIDIA driver at all; a script installs it now.

Each one has an entry with symptom, cause and check, and each check is a test that fails on the old state. After the fixes, a complete rebuild took about an hour and twenty minutes.

Roll-out and release check: the first findings were the checker's own

Code reaches the server through one script, deploy/rollout.sh, from a clean commit. The Caddyfile is validated before it is reloaded, and the public address comes from the configured domain. The script sets no firewall and no DNS entry.

The first roll-outs stopped for reasons the build had not shown: a file the release script left out of the package, and the language-model service from the build section. The loop was always the same: fix locally, commit, roll out again. The run that went through brought up every service, answered the health check from outside over HTTPS, redirected port 80 to HTTPS, wrote the first backup and passed the alarm check.

Then came the release check for the GPU stage, which reads documents through the deployed service and compares the output with expectations. The first run was red with 17 findings across 735 cells. Three of them were mistakes of the check itself: it asked the wrong port, compared against the wrong revision and expected a capability that does not exist. The real findings were tables on PDF and TIFF, an engine error on one hard page, words missing from a PDF, a character-engine result just outside its tolerance, and a video-memory comparison that counted two of four processes.

The second run had eight findings, the third only the character engine, the fourth none, over 739 cells. Behind the fixes were: a placeholder box for lines without a box that stuck out over the page, words lost on lines with a single box, and an unclear reference file for tables. An engine error now names the stage and the error class, and never the text of the page.

What a customer can forget when setting up a GPU server

• Ports 80 and 443 in the provider's firewall. The certificate request fails and the service looks dead from outside. The script deploy/ports-von-aussen.sh checks the ports from another machine, because the provider's firewall cannot be tested from the inside.
• The reply rules. The server reaches the network, and nothing comes back.
• The clock. A server whose clock is off fails certificate checks.
• Disk and memory. Models and Python environments need several GiB; the preflight reports it before the build starts.
• The driver on fresh bare metal. There is none.
• The roll-out does not do everything. It sets neither firewall nor DNS entry.

The steps for your own server are on Install the OCR server.

Sources