Inside the Monitoring Stack That Watches Your Site 24/7

Your e-commerce site just sold its last item of the day and the checkout page flashes a 500 error. Customers start posting on social media. The problem is not your code - it is your server missing a single database connection. Within a minute our monitoring stack has already raised the alarm, often before the first angry tweet lands.

We built a three-layer stack that runs on every node of our fleet. From the moment a service deviates from normal behaviour, flame-watchman, flame-sentinel and flame-guardian act in sequence. Below is the tour from the customer's point of view. You will see what gets noticed in 60 seconds, what is pinged immediately, and what we see before you even log in.

The 60-Second Sentinel

The Sentinel is a tiny heartbeat daemon that runs on each node. Every 60 seconds it posts a JSON packet to our Discord alert channel. The packet contains the node name, the current kernel version and a simple "alive" flag.

# Example of the payload sent by flame-sentinel
curl -X POST https://discord.com/api/webhooks/... \
  -H "Content-Type: application/json" \
  -d '{"node":"ember","kernel":"6.18.5","status":"ok"}'

If the packet stops arriving, the #alerts channel lights up with a red notice. That single missing heartbeat tells us the underlying host has vanished, network connectivity is broken, or the process itself crashed. Because the Sentinel runs locally, it detects loss of a server faster than any external polling service could.

From your perspective the first sign of trouble is usually a Discord message like:

⚠️ Sentinel silence on spark - last ping 02:13 UTC

You get the alert instantly, often before you notice a slowdown on your site. The absence of a signal is itself a signal, and we treat it with top priority.

Five-Minute Watchman Checks

Flame-watchman is the workhorse of our stack. Every five minutes it iterates over a static list of services defined per node. For each service it performs a set of checks:

• TCP connectivity - nc -z -w2 127.0.0.1 443
• HTTP response - curl -s -o /dev/null -w "%{http_code}" https://example.com/health
• SMTP handshake - nc -z -w2 127.0.0.1 25 && echo "EHLO test" | nc 127.0.0.1 25
• DNS resolution - dig +short example.com @127.0.0.1

The result of each check is written to a structured journal file /var/log/flame/watchman.log. An entry looks like:

2026-07-13T14:05:00Z ember nginx-http ok 200
2026-07-13T14:05:00Z ember postgres-tcp fail timeout

When a check fails, flame-watchman immediately forwards a concise alert to Discord. For example, a lost PostgreSQL port will generate:

❌ ember postgres-tcp failed - timeout (14:05 UTC)

Because the checks run on the same machine that hosts the services, they catch problems that external monitors miss, such as local firewall rules blocking a port, or a mis-configured loopback address. You will see the alert in the same #alerts channel that the Sentinel uses, giving you a single place to understand both loss of heartbeat and service failures.

Immediate Bans with flame-guardian and Guardian-X

Security events are handled by two cooperating daemons. flame-guardian watches the SSH auth log and the nginx access log in real time. When it identifies a brute-force pattern - five failed SSH attempts within 30 seconds, or a request for /wp-admin that returns a 404 repeatedly - it adds the offending IP to an nftables set.

# Sample rule added by flame-guardian
nft add rule inet filter blacklist ip saddr 203.0.113.45 drop

Bans are tiered. The first tier lasts five minutes, the second ten minutes, and so on up to a permanent "Guardian Jail". Each escalation is recorded in /var/log/flame/guardian.log.

Guardian-X sits one layer deeper, at the XDP kernel hook. It drops packets from the highest-tier bans before they reach the networking stack. This prevents the kernel from allocating any resources for the offending traffic.

// Pseudo-code snippet from Guardian-X
if (ip in permanent_blacklist) {
    return XDP_DROP;
}

Both daemons push ban events to Discord. You might receive a message like:

🔒 spark banned 198.51.100.22 - 5-minute tier (SSH brute-force)

Because the alerts are sent as soon as the logs record the offending line, you see the block before the attacker can finish a scan. The combination of nftables and XDP gives us a fast, low-overhead defense that runs entirely on our own hardware.

What You See Before You Log In

All three components feed the same Discord channels. The #alerts channel contains:

• Sentinel silence notices (every 60 seconds)
• Watchman failures (every five minutes)
• Guardian ban spikes (in real time)

The #health channel receives a summary of each Watchman cycle, for example:

🟢 ember Watchman cycle completed - 48 checks passed, 2 warnings

Because we keep the alerts in a chat platform, you can view them from any device without opening a ticket system. The moment you see a failure, you also have the journal entry and the offending log line ready for investigation.

If you prefer a manual snapshot, run flame doctor on your node. The command aggregates the status of network, services, certificates and disk usage into a single, readable report.

$ flame doctor
Network: OK
nginx: running (pid 3124)
PostgreSQL: 1/3 instances healthy
SSL cert: example.com expires in 42 days
Disk usage: /var 68%

The output is designed to be actionable. If a service is down, the corresponding line tells you exactly which check failed and provides the relevant log file path.

Trade-offs You Should Know

Our stack runs on three rented nodes in Germany, the US and Bulgaria. Because we do not own the datacenters, we rely on the providers' physical security. The software itself is open source and runs on Alpine Linux 3.23.4 with linux-virt 6.18.x. All alerts are delivered via Discord, which means you need an internet connection to see them, but it also gives us a low-cost, real-time channel without additional SaaS fees.

We do not promise 100 % uptime. The Watchman runs every five minutes, so a transient failure that resolves before the next cycle may not be logged. Likewise, a network outage that prevents the Sentinel from posting will generate a silence alert, but we cannot guarantee the exact moment of the outage. The design choices favor speed of detection and low overhead over exhaustive coverage.

Bottom Line

From the moment a problem appears on your site, our monitoring stack detects it within seconds. The Sentinel alerts you to a dead node instantly. The Watchman catches service-level issues every five minutes and logs the details you need to act. flame-guardian and Guardian-X block malicious traffic before it can do damage, and you see the ban in the same Discord feed.

Because we built each piece ourselves, the alerts are tied directly to our infrastructure. You get a clear, honest picture of what is happening on the server that hosts your site. That is the core of web hosting monitoring at TrueCore - simple, fast and transparent.