When I built a self-hosted analytics platform, the query engine was ClickHouse — and the first real decision wasn't which engine, it was where to run it. The tempting answer, since I already had a Kubernetes cluster, was to run it there on network-attached storage. I didn't. It sits on a dedicated bare-metal box, and that single choice is the difference between a warehouse that feels instant and one that feels like a cloud invoice. Here's the reasoning, what you give up, and the part people skip until it hurts: backing it up without corrupting your data.
Why not just run it in Kubernetes?
ClickHouse is a column store built for scanning huge tables, which makes it heavily IO-bound on the queries that matter. That's exactly where the deployment choice bites. Local NVMe on a dedicated box gives you somewhere around 5 GB/s sequential read. Network-attached cluster storage — the kind you get from a typical Kubernetes CSI volume — lands closer to 200–400 MB/s. On an analytics scan that's not a rounding error; it's a 10× difference in latency on the workload you actually run.
The rest compounds it. Merges and in-memory parts love RAM, so a box with 128 GB behaves very differently from a memory-capped pod. A CPU with a large cache (an AMD X3D part, in my case) suits the big working sets that analytics queries chew through. And the money runs the other way from the usual cloud story: a dedicated Hetzner box in the €120/month range outperforms a comparable managed cluster that quotes €400+/month. For an IO-bound column store, dedicated hardware is faster and cheaper — which is rare enough to notice.
What you give up
This isn't free, and pretending otherwise is how people get burned. A dedicated box is a second maintenance domain living outside Kubernetes — its own OS to patch, its own networking to wire back to the cluster. A single node is a single point of failure: until you add a replica, a dead host means downtime, not automatic failover. And nobody hands you backups — you own that end to end.
None of those are dealbreakers if you go in with eyes open. The SPOF is real but bounded, because the raw data lives in the lake (see the wider stack write-up) — a total loss of the box costs you the transformed state and ClickHouse's internals, not the source data. But rebuilding every transformation from scratch is still hours of unplanned work, which is exactly why the backup story below isn't optional.
Backing it up — the part that's easy to get wrong
The single most important thing: do not file-copy the data directory. ClickHouse merges MergeTree parts in the background constantly. Copy the files mid-merge and you get an inconsistent, sometimes corrupt snapshot that looks fine until you try to restore it. A plain cron of cp or rsync over the live data directory is a backup that lies to you.
The correct tool is clickhouse-backup (Altinity's maintained CLI). It issues a FREEZE on the MergeTree parts, which hardlinks a consistent point-in-time snapshot into a shadow path, then tars that and uploads it. Consistent by construction, no corruption, and it understands ClickHouse's on-disk layout instead of fighting it.
Three rules around it that matter as much as the tool:
Put the target off the box. If the host dies, the backup has to be somewhere else to be worth anything. S3-compatible object storage is the obvious primary. A cheap secondary — a weekly rsync to a storage box — covers the case where the object store itself has a bad day.
Automate it, don't remember it. A systemd timer running nightly, with a real retention policy (I keep 14 daily full backups). A backup you have to remember to run is a backup you don't have.
Set an RTO and RPO you actually mean. Mine are a 4–8 hour recovery time and 24 hours of acceptable data loss, which makes a daily backup genuinely sufficient — no need to over-engineer incremental streaming for a workload that tolerates a day.
The credentials for the backup job live in a 0600 config file on the box, separate from whatever profile you use from your laptop, and never in git. That's the boring part that's also the part that leaks if you're careless.
Would I do it this way again?
For an IO-bound analytics engine, without hesitation. Dedicated hardware is the rare choice that's faster and cheaper at once, and the trade-offs — a second thing to patch, a single node, your own backups — are all manageable with a few hours of setup and a nightly timer. Managed ClickHouse Cloud is the right answer if you have no ops capacity; if you do, a dedicated box plus a correct backup job gets you most of the way to a warehouse for a fraction of the price.
Frequently asked questions
Isn't a single ClickHouse node too risky for production?
It's a real single point of failure, yes. The mitigation is two-layered: correct, tested backups so a host loss is recoverable within your RTO, and — when uptime demands it — a second replica for automatic failover. For a lot of internal analytics, where a few hours of downtime is survivable, a well-backed-up single node is a reasonable Phase 1.
Can't I just snapshot the whole machine instead?
A dedicated bare-metal box usually has no hypervisor to snapshot, and even where a VM snapshot exists it isn't ClickHouse-consistent for parts that are mid-merge — the same corruption risk as a file copy. Use a tool that speaks MergeTree; a machine-level snapshot is a false sense of safety.
Why not managed ClickHouse Cloud and skip all this?
Cost and control. Managed removes the ops burden, which is worth real money if you don't have the capacity — but you pay a multiple for it and your data sits on someone else's infrastructure. If you can operate a Linux box and a database, self-hosting is dramatically cheaper and keeps the data yours.
How long does a restore actually take?
It depends on data volume and how fast you can pull from your backup target — which is exactly why you should time a real restore before you need one. An untested backup is a hypothesis, not a recovery plan.