Designing a Self-Hosted Backup Orchestrator with Go, Redis, and Rsync

Most backup setups I run into end up at one of two extremes.

On one side: raw Unix primitives (rsync, tar, cron, SSH). They work, but they don’t give you a control plane. You end up with a growing collection of scripts, state passed around via the filesystem, and a bunch of “run this, then check that, then tail this log” operator rituals.

On the other side: big backup suites that do everything, but often at the cost of opacity (harder to debug), proprietary storage formats (harder to exit), or just more moving parts than I want for a small self-hosted setup.

Design goal: I wanted something in the middle: a small system that stays close to standard tools, but behaves like a system (state, retries, scheduling, visibility, and restore workflows).

That project became JustBackup: a Go backup orchestrator that uses SSH and rsync for data movement, Redis for async task dispatching, and boring service boundaries to keep it maintainable as features grow.

If you want to inspect the implementation directly, the code is available on GitHub.

What it actually solves (and what it doesn’t)

This is not a “new way to copy files”. rsync already does that extremely well.

The real problem is making backups operable:

• define backup jobs as structured config, not conventions
• schedule and run them predictably (and idempotently)
• execute work asynchronously (backups are slow; UIs shouldn’t be)
• expose the same capabilities via API/CLI/UI
• treat restore and inspection as first-class workflows
• keep retention and hooks inside the same model

The value is simple: standard data movement underneath, explicit orchestration around it.

Design constraints: open tools, open formats, no trap doors

The interesting part for me is not Go vs. Rust vs. Python. It’s the constraint: everything should remain recoverable and inspectable without JustBackup.

Concretely:

• The data plane is standard: SSH (an open, ubiquitous protocol) + rsync.
• The storage model is intentionally “boring”: directories, files, and symlinks you can inspect with ls, find, du, rsync, and friends.
• No proprietary backup format that forces you to keep running the orchestrator forever to read your own backups.
• Redis is an off-the-shelf, open-source queue boundary, not an attempt to build a custom workflow engine.

If the app disappears, the backups should still be useful.

1. Script sprawl around routine backups

Most self-hosted backup systems start as a shell script and slowly turn into a pile of shell scripts: run, list, restore, purge, rotate, verify… plus a set of undocumented conventions held together by habit.

In this project, backup jobs are modeled as first-class domain entities with explicit fields such as:

• source path
• destination
• schedule
• exclude patterns
• incremental mode
• retention
• encryption
• pre/post hooks

Once those concerns are explicit, you can validate them, schedule them, expose them via HTTP, and execute them consistently. It also makes “diffing” behavior possible: the config becomes the contract.

2. Tight coupling between execution and user interaction

Long-running infrastructure work should not block the interface that triggers it.

Backups, file searches, restore operations, size calculations, and purge jobs are all asynchronous tasks. The user-facing layer should submit work and check status, not execute remote filesystem operations inline inside a request.

That’s why JustBackup uses Redis as a queue boundary. The API and scheduler publish work; workers consume tasks; results are handled separately. This keeps the execution model simple while still allowing the system to scale beyond a single synchronous process.

3. Operational friction around restore and inspection

A backup system that can only “store files somewhere” is incomplete.

In practice, people need to:

• browse stored files
• search across backup contents
• restore locally
• restore to another remote host
• understand whether a task failed and why

This project treats those workflows as part of the product, not as follow-up shell commands left to the operator.

4. Hidden maintenance work

Incremental backups solve one problem and create another: retention and cleanup.

JustBackup includes maintenance tasks for purge workflows so the system does not stop at “we created snapshots” and ignore the cost of keeping them forever. I care about that kind of completeness because operational systems usually fail at the edges, not at the happy path.

System shape

The architecture is intentionally pragmatic. I’m not optimizing for academic purity; I’m optimizing for boundaries that stay useful once the codebase grows.

Domain and application boundaries

The codebase is split into modules such as backup, auth, notification, maintenance, workerstats, and user, with domain interfaces separated from infrastructure implementations.

This is the structure I like in production systems:

• domain entities represent the core concepts
• application services orchestrate use cases
• infrastructure adapters handle Redis, Postgres, HTTP, crypto, and external processes
• the same use cases can be exposed through different interfaces without rewriting the core logic

This is also why the repository can switch between in-memory repositories in development and Postgres-backed repositories in production without rewriting service-level behavior.

That kind of separation is the same design pressure I wrote about in How I Structure a Fraud Detection Side Project So Iteration Stays Cheap: if the boundaries are honest, change stays cheaper for longer.

Async orchestration with Redis

The core execution flow is simple on purpose.

1. The API or scheduler decides that work needs to happen.
2. A publisher serializes that work into a WorkerTask and pushes it to Redis.
3. A worker consumes the task and executes the remote operation.
4. The worker publishes a result message.
5. A result consumer updates backup state, stores errors, emits domain events, and pushes live updates to the UI.

That pipeline is not exotic, but it’s effective.

It decouples orchestration from execution while keeping the transport understandable. I prefer this kind of architecture when the goal is reliability and operability, not novelty.

Backup execution flow

The worker path is where the design becomes more interesting.

A backup task follows a clear sequence:

1. prepare the destination
2. create an ephemeral workspace when needed
3. execute pre-backup hooks
4. run rsync
5. optionally compress and encrypt the artifact
6. execute post-backup hooks
7. calculate artifact size and publish the result

That explicit sequence matters for two reasons.

First, it makes failures easier to reason about because the execution stages are visible and bounded.

Second, it allows the system to support more than a simple directory copy. Hooks make it possible to back up data that must be prepared before transfer, such as generating a database dump into a temporary workspace before the actual synchronization step.

One detail I particularly like is the use of an ephemeral source path for internally generated artifacts. That allows the system to model “create backupable material first, then sync it” inside the same workflow instead of splitting it across unrelated scripts.

Incremental backups without a proprietary storage model

Another design choice was to keep the storage model understandable.

Incremental mode uses timestamped directories plus a latest symlink rather than introducing a complex custom format. When encryption is enabled, the workflow produces a compressed encrypted artifact and updates the relevant latest pointer.

That keeps the system close to standard filesystem semantics while still supporting useful operational behavior. It also means you can browse and copy the data without needing a special restore tool.

I tend to prefer this approach when building infrastructure products: push complexity into orchestration and safety, not into making the stored data impossible to inspect without the application.

Real-time visibility and event-driven side effects

The result consumer does more than just mark a task as finished.

It also:

• updates backup state
• persists execution errors
• emits domain events such as backup completed or backup failed
• broadcasts WebSocket notifications to connected clients

This makes the UI and notification layer reactive without forcing the core backup logic to know about each downstream concern.

That is a small but useful example of where event-driven design helps: the backup process should focus on backup behavior, while notifications and live status updates remain attached but decoupled.

Multiple operator surfaces from the same backend

One of the things I wanted from the beginning was for the system to be usable in more than one way.

So the project exposes the same core capabilities through:

• an HTTP API with OpenAPI/Swagger documentation
• a CLI for operator workflows such as bootstrap, run, search, files, and restore
• a web dashboard

This matters because infrastructure tools live in different contexts. Sometimes you want a browser. Sometimes you want a terminal. Sometimes you want automation through an API.

A backup platform should not force one interaction model when the underlying capabilities are the same.

Trade-Offs

This design is practical, but not free.

A few trade-offs are worth stating directly.

• Redis queues are simple and effective here, but they do introduce another runtime dependency.
• Keeping the transport lightweight means task durability and replay semantics are not as rich as in heavier workflow engines.
• Using rsync and SSH keeps the system transparent, but it also means remote host access, permissions, and network conditions remain part of the operational model.
• Supporting hooks makes the system much more flexible, but it also increases the need for disciplined execution contracts and observability.

I’m comfortable with those trade-offs because they match the goal. This is not trying to be a universal enterprise backup suite. It is a clean, operable system built on standard primitives, and you can walk away from it without losing access to your own data.

Why This Project Matters for My Profile

I do not think projects are useful as portfolio pieces just because they are technically broad. They become useful when the design decisions reveal how you think.

What I like about this repository is that it combines several concerns I care about professionally:

• modeling operational workflows as explicit application services
• designing asynchronous systems without overcomplicating the message flow
• building around open standards and existing tools instead of hiding everything behind abstraction theater
• exposing the same backend capability through API, CLI, and UI
• balancing delivery pragmatism with enough architectural structure to keep the codebase maintainable

That combination is close to how I work in real teams: backend-heavy, systems-aware, product-minded, and biased toward designs that stay understandable under change.

Conclusion

JustBackup started from a narrow question about backups, but the more interesting engineering question is broader: how do you turn a set of reliable low-level tools into a coherent operational product?

For me, the answer was to keep the data movement simple, make the workflows explicit, separate orchestration from execution, and design the system so that operators can interact with it from more than one surface.

That’s the kind of engineering work I enjoy most: not replacing solid primitives, but composing them into systems that are easier to run, easier to extend, and easier to trust. And doing it in a way that doesn’t trap you in a proprietary format or a vendor-specific control plane.