Metadata-Version: 2.5
Name: backup-orchestrator
Version: 0.19.0
Summary: Remote-first backup orchestration client and target-side server
Author-email: Tomasz Sieprawski <tomasz@sieprawski.pl>
Requires-Python: >=3.13
Provides-Extra: dev
Requires-Dist: build==1.3.0; extra == 'dev'
Requires-Dist: copier==9.17.1; extra == 'dev'
Requires-Dist: cryptography==50.0.1; extra == 'dev'
Requires-Dist: pytest-cov==7.1.0; extra == 'dev'
Requires-Dist: pytest==9.0.3; extra == 'dev'
Requires-Dist: reuse==5.1.1; extra == 'dev'
Requires-Dist: ruff==0.15.12; extra == 'dev'
Requires-Dist: twine==6.2.0; extra == 'dev'
Description-Content-Type: text/markdown

<!--
SPDX-FileCopyrightText: 2026 Tomasz Sieprawski <tomasz@sieprawski.pl>
SPDX-License-Identifier: LicenseRef-Proprietary
-->

# backup-orchestrator

Remote-first backup orchestration client and target-side server.

Current scope:
- installable Python package with local and SSH-remote orchestration commands
- lease wait, renewal, release, and rest-server lifecycle handling
- backup, snapshots, restore, and prune operations
- explicit password modes and optional persistent restic cache plumbing
- unit and integration tests
- release script and post-merge release workflow
- Forgejo workflows for checks, integration, release, and conventional commit validation

## Local Usage

Run checks:

```bash
./scripts/check.sh
```

Run the integration test:

```bash
./scripts/test-integration.sh
```

Run the release integration test:

```bash
./scripts/test-integration-release.sh
```

Run the manual restic and Borg benchmark harness:

```bash
./scripts/test-benchmark-restic-cache.sh --runs 5
```

Notes:

- this is a manual benchmark tool, not a normal CI test
- it runs real local `restic` and `borg` repositories against the same synthetic datasets
- it writes JSON output under `./benchmark-results` by default
- use it first on one machine locally, then reuse it on gator and zotac for cross-host measurements
- practical repetition guidance:
  - smoke or harness validation: `--runs 3`
  - first comparative pass: `--runs 10`
  - cheap scenario percentile runs: `--runs 30` to `--runs 100`
  - expensive large-repository scenarios: usually `--runs 5` to `--runs 20`
- current default decision-focused variants are:
  - restic: `default`, `cache-off`, `group-by-paths`, `explicit-parent`, `no-scan`, `ignore-ctime`, `compression-off`, `compression-max`
  - borg: `default`, `files-cache-disabled`
- restic runs use `--read-concurrency 4`; inode checking remains enabled because production sources are not FUSE or pCloud mounts
- use `--scenario-prefixes huge` for the most decision-relevant large Nextcloud-like comparisons
- for light local dry runs, keep `--huge-target-gib` modest and use a native Linux filesystem workdir
- for serious host runs, use a large disposable workdir on representative direct local storage; the completed `100 GiB` gator run is the current practical scale reference, while the slow USB `100 GiB` attempt exceeded 16 hours and larger runs should not be assumed feasible
- large runs need significantly more free workspace than the nominal `--huge-target-gib` target because source data, repository growth, cache state, temp files, and tool-specific working state coexist during the benchmark; plan for a substantial multi-times space multiplier rather than target-size-only headroom

Profile a real source tree before changing synthetic benchmark data:

```bash
python3 scripts/profile-backup-dataset.py /path/to/source --output source-profile.json
```

The profiler is read-only. It reports file-count, byte, extension, size-bucket, top-level-directory, and sampled content entropy/compressibility statistics to guide synthetic benchmark data design.

Run the CLI smoke command:

```bash
python -m backup_orchestrator
backup-orchestrator scaffold
```

The package also provides these operation families:

- `server`, `repos`, `repo`, `claim`, `renew`, and `release` for target-side
  repository and lease control;
- `backup`, `snapshots`, `ls`, `restore`, and `prune` for local rest-server sessions;
- `remote backup`, `remote snapshots`, `remote ls`, `remote restore`, and
  `remote prune` for SSH-tunneled target-side sessions;
- pass `--include-file PATH` to `restore` for a one-off partial restore;
- pass `--dry-run` to `restore` to preview the selected restore without writing files.

The downstream partial-restore wrapper accepts the same optional `--dry-run`
argument after its include-file argument.

Restore include files contain restic include patterns, not JSON or a verbatim
path list. Paths containing restic wildcard characters such as `*`, `?`, or
`[` must be escaped according to restic's pattern rules.

For remote usage, the target host must provide a configured
`backup-orchestrator` server and local wrapper command. The client requires an
SSH target, a known-hosts file, and the target server port. Use the downstream
adoption reference for the complete container and workflow contract.

Restic cache configuration:

- pass `--restic-cache-dir /path/to/persistent/cache` to local or remote
  `backup`, `snapshots`, `ls`, `restore`, or `prune`
- the value is exported as `RESTIC_CACHE_DIR` to the restic process
- downstreams own the persistent path, capacity, and container mount
- omit the option to retain restic's no-explicit-cache behavior
- `backup-orchestrator 0.17.0` is the first released version containing this
  cache contract

Downstream remote adoption:

- see [`docs/downstream-remote-adoption.md`](docs/downstream-remote-adoption.md)
  for the client-wrapper, target, workflow, cache, SSH container-path, and
  integration-test contract
- the Copier template and adoption tests are available under
  [`templates/remote-downstream`](templates/remote-downstream) and
  [`tests/test-copier-adoption.sh`](tests/test-copier-adoption.sh)

### Copier Template

`templates/remote-downstream/` is a Copier template for a downstream repository
that runs a local `backup-orchestrator` client in Docker Compose. The generated
client connects over SSH to the target host, uses the configured repository
name, and includes:

- backup, snapshots, ls, restore, partial restore, and prune wrappers;
- SSH, cache, source, and restore mounts;
- target-side server and local-wrapper examples;
- manual and scheduled Forgejo workflows;
- static checks and a real Forgejo workflow smoke test.

Install it into a downstream repository:

```bash
python3 -m pip install 'copier==9.17.1'
git clone https://code.sieprawski.pl/tsieprawski/backup-orchestrator.git backup-orchestrator-template
python3 -m copier copy --trust \
  backup-orchestrator-template/templates/remote-downstream \
  /path/to/downstream-repository
```

Answer the Copier questions, review the generated diff, and review the source,
repository, SSH, cache, restore, and scheduling values before using the
workflows. Copier writes `.copier-answers.yml` in the generated repository.

The generated workflow smoke test uses the pinned `setup-forgejo` fixture, but
that fixture is test infrastructure and is not generated into the downstream
runtime. Add it after generation:

```bash
cd /path/to/downstream-repository
git submodule add \
  https://code.sieprawski.pl/tsieprawski/setup-forgejo/ \
  vendor/setup-forgejo
git submodule update --init --recursive
```

Run the generated tests before enabling production scheduling:

```bash
./tests/test-adoption-contract.sh
./tests/test-forgejo-workflows.sh
```

## Licensing

All code in this repository is proprietary.
All rights reserved.

## Release

Merges to `main` are intended to trigger a package release to the local Forgejo
PyPI registry.

Version bumps are driven by conventional commits:
- breaking changes: major
- `feat`: minor
- `fix`, `ci`, `refactor`, `test`: patch
- `docs`, `chore`: no release
