- Rust 99.4%
- Makefile 0.3%
- Dockerfile 0.3%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| config | ||
| docs | ||
| src | ||
| tests | ||
| .dockerignore | ||
| .env.example | ||
| .gitignore | ||
| Cargo.lock | ||
| Cargo.toml | ||
| compose.beszel.yaml.example | ||
| compose.build.yaml | ||
| compose.yaml | ||
| Dockerfile | ||
| Makefile | ||
| README.md | ||
| rust-toolchain.toml | ||
Homelab Analyzer
A deterministic, self-hosted, read-only daily diagnostics CLI written in Rust 2024 (Rust 1.96+). Prometheus supplies numeric time series; optional Beszel integration supplies system inventory context. The operator runs the CLI and can review its reports in ChatGPT. No LLM, daemon, web server, remediation, infrastructure credentials, SSH client, or Docker socket is required by the analyzer.
Architecture
Operator -> CLI -> YAML rules
-> GET Prometheus range API -> pure statistical analysis
-> GET Beszel systems (optional context)
-> SQLite run + anomaly history
-> JSON + Markdown reports
analysis.rs contains deterministic analysis independent of HTTP and persistence;
timeline.rs (slots, intervals, counter increases), quality.rs (data-quality
causes), alerts.rs (Prometheus alerts) and incidents.rs (incident lifecycle) are
pure helpers it uses. insights.rs builds the executive summary and recommendations.
source.rs implements bounded HTTP timeouts, read-only data requests, a
restricted Beszel authentication POST, and synthetic signals. storage.rs commits runs and deduplicated observations transactionally.
report.rs writes each report atomically with an immutable run filename.
Local quick start
cd ~/Code/Rust/homelab-analyzer
cargo build --release --locked
./target/release/homelab-analyzer analyze --demo
./target/release/homelab-analyzer report --demo
./target/release/homelab-analyzer report --demo --format json
./target/release/homelab-analyzer healthcheck --demo
The synthetic scenario intentionally returns exit code 2: it includes a spike, 50% coverage, and a missing expected host. The spike is reported as an incident and as two statistical anomalies. No HTTP requests or service secrets are read in demo mode. Its default end time is fixed for reproducibility. Synthetic and live databases and report directories are separate. The complete demo can be run without LAN access once dependencies are installed.
The application does not auto-load .env locally. Export environment variables in
your shell; Compose loads .env. For example, in Bash:
cp .env.example .env
# Edit .env with verified LAN URLs; review config/queries.yaml for your inventory.
# Review the file before sourcing it into your shell.
set -a
source .env
set +a
./target/release/homelab-analyzer healthcheck
./target/release/homelab-analyzer analyze --format json
./target/release/homelab-analyzer report --format markdown
Commands: analyze, report, healthcheck and export (see "Reviewing reports in
ChatGPT"). All commands accept --config PATH, --data-dir PATH, --demo, and --format
(markdown, markdown-full or json). markdown is a compact summary for people
and ChatGPT: incidents, alerts, deviations, grouped data-quality causes and
recommendations. markdown-full adds per-series comparison and baseline tables.
The saved .md file is compact; the saved .json file has every detail.
The executable also provides --help, per-command help, and --version.
analyze --end 2026-10-27T00:00:00Z allows reproducible backfills. Explicit end
times require a timezone and are rounded down to the configured evaluation step.
report reads the most recent saved report without accessing any remote service.
healthcheck verifies query API access and optional Beszel inventory access;
it does not claim that monitored hosts or backups are healthy. It does not create
or check the SQLite database. analyze verifies local write access as it saves.
Exit codes: 0 = completed with adequate data (incidents may still be present),
1 = configuration/local storage/fatal source failure, 2 = degraded data or
optional source failure. With exit code 2 the JSON and Markdown reports are still
saved; stderr prints the report path and every reason from
data_quality.degraded_reasons. report returns 0 when an existing report is rendered,
even when that report describes degraded data. Consumers must inspect status,
executive_summary and active_incidents, not just the exit code. Status ok
describes data quality, not infrastructure health. Reports include schema_version: 2;
version 1 reports saved earlier remain readable.
Before v2 every live run exited 2: Prometheus keeps 15 days, so the 30-day baseline
never reached 80% coverage, and any unavailable baseline marked the run degraded.
Insufficient history is now a non-degrading insufficient_history notice.
Verify metrics before adding infrastructure rules
The default configuration is the verified rimonome profile:
27 verified rules for scrape availability/freshness, manwe/rimonome resources, Proxmox
storage, yane ZFS, SMART, Restic and firing Prometheus alerts, plus three v2 counter
rules for ZFS vdev and SMART error increases and Prometheus alert collection. Expressions and
expected labels were checked against the running API; the profile is specific to
this inventory. config/canary.yaml retains the generic vector(1) query API check.
Review the profile before using it in another environment. Static thresholds are
initial operator policy choices and are documented separately from observed facts.
- Set the actual Prometheus LAN URL and verify
healthcheck. - Inspect
GET /api/v1/label/__name__/valuesand the Prometheus query UI to find metrics actually exported by manwe, yane, Restic, and Alertmanager. - Inspect label sets and units using
GET /api/v1/query?query=..., then check history and gaps in the UI before committing each query. - Update the profile with verified gauge expressions, recording rules, or
rate/increaseexpressions for counters. Avoid raw cumulative counters. - Run a first analysis and inspect coverage, baseline availability, and report cardinality before using the report for daily review.
Example schema using a deliberately non-executable placeholder:
step_seconds: 900
rules:
- id: verified_capacity
query: REPLACE_WITH_VERIFIED_PROMQL
unit: percent
lower: 0
upper: 90
direction: high
mad_multiplier: 6
min_delta: 5
min_coverage: 0.8
min_baseline_samples: 24
breach_fraction: 0.05
stale_after_seconds: 1800
expected_labels: []
severity: warning # critical | warning | info for its incidents
counter: false # true: evaluate per-step increases of a cumulative counter
optional: false # true: absent series are notices, not degraded data
recommendation: null # operator text shown when the rule raises an incident
alerts: {enabled: true} at the top level reads GET /api/v1/rules?type=alert
and the ALERTS{alertstate="firing"} range for the current window. Label and
annotation values whose names contain password, passwd, secret, token, apikey,
api_key, authorization, cookie, credential, private_key or session are replaced by
[redacted], and user:password@ is removed from URLs. alerts.exclude_labels: [name]
drops further names entirely. Redaction happens before analysis, so secrets reach
neither reports nor SQLite.
Counter rules (counter: true) replace each value by its increase since the previous
slot. A decrease is a reset (reboot, zpool clear): it adds nothing, its time is
recorded in reset_at, and the post-reset value becomes the new reference, so a reset
is never reported as new errors. Errors counted between a reset and the next sample are
therefore not included; use node_boot_time_seconds if that matters. The current day continues
from the last 7d sample, so an increase at midnight is not lost. An incident is raised
when the window's total increase exceeds upper (default 0); lower is rejected.
MAD comparisons are skipped for counters; series[].counters reports start value, end
value, increase and resets for each window. Absolute values remain visible there.
Each breach interval carries its measured values (observation.values: value before
the interval, first, last, min, max and counter increase), and each incident has its
own values aggregated from its intervals, so two incidents on one counter never
show the same window total.
Counter fields and their scope
| Field | Scope |
|---|---|
incident.counter.incident_increase |
Increase inside this incident's breach intervals across all saved reports. Each evaluation slot is counted once: a later report adds only slots after the previous report's end, so overlapping windows are never summed twice. null for incidents saved before this field existed and starting before the current window. |
incident.counter.incident_increase_in_window |
The part of the incident's increase inside the current 24h window. |
incident.counter.window_total_increase |
Increase of the whole series in the current window, shared by every incident on that series. |
incident.counter.counter_resets, reset_at |
Real decreases of the series in the current window (reboot, zpool clear). They add nothing to any increase and never raise an incident. |
finding.details.window_total_increase, counter_resets, reset_at |
Same window totals on the series-level finding. increase and resets remain as deprecated aliases for existing consumers. |
incident.details |
Unchanged copy of the finding details, so for counters it holds window totals, not the incident's own values. Use incident.counter and incident.values. |
series.windows.current (counter rules) |
Statistics of per-step increases, used for coverage and baselines. |
series.raw_current (counter rules) |
Statistics of the raw cumulative counter values in the current window. |
series.counters.current |
Window start and end value, total increase, resets and reset times. |
In the export, metrics.json (schema homelab-analyzer-metrics/2) names the series
behind every statistics block: sample_statistics.series is value for gauges and
counter_value (raw cumulative values) for counters, per_step_increase holds the
per-step statistics of counters, and the 7d/30d blocks state series too. A counter
that rose from 2 to 4 in two steps therefore shows sample_statistics.max = 4,
per_step_increase.max = 1 and counter.window_total_increase = 2.
Unknown YAML fields, duplicate IDs, reversed bounds and nonfinite thresholds are
rejected. Label identity uses the complete returned label set, including
__name__ when present. expected_labels must contain exact complete label sets;
it is not a selector. Use it to detect a host absent from all retained history.
Series present in either baseline are checked even if absent from the current day.
New labels with no baseline remain explicitly unavailable for historical comparison.
Statistical assumptions and missing data
For analysis end T and step S, the current window is [T-24h, T). Baselines are
[T-8d, T-24h) and [T-31d, T-24h): 7 and 30 full preceding days, excluding the
current day. Range queries use the inclusive endpoint window_end-S to avoid
overlapping samples. Default S is 15 minutes; supported steps are 300–3600 seconds and must divide 86400 exactly.
Queries execute sequentially, with a 30-second server query timeout, a
35-second client timeout, a 10-second connection timeout, and a 16 MiB response limit.
The application uses blocking HTTP, without an application async runtime; reqwest
uses Tokio internally as an implementation detail. Keep query cardinality and configured rule count small.
Coverage is the number of unique finite evaluation slots divided by expected slots. NaN, infinity, off-grid and out-of-window samples do not count. No values are imputed or interpreted as zero. Native histograms are unsupported: convert them to scalar PromQL; responses containing them are marked degraded.
For each label set, the analyzer reports count, coverage, median, MAD, min, max,
first and last finite sample, and the window range. A baseline is usable only with
sufficient coverage and samples and no API errors or annotations. Short retention is
not extrapolated. An unavailable baseline is never an anomaly: its comparison is
skipped and unavailable_reason says why (source_issue, no_history,
insufficient_coverage, insufficient_samples). data_quality.baselines lists the
range and sample span used for each rule's 7d/30d baseline.
Data quality
data_quality.status is degraded only when degraded_reasons is non-empty; each
reason names its code, rule, labels and window:
| Code | Meaning | Degrades |
|---|---|---|
source_error / source_warning |
Prometheus query failed or returned annotations | yes |
exporter_unavailable |
series missing while its job/instance target was down or absent in up |
yes |
no_samples |
target was up (or unknown), but the series has no samples | yes |
incomplete_data / stale_data |
coverage below min_coverage or last sample too old |
yes |
alerts_unavailable / beszel_unavailable |
optional source failed | yes |
insufficient_history |
baseline lacks history (for example 30d with 15d retention) | no |
optional_metric_absent |
rule marked optional has no series |
no |
target_lookup_failed |
up lookup failed; missing-data causes unknown |
no |
up is queried once per run, only when some series lacks data.
data_quality.groups merges issues with the same code and window into one entry
with the affected rules and the gaps they share. Baseline gaps of at least one hour
are detected from the samples (gaps on each 7d/30d summary and on
data_quality.baselines) and classified as before_first_sample (retention or a
new series) or collection_gap (samples stopped mid-window, for example while
Prometheus was down). The per-rule issues remain in degraded_reasons and notices.
Incidents, anomalies and alerts
Static threshold breaches, counter increases and Prometheus alerts are incidents:
each has a stable id, severity, status (active/resolved), and two separate
parts. history is the lifecycle across saved reports: started_at, ended_at,
duration_seconds, total_breach_seconds, number of reports. current_window is
the evidence measured in this report only: breach intervals, breach seconds, and
whether the breach is ongoing at the window end.
Breach intervals come from evaluation samples: an interval starts at the first
breaching sample and ends at the next slot that holds a non-breaching sample or no
sample, so times have the resolution of step_seconds. Availability rules use
min_over_time(...[15m]), so a failure may have begun up to 15 minutes before the
reported start.
Breaches separated by at most incident_gap_seconds (top-level config, default 3600)
form one incident; a longer recovery ends it and the next breach opens a new
incident with a new id. 0 splits at every recovery. history.span_seconds runs
from the first breach to the last breach end (or the window end while active);
history.total_breach_seconds counts breach intervals only, breach_intervals
counts them, and pattern is continuous or intermittent. A flapping target
therefore reads as "intermittent: 7h 45m breach in 26 intervals over a 22h 30m
span", never as one uninterrupted outage. current_window contains only the
intervals that belong to that incident.
An incident continues into the next report when its next breach starts within the
gap, or exactly where an unobserved period between reports ended; a resolved
incident reopens with the same id under the same rule. Overlapping windows are not
counted twice. It resolves when a later
report evaluates the series without a breach. A series without samples keeps the
incident active, marked observed_in_current_window: false. Incident identity is the
source, rule, labels and kind, without configuration, so threshold tuning keeps the
id. Backfills of older windows do not rewrite lifecycles (tracking: window_only).
The JSON report contains executive_summary (measured facts only),
active_incidents, resolved_incidents (resolved by this run),
historical_incidents (resolved earlier, last 30 days, up to 20), anomalies
(MAD deviations, severity info), alerts (firing, pending and resolved within the
window, with labels, severity and annotations), data_quality and
recommendations. Recommendations are marked basis: operator_rule or
basis: heuristic; they are never measurements. findings and rules keep the
complete per-series detail. Alert annotations come from the active alert or, for
alerts that ended, from the rule template (annotations_source); values are
truncated to 1000 characters.
A historical outlier satisfies:
abs(sample - baseline_median) > max(min_delta, mad_multiplier * 1.4826 * baseline_MAD)
direction restricts the sign of the deviation. A finding requires at least
breach_fraction of observed current values to exceed the distance. Zero MAD
uses min_delta, with strict inequality; configure a meaningful floor in the
query's units. Historical checks require sufficient current coverage. Static
upper/lower checks use the same breach fraction and still flag observed breaches
on partial data, explicitly reporting partial confidence. Missing or stale data
is reported independently. Thresholds are strict (> / <).
This is an aggregate distribution baseline, not a forecasting model. It does not
adjust for weekday, time-of-day, seasonality, counter resets in raw counters, or
correlated samples. Median shifts and isolated spike fractions appear in the
report, but brief events below the configured breach fraction can be missed.
Prometheus range-query coverage represents evaluation availability, not raw
scrape completeness: lookback can reuse old samples. Add verified timestamp,
absent_over_time, or scrape-gap expressions for exporter freshness checks.
Prometheus warnings/info annotations make a source degraded and disable the corresponding historical baseline. Their counts are retained rather than raw messages, to avoid leaking sensitive upstream content. Transport/API errors are sanitized and included per window. One failed rule does not discard other results.
Beszel integration boundary
Official Beszel documentation confirms PocketBase APIs and the systems
collection, and warns that schemas may change in minor releases. This MVP uses
only GET /api/collections/systems/records, requesting id,name,status, with
pagination capped at 100 pages of 100 records. It validates the returned fields.
Unknown schema, authentication failures, and pagination overflow degrade the
report rather than inventing a healthy result.
Set BESZEL_URL and choose exactly one authentication mode:
- Account login: set
BESZEL_EMAILandBESZEL_PASSWORD, or useBESZEL_PASSWORD_FILEfor a mounted secret. Use a dedicated regular Beszel user with the Read only role and share the monitored systems with that account. - Existing user token: set
BESZEL_TOKENorBESZEL_TOKEN_FILE. Supply the raw PocketBase user token without aBearerprefix. Its expiry is still the operator's responsibility in this mode.
Example account configuration in the untracked .env:
BESZEL_URL=http://10.0.20.11:12900
BESZEL_EMAIL=analyzer@example.com
BESZEL_PASSWORD='replace-with-account-password'
Remove BESZEL_TOKEN and BESZEL_TOKEN_FILE when using an account. Do not set both
BESZEL_PASSWORD and BESZEL_PASSWORD_FILE. Conflicting modes or partial account
credentials are configuration errors, with no login attempt. Password contents
are not included in error messages. Password files preserve significant spaces
and remove only one final LF or CRLF line ending.
Account mode logs in once on every analyze or healthcheck invocation through
POST /api/collections/users/auth-with-password, then uses GET requests for system
inventory. This is the only permitted POST endpoint; no systems, alerts, metrics,
users, or infrastructure are modified. The resulting token stays in memory for
that invocation and is never written to the report or database. There is no need
to copy or renew a browser token between daily runs. Authentication failures,
including MFA challenges requiring interaction, produce a degraded Beszel result
without discarding the Prometheus report. No automatic retry is made for failed
credentials, and redirects are never followed. A token rejected during inventory
is reported as a source failure rather than triggering repeated login attempts.
report reads saved output without logging in; --demo never reads credentials
or makes authentication requests.
For a file-based password, copy compose.beszel.yaml.example to
compose.override.yaml and use this .env configuration instead:
BESZEL_URL=http://10.0.20.11:12900
BESZEL_EMAIL=analyzer@example.com
BESZEL_PASSWORD_FILE=/run/secrets/beszel_password
Create secrets/beszel_password locally with the account password. Keep it out of
Git and readable by runtime UID 1001; the optional Compose override mounts it as a
secret. Setting _FILE alone does not mount a host file into the container. Do not
set BESZEL_PASSWORD in file mode. The override is loaded automatically by Docker
Compose and is ignored by Git.
After publishing the updated image from the PC and updating configuration:
make pull
make healthcheck
make
Check that beszel.status is ok and systems contains the expected hosts. An
empty list can indicate that no systems were shared with the user. Creating a user
does not automatically grant access to existing systems. In the PocketBase admin
panel (/_/), open the systems collection, edit each intended system and add the
analyzer account to its users field, preserving existing users. See the
Beszel system-sharing instructions.
After changing access, run make healthcheck and then make (or make analyze)
to collect a fresh report. make report only reads the last saved result; rebuilding
the image or running a healthcheck does not update that saved report. Do not use
superuser credentials or the agent registration/universal token for this adapter.
Beszel inventory remains context only: its numeric historical collection names
and fields have deliberately not been guessed. Prometheus remains the numeric
analysis source.
Sources checked during implementation:
- Prometheus HTTP API
- Beszel REST API and stability notice
- PocketBase records API
- PocketBase token authentication
- Beszel readonly roles
Persistence and deduplication
Local SQLite stores immutable JSON runs, anomaly fingerprints, and unique run/fingerprint observations. Fingerprints include source identity, normalized configuration, rule ID, complete labels, and finding kind. The first/last seen analysis timestamps and occurrence count are attached to findings.
A run ID hashes the report before persistence metadata. Identical retries do not
increment counts; corrected or late data yields a new run. Backfills preserve
minimum first-seen and maximum last-seen timestamps. Counts represent distinct
observed runs, not days. Changing configuration deliberately starts a new anomaly
identity. Finding history is observation history; incident lifecycle is stored separately in
the incidents table (see above). No notifications or Alertmanager writes occur.
Reports live at data/reports/{live,synthetic}/<run_id>.{json,md}. Each file is
atomically replaced; readers should use the SQLite-backed report command if
both report files have not yet been written after a crash. Rerunning an identical
analysis repairs missing files. No automatic retention/deletion is performed;
back up and rotate the data volume externally.
Publishing on the PC and running on rimonome
make publish builds the Dockerfile on the PC and pushes the image to the internal
Forgejo registry. Rimonome pulls and runs the finished image without compilation.
Both hosts were verified as x86_64; build and runtime use linux/amd64.
One-time publisher login (Docker prompts for the password or access token):
docker login git.manwe.xyz
On the PC:
make publish
# Optional named release instead of latest:
make publish IMAGE_TAG=v0.1.1
The default image is git.manwe.xyz/akasiek/homelab-analyzer:latest. Override
IMAGE_REPOSITORY or IMAGE_TAG through the environment, .env, or Make command-line
variables. Publishing a named tag does not also update latest. Build and push
use the same Compose configuration and selected image reference. A failed build
stops the target before push. Publishing builds the current working tree, so commit
changes first when you want the published contents traceable to Git.
compose.build.yaml is a standalone build/publish configuration and does not load
runtime environment files or secret mounts. .env is still read by Compose for
image-variable interpolation when present. Registry credentials are managed by
Docker login and are never supplied as build arguments, copied into the image,
or added to the application .env. Only make publish pushes to the registry;
make build builds and tags the image locally.
Forgejo documents its image naming, login and push workflow in the container registry guide. Private packages also require a Docker login on rimonome using credentials with pull access. The repository-linked image source label is included in the image.
Docker Compose and LAN-only operation
On rimonome after committing/pushing the source changes and publishing the image:
cd ~/docker/homelab-analyzer
git pull --ff-only
# First installation only: copy .env.example to .env and configure LAN endpoints.
# For a private package, first run: docker login git.manwe.xyz
mkdir -p data
make pull
make
make pull downloads the image selected by IMAGE_REPOSITORY and IMAGE_TAG.
The default Compose file contains no build configuration. Normal commands use
--pull never: they run the already selected local image without an implicit
network update. Run make pull explicitly when adopting a new image. A first
installation therefore needs a pull before analysis. To pin a release, set
IMAGE_TAG in the server .env before pulling; use the same tag that was published.
Retain older published release tags when you need rollback.
make # Analyze using the downloaded image.
make demo # Analyze synthetic data using the downloaded image.
make report # Read the latest saved Markdown report.
make report ARGS='--format json'
make healthcheck # Check source connectivity; no build or pull.
make export # Analyze once and write exports/homelab-<UTC time>.zip.
make export-json # Same, JSON files only (no summary.md).
make export-summary # Same, summary.md and manifest.json only.
make pull # Download the selected published image.
make build # PC: build and tag locally, without publishing.
make publish # PC: build and publish the selected tag.
Additional analyzer options can be passed through ARGS, for example
make ARGS='--end 2026-10-27T00:00:00Z'. The Makefile preserves command failures:
an analyzer exit code of 2 (degraded data) is also reported by make as Error 2.
This still means the analysis generated a report; it is not a build failure.
The data volume remains available after the temporary container is removed.
Runtime Compose overrides, including the optional Beszel password-secret mount,
are applied automatically. The build/publish Compose file deliberately excludes
those runtime overrides. Rule files ship inside the image (/app/config), so
make publish on the PC followed by make pull on rimonome also updates the
configuration. To test rules without rebuilding, add
./config:/app/config:ro to the service volumes in a local
compose.override.yaml; while that mount exists it hides the image's rules. The current ./data:/data bind mount and UID/GID are preserved during this image-workflow migration.
Equivalent direct Compose commands for an already configured server:
docker compose pull analyzer
docker compose run --rm --pull never analyzer healthcheck
docker compose run --rm --pull never analyzer analyze
docker compose run --rm --pull never analyzer report --format json
# Synthetic smoke scenario (expected exit 2):
docker compose run --rm --pull never analyzer analyze --demo
This is a one-shot service; use make for each daily review. Do not use
compose up -d as a scheduler. No automatic schedule is installed. The analyzer
container has no access to Docker or SSH credentials. Publishing an image does
not update or restart anything on rimonome.
Reviewing reports in ChatGPT
cd ~/docker/homelab-analyzer
make export
make export runs one analysis (exactly like make, including saving it to SQLite)
and writes a single archive, for example exports/homelab-20261104-234500.zip, to
the host's ./exports directory (Compose mounts it at /exports; the Makefile creates
it, and it must be writable by UID 1001 like ./data). Upload that ZIP to ChatGPT.
The command prints the archive path; the exit code follows analyze (2 = degraded
data, archive still written). export accepts --contents full|json|summary and
--output-dir (or EXPORT_DIR).
| File | Content |
|---|---|
summary.md |
Short report: data sources, changes since the previous run, correlated events, incidents, alerts, deviations, data quality, recommendations, and a reading guide. No per-metric tables. |
incidents.json |
Incidents with id, severity, status, target host/service/disk, start, end, span and breach time, start/end/delta values, related Prometheus alerts, history and measurement quality; correlation groups that link events on one disk; changes since the previous run. |
metrics.json |
One line per series: current value, median, min, max, p05, p95 over 24h, 7d/30d comparison or the reason it is unavailable, counter increase and resets, coverage and flags. No raw samples. |
diagnostics.json |
The complete report (schema_version: 2), identical to --format json. |
manifest.json |
Export schema, application version, run id, creation time, window, SHA-256 and size of every file, and incomplete data sources. |
All files come from the same run and share its run_id. The archive is written to
a hidden .zip.partial file and renamed only when complete. Values of
PROMETHEUS_URL, BESZEL_URL (including their host names), tokens, the Beszel
e-mail and password are replaced by [redacted] in every file; alert labels and
annotations are already redacted during analysis.
Correlation groups join incidents that share an entity label value (device, disk,
disk_id, serial_number, pool, mountpoint; by-id names also match their
serial) and whose breaches are at most correlation.window_seconds (default 1800)
apart. correlation.aliases adds read-only queries whose series link several names
of one entity, for example disk="/dev/sdc" with its serial_number:
correlation:
window_seconds: 1800
aliases:
- query: smartmon_device_info{job="node",instance="manwe:9100"}
labels: [disk, serial_number]
The alias query is read once per run at the last evaluation slot; if it fails, the run is not degraded and incidents are correlated by exact label values only.
Copy the Markdown output into ChatGPT for interpretation. To read the last saved
report without collecting new metrics, use make report.
Use make report ARGS='--format json' when a structured report is useful.
Exit code 2 still means a
report was produced with degraded data; inspect its data-quality section.
ChatGPT interpretation is an operator action, outside the analyzer. The application never calls an LLM, uploads reports, or executes suggested infrastructure changes. An external scheduler may run the deterministic CLI later if explicitly configured.
The runtime uses UID/GID 1001, a read-only root filesystem, dropped capabilities,
no-new-privileges, PID/memory/CPU limits, and a small temporary filesystem. Only
/data persists; configuration is part of the read-only image. There are no published ports,
no host network, no Docker socket, and no runtime root privileges. The host
./data directory must exist and be writable by UID 1001 before running the
container. On rimonome, create it as that user; arrange matching ownership
explicitly on other hosts. The image build requires internet access.
Compose's bridge network allows outbound LAN connections but does not enforce
an egress allowlist. Use the server firewall/network policy to restrict egress to
the configured LAN services. LAN-only service binding/firewalls are infrastructure
responsibility; the analyzer opens no listening socket. For Docker DNS service
names, explicitly attach to the existing monitoring network in a local override,
or use actual LAN DNS names/IPs. Never assume localhost reaches the host.
Secrets may be supplied via environment variables or _FILE paths. Setting both
forms for one token is rejected. For Compose secrets, declare a secret file and
mount it through a local override, then set the corresponding _FILE variable to
/run/secrets/<name>. Keep secrets out of Git and restrict file permissions.
Environment files and data are excluded from Git and the build context. HTTP
clients reject redirects, use TLS verification, and ignore proxy environment
variables. Prefer HTTPS when the LAN is not fully trusted. Remote labels and
inventory remain potentially sensitive report contents; protect the data volume.
Development and verification
cargo fmt --check
cargo clippy --locked --all-targets -- -D warnings
cargo test --locked
Tests use synthetic time series, mocked API responses, and local loopback HTTP servers: outliers, zero MAD, directionality, missing history, incomplete data, nonfinite/duplicate samples, window boundaries, API failures, Beszel pagination/schema, report generation, SQLite retry/backfill deduplication, incident lifecycle across reports, counter increases and resets, alert collection, data-quality causes, and CLI exit codes against a fake Prometheus with 15-day retention. No live infrastructure is required or modified. Live Prometheus metric inventory, the installed Beszel version, and deployment connectivity require separate checks on rimonome. Local tests never contact live services.
Migration from the Python prototype
The implementation lives at ~/Code/Rust/homelab-analyzer. The original project
at ~/Code/Python/homelab-analyzer remains unchanged as a reference. Rust is the
maintained implementation for this MVP; Python, uv and pytest are not needed to
build or run it. The image contains the compiled release executable, system CA
certificates and the default configuration, without Cargo or a Python runtime.
Commands, YAML rules, exit codes and the JSON report's analysis fields retain the
prototype's meaning. tests/fixtures/python_demo.json is a captured reference from
the Python implementation; a Rust test compares the complete synthetic analysis
against it with numeric tolerance. The normal Rust tests do not execute Python.
The Rust project starts its own history; existing Python databases and reports are
not moved or overwritten. Hash encoding changed, so run IDs and fingerprints
should not be compared across implementations. Atomic SQLite writes also serialize
concurrent identical runs; report now opens the database strictly read-only.
The copied Compose configuration uses the existing local choice 1001:1001; the
Rust image and /data ownership match it. Existing volumes created with another
UID need administrator-managed ownership adjustment before reuse. This migration
does not change existing volumes or deploy anything to rimonome.
Local migration verification
cargo test --locked: 31 integration tests passed, including Python-reference parity, real loopback HTTP checks, concurrent SQLite deduplication and CLI flows.cargo fmt --checkand Clippy with-D warnings: passed.docker compose config --no-env-resolution --quiet: passed.- The release Docker image built successfully. Its synthetic healthcheck, analysis,
saved report and identical retry passed with network disabled, UID 1001,
read-only root, no capabilities and
no-new-privileges. SQLite remained writable on the isolated data volume. No Python, Cargo, SSH or Docker socket was present.
The original language migration did not contact live services. The subsequent rimonome profile was verified against the live Prometheus API; see profile evidence and limits. Beszel is disabled until an operator supplies its URL and authentication settings.