Observability

Live surfaces

• WebSocket log streaming and deployment logs, with level badges, filtering and download
• Web terminal (xterm) into running containers
• CPU / memory / network / block metrics sampled every 30 seconds, kept for 48 hours by default (NIXPLOY_METRICS_RETENTION_HOURS, up to 720)
• Replica breakdown and a 24h uptime chip
• A single push stream per tab updates deployment status, queue depth and service-status corrections, so dashboard screens carry no polling of their own

Prometheus

GET /api/metrics serves the calling API key's organization as Prometheus text exposition. A scrape is a read, so a read-only key is enough, and the same rate limits, key scopes and organization binding apply as everywhere else.

• Series: service CPU, memory and memory limit, service status, deployments by status, uptime probe state, and this process's queue depth
• Numbers come from the existing metrics store, so scraping faster than the 30-second sampler just sees the same point twice
• Services with no sample yet emit no CPU/memory series; the status gauge is always emitted, so status == 0 is the honest “it is down” signal
• One organization per key — there is deliberately no instance-wide dump that would expose every tenant's names

Service event timeline

Runtime → Events on every service page answers the question a Swarm panel usually cannot: why did it restart? One list of what happened to the service, newest first, filterable by failures, deploys or changes.

• Task started, task failed and killed rows, with the exit code and the daemon's own error string
• Status drift the reconciler had to correct — something changed the service outside Nixploy
• Every deployment transition, and the rollback or config change that came just before it
• Who changed what: settings, environment variables, source, build type, start and stop all carry the actor
• Deploys, rollbacks and kills are drawn on the metrics charts too, so a spike and its cause sit next to each other
• Read it from a terminal with nixploy events list <serviceId>, or over REST

Deploy Copilot reads the last events before a failure as context, so “explain this failed deploy” can see the out-of-memory kill four minutes earlier instead of guessing from the build log alone.

Alerts & incidents

• Per-service and host threshold alert rules
• Uptime probes with flip notifications
• Expiry warnings for uploaded TLS certificates, from 21 days out
• Incident timeline on the Monitoring page — acknowledge records who is looking at it and leaves the incident open; resolve closes it with an optional note
• Proposed remediations: a service that fails three tasks in ten minutes gets an incident carrying a suggested rollback — to the previous pinned image, or a compose stack's earlier snapshot — that waits for a person to apply or dismiss it. Nothing runs on its own, an out-of-memory loop is told to raise its limit rather than roll back, and one proposal per service per hour keeps it quiet
• Public status page at /status/<token>: chosen probes, their state, 90-day uptime and recent incident titles, on an unauthenticated link you can rotate or take offline
• Fleet overview across local and remote servers

Platform self-alerts

Organization thresholds watch tenant services; platform self-alerts watch the box the panel runs on, so an operator learns about a full disk from Slack instead of from a failed deploy. A check runs every five minutes.

• Host disk above 85% (warning) or 95% (critical) on the filesystem holding the config dir
• The oldest deployment still queued for more than 30 minutes
• An ACME certificate expiring in under 14 days, or already expired
• Traefik or Postgres below their desired replicas
• No successful instance backup in NIXPLOY_INSTANCE_BACKUP_ALERT_DAYS days (default 8, 0 disables)

Panel logs

• docker service logs -f nixploy — one line per event, prefixed with the subsystem in brackets ([deploy], [status-reconciler], [metrics-history], [platform-alerts], …)
• LOG_LEVEL is debug | info | warn | error; LOG_FORMAT=json emits one JSON object per line for Loki, Elastic or Datadog
• All three platform services rotate their json-file logs (10 MB × 3), so an unbounded log cannot fill a small host
• Build logs are not process logs: each deployment writes its own file under the config dir, streamed live to the UI and pruned after 30 days
• Secrets never reach the log — a line that contains a credential is a bug worth reporting

Runtime log history

The live log tab ends with the container. Runtime log history keeps what services printed: the worker collects docker logs every 30 s into hour files under the config directory (gzipped once the hour closes), with Docker's own timestamps and a level classified on the server. Every service page has Runtime → History, the Monitoring page has a Logs section across every service you can see, the CLI has nixploy logs search and MCP has get_runtime_logs.

• Query grammar: terms (all must match), "phrases", -excludes, level:error,warn, container:web, /regex/ (≤ 200 characters, quantified groups refused).
• Bounded: 2 000 lines per container per pass with an explicit marker line when more was dropped; a read stops after 500 000 lines or four seconds and says so.
• Retention per service, instance-wide: NIXPLOY_RUNTIME_LOG_RETENTION_DAYS (7) and NIXPLOY_RUNTIME_LOG_MAX_MB_PER_SERVICE (256); NIXPLOY_RUNTIME_LOGS=0 turns the harvester off.