Home

Zelos Operations Guide v0.8.1

Deployment Modes

1. Bare Metal (Python)

# Install from PyPI
pip install zelos-runtime

# Create config and run
cp zelos.yaml.example zelos.yaml    # edit config to your needs
python3 start.py                     # start Runtime + Dashboard

Or from source:

git clone https://github.com/AI-Zelos/zelos.git && cd zelos
pip install -e ".[dev]"
make dev

2. Docker

make build          # zelos:0.8.1
make run            # docker compose up -d
make stop           # docker compose down

# With Redis for persistent storage:
make run-storage    # includes redis profile

3. Kubernetes

kubectl apply -f deploy/k8s/deployment.yaml
kubectl apply -f deploy/k8s/service.yaml

K8s Probes

Example K8s config:

livenessProbe:
  httpGet:
    path: /live
    port: 9876
  initialDelaySeconds: 5
  periodSeconds: 15
readinessProbe:
  httpGet:
    path: /ready
    port: 9876
  initialDelaySeconds: 3
  periodSeconds: 10

Observability

Prometheus Metrics

Scrape http://<runtime>:9876/metrics for:

Grafana Dashboard

Import deploy/grafana/zelos-dashboard.json into Grafana. Requires Prometheus datasource configured.

Security

API Key Management

from zelos.security import APIKeyManager

mgr = APIKeyManager(
    max_failures=10,         # auto-revoke threshold
    failure_window_seconds=60.0,  # sliding window
    auto_revoke=True,        # enable brute-force protection
)

key = mgr.generate_key("admin", "production-admin-key")
# → "zelos_<128 hex chars>"  — save this, it's shown only once

mgr.validate(key)   # → {"role": "admin", ...}
mgr.revoke(key)     # → True

Anomaly Detection (v0.8.1)

• Tracks failed auth attempts per key hash
• Auto-revokes after max_failures within failure_window_seconds
• Configurable per deployment
• Failed attempts older than the window are pruned

Audit Log Export

logger.export_json_file("/var/log/zelos/audit.json")

Distributed Coordination (v0.8.1)

etcd Backend

from zelos.coordination import EtcdCoordinationBackend, CoordinationNode

backend = EtcdCoordinationBackend({"endpoints": "localhost:2379", "prefix": "/zelos/"})
backend.connect()
backend.register_node(CoordinationNode(node_id="node-1", host="10.0.0.1", port=9001))
backend.elect_leader("node-1", ttl_seconds=30)
leader = backend.get_leader()  # → "node-1"
backend.heartbeat("node-1")    # renew lease

NATS Message Bus

from zelos.messaging_nats import NatsMessageBus

bus = NatsMessageBus({"servers": ["nats://localhost:4222"]})
bus.connect()
bus.subscribe("zelos.events", lambda data, headers: process(data))
bus.publish("zelos.events", {"event_type": "task.completed"})

Go SDK

go get github.com/zelos/zelos-go

c := client.New("http://localhost:9876", "zk-client-dev")
health, _ := c.Health()
goal, _ := c.SubmitGoal("Build a landing page", "high")

Multi-Node Cluster Deployment

Architecture

┌──────────────────────────────────────────────┐
│                  Node 1 (Leader)             │
│  ZelosRuntime + etcd + NATS                  │
│  IP: 10.0.0.1:9876                           │
└──────────┬──────────┬────────────────────────┘
           │          │
     ┌─────▼───┐  ┌──▼──────┐
     │ Node 2  │  │ Node 3  │
     │ 10.0.0.2│  │ 10.0.0.3│
     └─────────┘  └─────────┘

Prerequisites

Step 1: Start Infrastructure

# etcd (all nodes connect to same cluster)
docker run -d --name etcd -p 2379:2379 \
  bitnami/etcd:latest

# NATS
docker run -d --name nats -p 4222:4222 \
  nats:latest

Step 2: Configure Each Node

# zelos-node-1.yaml
runtime:
  instance_id: "zelos-node-1"
  api:
    host: "10.0.0.1"
    port: 9876

distributed:
  enabled: true
  node_id: "node-1"
  peers: ["10.0.0.1:9877", "10.0.0.2:9877", "10.0.0.3:9877"]

storage:
  type: postgresql
  url: "postgresql://postgres:zelos@10.0.0.1:5432/zelos"

coordination:
  type: etcd
  endpoints: "10.0.0.1:2379"

messaging:
  type: nats
  servers: ["nats://10.0.0.1:4222"]

Step 3: Start Nodes

# Node 1 (expects to become leader)
python3 start.py --config zelos-node-1.yaml &

# Node 2
python3 start.py --config zelos-node-2.yaml &

# Node 3
python3 start.py --config zelos-node-3.yaml &

Step 4: Verify Cluster

curl http://10.0.0.1:9876/api/v1/cluster
# → {"is_leader": true, "peers": ["node-1", "node-2", "node-3"], "healthy": 3}

curl http://10.0.0.2:9876/api/v1/cluster
# → {"is_leader": false, "leader": "node-1"}

Leader Election Flow

1. All nodes register in etcd: /zelos/nodes/{node_id}
2. Smallest node_id wins (Bully algorithm)
3. Leader holds etcd lease, renews via heartbeat
4. If leader lease expires → new election triggered
5. Watchers (NATS subscribers) notified of leader change

Work Stealing

Node-2 idle (queue depth=0) → queries etcd for busiest node → steals READY tasks
Node-3 overloaded (queue depth=15) → Node-1 and Node-2 steal work

Failure Recovery

Troubleshooting

Health Check Components

GET /api/v1/health → {
  "status": "healthy",
  "components": {
    "kernel": "healthy",
    "plugins": {"total": N, "healthy": N},
    "agents": {"connected": N, "disconnected": N},
    "security": {"audit_events": N},
    "cluster": {"enabled": bool, "is_leader": bool}
  }
}

Backup & Recovery

Event Persistence

Enable PersistentEventStore with a storage backend to survive restarts:

from zelos.event_bus import PersistentEventStore
from zelos.storage import PostgreSQLStorageBackend

storage = PostgreSQLStorageBackend({"url": "postgresql://..."})
storage.connect()
store = PersistentEventStore(storage)

# After crash:
recovered = store.recover()  # replays all persisted events

State Recovery

Save Goal/Agent state before shutdown:

storage.set_state(f"goal-{goal_id}", goal_state)
# After restart:
saved = storage.get_state(f"goal-{goal_id}")

Configuration Reference

See zelos.yaml for all options. Key sections: