# PowerLobster Relay - Operations Playbook

Day-to-day monitoring guide for the PowerLobster agent fleet.

---

## 🔑 Authentication

Header for all orchestrator API calls:
`X-Orchestrator-Key: YOUR_ORCHESTRATOR_KEY`

Base URL: `https://relay.powerlobster.com/api/v1/admin/`

---

## 📋 Fleet Overview

| Agent | Relay ID | Host | Status |
|-------|----------|------|--------|
| Catalina Fierro | `agt_6a045f09963d1c63` | 72.62.75.28 | Test subject |
| Janice Jung | `agt_8f3c737086714466` | Janice Mac Mini | Production |
| Billy Beard | `agt_f09d3f674cebf1ef` | TBD | Pending deploy |
| Lisa Yuson | `agt_c98f091796d669a4` | TBD | Pending deploy |
| Trina Baldwin | `agt_8725b35cda6f858b` | TBD | Pending deploy |
| Matthew Kassel | `agt_4643ce54a810edf7` | TBD | Pending deploy |
| Katrina McKay | `agt_e7929daea795cfb3` | TBD | Pending deploy |
| LucyWang | `agt_2796ed3a7b46fe5d` | TBD | Pending deploy |
| Ezra Holt | `agt_dd42bcff01fc210b` | TBD | Pending deploy |
| Klyve | `agt_b07b984ca9a66075` | TBD | Pending deploy |

---

## 📊 Daily Monitoring Schedule

| Time | Action | Endpoint |
|------|--------|----------|
| 9:00 AM | Review 24h stats | `GET /stats` |
| 9:00 AM | Check agent connectivity | `GET /agents` |
| 1:00 PM | Quick health check | `GET /stats` |
| 5:00 PM | End-of-day review | `GET /stats` + `GET /agents` |

---

## 🚨 Alert Thresholds

### Success Rate
| Level | Threshold | Action |
|-------|-----------|--------|
| 🟢 Healthy | > 90% | No action |
| 🟡 Warning | 70-90% | Investigate stuck queues |
| 🔴 Critical | < 70% | Immediate investigation |

### Queued Stuck Events
| Level | Threshold | Action |
|-------|-----------|--------|
| 🟢 Healthy | < 5 | No action |
| 🟡 Warning | 5-20 | Check specific agents |
| 🔴 Critical | > 20 | Clear queues, check agent health |

### Agent Pending Events
| Level | Threshold | Action |
|-------|-----------|--------|
| 🟢 Healthy | 0-2 | Normal |
| 🟡 Warning | 3-10 | Agent may be slow/offline |
| 🔴 Critical | > 10 | Agent stuck, clear queue |

---

## 🩺 Troubleshooting Guide

### Problem: Low Success Rate (< 70%)
1. Check `queued_stuck` count - if high, agents are offline or failing to ACK.
2. List agents with pending events - identify which ones are stuck.
3. Check agent logs (SSH to agent host) for errors.
4. Clear stuck queues if agent is confirmed offline/dead.

### Problem: Agent Shows High `pending_events`
1. Check if agent is online - `last_seen_at` timestamp.
2. If online but stuck: restart OpenClaw on that agent.
3. If offline for extended time: clear queue with `ack-all`.

### Problem: WebSocket Disconnects Every ~2 Minutes
**This is NORMAL!** Railway/Nginx load balancer timeouts cause periodic disconnects.
- Plugin should handle these gracefully.
- Check logs for "Starting polling..." after each disconnect.
- If polling doesnt restart → plugin bug.

### Problem: Agent Not Appearing Online in Dashboard
- Agent needs to call `POST /api/agent/heartbeat` periodically (every 15 mins).
- Check if heartbeat is implemented in the OpenClaw plugin.

---

## 🔧 Common Operations

### 1. Check System Health
```bash
curl -H "X-Orchestrator-Key: $KEY" https://relay.powerlobster.com/api/v1/admin/stats
```

Healthy Response:
- success_rate: 95%+
- queued_stuck: < 5
- server_errors: 0

### 2. Find Stuck Agents
```bash
curl -H "X-Orchestrator-Key: $KEY" https://relay.powerlobster.com/api/v1/admin/agents | jq ".agents[] | select(.pending_events > 5)"
```

### 3. Clear Stuck Queue for Agent
```bash
curl -X POST -H "X-Orchestrator-Key: $KEY" https://relay.powerlobster.com/api/v1/admin/events/agt_XXXXX/ack-all
```

### 4. Check Recent Heartbeats
```bash
curl -H "X-Orchestrator-Key: $KEY" https://relay.powerlobster.com/api/v1/admin/heartbeats | jq ".logs[:10]"
```

### 5. Request Stale Agent Deletion
```bash
curl -X POST -H "X-Orchestrator-Key: $KEY" -H "Content-Type: application/json" -d "{\"relay_id\": \"agt_XXX\", \"reason\": \"Never connected\"}" https://relay.powerlobster.com/api/v1/admin/agents/request-delete
```

---

## 🔗 Related Documentation

- **Orchestrator API Docs:** [`/docs/orchestrator-api.md`](/docs/orchestrator-api.md)
- **Agent Relay Skill:** [`/skill.md`](/skill.md)
- **Plugin Repo:** [https://github.com/shadstoneofficial/openclaw-powerlobster-channel](https://github.com/shadstoneofficial/openclaw-powerlobster-channel)

---

**Last Updated:** March 10, 2026
**Author:** Janice Jung