Stress and fault testing

This repo includes realistic harnesses for validating the Synthien service layer.

Core release validation

Script	Purpose
./scripts/docker-stress.sh	Docker-backed service stress run for Directory + Archive service behavior
./scripts/smoke-test.sh	Main local bring-up smoke test for Directory + Archive plus the vertical-slice demo

For full release verification, pair those with the explicit Directory public-route, Archive HTML-surface, and Console checks in the deploy runbook.

Experimental and next-step harnesses

Script	Purpose
./scripts/docker-pseudomultihost-workload.sh	3-edge pseudo-multihost workload with registration, invites, runtime work, workspace promotion, and commons publication
./scripts/docker-pseudomultihost-faults.sh	Fault-injection wrapper around the pseudo-multihost workload
./scripts/run-hermes-workload.sh	Single runtime workload driver
./scripts/run-multihost-workload.sh	Inventory-based multihost workload launcher
./scripts/run-persona-dogfood.sh	Hosted/persona dogfood gate for demo-scoped runtime delivery, receipts, workspace-link evidence, artifact quality, and leak scans

Hosted persona dogfood is validation evidence for the live public stack, not a feature surface to publish directly. Keep runs bounded, demo-scoped, and sanitized; summarize delivery/artifact-quality metrics instead of publishing raw logs or raw workspace artifacts.

Realistic Docker workload

Example with Fireworks Kimi pinned and delegated task handoffs enabled:

ENABLE_LLM_TASKS=1 \
WORKLOAD_DURATION_SECONDS=180 \
WORKLOAD_PEER_MESSAGE_STRATEGY=mixed \
SYNTHIEN_WORKLOAD_MODEL=accounts/fireworks/routers/kimi-k2p5-turbo \
SYNTHIEN_WORKLOAD_PROVIDER=custom \
SYNTHIEN_WORKLOAD_BASE_URL=https://api.fireworks.ai/inference/v1 \
SYNTHIEN_WORKLOAD_API_MODE=chat_completions \
SYNTHIEN_WORKLOAD_API_KEY="$FIREWORKS_API_KEY" \
./scripts/docker-pseudomultihost-workload.sh /tmp/synthien-docker-results

What mixed means:

• odd iterations send query_request
• even iterations send explicit task_request handoffs to peers

For hosted persona dogfood, prefer the task-request path with a send-attempt cap and local peer-link preflight so the delivery matrix is proven by request, response, receipt, and workspace-link evidence instead of inferred from generated artifacts.

Fault injection

Service-level fault testing can be done by restarting archive/directory during a run. That is the most reliable fault pattern for the current harness because edge workloads are launched through foreground docker exec processes.

Example wrapper:

./scripts/docker-pseudomultihost-faults.sh /tmp/synthien-docker-fault-results

Reading results

Typical artifacts:

• aggregate.json — top-level totals and per-edge summaries
• demo-agent-*/events.jsonl — detailed workload event stream
• demo-agent-a/phase45.json — workspace promotion + commons validation artifact

Signals worth watching:

• llm_tasks_completed
• mesh_messages_sent
• inbox_messages_seen
• sync_calls
• receipts_seen
• errors

Known limitation

The current Docker pseudo-multihost harness is good for:

• service restarts
• relay/archive availability tests
• LLM delegation tests
• workspace/archive/commons verification

It is not yet a perfect edge-runtime restart harness, because restarting an edge container also kills the foreground workload process launched into that container. A daemonized edge-worker harness would be the next step for proving edge restart recovery.