Testing Methodology

Command guides

We compare commands with current official documentation, then check that each guide has a stated expected result and an escalation path. Version-dependent behavior is dated.

Troubleshooting

Diagnostic guides begin with the symptom and move through the smallest useful ladder: overall status, endpoint reachability, service state, configuration diagnostics, channel probe, then logs.

Browser tools

Clawdbook tools use redacted fixtures that cover valid input, invalid syntax, warnings, reset behavior, and copy or generated output. They run locally and never claim to replace the official runtime validator.

Model benchmarks

A publishable benchmark records the exact model tag, quantization, hardware, memory, provider version, OpenClaw version, context configuration, task set, run count, raw outcomes, and scoring code.

We separate task completion, parameter errors, loop rate, latency, and memory use. A model does not receive a “best” label from one successful prompt.

What we do not do

• Invent benchmark scores.
• Backfill a result from community sentiment.
• Claim 100% safety from static analysis.
• Mark a page tested because only its publication date changed.