Add: dummy backend for behavioural testing
This commit is contained in:
100
docs/RANDOM_BACKEND_TESTING.md
Normal file
100
docs/RANDOM_BACKEND_TESTING.md
Normal file
@@ -0,0 +1,100 @@
|
||||
# Random Backend API Testing
|
||||
|
||||
Date: 2026-02-13
|
||||
|
||||
## Purpose
|
||||
|
||||
This workflow provides randomized backend-like API traffic against a running Sprimo frontend.
|
||||
It focuses on command endpoints and mixes valid and invalid requests to verify transport and
|
||||
runtime resilience.
|
||||
|
||||
Primary targets:
|
||||
|
||||
- `POST /v1/command`
|
||||
- `POST /v1/commands`
|
||||
|
||||
Supporting checks:
|
||||
|
||||
- `GET /v1/health`
|
||||
- `GET /v1/state` (periodic sampling)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Frontend runtime is already running (`sprimo-app` or `sprimo-tauri`).
|
||||
- Python is available.
|
||||
- Auth token and port are available from config or passed via CLI flags.
|
||||
|
||||
By default, the tester discovers config at:
|
||||
|
||||
- Windows: `%APPDATA%/sprimo/config/config.toml` (legacy fallback: `%APPDATA%/sprimo/config.toml`)
|
||||
- macOS: `~/Library/Application Support/sprimo/config/config.toml`
|
||||
- Linux: `~/.config/sprimo/config/config.toml`
|
||||
|
||||
## Quick Start
|
||||
|
||||
```powershell
|
||||
just random-backend-test
|
||||
```
|
||||
|
||||
Strict mode (non-zero exit if unexpected outcomes appear):
|
||||
|
||||
```powershell
|
||||
just random-backend-test-strict
|
||||
```
|
||||
|
||||
## CLI Examples
|
||||
|
||||
Run against explicit host/port/token:
|
||||
|
||||
```powershell
|
||||
python scripts/random_backend_tester.py --host 127.0.0.1 --port 32145 --token "<token>"
|
||||
```
|
||||
|
||||
Deterministic run with higher invalid traffic:
|
||||
|
||||
```powershell
|
||||
python scripts/random_backend_tester.py --seed 42 --invalid-probability 0.35 --duration-seconds 45
|
||||
```
|
||||
|
||||
Write machine-readable summary:
|
||||
|
||||
```powershell
|
||||
python scripts/random_backend_tester.py --json-summary dist/random-backend-summary.json
|
||||
```
|
||||
|
||||
## Important Flags
|
||||
|
||||
- `--duration-seconds`: total run time
|
||||
- `--interval-ms`: delay between random requests
|
||||
- `--batch-probability`: ratio of `/v1/commands` usage
|
||||
- `--max-batch-size`: max commands per batch request
|
||||
- `--invalid-probability`: inject malformed/invalid payloads
|
||||
- `--unauthorized-probability`: inject auth failures
|
||||
- `--state-sample-every`: periodic authenticated `/v1/state` checks
|
||||
- `--strict`: fail run on unexpected outcomes
|
||||
- `--health-check`: verify API liveness before random traffic
|
||||
|
||||
## Expected Result Pattern
|
||||
|
||||
In mixed mode, typical status distribution includes:
|
||||
|
||||
- `202` for valid command requests
|
||||
- `400` for malformed/invalid payloads
|
||||
- `401` for missing/invalid auth
|
||||
|
||||
Unexpected outcomes that should be investigated:
|
||||
|
||||
- `5xx` responses
|
||||
- repeated transport failures/timeouts
|
||||
- strict mode failures (`unexpected_outcomes > 0`)
|
||||
|
||||
## Evidence Guidance
|
||||
|
||||
When used for issue verification, record:
|
||||
|
||||
- command used (including seed/probabilities)
|
||||
- summary output (status buckets, unexpected outcomes, transport errors)
|
||||
- linked issue file under `issues/issueN.md`
|
||||
|
||||
This test complements `cargo check --workspace`, `cargo test --workspace`, and
|
||||
`just qa-validate`; it does not replace them.
|
||||
Reference in New Issue
Block a user