311 lines
13 KiB
Markdown
311 lines
13 KiB
Markdown
# 📘 `grel-rs` Technical Design Document
|
||
**Binary:** `grel` | **Repository:** `grel-rs`
|
||
**Target Platforms:** Linux, Windows (macOS optional)
|
||
**Core Philosophy:** Forge-agnostic, pacman-familiar, performance-first, secure, extensible via traits.
|
||
|
||
---
|
||
|
||
## 1. Overview & Goals
|
||
`grel` (Global/General Release) is a terminal-native, high-performance package manager for downloading and managing binary releases from Git forges. It abstracts provider-specific APIs into a unified resolution/download pipeline, supports HTTP/SOCKS5 proxies, caches optimal DNS endpoints, downloads assets in parallel, and presents a polished, pacman-compatible UX.
|
||
|
||
**Key Requirements Met:**
|
||
- ✅ Pacman-style CLI (`-S`, `-Syu`, `-Ss`, `--noconfirm`)
|
||
- ✅ Proxy support (HTTP/HTTPS/SOCKS5) with env/config fallback
|
||
- ✅ DNS/IP latency caching + RTT probing for CDN optimization
|
||
- ✅ Parallel asset downloads with multi-progress UI
|
||
- ✅ Interactive asset selection (pacman/artix style) with TTY/CI fallback
|
||
- ✅ Cross-platform (Linux/Windows) with safe extraction & PATH integration
|
||
- ✅ Extensible provider architecture (GitHub, GitLab, Codeberg, Gitea, self-hosted)
|
||
|
||
---
|
||
|
||
## 2. Workspace Architecture
|
||
```
|
||
grel-rs/
|
||
├── Cargo.toml # Workspace root (resolver = "2")
|
||
├── crates/
|
||
│ ├── grel-cli/ # CLI parsing, TUI orchestration, prompts, progress
|
||
│ ├── grel-core/ # Resolution, versioning, asset matching, upgrade state
|
||
│ ├── grel-providers/ # Forge trait, registry, API implementations (modules)
|
||
│ ├── grel-network/ # HTTP client, proxy routing, IP cache resolver, downloader
|
||
│ ├── grel-cache/ # SQLite state, IP cache, artifact storage, TTL eviction
|
||
│ └── grel-config/ # Layered config (CLI > env > TOML > defaults), migrations
|
||
├── tests/ # Integration, mock servers, e2e fixtures
|
||
└── scripts/ # CI, release, benchmark helpers
|
||
```
|
||
|
||
**Crate Boundaries:**
|
||
| Crate | Owns | Depends On |
|
||
|-------|------|------------|
|
||
| `grel-cli` | CLI, UI, command routing, prompts | `grel-core`, `grel-config`, `dialoguer`, `indicatif` |
|
||
| `grel-core` | Package resolution, asset matching, upgrade logic | `grel-providers`, `grel-cache`, `semver` |
|
||
| `grel-providers` | API clients, pagination, rate-limit handling | `reqwest`, `async-trait`, `chrono` |
|
||
| `grel-network` | Proxy, IP resolver, parallel downloads, streaming | `reqwest`, `hickory-resolver`, `dashmap`, `tokio` |
|
||
| `grel-cache` | SQLite state, IP cache, artifact storage, LRU | `sqlx`, `sha2`, `tar`/`zip`, `directories` |
|
||
| `grel-config` | Settings loading, validation, defaults, migrations | `figment`, `serde`, `toml` |
|
||
|
||
---
|
||
|
||
## 3. Core Abstractions & Data Flow
|
||
All forges implement a single normalized contract. `grel-core` parses user input → providers return forge-agnostic structs → `grel-network` downloads blindly.
|
||
|
||
```rust
|
||
pub enum ForgeType { GitHub, GitLab, Gitea, Codeberg, SelfHosted(String) }
|
||
|
||
pub struct PackageRef {
|
||
pub forge: ForgeType,
|
||
pub owner: String, pub repo: String,
|
||
pub version: Option<semver::Version>,
|
||
pub prerelease: bool,
|
||
}
|
||
|
||
pub struct ResolvedRelease {
|
||
pub version: semver::Version,
|
||
pub published_at: chrono::DateTime<Utc>,
|
||
pub is_prerelease: bool,
|
||
pub assets: Vec<DownloadAsset>,
|
||
pub release_notes: Option<String>,
|
||
}
|
||
|
||
pub struct DownloadAsset {
|
||
pub filename: String,
|
||
pub download_url: url::Url,
|
||
pub size_bytes: u64,
|
||
pub sha256: Option<String>,
|
||
pub asset_type: AssetKind, // Binary, Archive, Source, Other
|
||
}
|
||
```
|
||
|
||
**Flow:**
|
||
```
|
||
CLI → PackageRef → ProviderRegistry → Provider.resolve() → ResolvedRelease
|
||
→ AssetResolver (tiered match + checksum pairing) → SelectedRelease
|
||
→ Network (IP cache → proxy client → parallel download)
|
||
→ Cache (verify sha256 → extract → atomic rename → update state.sqlite)
|
||
→ CLI (progress summary → exit)
|
||
```
|
||
|
||
---
|
||
|
||
## 4. CLI Specification & UX Design
|
||
`clap 4` with subcommands. Pacman short flags supported via aliases.
|
||
|
||
| Command | Aliases | Description |
|
||
|---------|---------|-------------|
|
||
| `grel sync foo/bar` | `S`, `-S` | Download & install latest |
|
||
| `grel sync foo/bar@1.2.3` | `-S` | Pin version |
|
||
| `grel search keyword` | `Ss`, `-Ss` | Search across registered forges |
|
||
| `grel upgrade` | `Syu`, `-Syu` | Refresh + upgrade installed |
|
||
| `grel info foo/bar` | `Si`, `-Si` | Show release metadata |
|
||
| `grel list` | `Q`, `-Q` | List installed packages |
|
||
| `grel remove foo/bar` | `R`, `-R` | Uninstall |
|
||
| `grel clean` | `Sc`, `-Sc` | Purge artifact/metadata cache |
|
||
| `grel path add` | - | Inject `bin/` dir into shell/Windows PATH |
|
||
|
||
**UX Features:**
|
||
- `tabled` for search/list output
|
||
- `indicatif::MultiProgress` for parallel downloads
|
||
- `owo-colors` for status: 🟢 ✅, 🟡 ⚠️, 🔴 ❌, 🔵 ℹ️
|
||
- `--json` for machine-readable output
|
||
- `--noconfirm` / `-y` skips all prompts
|
||
|
||
---
|
||
|
||
## 5. PATH Integration & Post-Install Behavior
|
||
**Problem:** User-space installs aren't in `$PATH` by default. Pacman installs globally (`/usr/bin`).
|
||
|
||
**Solution:** Dedicated `bin/` directory + explicit shell integration.
|
||
```
|
||
~/.local/share/grel/ (Linux) → bin/
|
||
%LOCALAPPDATA%\grel\ (Windows) → bin\
|
||
```
|
||
|
||
**Implementation:**
|
||
1. `grel sync` extracts binaries to `~/.local/share/grel/bin/` (or equivalent)
|
||
2. `grel path add` detects shell (`$SHELL`, `$PROFILE`) and appends:
|
||
```bash
|
||
export PATH="$HOME/.local/share/grel/bin:$PATH" # bash/zsh
|
||
$env:Path = "$env:LOCALAPPDATA\grel\bin;" + $env:Path # PowerShell
|
||
```
|
||
3. Windows: Uses registry `HKCU\Environment\Path` + broadcasts `WM_SETTINGCHANGE`
|
||
4. First-run hint: `💡 Run grel path add to enable global command access`
|
||
5. **Why not `/usr/local/bin`?** Avoids `sudo`, respects XDG, prevents system package conflicts.
|
||
|
||
---
|
||
|
||
## 6. Configuration vs State Management
|
||
**Strict Separation:**
|
||
| Layer | Format | Location | Purpose |
|
||
|-------|--------|----------|---------|
|
||
| **User Config** | `config.toml` | `~/.config/grel/` (Linux) / `%APPDATA%\grel\` (Win) | Proxy, tokens, concurrency, asset preferences |
|
||
| **Program State** | `state.sqlite` | `~/.local/share/grel/db/` | Installed packages, IP cache, ETags, last-checked |
|
||
| **Artifact Cache** | Files | `~/.cache/grel/artifacts/` | Downloaded archives (LRU evicted, 2GB default) |
|
||
|
||
**`config.toml` Example:**
|
||
```toml
|
||
[general]
|
||
version = 1
|
||
max_concurrent = 4
|
||
proxy = "http://127.0.0.1:7890"
|
||
|
||
[assets]
|
||
default_selection_policy = "largest" # first | largest | preferred_format
|
||
prefer_formats = ["*.tar.gz", "*.zip", "*.exe"]
|
||
ignore_patterns = ["*source*", "*.deb", "*.rpm"]
|
||
|
||
[upgrade]
|
||
check_interval_hours = 6
|
||
max_parallel_checks = 10
|
||
```
|
||
|
||
**Migrations:**
|
||
- Config: `version` field in TOML. On load, `grel-config` runs migration functions (rename keys, add defaults, warn on breaking changes).
|
||
- SQLite: `sqlx migrate` with timestamped files. Applied automatically on startup. Zero-downtime schema upgrades.
|
||
|
||
---
|
||
|
||
## 7. Asset Resolution & Selection Pipeline
|
||
Releases contain multiple formats. `grel` uses **deterministic tiered matching** + **interactive fallback**.
|
||
|
||
### Tiered Matching (Order Matters)
|
||
1. **Exact:** `{repo}-{version}-{os}-{arch}.{fmt}`
|
||
2. **OS+Arch:** `*{os}*{arch}*.{fmt}`
|
||
3. **OS Only:** `*{os}*.{fmt}`
|
||
4. **Universal Archive:** `*.{tar.gz,tar.xz,zip}`
|
||
5. **Universal Binary:** `*.{exe,elf,dll,dylib}`
|
||
|
||
### Checksum & Signature Auto-Pairing
|
||
If main asset is `foo-1.0.0-linux-amd64.tar.gz`, `grel` searches for:
|
||
- `foo-1.0.0-linux-amd64.tar.gz.sha256`
|
||
- `sha256sums.txt` (parses line matching filename)
|
||
Automatically downloads & verifies before extraction.
|
||
|
||
### Interactive Selection (Pacman/Artix Style)
|
||
- Uses `dialoguer::Select` with numbered list
|
||
- Default marker: `(default)` shown next to highest-priority match
|
||
- `--noconfirm` / `-y` skips prompt, uses default
|
||
- **TTY Detection:** If not interactive (CI/pipes), auto-selects default + prints `ℹ️ Non-interactive mode. Using default: <filename>` to stderr
|
||
- Timeout fallback (configurable): auto-selects default if no input within `prompt_timeout_secs`
|
||
|
||
**CLI Overrides:**
|
||
```bash
|
||
grel sync foo/bar --asset custom.exe
|
||
grel sync foo/bar --platform linux/aarch64
|
||
grel sync foo/bar --list-assets # Dry-run table
|
||
```
|
||
|
||
---
|
||
|
||
## 8. Networking, Proxy & DNS/IP Caching
|
||
### Proxy Support
|
||
- Native via `reqwest::Proxy::all()`, `http()`, `https()`
|
||
- Priority: `--proxy` > `GREL_PROXY` > `http_proxy`/`all_proxy` > `config.toml`
|
||
- Supports `http://user:pass@host`, `socks5://...`, `socks5h://...`
|
||
|
||
### DNS/IP Caching
|
||
⚠️ **No hardcoded IPs.** CDNs rotate frequently. Instead:
|
||
1. Resolve `A/AAAA` via `hickory-resolver`
|
||
2. Parallel `TcpStream::connect` probes to all IPs on `443`
|
||
3. Store fastest IP + RTT in SQLite with TTL (default `300s`)
|
||
4. Use `reqwest::ClientBuilder::resolve(host, ip)` for routing
|
||
5. Background refresh thread evicts stale entries, pre-warms hot domains
|
||
|
||
---
|
||
|
||
## 9. Rate Limiting & `-Syu` Optimization
|
||
GitHub allows 60 req/hr unauthenticated, 5000/hr authenticated. `grel` handles this gracefully:
|
||
|
||
1. **ETag Caching:** `If-None-Match` → `304 Not Modified` **does not count** against rate limits. Cached ETags stored in `state.sqlite`.
|
||
2. **Smart Polling:** Only recheck packages where `last_checked < now() - check_interval_hours`.
|
||
3. **Rate Limit Awareness:** Parse `X-RateLimit-Remaining` & `X-RateLimit-Reset`. If `< 50`, sleep until reset or queue remaining checks.
|
||
4. **Auth Token:** Strongly recommended. `grel` prompts on first run: `🔑 Set GREL_GITHUB_TOKEN for 5000 req/hr`.
|
||
5. **`--no-api` Fallback:** Skips remote checks, uses local manifest + cache timestamps.
|
||
6. **Parallel Batch Checks:** `max_parallel_checks` limits concurrent API calls to avoid hitting burst limits.
|
||
|
||
**`-Syu` Flow:**
|
||
```
|
||
Load state → Filter stale → Batch API (ETag) → Parse 200/304 → Build diff → Prompt → Download → Verify → Update
|
||
```
|
||
|
||
---
|
||
|
||
## 10. Cross-Platform Implementation Notes
|
||
| Aspect | Linux | Windows | Implementation |
|
||
|--------|-------|---------|----------------|
|
||
| Paths | `~/.local/share/grel` | `%LOCALAPPDATA%\grel` | `directories::ProjectDirs` |
|
||
| Archives | `tar` + `zstd`/`xz`/`gz` | `zip` + `tar` | Feature-gated extraction |
|
||
| Exec Perms | `0o755` | `FILE_ATTRIBUTE_ARCHIVE` | `cfg(unix)` vs `cfg(windows)` |
|
||
| Path Safety | Reject `..`, absolute paths | Same | `sanitize-filename` + strict validation |
|
||
| Executable Suffix | None | `.exe` | Auto-append if `cfg(windows)` & missing |
|
||
| Shell Integration | `.bashrc`, `.zshrc`, fish | `$PROFILE`, Registry | `grel path add` |
|
||
|
||
---
|
||
|
||
## 11. Security & Reliability Guarantees
|
||
- ✅ **TLS:** `rustls` only (no native OpenSSL)
|
||
- ✅ **Checksums:** Verify `sha256` before extraction
|
||
- ✅ **Archive Safety:** Reject `..`, absolute paths, symlinks to outside dest
|
||
- ✅ **Atomic Installs:** Download to temp dir → verify → `rename` → register in DB
|
||
- ✅ **No Auto-Execute:** Binaries installed but not run unless invoked by user
|
||
- ✅ **Token Security:** Loaded from env/config, never logged or serialized to stdout
|
||
- ✅ **Graceful Degradation:** Network errors → retry with backoff, rate limits → sleep/queue, missing assets → clear error + suggestions
|
||
|
||
---
|
||
|
||
## 12. Dependency Matrix
|
||
```toml
|
||
# Core & Async
|
||
tokio = { version = "1", features = ["full"] }
|
||
futures = "0.3"
|
||
async-trait = "0.1"
|
||
|
||
# CLI & UX
|
||
clap = { version = "4", features = ["derive", "wrap_help", "string"] }
|
||
indicatif = { version = "0.17", features = ["tokio"] }
|
||
tracing-indicatif = "0.3"
|
||
dialoguer = "0.11"
|
||
tabled = "0.16"
|
||
owo-colors = "4"
|
||
|
||
# Network & Proxy
|
||
reqwest = { version = "0.12", features = ["rustls-tls", "json", "socks", "stream"] }
|
||
hickory-resolver = "0.24"
|
||
dashmap = "6"
|
||
|
||
# Cache & Storage
|
||
sqlx = { version = "0.8", features = ["runtime-tokio-rustls", "sqlite"] }
|
||
sha2 = "0.10"
|
||
directories = "5"
|
||
tar = "0.4"
|
||
zip = "2"
|
||
zstd = "0.13"
|
||
xz2 = "0.1"
|
||
bzip2 = "0.5"
|
||
sanitize-filename = "0.5"
|
||
|
||
# Config & Utils
|
||
figment = { version = "0.10", features = ["toml", "env"] }
|
||
semver = "1"
|
||
url = "2"
|
||
chrono = { version = "0.4", features = ["serde"] }
|
||
serde = { version = "1", features = ["derive"] }
|
||
anyhow = "1"
|
||
thiserror = "2"
|
||
tracing = "0.1"
|
||
tracing-subscriber = { version = "0.3", features = ["env-filter", "fmt"] }
|
||
```
|
||
*Note: Rust 1.70+ required for `std::io::IsTerminal` (replaces `atty`).*
|
||
|
||
---
|
||
|
||
## 13. Testing & CI Strategy
|
||
- **Unit:** Parsing, version comparison, cache TTL math, asset matching tiers
|
||
- **Mocked Providers:** `wiremock` for API responses (200, 304, 403, 429, 500)
|
||
- **Integration:** Download test fixtures → extract → verify structure/checksums
|
||
- **Cross-Platform CI:** `ubuntu-latest`, `windows-latest` (GitHub Actions)
|
||
- **Performance:** `criterion` for DNS resolution + parallel download throughput
|
||
- **Fuzzing:** `cargo-fuzz` on `PackageRef` parser + archive path validation
|
||
- **Non-Interactive Tests:** `echo "" | grel sync foo/bar` must not hang
|
||
|
||
---
|