manbo/blog-post-backup
1
0
Fork 0
Code Issues 88 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Update designs/grel-rs/TECHNICAL_DESIGN.md

Browse Source
This commit is contained in:
manbo
2026-04-12 00:08:25 +08:00
parent 90393ee190
commit 7882ccc435
1 changed files with 339 additions and 312 deletions
Download Patch File Download Diff File Expand all files Collapse all files

205
designs/grel-rs/TECHNICAL_DESIGN.md
View File

@@ -1,22 +1,24 @@
# 📘 `grel-rs` Technical Design Document (v3.0) # 📘 `grel-rs` Technical Design Document
**Binary:** `grel` | **Repository:** `grel-rs` **Binary:** `grel` | **Repository:** `grel-rs`
**Target Platforms:** Linux, Windows (macOS optional) **Target Platforms:** Linux, Windows (macOS optional)
**Core Philosophy:** Pure CLI, deterministic asset resolution, pacman/Artix-familiar UX, explicit over implicit, robust upgrade edge-case handling. **Core Philosophy:** Pure CLI, deterministic asset resolution, explicit over implicit, robust upgrade handling, transparent user control.
--- ---
## 1. Overview & Goals ## 1. Overview & Goals
`grel` (Global/General Release) is a terminal-native, high-performance package manager for downloading and managing binary releases from Git forges. It abstracts provider-specific APIs into a unified resolution pipeline, supports HTTP/SOCKS5 proxies, caches optimal DNS endpoints, downloads assets in parallel, and delivers a transparent, scriptable, pacman-compatible UX. `grel` is a terminal-native, high-performance release downloader and package manager for Git forges. It abstracts provider APIs into a unified pipeline, supports proxies, caches DNS/IPs for CDN routing, downloads in parallel, and delivers a transparent, scriptable, pacman-compatible UX.
**Key Requirements Met:** **Key Requirements Met:**
- ✅ Pure CLI (no TUI/GUI), standard `std::io` prompts - ✅ Pure CLI (no TUI/GUI), standard `std::io` prompts & warnings
- ✅ Deterministic asset resolution: **filter → sort → explicit prompt** (zero scoring magic) - ✅ Deterministic resolution: **strict filters → priority sorting → explicit policy fallback** (zero scoring)
- ✅ Robust parsing of non-standard release names via `Os`/`Arch` enums + alias mapping - ✅ `exclude_keywords` config to block installer/setup/bundle artifacts
- ✅ Format control: `.deb`/`.rpm`/`.msi`/`.dmg` disabled by default - ✅ Warning system for unmanaged or extra-step packages (applies to `ignore_formats` overrides & keyword matches)
- ✅ Pacman/Artix-style numbered selection with `[default]`, TTY/CI fallback, `-y`/`--noconfirm` - ✅ Configurable `download_dir` for unmanaged packages (defaults to OS `Downloads/`)
- ✅ `-Syu` resilience: detects filename renames, handles repo 404s via orphan tracking, explicit migration path - ✅ `default_selection_policy` (`first` | `largest`) as explicit fallback, overridden by detailed flags
- ✅ Cross-platform PATH integration (`grel path add`), XDG-compliant, no `sudo` - ✅ `-Syu` resilience: filename rename detection, orphan tracking, explicit migration
- ✅ Forge-agnostic provider routing (public + self-hosted URLs) - ✅ Cross-platform PATH integration, XDG-compliant, no `sudo`
--- ---
@@ -46,69 +48,64 @@ grel-rs/
--- ---
## 3. CLI Specification & Pure UX ## 3. CLI Specification & Warning Flow
No TUI frameworks. All interaction uses standard terminal I/O with predictable, pipe-safe behavior. All interaction uses standard terminal I/O. Unmanaged packages trigger explicit warnings & confirmation.
### Commands & Aliases ### Warning & Confirmation Flow
| Command | Aliases | Description | When an asset matches `exclude_keywords` or falls under an overridden `ignore_formats`:
|---------|---------|-------------|
| `grel sync foo/bar` | `S`, `-S` | Download & install latest |
| `grel sync foo/bar@1.2.3` | `-S` | Pin version |
| `grel search keyword` | `Ss`, `-Ss` | Search across registered forges |
| `grel upgrade` | `Syu`, `-Syu` | Refresh + upgrade installed |
| `grel info foo/bar` | `Si`, `-Si` | Show release metadata |
| `grel list` | `Q`, `-Q` | List installed packages (shows status) |
| `grel remove foo/bar` | `R`, `-R` | Uninstall & clean DB |
| `grel clean` | `Sc`, `-Sc` | Purge artifact/metadata cache |
| `grel path add` | - | Generate shell/registry snippets for `$PATH` |
| `grel migrate old/path new/path` | - | Remap repo path in state DB |
### Pure CLI Prompt (Artix/Pacman Compatible)
``` ```
:: 3 compatible asset(s) found for linux/x86_64 ⚠️ Asset "foo-setup-1.0.0.exe" matches excluded keyword "setup".
1) foo-1.0.0-linux-x86_64.tar.gz | 12.4 MB | tar.gz (default) ℹ️ grel cannot manage installers directly. Package will download to ~/Downloads/.
2) foo-1.0.0-linux-x86_64-musl.tar.gz | 10.2 MB | tar.gz :: Proceed with download? [y/N]: _
3) foo-1.0.0-linux-amd64.tar.gz | 11.8 MB | tar.gz
:: Select asset to download [1]: _
``` ```
- Accepts `1`–`3` or `Enter` (selects `[1]`) - `y`/`Enter` → Downloads to `download_dir`, marks `is_managed = false` in DB
- Invalid input → `:: Invalid selection. Enter a number [1-3]: ` - `N`/`n`/`Esc` → Aborts sync for this package, continues with others
- `--noconfirm` / `-y` → skips prompt, selects `[1]` - `--noconfirm` / `-y` → Auto-accepts, prints `ℹ️ Non-interactive: accepted unmanaged asset` to `stderr`
- **Non-Interactive Fallback:** `!std::io::stdin().is_terminal()` → auto-selects `[1]`, prints `ℹ️ Non-interactive mode. Using: <filename>` to `stderr`, never blocks CI/pipes. - **Never blocks CI/pipes:** `!std::io::stdin().is_terminal()` → auto-accepts, logs to `stderr`
--- ---
## 4. Asset Resolution Pipeline (Deterministic) ## 4. Asset Resolution Pipeline (Deterministic)
**No scoring. No fuzzy logic.** Strict filtering → transparent sorting → explicit selection. **No scoring. No fuzzy logic.** Strict filtering → transparent priority → policy fallback.
### Step 1: Tokenization ### Precedence Rules (Strict → Override → Fallback)
Filenames are split on non-alphanumeric characters. Tokens are matched against case-insensitive alias maps for `Os` and `Arch`. | Priority | Mechanism | Override Capability |
```rust |----------|-----------|---------------------|
// foo-v1.2.3-win64-setup.exe → Os::Windows, Arch::X86_64 | 1️⃣ | OS/Arch exact match or `Unknown` | None |
// bar_1.0.0_linux_amd64.tar.gz → Os::Linux, Arch::X86_64 | 2️⃣ | `exclude_keywords` filter | None (hard block unless CLI `--allow-keyword`) |
// app.Darwin.arm64.zip → Os::MacOS, Arch::Aarch64 | 3️⃣ | `ignore_formats` filter | Overridable via CLI/config |
``` | 4️⃣ | `arch_priority` index | Overrides `default_selection_policy` |
Unknown tokens fall back to `Os::Unknown("...")` or `Arch::Unknown("...")` → never crash, just filter out later. | 5️⃣ | `prefer_formats` index | Overrides `default_selection_policy` |
| 6️⃣ | `default_selection_policy` | **Only applies to remaining ties** |
### Step 2: Strict Filtering ### Step 1: Strict Filtering
Assets are filtered in fixed order. Failure at any step → discard.
```rust ```rust
1. OS matches target OR is Unknown assets.iter()
2. Arch matches priority list OR fallback allowed (32-bit on 64-bit) .map(|a| AssetTokens::from_filename(&a.filename))
3. Format NOT in `ignore_formats` (deb/rpm/msi/dmg disabled by default) .filter(|t| t.os == target.os || t.os == Os::Unknown)
.filter(|t| !keyword_excluded(t, &config.exclude_keywords))
.filter(|t| arch_matches_priority(t, &config.arch_priority, config.fallback_to_32bit))
.filter(|t| !format_ignored(t, &config.ignore_formats))
.collect()
``` ```
### Step 3: Deterministic Sorting ### Step 2: Deterministic Sorting
Remaining assets sorted by explicit cascade: Sorted by explicit cascade:
1. `arch_priority` index (lower = better) 1. `arch_priority` index
2. `prefer_formats` index 2. `prefer_formats` index
3. Lexicographic filename 3. Lexicographic filename
4. Size descending (tie-breaker) 4. Size descending
### Step 4: Selection ### Step 3: Policy Fallback (`default_selection_policy`)
- `0` assets → `❌ No compatible assets. Run grel sync --list-assets` Only applied if `>1` asset survives sorting and remains tied.
- `1` asset → Auto-select, log `ℹ️ Selected: <filename>` - `first` → picks top of sorted list
- `>1` assets → Pure CLI numbered prompt (see §3) - `largest` → picks by `size_bytes` descending
- **Never overrides** arch/format priority or keyword filters.
### Step 4: Selection & Warning
- `0` → `❌ No compatible assets`
- `1` → Check if unmanaged → warn/confirm → auto-select or prompt
- `>1` → Pure CLI numbered prompt → warn/confirm if unmanaged
--- ---
@@ -129,55 +126,85 @@ pub enum Arch { X86_64, Aarch64, I686, ArmV7, ArmV6, Riscv64, S390x, PowerPC64,
--- ---
## 6. Configuration vs State Management ## 6. Configuration Structure (Updated)
**Strict separation.** Humans edit TOML. Machines manage SQLite. Strict separation of human-editable TOML and machine-managed SQLite.
### `~/.config/grel/config.toml`
```toml ```toml
# ~/.config/grel/config.toml
[general] [general]
version = 1 version = 1
max_concurrent = 4 max_concurrent = 4
proxy = "http://127.0.0.1:7890" proxy = "http://127.0.0.1:7890"
[assets] [assets]
default_selection_policy = "largest" # first | largest (tiebreaker only)
exclude_keywords = ["setup", "installer", "portable", "bundle", "nupkg"]
ignore_formats = ["*.deb", "*.rpm", "*.msi", "*.dmg", "*.pkg", "*.AppImage"] ignore_formats = ["*.deb", "*.rpm", "*.msi", "*.dmg", "*.pkg", "*.AppImage"]
prefer_formats = ["*.tar.gz", "*.tar.xz", "*.zip", "*.exe"] prefer_formats = ["*.tar.gz", "*.tar.xz", "*.zip", "*.exe"]
arch_priority = ["x86_64", "aarch64", "x86", "armv7"] arch_priority = ["x86_64", "aarch64", "x86", "armv7"]
fallback_to_32bit = true fallback_to_32bit = true
prefer_musl = false prefer_musl = false
[migrations] [paths]
"legacy/old-tool" = "new-org/old-tool" install_root = "~/.local/share/grel"
bin_dir = "~/.local/share/grel/bin"
download_dir = "~/Downloads" # Fallback for unmanaged/extra-op packages
[upgrade] [upgrade]
check_interval_hours = 6 check_interval_hours = 6
max_parallel_checks = 10 max_parallel_checks = 10
[migrations]
"legacy/old-tool" = "new-org/old-tool"
``` ```
### `state.sqlite` Schema (`installed` table) ---
## 7. State Database Schema (`state.sqlite`)
```sql ```sql
CREATE TABLE installed ( CREATE TABLE installed (
id INTEGER PRIMARY KEY AUTOINCREMENT, id INTEGER PRIMARY KEY AUTOINCREMENT,
forge TEXT NOT NULL, -- 'github', 'gitlab', or self-hosted base URL
forge TEXT NOT NULL,
owner TEXT NOT NULL, owner TEXT NOT NULL,
repo TEXT NOT NULL, repo TEXT NOT NULL,
version TEXT NOT NULL, -- Strict semver version TEXT NOT NULL,
asset_filename TEXT NOT NULL, -- Exact remote filename used asset_filename TEXT NOT NULL,
checksum TEXT, checksum TEXT,
install_path TEXT NOT NULL, install_path TEXT NOT NULL, -- Actual path (bin/ or download_dir)
is_managed BOOLEAN NOT NULL DEFAULT 1,
status TEXT NOT NULL DEFAULT 'active' CHECK (status IN ('active', 'orphaned', 'migrated')), status TEXT NOT NULL DEFAULT 'active' CHECK (status IN ('active', 'orphaned', 'migrated')),
orphaned_at INTEGER, -- Unix timestamp when marked unreachable orphaned_at INTEGER,
last_checked INTEGER, -- For smart polling intervals last_checked INTEGER,
installed_at INTEGER DEFAULT (strftime('%s', 'now')) installed_at INTEGER DEFAULT (strftime('%s', 'now'))
); );
CREATE UNIQUE INDEX idx_pkg_unique ON installed(forge, owner, repo); CREATE UNIQUE INDEX idx_pkg_unique ON installed(forge, owner, repo);
CREATE INDEX idx_status ON installed(status); CREATE INDEX idx_status ON installed(status);
``` ```
- `is_managed`: `true` = auto-extracted/linked to `bin/`; `false` = left in `download_dir`
- `install_path`: Stores actual destination for accurate cleanup/migration
--- ---
## 7. `-Syu` Upgrade Logic & Edge Cases ## 8. Download Path & Installation Behavior
### 7.1 Filename Change Detection | Asset Type | Destination | Management |
|------------|-------------|------------|
| Standard binary/archive (`.tar.gz`, `.zip`, `.exe`) | `bin_dir/` (Linux) / `bin\` (Win) | ✅ Managed (extracted, checksummed, linked) |
| Unmanaged/Extra-op (`.msi`, `.deb`, matched keywords) | `download_dir` (default: `~/Downloads`) | ⚠️ Download-only, no extraction/execution |
| CLI Override (`--output-dir ~/tmp`) | User-specified path | ✅ Respected regardless of type |
**Cross-Platform Default Resolution:**
```rust
fn default_download_dir() -> PathBuf {
directories::UserDirs::new()
.map(|d| d.download_dir().clone())
.unwrap_or_else(|| std::env::temp_dir().join("grel-downloads"))
}
```
---
## 9. `-Syu` Upgrade Logic & Edge Cases
### 9.1 Filename Change Detection
Authors often rename assets (`x86_64` → `amd64`, `.tar.gz` → `.tar.xz`). Authors often rename assets (`x86_64` → `amd64`, `.tar.gz` → `.tar.xz`).
1. Load stored `asset_filename` from DB 1. Load stored `asset_filename` from DB
2. Resolve new release → filter → sort → get `new_best` 2. Resolve new release → filter → sort → get `new_best`
@@ -188,7 +215,7 @@ Authors often rename assets (`x86_64` → `amd64`, `.tar.gz` → `.tar.xz`).
``` ```
4. Download → verify → extract → **update DB record** with new filename. Proceeds safely. 4. Download → verify → extract → **update DB record** with new filename. Proceeds safely.
### 7.2 Repository Rename / 404 Handling ### 9.2 Repository Rename / 404 Handling
1. Provider fetch → `404` or unreachable 1. Provider fetch → `404` or unreachable
2. Mark `status = 'orphaned'`, set `orphaned_at = now()` 2. Mark `status = 'orphaned'`, set `orphaned_at = now()`
3. Skip in future `-Syu` runs 3. Skip in future `-Syu` runs
@@ -199,7 +226,7 @@ Authors often rename assets (`x86_64` → `amd64`, `.tar.gz` → `.tar.xz`).
``` ```
5. **No auto-migration.** Explicit user action required. 5. **No auto-migration.** Explicit user action required.
### 7.3 Orphaned Visibility ### 9.3 Orphaned Visibility
```bash ```bash
$ grel list $ grel list
:: Installed packages (3 active, 1 orphaned) :: Installed packages (3 active, 1 orphaned)
@@ -210,7 +237,7 @@ $ grel list
--- ---
## 8. Networking, Proxy & DNS/IP Caching ## 10. Networking, Proxy & DNS/IP Caching
- **Proxy Priority:** `--proxy` > `GREL_PROXY` env > `http_proxy`/`all_proxy` > `config.toml` - **Proxy Priority:** `--proxy` > `GREL_PROXY` env > `http_proxy`/`all_proxy` > `config.toml`
- **DNS/IP Cache:** - **DNS/IP Cache:**
1. Resolve `A/AAAA` via `hickory-resolver` 1. Resolve `A/AAAA` via `hickory-resolver`
@@ -221,7 +248,7 @@ $ grel list
--- ---
## 9. Rate Limiting & ETag Optimization ## 11. Rate Limiting & ETag Optimization
GitHub: 60/hr unauth, 5000/hr auth. `grel` handles gracefully: GitHub: 60/hr unauth, 5000/hr auth. `grel` handles gracefully:
1. **ETag Caching:** `If-None-Match` → `304 Not Modified` **free**. Cached in DB. 1. **ETag Caching:** `If-None-Match` → `304 Not Modified` **free**. Cached in DB.
2. **Smart Polling:** Only checks packages where `last_checked < now() - check_interval_hours` 2. **Smart Polling:** Only checks packages where `last_checked < now() - check_interval_hours`
@@ -231,7 +258,7 @@ GitHub: 60/hr unauth, 5000/hr auth. `grel` handles gracefully:
--- ---
## 10. PATH Integration & Post-Install ## 12. PATH Integration & Post-Install
- Installs to `~/.local/share/grel/bin/` (Linux) / `%LOCALAPPDATA%\grel\bin\` (Windows) - Installs to `~/.local/share/grel/bin/` (Linux) / `%LOCALAPPDATA%\grel\bin\` (Windows)
- `grel path add` prints exact shell/registry snippets: - `grel path add` prints exact shell/registry snippets:
```bash ```bash
@@ -245,17 +272,17 @@ GitHub: 60/hr unauth, 5000/hr auth. `grel` handles gracefully:
--- ---
## 11. Security & Reliability ## 13. Security & Reliability
- ✅ TLS: `rustls` only (no native OpenSSL) - ✅ TLS: `rustls` only
- ✅ Checksums: `sha256` verification before extraction - ✅ Checksums: `sha256` before extraction (managed packages only)
- ✅ Archive safety: Reject `..`, absolute paths, external symlinks - ✅ Archive safety: Reject `..`, absolute paths, external symlinks
- ✅ Atomic installs: Temp dir → verify → `rename` → DB update - ✅ Atomic installs: Temp dir → verify → `rename` → DB update
- ✅ No auto-exec: Binaries installed but not run unless invoked - ✅ Unmanaged warnings: Explicit user consent required (unless `--noconfirm`)
- ✅ Graceful degradation: Network errors → retry/backoff, rate limits → sleep/queue, missing assets → clear error + suggestions - ✅ Graceful degradation: Network errors → retry/backoff, rate limits → sleep/queue, missing assets → clear error + suggestions
--- ---
## 12. Dependency Matrix ## 14. Dependency Matrix
```toml ```toml
# Core & Async # Core & Async
tokio = { version = "1", features = ["full"] } tokio = { version = "1", features = ["full"] }
@@ -300,12 +327,12 @@ tracing-subscriber = { version = "0.3", features = ["env-filter", "fmt"] }
--- ---
## 13. Testing & CI Strategy ## 15. Testing & CI Strategy
- **Unit:** Token extraction from 20+ messy filenames, filter/sort determinism, config parsing, migration logic - **Unit:** Keyword exclusion, policy fallback precedence, filter/sort determinism, config parsing
- **Mocked Providers:** `wiremock` for 200/304/403/429/500, rate limit header injection - **Mocked Providers:** `wiremock` for 200/304/403/429/500, rate limit headers
- **Integration:** Download fixtures → extract → verify checksums & paths, `grel migrate` DB state changes - **Integration:** Unmanaged package download → `download_dir` verification, `grel migrate` state changes
- **Non-Interactive:** `echo "" | grel sync foo/bar` must not hang, must log to stderr - **Non-Interactive:** `echo "" | grel sync foo/bar` must auto-accept with stderr warning
- **Cross-Platform CI:** `ubuntu-latest`, `windows-latest` (GitHub Actions) - **Cross-Platform CI:** `ubuntu-latest`, `windows-latest`
- **Performance:** `criterion` for DNS/IP cache + parallel download throughput - **Performance:** `criterion` for DNS/IP cache + parallel download throughput
- **Fuzzing:** `cargo-fuzz` on `AssetTokens::from_filename()` + archive path validation - **Fuzzing:** `cargo-fuzz` on `AssetTokens::from_filename()` + archive path validation