manbo/blog-post-backup
1
0
Fork 0
Code Issues 88 Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
a969692d048468e808cfd45f77f2dbf2c7d4150b
blog-post-backup/designs/grel-rs/TECHNICAL_DESIGN.md

13 KiB
Raw Blame History

📘 grel-rs Technical Design Document

Binary: grel | Repository: grel-rs
Target Platforms: Linux, Windows (macOS optional)
Core Philosophy: Forge-agnostic, pacman-familiar, performance-first, secure, extensible via traits.

1. Overview & Goals

grel (Global/General Release) is a terminal-native, high-performance package manager for downloading and managing binary releases from Git forges. It abstracts provider-specific APIs into a unified resolution/download pipeline, supports HTTP/SOCKS5 proxies, caches optimal DNS endpoints, downloads assets in parallel, and presents a polished, pacman-compatible UX.

Key Requirements Met:

  • ✅ Pacman-style CLI (-S, -Syu, -Ss, --noconfirm)
  • ✅ Proxy support (HTTP/HTTPS/SOCKS5) with env/config fallback
  • ✅ DNS/IP latency caching + RTT probing for CDN optimization
  • ✅ Parallel asset downloads with multi-progress UI
  • ✅ Interactive asset selection (pacman/artix style) with TTY/CI fallback
  • ✅ Cross-platform (Linux/Windows) with safe extraction & PATH integration
  • ✅ Extensible provider architecture (GitHub, GitLab, Codeberg, Gitea, self-hosted)

2. Workspace Architecture

grel-rs/
├── Cargo.toml                  # Workspace root (resolver = "2")
├── crates/
│   ├── grel-cli/               # CLI parsing, TUI orchestration, prompts, progress
│   ├── grel-core/              # Resolution, versioning, asset matching, upgrade state
│   ├── grel-providers/         # Forge trait, registry, API implementations (modules)
│   ├── grel-network/           # HTTP client, proxy routing, IP cache resolver, downloader
│   ├── grel-cache/             # SQLite state, IP cache, artifact storage, TTL eviction
│   └── grel-config/            # Layered config (CLI > env > TOML > defaults), migrations
├── tests/                      # Integration, mock servers, e2e fixtures
└── scripts/                    # CI, release, benchmark helpers

Crate Boundaries:

Crate Owns Depends On
grel-cli CLI, UI, command routing, prompts grel-core, grel-config, dialoguer, indicatif
grel-core Package resolution, asset matching, upgrade logic grel-providers, grel-cache, semver
grel-providers API clients, pagination, rate-limit handling reqwest, async-trait, chrono
grel-network Proxy, IP resolver, parallel downloads, streaming reqwest, hickory-resolver, dashmap, tokio
grel-cache SQLite state, IP cache, artifact storage, LRU sqlx, sha2, tar/zip, directories
grel-config Settings loading, validation, defaults, migrations figment, serde, toml

3. Core Abstractions & Data Flow

All forges implement a single normalized contract. grel-core parses user input → providers return forge-agnostic structs → grel-network downloads blindly.

pub enum ForgeType { GitHub, GitLab, Gitea, Codeberg, SelfHosted(String) }

pub struct PackageRef {
    pub forge: ForgeType,
    pub owner: String, pub repo: String,
    pub version: Option<semver::Version>,
    pub prerelease: bool,
}

pub struct ResolvedRelease {
    pub version: semver::Version,
    pub published_at: chrono::DateTime<Utc>,
    pub is_prerelease: bool,
    pub assets: Vec<DownloadAsset>,
    pub release_notes: Option<String>,
}

pub struct DownloadAsset {
    pub filename: String,
    pub download_url: url::Url,
    pub size_bytes: u64,
    pub sha256: Option<String>,
    pub asset_type: AssetKind, // Binary, Archive, Source, Other
}

Flow:

CLI → PackageRef → ProviderRegistry → Provider.resolve() → ResolvedRelease
   → AssetResolver (tiered match + checksum pairing) → SelectedRelease
   → Network (IP cache → proxy client → parallel download)
   → Cache (verify sha256 → extract → atomic rename → update state.sqlite)
   → CLI (progress summary → exit)

4. CLI Specification & UX Design

clap 4 with subcommands. Pacman short flags supported via aliases.

Command Aliases Description
grel sync foo/bar S, -S Download & install latest
grel sync foo/bar@1.2.3 -S Pin version
grel search keyword Ss, -Ss Search across registered forges
grel upgrade Syu, -Syu Refresh + upgrade installed
grel info foo/bar Si, -Si Show release metadata
grel list Q, -Q List installed packages
grel remove foo/bar R, -R Uninstall
grel clean Sc, -Sc Purge artifact/metadata cache
grel path add - Inject bin/ dir into shell/Windows PATH

UX Features:

  • tabled for search/list output
  • indicatif::MultiProgress for parallel downloads
  • owo-colors for status: 🟢 ✅, 🟡 ⚠️, 🔴 ❌, 🔵 ℹ️
  • --json for machine-readable output
  • --noconfirm / -y skips all prompts

5. PATH Integration & Post-Install Behavior

Problem: User-space installs aren't in $PATH by default. Pacman installs globally (/usr/bin).

Solution: Dedicated bin/ directory + explicit shell integration.

~/.local/share/grel/ (Linux)   →  bin/
%LOCALAPPDATA%\grel\ (Windows) →  bin\

Implementation:

  1. grel sync extracts binaries to ~/.local/share/grel/bin/ (or equivalent)
  2. grel path add detects shell ($SHELL, $PROFILE) and appends: 
    export PATH="$HOME/.local/share/grel/bin:$PATH"  # bash/zsh
$env:Path = "$env:LOCALAPPDATA\grel\bin;" + $env:Path  # PowerShell
  3. Windows: Uses registry HKCU\Environment\Path + broadcasts WM_SETTINGCHANGE
  4. First-run hint: 💡 Run grel path add to enable global command access
  5. Why not /usr/local/bin? Avoids sudo, respects XDG, prevents system package conflicts.

6. Configuration vs State Management

Strict Separation:

Layer Format Location Purpose
User Config config.toml ~/.config/grel/ (Linux) / %APPDATA%\grel\ (Win) Proxy, tokens, concurrency, asset preferences
Program State state.sqlite ~/.local/share/grel/db/ Installed packages, IP cache, ETags, last-checked
Artifact Cache Files ~/.cache/grel/artifacts/ Downloaded archives (LRU evicted, 2GB default)

config.toml Example:

[general]
version = 1
max_concurrent = 4
proxy = "http://127.0.0.1:7890"

[assets]
default_selection_policy = "largest"  # first | largest | preferred_format
prefer_formats = ["*.tar.gz", "*.zip", "*.exe"]
ignore_patterns = ["*source*", "*.deb", "*.rpm"]

[upgrade]
check_interval_hours = 6
max_parallel_checks = 10

Migrations:

  • Config: version field in TOML. On load, grel-config runs migration functions (rename keys, add defaults, warn on breaking changes).
  • SQLite: sqlx migrate with timestamped files. Applied automatically on startup. Zero-downtime schema upgrades.

7. Asset Resolution & Selection Pipeline

Releases contain multiple formats. grel uses deterministic tiered matching + interactive fallback.

Tiered Matching (Order Matters)

  1. Exact: {repo}-{version}-{os}-{arch}.{fmt}
  2. OS+Arch: *{os}*{arch}*.{fmt}
  3. OS Only: *{os}*.{fmt}
  4. Universal Archive: *.{tar.gz,tar.xz,zip}
  5. Universal Binary: *.{exe,elf,dll,dylib}

Checksum & Signature Auto-Pairing

If main asset is foo-1.0.0-linux-amd64.tar.gz, grel searches for:

  • foo-1.0.0-linux-amd64.tar.gz.sha256
  • sha256sums.txt (parses line matching filename) Automatically downloads & verifies before extraction.

Interactive Selection (Pacman/Artix Style)

  • Uses dialoguer::Select with numbered list
  • Default marker: (default) shown next to highest-priority match
  • --noconfirm / -y skips prompt, uses default
  • TTY Detection: If not interactive (CI/pipes), auto-selects default + prints ℹ️ Non-interactive mode. Using default: <filename> to stderr
  • Timeout fallback (configurable): auto-selects default if no input within prompt_timeout_secs

CLI Overrides:

grel sync foo/bar --asset custom.exe
grel sync foo/bar --platform linux/aarch64
grel sync foo/bar --list-assets  # Dry-run table

8. Networking, Proxy & DNS/IP Caching

Proxy Support

  • Native via reqwest::Proxy::all(), http(), https()
  • Priority: --proxy > GREL_PROXY > http_proxy/all_proxy > config.toml
  • Supports http://user:pass@host, socks5://..., socks5h://...

DNS/IP Caching

⚠️ No hardcoded IPs. CDNs rotate frequently. Instead:

  1. Resolve A/AAAA via hickory-resolver
  2. Parallel TcpStream::connect probes to all IPs on 443
  3. Store fastest IP + RTT in SQLite with TTL (default 300s)
  4. Use reqwest::ClientBuilder::resolve(host, ip) for routing
  5. Background refresh thread evicts stale entries, pre-warms hot domains

9. Rate Limiting & -Syu Optimization

GitHub allows 60 req/hr unauthenticated, 5000/hr authenticated. grel handles this gracefully:

  1. ETag Caching: If-None-Match → 304 Not Modified does not count against rate limits. Cached ETags stored in state.sqlite.
  2. Smart Polling: Only recheck packages where last_checked < now() - check_interval_hours.
  3. Rate Limit Awareness: Parse X-RateLimit-Remaining & X-RateLimit-Reset. If < 50, sleep until reset or queue remaining checks.
  4. Auth Token: Strongly recommended. grel prompts on first run: 🔑 Set GREL_GITHUB_TOKEN for 5000 req/hr.
  5. --no-api Fallback: Skips remote checks, uses local manifest + cache timestamps.
  6. Parallel Batch Checks: max_parallel_checks limits concurrent API calls to avoid hitting burst limits.

-Syu Flow:

Load state → Filter stale → Batch API (ETag) → Parse 200/304 → Build diff → Prompt → Download → Verify → Update

10. Cross-Platform Implementation Notes

Aspect Linux Windows Implementation
Paths ~/.local/share/grel %LOCALAPPDATA%\grel directories::ProjectDirs
Archives tar + zstd/xz/gz zip + tar Feature-gated extraction
Exec Perms 0o755 FILE_ATTRIBUTE_ARCHIVE cfg(unix) vs cfg(windows)
Path Safety Reject .., absolute paths Same sanitize-filename + strict validation
Executable Suffix None .exe Auto-append if cfg(windows) & missing
Shell Integration .bashrc, .zshrc, fish $PROFILE, Registry grel path add

11. Security & Reliability Guarantees

  • ✅ TLS: rustls only (no native OpenSSL)
  • ✅ Checksums: Verify sha256 before extraction
  • ✅ Archive Safety: Reject .., absolute paths, symlinks to outside dest
  • ✅ Atomic Installs: Download to temp dir → verify → rename → register in DB
  • ✅ No Auto-Execute: Binaries installed but not run unless invoked by user
  • ✅ Token Security: Loaded from env/config, never logged or serialized to stdout
  • ✅ Graceful Degradation: Network errors → retry with backoff, rate limits → sleep/queue, missing assets → clear error + suggestions

12. Dependency Matrix

# Core & Async
tokio = { version = "1", features = ["full"] }
futures = "0.3"
async-trait = "0.1"

# CLI & UX
clap = { version = "4", features = ["derive", "wrap_help", "string"] }
indicatif = { version = "0.17", features = ["tokio"] }
tracing-indicatif = "0.3"
dialoguer = "0.11"
tabled = "0.16"
owo-colors = "4"

# Network & Proxy
reqwest = { version = "0.12", features = ["rustls-tls", "json", "socks", "stream"] }
hickory-resolver = "0.24"
dashmap = "6"

# Cache & Storage
sqlx = { version = "0.8", features = ["runtime-tokio-rustls", "sqlite"] }
sha2 = "0.10"
directories = "5"
tar = "0.4"
zip = "2"
zstd = "0.13"
xz2 = "0.1"
bzip2 = "0.5"
sanitize-filename = "0.5"

# Config & Utils
figment = { version = "0.10", features = ["toml", "env"] }
semver = "1"
url = "2"
chrono = { version = "0.4", features = ["serde"] }
serde = { version = "1", features = ["derive"] }
anyhow = "1"
thiserror = "2"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "fmt"] }

Note: Rust 1.70+ required for std::io::IsTerminal (replaces atty).

13. Testing & CI Strategy

  • Unit: Parsing, version comparison, cache TTL math, asset matching tiers
  • Mocked Providers: wiremock for API responses (200, 304, 403, 429, 500)
  • Integration: Download test fixtures → extract → verify structure/checksums
  • Cross-Platform CI: ubuntu-latest, windows-latest (GitHub Actions)
  • Performance: criterion for DNS resolution + parallel download throughput
  • Fuzzing: cargo-fuzz on PackageRef parser + archive path validation
  • Non-Interactive Tests: echo "" | grel sync foo/bar must not hang
Reference in New Issue View Git Blame Copy Permalink