From ca5ced7ac1b2ab9d0c7d2a6176e99ddfb92062e0 Mon Sep 17 00:00:00 2001 From: tegwick Date: Sat, 2 May 2026 22:08:38 +0200 Subject: [PATCH] This seems to be our first runnable version on railiance01 --- .claude/scheduled_tasks.lock | 1 + HaskellVibePrimer.md | 426 +++++++++++++++++++++++++++++++++++ 2 files changed, 427 insertions(+) create mode 100644 .claude/scheduled_tasks.lock create mode 100644 HaskellVibePrimer.md diff --git a/.claude/scheduled_tasks.lock b/.claude/scheduled_tasks.lock new file mode 100644 index 0000000..90f1bd9 --- /dev/null +++ b/.claude/scheduled_tasks.lock @@ -0,0 +1 @@ +{"sessionId":"7a72372c-e488-456c-baba-1e60d38649cf","pid":9468,"procStart":"127351","acquiredAt":1777404376811} \ No newline at end of file diff --git a/HaskellVibePrimer.md b/HaskellVibePrimer.md new file mode 100644 index 0000000..eea9b15 --- /dev/null +++ b/HaskellVibePrimer.md @@ -0,0 +1,426 @@ +# Haskell Vibe Primer +## Hard-won lessons for coding agents working on inter-hub and IHP projects + +--- + +## Quick orientation + +Inter-hub is an IHP (Integrated Haskell Platform) v1.5 application on GHC 9.10.3, managed via Nix/devenv. It has two distinct compilation modes with fundamentally different behaviour: + +| Mode | Command | Compiler | Output | Crashes? | +|------|---------|----------|--------|----------| +| Dev | `devenv up` → ghcid | GHCi byte-code | In-memory | No (byte-code avoids static linker) | +| Production | `nix build .#docker` | Native (cabal) | OCI image | Yes, if archive bug present | + +A build that works in `devenv up` does **not** guarantee `nix build .#docker` will succeed. The production build hits code paths that ghcid never reaches. + +The build host for production is **haskelseed** (192.168.178.135, root/hcs26!x), not CoulombCore. Do not try to run `nix build` on CoulombCore. + +--- + +## Part 1 — Known GHC 9.10.3 bugs in this project + +### Bug 1: `module M` re-export causes `.hi` overflow (FIXED in flake.nix) + +**Symptom:** +``` +Data.Binary.Get.runGet at position ~287000000: not enough bytes +``` +Crash occurs while GHC reads a `.hi` (interface) file. + +**Cause:** IHP generates a hub module `Generated/ActualTypes.hs` that originally used: +```haskell +module Generated.ActualTypes + ( module Generated.Foo + , module Generated.Bar + ... -- 61 sub-modules + ) where +import Generated.Foo +import Generated.Bar +``` + +The `module M` re-export syntax causes GHC to **embed the full exported interface of every sub-module verbatim** into `ActualTypes.hi`. With 61 sub-modules and thousands of typeclass instances, the resulting `.hi` reaches ~287 MB — exceeding GHC 9.10.3's `Data.Binary.Get` 274 MB deserialization limit. + +**Fix (active in `flake.nix`):** The `inter-hub-models` postUnpack overlay rewrites the export list to explicit `T(..)` per-type exports: +```haskell +module Generated.ActualTypes + ( Foo(..) + , FooId(..) + , Bar(..) + , ... + ) where +``` +Explicit re-exports store only compact name-references in `.hi` (not embedded sub-interfaces). `ActualTypes.hi` drops from ~287 MB to ~51 KB. + +**Rule: NEVER use `module M` re-export syntax for generated hub modules with many sub-modules.** Always use explicit `T(..)` exports. + +```haskell +-- BAD: embeds sub-interfaces into hub .hi +module MyHub (module Sub1, module Sub2) where + +-- GOOD: stores compact references +module MyHub (Foo(..), Bar, baz) where +``` + +--- + +### Bug 2: Truncated `libHSghc-9.10.3-5702.a` in Nix store (FIXED on haskelseed) + +**Symptom:** +``` +panic! (the 'impossible' happened) + GHC version 9.10.3: + Data.Binary.Get.runGet at position 287686318: not enough bytes +``` +Crash occurs **after** `[477 of 477] Compiling Generated.WidgetVersionInclude` — all modules compiled, crash in post-compilation step. + +**Cause:** The Nix-provisioned static archive for the GHC compiler-as-library is truncated: +- Truncated: `ffg3yf2.../lib/.../ghc-9.10.3-5702/libHSghc-9.10.3-5702.a` — 287,768,576 bytes +- Full archive: `ffg3yf2.../ghc-9.10.3-partial/lib/.../ghc-9.10.3-5702/libHSghc-9.10.3-5702.a` — 289,295,782 bytes + +The last AR entry (`Expr.o`) claims 517,544 bytes but only 82,258 are present. GHC's `readAr` (via `Data.Binary.Get.runGet`) panics when it tries to read the full entry. + +Why GHC reads this archive at all: The global `ghc-with-packages` package db is searched during the build (cabal configure does not pass `--package-db=clear`). That db registers the `ghc` compiler-as-library package with `library-dirs` pointing to the directory containing the truncated archive symlink. GHC loads this archive during internal post-compilation processing — even when the package has **no Template Haskell**. + +**What does NOT fix it:** +- `-fomit-interface-pragmas` — reduces `.hi` size, unrelated to archive loading +- `--disable-shared` — affects output type, not input archive reading +- `-fexternal-interpreter -pgmi ghc-iserv-dyn` — routes TH to external process, but inter-hub-models has **zero TH**; the archive read is from a different code path + +**Fix applied on haskelseed (2026-05-02):** +```bash +TRUNCATED="/nix/store/ffg3yf2ypnbz3hc31y7nglrkihz0if01-ghc-9.10.3/lib/ghc-9.10.3/lib/x86_64-linux-ghc-9.10.3/ghc-9.10.3-5702/libHSghc-9.10.3-5702.a" +FULL="/nix/store/ffg3yf2ypnbz3hc31y7nglrkihz0if01-ghc-9.10.3/ghc-9.10.3-partial/lib/ghc-9.10.3/lib/x86_64-linux-ghc-9.10.3/ghc-9.10.3-5702/libHSghc-9.10.3-5702.a" +chmod u+w "$(dirname "$TRUNCATED")" +cp "$FULL" "$TRUNCATED" +chmod a-w "$TRUNCATED" "$(dirname "$TRUNCATED")" +``` + +**If the fix is lost** (flake lock update changing GHC version): +```bash +# Check archive size before building — should be ~289 MB +wc -c /nix/store/HASH-ghc-9.10.3/lib/ghc-9.10.3/lib/x86_64-linux-ghc-9.10.3/ghc-9.10.3-5702/libHSghc-9.10.3-5702.a +# If ~287 MB, look for ghc-9.10.3-partial/ in same store path and apply patch +``` + +--- + +### Bug 3: `--disable-shared` causes missing `.dyn_hi` files (FIXED in flake.nix) + +If `inter-hub-models` is built with `--disable-shared`, it produces only `.hi` files but no `.dyn_hi` (dynamic interface files). The dependent `inter-hub-lib` package (built with `--enable-shared` by default in NixPkgs) requires `.dyn_hi` from its dependencies and fails with: +``` +Failed to load dynamic interface file for Generated.Foo: + .../Generated/Foo.dyn_hi: does not exist +``` + +Do not add `--disable-shared` to the inter-hub-models configureFlags unless you also suppress `--enable-shared` in inter-hub-lib (and its transitive dependents). The archive fix (Bug 2) is the correct solution; `--disable-shared` was a failed hypothesis. + +--- + +## Part 2 — IHP compilation architecture + +### Compilation layers (from CLAUDE.md) + +``` +Layer 1 — stable core (compile once, expensive to change): + build/Generated/Types.hs IHP auto-generated from Schema.sql + Web/Types.hs controller/action type definitions + +Layer 2 — helpers (only touch if Layer 1 is clean): + Application/Helper/*.hs 12 helper modules + +Layer 3 — working surface (most errors live here): + Web/Controller/*.hs 59 controllers + Web/View/**/*.hs 120 views + +Layer 4 — wiring (fix last): + Web/FrontController.hs + Web/Routes.hs +``` + +**Fix errors bottom-up, one module at a time.** Never touch Layer 1 during an error-fix loop. A single change to `Web/Types.hs` invalidates all 59 controllers and 120 views simultaneously — equivalent to a full rebuild. + +### Package split + +The project compiles as two separate Cabal packages: + +| Package | Source | Contains | +|---------|--------|----------| +| `inter-hub-models` | `inter-hub-models-src/build/Generated/` | All IHP-generated types, ~477 modules | +| `inter-hub-lib` | `inter-hub-lib-src/` | Application code: `Application/`, `Config/`, `Web/` | + +Critical: **`inter-hub-lib-src` has no `build/` directory.** Generated modules come from the `inter-hub-models` package, not from the lib source tree. Any script that assumes lib-src contains `build/Generated/` will fail. To access generated file content from within inter-hub-lib's build scripts, find inter-hub-models-src dynamically: +```bash +_models_src=$(find /nix/store -maxdepth 1 -name "*-inter-hub-models-src" ! -name "*.drv" | head -1) +``` + +### Generated code — what changes when schema changes + +When `Application/Schema.sql` changes and you regenerate via the IHP IDE: +- `build/Generated/Types.hs` — one-liner re-export hub (regenerated, do not edit) +- `build/Generated/ActualTypes.hs` — type hub (regenerated, patched by postUnpack overlay) +- `build/Generated/Foo.hs` — one per table, data type + instances +- `build/Generated/FooInclude.hs` — `type instance Include` for eager loading +- `build/Generated/ActualTypes/Foo.hs` — sub-module used by ActualTypes hub + +After any schema change, check whether the postUnpack in `flake.nix` still correctly rewrites `Generated.ActualTypes` — run `nix build --keep-failed` and inspect the rewritten file. + +--- + +## Part 3 — Nix/devenv specifics + +### How the Nix overlay works + +The `flake.nix` has an `overlays = lib.mkAfter [...]` block in `devenv.shells.default` that overrides the Haskell package set. It intercepts derivations by `pname`: + +```nix +mkDerivation = args: + let drv = hprev.mkDerivation args; + in if (args.pname or "") == "inter-hub-models" + then drv.overrideAttrs (old: { ... }) + else if (args.pname or "") == "inter-hub-lib" + then drv.overrideAttrs (old: { ... }) + else drv; +``` + +`postUnpack` runs after the source is unpacked but before configure. `postConfigure` runs after `setup configure`. Both run inside the Nix sandbox. + +### Environment variables in Nix builds + +- `$sourceRoot` — relative path to unpacked source (e.g., `inter-hub-models-src`) +- `$TMPDIR` — build-specific temp directory (e.g., `/nix/var/nix/builds/nix-PID-RND/`) +- `$NIX_BUILD_CORES` — parallelism (8 on haskelseed); NixPkgs Haskell builder passes `--ghc-option=-j$NIX_BUILD_CORES` by default +- `$NIX_BUILD_TOP` — may not be set; prefer `$TMPDIR` + +### Configure flags ordering matters + +NixPkgs Haskell builder prepends its own configureFlags before yours. Your overlay's flags are **appended** via `(old.configureFlags or []) ++ [ your-flags ]`. For boolean flags (`--enable-X` / `--disable-X`), the last one wins. For `--ghc-option=-jN`, all are passed and GHC uses the last one. + +Example of NixPkgs flags you must know about: +``` +--enable-shared +--enable-split-sections +--ghc-option=-j8 (from NIX_BUILD_CORES) +``` + +To override: append `--disable-shared`, `--disable-split-sections`, `--ghc-option=-j1` in your configureFlags. But beware: disabling shared breaks `.dyn_hi` for downstream packages. + +### The `ghc-with-packages` symlink chain + +The GHC wrapper used in builds (`ghc-with-packages`) exposes packages from both the wrapped GHC derivation and registered Haskell packages. Its `lib/` directory contains symlinks back into the unwrapped GHC derivation. When patching Nix store archives, verify the symlink chain resolves correctly: +```bash +readlink -f /nix/store/WRAP-ghc-9.10.3-with-packages/lib/.../libHSghc.a +wc -c /nix/store/WRAP-ghc-9.10.3-with-packages/lib/.../libHSghc.a +``` + +### Diagnosing `nix build` failures efficiently + +```bash +# Get full build log for a specific derivation +nix log /nix/store/HASH-inter-hub-models-0.1.0.drv + +# What derivations would be built (dry run) +nix build .#docker --dry-run + +# Keep failed build artifacts for inspection +nix build .#docker --keep-failed + +# Check if a store path exists (was the drv already built?) +ls /nix/store/EXPECTED-OUTPUT-HASH 2>/dev/null && echo "cached" || echo "not built" +``` + +--- + +## Part 4 — Haskell type system patterns + +### Interface files (`.hi` and `.dyn_hi`) + +GHC produces two kinds of interface files: +- `.hi` — static interface, for non-shared compilations +- `.dyn_hi` — dynamic interface, required when compiling with `--enable-shared` / `-dynamic-too` + +A package compiled with `--disable-shared` produces `.hi` only. Any package depending on it that was compiled with `--enable-shared` will fail to load the `.dyn_hi`. Always maintain consistent shared/static settings across a package dependency chain. + +### Type families vs Template Haskell + +`type instance` declarations (like `Include "widgetId" ...`) are **type-level computations processed entirely by GHC during type-checking**. They are NOT Template Haskell, do NOT require the TH runtime, and do NOT trigger archive loading via the internal linker. + +Template Haskell requires the TH evaluator (internal or external interpreter) only for modules with: +- `{-# LANGUAGE TemplateHaskell #-}` pragma +- Splice expressions `$(...)` or quasi-quotes `[| ... |]` +- `deriveGeneric`, `makeLenses`, etc. (lens/generics libraries) + +If a module has none of these, `-fexternal-interpreter` has no meaningful effect on it. + +### `module M` re-export vs explicit exports + +```haskell +-- EMBEDS sub-interface into .hi (dangerous for large hubs): +module Hub (module Sub) where +import Sub + +-- Stores compact name-reference in .hi (correct for large hubs): +module Hub (Foo(..), Bar, baz) where +import Sub (Foo(..), Bar, baz) +``` + +The `module M` form is fine for small modules (< 10 re-exported sub-modules with modest interfaces). It becomes a GHC 9.10.3 bomb for generated hubs with 50+ sub-modules full of typeclass instances. + +### AR archive format and `readAr` + +GHC's internal static linker uses `Data.Binary.Get.runGet` to read AR archives (`.a` files). Unlike `ar t` (which only reads headers), `readAr` reads **all entry content**. A truncated archive that passes `ar t` will still panic `readAr`. If you see: + +``` +panic! (the 'impossible' happened) + Data.Binary.Get.runGet at position N: not enough bytes +``` + +Run `ar tv /path/to/suspect.a | tail -5` — if the last entry shows a size, check whether `wc -c /path/to/suspect.a` is substantially smaller than expected. + +--- + +## Part 5 — Build compile tools and discipline + +### Compile check scripts + +Inside `devenv shell`: +```bash +scripts/compile-check # full build via ghcid (all layers), errors → /tmp/ihub-compile-errors.txt +scripts/compile-check --bg # background-friendly (no colour/title escape codes) +scripts/compile-check-core # Layer 1+2 only — verify clean base +``` + +### Error-fix discipline + +1. Fix **bottom-up**: Layer 1 → 2 → 3 → 4 +2. Fix **one module at a time** — wait for ghcid to reload before touching the next module +3. **Never change Layer 1 (Web/Types.hs, Generated/Types.hs) while debugging Layer 3 errors** — it restarts the entire recompile +4. `Generated/Types.hs` is auto-generated from `Application/Schema.sql`. Edit the schema in IHP IDE, not the generated file + +### Reading GHC error messages + +GHC errors reference **source locations**, not compiled output. When you see: +``` +Web/Controller/Foo.hs:42:5: error: + • Couldn't match expected type 'UUID' with actual type 'Text' +``` + +The fix is in `Web/Controller/Foo.hs:42`, not in any generated file. + +When you see: +``` +Failed to load interface for 'Generated.Foo' +``` +Check that `Generated.Foo` is in the `inter-hub-models` package and that models compiled successfully first. + +### Parallel compilation flag (`-j`) + +GHC's `-j` flag controls **parallel module compilation** within a single package. On haskelseed (2 CPU, ~3.8 GiB RAM), NixPkgs sets `-j8` (from `NIX_BUILD_CORES=8`). The overlay overrides to `-j1` for the models package to prevent OOM on the constrained host. Do not increase this without verifying available memory: +```bash +free -m # on haskelseed +``` + +The env var `GHCRTS = "-A32m -M2g"` caps the GHC heap at 2 GiB and sets minor GC allocation to 32 MB. These are set in `devenv.shells.default.env`. + +--- + +## Part 6 — Guardrails: expensive mistakes to avoid + +### NEVER do these + +| Action | Why it's expensive | +|--------|-------------------| +| Edit `build/Generated/Types.hs` directly | Regenerated on next schema sync; change is lost. Also, editing it changes Layer 1 and invalidates all 477 modules | +| Add `module M` re-export syntax to a new generated hub module | Embeds sub-interfaces → `.hi` overflow → GHC crash | +| Change `Web/Types.hs` or `Web/Types.hs` during a Layer 3 error loop | Restarts compile of all 59 controllers + 120 views | +| Add `--disable-shared` to models without removing it from lib | Missing `.dyn_hi` crash in every downstream package | +| Hardcode Nix store hashes in postUnpack scripts | Hash changes with every schema change; use `find /nix/store` instead | +| Run `nix build .#docker` on CoulombCore | Insufficient RAM (< 4 GiB), will OOM during GHC compilation | +| Trust `ar t` to validate an archive | `ar t` reads only headers; GHC reads content. Use `wc -c` and compare to expected size | +| Assume `devenv up` success = `nix build` success | Different compilation modes; static linker issues only surface in `nix build` | + +### Before modifying `flake.nix` overlays + +1. **Check what derivation hash the current flake produces**: `nix build .#docker --dry-run` +2. **Understand layer dependencies**: changing inter-hub-models configureFlags invalidates models AND all downstream (lib, binaries, docker image) +3. **Test postUnpack scripts locally first**: simulate with `bash -n yourscript.sh` and run `awk` commands against sample files +4. **Verify `.dyn_hi` will be produced** if `--enable-shared` is set (which is the NixPkgs default) + +### Diagnosing a new crash before trying fixes + +1. Get the full log: `nix log /nix/store/HASH-inter-hub-models-0.1.0.drv` +2. Find the crash position: `Data.Binary.Get.runGet at position N` +3. Determine which file is being read at that position (check file sizes of `.hi`, `.a`, `.conf` etc.) +4. Check `ar tv suspect.a | tail -5` to see if the last AR entry header is valid +5. Only then propose a fix — flag-based fixes almost never work for archive truncation issues + +--- + +## Part 7 — IHP-specific Haskell patterns + +### IHP record access + +IHP generates `HasField` instances. Access fields with: +```haskell +record.fieldName -- accessor (uses OverloadedRecordDot or get) +record |> set #fieldName value -- functional update +``` + +### IHP type family: `Include` + +```haskell +-- Generated/WidgetVersionInclude.hs (typical): +type instance Include "widgetId" (WidgetVersion' widgetId) = + WidgetVersion' (GetModelById widgetId) +``` + +This is pure type-level. `Include` is a closed type family that fills in the concrete type for an association when you do eager loading with `fetchRelated`. It is NOT Template Haskell. + +### IHP controller pattern + +```haskell +-- Each action is a function returning an IO Response (via implicit context) +action ShowWidgetAction { widgetId } = do + widget <- fetch widgetId + render ShowView { .. } +``` + +Controller type errors almost always mean a mismatch in `Web/Types.hs` — the action type declared there must match the implementation. + +### Schema → Generated code cycle + +``` +Edit Application/Schema.sql + → IHP IDE at localhost:8001 generates build/Generated/ + → Regenerated: Types.hs, ActualTypes.hs, Foo.hs, FooInclude.hs + → Also regenerates Web/Types.hs (action types) and Web/Routes.hs + → Run compile-check to verify +``` + +After schema changes, always verify the postUnpack overlay in `flake.nix` still produces valid Haskell. The `_types` / `_exports` awk scripts in postUnpack depend on the generated file structure remaining stable. + +--- + +## Quick reference card + +``` +# Build status checks (on haskelseed) +nix build .#docker --dry-run # what would be built? +nix log /nix/store/HASH-inter-hub-models-0.1.0.drv # full build log +wc -c /nix/store/ffg3yf2.../libHSghc-9.10.3-5702.a # verify archive size (must be ~289 MB) + +# If archive is truncated (287 MB), apply patch: +# Full archive is at same derivation's ghc-9.10.3-partial/ subdirectory +# See project_ghc_build_fix.md in Claude memory for exact commands + +# Compilation layers (in devenv shell) +scripts/compile-check-core # Layer 1+2 only +scripts/compile-check # Full build, errors → /tmp/ihub-compile-errors.txt + +# Schema regeneration +# Use IHP IDE at localhost:8001, then run compile-check + +# Production build → push → deploy +nix build .#docker +skopeo copy docker-archive:result docker://92.205.130.254:32166/coulomb/inter-hub:$(git rev-parse --short HEAD) +```