# ProGramerly — the AI and ML structure

Ionity (Pty) Ltd | AEDI — Policy 986 AED
Author: Johan Wilhelm van Antwerp · <https://www.ionity.today>
Applies to ProGramerly 3.4.0 and later.

Everything in this document runs on the machine ProGramerly is installed on.
No prompt, no data set, no page and no reading is sent anywhere. Where a
capability is missing, the application says which catalogue item installs it
rather than reaching for a network service.

---

## 1. The shape of it

```
                      the operator
                           │
        ┌──────────────────┴──────────────────┐
        │            the shell                │
        │  ask box · orb · brief · workspaces │
        └───────┬───────────────────┬─────────┘
                │                   │
        services/dome.js      services/ocr.js
       (what can be known)   (what can be read)
                │                   │
        ┌───────┴───────┐    ┌──────┴───────┐
        │ the registry  │    │ the readers  │
        │ datasets.json │    │ engines +    │
        │ dome.json     │    │ vision models│
        └───────┬───────┘    └──────┬───────┘
                │                   │
        ┌───────┴───────────────────┴────────┐
        │           services/ai.js           │
        │  endpoints · models · chat · GPU   │
        └───────────────┬────────────────────┘
                        │
                  Ollama, locally
```

Three files are the contract:

| File | What it fixes |
| --- | --- |
| `src/main/data/dome.json` | The structure: 5 strata, 26 segments, what each segment reads and what to watch for |
| `src/main/data/datasets.json` | The boundary: the 34 data sets AEDi may read, their class, and the 14 standing questions |
| `src/main/catalog/catalog.json` | What can be installed: every model, engine and framework, by profile |

Nothing outside `datasets.json` is servable. `dome.dataset('etc.passwd')` is
refused because it is not in the registry — not because a filter caught it.

---

## 2. Models

### 2.1 Roles

Every curated model in `services/ai.js` declares a **role**, and the rest of
the application reads that role rather than guessing from the name:

| Role | Used by | Models |
| --- | --- | --- |
| `chat` | the ask box, the launch brief, every Ask control | llama3.2 (1b/3b), llama3.1:8b, mistral:7b, **gemma3:1b, gemma2:2b, gemma3:4b, gemma4:e2b, gemma4:e4b** |
| `code` | code questions in the AI workspace | qwen2.5-coder (7b/14b) |
| `reason` | longer analysis | deepseek-r1:8b, phi4:14b |
| `vision` | the Reading workspace, OCR | **moondream, granite3.2-vision**, llava:7b, llama3.2-vision:11b, gemma3 (4b and up) |
| `embed` | retrieval | nomic-embed-text |

`sizeBytes` sits beside the human size so a fit check is arithmetic, not
string parsing.

### 2.2 Gemma

Gemma is a first-class family in ProGramerly 3.4.0.

| Tag | Size | What it is for |
| --- | --- | --- |
| `gemma3:1b` | ~815 MB | Pocket size. Quick triage on any machine. **Text only** |
| `gemma2:2b` | ~1.6 GB | Small, steady instruction following |
| `gemma3:4b` | ~3.3 GB | The middle step; multimodal, so it can also read a page |
| `gemma4:e2b` | ~7.2 GB | The everyday Ionity model above the starter — the default to move up to |
| `gemma4:e4b` | ~9.6 GB | Long reasoning and document work, where the memory is there |

Two catalogue items install them:

- **`ollama-gemma`** — `gemma4:e2b` + `gemma3:1b` (in the `full` and `ai` profiles)
- **`ollama-gemma-large`** — `gemma4:e4b` + `gemma3:4b` (opt-in)

Multimodality is stated exactly: Gemma 3 reads images from 4b upwards, and
`gemma3:1b` is listed as text only. The application never offers a model as a
reader when it cannot read.

---

## 3. Reading — OCR and vision

`src/main/services/ocr.js` offers two kinds of reader and always says which
one answered.

### 3.1 OCR engines

| Engine | Installed by | Character |
| --- | --- | --- |
| **Tesseract** | `ocr-engine` (winget / brew / apt / dnf / pacman) | Deterministic. Returns the characters it found. Best on clean scans and screenshots |
| **RapidOCR** | `ocr-toolkit` (managed venv) | ONNX runtime, CPU only, no system dependency, good on mixed layouts |
| **EasyOCR** | `ocr-toolkit` (managed venv) | Heavier and slower, better on photographs and odd fonts |

### 3.2 Vision models

| Model | Size | Installed by |
| --- | --- | --- |
| **moondream** | ~1.7 GB | `ollama-ocr` — the tiny reader; a screenshot in seconds, on CPU |
| **granite3.2-vision** | ~2.4 GB | `ollama-ocr` — built for documents: tables, forms, invoices, scans |
| llava:7b | ~4.7 GB | `ollama-models` |
| llama3.2-vision:11b | ~7.9 GB | curated list |

A vision model can be asked a question about the page — *"what is the invoice
total and its date"* — instead of transcribing it. An OCR engine cannot; it
returns characters.

### 3.3 What happens when you read a page

1. The file is checked: it must be an image or a PDF, and at most 40 MB.
2. A PDF is rasterised first, at 200 dpi, up to 20 pages, with PyMuPDF in the
   managed environment. Without PyMuPDF the application says so and stops.
3. The chosen reader runs, page by page, streaming text back on `ocr:token`.
4. The result carries the engine, the page count, the elapsed time, the mean
   confidence where the reader reports one, and a note saying whether this was
   characters found or a model's reading.

Nothing is written unless you save it. Saved readings go to
`<userData>/readings`.

### 3.4 The honest part

- An engine that is not installed is reported with **the reason and the
  catalogue item that installs it**, never hidden.
- The reading note distinguishes *"these are the characters it found"* from
  *"this is a model's reading — check anything that matters"*.
- Illegible text is asked for as `[illegible]`, not guessed.

---

## 4. The data registry — what a model may see

Each of the 34 sets carries a **class**, and every figure the model quotes
carries the class of the set it came from:

| Class | Meaning |
| --- | --- |
| `measured` | Read from a sensor, the OS or a live scan on this machine, now |
| `catalogue` | Curated data shipped inside the application |
| `manifest` | A pinned record — sizes and SHA-256 digests |
| `state` | What this installation has recorded about itself |
| `computed` | Derived from other sets. Never a reading |

Sets marked `cost: scan` (the registry probe, the installed-tool probe, the
development-root walk, the port scan, the reclaim scan, the repository sweep,
the Python environment sweep and the reader probe) are skipped by the quick
pass and read in full behind it. A segment whose sets have not been scanned
reports **"not scanned yet"** rather than being scored from nothing.

### 4.1 The intelligence stratum

| Segment | Code | Reads | Says |
| --- | --- | --- | --- |
| Local models | MDL | `ollama.models`, `ai.endpoints` | What is held and what is loaded |
| **Reading and OCR** | **OCR** | `ocr.engines`, `ollama.models` | What can turn a picture of words into words |
| Acceleration | ACC | `gpu.devices` | What the models will actually run on |
| Python environments | PY | `python.envs` | Where the ML stack lives |
| Data sets | DAT | `dome.datasets` | The registry describing itself |
| Presets | PRE | `dome.presets` | The standing questions |

### 4.2 The presets

Eleven standing questions, each declaring the sets it reads: state of the
machine, thermal and airflow brief, which curve to run, disk pressure and
reclaim, toolchain gaps, repository standing, model fit, **what can this
machine read**, integrity and privilege, explain the last scan, and *what can
you see*.

---

## 5. The ML stack

Installed into one managed virtual environment — `<devRoot>/.venvs/ionity` —
so the whole set is removable in one folder and the system Python is never
touched.

| Item | Contents |
| --- | --- |
| `python-venv` | The managed environment itself |
| `ml-toolkit` | numpy, pandas, scipy, scikit-learn, matplotlib, seaborn, JupyterLab, ipykernel, OpenCV, Pillow, polars, pyarrow |
| `pytorch` | torch, torchvision, torchaudio |
| `transformers` | transformers, datasets, accelerate |
| `tensorflow` | TensorFlow + Keras |
| `agent-frameworks` | LangChain, LangGraph, LlamaIndex, CrewAI |
| **`ocr-toolkit`** | rapidocr-onnxruntime, easyocr, pytesseract, PyMuPDF, pdf2image, Pillow |

---

## 6. Startup

1. **The intro** — its own frameless window. The DOME assembles arc by arc
   from this machine's real facts (host, OS, cores, memory, catalogue size,
   strata, data sets, presets, tools).
2. **The DOME boot** — the startup of the official IONITY Ai-OS DOME build,
   carried into the shell window: the mark, the wordmark line, the fill and
   the step list. Each step is ticked by the milestone it names —
   *verifying the integrated tools, loading the analysis presets, reading
   sensors and volumes, waking AEDi - the local core, assembling the dome* — so the
   fill is the share of real work done and cannot run ahead of the machine. A
   20-second ceiling clears it whatever happens.
3. **The shell** — and, if the launch brief is on and a model is present, the
   machine reads itself once and reports on the deck.

---

## 7. Where each piece lives

| Path | What it is |
| --- | --- |
| `src/main/services/ai.js` | Endpoints, models, pulls, chat, benchmark, GPU, the curated list and the vision rule |
| `src/main/services/ocr.js` | The readers, PDF rasterising, reading, saving |
| `src/main/services/dome.js` | The readers behind the registry, the scorers, the cache, the brief builder |
| `src/main/data/dome.json` | 5 strata, 26 segments |
| `src/main/data/datasets.json` | 34 data sets, 6 score sources, 14 presets |
| `src/main/catalog/catalog.json` | 116 installable items in 19 groups |
| `src/renderer/apps.js` | The DOME surface, the fan surface, the Reading surface |
| `src/renderer/shell.js` | The shell, the boot, the brief, the orb, the options |

---

Governance: Policy 986 AED · Licence AED 900
© 2018–2026 Antwerp Designs | Ionity (Pty) Ltd — All rights reserved — TM
*Building Tomorrow, Today.*

## 3.5.0 — AEDi, and the sets behind Relations, Environments and System

**AEDi** is the name of the local core in the product; **Ollama** is the engine
that serves it, and the app names both wherever the core is mentioned (`AEDi ·
gemma3:4b`, "powered by Ollama on this machine"). The AEDi tick-box in Software
installs `ollama` plus one model item (`ollama-small`, `ollama-gemma` or
`ollama-models`) with the rest of the run.

Five sets joined the registry, all class `measured`, all cost `scan`:

| Set | Reader | Source |
| --- | --- | --- |
| `system.processes` | `system.processes()` | Win32_Process + Get-Process · `ps -eo` |
| `system.services` | `system.services()` | Win32_Service · `systemctl` · `launchctl` |
| `system.startup` | `system.startup()` | Run keys · Startup folders · Task Scheduler · autostart · systemd user units · LaunchAgents |
| `system.listeners` | `system.listeners()` | Get-NetTCPConnection / Get-NetUDPEndpoint · `ss` · `/proc/net` · `lsof` |
| `envs.all` | `envs.list()` | `pyvenv.cfg` · `conda env list` · `package.json` · compose files · `docker compose ls` |

Three presets joined with them: *what is running, seen and unseen* (system),
*what starts by itself* (system), *environments on this machine* (toolchain).

**Score sources.** Every DOME percentage now carries one of six words —
`measured` (a raw reading rescaled), `computed` (a stated formula over real
readings), `assessment` (a present / absent / both bucket), `state` (what this
install recorded), `manifest` (pinned digests), `catalogue` (shipped data) — and
a constant is never `measured`. The word is shown beside the percentage on the
dome and passed to the model in the brief.

**Relations asks** go through `graph:ask`: the node's own facts and relations
(`graph.describe`) are appended to a `dome.brief` built from the sets its kind
belongs to, so the model answers about *this* process or *this* model with the
live figures around it — and, as everywhere else, says so when it cannot.

**Environment recipes** go through `env:recipe`: `RECIPE_SYSTEM` asks the model
for one JSON object (`kind`, `name`, `python`, `packages`, `deps`, `devDeps`,
`services[]`, `start`, `git`, `why`); `parseRecipe` validates it; the operator
sees the filled form and presses Create. The model never creates anything
itself.

## 3.6.0 — AEDi Predict: the forecast, its classes, and what the model may say

**Predict** (`src/main/services/predict.js`) is not a model call. It is a small,
inspectable estimator that runs before an install and learns after it:

| Question | Method | Class of the answer |
| --- | --- | --- |
| Will item *i* install cleanly here? | Prior `p₀` from `predict-priors.json` (per item, else per group), multiplied down for a missing engine (×0.35, or ×0.85 when Chocolatey can stand in for winget), an unanswering host (×0.15; ×0.03 when every host is silent), disk past the 8 GB reserve (×0.2), no elevation on an `elevate` item (×0.6); then shrunk toward this machine's record: `p = (p₀·W + ok + ½·partial) / (W + n)`, `W = 3`. | `assumed` until `n > 0`, then `learned` |
| How long? | Prior minutes × the machine's speed factor (an EMA of actual/assumed over runs, α = 0.35, first sample taken whole); once an item has two timings here, its learned median instead. | `assumed` → `computed` → `learned` |
| How much disk? | Σ prior GB against the system disk's free space minus the reserve. | GB `assumed`, free `measured`, after `computed` |
| Where is the machine going? | One sample per five minutes (`samples.jsonl`); least-squares slope of free bytes over the last seven days per fixed disk once six samples span two hours; days-to-full = free / −slope. 24-hour memory and CPU averages. | `measured` now, `computed` trend, `learned` history |
| What belongs next? | The `affinity.pairs` table in the priors (stated relations), profile completion at ≥ 60 %, and the dependencies a queue will pull in. | `stated` — never a statistic |

The engine reads its facts itself — `has()` for each engine binary and a TCP
connect with a 2.5 s timeout to each package host — and caches them for a
minute. An engine that an earlier item in the same queue provides (`node-system`
before an npm item, `ollama` before a model pull) is counted as present.

**What the model may do.** `predict:explain` hands the pack to the preferred
local model through `ai.chat` with a system prompt that says: use only the
figures given, never invent a number, never round a class up, six sentences,
end with one action. The reply streams on `predict:token`. With no model the
handler returns the same sentence the rest of the app uses. The forecast is
complete without the model; the model only reads it back.

**Learning is local and disposable.** `<userData>/predict/history.json` holds
per-item counts and the last twelve durations, the last sixty runs with their
predicted and actual figures, and the speed factor. `predict:reset` deletes it
and the samples. Nothing is sent anywhere.

**Tests.** `scripts/test-predict.js` injects facts (all engines up / npm missing /
offline / 20 GB free) and asserts the direction of every multiplier, the
posterior after a failure, the speed factor after a slow run, the learned median
after two timings, a ~2 GB/day regression with days-to-full, the suggestion
rules and the explain prompt. `scripts/ui-check-360.js` drives the surface in
the running application.