From d8f8acf29b662eead03ae41cc249ea2737ad0af0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Paul=20L=C3=BCtzner?= Date: Tue, 1 Sep 2026 12:54:09 +0200 Subject: [PATCH] feat(ai): tunable keep-alive per model, and a warm-up button MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first "Help me write" after a quiet spell shows nothing for tens of seconds while the Ollama host reads twenty gigabytes back off disk. How long it waits before evicting a model is OLLAMA_KEEP_ALIVE — an environment variable on a machine plMail can talk to but cannot configure. The API takes the same setting per request and a per-request value wins, so this is the only lever available from inside the app. Two nullable columns on ai_settings, one per model, because the two are opposite trades: the writing model is eighteen gigabytes and asked for occasionally, the embedding model a few hundred megabytes and asked for on every search and every indexed message. Both ship at 5m, which is Ollama's own default, so an upgrade changes nothing about how a host behaves — the boxes just say out loud what was already happening. -1 is recommended for the search model in the help text and the handbook rather than shipped, because plMail cannot see how much memory that host has. Empty means send no keep_alive at all and let the host decide, which is a real selectable state and not an unfinished one. The wire representation is the part that is easy to get wrong: Ollama parses a JSON string as a Go duration, and a Go duration needs a unit, so "-1" and "3600" as strings are a 400 on every model call from a setting that reads as correct. KeepAlive::forBody() is the only thing that puts the value in a body — int for the numeric spellings, string for the durations, null to omit the field entirely, since an explicit null is read as zero and unloads immediately. One helper adds it to all four bodies so no call site can get that wrong. Validated with the same expression the body is built from, refused rather than coerced, and the message lives in the validators domain where Symfony actually looks for it. Warm up now: POST /admin/ai/warmup issues the no-prompt /api/generate, times the round trip and reports it. PHP's 30s execution limit is raised for it — a cold load is not 30 seconds — and the upstream timeout sits below that so it gives up first. A host that is unreachable and a model that is not on the host are distinct, translated answers. Also carries both columns through the configuration backup, and adds the AI feature to the README, which had never mentioned it. Fixes a search test that asserted how fast the machine was: it gave the last-resort body pass a one-millisecond budget and one row to scan, so on quick hardware the pass finished in time and the test failed. It now provokes a genuine statement_timeout cancellation with pg_sleep, which exercises the same guard without a clock in the assertion. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 46 ++++ README.md | 3 + .../controllers/admin/ai_warmup_controller.js | 170 +++++++++++++++ docs/de/features/admin.md | 33 ++- docs/features/admin.md | 29 +++ migrations/Version20260903150000.php | 84 ++++++++ src/Controller/Admin/AiSettingsController.php | 87 ++++++++ src/Domain/Ai/KeepAlive.php | 148 +++++++++++++ src/Domain/DTO/Ai/AiWarmUp.php | 54 +++++ src/Entity/Ai/AiSettings.php | 63 ++++++ src/Form/Admin/AiSettingsType.php | 63 ++++++ src/Service/Ai/AiAssistant.php | 76 ++++++- src/Service/Ai/OllamaClient.php | 199 +++++++++++++++++- src/Service/Backup/ConfigBackupDatabase.php | 12 ++ templates/admin/ai/_frame.html.twig | 85 +++++++- tests/Form/Admin/AiKeepAliveTest.php | 73 +++++++ tests/Repository/Mail/MessageSearchTest.php | 54 ++++- tests/Service/Ai/OllamaKeepAliveTest.php | 137 ++++++++++++ translations/messages.de.yaml | 17 ++ translations/messages.en.yaml | 17 ++ translations/messages.en_PI.yaml | 17 ++ translations/validators.de.yaml | 1 + translations/validators.en.yaml | 1 + translations/validators.en_PI.yaml | 1 + 24 files changed, 1443 insertions(+), 27 deletions(-) create mode 100644 assets/controllers/admin/ai_warmup_controller.js create mode 100644 migrations/Version20260903150000.php create mode 100644 src/Domain/Ai/KeepAlive.php create mode 100644 src/Domain/DTO/Ai/AiWarmUp.php create mode 100644 tests/Form/Admin/AiKeepAliveTest.php create mode 100644 tests/Service/Ai/OllamaKeepAliveTest.php diff --git a/CHANGELOG.md b/CHANGELOG.md index f5e95154..b3609511 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,52 @@ so anything that changes the schema irreversibly is called out explicitly. The published image tags: `latest` follows the most recent release below, `main` follows the tip of the default branch, and `sha-…` pins one commit. +## v0.2.15 — 2026-09-01 + +**Adds two columns to the AI settings table; the migrations run on boot. Nothing is removed and +nothing is rewritten, and the values it fills in are the ones your model host was already using — so +this release changes how the AI feels, not what it does.** + +### Added + +- **How long a model stays loaded is yours to set.** The first "Help me write" after a quiet spell + used to take a small age and show nothing at all while it did. That is not the model thinking: it + is your Ollama host reading twenty gigabytes back off disk, because it had dropped the model out of + memory a few minutes after the last request. How long it waits before doing that is + `OLLAMA_KEEP_ALIVE`, an environment variable on the host — a machine plMail can talk to but cannot + configure, so until now the only way to change it was to go and edit the host and restart it. + + Admin → AI now has a box under each model. Write a duration — `5m`, `30m`, `4h` — or a plain number + of seconds; `-1` keeps the model loaded for good and `0` unloads it after every request. Longer + means a faster first answer, and the host keeps that memory reserved for the whole time either way. + Every request plMail makes carries the setting, so it covers drafting, summaries, sorting mail into + tabs and the search index alike. + + **Both boxes start at `5m`, which is exactly what Ollama does on its own** — so nothing about your + host changes until you decide it should. The boxes simply say out loud what was already happening. + + **`-1` is worth considering for the search model.** It is small, and it is asked for constantly — + every search by meaning, and every message being indexed — so pinning it usually costs a rounding + error of memory and saves a wait every single time. It is not the shipped setting only because + plMail cannot see how much memory your host has, and that is not a decision to make on your behalf. + + **Leaving a box empty is a real answer too**, and it means "send nothing, and let the host decide". + If you have set `OLLAMA_KEEP_ALIVE` deliberately, that is how you keep it. + +- **Warm up now.** A button beside the writing model that loads it straight away and tells you how + long it took. "Model loaded in 41 s" means it came off disk; "loaded in 230 ms — it was already in + memory" means the click bought nothing. Useful after a restart, and the only way to find out what a + cold load actually costs on your hardware before deciding how long to hold it. A host that cannot + be reached and a model that is not on the host are told apart, because they send you to two + different places. + +### Fixed + +- **A search test asserted how fast your machine was.** The guard that stops an expensive + last-resort search pass from running away is real and unchanged; the test proving it fired had a + one-millisecond budget and one row to scan, so on quick hardware the pass simply finished in time + and the test failed. It now provokes a genuine cancellation instead of hoping for one. + ## v0.2.14 — 2026-08-31 **Adds one table; the migrations run on boot. Nothing is removed and nothing is rewritten. Your diff --git a/README.md b/README.md index e8b9bbba..0f3d29ad 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,9 @@ provider doesn't. week grid, invitations answered in place, share and booking links. - **Files where you keep them** — attach from and save to Drive, Photos, OneDrive, Dropbox, Nextcloud and Immich. +- **A language model, if you want one** — point plMail at an Ollama box on your own network and it + can search by meaning, draft replies, summarise a thread and sort mail into tabs. Off by default, + each of the four switched on separately, and nothing ever leaves your network. - **Yours** — two-factor auth, per-app JMAP passwords, 41 themes, English, German and Pirate. ## A look around diff --git a/assets/controllers/admin/ai_warmup_controller.js b/assets/controllers/admin/ai_warmup_controller.js new file mode 100644 index 00000000..62d3ce58 --- /dev/null +++ b/assets/controllers/admin/ai_warmup_controller.js @@ -0,0 +1,170 @@ +import { Controller } from "@hotwired/stimulus"; + +/** + * "Warm up now" — asks the host to load the writing model, and says what happened. + * + * Values: + * url — POST endpoint, /admin/ai/warmup + * csrf — token minted for `admin-ai-warmup`, NOT the shared `ajax` one + * idleLabel — the button's resting text, restored after a result has been read + * busyLabel — while the host is loading; see below, this is not a spinner + * failedLabel — the fallback when the answer carries no message of its own + * + * Targets: + * button — the control itself, disabled while a request is in flight + * label — the span inside it whose text changes + * icon — the resting icon's wrapper; hidden while busy + * spinner — the spinner's wrapper; shown while busy + * result — where the sentence from the server is written + * + * WHY THE STATES ARE WORDS AND NOT A SPINNER ALONE + * ──────────────────────────────────────────────── + * This request can legitimately take forty seconds and produce nothing at all + * while it does — the host is reading eighteen gigabytes off disk and says + * nothing until it has finished. A spinner during that is indistinguishable + * from a dead button, which is the exact report compose/ai_assist_controller + * was built to answer, and the answer there was the same one: name the state. + * The spinner is here as well, but it is the decoration and the words are the + * information. + * + * WHY THE SENTENCE COMES FROM THE SERVER + * ────────────────────────────────────── + * Every outcome is a sentence in three languages, and one of them carries the + * host's own refusal in it. Assembling those here would put the copy in a file + * no translator opens — the rule admin/ai_status_controller states for the same + * page. This file decides WHEN to ask and what the button looks like while it + * is asking, and nothing about what it says. + * + * WHAT IT DOES NOT DO + * ─────────────────── + * It never talks to the model host: different origin, an address usually only + * the server can route to, and `connect-src 'self'` refuses it in production + * regardless. + */ +export default class extends Controller { + static targets = ["button", "label", "icon", "spinner", "result"]; + + static values = { + url: String, + /** + * Taken from the template rather than from the `csrf-token` meta tag, + * because it is a different token and not a different way of reading + * the same one — the distinction assets/csrf.js draws for + * settings/account_order_controller. This endpoint makes another + * machine reserve eighteen gigabytes; a token good for it should not + * also be good for everything else on the page. + */ + csrf: String, + idleLabel: { type: String, default: "Warm up now" }, + busyLabel: { type: String, default: "Loading the model…" }, + failedLabel: { type: String, default: "No answer" }, + }; + + /** + * Whether a request is in flight. + * + * The button is disabled while it is, so this is belt and braces — but the + * braces earn their place: a second preload of a model already loading does + * not queue behind the first, it makes the host do the work twice, and the + * host is the machine this button exists to be considerate of. + */ + #busy = false; + + /** + * Abandoned if the frame is swapped out from under it — this card lives + * inside a Turbo Frame that re-renders on every save, and a fetch that + * outlives its controller would write into elements no longer on the page. + */ + #abort = null; + + disconnect() { + if (null !== this.#abort) { + this.#abort.abort(); + this.#abort = null; + } + } + + async warmUp() { + if (true === this.#busy) { + return; + } + + this.#busy = true; + this.#abort = new AbortController(); + this.#setBusy(true); + + try { + const response = await fetch(this.urlValue, { + method: "POST", + headers: { "Content-Type": "application/json", "X-CSRF-Token": this.csrfValue }, + signal: this.#abort.signal, + }); + + if (false === response.ok) { + throw new Error(`Request failed (${response.status}).`); + } + + const payload = await response.json(); + + this.#show(payload?.message ?? this.failedLabelValue, true === payload?.ok); + } catch (error) { + // An aborted fetch is the frame being replaced, not a failure — + // there is nothing left on screen to report it to. + if ("AbortError" === error?.name) { + return; + } + + // Everything the server can answer already arrives as a translated + // sentence, so reaching here means the endpoint itself did not + // answer: a session that expired, or a container restarting. One + // label covers both, because the fix for both is to try again. + this.#show(this.failedLabelValue, false); + } finally { + this.#busy = false; + this.#abort = null; + this.#setBusy(false); + } + } + + // ── Private ─────────────────────────────────────────────────────────── + + #setBusy(busy) { + if (true === this.hasButtonTarget) { + this.buttonTarget.disabled = busy; + } + + if (true === this.hasLabelTarget) { + this.labelTarget.textContent = busy ? this.busyLabelValue : this.idleLabelValue; + } + + // The `hidden` attribute on the WRAPPERS, not on the icons — Font + // Awesome's own display rule beats it on an . See the template. + if (true === this.hasIconTarget) { + this.iconTarget.hidden = busy; + } + + if (true === this.hasSpinnerTarget) { + this.spinnerTarget.hidden = false === busy; + } + + if (true === busy && true === this.hasResultTarget) { + // Cleared on the way IN rather than on the way out, so the previous + // answer stays readable until a new one is actually being fetched. + this.resultTarget.textContent = ""; + this.resultTarget.hidden = true; + } + } + + #show(message, ok) { + if (false === this.hasResultTarget) { + return; + } + + // textContent, never innerHTML. One of these messages carries the + // host's own error string, which is text off another machine. + this.resultTarget.textContent = message; + this.resultTarget.hidden = false; + this.resultTarget.classList.toggle("text-emerald-500", ok); + this.resultTarget.classList.toggle("text-danger", false === ok); + } +} diff --git a/docs/de/features/admin.md b/docs/de/features/admin.md index 270bbf98..2c0778f4 100644 --- a/docs/de/features/admin.md +++ b/docs/de/features/admin.md @@ -1,4 +1,4 @@ - + # Administration @@ -336,6 +336,37 @@ Es sagt dir den Unterschied zwischen „unter der Adresse hat nichts geantwortet geantwortet, hat aber nicht das Modell, das du eingetragen hast“ — und das führt an zwei ganz verschiedene Stellen. +### Wie lange die Modelle geladen bleiben + +Ollama wirft ein Modell ein paar Minuten nach der letzten Anfrage aus dem Speicher, und es wieder +von der Platte zu lesen ist die ganze Wartezeit beim ersten „Hilf mir schreiben“ nach einer Pause — +ein 20-GB-Modell braucht dafür Dutzende Sekunden, in denen der Host überhaupt nichts liefert. Mit +**Schreib-Modell geladen lassen für** und **Such-Modell geladen lassen für** legst du fest, wie lange +plMail den Host bittet, das jeweilige Modell zu halten. + +Trag eine Dauer ein — `5m`, `30m`, `4h` — oder einfach eine Zahl in Sekunden. `-1` lässt das Modell +dauerhaft geladen, `0` entlädt es nach jeder Anfrage. Je länger, desto schneller kommt die erste +Antwort; der Host hält den Speicher dafür die ganze Zeit belegt, ob ihn jemand braucht oder nicht. + +Beide starten bei `5m` — das ist Ollamas eigene Voreinstellung. plMail ändert also von sich aus +nichts daran, wie dein Host sich verhält; die Felder sagen nur, was ohnehin schon passiert. + +**Für das Such-Modell lohnt sich `-1`**, und es ist nur deshalb nicht voreingestellt, weil plMail +nicht sehen kann, wie viel Speicher dein Host hat. Dieses Modell ist klein und wird ständig +gebraucht — bei jeder Suche nach Bedeutung und bei jeder Nachricht, die indiziert wird. Es +festzuhalten kostet meist kaum Speicher und spart jedes Mal eine Wartezeit. Wenn der Host den +Speicher übrig hat, stell es um. + +Lass ein Feld **leer**, dann schickt plMail gar nichts mit, und es gilt, was am Host selbst in +`OLLAMA_KEEP_ALIVE` steht. plMail kann diese Variable nicht lesen — ein leeres Feld ist also die +einzige Möglichkeit zu sagen „das überlasse ich dem Host“. + +**Jetzt vorladen** lädt das Schreib-Modell sofort und sagt dir, wie lange das gedauert hat. „Modell +in 41 s geladen“ heißt: es kam von der Platte. „Modell in 30 ms geladen“ heißt: es war schon im +Speicher und der Klick hat nichts gebracht. Praktisch nach einem Neustart — und um herauszufinden, +was ein kalter Ladevorgang auf deiner Hardware wirklich kostet, bevor du entscheidest, wie lange du +das Modell hältst. + ### Die vier Funktionen sind einzeln schaltbar Sie kosten sehr Unterschiedliches, und es hat sehr unterschiedliche Folgen, wenn das Modell danebenliegt. diff --git a/docs/features/admin.md b/docs/features/admin.md index 2a2ae029..a769f7ad 100644 --- a/docs/features/admin.md +++ b/docs/features/admin.md @@ -298,6 +298,35 @@ on the host first — plMail never downloads a model. tell you the difference between "nothing answered at that address" and "that host answered but is not holding the model you named", which send you to two different places. +### How long each model stays loaded + +Ollama drops a model out of memory a few minutes after the last request, and reading it back off +disk is the whole of the wait on the first "Help me write" after a quiet spell — a 20 GB model takes +tens of seconds to load, during which the host produces nothing at all. The **keep the writing model +loaded for** and **keep the search model loaded for** fields are how long plMail asks the host to +hold each one. + +Write a duration — `5m`, `30m`, `4h` — or a plain number of seconds. `-1` keeps the model loaded +indefinitely, `0` unloads it as soon as each request finishes. Longer means a faster first answer, +and the host keeps that memory reserved for the whole time whether anyone uses it or not. + +Both ship at `5m`, which is Ollama's own default — so out of the box plMail changes nothing about +how your host behaves, and the boxes simply say what is already happening. + +**`-1` is worth considering for the search model**, and it does not ship that way only because plMail +cannot see how much memory your host has. That model is small and asked for constantly — every +search by meaning, and every message being indexed — so pinning it usually costs a rounding error of +memory and saves a wait every single time. If the host has the memory to spare, set it. + +Leave either box **empty** to send nothing at all, in which case whatever `OLLAMA_KEEP_ALIVE` says +on the host itself applies. plMail cannot read that variable, so an empty box is the only way to say +"leave this to the host". + +**Warm up now** loads the writing model straight away and tells you how long it took. "Loaded in +41 s" means it came off disk; "loaded in 30 ms" means it was already resident and the click bought +nothing. Useful after a restart, and useful for finding out what a cold load actually costs on your +hardware before you decide how long to hold it. + ### The four features are separate switches They have very different costs, and very different consequences when the model is wrong. diff --git a/migrations/Version20260903150000.php b/migrations/Version20260903150000.php new file mode 100644 index 00000000..caef6d36 --- /dev/null +++ b/migrations/Version20260903150000.php @@ -0,0 +1,84 @@ +addSql('ALTER TABLE ai_settings ADD chat_keep_alive VARCHAR(16) DEFAULT \'5m\''); + $this->addSql('ALTER TABLE ai_settings ADD embedding_keep_alive VARCHAR(16) DEFAULT \'5m\''); + } + + public function down(Schema $schema): void + { + // Nothing to preserve. Dropping these puts every call back to sending + // no keep_alive field, which is what the code did before them and what + // the host's own default is for. + $this->addSql('ALTER TABLE ai_settings DROP chat_keep_alive'); + $this->addSql('ALTER TABLE ai_settings DROP embedding_keep_alive'); + } +} diff --git a/src/Controller/Admin/AiSettingsController.php b/src/Controller/Admin/AiSettingsController.php index 3034c491..63f3541f 100644 --- a/src/Controller/Admin/AiSettingsController.php +++ b/src/Controller/Admin/AiSettingsController.php @@ -21,6 +21,7 @@ use Symfony\Component\HttpFoundation\Request; use Symfony\Component\HttpFoundation\Response; use Symfony\Component\Routing\Attribute\Route; use Symfony\Component\Security\Http\Attribute\IsGranted; +use Symfony\Contracts\Translation\TranslatorInterface; /** * Admin → AI: which model host this installation talks to, and what it may do. @@ -47,6 +48,12 @@ final class AiSettingsController extends AbstractController { use ChecksCsrf; + /** + * Above OllamaClient::PRELOAD_TIMEOUT, so the upstream gives up first and + * this only ever catches a request that has escaped every other bound. + */ + private const int WARMUP_TIME_LIMIT_SECONDS = 300; + public function __construct( private readonly AiSettingsRepository $settings, private readonly AiAssistant $assistant, @@ -54,6 +61,7 @@ final class AiSettingsController extends AbstractController private readonly AiPerformancePanel $panel, private readonly EmbeddingBackfill $backfill, private readonly PromptLibrary $prompts, + private readonly TranslatorInterface $translator, ) { } @@ -81,6 +89,14 @@ final class AiSettingsController extends AbstractController $settings->apiToken = $token; } + // The two keep-alive fields are NOT normalised here, and that is + // deliberate rather than an omission. An emptied TextType bound to + // a nullable property yields null already, and KeepAlive::PATTERN + // has no room in it for surrounding whitespace — so by the time a + // form is valid the value is null or exactly one of the accepted + // spellings. A second pass would imply the first cannot be trusted. + // The writer that DOES need it is the config restore, which reads a + // file rather than a validated form; see ConfigBackupDatabase. $this->entityManager->persist($settings); $this->entityManager->flush(); @@ -217,6 +233,77 @@ final class AiSettingsController extends AbstractController ]); } + /** + * Load the writing model now, and say how long it took. + * + * The one control on this page that asks the host to DO something rather + * than to report. Ollama's `/api/generate` with a model and no prompt loads + * the weights and returns, so the round trip is the cold-load cost — which + * is the number the keep-alive field above is really about, and the only + * way to see it without making somebody go and click "Help me write" and + * count. + * + * WHY PHP'S EXECUTION LIMIT IS RAISED HERE + * ──────────────────────────────────────── + * It is 30 s in this stack, and a cold load of an eighteen-gigabyte model is + * not. Without this the button 500s with MaxExecutionTimeError on precisely + * the slow load it exists to measure, which is the same failure + * ComposeAssistController documents at length — and the same fix, a + * POSITIVE value rather than 0, because 0 cancels the timer without taking + * down the one already armed and it fires anyway while blocked on the + * socket. Above OllamaClient::PRELOAD_TIMEOUT, so the upstream is what + * gives up first and this is only ever the backstop. + * + * A SENTENCE, NOT A KEY, and this is the one place on this page that + * answers that way. The panel next door ships rendered HTML because it has + * six states and a table; this has one line of text on a button, and + * shipping a fragment for it would be markup where a string will do. What + * it must not do is build the sentence in JavaScript — that is where copy + * goes to be missed by translators — so the translation happens here and + * the controller sends the finished line, the way ComposeAttachmentController + * answers its uploads. + */ + #[Route('/warmup', name: 'warmup', methods: ['POST'])] + public function warmUp(Request $request): JsonResponse + { + // Its own token id rather than the shared `ajax` one, the rule + // ChecksCsrf states: this makes another machine reserve eighteen + // gigabytes of memory, which is not something a single stolen token + // should be good for alongside everything else. + $this->assertCsrf($request, 'admin-ai-warmup'); + + set_time_limit(self::WARMUP_TIME_LIMIT_SECONDS); + + $result = $this->assistant->warmUp(); + + if (true === $result->loaded) { + return $this->json([ + 'ok' => true, + 'millis' => $result->milliseconds, + 'message' => $this->translator->trans( + // Milliseconds below a second, whole seconds above it. Not + // a decimal in either case: the distinction that matters is + // "already resident" against "read off disk", and those are + // 40 ms and 40 s. A shared format would print 0.0 s for the + // first, which is the one reading that tells you nothing. + $result->milliseconds < 1000 ? 'admin.ai.warmup.loaded_ms' : 'admin.ai.warmup.loaded_seconds', + ['%count%' => $result->milliseconds < 1000 + ? $result->milliseconds + : (int) round($result->milliseconds / 1000)], + ), + ]); + } + + return $this->json([ + 'ok' => false, + 'millis' => $result->milliseconds, + 'message' => $this->translator->trans( + 'admin.ai.warmup.' . (string) $result->reason, + $result->reasonParams, + ), + ]); + } + /** * What the seven prompts say, when an administrator does not want what we * ship. diff --git a/src/Domain/Ai/KeepAlive.php b/src/Domain/Ai/KeepAlive.php new file mode 100644 index 00000000..d4ab560c --- /dev/null +++ b/src/Domain/Ai/KeepAlive.php @@ -0,0 +1,148 @@ + $reasonParams + */ + public function __construct( + public bool $loaded, + public int $milliseconds, + public ?string $reason = null, + public array $reasonParams = [], + ) { + } + + public static function loaded(int $milliseconds): self + { + return new self(true, $milliseconds); + } + + /** + * @param array $params + */ + public static function failed(string $reason, array $params = [], int $milliseconds = 0): self + { + return new self(false, $milliseconds, $reason, $params); + } +} diff --git a/src/Entity/Ai/AiSettings.php b/src/Entity/Ai/AiSettings.php index 14c75995..31b89b45 100644 --- a/src/Entity/Ai/AiSettings.php +++ b/src/Entity/Ai/AiSettings.php @@ -4,6 +4,7 @@ declare(strict_types=1); namespace App\Entity\Ai; +use App\Domain\Ai\KeepAlive; use App\Domain\Trait\TimestampableTrait; use App\Entity\Embeddable\AiPrompts; use App\Infrastructure\Doctrine\Type\EncryptedStringType; @@ -94,10 +95,47 @@ class AiSettings #[ORM\Column(name: 'chat_model', length: 128, nullable: true)] public ?string $chatModel = null; + /** + * How long the host is asked to hold the writing model after a request. + * + * The setting people actually feel. This model is around eighteen + * gigabytes, so a request that arrives after it has been evicted spends the + * first thirteen seconds or more reading it off disk and produces nothing + * at all while it does — which is the "the AI button does nothing" report, + * every time. Longer here buys a faster first answer; the price is that the + * host keeps the memory reserved for the whole window whether anybody + * writes anything or not, and on a single-GPU box that memory is the + * scarce thing. + * + * NULL IS A REAL SETTING AND NOT AN UNFINISHED ONE. It means send no + * keep_alive field, so whatever the operator put in OLLAMA_KEEP_ALIVE on + * the host stands — a choice somebody may have made deliberately and which + * plMail has no way to read and no business overriding by accident. See + * {@see KeepAlive}, which owns the format and the wire representation. + */ + #[ORM\Column(name: 'chat_keep_alive', length: 16, nullable: true, options: ['default' => KeepAlive::DEFAULT_CHAT])] + public ?string $chatKeepAlive = KeepAlive::DEFAULT_CHAT; + /** The model that turns text into vectors. Usually a much smaller one. */ #[ORM\Column(name: 'embedding_model', length: 128, nullable: true)] public ?string $embeddingModel = null; + /** + * The same question for the search model, and a separate answer to it. + * + * Its own column rather than one setting for both, because the two models + * are nothing alike even when the value is. This one is a few hundred + * megabytes and it is asked for constantly — every search by meaning, and + * every message the backfill walks — so paying a cold load for it is close + * to the worst bargain on offer, and `-1` is usually right here where it + * would rarely be right above. It is not what SHIPS, for the reason + * {@see KeepAlive::DEFAULT_EMBEDDING} gives: plMail cannot see how much + * memory that host has, and `-1` is the one value that never gives any + * back. + */ + #[ORM\Column(name: 'embedding_keep_alive', length: 16, nullable: true, options: ['default' => KeepAlive::DEFAULT_EMBEDDING])] + public ?string $embeddingKeepAlive = KeepAlive::DEFAULT_EMBEDDING; + /** * How many components the embedding model produces. * @@ -215,4 +253,29 @@ class AiSettings AiFeature::Summary => $this->summaryEnabled, }; } + + /** + * How long to ask the host to hold the model this feature uses. + * + * The same split enabledFor() makes and for the same reason: a feature does + * not get "the" model, it gets the one chosen for its job, and the setting + * that governs how long it stays resident has to follow the model rather + * than the feature. Three of the four features share the writing model, so + * three of the four arms share its answer — asking per feature is what + * stops a caller having to know that. + * + * The raw stored spelling, not the wire value. {@see KeepAlive::forBody()} + * is the one place that turns it into something a request body can carry, + * because the difference between the number -1 and the string "-1" matters + * to the host and to nothing else. + */ + public function keepAliveFor(AiFeature $feature): ?string + { + return match ($feature) { + AiFeature::Search => $this->embeddingKeepAlive, + AiFeature::Categorise, + AiFeature::WritingHelp, + AiFeature::Summary => $this->chatKeepAlive, + }; + } } diff --git a/src/Form/Admin/AiSettingsType.php b/src/Form/Admin/AiSettingsType.php index 143a7f80..271980e1 100644 --- a/src/Form/Admin/AiSettingsType.php +++ b/src/Form/Admin/AiSettingsType.php @@ -4,6 +4,7 @@ declare(strict_types=1); namespace App\Form\Admin; +use App\Domain\Ai\KeepAlive; use App\Entity\Ai\AiSettings; use App\Form\PasswordManagerIgnore; use Symfony\Component\Form\AbstractType; @@ -76,12 +77,27 @@ final class AiSettingsType extends AbstractType 'required' => false, 'attr' => ['placeholder' => 'llama3.1:8b', 'autocomplete' => 'off'], ]) + // Directly under the model it governs rather than grouped with the + // other keep-alive below, because the pair is one decision: which + // model, and for how long it stays loaded. Two adjacent duration + // fields under two adjacent model fields would be four boxes in + // which it is genuinely hard to see which belongs to which. + ->add('chatKeepAlive', TextType::class, self::keepAlive( + 'admin.ai.field.chat_keep_alive', + 'admin.ai.field.chat_keep_alive_help', + KeepAlive::DEFAULT_CHAT, + )) ->add('embeddingModel', TextType::class, [ 'label' => 'admin.ai.field.embedding_model', 'help' => 'admin.ai.field.embedding_model_help', 'required' => false, 'attr' => ['placeholder' => 'nomic-embed-text', 'autocomplete' => 'off'], ]) + ->add('embeddingKeepAlive', TextType::class, self::keepAlive( + 'admin.ai.field.embedding_keep_alive', + 'admin.ai.field.embedding_keep_alive_help', + KeepAlive::DEFAULT_EMBEDDING, + )) // The four below are separate because they have very different // costs and very different appetites for being wrong. See // AiSettings for the argument. @@ -107,6 +123,53 @@ final class AiSettingsType extends AbstractType ]); } + /** + * The two keep-alive fields, which differ only in what they say. + * + * WHY THE VALIDATION IS A REGEX AND NOT A COERCION + * ──────────────────────────────────────────────── + * A wrong value here fails at the far end, hours later, on somebody else's + * request: Ollama answers 400 for a duration it cannot parse, and the only + * symptom is that the writing help stopped working. Quietly rewriting "5 + * min" into "5m" would hide that, and quietly falling back to the default + * would hide it better — the field would show one thing and the host would + * be told another, which is the single worst outcome available for a + * setting whose entire job is to be believed. + * + * So it is refused at the moment somebody can still fix it, with the + * accepted spellings in the message. {@see KeepAlive::PATTERN} is the same + * expression the request body is built from, so the form and the wire + * cannot come to disagree about what is legal. + * + * The message lives in the `validators` domain, which is not the domain + * this form declares — Symfony translates constraint output there + * regardless, and a key put in `messages` renders on screen as the raw key. + * See CODESTYLE §8.2, and the `base_url_invalid` message directly above, + * which has that bug. + * + * @return array + */ + private static function keepAlive(string $label, string $help, string $shipped): array + { + return [ + 'label' => $label, + 'help' => $help, + // Empty is a real, selectable answer — "say nothing and let the + // host's own OLLAMA_KEEP_ALIVE stand" — so the field is optional + // and the placeholder shows what plMail ships rather than what is + // required. + 'required' => false, + 'attr' => ['placeholder' => $shipped, 'autocomplete' => 'off'], + 'constraints' => [ + new Regex( + pattern: KeepAlive::PATTERN, + message: 'admin.ai.keep_alive_invalid', + match: true, + ), + ], + ]; + } + public function configureOptions(OptionsResolver $resolver): void { $resolver->setDefaults([ diff --git a/src/Service/Ai/AiAssistant.php b/src/Service/Ai/AiAssistant.php index 6d0fe77f..3aa90c61 100644 --- a/src/Service/Ai/AiAssistant.php +++ b/src/Service/Ai/AiAssistant.php @@ -8,6 +8,7 @@ use App\Domain\DTO\Ai\AiCallTiming; use App\Domain\DTO\Ai\AiChatResult; use App\Domain\DTO\Ai\AiEmbedResult; use App\Domain\DTO\Ai\AiProbe; +use App\Domain\DTO\Ai\AiWarmUp; use App\Domain\Enum\Ai\AiCallFeature; use App\Entity\Ai\AiFeature; use App\Entity\Ai\AiSettings; @@ -152,7 +153,12 @@ final readonly class AiAssistant $model = (string) $settings->embeddingModel; - $result = $this->client->embed((string) $settings->baseUrl, $model, $text); + $result = $this->client->embed( + (string) $settings->baseUrl, + $model, + $text, + $settings->keepAliveFor(AiFeature::Search), + ); // Recorded here rather than in the client, because the client knows the // numbers and this knows what they were FOR. @@ -194,7 +200,13 @@ final readonly class AiAssistant $model = (string) $settings->chatModel; - $result = $this->client->chat((string) $settings->baseUrl, $model, $messages, $temperature); + $result = $this->client->chat( + (string) $settings->baseUrl, + $model, + $messages, + $temperature, + $settings->keepAliveFor($feature), + ); $this->recorder->record( AiCallFeature::forChat($feature), @@ -261,7 +273,13 @@ final readonly class AiAssistant return $this->recorded( AiCallFeature::forChat($feature), $model, - $this->client->chatStream((string) $settings->baseUrl, $model, $messages, $temperature), + $this->client->chatStream( + (string) $settings->baseUrl, + $model, + $messages, + $temperature, + $settings->keepAliveFor($feature), + ), ); } @@ -414,4 +432,56 @@ final readonly class AiAssistant return $this->client->probe($target); } + + /** + * Load the writing model now, so the next person to ask does not wait. + * + * WHY THE MASTER SWITCH IS NOT CONSULTED + * ────────────────────────────────────── + * probe()'s reasoning, and the same situation: an administrator setting + * this up has not turned anything on yet, and the cost of a cold load is + * precisely what they are trying to find out before they do. A button that + * answered "disabled" would refuse at the only moment the number is worth + * measuring. + * + * A HOST AND A MODEL ARE STILL REQUIRED, and they get separate answers + * because they are separate mistakes with separate fixes — one is a field + * above this button, the other is a different field above this button, and + * "nothing happened" would send somebody to check the network for neither + * of them. + * + * WHAT IT DELIBERATELY DOES NOT DO + * ──────────────────────────────── + * It reads the SAVED settings, where the Test button reads what is on + * screen. The two are different errands: Test answers "is this address any + * good" before you commit to it, while this one asks a host to reserve + * eighteen gigabytes on behalf of the configuration that is actually in + * force. Warming a model nothing is going to use — because the form has not + * been saved — would be the wrong machine doing real work for a setting + * that does not exist yet. + * + * It also warms only the writing model. The search model is the one that is + * cheap to pin and is pinned by default, and a second button for it would + * be a control whose correct use is "never". + */ + public function warmUp(): AiWarmUp + { + $settings = $this->settings(); + + if (false === $settings->isConfigured()) { + return AiWarmUp::failed('no_host'); + } + + $model = $settings->chatModel; + + if (null === $model || '' === trim($model)) { + return AiWarmUp::failed('no_model'); + } + + return $this->client->preload( + (string) $settings->baseUrl, + trim($model), + $settings->keepAliveFor(AiFeature::WritingHelp), + ); + } } diff --git a/src/Service/Ai/OllamaClient.php b/src/Service/Ai/OllamaClient.php index ca6c738e..76871c8a 100644 --- a/src/Service/Ai/OllamaClient.php +++ b/src/Service/Ai/OllamaClient.php @@ -4,10 +4,12 @@ declare(strict_types=1); namespace App\Service\Ai; +use App\Domain\Ai\KeepAlive; use App\Domain\DTO\Ai\AiCallTiming; use App\Domain\DTO\Ai\AiChatResult; use App\Domain\DTO\Ai\AiEmbedResult; use App\Domain\DTO\Ai\AiProbe; +use App\Domain\DTO\Ai\AiWarmUp; use App\Domain\DTO\Ai\LoadedModel; use App\Domain\DTO\Ai\OllamaModel; use DateTimeImmutable; @@ -91,6 +93,24 @@ final readonly class OllamaClient */ private const float PROBE_TIMEOUT = 5.0; + /** + * Longer than GENERATE_TIMEOUT, which looks wrong and is not. + * + * Every one of these timeouts is an IDLE timeout — the longest the host may + * go without saying anything — and on a streamed completion the first token + * resets it, so 120 s covers a cold load and then never applies again. + * {@see preload()} has no tokens. It is the one call that is ENTIRELY the + * cold load: the host says nothing whatsoever until eighteen gigabytes are + * in memory, and off a spinning disk that is comfortably past two minutes. + * Held to 120 s the button would report a timeout for a load that was + * proceeding perfectly well, which is the exact confusion it exists to end. + * + * Below the caller's own ceiling, so it is this that gives up first — see + * AiSettingsController::warmUp(), which raises PHP's execution limit above + * this and says why. + */ + private const float PRELOAD_TIMEOUT = 240.0; + public function __construct( private HttpClientInterface $http, private LoggerInterface $logger, @@ -263,22 +283,27 @@ final readonly class OllamaClient * as broken rather than as out of date. * * Both attempts collapse into ONE result, and therefore one recorded call. + * + * The keep-alive rides on both. The older dialect predates the field and + * ignores what it does not recognise, so sending it costs nothing there and + * omitting it would mean the setting silently stopped applying on exactly + * the hosts that pay the most for a reload. */ - public function embed(string $baseUrl, string $model, string $text): AiEmbedResult + public function embed(string $baseUrl, string $model, string $text, ?string $keepAlive = null): AiEmbedResult { - $modern = $this->tryEmbed($baseUrl, '/api/embed', [ + $modern = $this->tryEmbed($baseUrl, '/api/embed', self::withKeepAlive([ 'model' => $model, 'input' => $text, - ], static fn (array $body): ?array => $body['embeddings'][0] ?? null); + ], $keepAlive), static fn (array $body): ?array => $body['embeddings'][0] ?? null); if (true === $modern->succeeded) { return $modern; } - $legacy = $this->tryEmbed($baseUrl, '/api/embeddings', [ + $legacy = $this->tryEmbed($baseUrl, '/api/embeddings', self::withKeepAlive([ 'model' => $model, 'prompt' => $text, - ], static fn (array $body): ?array => $body['embedding'] ?? null); + ], $keepAlive), static fn (array $body): ?array => $body['embedding'] ?? null); if (true === $legacy->succeeded) { // The older dialect answered, so the embedding worked. Its timings @@ -313,13 +338,13 @@ final readonly class OllamaClient * * @param list $messages */ - public function chat(string $baseUrl, string $model, array $messages, ?float $temperature = null): AiChatResult + public function chat(string $baseUrl, string $model, array $messages, ?float $temperature = null, ?string $keepAlive = null): AiChatResult { - $payload = [ + $payload = self::withKeepAlive([ 'model' => $model, 'messages' => $messages, 'stream' => false, - ]; + ], $keepAlive); if (null !== $temperature) { $payload['options'] = ['temperature' => $temperature]; @@ -434,13 +459,13 @@ final readonly class OllamaClient * * @return \Generator */ - public function chatStream(string $baseUrl, string $model, array $messages, ?float $temperature = null): \Generator + public function chatStream(string $baseUrl, string $model, array $messages, ?float $temperature = null, ?string $keepAlive = null): \Generator { - $payload = [ + $payload = self::withKeepAlive([ 'model' => $model, 'messages' => $messages, 'stream' => true, - ]; + ], $keepAlive); if (null !== $temperature) { $payload['options'] = ['temperature' => $temperature]; @@ -531,6 +556,158 @@ final readonly class OllamaClient return AiChatResult::ok($whole, $timing); } + /** + * Load a model into the host's memory and generate nothing. + * + * A `/api/generate` with a model and NO prompt is Ollama's documented way + * to say "have this ready". The host reads the weights in and answers as + * soon as they are resident, so the round trip IS the cold-load cost — the + * number an operator actually wants, and one nothing else here can report: + * /api/ps says whether a model is loaded but never how long getting it + * there took. + * + * It is a WRITE in every sense that matters — it reserves gigabytes on + * somebody else's machine for as long as the keep-alive says — which is why + * it is only ever reached from an administrator pressing a button, and why + * nothing polls it. ps() is the passive question. + * + * NOT RECORDED IN THE METRICS TABLE, and that is deliberate rather than + * forgotten. AiCallRecorder accounts for WORKLOADS — five of them, each + * gated by a feature — and this is not one: it generates no tokens, it + * serves no user's request, and giving it a sixth AiCallFeature case would + * put a row in the throughput table with nothing to put in any of its + * columns. The timing goes back to the button that asked for it, which is + * the only reader there is. + * + * Answers a value object rather than throwing, like probe(): the caller is + * an administrator who wants to be told what happened, in the same three + * shapes — it worked, nothing answered, or something answered and refused. + */ + public function preload(string $baseUrl, string $model, ?string $keepAlive = null): AiWarmUp + { + $started = microtime(true); + + try { + $response = $this->http->request('POST', $this->url($baseUrl, '/api/generate'), [ + 'json' => self::withKeepAlive([ + 'model' => $model, + 'stream' => false, + ], $keepAlive), + 'timeout' => self::PRELOAD_TIMEOUT, + ]); + + $status = $response->getStatusCode(); + + if (200 !== $status) { + // 404 is its own answer because it is the one an operator can + // fix from where they are standing: the host is up, it is + // talking, and the model named in the settings above is simply + // not pulled on it. Told "the host is unreachable" they would + // go and check the network instead. + if (404 === $status) { + return AiWarmUp::failed('model_missing', [ + 'model' => $model, + 'error' => self::refusal($response->getContent(false)), + ], self::elapsed($started)); + } + + return AiWarmUp::failed('status', [ + 'status' => $status, + 'error' => self::refusal($response->getContent(false)), + ], self::elapsed($started)); + } + + // Read so the response is complete before the clock is stopped — + // the status line arrives with the headers, and on this endpoint + // that is the whole of the wait. Not read for its content: a + // preload answers `done: true` with an empty response, which is + // success and would fail any check for text. + $response->getContent(false); + + return AiWarmUp::loaded(self::elapsed($started)); + } catch (TimeoutExceptionInterface $exception) { + $this->logger->warning('OllamaClient: preload timed out', [ + 'model' => $model, + 'error' => $exception->getMessage(), + 'exception' => $exception, + ]); + + return AiWarmUp::failed('timeout', [], self::elapsed($started)); + } catch (HttpClientException $exception) { + $this->logger->warning('OllamaClient: preload could not reach the host', [ + 'model' => $model, + 'error' => $exception->getMessage(), + 'exception' => $exception, + ]); + + return AiWarmUp::failed('unreachable', ['error' => $exception->getMessage()], self::elapsed($started)); + } catch (Throwable $exception) { + $this->logger->error('OllamaClient: preload failed unexpectedly', [ + 'model' => $model, + 'error' => $exception->getMessage(), + 'exception' => $exception, + ]); + + return AiWarmUp::failed('unexpected', ['error' => $exception->getMessage()], self::elapsed($started)); + } + } + + /** + * The request body with `keep_alive` in it, or exactly as it was. + * + * One helper for all four bodies, because the rule is the same everywhere + * and the thing being avoided is a fifth call site that sends `null` — an + * explicit null in the JSON is NOT the same as omitting the field: Ollama + * decodes it to a zero duration and unloads the model the moment the + * request finishes, which is the exact opposite of "we did not want to + * say". + * + * @param array $payload + * + * @return array + */ + private static function withKeepAlive(array $payload, ?string $keepAlive): array + { + $value = KeepAlive::forBody($keepAlive); + + if (null === $value) { + return $payload; + } + + $payload['keep_alive'] = $value; + + return $payload; + } + + /** Whole milliseconds, which is the resolution anybody reads this at. */ + private static function elapsed(float $started): int + { + return (int) round((microtime(true) - $started) * 1000); + } + + /** + * What the host said when it refused, short enough to put on a page. + * + * Ollama answers `{"error": "model 'qwen3' not found"}`, and that sentence + * is the most useful thing on the screen — it names the model as the host + * understood it, which is how a trailing space or a wrong tag gets found. + * The raw body is the fallback for a reverse proxy answering its own HTML, + * clipped because that can be a whole error page. + * + * Safe to show here in a way it would not be from chat(): this request body + * carries a model name and nothing else. No mail has ever been near it. + */ + private static function refusal(string $body): string + { + $decoded = json_decode($body, true); + + if (true === is_array($decoded) && true === is_string($decoded['error'] ?? null)) { + return $decoded['error']; + } + + return mb_substr(trim($body), 0, 200); + } + /** * @param array $payload * @param callable(array): (list|null) $pluck diff --git a/src/Service/Backup/ConfigBackupDatabase.php b/src/Service/Backup/ConfigBackupDatabase.php index 7f601773..b3f53ebf 100644 --- a/src/Service/Backup/ConfigBackupDatabase.php +++ b/src/Service/Backup/ConfigBackupDatabase.php @@ -12,6 +12,7 @@ use App\Entity\Push\FcmConfig; use App\Repository\Integration\IntegrationProviderConfigRepository; use App\Repository\Integration\MailProviderConfigRepository; use App\Repository\Push\FcmConfigRepository; +use App\Domain\Ai\KeepAlive; use App\Entity\Ai\AiSettings; use App\Entity\Monitoring\LogSettings; use App\Repository\Ai\AiSettingsRepository; @@ -238,7 +239,9 @@ final readonly class ConfigBackupDatabase 'baseUrl' => $settings->baseUrl, 'apiToken' => $settings->apiToken, 'chatModel' => $settings->chatModel, + 'chatKeepAlive' => $settings->chatKeepAlive, 'embeddingModel' => $settings->embeddingModel, + 'embeddingKeepAlive' => $settings->embeddingKeepAlive, 'embeddingDimensions' => $settings->embeddingDimensions, 'searchEnabled' => $settings->searchEnabled, 'categorisationEnabled' => $settings->categorisationEnabled, @@ -282,6 +285,15 @@ final readonly class ConfigBackupDatabase $settings->apiToken = $this->string($values, 'apiToken'); $settings->chatModel = $this->string($values, 'chatModel'); $settings->embeddingModel = $this->string($values, 'embeddingModel'); + // Through KeepAlive::normalised() rather than string() alone, because + // this is the one writer that is not the form. A file can carry + // whitespace where the form's constraint would have refused it, and a + // column holding " " shows an administrator a value they cannot save + // again. Absent from an older backup means null, which is the + // host's-own-default state and the right answer for a file written + // before this setting existed. + $settings->chatKeepAlive = KeepAlive::normalised($this->string($values, 'chatKeepAlive')); + $settings->embeddingKeepAlive = KeepAlive::normalised($this->string($values, 'embeddingKeepAlive')); $settings->embeddingDimensions = is_int($values['embeddingDimensions'] ?? null) ? $values['embeddingDimensions'] : null; $settings->searchEnabled = true === ($values['searchEnabled'] ?? false); $settings->categorisationEnabled = true === ($values['categorisationEnabled'] ?? false); diff --git a/templates/admin/ai/_frame.html.twig b/templates/admin/ai/_frame.html.twig index a50522f5..bcd1a505 100644 --- a/templates/admin/ai/_frame.html.twig +++ b/templates/admin/ai/_frame.html.twig @@ -140,7 +140,90 @@ poll would empty a field mid-word. {{ form_start(form, { attr: { 'data-turbo-frame': 'admin-ai' } }) }}
- {{ form_widget(form) }} + {# Rendered row by row rather than with form_widget(form), + for one reason: the warm-up button has to sit with the + writing model and its keep-alive, and a single + form_widget emits all eleven rows in one go with nowhere + to put anything between them. The wrapper carries the + space-y-5 that modal_form_theme's form_widget_compound + adds for a root form, and form_rest() below still emits + the CSRF token — dropping it is how this kind of + hand-rendered form silently stops validating. #} +
+ {{ form_errors(form) }} + + {{ form_row(form.isEnabled) }} + {{ form_row(form.baseUrl) }} + {{ form_row(form.apiToken) }} + + {{ form_row(form.chatModel) }} + {{ form_row(form.chatKeepAlive) }} + + {# ── Warm up now ────────────────────────────────── + Here rather than beside Save, because it is about + the two fields above it and nothing else on the + page: it loads THAT model and holds it for THAT + long. Beside Save it would read as another way of + submitting the form, which it is not — it writes + nothing, and it acts on what is stored rather than + on what is on screen. + + type="button", so a stray Enter in a text field + above cannot make it the form's default submit. #} +
+ + + + {# aria-live, because the answer arrives up to a + minute after the click and a screen reader user + who has moved on would otherwise never learn + what happened. Polite: it is a result, not an + interruption. #} + +
+ + {{ form_row(form.embeddingModel) }} + {{ form_row(form.embeddingKeepAlive) }} + + {{ form_row(form.searchEnabled) }} + {{ form_row(form.categorisationEnabled) }} + {{ form_row(form.writingHelpEnabled) }} + {{ form_row(form.summaryEnabled) }} + + {{ form_rest(form) }} +
{# Same form, different action. `formaction` so the diff --git a/tests/Form/Admin/AiKeepAliveTest.php b/tests/Form/Admin/AiKeepAliveTest.php new file mode 100644 index 00000000..7481fd27 --- /dev/null +++ b/tests/Form/Admin/AiKeepAliveTest.php @@ -0,0 +1,73 @@ +forms = self::getContainer()->get('form.factory'); + } + + public function testTheAcceptedSpellingsAreAccepted(): void + { + foreach (['5m', '30m', '4h', '3600', '0', '-1', ''] as $value) { + $form = $this->submit($value); + + self::assertTrue($form->isValid(), $value . ' → ' . (string) $form->getErrors(true, false)); + } + } + + /** + * "5 min" is what somebody types who has not read the help text, and + * "5M" is what somebody types who has — Go's duration parser accepts + * neither, so neither may be accepted here. + */ + public function testAnythingElseIsRefusedRatherThanQuietlyReplaced(): void + { + foreach (['5 min', '5M', 'forever', '5m30s', '-2'] as $value) { + $form = $this->submit($value); + + self::assertFalse($form->isValid(), $value . ' was accepted'); + } + } + + private function submit(string $keepAlive): FormInterface + { + $form = $this->forms->create(AiSettingsType::class, new AiSettings(), ['csrf_protection' => false]); + + $form->submit([ + 'baseUrl' => 'http://10.0.0.5:11434', + 'chatKeepAlive' => $keepAlive, + ], false); + + return $form; + } +} diff --git a/tests/Repository/Mail/MessageSearchTest.php b/tests/Repository/Mail/MessageSearchTest.php index a978087f..dd65824f 100644 --- a/tests/Repository/Mail/MessageSearchTest.php +++ b/tests/Repository/Mail/MessageSearchTest.php @@ -906,28 +906,60 @@ final class MessageSearchTest extends KernelTestCase } /** - * The same search, with a budget no query can meet: nothing found, and - * nothing thrown. + * A body pass that blows its budget: the cheap results stand, and nothing + * is thrown. * - * Before this, the pass ran unbounded and a live mailbox answered with - * `MaxExecutionTimeError: Maximum execution time of 30 seconds exceeded`. - * Losing the infix is the deliberate price; a 500 was not a price anybody - * chose. + * Before this guard existed the pass ran unbounded and a live mailbox + * answered with `MaxExecutionTimeError: Maximum execution time of 30 + * seconds exceeded`. Losing the infix is the deliberate price; a 500 was + * not a price anybody chose. + * + * IT NO LONGER ASKS POSTGRES TO BE SLOW, AND THAT IS THE FIX + * ───────────────────────────────────────────────────────── + * This used to build a repository with `rescueTimeoutMs: 1`, run a real + * search and assert it came back empty — which asserts how fast the machine + * is, not what this code does. One seeded row is a scan Postgres finishes + * well inside a millisecond, so on a quick box the pass SUCCEEDS, finds the + * row the test called "the one row only the expired pass could find", and + * fails with "actual size 1 matches expected size 0". Green on slow + * hardware and red on fast is not a test of anything. + * + * `pg_sleep` removes the race instead of widening the assertion. One second + * against a one-millisecond budget cannot finish in time on any machine, so + * the cancel is certain — and it is a REAL cancel, arriving as SQLSTATE + * 57014 through the same path a genuine overrun takes. All three halves of + * the guard are exercised: the budget is set, the statement is killed, and + * the fallback rows are returned instead of the exception escaping. + * + * The rescue callable is handed in by reflection because the method is + * private and should stay private — the same access this file already takes + * for buildSearchSql(), and for the same reason: the seam is where the bug + * lives. Going through searchPage() instead is what forced the timing + * assumption in the first place. */ public function testAnExpiredBodyPassGivesUpInsteadOfFailing(): void { - $this->seedMessage('Quarterly figures', body: 'The pelican audit is attached.'); - $impatient = new MessageThreadRepository( self::getContainer()->get(ManagerRegistry::class), new FreeTextCompiler(), rescueTimeoutMs: 1, ); - $page = $impatient->searchPage($this->user, $this->parser->parse('elica')); + $connection = self::getContainer()->get(EntityManagerInterface::class)->getConnection(); - self::assertCount(0, $page->threads, 'the one row only the expired pass could find'); - self::assertSame(0, $page->total); + // What the cheap pass already found. Identity is the assertion below: + // the guard must hand back exactly this, not an empty page. + $fallback = [['thread_id' => 7]]; + + $rows = new \ReflectionMethod(MessageThreadRepository::class, 'rescueRows')->invoke( + $impatient, + // The same connection rescueRows() put the budget on — a second one + // would be a different session and would never see the timeout. + static fn (): array => $connection->fetchAllAssociative('SELECT pg_sleep(1)'), + $fallback, + ); + + self::assertSame($fallback, $rows, 'the cheap results have to stand when the rescue pass is killed'); } /** diff --git a/tests/Service/Ai/OllamaKeepAliveTest.php b/tests/Service/Ai/OllamaKeepAliveTest.php new file mode 100644 index 00000000..eb87cee3 --- /dev/null +++ b/tests/Service/Ai/OllamaKeepAliveTest.php @@ -0,0 +1,137 @@ +client(static function (string $method, string $url, array $options) use (&$body): MockResponse { + $body = json_decode((string) $options['body'], true); + + return new MockResponse(json_encode(['message' => ['content' => 'hello']])); + }); + + $client->chat('http://h:11434', 'm', [['role' => 'user', 'content' => 'hi']], null, ''); + + self::assertIsArray($body); + self::assertArrayNotHasKey('keep_alive', $body); + } + + /** + * And the embedding path carries its own, which is a different column. + */ + public function testTheEmbeddingCallCarriesTheSearchModelsSetting(): void + { + $body = null; + + $client = $this->client(static function (string $method, string $url, array $options) use (&$body): MockResponse { + $body = json_decode((string) $options['body'], true); + + return new MockResponse(json_encode(['embeddings' => [[0.1, 0.2]]])); + }); + + $client->embed('http://h:11434', 'nomic-embed-text', 'some mail', '-1'); + + self::assertIsArray($body); + self::assertSame(-1, $body['keep_alive']); + } + + /** + * The preload posts a model and NO prompt, which is what makes it a load + * rather than a generation. + */ + public function testWarmingUpAsksForAModelAndNoPrompt(): void + { + $body = null; + $seen = null; + + $client = $this->client(static function (string $method, string $url, array $options) use (&$body, &$seen): MockResponse { + $seen = $url; + $body = json_decode((string) $options['body'], true); + + return new MockResponse(json_encode(['done' => true])); + }); + + $result = $client->preload('http://h:11434', 'qwen3:30b', '5m'); + + self::assertTrue($result->loaded); + self::assertStringContainsString('/api/generate', (string) $seen); + self::assertIsArray($body); + self::assertArrayNotHasKey('prompt', $body); + self::assertSame('5m', $body['keep_alive']); + } + + /** + * "The host is not there" and "the host is there and has never heard of + * that model" send an administrator to two different fields, so they are + * two different answers. + */ + public function testAMissingModelIsNotTheSameAnswerAsAMissingHost(): void + { + $refused = $this->client(static fn (): MockResponse => new MockResponse( + json_encode(['error' => "model 'qwen3:30b' not found"]), + ['http_code' => 404], + )); + + $missing = $refused->preload('http://h:11434', 'qwen3:30b'); + + self::assertFalse($missing->loaded); + self::assertSame('model_missing', $missing->reason); + // The host's own sentence, because it names the model as the host read + // it — which is how a trailing space or a wrong tag gets found. + self::assertSame("model 'qwen3:30b' not found", $missing->reasonParams['error']); + + $down = $this->client(static function (): MockResponse { + throw new TransportException('Connection refused'); + }); + + self::assertSame('unreachable', $down->preload('http://h:11434', 'qwen3:30b')->reason); + } + + private function client(callable $handler): OllamaClient + { + return new OllamaClient(new MockHttpClient($handler), new NullLogger()); + } +} diff --git a/translations/messages.de.yaml b/translations/messages.de.yaml index 0c986ef2..15c86dd4 100644 --- a/translations/messages.de.yaml +++ b/translations/messages.de.yaml @@ -95,8 +95,12 @@ admin: base_url_invalid: 'Das muss mit http:// oder https:// anfangen.' categorisation: 'Mail in Tabs einsortieren' categorisation_help: 'Läuft bei jeder eingehenden Nachricht. Immer nur als Stichentscheid — zuerst entscheiden die normalen Regeln.' + chat_keep_alive: 'Schreib-Modell geladen lassen für' + chat_keep_alive_help: '5m, 30m, 4h oder einfach eine Zahl in Sekunden. -1 lässt es dauerhaft geladen, 0 entlädt es nach jeder Anfrage. Je länger, desto schneller kommt die erste Antwort — der Host hält den Speicher aber die ganze Zeit belegt. Leer lassen, um das zu nehmen, was am Host selbst eingestellt ist.' chat_model: 'Schreib-Modell' chat_model_help: 'Für Antworten, Zusammenfassungen und das Einsortieren. Nimm ein instruction-tuned Modell.' + embedding_keep_alive: 'Such-Modell geladen lassen für' + embedding_keep_alive_help: 'Gleiches Format wie oben. Dieses Modell ist klein und wird ständig gebraucht, deshalb kostet -1 — dauerhaft geladen — meist kaum etwas und spart bei jeder Suche eine Wartezeit; wenn der Host den Speicher übrig hat, lohnt es sich. Leer lassen, um das zu nehmen, was am Host selbst eingestellt ist.' embedding_model: 'Such-Modell' embedding_model_help: 'Wandelt Mail in Vektoren um, damit die Suche Bedeutung statt Wörter findet.' enabled: 'Sprachmodell verwenden' @@ -211,6 +215,19 @@ admin: on: 'An' paused: 'Pausiert' test: 'Testen' + warmup: + button: 'Jetzt vorladen' + failed: 'Keine Antwort. Versuch es noch einmal.' + loaded_ms: 'Modell in %count% ms geladen — es war schon im Speicher.' + loaded_seconds: 'Modell in %count% s geladen.' + loading: 'Modell wird geladen …' + model_missing: 'Auf dem Host gibt es kein Modell namens „%model%“. Zieh es dort, oder trag eines ein, das da ist. (%error%)' + no_host: 'Trag zuerst einen Modell-Host ein und speichere.' + no_model: 'Trag zuerst ein Schreib-Modell ein und speichere.' + status: 'Der Host hat mit HTTP %status% abgelehnt. (%error%)' + timeout: 'Der Host hat nach vier Minuten immer noch geladen, plMail hat aufgehört zu warten. Gut möglich, dass er fertig wird — vermutlich ist das Modell aber zu groß für seinen Speicher.' + unexpected: 'Etwas ist schiefgegangen, bevor der Host antworten konnte. (%error%)' + unreachable: 'Unter der Adresse hat nichts geantwortet. (%error%)' config_backup: change: absent: 'hier noch nicht gesetzt' diff --git a/translations/messages.en.yaml b/translations/messages.en.yaml index 7b7432fa..ce3f9754 100644 --- a/translations/messages.en.yaml +++ b/translations/messages.en.yaml @@ -95,8 +95,12 @@ admin: base_url_invalid: 'This has to start with http:// or https://.' categorisation: 'Sort mail into tabs' categorisation_help: 'Runs on every message that arrives. Only ever a tie-breaker — the ordinary rules decide first.' + chat_keep_alive: 'Keep the writing model loaded for' + chat_keep_alive_help: '5m, 30m, 4h, or a plain number of seconds. -1 keeps it loaded for good, 0 unloads it after every request. Longer means the first reply comes back sooner; the host keeps the memory reserved for the whole time either way. Leave empty to use whatever the host itself is set to.' chat_model: 'Writing model' chat_model_help: 'Used for replies, summaries and sorting. Something instruction-tuned.' + embedding_keep_alive: 'Keep the search model loaded for' + embedding_keep_alive_help: 'Same format as above. This model is small and asked for constantly, so -1 — loaded for good — usually costs little and saves a wait on every search; it is worth setting if the host has the memory to spare. Leave empty to use whatever the host itself is set to.' embedding_model: 'Search model' embedding_model_help: 'Turns mail into vectors so search can find meaning rather than words.' enabled: 'Use a language model' @@ -211,6 +215,19 @@ admin: on: 'On' paused: 'Paused' test: 'Test' + warmup: + button: 'Warm up now' + failed: 'No answer. Try again.' + loaded_ms: 'Model loaded in %count% ms — it was already in memory.' + loaded_seconds: 'Model loaded in %count% s.' + loading: 'Loading the model…' + model_missing: 'That host has no model called “%model%”. Pull it there, or name one that is on it. (%error%)' + no_host: 'Set a model host first, and save.' + no_model: 'Name a writing model first, and save.' + status: 'The host refused with HTTP %status%. (%error%)' + timeout: 'The host was still loading after four minutes and plMail stopped waiting. It may well finish; the model is very likely too big for the memory it has.' + unexpected: 'Something went wrong before the host could answer. (%error%)' + unreachable: 'Nothing answered at that address. (%error%)' config_backup: change: absent: 'nothing set here yet' diff --git a/translations/messages.en_PI.yaml b/translations/messages.en_PI.yaml index efd9e7ed..855d035c 100644 --- a/translations/messages.en_PI.yaml +++ b/translations/messages.en_PI.yaml @@ -95,8 +95,12 @@ admin: base_url_invalid: 'That must begin with http:// or https://.' categorisation: 'Sort mail into holds' categorisation_help: "Runs on ev'ry message that arrives. Only breaks a tie — the ordinary rules speak first." + chat_keep_alive: "Keep the scrivener aboard for" + chat_keep_alive_help: "5m, 30m, 4h, or a plain count o' seconds. -1 keeps it aboard for good, 0 puts it ashore after ev'ry errand. Longer means the first reply comes quicker — but the berth stays taken the whole while. Leave it empty to sail by whatever the host itself says." chat_model: 'Scrivener model' chat_model_help: "For replies, summaries an' sortin'. Somethin' instruction-tuned." + embedding_keep_alive: "Keep the diviner aboard for" + embedding_keep_alive_help: "Same shape as above. This one be small an' called for constantly, so -1 — aboard for good — mostly costs barely a crumb an' spares a wait on ev'ry search; worth settin' if the host has the hold to spare. Leave it empty to sail by whatever the host itself says." embedding_model: "Divinin' model" embedding_model_help: "Turns mail into vectors so the search finds meanin', not words." enabled: 'Consult an oracle' @@ -211,6 +215,19 @@ admin: on: 'On' paused: 'Becalmed' test: 'Test' + warmup: + button: 'Rouse it now' + failed: "Nothin' came back. Try again." + loaded_ms: 'Model roused in %count% ms — it were already aboard.' + loaded_seconds: 'Model roused in %count% s.' + loading: "Rousin' the model…" + model_missing: "That host carries no model named “%model%”. Haul it aboard there, or name one that be. (%error%)" + no_host: "Name a model host first, an' stow it." + no_model: "Name a scrivener model first, an' stow it." + status: 'The host refused with HTTP %status%. (%error%)' + timeout: "Four minutes an' the host were still haulin', so plMail stopped waitin'. It may yet finish — like as not the model be too heavy for the hold it has." + unexpected: "Somethin' went wrong afore the host could answer. (%error%)" + unreachable: "Nothin' answered at that address. (%error%)" db: active: empty: 'No queries afoot right now.' diff --git a/translations/validators.de.yaml b/translations/validators.de.yaml index 8a0da5aa..aadf89a3 100644 --- a/translations/validators.de.yaml +++ b/translations/validators.de.yaml @@ -1,5 +1,6 @@ # Symfony übersetzt Validierungsmeldungen in der Domain `validators`, nicht in # `messages` — ein Schlüssel in messages.*.yaml erscheint unübersetzt. +admin.ai.keep_alive_invalid: Nimm eine Dauer wie 5m, 30m oder 4h, eine reine Sekundenzahl oder -1, damit das Modell dauerhaft geladen bleibt — leer lassen nimmt die Einstellung des Hosts. admin.integrations.client_id_missing: Trage die Client-ID ein, die zu diesem Secret gehört. admin.integrations.secret_missing: Trage auch das Client-Secret ein — mit der Client-ID allein kann sich niemand anmelden. admin.users.password_too_short: Mindestens {{ limit }} Zeichen — dieses Passwort wählst du für jemand anderen. diff --git a/translations/validators.en.yaml b/translations/validators.en.yaml index 8616f940..2323c866 100644 --- a/translations/validators.en.yaml +++ b/translations/validators.en.yaml @@ -1,6 +1,7 @@ # Constraint messages. Symfony translates validation output in the `validators` # domain, not `messages` — a key put in messages.*.yaml renders as the raw key, # which is how "setup.install.password_too_short" ended up on screen. +admin.ai.keep_alive_invalid: Use a duration like 5m, 30m or 4h, a plain number of seconds, or -1 to keep the model loaded for good — leave it empty to use the host's own setting. admin.integrations.client_id_missing: Enter the client ID that goes with this secret. admin.integrations.secret_missing: Enter the client secret too — a client ID on its own cannot sign anyone in. admin.users.password_too_short: Use at least {{ limit }} characters — this password is being chosen for someone else. diff --git a/translations/validators.en_PI.yaml b/translations/validators.en_PI.yaml index 0118eacc..4dae9bf2 100644 --- a/translations/validators.en_PI.yaml +++ b/translations/validators.en_PI.yaml @@ -1,4 +1,5 @@ # Symfony be translatin' validation messages in the `validators` domain. +admin.ai.keep_alive_invalid: "Give a spell like 5m, 30m or 4h, a plain count o' seconds, or -1 to keep it aboard for good — leave it empty an' the host's own word stands." admin.integrations.client_id_missing: Name th' client ID that goes with this secret. admin.integrations.secret_missing: Enter th' client secret an' all — an ID alone signs nobody aboard. admin.users.password_too_short: "{{ limit }} marks at least — ye be choosin' this one for somebody else."