mirror of
https://github.com/karatektus/pl_mail.git
synced 2026-10-25 10:04:18 +00:00
cd7828395d5c37c05baae9fe35f1d177a70945ec
24
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
cd7828395d |
Send the model host extra headers
A bearer token is not all that stands in front of a model. A reverse proxy with basic auth, a zero-trust gateway that wants a client id and a secret, a routing gateway with a key of its own, a provider that takes a project by header: each of those made plMail unusable with it. Admin → AI has an Extra headers box under the token, one `Name: value` per line. ConnectionHeaders reads them strictly while somebody can still fix what they typed, and forgivingly on the way to the wire, so a column that cannot be read costs the headers and not the feature. They are stored encrypted and ride on AiConnection, so both clients send them on every request without knowing they exist. The values are never shown again; the page lists the names on file. An empty box keeps them, typing replaces them, a tick box removes them, and a change of address or provider drops them as it drops the token. A typed Authorization replaces the bearer. The headers the HTTP client writes itself are refused. Fixed values only. The idea of extra headers comes from #58; its per-task session header is left out, because nothing plMail does needs one. Co-authored-by: shady <shady2k@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3c7a03b843 |
Talk to an OpenAI-compatible endpoint as well as Ollama
The language model had to be an Ollama container. Admin → AI now has a Provider field, and the second choice is the request format OpenAI published and nearly everything else has copied: llama.cpp, vLLM, LM Studio, LiteLLM, the hosted providers, and Ollama itself under /v1. OpenAiClient speaks three paths, /chat/completions, /embeddings and /models, and answers in the value objects and error categories OllamaClient already uses. AiAssistant reads the connection for the feature being served and picks the client; nothing above it knows there are two. The body carries only what the format defines, because several servers answer 400 to a field they do not know. The stream is read a line at a time, skips comment lines, does not require [DONE], and fails on an error object delivered inside a 200. Redirects are never followed. What only Ollama can be asked is not asked of the other dialect: no keep-alive is sent, the composer is not told to expect a cold start, Warm up refuses with its own reason, and the panel reads no /api/ps. The page puts those controls away while the other provider is chosen and keeps their values. A stored token is dropped when the provider changes, as it is when the address does. One column on ai_settings, defaulting to ollama; the migration is reversible. The SSE handling, the strict reading of an embedding and the rule that a key stays with its endpoint come from #58. Co-authored-by: shady <shady2k@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5.5
|
351c92b8a3 |
Send the model host its token
Admin → AI has a Token field for an Ollama behind a reverse proxy. The token was stored, encrypted, and never sent: every call site passed the address alone. A host that asks for one answered 401 to everything while the page said a token was on file. The address and the token now travel as one AiConnection, and OllamaClient scopes its HTTP client to it, so every request carries the header — including the fallback embedding dialect and the version read behind a probe. Redirects are not followed once a token is attached. The token belongs to the address it was typed for. The form drops a stored token when the address changes and none is typed, in Admin → AI and in the setup wizard alike; the rule lived in both controllers and is now in the form. The Test button binds to a copy of the settings, shows the typed token to the typed address, and asks nobody without a valid CSRF token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5.5
|
3a6f4db0f1 |
Measure mail arrival from when the provider accepted it, and show the slowest
Admin → Performance measured "how late mail arrives" from the date a message carries, which is the sender's clock. A newsletter dated half an hour before it was sent, or mail held back by greylisting, was reported as plMail being that late. The delay is now taken from the newest Received: header. Each message is stamped as it is first stored with when the provider accepted it, what started the sync that fetched it (push, IDLE, schedule, sync button) and where it was filed. Inbox mail and the rest are counted apart, since providers only announce the inbox, and the slowest messages of the period are listed in a section that starts collapsed. An upgrade task gives the last week of mail its accept time from the stored headers. What fetched older mail cannot be recovered and shows as a dash. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5.5
|
2dfc9b3b9b |
Give the admin panel a page for how long things take
The line under the hold switch in Admin → AI said "typically 6 s, 95%
within 13 s": enough to see that one message took twice as long as the
rest, and nothing about why. System says whether plMail is working;
nothing said how long anything took.
Admin → Performance, directly below System:
How late mail arrives per account, provider-received to stored.
Seconds is push working; minutes is the
quarter-hourly poll, which "works" and so
showed up nowhere. Imports and plMail's own
sent copies are not arrivals.
What each worker has the queue numbers by the process that owns
waiting them, a worker's queues in the order it
looks at them. Live; the transport keeps no
history.
New mail waiting for one row per held message, its wait split
the assistant three ways: queued, the model call, and how
much of the call was the model loading.
Model host moved from Admin → AI, unchanged.
To split a wait, the three parts have to be recorded with the message:
ai_queue_ms, ai_call_ms, ai_load_ms. ai_call_metric has the same
numbers and no idea which message they were for, and its timestamps are
the database's clock where the message's are PHP's, so a join by time
is not one to rely on. AiAssistant::ask() returns the whole result for
that; chat() is ask()?->content.
The model host's card and the search index backfill were one fragment
polled by one endpoint. They are about different things — a measurement
and a job with Start and Pause on it — so the fragment renders either
half, each page names the one it wants, and a caller that names neither
gets both, as before.
BackgroundProcesses::consumers() is the worker-to-queue map, read by
all() to start the processes and by this page to label them. They were
two facts that had to agree.
Rows name accounts. Nothing about any mail's content is shown.
Not looked at in a browser; the E2E card-inset spec covers the section
and has been given its frame.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Paul LütznerandClaude Opus 5.5
|
88b1744385 |
Leave the ingest worker to fetch mail, and nothing else
Adding an account put new mail behind hours of follow-up work. What
plMail does to a message once it has it — a model call where the
assistant sorts, event extraction, insight cards, receipts — was queued
on `ingest`, beside the syncs, for every message of the import, oldest
first. One consumer serves that queue. 988 envelopes pending and the
oldest an hour old was a mailbox being asked about newsletters from
2019 while mail on the server waited to be fetched.
`ingest` is now what talks to a mail provider. The rest moved:
enrich_live worker-live follow-up work for mail that has just
arrived on an account whose import is
over; never behind an import
enrich worker-enrich the same for an import's recent mail,
and search indexing
enrich_backlog worker-enrich old mail for the assistant, consumed
only when `enrich` is empty
release worker-release ends a hold (below); nothing else
Which queue is decided per batch, not per message class: the pipeline
asks InitialImportState once, carries the answer on PostIngestResult,
and EnrichmentRouter turns it into a stamp.
Only mail from the last 30 days (ENRICH_RECENT_DAYS) gets follow-up
work on arrival. Older mail is stored, threaded and filed by the rules.
The assistant is asked about it afterwards — newest first, a bounded
handful per run, and not until every account of that person has
finished importing (app:ai:classify-backlog, ClassificationCatchUp).
For the other steps `app:backfill` remains the way to read history.
ExtractInsightsStep and ProposeEventsStep ran their work inside the
sync and argued for it; both dispatch now.
New mail no longer appears in one tab and moves to another. For
somebody sorting by assistant the rules' category is a guess about to
be replaced, so where the verdict could still change it the message is
held: stamped before threading, its new thread given no category, which
every tab query already reads as "in no tab". The answer releases it.
Failing that a timed release does, after AiSettings::holdMaxSeconds —
60 by default — and the mail is shown where the rules put it; the model
call carries on and moves the mail when it answers. A message is asked
about once. Admin → AI has the switch, the ceiling, and the measured
delay over the last week beside them.
Three things about the release that were each wrong once first:
- It ran on the live worker, so it could not fire while that worker
was inside the call that was taking too long. It was moved to
export, which mixed mail appearing with mail leaving. It has a
queue and a worker of its own.
- The call was cut off at the ceiling and the message asked about
again later. The hold and the question are separate; only the hold
has a deadline.
- A late verdict was stored and moved nothing. It moves the mail.
Both workers still end at one model host, so bulk classification looks
at the live queue before each call and stands aside (LiveMailPriority).
One migration: two columns on `message`, two on `ai_settings`.
Not verified against a real import or a live model host: the tests pin
the queues, the hold and the release against in-memory transports and a
mocked host. Known and left: IMAP still imports oldest first; a reply
into an existing thread cannot be hidden; a Gmail account counts as
importing for up to an hour after its last message lands.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Paul LütznerandClaude Opus 5.5
|
8289891830 |
feat: failed requests raise a toast instead of breaking the page
A failed Turbo response used to be rendered: the error page drawn as the page (or merged under it — a picked PDF avatar left plMail's 500 hanging off the settings screen), or "Content missing" in a frame. - request_errors.js catches every Turbo visit, form and frame. A failed response is not rendered; the page stays as it was and an error toast says what happened. 422 (a form with its errors), 401/403 (the session guard) and elements marked data-turbo-errors="self" (the compose dock) are left alone. A network failure gets its own words; a frame response of the wrong shape is visited as a page. - requestFailed() gives controllers' own fetch() calls the same toast. An audit of the 60 calls fixed the 22 files whose user-initiated requests failed silently (star, archive, labels, drag and drop, bulk actions, reorders, appearance save, undo/cancel send, and more), and left background polling and preferences quiet on purpose. - Every request gets an 8-hex id: X-Request-Id on the response, `id` on its log entries, "Ref. …" in the toast with a copy button, and a Reference filter in Admin → Logs. A proxy's own id is kept. - toast.js clones the server's toast partial, so a script's toast is the same pane and theme as a streamed one. The sync button uses it, and its hardcoded English is now translated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
d53e7c4963 |
Report browser errors to the admin panel, in a card of their own
A JavaScript fault was visible only to whoever had the console open, which on a self-hosted mailbox means nobody. Uncaught exceptions and rejected promises from plMail's own scripts now reach Admin → Logs, together with whatever the Content-Security-Policy blocked. Its own card rather than rows in the log, because it holds a different kind of thing. A server error is one event in one request; a broken line in a controller runs on every page load for every user. So a row is a distinct fault — message, file, line — with a count and a lifetime, and the same fault reported again increments it. Merged into the log, one bug would be four hundred rows and the log would be unreadable past it. THE FILTER IS THE FEATURE. Most of what window.onerror sees is not ours: the console screenshot that prompted this was an ad-attribution metrics script failing inside requestIdleCallback, injected by an extension. A panel that accepted that fills with other people's bugs and stops being read, which is the same as not having one. A report is kept only when it can be attributed to a script this server sent — an injected script has no URL (Chrome's `VM947`), and a cross-origin one arrives as the bare string `Script error.` with no file, line or message. The rule is applied in the browser and again in the recorder, because the browser half is part of the code being reported on. CSP violations come to the same card through /csp-report. That endpoint cannot be CSRF-protected and is public in security.yaml, because the browser generates the request itself: no token, no header we chose, and on a login page no session. The rate limiter is the bound. It is worth having because the full policy is enforced in production, so what arrives is what this installation actually refused to load in a real browser. The count means page loads, not executions — the client reports each fault once per page, capped at five. Measured rather than assumed: throwing the same error twice on one page records one sighting. Documented that way on both sides after the first version claimed the server would count what the client never sent. `column` is a reserved word in Postgres, so the field is `column_number`. The quoted DDL created the table happily and every insert into it was a syntax error — same trap User::$usr already avoids. Verified on a running stack by throwing real errors in a real browser: an error from a script on this origin recorded, an anonymous injected one refused, the card rendering both kinds grouped with counts. Its first catch was a false alarm, and a useful one to record. It reported the Mercure connection as blocked by `connect-src` — which was true, and was me browsing `localhost` while MERCURE_PUBLIC_URL names `127.0.0.1`. Different origins, so the app's own policy refused its own hub. Browsing the host the config actually names produces no violation at all. The compose comment calling that URL "same-origin" is correct on its own terms; it just stops being true if you reach the stack by another name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
41362290bb |
Release v0.2.20
Twenty-six commits since v0.2.19, of which the changelog had one entry — the rest written up here. The handbook needed two corrections rather than additions. features/mail.md still said a sender you have written to is pulled into Primary "regardless of any bulk-mail header", which stopped being true when the mailing limit went in and directly contradicted the paragraph two sections above it. features/admin.md described Reported mail as one pile of missed insights with a scope drop-down, which is now two kinds in one list with a selection. Both German pages follow, and both markers are re-hashed. README gains the sorting choice and says the summary can take a whole conversation. Screenshots are untouched: nothing in the seven views they show changed this cycle — the new interface is in Settings and in Admin → Reported mail, neither of which is photographed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
ffe9457b42 |
Give the search model its query instruction and its own threshold
Search by meaning was answering "mails about food" with a page of login links
and newsletters, ranked confidently, with nothing anywhere saying it had gone
wrong. Two settings decide whether it works and plMail had neither.
THE QUERY INSTRUCTION. Embedding models are asymmetric: they are trained to be
told which side of a search they are looking at. An unlabelled query is treated
as one more document, and it lands nearest whatever is most boilerplate — in a
mailbox, login links and newsletters. Measured on qwen3-embedding:0.6b over
twelve German and English messages, five about food:
query non-food ranked above the best food mail
"mails about food" 5 (raw) 0 (instructed)
"essen" 7 (raw) 0 (instructed)
"restaurant reservation" 2 (raw) 0 (instructed)
"essen" — the least ambiguous word available — put every food mail below every
non-food mail. With the model's own instruction, perfect separation on all
three. It goes on the QUERY ONLY, so stored vectors are untouched and nothing
has to be re-indexed.
THE THRESHOLD. 0.55 was one constant for every installation, and cosine
similarity is not comparable between models — or even between the instructed
and uninstructed forms of one model. Best threshold by F1, same corpus:
qwen3-embedding:0.6b no instruction 0.20 F1 0.59 precision 0.42
qwen3-embedding:0.6b with instruction 0.42 F1 0.85 precision 1.00
nomic-embed-text no instruction 0.45 F1 0.74 precision 0.65
all-minilm no instruction 0.20 F1 0.71 precision 0.77
0.55 is above all of them. So it travels on SemanticSearch next to the model and
width it has to agree with, and it is a setting rather than a release.
Both are columns, and EmbeddingPreset holds the measured pair for each model the
release was tested against, with a line on what it is good and bad at. The model
field stays free text — a closed dropdown would go stale and refuse something
that works — and choosing a preset fills all three fields at once, because they
are one decision written in three boxes: any two from different models give a
search that is confidently wrong rather than visibly broken.
qwen3-embedding:0.6b becomes the documented default. Multilingual matters more
than any benchmark for a mailbox that is not all in one language, and it was the
only combination measured at precision 1.00.
Existing installations keep 0.55 and no instruction. Backfilling 0.42 would
apply Qwen's number to whatever model is configured, and on all-minilm (0.20)
that is twice as tight as it should be — a migration is the wrong place to
change what somebody's search returns. The panel now shows both values and sets
them on one click.
The instruction field is trim: false, which a test caught rather than a guess:
every working instruction ends in a separator, Qwen's is "…\nQuery: ", and
Symfony's default trim turned it into "Query:mails about food".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Paul LütznerandClaude Opus 5
|
d8f8acf29b |
feat(ai): tunable keep-alive per model, and a warm-up button
The first "Help me write" after a quiet spell shows nothing for tens of seconds while the Ollama host reads twenty gigabytes back off disk. How long it waits before evicting a model is OLLAMA_KEEP_ALIVE — an environment variable on a machine plMail can talk to but cannot configure. The API takes the same setting per request and a per-request value wins, so this is the only lever available from inside the app. Two nullable columns on ai_settings, one per model, because the two are opposite trades: the writing model is eighteen gigabytes and asked for occasionally, the embedding model a few hundred megabytes and asked for on every search and every indexed message. Both ship at 5m, which is Ollama's own default, so an upgrade changes nothing about how a host behaves — the boxes just say out loud what was already happening. -1 is recommended for the search model in the help text and the handbook rather than shipped, because plMail cannot see how much memory that host has. Empty means send no keep_alive at all and let the host decide, which is a real selectable state and not an unfinished one. The wire representation is the part that is easy to get wrong: Ollama parses a JSON string as a Go duration, and a Go duration needs a unit, so "-1" and "3600" as strings are a 400 on every model call from a setting that reads as correct. KeepAlive::forBody() is the only thing that puts the value in a body — int for the numeric spellings, string for the durations, null to omit the field entirely, since an explicit null is read as zero and unloads immediately. One helper adds it to all four bodies so no call site can get that wrong. Validated with the same expression the body is built from, refused rather than coerced, and the message lives in the validators domain where Symfony actually looks for it. Warm up now: POST /admin/ai/warmup issues the no-prompt /api/generate, times the round trip and reports it. PHP's 30s execution limit is raised for it — a cold load is not 30 seconds — and the upstream timeout sits below that so it gives up first. A host that is unreachable and a model that is not on the host are distinct, translated answers. Also carries both columns through the configuration backup, and adds the AI feature to the README, which had never mentioned it. Fixes a search test that asserted how fast the machine was: it gave the last-resort body pass a one-millisecond budget and one row to scan, so on quick hardware the pass finished in time and the test failed. It now provokes a genuine statement_timeout cancellation with pg_sleep, which exercises the same guard without a clock in the assertion. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
e7f4513574 |
fix(backup): the config backup had stopped carrying what it promises
A restore looked like it worked. That is the whole shape of this: nothing threw, the review page listed a clean plan, and the parts that had gone missing produced no row to warn about, because a section that does not exist has nothing to say. ── A removed user came back to life ──────────────────────────────────────── The restorer's own docblock and the handbook both say a soft-deleted user counts as existing, "because they hold the email address against a unique index". They had stopped holding it. Removal frees the address on purpose — the admin delete action rewrites it to `deleted-<id>@invalid` — so existingId() answered null for every removed user, the plan called them Absent, and the restore created them afresh and LIVE: old password hash, TOTP secret, recovery codes, every mailbox credential. Listed as an ordinary creation, indistinguishable from a new colleague. The handbook steers operators straight into it. The rewrite destroys the only copy of the address, so nothing could match it back; `email_at_deletion` is that copy, written at the same moment. Rows tombstoned before the migration stay unrecognisable, which is stated and held by a test rather than papered over. The regression test that was supposed to catch this passed throughout, because it wrote `deleted_at` and left the email in place — a row the application cannot produce. It now tombstones the way the delete action does. Its sibling: a violated unique index escaped as an uncaught 500 with no message. api_token.token_hash is unique across the WHOLE install and a soft delete cascades nothing, so a removed user's tokens still hold those values. Translated into a stated failure with a sentence attached. Not demoted like other failed writes — a violated constraint aborts the transaction, so there is no "rest" to let through, and stopping with a reason is the honest shape. ── Four things a person configured never reached the file ────────────────── The assistant configuration in full: model host, both models, the reverse-proxy token that exists in plaintext nowhere else, the feature switches, and every prompt an administrator rewrote. The per-user half had been carried since the day it shipped; the instance-level half it depends on had not, so a restore returned everyone's preferences pointing at a host that was gone. Account signatures and read-receipt defaults — and on the instance this was found on, `compose.signature` was the ONLY account setting in use, so every export it produced held an empty settings bag for every account. The per-alias overrides key themselves by alias row id, which names somebody else's alias on the target, so they now travel inside their alias and are rewritten under the new id after the flush, the way `calendar.target_id` already was. Five user settings, the chosen log level, and the OAuth scopes that describe what the provider granted to the refresh token the file already carries. ── Why none of it was caught, which is the part worth keeping ────────────── Every guard pointed the wrong way. The env test builds its expectation by calling the constant under test. The users test is fourteen assertArrayNotHasKey calls — it fails if somebody ADDS a field, never if one goes missing. The version assertion compares a constant against its own echo. A backup's characteristic defect is omission, and nothing could fail in that direction. Two tests now compare the backup against the application rather than against itself, and fail closed: every User and Account SETTING_ constant must be carried or excluded on the record with its reason, and the database section must carry every admin-configured table. The second would have gone red the day AiSettings landed. A round-trip test pins the set of things a restore drops to exactly two, each deliberate, so the set can only change on purpose. ── Also ─────────────────────────────────────────────────────────────────── The KDF had a ceiling and no floor, so `memlimit: 1` passed every check and took a SodiumException out onto a blank error page — through the anonymous /install/restore among others. Both memzero paths now run under try/finally, the abnormal case they exist for. App passwords keep the date they were created rather than the date they were restored. And the admin handbook said the file contains "no user accounts". It contains every one, with their secrets — published in two languages. Corrected, along with the inventories, which now list what actually travels. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
755f3cf257 |
feat(users): an administrator can run the whole life of an account, and you can change your own password
Managing users was reachable and worked — list, search, create, promote, demote, remove — but three things around it did not, and one of them was load-bearing. ── You could not change your own password ────────────────────────────────── Nowhere. Not in Settings, not by mail, not at all; only an administrator setting one for you. Settings → Security now leads with a Password card that asks for the current password (an unmapped UserPassword constraint, so it is checked against the signed-in token rather than against anything the form was handed) and takes the new one twice. The route carries no id and the form has no data_class, so it structurally cannot become the admin form by accident. The floor those passwords are held to was three literal 12s in three files that happened to agree. It is now User::PASSWORD_MIN_LENGTH, read by all three. ── An account could only be removed, never switched off ──────────────────── Nothing sat between "leave it alone" and "destroy it", so the answer to "this person has left, keep their mail" was to delete them. There is a deactivated_at now, and a UserChecker registered on BOTH firewalls — the JMAP one included, because an app password outlives the browser session it was minted from and a check only on `main` would shut somebody out of the web UI while every mail client they had already connected kept syncing. The refusal is in checkPostAuth rather than checkPreAuth on purpose. checkPreAuth runs before the password is verified, so refusing there would answer "is this address a suspended account here?" to anyone who types an address — the question hide_user_not_found exists to keep unanswerable. After verification, the only person told is the one who could otherwise have signed in. Symfony runs user checkers at authentication only, so a session already open would have carried on until it happened to end. UserProvider::refreshUser() refuses the same two states on every request. Suspension also travels through config backup, which would otherwise have handed a suspended account its sign-in back on restore. ── Two smaller lies ──────────────────────────────────────────────────────── A search under three characters silently returned the unfiltered list, which a QueryBuilder has no way to complain about, so the floor was dropped rather than kept as something that could only mislead. And an admin declining to demote themselves got a success toast for a change that never happened; the refusal now says so, while the rest of the form still saves. ── Restoring a removed user is deliberately absent ───────────────────────── Removal overwrites the address, the name and the hash specifically to free the unique address for reuse, so a restore would have to invent all three and the original address may by then belong to somebody else. Doing it properly means snapshot columns and a guided flow that asks for a fresh address — a different change, which alters what removal means. Documented as absent on the screen an administrator would look for the button on, because switching an account off is the answer to the question people are actually asking. Also fixed en route: the modal form theme dropped help_translation_parameters, so the first field to use one rendered "at least %count% characters" on screen. And GermanIsInformalTest only ever globbed messages.de.yaml, leaving validators and the new security catalogue — exactly where a translator reaches for "Sie" — outside the test that exists to catch it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
661d3bbf25 |
feat(ai): summarise a conversation, when somebody asks
On demand, and that is the design rather than a first step. A summary spends the 20.3 GiB writing model for about a minute cold, on one GPU already shared with the indexer and the composer. Summarising every arriving thread would spend days of it on conversations nobody opens. FRESHNESS IS DERIVED, NOT MAINTAINED The stored row carries a SHA-256 of the exact transcript that was sent, plus the model tag and a prompt version, and freshness is that hash matching what the thread would send today. Nothing has to remember to invalidate anything. lastMessageAt was the obvious key and is the wrong one: it only ever moves FORWARD, so a deleted message leaves a summary describing mail that is no longer there, silently and for ever. The hash also makes a prompt change or a budget change invalidate every stored summary by itself, which is correct and free. CASCADE, following message_embedding rather than MailInsight. The two precedents point opposite ways and both are right — an insight outlives its message because "a deleted shipping confirmation does not un-ship the parcel", an embedding does not because it describes one. A summary describes a thread. THE BUDGET IS MEASURED, NOT PICKED 8000 characters, against two ceilings and the smaller one is silent. The visible one is the wait before the first token — load plus prompt eval, about 42s at this budget, a third of the client's idle timeout. The quiet one is the context window: OllamaClient sends no num_ctx and Ollama's default is 4096 tokens, so 12000 characters sits exactly on the line, and an install running the default would drop the HEAD of the transcript — the part the reader most wants and the part the transcript is assembled forwards to protect. No error, no notice. The fourth AiFeature case landed LAST, after every arm that reads it. Eight exhaustive matches key on that enum and none of them fails at compile time; AiPermissions::anyAdminEnabled() loops the cases and the settings nav calls it on every settings page, so the case arriving early would have taken the whole settings area down rather than the AI section. Verified against the real host, not a mock: a German thread returns a German summary, a reopened thread renders the stored row with no new model call, a reply greys it and offers to run again, and changing the chat model hides it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
92d6e8ffab |
feat(ai): index new mail when the model is warm, and once a night
New mail was embedded the moment it arrived. That is the wrong trade on a box
with one GPU: it is rare to search for mail that has just landed, and paying to
index every arrival competes with the two things somebody is actually waiting
on.
Two triggers replace it. A nightly sweep is the backstop, and a search is the
invitation — a search has just loaded the embedding model, so the cheapest
possible moment to index is immediately after one, when the model is warm and
nobody is waiting on it any more.
THE PART THAT MADE THIS MORE THAN A SCHEDULE CHANGE
The backfill stood aside after ANY interactive AI use, and a search counted.
That is precisely backwards for search, and it took both halves of the signal to
say so: lastInteractiveCallAt() matched
`feature IN ('search_query','writing_help')`, and the request subscriber stamped
`app_mail_search` alongside `app_compose_assist`.
The distinction the old code could not make is that there are two models with
completely different costs. The composer spends the 20.3 GiB writing model,
which is worth standing aside for and is the case the yielding was written for.
Search spends the sub-gigabyte embedder — the same model indexing uses. So a
finished search is not a reason to keep away, it is the best moment there is.
Both docblocks now cross-reference each other, so restoring one half alone reads
as obviously wrong.
The opportunistic trigger claims its rate-limit marker BEFORE it queries, so the
mailbox scan is inside the window rather than only the dispatch, and a search
that produced no vector at all spends nothing — one unreachable moment must not
cost the next five minutes of indexing.
EmbedMailStep is deleted rather than emptied. An empty step would still be built
and iterated on every ingest batch while carrying a name asserting that
embedding happens on arrival — an active lie in the one directory somebody reads
to find out what happens to new mail.
The finder matches on model and not on width, deliberately. What consumes its
ids asks about the model alone, so a width-aware finder would hand over a full
budget the handler then drops as already done: a nightly sweep that reports
itself busy and indexes nothing, for ever. The width mismatch already has an
owner in the coverage report's "stale" count, which tells the user to re-index.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Paul LütznerandClaude Opus 5
|
672b38d6c2 |
test(ai): pin the refusals, and document the feature for the people using it
TESTS
The refusals are the half worth having. This produces text that goes into
somebody's outgoing mail and vectors that decide what a search finds, so
the failure that matters is not a clumsy sentence — it is a model
inventing a message because it was handed nothing to work from, or a
search quietly getting worse for an install that never switched any of
this on.
· WritingAssistant: the three rewriting tasks refuse an empty draft
without spending a request, a reply can still be drafted from the
message being answered alone, a code fence is stripped and prose that
merely mentions backticks is not, and the task list is closed —
asserted as a count, because it is a security property rather than
tidiness.
· SemanticQuery: every reason a vector might be missing — off,
unconfigured, host refusing the connection, query too short, a vector
that cannot be normalised — arrives at the same null, because search
must never be worse than it was before this feature existed.
DOCS
Admin -> AI, in both handbooks, written for somebody deciding whether
they want this rather than for somebody who has already decided. The
three switches are a table with a "when it runs" column, because that is
the difference that matters: sorting runs unattended on every arriving
message and the other two do not.
Says plainly that semantic search needs one pass over the existing
mailbox and that switching it on does nothing for mail already there —
which is the thing most likely to be reported as "it does not work".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Paul LütznerandClaude Opus 5
|
5f648864ab |
feat(admin): set the log level from the page that shows the logs
Asked for directly: "let the admin panel set the log level. i do not want to mess with env vars." APP_DB_LOG_LEVEL meant editing a file on the host and restarting the stack, and the moment it is actually wanted is the moment that is most awkward — something is wrong, the answer is one level further down, and restarting throws away whatever was in flight. So the level gets turned down, forgotten, and left down. WHAT IT IS NOT Not another control in the filter row. That row narrows what is SHOWN out of what was stored; this decides what gets STORED. They look alike and mean opposite things, and side by side they would invite an admin to change the wrong one and conclude the log was broken. Its own row, its own form, its own wording. NULL IS NOT "TODAY'S VALUE" A stored level of null means "follow the environment", which is deliberately different from copying APP_DB_LOG_LEVEL's current value into the row. An install that has never chosen keeps following its configuration including later changes to it, and there is a way back to that after a level has been set — which storing the resolved value would have removed, silently, on the day the env var next changed. PER RECORD, NOT PER PROCESS Monolog fixes a handler's level at construction. That is right for one configured in a file and useless for one an administrator can turn down from the page they are reading, so DoctrineLogHandler decides in isHandling() instead, and the constructor's level is now only the answer when no resolver was given — which keeps directly-constructed handlers, and every existing test, behaving as they did. LogLevelResolver caches for ten seconds. A query per log line would put a database round-trip inside the thing that runs when the database is in trouble; a value cached for the life of the process would leave every long-lived worker ignoring the change until the next deploy, and the worker logs are usually the ones it was lowered for. It uses DBAL, not the ORM, for the same reason the handler does, and every failure — no table yet, no database, a name that is not a level — falls back to the environment rather than throwing. A logger that throws while deciding whether to log is worse than one briefly out of date. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
e8f4033f03 |
docs: changelog and log-entry reference for v0.1.36
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
e74fa116e5 |
docs: the handbook learns about reported mail and three new shapes
The radar section gains the three new sources and the one that behaves differently from the rest — a one-time code carries an expiry rather than an occurrence, and carries no time at all when the mail states none. Reporting gets its own subsection in both languages, and it leads with what reporting costs: the report carries a copy of the mail, that copy is readable and downloadable by an administrator, and on an installation you do not run yourself that is worth knowing before you press the button. Admin gains the panel, the export and the reason downloading marks nothing handled. Two traps added per page, both things that actually bite: reporting hands over a copy of the mail, a code with no stated lifetime gets no expiry, and the export — unlike the config backup — is plaintext mail. German re-hashed from the English it was made from. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Fable 5
|
567db06844 |
Restore a config backup by writing it, not by printing it
The import classified every environment value as unwritable and handed
back two dozen lines to paste into .env.local. That was built on a
premise that is wrong for the way plMail is deployed: nobody hand-edits
those. frankenphp/generate-secrets.sh mints them on first run into
var/secrets/generated.env, every service mounts the volume that file is
on, the entrypoint loads it before it execs the server, and the app
process can write it. So the import writes it.
What the entrypoint actually promises, and what everything here follows
from: a real environment variable wins over the generated file, which
wins over .env.local and .env, with an empty value counting as absent at
every level. The shell (load_generated_secrets) and PHP
(config/bootstrap_generated_secrets.php) apply exactly the same rule.
ConfigBackupObstacle — "why can plMail not write this" — becomes
ConfigBackupDisposition, "what becomes of this": applied,
applied-on-restart, shadowed-by-compose, external, kept-deliberately,
not-writable. The restart instruction is now one sentence for the whole
plan rather than one line per value, because it is one action.
Three fates survive, each forced by the deployment rather than by
caution:
- POSTGRES_PASSWORD, the postgres_password file and DATABASE_URL stay
external. generate-secrets.sh does rewrite that file from
generated.env on every boot, but the Postgres image reads
POSTGRES_PASSWORD_FILE only at initdb — on a database that already
exists the role keeps the password it was created with, so writing
one without the other is a stack that cannot authenticate.
- APP_ENCRYPTION_KEY is kept on purpose: the import re-encrypts the
backup's credentials under the key in force here, and swapping it
underneath them would make the rows it just wrote unreadable. It is
a note now, not a chore — the line is still offered for the operator
who is also restoring the old database.
- A name pinned non-empty in the process environment beats the
generated file at the next start. Detected by comparing the live
value against what the file holds, because the entrypoint exports
that file's own contents and "is it in getenv" distinguishes
nothing. On the stock compose.yaml that is three names.
The /install path is the one this was for: upload and password are the
whole job, everything lands in this instance's own secrets file and
database, and the page asks for one restart instead of a wall.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Paul LütznerandClaude Fable 5
|
1adac1fe40 |
Carry an instance's configuration in one password-sealed file
Export from the admin area: env secrets, the JWT keypair and the database-held provider credentials - FCM included, decrypted inside the envelope because the restore target holds a different encryption key - sealed with Argon2id and secretbox at parameters a Raspberry Pi can afford, in a documented format a php one-liner can open without plMail. Import shows its plan before touching anything: what it can write is written and re-encrypted under this instance's key, what it cannot - process environment, the Postgres role's own password - is rendered as the exact lines to place by hand. The install wizard offers the same restore before the first account exists, behind the same guard that closes /install forever afterwards. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Fable 5
|
287c93805d |
Record where every push went, and stop scheduled sends being invisible
Every attempt both senders make lands in push_delivery - transport, device, outcome, the error name that tells a dead phone from a Firebase outage, latency - browsable under /admin/push/deliveries and echoed per device in the user's notification settings. Nothing of the payload past its @type is stored, and the record survives the subscription it is about. A held submission now answers EmailSubmission/get as pending with its real release time, a cancelled one as canceled, and the changes feed reports both transitions - the release time used to exist only in the create response, which forced clients to keep schedules device-local. Cancelling a never-submitted mail is refused instead of arming a flag the next real send would silently trip over. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Fable 5
|
e5596a9b47 |
Send push over FCM for clients that ask for it, configured from the admin page
A second transport beside Web Push: PushSubscription/set create accepts a vendor fcmToken, the dispatcher fans out per transport, and delivery goes through FCM HTTP v1 with an OAuth token minted from the service account the admin pasted - no new dependency, openssl and the http-client do it. The admin page takes the service-account JSON (encrypted at rest) and the instance's google-services.json, refuses mismatched pairs, and the session advertises fcm plus the public client config a stock Play Store build needs to initialise Firebase at runtime against this very instance. Committed by the supervisor: the implementing session hit its usage limit after writing code and tests but before running them; results recorded in the merge that follows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
4d103cfbdf |
Write the handbook, and make it fail the build when it goes stale
README says what plMail is and gets you running; CONTRIBUTING says how to
change it. Nothing covered the part in between — using it, operating it,
and registering the Google and Microsoft applications it talks to. That
is docs/: 34 pages across features, installation, providers and
internals, mirrored to the GitHub wiki.
The wiki is a mirror and never a source. bin/mirror-wiki.php flattens the
tree into the wiki's flat namespace, rewrites every relative link,
generates the sidebar from the index so the two cannot disagree, and
stamps each page with a footer saying where it came from. A browser edit
survives until the next push to main. That cost is deliberate:
documentation not versioned with the code drifts from it, and the drift
is invisible until somebody follows an instruction that stopped being
true.
Which is the whole problem, so the checkable half is checked.
DocumentationCoverageTest fails the build when a command is missing from
the reference or documented and gone, when a variable in .env is absent
from the configuration page, when a relative link dangles, when a page is
unreachable from the index or promised and missing, or when a page has no
"Things that bite" section. It cannot check whether a paragraph is TRUE,
and says so — this is the floor, so review attention goes to the part a
machine cannot have an opinion about. CONTRIBUTING has claimed since it
was written that the command table "is part of the definition of done";
nothing enforced it until now, and it was wrong in three places.
Writing it found things. They are fixed here rather than documented
around, because a handbook describing a bug accurately is still a
handbook nobody can follow.
compose.yaml mounted no volume for var/attachments, var/raw
or var/uploads. Workers write those during sync
and the web container serves them back, so each
kept its own copy in its own layer — downloads
404, and everything died on a recreate. The
override and the TrueNAS file both mount them;
the file the README tells an operator to run did
not. The same omission cost real data once
before. See the CHANGELOG for moving what is
currently stranded.
app:calendar:materialise new, nightly. Occurrences are drawn when an
event is saved and nothing moved that window
again, so a weekly standup ran out of dates in
two years and its reminders stopped with them —
silently. The query it runs had existed since
the materialiser did, documented as exactly this
sweep, and had never been called.
Microsoft wizard asked for three of the nine permissions plMail
requests, and a token already issued is not
upgraded when the rest are added later.
GmailWatchService told you to grant Pub/Sub Publisher to
gmail-api@system.gserviceaccount.com. The
account is gmail-api-push@; a topic granted
exactly what the docblock asked for never
delivered a notification.
IcsExporter said it wrote no VALARM. It has written them
since alerts landed.
CLIENT_DEVELOPMENT.md said there was no JMAP calendar API and that
SearchSnippet/get was absent. Both shipped.
CONTRIBUTING said only app:calendar:push registers a push
channel; subscribing has done it since the
subscribe flow learned to.
Three docblocks named getters removed long ago, or described a
method two below the one they sat on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|