mirror of
https://github.com/karatektus/pl_mail.git
synced 2026-10-25 01:54:10 +00:00
71e2176fd138bda22fa056a9bb0084e021cd5fb9
127
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Paul LütznerandClaude Opus 5.5
|
71e2176fd1 |
Let the inbox be one list, without category tabs
Some people do not want their inbox divided. A third control on the sorting card, Inbox tabs, switches the tabs off per person; on by default and true on every existing row, so nothing changes by upgrading. One column, user.category_tabs. Off changes what is drawn and nothing about what is stored. Mail is still filed under a category as it arrives, so switching back shows sorted tabs at once and nothing has to be re-filed in either direction, which is also why the control queues no ResortMailbox. That includes the assistant: somebody who leaves it sorting into tabs they have switched off has decided that, and the card says so when both are true rather than preventing it. The inbox query takes a null category for the whole inbox, through one helper shared by the list and its count. ListViewResolver decides from the person's setting and not from the value the page posts, so "select all in this view" is the list the server would have drawn: an untabbed page posts an empty value, and reading that as Primary would have acted on part of a selection. What only exists for tabs goes with them: the strip and its drop targets, the per-category keys in the counts payload and the grouped reads behind them, and the panel in message details that explains a filing and takes "wrong tab" reports. The JMAP session publishes the choice as inboxTabs beside the appearance hint. Thread.category is served either way, so a client drawing tabs from it had no other way to learn not to. Closes #29 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5.5
|
98ea3948f4 |
Only let mail change the calendar when it comes from where the event came from
Reported in #34, and both findings held up: eight tests written from the report's scenarios fail on the code as it was. An invitation was found by its UID and then judged by SEQUENCE and arrival time alone. Nothing asked who had sent the update, and a UID is not a secret — everyone invited holds it, and so does anyone a copy was forwarded to. Any of them could cancel the meeting or move it and swap its link, and the reader's earlier "accepted" was carried over on top. A meeting mirrored from a connected calendar was rewritable the same way, which the report did not mention. A booking read out of a mail's markup was keyed on the From line's domain, which nobody had checked, and drawn at once. One rule closes both, in EventReconciler::mayApplyUnasked(). It asks about the sender of the MAIL, never the ORGANIZER line, which a forger copies along with the rest of the file: an invitation's update from an address the event's content has already come from, or from the organiser it names. "Already come from" rather than "is the organiser" because a calendar service sends the invitation and its updates from one address of its own. a booking only when the receiving server vouches for the From domain. a From that failed nobody, the reader's own address included. A refused claim is held, not dropped: filed unapplied with a reason, shown on the message's card, and applied when the reader says so. The rule is a judgment about a sender and the reader may know better. SenderAuthentication reads the Authentication-Results header and checks nothing itself; its job is deciding whose header to believe, since a sender can write one too. mx.google.com for Gmail, the single nameless header for Microsoft, and for IMAP a server on the same registrable domain as the account's IMAP host — learning the name from the mailbox or asking for it in a setting were both considered and turned down in the issue. Mail in Spam or the bin is no longer read for events at all, and a message taken back out is read then. What it does not catch: on an account whose server reports nothing, a forged From is indistinguishable from the real sender, and holding every such update would hold every update a self-hosted mailbox receives. There the address is taken at its word. The handbook says so. Two things turned up on the way. A copy of a meeting on a second calendar has no source links of its own, so the first version of the rule held every update on the copy while its sibling took it — caught by SharedEventUpdateTest, and the senders are now asked across every copy under the UID. And there is no HTTP-level test of the new "apply" action; its logic is covered through the reconciler. One new nullable column, event_source_link.hold_reason. Nothing already on a calendar changes. Refs #34 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5.5
|
09600802db |
Let a message be written once and inserted as a template
Settings → Templates is a new section: a tree on the left, one editor
beside it. The tree opens with one folder per mail account, and those
are not stored — an account folder is the account, drawn as a folder,
so the mirror cannot drift. Folders the user makes go inside an
account, inside one another or beside the accounts. Removing an
account or a folder keeps the templates in it and moves them up.
A template holds variables: the recipient's first name, name and
address, the sender's name and address, a signature, and dates with an
offset and a format of their own. They are stored as text tokens in
the body, because the body goes through the inbound-mail sanitiser and
a marked element would not survive it; the editor draws them as chips
and turns them back before it posts.
The compose window gets a button beside the signature one. Dates,
sender and signature are filled in by the server when a template is
inserted. The recipient is not: a template is usually picked before
anyone is addressed, so recipient variables arrive as markers and the
window fills them when To has somebody in it, and asks before sending
with one still open. A template's signature replaces the window's
rather than joining it, and one the template names is pinned so a
change of From leaves it alone.
TemplateTokens the notation, read the same way in PHP and in
assets/compose/template_variables.js
TemplateRenderer what each variable becomes, and who fills what
TemplateLibrary the tree, and the only writer of where a
template or folder sits
One bug fell out of running it in a browser. Chromium will not put a
caret between a non-editable element and the edge of its block, so a
template ending in a variable could not be given its full stop — the
keystrokes were dropped. The editor keeps a zero-width space beside
each chip and strips them before saving.
Two new tables, mail_template and template_folder. Web only so far:
nothing is published over JMAP yet.
Refs #26
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Paul LütznerandClaude Opus 5.5
|
ec938bca99 |
Count the sessions a public demo hands out, without keeping an address
Admin → Demo visitors, in demo mode only: sessions started in the last hour, 24 hours, 7 days and 30 days, the different visitors behind them, the total and how many have a mailbox now. The users cannot be counted because the reaper deletes them, so /demo writes a row of its own. That row holds a keyed hash of the address with its last part cut off (a /24 or a /48), never the address, and the reaper removes the hash after 30 days. The demo's privacy notice describes it. Outside demo mode the section is not in the nav, its route answers 404 and nothing is recorded; the table exists and stays empty. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5.5
|
3a6f4db0f1 |
Measure mail arrival from when the provider accepted it, and show the slowest
Admin → Performance measured "how late mail arrives" from the date a message carries, which is the sender's clock. A newsletter dated half an hour before it was sent, or mail held back by greylisting, was reported as plMail being that late. The delay is now taken from the newest Received: header. Each message is stamped as it is first stored with when the provider accepted it, what started the sync that fetched it (push, IDLE, schedule, sync button) and where it was filed. Inbox mail and the rest are counted apart, since providers only announce the inbox, and the slowest messages of the period are listed in a section that starts collapsed. An upgrade task gives the last week of mail its accept time from the stored headers. What fetched older mail cannot be recovered and shows as a dash. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5.5
|
2dfc9b3b9b |
Give the admin panel a page for how long things take
The line under the hold switch in Admin → AI said "typically 6 s, 95%
within 13 s": enough to see that one message took twice as long as the
rest, and nothing about why. System says whether plMail is working;
nothing said how long anything took.
Admin → Performance, directly below System:
How late mail arrives per account, provider-received to stored.
Seconds is push working; minutes is the
quarter-hourly poll, which "works" and so
showed up nowhere. Imports and plMail's own
sent copies are not arrivals.
What each worker has the queue numbers by the process that owns
waiting them, a worker's queues in the order it
looks at them. Live; the transport keeps no
history.
New mail waiting for one row per held message, its wait split
the assistant three ways: queued, the model call, and how
much of the call was the model loading.
Model host moved from Admin → AI, unchanged.
To split a wait, the three parts have to be recorded with the message:
ai_queue_ms, ai_call_ms, ai_load_ms. ai_call_metric has the same
numbers and no idea which message they were for, and its timestamps are
the database's clock where the message's are PHP's, so a join by time
is not one to rely on. AiAssistant::ask() returns the whole result for
that; chat() is ask()?->content.
The model host's card and the search index backfill were one fragment
polled by one endpoint. They are about different things — a measurement
and a job with Start and Pause on it — so the fragment renders either
half, each page names the one it wants, and a caller that names neither
gets both, as before.
BackgroundProcesses::consumers() is the worker-to-queue map, read by
all() to start the processes and by this page to label them. They were
two facts that had to agree.
Rows name accounts. Nothing about any mail's content is shown.
Not looked at in a browser; the E2E card-inset spec covers the section
and has been given its frame.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Paul LütznerandClaude Opus 5.5
|
88b1744385 |
Leave the ingest worker to fetch mail, and nothing else
Adding an account put new mail behind hours of follow-up work. What
plMail does to a message once it has it — a model call where the
assistant sorts, event extraction, insight cards, receipts — was queued
on `ingest`, beside the syncs, for every message of the import, oldest
first. One consumer serves that queue. 988 envelopes pending and the
oldest an hour old was a mailbox being asked about newsletters from
2019 while mail on the server waited to be fetched.
`ingest` is now what talks to a mail provider. The rest moved:
enrich_live worker-live follow-up work for mail that has just
arrived on an account whose import is
over; never behind an import
enrich worker-enrich the same for an import's recent mail,
and search indexing
enrich_backlog worker-enrich old mail for the assistant, consumed
only when `enrich` is empty
release worker-release ends a hold (below); nothing else
Which queue is decided per batch, not per message class: the pipeline
asks InitialImportState once, carries the answer on PostIngestResult,
and EnrichmentRouter turns it into a stamp.
Only mail from the last 30 days (ENRICH_RECENT_DAYS) gets follow-up
work on arrival. Older mail is stored, threaded and filed by the rules.
The assistant is asked about it afterwards — newest first, a bounded
handful per run, and not until every account of that person has
finished importing (app:ai:classify-backlog, ClassificationCatchUp).
For the other steps `app:backfill` remains the way to read history.
ExtractInsightsStep and ProposeEventsStep ran their work inside the
sync and argued for it; both dispatch now.
New mail no longer appears in one tab and moves to another. For
somebody sorting by assistant the rules' category is a guess about to
be replaced, so where the verdict could still change it the message is
held: stamped before threading, its new thread given no category, which
every tab query already reads as "in no tab". The answer releases it.
Failing that a timed release does, after AiSettings::holdMaxSeconds —
60 by default — and the mail is shown where the rules put it; the model
call carries on and moves the mail when it answers. A message is asked
about once. Admin → AI has the switch, the ceiling, and the measured
delay over the last week beside them.
Three things about the release that were each wrong once first:
- It ran on the live worker, so it could not fire while that worker
was inside the call that was taking too long. It was moved to
export, which mixed mail appearing with mail leaving. It has a
queue and a worker of its own.
- The call was cut off at the ceiling and the message asked about
again later. The hold and the question are separate; only the hold
has a deadline.
- A late verdict was stored and moved nothing. It moves the mail.
Both workers still end at one model host, so bulk classification looks
at the live queue before each call and stands aside (LiveMailPriority).
One migration: two columns on `message`, two on `ai_settings`.
Not verified against a real import or a live model host: the tests pin
the queues, the hold and the release against in-memory transports and a
mocked host. Known and left: IMAP still imports oldest first; a reply
into an existing thread cannot be hidden; a Gmail account counts as
importing for up to an hour after its last message lands.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Paul LütznerandClaude Opus 5.5
|
f0380c47fa |
Make the wax seal the icon an account starts on
The default icon was the pl mark. It is the wax seal now, in its own
design: the mark pressed into wax is still recognisably plMail, and it
is a picture rather than two letters, which is what an icon on a
catalogue page or a home screen has to be.
Three things move together:
LogoMotif::DEFAULT the seal, for new accounts and for pages
with nobody signed in
Appearance::$logoOriginal true by default, so the seal arrives the
way a picked motif does and not in the
theme's colourway
public/icons/ favicon.svg and the four PNGs redrawn as
the seal on its tile
A migration moves the column defaults and every account that never
chose an icon: the pl mark, still in berry, still linked to the theme.
That cannot tell an untouched account from one that deliberately stayed
on the mark; those get the seal too and can pick the mark again.
The static files are drawn by hand from the motif's geometry, so a test
now holds favicon.svg to the default icon's own colours.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Paul LütznerandClaude Opus 5.5
|
6edc6c324c |
fix(deploy): the cache table exists before any worker reads it
A fresh install logged a warning about once a second from the worker, until an administrator happened to press "Restart workers": Failed to fetch key "workers.restart_requested_timestamp": ... relation "cache_items" does not exist The restart signal and the notice throttle live in cache pools on Symfony's DBAL adapter, which creates cache_items on its first write. Every Messenger consumer reads the signal on every loop, long before anything writes it, and each failed read is logged. The migrations leave the table out on purpose, because the adapter owns its DDL. app:db:migrate now has each DBAL adapter create its table when it is missing, after the migrations and inside the advisory lock, so php and the worker cannot race each other to it. The adapter still owns the DDL: it names its table through configureSchema() and creates it with createTable(). Best effort, like the statement statistics: a failure is a warning, not a container that will not start. Existing installs get the table on their next boot. The adapters are collected by class in a compiler pass rather than injected by pool id. In dev and test the profiler wraps every pool in a TraceableAdapter, which has no createTable(). It is the lookup DoctrineBundle uses for its schema listener, and it covers any DBAL pool added later. Checked on two fresh compose.yaml + compose.demo.yaml stacks: 95 warnings in 75 seconds without this, none with it, and Restart workers still reaches every consumer through the table created at boot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5.5
|
4cbdb4f511 |
feat(admin): a version checker, with a channel to follow
Admin → Updates: an administrator picks which builds count as an update (Releases, the `latest` image; Main, the `main` image; or Off), and every hour app:updates:check asks whether that channel has a newer one. Until somebody picks, an installation follows the tag its own build came from, so a `main` image follows main and an unbuilt checkout asks nothing of anyone. The registry decides what exists: the channel's image is read the way `docker pull` finds it (OCI distribution API, anonymous token from the 401's challenge, the index entry for this architecture, the config's labels), so a commit that builds no image is never announced. GitHub's compare decides which way two builds differ, so a main build following releases is not told to go back to an older release, and its commit list is the page's "what's new". An image from anywhere but GitHub falls back to what the version numbers alone can say. Each new build is announced once: a push to every administrator's registered devices (a new UpdateAvailable payload the service worker shows), a badge beside the build chip in the admin header, and a dot on the nav entry. A check that gets no answer keeps the last one and says why, and the hourly command exits 0 for it: an installation behind a firewall is not a fault. Images are stamped with where they are published (APP_IMAGE), so a fork that publishes its own checks its own. The channel travels in config backups; what the last check found does not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5.5
|
24d7bdfc07 |
feat(appearance): the logo is an icon, then its paint
Appearance gains logoMotif (default pl) and logoOriginal (default false), with Version20260923223000 adding both columns so every account keeps the mark it has. effectiveLogoPaint() says what the chosen icon wears: for the pl mark it is effectiveLogoStyle(), unchanged; a motif wears its own design while logoOriginal holds, then the theme's namesake colourway while linked and the chosen one when not. A linked motif on a classic theme falls back to its original rather than to Berry. Export, import and the settings save carry both fields; an unknown motif keeps the current one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5.5
|
cf7b971d0d |
perf(db): index message_thread for badges, list order and snooze sweep
The dozen sidebar/list aggregates per page and the per-minute snooze sweep had only account_id to work with. Adds partial indexes for unread, starred and snoozed threads and (account_id, last_message_at DESC, id DESC) for the list ORDER BY. Applied against the test database in a rolled-back transaction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5.5
|
12842bb3be |
fix(imap): index unread rows per folder for the post-sync recount
Every folder sync ends with countUnseenForMailbox(), `mailbox_id = ? AND seen_at IS NULL`, which visited every row of the folder to test seen_at. A partial index on message(mailbox_id) WHERE seen_at IS NULL holds only the unread rows. Migration-only, like the table's other partial indexes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5.5
|
b9238ab019 |
fix(imap): stop re-probing genuine multi-folder copies every sync
repairRelocated() probed every row whose Message-ID was also filed in another folder. A message that genuinely lives in two folders survived every probe, collapse() returned 0, and it was back on the work list next sync, costing a round trip per copy per poll and filling the 100-row slice so real ghosts behind it waited. Confirmed copies now get message.copies_confirmed_at (new migration) and are left out of findMessageIdsAlsoFiledElsewhere() for 30 days. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5.5
|
07b6724e8c |
fix(imap): a message that fails to build is retried instead of lost
The catch around buildMessage() logged and moved on without holding the high-water mark, so lastSeenUid passed the failed UID and it was never fetched again. The clamp that did exist for empty fetches was per batch and got overwritten by the next batch's higher UIDs. Now the lowest skipped UID is tracked for the whole run, both failure paths hold the mark below it, and a per-mailbox counter (mailbox.failed_uid / failed_uid_attempts, new migration) lets one message that always fails go after five syncs so it cannot block the folder. A present but unparseable Date header made webklex throw while building the message, failing the fetch for the whole folder. The client now configures a fallback date (the epoch), and MessageSyncer::dateFromHeader() reads that, or any throw, as "no date" and uses the arrival time. Attachment parts are persisted only after the message is fully built; before, a second attachment that threw left the first part managed and pointing at an unpersisted message, and the batch flush failed on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5.5
|
63b236b134 |
fix(imap): a folder missing from one LIST no longer deletes its mail
MailboxSyncer removed every stored folder the server's LIST did not name, and message.mailbox_id cascades, so an empty or partial LIST, or a rename done in another client, deleted every message of those folders in one poll. Now an empty listing is refused outright, a listing that would leave out most of the stored folders is refused like VanishedMessageReconciler's bulk guard, and a folder that is really gone is only marked (mailbox.missing_since and missing_syncs, new migration) and skipped by the sync loop. It is removed only after three consecutive listings and 24 hours, and its messages go through MessageEraser so JMAP clients get a destroyed change for each. A new folder with the same special use or the same UIDVALIDITY as exactly one missing folder is treated as a rename: the row keeps its mail and takes the new path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5.5
|
7a7673c0e8 |
perf(jmap): Email/query pages in SQL instead of reading the mailbox
Every call read and sorted every matching message of the account for PHP to slice a page from, and received_at had no index. OFFSET/LIMIT now go into the query; a collapsed list is collapsed first with DISTINCT ON so position still counts threads; total is a separate COUNT, skipped when the client sends calculateTotal: false. New index on (account_id, received_at DESC NULLS LAST, id DESC) for the default sort. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5.5
|
bdcce85acb |
fix(jmap): state tokens follow commit order per account
The change log's identity value is the state token and is drawn at INSERT, so two writers of one account could commit out of order and a client holding the later token never saw the earlier change. A BEFORE INSERT trigger now takes a per-account advisory xact lock and draws the number after it. A trigger because Doctrine fires no event inside its flush transaction before the inserts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
5a620f4be7 |
Run one-time repairs once per installation, and repair the lost bodies
The previous commit stops bodies being filed as attachments. It does nothing for the mail already in the mailbox, which stays blank — and there was nowhere for that kind of work to live. WHY NOT THE FOUR PLACES IT COULD HAVE GONE. A migration runs exactly once on update and is the right home for schema, but this repair needs the attachment store and the body sanitizer, and a migration reaching into the container breaks the day one of them moves — halfway through an update, with the schema half applied. The entrypoint is downtime: it runs before the server accepts a request, so a walk over every message put there means an install that updates at eight in the evening is unreachable until it finishes. BackfillTaskInterface is deliberately manual, which is right for "regenerate the categories because the classifier changed" and wrong for damage nobody should read a changelog to discover. BackgroundJob is per-user, and this is not owned by a user. So: a ledger, in the same shape as Doctrine's own migration_versions and readable the same way when the application will not start. A task claims its row BEFORE running, so a container killed mid-walk leaves evidence rather than looking like one that never started; three attempts and it stops rather than taking the worker down every ten minutes forever. The runner re-reads the row afterwards because tasks clear the EntityManager between batches and the claimed instance is detached by then — writing through it would flush nothing and the task would run again for ever. Scheduled every ten minutes, and what runs every ten minutes is the QUESTION, not the work: one indexed lookup per registered task against a table with single-digit rows. What it buys is the gap between an update landing and the repair happening. The repair is the first task and implements both interfaces — automatic for everybody, and app:backfill misfiled-bodies for anyone who wants it now. It needs no mail server: persistAttachment() stored the bytes on the way past, so the body has been on disk the whole time it was missing from the screen. It does lose the declared charset, which MessagePart does not keep, so CharsetHelper works it out from the bytes — exact for UTF-8 and a reasoned guess otherwise, against a round trip per message to a server with nothing else to say. EXISTS rather than a join in the lookup, and not as a preference: a multipart/alternative can have both halves misfiled, and SELECT DISTINCT cannot run at all here — Message carries json columns and Postgres has no equality operator for json. The repository test caught that on its first run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
fb97a5a6b4 |
Recognise Google's weekly Testing expiry, and warn before the next one
A Gmail account that stops every few days is almost always one thing: Google expires refresh tokens issued by an OAuth app whose consent screen is still in "Testing" publishing status, after about a week. That is the state nearly every self-hosted install is in, because leaving it there is the documented way to avoid Google's verification review for scopes of Gmail's reach. So the account works, stops seven days later, gets reconnected, and stops again. Forever. Google gives no clue which case you are in. What arrives is `invalid_grant` — the same code it returns for a revoked consent, a changed password, and a token that aged out. Four causes, one error, and only one of them is fixed by changing a setting rather than by signing in again every week. THE ONLY THING THAT SEPARATES THEM IS THE INTERVAL, and nothing was recording it. Two columns now do: when the current grant was issued, and how many hours the previous one survived. The second is kept ACROSS the reconnect on purpose — it is the only evidence that survives one, and the moment somebody has just fixed it is the one moment they are looking at the account and could act on the cause. Two cards come out of it. A sign-in that died after about a week gets one naming that cause and the real fix instead of the generic "sign in again" for the fifth time. And once the pattern has been seen once, a warning appears a day or two BEFORE the next one runs out, while nothing is broken yet. The inference needs both halves and says so. Without the history it would nag every healthy Google account weekly; without the age it would fire the moment somebody reconnected, and a warning visible five days out of seven is furniture. The window is 4 to 9½ days: wide enough for a death noticed a day late on the next sync, narrow enough to exclude a grant that ran for three months and was then revoked — which is the claim that must never be made, and is pinned in WeeklyGrantExpiryTest in both directions. An account connected before this was recorded is told nothing at all. Verified on a running stack in all three states: the warning on a live six-day grant with a weekly history, the diagnostic card on a dead one, and the plain card with no mention of Testing on a grant that lasted ninety days. The last found a real bug — the check was wired into inspectAccount(), which the settings page never calls, so the card rendered nowhere. It also must not register as a CAUSE, or every calendar on the account would grey out and blame a card whose first sentence says mail is still arriving. The handbook said "publishing and going through verification is what removes that", which conflates two separate settings and is what makes people accept the weekly reconnect as the price of not being verified. Publishing status and verification status are different fields; an app published without verification works, with a warning screen the first time. Corrected in English and German, with a new section in features/health.md describing both cards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
d53e7c4963 |
Report browser errors to the admin panel, in a card of their own
A JavaScript fault was visible only to whoever had the console open, which on a self-hosted mailbox means nobody. Uncaught exceptions and rejected promises from plMail's own scripts now reach Admin → Logs, together with whatever the Content-Security-Policy blocked. Its own card rather than rows in the log, because it holds a different kind of thing. A server error is one event in one request; a broken line in a controller runs on every page load for every user. So a row is a distinct fault — message, file, line — with a count and a lifetime, and the same fault reported again increments it. Merged into the log, one bug would be four hundred rows and the log would be unreadable past it. THE FILTER IS THE FEATURE. Most of what window.onerror sees is not ours: the console screenshot that prompted this was an ad-attribution metrics script failing inside requestIdleCallback, injected by an extension. A panel that accepted that fills with other people's bugs and stops being read, which is the same as not having one. A report is kept only when it can be attributed to a script this server sent — an injected script has no URL (Chrome's `VM947`), and a cross-origin one arrives as the bare string `Script error.` with no file, line or message. The rule is applied in the browser and again in the recorder, because the browser half is part of the code being reported on. CSP violations come to the same card through /csp-report. That endpoint cannot be CSRF-protected and is public in security.yaml, because the browser generates the request itself: no token, no header we chose, and on a login page no session. The rate limiter is the bound. It is worth having because the full policy is enforced in production, so what arrives is what this installation actually refused to load in a real browser. The count means page loads, not executions — the client reports each fault once per page, capped at five. Measured rather than assumed: throwing the same error twice on one page records one sighting. Documented that way on both sides after the first version claimed the server would count what the client never sent. `column` is a reserved word in Postgres, so the field is `column_number`. The quoted DDL created the table happily and every insert into it was a syntax error — same trap User::$usr already avoids. Verified on a running stack by throwing real errors in a real browser: an error from a script on this origin recorded, an anonymous injected one refused, the card rendering both kinds grouped with counts. Its first catch was a false alarm, and a useful one to record. It reported the Mercure connection as blocked by `connect-src` — which was true, and was me browsing `localhost` while MERCURE_PUBLIC_URL names `127.0.0.1`. Different origins, so the app's own policy refused its own hub. Browsing the host the config actually names produces no violation at all. The compose comment calling that URL "same-origin" is correct on its own terms; it just stops being true if you reach the stack by another name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Claude
|
044aa01846 |
Reported mail is one feature: one list, one triage, one export you choose
Two kinds of report — a mail an extractor should have understood and did not, and a mail somebody found in the wrong tab — were two tables behind two panels with two export buttons. That split made sense to whoever wrote them and to nobody using them: an admin opening this section is asking "what is still mine to do", and that question does not have two answers. The tables stay apart, because the evidence really is different: a wrong tab is a disagreement between three classifiers, a missed insight is a mail body a parser could not read, and making them inherit would put a dozen always-null columns on each row and tie a schema change on one to the other for ever. What is unified is everything above them. ReportedMail carries the handful of fields a triage list is built from and the entity alongside for the fold and the export; ReportedMailProvider is the only place that knows there are two tables; the panel, the buttons and the file are written against the pair. Category reports gained handledAt, which is what makes them belong in the same list — a worklist you cannot cross anything off is not a worklist. Before this the only way to stop looking at one was to delete every one of them, taking the evidence for a rule away at the moment somebody might want to check it again. The export's scope becomes the selection. "Everything, or only the unhandled part" answers the question an admin has on their first visit and none of the ones they have afterwards; the one that comes up is "these six, the ones about the shop". So the list filters — by status, and by the problem itself, with the chips derived from the rows because for a wrong tab the problem IS a pair of categories and only this install knows which pairs it has produced — and the export takes what is ticked. Filtering never unticks anything: a row hidden by a chip is out of the selection while it is hidden and comes back ticked, so a selection built across two chips survives. An empty post still means everything, which is what a browser without JavaScript sends. Each record now carries its kind, every field its kind has, and the same one-line rendering the panel shows and the copy button copies. The document is plmail.reported-mail version 2 — a new name rather than a bumped version alone, so a consumer written against version 1 fails on the name instead of half-reading a file whose records it only understands some of. The row is a <div> and not a <label>, though a label around the whole row would give the checkbox a better hit target: a label may not contain a <form> or a <details> and both are there. Chromium happens not to forward label activation from a nested button, which means the invalid version works right up until the row is read by something that is not Chromium. Verified on a running stack: six reports of both kinds in one list, the status chip hiding the handled one, select-all writing five keys into the export form, a problem chip narrowing that to one without losing the other four, and the export returning scope:selection with the full record and its line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Claude
|
d42012cedf |
Report the category the tabs actually sort by, and the evidence behind it
The first real miscategorisation report read `filed:updates should:updates` about a conversation its owner was reading in Primary. Both halves of that were wrong in the same way: `filed` was `message.category`, and the mailbox lists threads and filters the tabs on `thread.category`. A thread takes one category from its newest message and a pin overrides it outright, so on any conversation with more than one message the two are free to differ — and it is the thread that decides which tab somebody is looking at. So the report described a column nobody can see, and the popover's "category" line did too. Both now read the thread, falling back to the message for one not yet threaded, and both say so when the two disagree: the panel explains that the conversation takes its category from its newest message, which is the difference between a panel that looks broken and one that is informative. The report row was also too thin to act on. It carried the four verdicts and none of the inputs they came from, so a line could be entirely true and still leave nobody able to say what to change. Added: the message's own category when its thread disagrees, whether a pin is holding it, whether the reader had asked for the provider to be overruled, whether the model was asked at all (`model:-` could not tell "never asked" from "asked, no answer"), whether there was any plain text for it to read, and List-Id's value — the one bulk header whose value is a rule-writing input rather than a tracking link. The card moves from Admin → AI into Reported mail, where somebody who has just pressed Report goes looking for it, and is grouped by `filed → should` rather than listed by date: twenty reports are rarely twenty problems, and the fix for one group is not the fix for another. Each group and each row can be selected, and the selection can be copied or downloaded as a text file — so whoever is changing one rule can take the reports about that rule and leave the rest. A report that agrees with where the mail already is now 409s rather than landing as a row saying nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
fabaf49a80 |
Let a reader report mail as filed in the wrong tab
Categorisation is decided by three things that disagree — a provider's labels, a header cascade, and a model — and whether the answer is any good is a judgement only the owner of the mailbox can make. Everything else in this feature can be measured against fixtures somebody invented. This is the one route by which real mail, wrongly filed, becomes something a rule or a prompt can be changed on the strength of. The control sits at the bottom of the header popover, where the three answers already are. A SELECT AND A BUTTON rather than five buttons: knowing a message was misfiled is a fraction as useful as knowing where it belonged — "promotions, not primary" is a rule somebody can act on, "wrong" is a shrug. The tab it is already in is not offered, because "it should be where it is" is not a report. It records the DECISION and its evidence, not the message: what each of the three said, what the rules matched on, which of them was deciding, which bulk headers were present, the sender and the subject. Not the body — the most sensitive part of a message and the least useful here, since most marketing mail has no plain-text part at all and the classifier sees "(no plain text part)", so the column would be empty for exactly the mail most often reported. Sender and subject ARE kept, and on a multi-user installation an administrator reading these sees them for other people's mail. That is a real cost, taken deliberately: a report saying only "something was wrong" is not worth the button, and it is why nothing is recorded without somebody pressing it. The reports collect in Admin → AI rather than a nav entry of their own, because that is where somebody ACTS on them — the prompt and the model are a card above. The product is the TEXT: one line per report, built in the entity rather than the template because a format assembled in Twig would change whenever the table's layout did. Copy takes the block; clearing is a deliberate press, since these are observations nothing regenerates and a pattern across twenty of them over two months is the whole value. No foreign key to `message`: a report outlives the mail it is about. Somebody deletes the message, the pattern it was evidence of does not go away. Verified on the stack: the control renders, a report returns 204, a hand-edited category is refused with 400, the line appears in the admin card, and clearing empties it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
ed4a483ac3 |
Let each person choose what sorts their mail into tabs
Two decisions, and they were both taken for everybody by a constant. WHAT DECIDES. The rules read headers — a mailing-list header, an unsubscribe link, a sender you have written to — and answer the same way every time for the same mail, on any installation, with nothing switched on. The assistant reads the message and decides what it is for: better at mail that does not announce itself, and occasionally confidently wrong. Which is preferable is genuinely a matter of taste, which is why it is now a setting rather than something argued about once. Choosing the assistant puts its verdict ABOVE the header cascade — which is what choosing it has to mean. Read only as a tie-break, a newsletter carrying List-Unsubscribe is Promotions whatever the model made of it. Below the correspondent rule, though, and that placement is the one to keep: "you have written to this person" is a fact about the reader rather than a guess about the mail, and a model above it files a colleague under Promotions on one bad guess — the only mistake in this cascade anybody would actually notice. WHETHER IT OUTRANKS GMAIL. A Gmail account arrives already sorted, and plMail has always treated those labels as authoritative — its own rules never even ran. That is the right default and it is also exactly what somebody who dislikes Google's sorting could not escape, on the account where they most wanted to. Per person, not per installation, for AiPreferences' reason: an administrator decides what is available and a reader decides what happens to their own mail. Its own embeddable rather than two more fields in the AI bag, because only half of it is about the assistant — the provider half works on an installation with no model at all, and folding it in would file a setting where nobody would look for it. ONE BEHAVIOUR CHANGE, NAMED RATHER THAN DISCOVERED. The model's verdict used to be consulted as a silent tie-break: no setting turned it on, none turned it off, and nothing on screen said it had happened. It is now what Assistant means, and Rules means what it says — so an installation with categorisation on loses that tie-break until somebody chooses. The alternative is a model quietly sorting mail for people who never asked, which is the arrangement this ends. Said in the migration, in the handbook, and in an upgrade note. Nothing is re-sorted. The category is recomputed from stored data, so a mailbox keeps what it has until `app:backfill category` runs — an operator's call, not a settings toggle's, and the card says so. Verified on the stack rather than asserted: the card renders, the assistant option is disabled with its reason when categorisation is off, and a posted change lands in the column. The four combinations of the two settings pick three different categories for one fixture, which is what makes them worth having. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
2bfc30ad8d |
Offer to summarise the whole conversation, and send a context window with it
The card now says when a summary was written from part of a thread, and the way out sits beside the notice that explains why it is there. num_ctx IS THE WHOLE MECHANISM. This client has never sent one — four docblocks across the codebase size their budgets around "OllamaClient sends no num_ctx, so the model's default decides what survives", and that default is 4096 tokens with everything past it dropped silently. So "send the whole conversation" without also asking for somewhere to put it would send more, have it truncated at a point nobody chose, and spend a much longer wait to produce no more than before. It is sent only when a caller asks, so every existing call keeps its shape. FULL IS STILL BOUNDED, and that is the honest part rather than a caveat. A context window is memory: Ollama allocates the KV cache for the whole of it whether the prompt fills it or not, so an unbounded "send everything" lets one absurd thread ask a modest host for an allocation it cannot make — and a failed allocation costs the reader the entire wait and produces nothing, where a clipped transcript produces a summary and a sentence saying it is partial. The ceiling is 96,000 characters against a 32,768-token window, twelve times the ordinary budget. Past it the run is full AND partial, and the button withdraws rather than inviting the same wait twice. The window is SIZED to the transcript rather than fixed, since it is reserved whole either way and a two-message thread should not cost what a hundred-message one does. Sized at 3.0 characters per token, deliberately below the 3.55 measured on real German business mail: this divides to get tokens, so erring low overestimates the need and asks for slightly more room than strictly required. Erring the other way would silently truncate inside the one feature whose entire purpose is not to. thread_summary gains full_context, and it cannot be derived like the rest of the card. "Is this thread too long to send whole" is a question about the thread and comes free from the transcript the freshness check already builds. "Did THIS summary see all of it" is a question about the row, and the thread cannot answer it — a thread that is still too long is still too long after somebody presses the button and gets a summary that did see everything. Without the column that press would appear to do nothing: new text, same notice, much longer wait. A press and not a setting, deliberately. It is a fair thing to spend once on a thread somebody cares about and a poor default for every summary in a mailbox, so the cost is written into the label and nothing is remembered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
ffe9457b42 |
Give the search model its query instruction and its own threshold
Search by meaning was answering "mails about food" with a page of login links
and newsletters, ranked confidently, with nothing anywhere saying it had gone
wrong. Two settings decide whether it works and plMail had neither.
THE QUERY INSTRUCTION. Embedding models are asymmetric: they are trained to be
told which side of a search they are looking at. An unlabelled query is treated
as one more document, and it lands nearest whatever is most boilerplate — in a
mailbox, login links and newsletters. Measured on qwen3-embedding:0.6b over
twelve German and English messages, five about food:
query non-food ranked above the best food mail
"mails about food" 5 (raw) 0 (instructed)
"essen" 7 (raw) 0 (instructed)
"restaurant reservation" 2 (raw) 0 (instructed)
"essen" — the least ambiguous word available — put every food mail below every
non-food mail. With the model's own instruction, perfect separation on all
three. It goes on the QUERY ONLY, so stored vectors are untouched and nothing
has to be re-indexed.
THE THRESHOLD. 0.55 was one constant for every installation, and cosine
similarity is not comparable between models — or even between the instructed
and uninstructed forms of one model. Best threshold by F1, same corpus:
qwen3-embedding:0.6b no instruction 0.20 F1 0.59 precision 0.42
qwen3-embedding:0.6b with instruction 0.42 F1 0.85 precision 1.00
nomic-embed-text no instruction 0.45 F1 0.74 precision 0.65
all-minilm no instruction 0.20 F1 0.71 precision 0.77
0.55 is above all of them. So it travels on SemanticSearch next to the model and
width it has to agree with, and it is a setting rather than a release.
Both are columns, and EmbeddingPreset holds the measured pair for each model the
release was tested against, with a line on what it is good and bad at. The model
field stays free text — a closed dropdown would go stale and refuse something
that works — and choosing a preset fills all three fields at once, because they
are one decision written in three boxes: any two from different models give a
search that is confidently wrong rather than visibly broken.
qwen3-embedding:0.6b becomes the documented default. Multilingual matters more
than any benchmark for a mailbox that is not all in one language, and it was the
only combination measured at precision 1.00.
Existing installations keep 0.55 and no instruction. Backfilling 0.42 would
apply Qwen's number to whatever model is configured, and on all-minilm (0.20)
that is twice as tight as it should be — a migration is the wrong place to
change what somebody's search returns. The panel now shows both values and sets
them on one click.
The instruction field is trim: false, which a test caught rather than a guess:
every working instruction ends in a separator, Qwen's is "…\nQuery: ", and
Symfony's default trim turned it into "Query:mails about food".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Paul LütznerandClaude Opus 5
|
0e7741b4de |
Let a Gmailified message reach the Gmail account that carries it
Google can fetch another mailbox into Gmail. With both connected here
every message arrives twice, and SyncGmailMessageBatchHandler already
does the right thing with that: it merges rather than skipping, so the
IMAP row keeps its location and its flags and gains what the Gmail copy
knows — the gmailId, the label ids, the translated labels. Its own
comment calls it "Gmailify dedup: merge, don't skip".
The result is a row that belongs to the IMAP account and carries a Gmail
identity, and every push to Gmail then asked the wrong account. It read
`accountOf($message)->isGmail()`, which for a merged row is the IMAP
account, so the answer was always no — and the message's usable gmailId
went unused with a Gmail account sitting beside it. Archive, trash,
restore, star, mark-read and every label change: applied locally, applied
on IMAP, never anywhere Google could see.
Reported as a mail dragged out of spam that came back. Hetzner obeyed;
Gmail was never told, and the next sync restored Gmail's opinion.
The account that CAN act is the carrier — the Gmail account whose sync
recognised the message — and nothing recorded which one it was. It also
cannot be re-derived in general: a gmailId is issued by one Gmail
mailbox and means nothing to another, so an install with two Google
accounts has no answer to reconstruct. So it is stored, by the one place
that knows, at the moment it stamps the id on.
message.gmail_carrier_account_id nullable, ON DELETE SET NULL —
disconnecting the Gmail account must
not take the IMAP mailbox's mail with
it, and a row that stops being pushed
to Gmail afterwards is correct.
enrichExisting() writes it, and only when the carrier
is somebody else: a native Gmail row's
own account is already the one that
can act.
gmailAccountFor() the message's own account when that is
a Gmail one, else the carrier — and a
carrier that has since been
re-authenticated as something else is
refused rather than used.
The backfill takes the unambiguous half on purpose. One Google account
on the owner means there is only one account the id could have come
from; two or more means a guess, so those are left null and filled the
next time the Gmail sync sees the message.
Both tests were run against the unfixed propagator first: the Gmailify
one fails with "actual size 0 matches expected size 1", which is the bug
exactly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Paul LütznerandClaude Opus 5
|
8e05c4f557 |
Let a conversation be dragged into a folder or onto a category tab
Filing mail meant a menu: "Label as" for a folder, and for a category
nothing at all — the inbox tabs were computed from headers and there was
no way to disagree with them. Both are now a drag.
The two ends of the gesture are in different parts of the document — the
rows are in the list frame, the folders are in a sidebar that is rendered
twice, and the tab strip is a third place — so mail--dnd listens on
<body>. It uses HTML5 drag rather than pointer events, unlike the
calendar's block-dragging: this gesture has a discrete destination, so
the platform's own auto-scroll, drag image, refusal cursor and Escape
are worth more than the live preview a pointer implementation buys. The
cost is touch, which is smaller than it sounds — on a phone the sidebar
is a closed drawer while the list is on screen.
A move is deliberately not "attach the destination label". Nobody drags
something onto a folder and expects to find it in both places, so
ThreadStatusUpdater::move() detaches the location labels the mail
carries and leaves the tags alone; which labels count as a location is
the one judgement in it, and getting it wrong in the generous direction
would strip the user's own labels off. The three system destinations
delegate to archive/trash/restore, which already own their provider
semantics.
A drop posts to the bulk endpoint the toolbar's buttons post to, and for
the reason that endpoint exists: a row-shaped response cannot fix the
pager or the empty state, and a private route for drags would have been
a second implementation of what happens to the list afterwards.
MessageThread category_pinned_at, so a category chosen by hand
survives the next message in the thread — and
a run of `app:backfill category`, which is the
same overwrite applied to a whole account at
once.
LabelRole acceptsMoves(), read by the sidebar to decide
which rows offer a drop and by the controller to
refuse the rest. An interface that hides a
control while the route stays open is a habit,
not a rule.
SidebarCounts roleLabelId(), because the system rows are
anchors to routes and a route is not a thing a
conversation can be moved into.
inbox.html.twig ghost tabs for the categories nothing is filed
under yet, revealed only during a drag. Without
them the only categories you could move mail
into were the ones it might already be in.
Three things the browser found that no unit test could. The list
animates in, so a box measured on arrival belongs to whichever row was
passing through that coordinate — the drag worked perfectly and moved a
different conversation. A multi-step mouse move dispatches one drag
event per step and the last one lags, so the highlight belonged to the
tab next door. And enter/leave cannot track a target made of several
children during a native drag, where relatedTarget is routinely null;
dragover alone can, because "somewhere else" is simply the next one
arriving.
Deliberately not done: an inbox where every message is Primary has no
tab strip, so it has nowhere to drop and cannot acquire a second
category. Rendering the strip there anyway, hidden until a drag, was
built and taken out — it makes the list jump under the pointer that has
just picked a row up. The honest fix is a preference, and that is a
different change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Paul LütznerandClaude Opus 5
|
d8f8acf29b |
feat(ai): tunable keep-alive per model, and a warm-up button
The first "Help me write" after a quiet spell shows nothing for tens of seconds while the Ollama host reads twenty gigabytes back off disk. How long it waits before evicting a model is OLLAMA_KEEP_ALIVE — an environment variable on a machine plMail can talk to but cannot configure. The API takes the same setting per request and a per-request value wins, so this is the only lever available from inside the app. Two nullable columns on ai_settings, one per model, because the two are opposite trades: the writing model is eighteen gigabytes and asked for occasionally, the embedding model a few hundred megabytes and asked for on every search and every indexed message. Both ship at 5m, which is Ollama's own default, so an upgrade changes nothing about how a host behaves — the boxes just say out loud what was already happening. -1 is recommended for the search model in the help text and the handbook rather than shipped, because plMail cannot see how much memory that host has. Empty means send no keep_alive at all and let the host decide, which is a real selectable state and not an unfinished one. The wire representation is the part that is easy to get wrong: Ollama parses a JSON string as a Go duration, and a Go duration needs a unit, so "-1" and "3600" as strings are a 400 on every model call from a setting that reads as correct. KeepAlive::forBody() is the only thing that puts the value in a body — int for the numeric spellings, string for the durations, null to omit the field entirely, since an explicit null is read as zero and unloads immediately. One helper adds it to all four bodies so no call site can get that wrong. Validated with the same expression the body is built from, refused rather than coerced, and the message lives in the validators domain where Symfony actually looks for it. Warm up now: POST /admin/ai/warmup issues the no-prompt /api/generate, times the round trip and reports it. PHP's 30s execution limit is raised for it — a cold load is not 30 seconds — and the upstream timeout sits below that so it gives up first. A host that is unreachable and a model that is not on the host are distinct, translated answers. Also carries both columns through the configuration backup, and adds the AI feature to the README, which had never mentioned it. Fixes a search test that asserted how fast the machine was: it gave the last-resort body pass a one-millisecond budget and one row to scan, so on quick hardware the pass finished in time and the test failed. It now provokes a genuine statement_timeout cancellation with pg_sleep, which exercises the same guard without a clock in the assertion. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
39dec36066 |
feat(calendar): the calendars themselves have a state too, and ifInState works
Calendar/get was the last method answering with the constant "fixed", left that way when CalendarEvent/changes landed on the argument that the gap was small: a user has a handful of calendars, Calendar/get returns all of them by default, so re-running it is cheap. True about cost, and beside the point. A state that never moves is a promise that nothing changed, and a client is entitled to believe it — so renaming a calendar left the old name in the sidebar until something unrelated forced a reload. calendar_change_log now records the collections as well. event_id and event_uid become nullable and their absence is what marks a row as being about the calendar itself — one discriminator, because an explicit target column beside a nullable id is a second thing to keep true and the two would eventually disagree in silence. A sibling table was the alternative and was rejected for costing a second sequence, a second repository and a second copy of the token arithmetic, which is the part with the subtle guards in it and therefore the last part worth having two of. Every reader filters, and that filter is load-bearing. sync-collection and CalendarEvent/changes take the rows with an event; Calendar/changes takes the rest. A reader that forgot would report a renamed calendar as a changed event and send a CalDAV client after an href built from a null UID, so both directions are asserted rather than only the happy one. App\Jmap\Calendar\CalendarState is deleted. Nothing answers with a constant any more, so the class explaining why anything did has no subject; the history it carried is folded into CalendarChangeLog and the listener, where someone meeting the design will actually be standing. ifInState is honoured rather than refused, and that reversal is the point rather than a side effect. The refusal was correct while the state was a constant — a guard that can never fail is worse than no guard, because a client relies on it — and with a real token the promise is one the server can keep. A client that read at one state and writes expecting it to still hold now gets stateMismatch instead of being allowed to overwrite what it never saw. Same protection If-Match gives a CalDAV client, from the same number. One bug fell out, and it is the event tombstone's mirror image. queueCollection deferred the calendar id to postFlush for every kind of change, which is right for an insert that has not got one yet and wrong for a delete: Doctrine clears the identifier once the row is gone, so the row recording a removal was the one silently dropped. Taken eagerly now wherever there is one. Also updates src/Jmap/README.md — 29 methods, and a note on why calendars count in their own log. No deployment change. One migration, relaxing two columns to nullable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
2790c13a82 |
feat(calendar): a client can ask what changed, and be told the truth
CalendarState was the string "fixed", and its docblock explained that this
was a confession rather than a design: events are written from the sync
engine, from extraction, from the web editor and from CalendarEvent/set,
and a log recording a quarter of the writes "is not a weaker version of
it, it is a lie with a number on it". So canCalculateChanges was false,
there was no CalendarEvent/changes, and every client re-ran its query.
calendar_change_log closes that. The rows are written by a Doctrine
listener rather than by a recorder the writers call, and that is the whole
point — CalendarEvent exposes public mutable properties, so fields move
wherever somebody assigns one and lets the flush carry it. An RSVP in
InviteResponder, a SEQUENCE bump in EventReconciler, the recurrence flags
in RecurrenceMaterialiser: none pass through CalendarEventWriter, so a
recorder wired into write() would have missed all three and every deletion
besides. Doctrine already knows what changed; asking it is the only way to
be complete without asking every future caller to remember.
CalendarChangeLog the log. Its own table rather than new
JmapObjectType cases on jmap_change_log, which
CalendarState suggested: that table is keyed by
account_id, and CalendarAccountResolver says a
user can delete their last account and keep a
calendar. CalDAV also counts per collection, and
there is nowhere to put the collection.
the listener onFlush collects, postFlush writes. An ignore
list, not an allow list, so a field added later
defaults to being logged — over-recording costs a
wasted conditional GET, under-recording is the
silent failure.
CalendarChangeReader tokens and deltas, protocol-neutral, in both
scopes: per user for JMAP, per calendar for the
CalDAV server this is the groundwork for.
Two bugs fell out while building it.
The first was mine and is why postFlush writes SQL rather than persisting
entities. Flushing from inside postFlush re-runs the whole unit of work,
and the second pass sees what the first left behind: EventDismisser
removes an event while an EventSourceLink still points at it, which is
legal once and throws "a new entity was found through the relationship"
twice. It broke every calendar deletion in the suite and five demo-flow
tests. A plain INSERT stays out of the UnitOfWork entirely, so this
listener cannot make somebody else's flush fail.
The second is subtler: a flush that throws never reaches postFlush, so the
queued rows survived to be written by the next flush that succeeded —
logging changes the database had rolled back. onFlush clears the queue as
it starts. A worker whose EntityManager is reset and carries on is exactly
where that would have bitten.
CalendarEvent/get, /query and /set now return real sequences. Calendar/get
still does not: the log records events, not the collections holding them,
so a rename or a recolour moves nothing. That is a smaller gap than it
sounds — a user has a handful of calendars, Calendar/get returns all of
them by default, and CalDAV discovers the collection set with a PROPFIND
that holds no token. CalendarState now says so instead of describing a
world that no longer exists.
Also marks config/reference.php optional in phpstan.dist.neon. It is
gitignored and generated, every sibling entry in that exclude list already
carries (?), and without it PHPStan refuses to start on a clean checkout.
No deployment change. One new table, created on boot.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Paul LütznerandClaude Opus 5
|
e7f4513574 |
fix(backup): the config backup had stopped carrying what it promises
A restore looked like it worked. That is the whole shape of this: nothing threw, the review page listed a clean plan, and the parts that had gone missing produced no row to warn about, because a section that does not exist has nothing to say. ── A removed user came back to life ──────────────────────────────────────── The restorer's own docblock and the handbook both say a soft-deleted user counts as existing, "because they hold the email address against a unique index". They had stopped holding it. Removal frees the address on purpose — the admin delete action rewrites it to `deleted-<id>@invalid` — so existingId() answered null for every removed user, the plan called them Absent, and the restore created them afresh and LIVE: old password hash, TOTP secret, recovery codes, every mailbox credential. Listed as an ordinary creation, indistinguishable from a new colleague. The handbook steers operators straight into it. The rewrite destroys the only copy of the address, so nothing could match it back; `email_at_deletion` is that copy, written at the same moment. Rows tombstoned before the migration stay unrecognisable, which is stated and held by a test rather than papered over. The regression test that was supposed to catch this passed throughout, because it wrote `deleted_at` and left the email in place — a row the application cannot produce. It now tombstones the way the delete action does. Its sibling: a violated unique index escaped as an uncaught 500 with no message. api_token.token_hash is unique across the WHOLE install and a soft delete cascades nothing, so a removed user's tokens still hold those values. Translated into a stated failure with a sentence attached. Not demoted like other failed writes — a violated constraint aborts the transaction, so there is no "rest" to let through, and stopping with a reason is the honest shape. ── Four things a person configured never reached the file ────────────────── The assistant configuration in full: model host, both models, the reverse-proxy token that exists in plaintext nowhere else, the feature switches, and every prompt an administrator rewrote. The per-user half had been carried since the day it shipped; the instance-level half it depends on had not, so a restore returned everyone's preferences pointing at a host that was gone. Account signatures and read-receipt defaults — and on the instance this was found on, `compose.signature` was the ONLY account setting in use, so every export it produced held an empty settings bag for every account. The per-alias overrides key themselves by alias row id, which names somebody else's alias on the target, so they now travel inside their alias and are rewritten under the new id after the flush, the way `calendar.target_id` already was. Five user settings, the chosen log level, and the OAuth scopes that describe what the provider granted to the refresh token the file already carries. ── Why none of it was caught, which is the part worth keeping ────────────── Every guard pointed the wrong way. The env test builds its expectation by calling the constant under test. The users test is fourteen assertArrayNotHasKey calls — it fails if somebody ADDS a field, never if one goes missing. The version assertion compares a constant against its own echo. A backup's characteristic defect is omission, and nothing could fail in that direction. Two tests now compare the backup against the application rather than against itself, and fail closed: every User and Account SETTING_ constant must be carried or excluded on the record with its reason, and the database section must carry every admin-configured table. The second would have gone red the day AiSettings landed. A round-trip test pins the set of things a restore drops to exactly two, each deliberate, so the set can only change on purpose. ── Also ─────────────────────────────────────────────────────────────────── The KDF had a ceiling and no floor, so `memlimit: 1` passed every check and took a SodiumException out onto a blank error page — through the anonymous /install/restore among others. Both memzero paths now run under try/finally, the abnormal case they exist for. App passwords keep the date they were created rather than the date they were restored. And the admin handbook said the file contains "no user accounts". It contains every one, with their secrets — published in two languages. Corrected, along with the inventories, which now list what actually travels. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
755f3cf257 |
feat(users): an administrator can run the whole life of an account, and you can change your own password
Managing users was reachable and worked — list, search, create, promote, demote, remove — but three things around it did not, and one of them was load-bearing. ── You could not change your own password ────────────────────────────────── Nowhere. Not in Settings, not by mail, not at all; only an administrator setting one for you. Settings → Security now leads with a Password card that asks for the current password (an unmapped UserPassword constraint, so it is checked against the signed-in token rather than against anything the form was handed) and takes the new one twice. The route carries no id and the form has no data_class, so it structurally cannot become the admin form by accident. The floor those passwords are held to was three literal 12s in three files that happened to agree. It is now User::PASSWORD_MIN_LENGTH, read by all three. ── An account could only be removed, never switched off ──────────────────── Nothing sat between "leave it alone" and "destroy it", so the answer to "this person has left, keep their mail" was to delete them. There is a deactivated_at now, and a UserChecker registered on BOTH firewalls — the JMAP one included, because an app password outlives the browser session it was minted from and a check only on `main` would shut somebody out of the web UI while every mail client they had already connected kept syncing. The refusal is in checkPostAuth rather than checkPreAuth on purpose. checkPreAuth runs before the password is verified, so refusing there would answer "is this address a suspended account here?" to anyone who types an address — the question hide_user_not_found exists to keep unanswerable. After verification, the only person told is the one who could otherwise have signed in. Symfony runs user checkers at authentication only, so a session already open would have carried on until it happened to end. UserProvider::refreshUser() refuses the same two states on every request. Suspension also travels through config backup, which would otherwise have handed a suspended account its sign-in back on restore. ── Two smaller lies ──────────────────────────────────────────────────────── A search under three characters silently returned the unfiltered list, which a QueryBuilder has no way to complain about, so the floor was dropped rather than kept as something that could only mislead. And an admin declining to demote themselves got a success toast for a change that never happened; the refusal now says so, while the rest of the form still saves. ── Restoring a removed user is deliberately absent ───────────────────────── Removal overwrites the address, the name and the hash specifically to free the unique address for reuse, so a restore would have to invent all three and the original address may by then belong to somebody else. Doing it properly means snapshot columns and a guided flow that asks for a fresh address — a different change, which alters what removal means. Documented as absent on the screen an administrator would look for the button on, because switching an account off is the answer to the question people are actually asking. Also fixed en route: the modal form theme dropped help_translation_parameters, so the first field to use one rendered "at least %count% characters" on screen. And GermanIsInformalTest only ever globbed messages.de.yaml, leaving validators and the new security catalogue — exactly where a translator reaches for "Sie" — outside the test that exists to catch it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
5f8ae26be4 |
feat(ai): the system prompts are the administrator's, and summaries follow them
Seven prompts were constants, so tuning one meant a deploy. Admin -> AI now holds all of them, with the shipped wording as the placeholder and a way back to it. EMPTY = NULL = SHIPPED, and that one rule carries the whole design. Reset is a DELETION rather than a copy, so a later release can still improve the wording instead of every installation being pinned to whatever shipped the day it first saved. It is also why the language rule can be reworded but not removed: there is no special case, an empty box simply is not an override. THE HAND-BUMPED VERSION HAD TO GO thread_summary carried PROMPT_VERSION, an integer whose docblock said "bumped whenever SYSTEM_PROMPT changes". That cannot survive an editable prompt: an administrator tunes it, nobody bumps anything, and every stored summary keeps the old prompt's output for ever while the page calls it fresh. It is now a digest of the EFFECTIVE prompt, which is the same move the transcript hash already made — edit a prompt, edit the language rule, clear either, or take an upgrade that improves the shipped text, and each is a different string on the wire, so a different digest, so the stale rows stop matching by construction. WritingTask::systemPrompt() is deleted rather than kept. A method by that name returning something that is not the system prompt on any installation that edited one is precisely the drift this repository keeps collapsing. The cap is 1200 and not AiPreferences' 600, because the summary prompt we ship is 587 characters: a cap the shipped text already fills only ever bites the person trying to improve it. Verified against the real host — an override changed the answer and clearing the box changed it back. Worth recording the negative result too: rewriting the language rule to demand English did NOT make this model answer a German mail in English. The rule is a nudge it already agrees with; the bug it was written for was the ABSENCE of any rule, which is now unreachable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
661d3bbf25 |
feat(ai): summarise a conversation, when somebody asks
On demand, and that is the design rather than a first step. A summary spends the 20.3 GiB writing model for about a minute cold, on one GPU already shared with the indexer and the composer. Summarising every arriving thread would spend days of it on conversations nobody opens. FRESHNESS IS DERIVED, NOT MAINTAINED The stored row carries a SHA-256 of the exact transcript that was sent, plus the model tag and a prompt version, and freshness is that hash matching what the thread would send today. Nothing has to remember to invalidate anything. lastMessageAt was the obvious key and is the wrong one: it only ever moves FORWARD, so a deleted message leaves a summary describing mail that is no longer there, silently and for ever. The hash also makes a prompt change or a budget change invalidate every stored summary by itself, which is correct and free. CASCADE, following message_embedding rather than MailInsight. The two precedents point opposite ways and both are right — an insight outlives its message because "a deleted shipping confirmation does not un-ship the parcel", an embedding does not because it describes one. A summary describes a thread. THE BUDGET IS MEASURED, NOT PICKED 8000 characters, against two ceilings and the smaller one is silent. The visible one is the wait before the first token — load plus prompt eval, about 42s at this budget, a third of the client's idle timeout. The quiet one is the context window: OllamaClient sends no num_ctx and Ollama's default is 4096 tokens, so 12000 characters sits exactly on the line, and an install running the default would drop the HEAD of the transcript — the part the reader most wants and the part the transcript is assembled forwards to protect. No error, no notice. The fourth AiFeature case landed LAST, after every arm that reads it. Eight exhaustive matches key on that enum and none of them fails at compile time; AiPermissions::anyAdminEnabled() loops the cases and the settings nav calls it on every settings page, so the case arriving early would have taken the whole settings area down rather than the AI section. Verified against the real host, not a mock: a German thread returns a German summary, a reopened thread renders the stored row with no new model call, a reply greys it and offers to run again, and changing the chat model hides it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
2348b75621 |
feat(db): pgvector as an opt-in image, built without changing libc
The original rejection of pgvector was tested rather than reasoned, and it stands: pgvector/pgvector:pg18 means musl to glibc on an existing data volume, and 44 collation-dependent indexes — 18 of them UNIQUE — are then walked with the wrong comparator, silently, because musl records no collation version so Postgres's own mismatch check never fires. So the extension is compiled INTO the alpine image instead. libc never changes, and there is nothing to rebuild. Verified end to end rather than argued: an existing volume migrated on stock alpine, stopped, restarted on the pgvector image, then bt_index_check(heapallindexed => true) across 175 btree indexes — zero errors. Two build flags carry more weight than they look. OPTFLAGS="" because pgvector's Makefile defaults to -march=native, which bakes the build host's instruction set into vector.so and gives SIGILL mid-query on a host without it, with an empty log. with_llvm=no because a missing .bc file makes the JIT skip inlining silently, and inlining does not apply to index scans anyway. Pinned at v0.8.6, not the earliest that works. v0.8.0 does not compile against Postgres 18 at all (vacuum_delay_point gained a parameter); v0.8.1 is the first that does. But 0.8.3 fixed possible index corruption with HNSW vacuuming and 0.8.4 fixed an unrepaired HNSW graph, and message_embedding is ON DELETE CASCADE from message on a system that deletes mail in bulk — so vacuum is a hot path here and shipping the earliest that merely compiles would ship two known corruption bugs. THE HANDBOOK'S 24x WAS WRONG AND IS CORRECTED TO 1.75x. The SIMD dot product is microseconds; what dominates is that embedding is real[], so every row pays a 4KB detoast and a fresh real[] to vector conversion. 74,000 rows: 7.70s becomes 4.51s. NO INDEX IS CREATED, and that is the finding rather than an omission. An HNSW index answers ORDER BY ... LIMIT k and the search asks distance <= 0.45, which no ANN index serves. The expression form needed to build one at all is (embedding::vector(1024)), which Postgres treats as a DIFFERENT expression from the inlined function's (embedding::vector) — so the index exists, the planner ignores it, and nothing says so. And it costs 7m40s and 578MB to build at the maintenance_work_mem this stack ships, behind the one advisory lock every entrypoint waits on: an eight-minute dead stack, not a slow migration. The measured route to an index is written down in ai-assist.md rather than half taken. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
037219e307 |
perf(search): compute the vector distance once, on candidates only
Semantic search measured 12.9s mean and 47s max in production, and it was getting worse as indexing progressed — 61.9% indexed then, projecting to ~19.6s at 100%. The arithmetic was exact: plmail_embed_distance costs 0.132ms a call at 1024 dimensions, and one search made about 92,000 of them. Two faults multiplied. The semantic ARM was correctly capped at SEMANTIC_CANDIDATES. The outer `LEFT JOIN message_embedding sem` was not capped at all, so the rank expression scored every embedded message the query matched — including rows only a keyword had found. And it scored them TWICE. The comment above $semanticRank said Postgres matches identical aggregate calls and evaluates the argument once per row, which is true, and the code carefully built one string to get it. DBAL expands the same named :queryVector into a separate positional parameter at each occurrence, so what reached the server were two textually different expressions. Production's own pg_stat_statements showed it: $7(...$3 AS real[]) beside $10(...$4 AS real[]). Both go away by construction rather than by care. The distance moves into a materialised CTE over the candidates, the arm reads from it, the outer query joins it, and plmail_embed_distance and :queryVector each now appear exactly once in the whole statement. Measured on 20,000 rows: 12,616 calls and 3,781ms become 2,000 calls and 602ms. MATERIALIZED is load-bearing, not decoration — without it the planner inlines the CTE at both references and 3.6x of that goes back. Two CTEs rather than one for a neighbouring reason: with the threshold applied inside a derived table the subquery is flattened, similarity lands in both the filter and the target list, and every row that PASSES is scored twice. The function's cost is corrected too, and derived rather than guessed: calibrating two index-only scans against their planner costs gives ~1.5us per cost unit, so 132us is about 33,000 of them. Rounded up to 50,000, because underestimating is the failure being fixed. Only the plpgsql body — the pgvector body inlines before costing. The cheap pass now runs under a statement_timeout and drops the vector for the rest of the search when it expires, so a pathological plan is paid once rather than up to four times. ONE BEHAVIOUR CHANGE, deliberate. Ranking moves for 3,966 of 5,308 rows in the benchmark, every one a keyword row: the old unbounded join had no threshold and no recency bound, so GREATEST(ts_rank, 1 - distance) took the vector's number on rows the vector had never found. 80.5% of keyword rows were ranked by a below-threshold similarity averaging 0.308 against a true ts_rank of 0.067. All 308 semantic-only rows keep their rank exactly, and EXCEPT ALL in both directions on (thread_id, semantic_only) is empty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
cdbe2d05bb |
feat(ai): let each person configure the assistant for themselves
The administrator decides what an installation offers; this decides what one person uses of it. Settings → Assistant: three feature switches, two notes the model is given before it writes, and how much of a conversation it reads. THE CEILING AND THE FLOOR AiSettings stays a ceiling and no user value can get past a false in it. The AND is written in exactly one place, AiPermissions, and every one of the four things in src/ that builds content for a model now takes the user in its SIGNATURE — so it is not possible to build a prompt without having named whose mail it is. A worker that cannot work out whose mail it is holding REFUSES: two of them run unattended over somebody's correspondence, and "we could not tell" is not a reason to send it to a model. That reaches into the transport. EmbedMessagesMessage now carries the user id, which for one deployment window means already-queued batches are refused and picked up by the nightly sweep instead. HOW MUCH IT READS An enum, not a switch: Nothing / The message / The conversation. Whole-thread is new capability rather than a restriction — the model stops answering a conversation it has only seen the last turn of. An oversized thread loses its OLDEST turns and always keeps the message being replied to. There is no privacy control here and the copy does not pretend otherwise. The model runs on the operator's own machine; what more context costs is time, and the hint says so. A "do not send the subject" switch was designed and dropped for the same reason — its only honest label would have been "make my search worse". WHAT COULD NOT BE ENFORCED, SAID PLAINLY A persona can talk a model out of the app's own rules; models have no reliable precedence inside one system message. So the hint says "Preferences, not instructions — the app's own rules come first", which is a statement of intent rather than a promise. What IS mechanical: placement, a length clamped on write AND re-clamped on read (hooks are skipped on hydration), a blast radius of writing help alone, and provenance from the entity rather than the POST body. Switching search off stops indexing; it does not forget. The copy says stops. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
377d6026b8 |
feat(ai): make the language model observable, and the composer honest
"I click the AI button in the composer and nothing happens" turned out not to be about the model at all. #editor() queried data-compose--toolbar-target while the composer renders data-compose--compose-toolbar-target, so run() returned at its first guard: no request, no status, no error. The feature had never worked once. #subject() was wrong the same way and quieter — every prompt went out with an empty subject. And the status element lived INSIDE the dropdown that the same click closes, so even once working it would have written into a hidden subtree. The rest of this is what makes the next such thing findable. RECORDING One row per model call, tagged by the workload that caused it, and nothing about what was in it — no prompt, no completion, no message id, no subject, no address. error_kind is a category from a closed set for the same reason: an HTTP client's exception message routinely quotes the request body back, and here that is somebody's mail. OllamaClient was discarding Ollama's whole timing block; it now returns it, and AiAssistant — the only place that knows both the feature and the model — writes the row. DBAL and deliberately no entity, following EmbeddingStore: BackfillEmbeddings clears the EntityManager after every chunk, so an unflushed ORM row would be detached and silently dropped, fifty at a time, for as long as a backfill runs — which is exactly the window the panel exists to show. AiFeature could not carry this. A four-character search query somebody is waiting on and two thousand characters of mail body written unattended a hundred thousand times both went through embed() with no discriminator, and averaged together the second buries the first — the panel would report backfill throughput under the heading "search". AiCallFeature splits them. THE PANEL Live state from /api/ps, which is passive and safe to poll. Six states that look different, and the one that justifies the work is "partly on the CPU": size_vram < size means layers spilled, throughput collapses, and the model still answers correctly — invisible everywhere else, so everything is simply slow forever. THE COMPOSER Tokens stream into a provisional surface and commit in one operation, because hundreds of per-token insertions destroy the undo stack and Ctrl+Z is the first thing anyone tries. Waiting, generating, done and failed are distinct states with reasons: outside the keep-alive window the first thirteen seconds produce nothing while 20 GiB loads, and a featureless spinner for that IS the bug. Cancellation propagates to the host — an abandoned draft otherwise holds the GPU and everything else queues behind it. SEARCH Silent degradation to keyword-only is replaced by the reason it degraded, and rows found by meaning say so. The semantic arm and its join are now matched on model AND width: vectors from different models are not comparable, and after somebody changes the search model every existing vector is dead weight. On a pgvector install comparing different widths raises an error — a 500 on /mail/search caused by an admin dropdown — and on the shipped loop it silently compares the overlap, which is worse. Coverage is stated where results are shown, because semantic search over an 8%-indexed mailbox is a trap. BACKFILL Progress is counted from the database, never accumulated, so a killed worker resumes. It stands aside while anyone is using the AI and comes back on its own: both compete for one integrated GPU, and a batch in flight when somebody presses the button puts them straight back to "nothing happens". Also carries the cron line for app:jobs:reap from the commit before this one. Verified against the real host: reachable, both configured models installed, timings parsed, 1024-dimension embeddings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
92004ee576 |
fix(jobs): background work would not open, and finished work never left
Two faults in the same indicator, reported together as "it fades in and out but clicking does nothing" plus three "Marking as read" jobs stuck at partial progress that had stopped weeks earlier. The menu was hidden by Tailwind's `hidden` CLASS while ui--dropdown opens a menu by clearing the `hidden` PROPERTY and never touches classes. So the panel was display:none for the whole life of the page — and because the attribute was absent, the first click read the menu as already open and CLOSED it. No error, no console warning, nothing to search for. Every other dropdown in the app uses the attribute; this was the one that did not, and a static test now says so. The stuck jobs were a different fault with the same face. findVisibleForUser matched `state IN (queued, running)` with no time bound of any kind, so "still going" and "abandoned" were literally the same row and no amount of waiting would ever change it. A job that fails properly calls finish() and clears after five minutes; these never reached that line — a killed worker, a restart mid-deploy, a fatal that bypassed the failure path. The entity had no progress timestamp, so a long legitimate job could not be told from an abandoned one by time. It has one now, stamped by begin() and advance() so the counter and the stamp cannot come apart, the query is bounded by it, and app:jobs:reap marks the stragglers failed so they are seen once and then go. The migration backfills last_progress_at from created_at, which is the only value that gets both directions right: a job stranded weeks ago is instantly outside the window and swept, while a bulk action started two minutes before the deploy is inside it and left alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
6c33316f33 |
feat(settings): a second opacity slider, for windows and menus
One slider drove every translucent surface, and translucency multiplies: a composer at 0.7 over a pane at 0.7 left about a tenth of the wallpaper landing in the middle of the text being read. So the knob splits. Structural surfaces — sidebar, top bar, main and calendar panes — keep --pane-alpha. Floating ones — the compose window, modals, confirm dialogs, toasts and twelve menus — take --popover-alpha, and both floating utilities re-point --pane-alpha to it for their own subtree, so an inset strip inside the composer belongs to the composer rather than to the sidebar behind it. The guarantee is a floor, not just a default: 0.5, so a popover can never be made as see-through as a pane. Boxed spends the gap at 0.9 over its panes' 0.7 — about 3% of the background survives both layers instead of 10%. The old stored value is deliberately NOT copied across. Twelve of the sixteen floating surfaces never honoured it: `popover` has been hardcoded opaque since it was written, for exactly the illegibility reason being reported. Backfilling 0.7 would have introduced the fault in twelve places to preserve it in four. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
1b6c7d9916 |
feat(ai): somewhere to keep a vector, and one way to compare two
Storage for semantic search. Nothing searches yet — this is the part the search has to stand on, and most of its design was settled by measurement rather than by preference. NO pgvector, AND THAT IS THE RECOMMENDATION NOT THE COMPROMISE The shipped image is postgres:18-alpine, which has no vector extension. Moving to pgvector/pgvector:pg18 means musl -> glibc on an existing data volume, and that was tested rather than reasoned about: the swap returns WRONG ROWS from an index scan with an empty log. musl records no collation version, so datcollversion is NULL, so Postgres's mismatch check never fires at all. 44 collation-dependent indexes, 18 of them UNIQUE, silently walked with the wrong comparator. On a stack whose upgrade doc says "Starting the new image IS running it", that is not a hazard to accept for a feature nobody has asked to be mandatory. ONE FUNCTION NAME, TWO POSSIBLE BODIES The application only ever writes plmail_embed_distance(a, b). The MIGRATION picks the body: a plpgsql loop normally, and a one-line SQL cast to `vector` if the extension happens to exist — which Postgres inlines, so even an HNSW expression index over the same real[] column is reachable. Detection lives in SQL, once. No PHP branch, no second query builder, and the same tests exercise whichever body is installed. Measured: 0.11ms per 768-dimension row, so 56ms over 500 candidates — cheaper than the full-text pass that produces them. Over 100,000 rows the same loop is 11 seconds, which is worse than the 9-10s pathology the search UNION was rewritten to escape. So it is only ever allowed to run over a bounded candidate set, and that is a rule the search integration has to keep rather than a hope. ONE ROW PER MESSAGE, IN A TABLE OF ITS OWN Per chunk would multiply every (thread, matching message) row before the GROUP BY collapsed it again — precisely the row multiplication removed from the label joins in v0.1.40, whose defining property is that the answers stay correct and only the clock shows it. A table rather than a column because a 768-dim float4 array is 3,092 bytes, past the TOAST threshold, so every vector is stored out of line — 391 MB for 100,000 messages, measured. That does not belong on `message`. NORMALISED ON THE WAY IN Unit length at write time is what lets the distance function be a dot product with no square root per row, and the arithmetic dominates I/O about six to one in that loop. The test reads the value back OUT of Postgres to check the length, because a vector that came back denormalised would make every distance subtly wrong in a way nothing reports — the search would simply rank badly. A zero-length or non-finite vector is refused rather than stored: one NaN in that column poisons every ORDER BY that touches it, and it would be blamed on the search rather than on the model that produced it. DBAL, no entity, on purpose. real[] is not a type Doctrine can name, and teaching it one means a mapping_types entry so the comparator stops asking to rewrite the column on every diff — the workaround already carried for tsrange. Nothing here needs an object. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
ac5b5360fe |
feat(ai): a second opinion on mail no rule recognised
Categorisation, built so that the model can only ever break a tie. WHERE IT IS ALLOWED TO COUNT At the very bottom of the cascade, after every rule has declined. At that point the honest answer is "we do not know, so Primary", and that is the one place another opinion helps without doing harm. Anywhere above it, the model could only overrule something that actually matched a header — which would make a tab's contents depend on which model happened to be installed the week the mail arrived, with nothing on the row to say so, decided unattended on every message that arrives. MailClassificationTest asserts it cannot: a matched header, a correspondent and a Gmail label each outrank a verdict that disagrees. STORED BESIDE THE CATEGORY, NEVER IN IT `category` is what the deterministic rules produced and what every list query filters on, and it has to stay explicable without reaching for a machine that may since have been switched off. So the verdict lives in its own column and the cascade reads it only in the branch it would otherwise have answered "default" from. Which makes switching the feature off sufficient on its own: nothing clears the column, nothing needs re-processing, and a stale verdict from a model removed months ago is inert as long as a rule recognises the mail. There is a test for exactly that. NO CALL ON ANY READ PATH MessageCategorizer never talks to anything — it reads a verdict stored asynchronously. explain() is called from the details panel, from ingest and behind list rendering; an HTTP round trip in there would put another machine on the path of drawing a page. The one request lives in a Messenger handler, on a worker, after the transaction, over a batch, for a subset. The step that dispatches it checks the feature is on before enqueueing anything, and the handler checks again — not redundantly: almost every install has this off, and without the first check every one of them would queue a job per batch for a handler whose whole body is `return`; without the second, a job already queued when somebody switches it off would still run. WHAT IT REFUSES TO ASK ABOUT Anything a rule recognised — the answer is known, explicable, and would be discarded anyway. That test needs the correspondent set, which is a query, so it happens in the handler rather than in the step that runs inside a sync holding a mailbox connection. The attempt is stamped even when the answer is unusable, so a backfill can tell "asked, got nothing" from "never asked" and does not retry the same unanswerable message forever. An ambiguous answer — two category names in one reply — is discarded rather than resolved by first match: the model has not answered the question, and leaving the mail where the rules put it is the honest result. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
64ac8f3704 |
feat(ai): the foundation — a model host, a gateway, and an admin section
First slice of the optional AI feature set. Nothing user-facing does anything yet; this is the part everything else stands on, and it is built so that an installation which never touches it cannot tell the difference. OFF IS A REAL STATE, NOT AN UNCONFIGURED ONE Every column starts off or empty and plMail is a complete mail client in that state: no degraded search, no empty panels, no nagging. That is the state every existing install is in and the state most will stay in, so the expensive thing to get wrong is not a bad completion — it is a request being made at all by somebody who never asked for this. The tests that matter most assert what did NOT happen: no HTTP call, no exception, a null the caller already handles because the host can equally be off at the wall. THREE SWITCHES, NOT ONE Writing help is asked for once, deliberately, by somebody looking at the result. Categorisation runs on every message that arrives, unattended, and a confidently wrong model quietly misfiles mail. Embedding runs over the whole mailbox and then on everything new. Wanting the first and not the third is reasonable, and a single master switch would make that choice for people. The master switch overrules all three, so everything can be stopped in one action without losing the configuration under it — the difference FcmConfig draws between "not using this" and "lost the key". WHY OllamaClient IS NOT ImageProxyFetcher That class exists to REFUSE private addresses: it takes a URL out of a message somebody was sent, so a link to 10.0.0.5 is an attacker asking this server to fetch something only this server can reach. Here the private address IS the feature — it is typed by an administrator into a form only an administrator can open and stored as configuration. So OllamaClient performs no address validation at all, and the safety is entirely in where the URL is allowed to come from. The two must never share a policy and must never share a code path: a helper both called would eventually be relaxed for one and quietly weaken the other. Said in both docblocks, because that is the comment somebody will be tempted to delete. DIALECT /api/embed with a fallback to /api/embeddings — a host set up a year ago speaks the older one, and refusing to work against it would be reported as broken rather than as out of date. Chat is unstreamed: everything asked for here is short, wanted whole, and inserted into a form. Components are cast to float because whole numbers decode from JSON as int, and a mixed list is a valid embedding and an invalid list<float>. The admin section is registered in all four of the hand-maintained lists that make one exist — the allow-list, the nav, the refresh-hint suppression and the section body. Three of four renders a broken page. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
5f648864ab |
feat(admin): set the log level from the page that shows the logs
Asked for directly: "let the admin panel set the log level. i do not want to mess with env vars." APP_DB_LOG_LEVEL meant editing a file on the host and restarting the stack, and the moment it is actually wanted is the moment that is most awkward — something is wrong, the answer is one level further down, and restarting throws away whatever was in flight. So the level gets turned down, forgotten, and left down. WHAT IT IS NOT Not another control in the filter row. That row narrows what is SHOWN out of what was stored; this decides what gets STORED. They look alike and mean opposite things, and side by side they would invite an admin to change the wrong one and conclude the log was broken. Its own row, its own form, its own wording. NULL IS NOT "TODAY'S VALUE" A stored level of null means "follow the environment", which is deliberately different from copying APP_DB_LOG_LEVEL's current value into the row. An install that has never chosen keeps following its configuration including later changes to it, and there is a way back to that after a level has been set — which storing the resolved value would have removed, silently, on the day the env var next changed. PER RECORD, NOT PER PROCESS Monolog fixes a handler's level at construction. That is right for one configured in a file and useless for one an administrator can turn down from the page they are reading, so DoctrineLogHandler decides in isHandling() instead, and the constructor's level is now only the answer when no resolver was given — which keeps directly-constructed handlers, and every existing test, behaving as they did. LogLevelResolver caches for ten seconds. A query per log line would put a database round-trip inside the thing that runs when the database is in trouble; a value cached for the life of the process would leave every long-lived worker ignoring the change until the next deploy, and the worker logs are usually the ones it was lowered for. It uses DBAL, not the ORM, for the same reason the handler does, and every failure — no table yet, no database, a name that is not a level — falls back to the environment rather than throwing. A logger that throws while deciding whether to log is worse than one briefly out of date. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
561fc3f372 |
feat(mail): a message that bounced now says so, on the message that bounced
plMail could tell you a message had been READ. It could not tell you the opposite. A bounce arrived in the Inbox as a machine-generated mail titled "Undelivered Mail Returned to Sender" whose body is an SMTP transcript, and the message it was about went on sitting in Sent saying "Sent" — which at that point is a lie. The one delivery outcome a person has to act on was the one outcome attached to nothing. The structure was already here. A DSN is ReadReceiptCorrelator's twin: same multipart/report envelope, same Original-Message-ID lookup, same "stamp the sent row" payoff. BounceCorrelator is a separate service anyway, because the two disagree on every judgement that matters. ONLY `Action: failed` A DSN reports four things and three of them are not bounces. `delayed` in particular is a message STILL IN FLIGHT — Postfix emits one after four hours and keeps trying for five days — and stamping it as undelivered would tell the user their mail failed at the exact moment it is being retried, with nothing to clear the stamp when it went through. `delivered` and `relayed` are good news and set nothing. THE BOUNCE IS NOT FILED AWAY A read receipt is noise once correlated and gets marked read and taken off the Inbox. A bounce stays exactly where it landed, unread. Its body is routinely the only readable statement of what went wrong, and a failure the user may have to act on is not something to tidy up on their behalf. NOTHING IS CLASSIFIED The status is stored as the reporting MTA wrote it — "5.1.1", "4.4.7" — and never reduced to a boolean "permanent". No address is disabled, nothing is removed from a contact list, no resend is attempted. plMail records what one server said about one delivery attempt. Deciding an address is dead on that basis is a conclusion this codebase has been wrong about before, and the sweep that led here said so repeatedly. A BOUNCE WAS BEING READ AS A READ RECEIPT Found by the first fixture written against this, and reachable before it. Both are multipart/report and both carry Original-Message-ID, so a DSN satisfied ReadReceiptCorrelator's body fallback, got correlated there, and put "Read at …" on a message that was never delivered — not merely missing information but the precise opposite of the truth. The two field blocks have mutually exclusive discriminators, so that is what tells them apart now: `Disposition:` is an MDN, `Action:` is a DSN. Both arms of the detection refuse a delivery-status report first. ON THE DEMO The demo seeds a reply sent to a typo'd domain and the MAILER-DAEMON answer four minutes later, correlated by the real BounceCorrelator rather than by writing the columns directly. A demo that stamps the result itself would stay green while the feature broke; this way the red panel disappearing from the demo IS the alarm. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
91551b2bd2 |
feat(health): count the times an account had to list its whole mailbox again
Both providers hand out a sync cursor that expires, and the answer to either is the same: throw the cursor away and re-enumerate everything. v0.1.32 made that recovery correct. Nothing recorded that it happened. Once in a while it is housekeeping — a mailbox that sat untouched past the provider's retention window. Repeatedly it means something else: the account is not being synced inside that window at all, or a worker is dying mid-batch and never getting far enough to store a new cursor. The symptom either way is a mailbox that is intermittently behind, which from outside looks like nothing whatsoever. So the account now carries fullResyncCount and lastFullResyncAt, written where the two syncers already handle expiry — Gmail's 404/410 on a stale historyId, Graph's resyncRequired. The health page attaches it as a FACT on the existing "sync failing" card rather than as a card of its own, because on its own it is not a problem and a card would make it read as one. HealthFact gained labelParams for it. The inspector still emits keys and parameters and never a translated string, which is the rule that keeps the health page translatable and testable in one language. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
b939fdae83 |
feat(mail): acting on a whole view runs as a job, with an indicator that says so
Selecting 5,455 conversations and marking them read died at `Maximum execution time of 30 seconds exceeded`. The request resolved the view, hydrated every thread AND every message, checked ownership per thread, then wrote — all inside a request with thirty seconds to live. The user got a broken page and no way to tell how much had happened. Chunking the Gmail call in v0.1.30 fixed the API refusing the batch. It did not make the work fit in a web request, because nothing could. A whole-view action now creates a job and returns. A list of ids stays inline, deliberately: it is bounded by the rows on screen, finishes in milliseconds, and answering it with "started" would make every ordinary archive feel slower than it is. THE JOB IS A ROW Same reasoning as MailRule's run state: a job over a large mailbox takes minutes and the person who started it closes the tab. The row answers "is it done?" afterwards, on any device. Mercure only nudges a page that is still open, and carries no state — a missed nudge costs one stale render, never a wrong number. The handler chunks for progress rather than speed: the counter is flushed per chunk so the indicator moves, a worker killed mid-run leaves what it did done and recorded, and the EntityManager is cleared each time, which is what stops a run over thousands of threads being killed for memory instead of time. THE INDICATOR EXISTS ONLY WHEN IT SHOULD Nothing in the topbar when there are no jobs — a permanent fixture is one more thing to learn to ignore. A job that SUCCEEDED does not linger either: it has already changed the list the user is looking at, and an indicator appearing and disappearing on its own is a topbar that reflows for no reason. A failure stays for five minutes, because nothing else will say so. It renders from a Twig function rather than a frame with a src. A frame that fetches itself costs a request per page load for every user to answer "nothing is happening" — and the appearance preview's assertion that it loads no mail caught exactly that, correctly. WHERE THE TEST LIVES The offload contract is a controller test, not an E2E one. It was an E2E test first and it was a bad citizen: a view selection spans every account by construction, so acting on one in the shared fixture mailbox reaches every other spec's mail, and the job's timing belongs to the environment rather than the spec. That cost two suites' worth of unrelated failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Paul LütznerandClaude Opus 5
|
25d2ec3817 |
feat(health): partial grants on connected services, the account behind a failed job, and what the mail server said
Three of the discarded signals from the sweep, all the same shape: the app had the fact in hand and threw it away. CONNECTED SERVICES HAD THE MAIL ACCOUNTS' OLD BLINDNESS A file store or photo library can grant part of what was asked for and return a token that works perfectly for the part it granted. Google Photos is the case that makes it concrete: an unverified app is routinely refused `photoslibrary.readonly` while being given `appendonly`, so saving to a library works, browsing one is empty, and nothing connects that to a consent screen clicked through last week. Recorded and compared now, with `offline_access` excluded — it asks for a refresh token rather than access to anything, and Microsoft does not echo it, which is the same false positive the mail side had to unpick. A 403 from Google Drive or Photos is also no longer reported as a missing permission when it is a rate limit. messageForStatus() read the status and never the body, so `userRateLimitExceeded` — a burst of saves, or a share-link action issuing two calls per file — told the user to re-grant a permission they already had, on a connection that would work again in a minute. "50 BACKGROUND JOBS WERE GIVEN UP ON" NOW SAYS WHOSE Nine of the nineteen message classes carry a public accountId, and it was discarded with the rest of the envelope. The card could not say that all fifty belonged to one mailbox whose grant could not write — which was the entire answer on the install that prompted this, sitting in the envelopes the whole time. Resolved in one query, not one per row. THE MAIL SERVER CAN FINALLY SAY SOMETHING RFC 3501 §7.1: the text of an `* OK [ALERT]` response "MUST be presented to the user". plMail read those lines off the socket in its IDLE loop and dropped them, which made it a MUST-violating client and lost the one channel a server has for saying what nothing else will. Quota is why it matters, and plMail's own preset list is full of German consumer ISPs with small free tiers. An over-quota mailbox connects fine and syncs fine; it just stops receiving, while every other signal in the app looks healthy. Shown verbatim — the server wrote a sentence for a person to read — and with no repair, because plMail cannot empty somebody's mailbox and a button that did nothing would be worse than none. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |