The original rejection of pgvector was tested rather than reasoned, and it
stands: pgvector/pgvector:pg18 means musl to glibc on an existing data volume,
and 44 collation-dependent indexes — 18 of them UNIQUE — are then walked with
the wrong comparator, silently, because musl records no collation version so
Postgres's own mismatch check never fires.
So the extension is compiled INTO the alpine image instead. libc never changes,
and there is nothing to rebuild. Verified end to end rather than argued: an
existing volume migrated on stock alpine, stopped, restarted on the pgvector
image, then bt_index_check(heapallindexed => true) across 175 btree indexes —
zero errors.
Two build flags carry more weight than they look. OPTFLAGS="" because
pgvector's Makefile defaults to -march=native, which bakes the build host's
instruction set into vector.so and gives SIGILL mid-query on a host without it,
with an empty log. with_llvm=no because a missing .bc file makes the JIT skip
inlining silently, and inlining does not apply to index scans anyway.
Pinned at v0.8.6, not the earliest that works. v0.8.0 does not compile against
Postgres 18 at all (vacuum_delay_point gained a parameter); v0.8.1 is the first
that does. But 0.8.3 fixed possible index corruption with HNSW vacuuming and
0.8.4 fixed an unrepaired HNSW graph, and message_embedding is ON DELETE CASCADE
from message on a system that deletes mail in bulk — so vacuum is a hot path
here and shipping the earliest that merely compiles would ship two known
corruption bugs.
THE HANDBOOK'S 24x WAS WRONG AND IS CORRECTED TO 1.75x. The SIMD dot product is
microseconds; what dominates is that embedding is real[], so every row pays a
4KB detoast and a fresh real[] to vector conversion. 74,000 rows: 7.70s becomes
4.51s.
NO INDEX IS CREATED, and that is the finding rather than an omission. An HNSW
index answers ORDER BY ... LIMIT k and the search asks distance <= 0.45, which
no ANN index serves. The expression form needed to build one at all is
(embedding::vector(1024)), which Postgres treats as a DIFFERENT expression from
the inlined function's (embedding::vector) — so the index exists, the planner
ignores it, and nothing says so. And it costs 7m40s and 578MB to build at the
maintenance_work_mem this stack ships, behind the one advisory lock every
entrypoint waits on: an eight-minute dead stack, not a slow migration. The
measured route to an index is written down in ai-assist.md rather than half
taken.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Semantic search measured 12.9s mean and 47s max in production, and it was
getting worse as indexing progressed — 61.9% indexed then, projecting to ~19.6s
at 100%. The arithmetic was exact: plmail_embed_distance costs 0.132ms a call
at 1024 dimensions, and one search made about 92,000 of them.
Two faults multiplied.
The semantic ARM was correctly capped at SEMANTIC_CANDIDATES. The outer
`LEFT JOIN message_embedding sem` was not capped at all, so the rank expression
scored every embedded message the query matched — including rows only a keyword
had found.
And it scored them TWICE. The comment above $semanticRank said Postgres matches
identical aggregate calls and evaluates the argument once per row, which is
true, and the code carefully built one string to get it. DBAL expands the same
named :queryVector into a separate positional parameter at each occurrence, so
what reached the server were two textually different expressions. Production's
own pg_stat_statements showed it: $7(...$3 AS real[]) beside $10(...$4 AS
real[]).
Both go away by construction rather than by care. The distance moves into a
materialised CTE over the candidates, the arm reads from it, the outer query
joins it, and plmail_embed_distance and :queryVector each now appear exactly
once in the whole statement. Measured on 20,000 rows: 12,616 calls and 3,781ms
become 2,000 calls and 602ms.
MATERIALIZED is load-bearing, not decoration — without it the planner inlines
the CTE at both references and 3.6x of that goes back. Two CTEs rather than one
for a neighbouring reason: with the threshold applied inside a derived table the
subquery is flattened, similarity lands in both the filter and the target list,
and every row that PASSES is scored twice.
The function's cost is corrected too, and derived rather than guessed:
calibrating two index-only scans against their planner costs gives ~1.5us per
cost unit, so 132us is about 33,000 of them. Rounded up to 50,000, because
underestimating is the failure being fixed. Only the plpgsql body — the pgvector
body inlines before costing.
The cheap pass now runs under a statement_timeout and drops the vector for the
rest of the search when it expires, so a pathological plan is paid once rather
than up to four times.
ONE BEHAVIOUR CHANGE, deliberate. Ranking moves for 3,966 of 5,308 rows in the
benchmark, every one a keyword row: the old unbounded join had no threshold and
no recency bound, so GREATEST(ts_rank, 1 - distance) took the vector's number on
rows the vector had never found. 80.5% of keyword rows were ranked by a
below-threshold similarity averaging 0.308 against a true ts_rank of 0.067. All
308 semantic-only rows keep their rank exactly, and EXCEPT ALL in both
directions on (thread_id, semantic_only) is empty.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The administrator decides what an installation offers; this decides what one
person uses of it. Settings → Assistant: three feature switches, two notes the
model is given before it writes, and how much of a conversation it reads.
THE CEILING AND THE FLOOR
AiSettings stays a ceiling and no user value can get past a false in it. The
AND is written in exactly one place, AiPermissions, and every one of the four
things in src/ that builds content for a model now takes the user in its
SIGNATURE — so it is not possible to build a prompt without having named whose
mail it is. A worker that cannot work out whose mail it is holding REFUSES: two
of them run unattended over somebody's correspondence, and "we could not tell"
is not a reason to send it to a model.
That reaches into the transport. EmbedMessagesMessage now carries the user id,
which for one deployment window means already-queued batches are refused and
picked up by the nightly sweep instead.
HOW MUCH IT READS
An enum, not a switch: Nothing / The message / The conversation. Whole-thread
is new capability rather than a restriction — the model stops answering a
conversation it has only seen the last turn of. An oversized thread loses its
OLDEST turns and always keeps the message being replied to.
There is no privacy control here and the copy does not pretend otherwise. The
model runs on the operator's own machine; what more context costs is time, and
the hint says so. A "do not send the subject" switch was designed and dropped
for the same reason — its only honest label would have been "make my search
worse".
WHAT COULD NOT BE ENFORCED, SAID PLAINLY
A persona can talk a model out of the app's own rules; models have no reliable
precedence inside one system message. So the hint says "Preferences, not
instructions — the app's own rules come first", which is a statement of intent
rather than a promise. What IS mechanical: placement, a length clamped on write
AND re-clamped on read (hooks are skipped on hydration), a blast radius of
writing help alone, and provenance from the entity rather than the POST body.
Switching search off stops indexing; it does not forget. The copy says stops.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reported three times, because it was never one slip. _card_chrome.html.twig has
said px-4 since it was written, and only two admin templates used it — every
other card hand-rolled px-5, px-6, px-3.5 or px-4 by hand. Consistency was
accidental, and each new card was a fresh coin flip.
Twenty-eight cards now take their band and body from the macros, and the labels
and prose between cards take the same inset the cards do. What is deliberately
left alone is written down where it is: tinted cards cannot use the macro
because two border colour utilities on one element are resolved by stylesheet
order rather than attribute order, row-shaped summaries would change height,
and dense list rows would end up padded twice — 32px from the card edge to the
first word, which the chrome docblock warns about.
Two guards, because one would not hold.
The static test is a threshold and not a blocklist: inside templates/admin no
class attribute may inset further than the macros' px-4. Buttons, chips and
inputs live below that line by construction, so the rule needs no exception
list to maintain.
The rendered test measures. It walks every section in SECTIONS, resolves each
to its turbo-frame id so a renamed section fails loudly instead of silently
re-measuring the previous one, and descends until it finds something actually
padded — so a padded box, a list of self-padding rows and a full-bleed table
all answer the same question. It collects every offence before asserting, so
one run names every wrong card rather than the first.
Both were proven to fail against the old markup before being trusted. A guard
that has never failed is not known to work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
In a 56px rail every icon centred on 28 except Inbox and Starred, which sat on
18. Measured, because ten pixels is exactly the amount that looks like nothing
in isolation and wrong in a column.
Two causes, both structural. Those two are the rows that carry an unread badge,
so they are a <div class="nav-item"> wrapping the link rather than a bare <a> —
and the rail rules target `nav > a`, so they never reached them. The
`justify-between` that spaces a label from its badge then put the icon at the
START of the row once .rail-hide had taken the badge away.
The new-mail dot was the other half: a flex ITEM sitting after the label, so
hiding the label left it holding a slot and a gap. Lifted out of the flow and
pinned to the icon's corner rather than hidden — a collapsed rail is precisely
where "something arrived in here" is the only signal left, since the count
badge has already gone. Hiding it would have centred the icon by removing the
one thing the row still had to say.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three small things, all reported from use.
The writing help answered German mail in English, and proofreading a German
draft quietly translated it. The instructions are written in English and a model
reads that as evidence of the language it is meant to answer in — so the rule
has to name that pull and refuse it, not merely ask for the right language. The
proofread case is the serious one: a task asked to fix punctuation returned a
different message in a different language, and the writer would have had to
notice before sending it. Verified against the real host: a German mail now gets
a German reply, and proofreading fixes the umlauts without touching a word.
"Draft a reply" was offered on a new message, which has nothing to reply to. The
server already refused it, so the option was always there and never worked —
which reads as a broken feature rather than an inapplicable one. The rewriting
tasks stay, because they act on whatever is in the body.
And the last force-click goes. `force` skips the hit-target check, and skipping
it is how the snooze button came to be pressed while it was invisible: the event
went to its coordinates, landed on the ROW, and the failure said "the menu did
not open" — true, and silent about why. The shared helper now re-hovers on every
attempt (a row replaced under a stationary pointer never receives :hover, so
retrying alone re-reads the same nothing) and clicks honestly. snoozed-list
stops carrying its own copy of that helper.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
New mail was embedded the moment it arrived. That is the wrong trade on a box
with one GPU: it is rare to search for mail that has just landed, and paying to
index every arrival competes with the two things somebody is actually waiting
on.
Two triggers replace it. A nightly sweep is the backstop, and a search is the
invitation — a search has just loaded the embedding model, so the cheapest
possible moment to index is immediately after one, when the model is warm and
nobody is waiting on it any more.
THE PART THAT MADE THIS MORE THAN A SCHEDULE CHANGE
The backfill stood aside after ANY interactive AI use, and a search counted.
That is precisely backwards for search, and it took both halves of the signal to
say so: lastInteractiveCallAt() matched
`feature IN ('search_query','writing_help')`, and the request subscriber stamped
`app_mail_search` alongside `app_compose_assist`.
The distinction the old code could not make is that there are two models with
completely different costs. The composer spends the 20.3 GiB writing model,
which is worth standing aside for and is the case the yielding was written for.
Search spends the sub-gigabyte embedder — the same model indexing uses. So a
finished search is not a reason to keep away, it is the best moment there is.
Both docblocks now cross-reference each other, so restoring one half alone reads
as obviously wrong.
The opportunistic trigger claims its rate-limit marker BEFORE it queries, so the
mailbox scan is inside the window rather than only the dispatch, and a search
that produced no vector at all spends nothing — one unreachable moment must not
cost the next five minutes of indexing.
EmbedMailStep is deleted rather than emptied. An empty step would still be built
and iterated on every ingest batch while carrying a name asserting that
embedding happens on arrival — an active lie in the one directory somebody reads
to find out what happens to new mail.
The finder matches on model and not on width, deliberately. What consumes its
ids asks about the model alone, so a width-aware finder would hand over a full
budget the handler then drops as already done: a nightly sweep that reports
itself busy and indexes nothing, for ever. The width mismatch already has an
owner in the coverage report's "stale" count, which tells the user to re-index.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The chrome macro puts a card's band and its body on the same px-4, and both AI
cards hand-rolled px-5 for the body — so every heading and the text under it
started four pixels apart, down the whole page.
The cards collapse now, and the settings one starts collapsed once a host and a
model are on file. Settings are a thing you FINISH: after that the card is a
wall of filled-in fields standing between the administrator and the two panels
below it, which are the ones worth coming back to. It opens itself for the three
cases that still want attention — nothing configured yet, the moment after a
save, and a form that came back refused.
Native <details>, like the settings pages already use: keyboard-operable and
announced with nothing added, no JavaScript, and nothing for the policy to
refuse.
The test renders admin/ai/_frame.html.twig, which nothing did before — the panel
test renders only the telemetry partial, so the settings card, the form and the
expression deciding the fold had no coverage at all.
Its first version was worthless and is worth recording: the summary's chevron
carries `group-open:rotate-90`, so searching the markup for "open" was true
whatever the card did. The assertion passed, its negation could never pass, and
neither was looking at the thing under test. It matches the opening tag now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three faults in the writing help, all of them things that only appear once it is
actually being used.
PHP's max_execution_time is 30 seconds for the web SAPI while the model call is
allowed 120, so the limit always belonged to the wrong side of the request. A
writing model that has fallen out of the keep-alive window needs about thirteen
seconds to be read off disk BEFORE the first token exists, and a reply worth
reading is another twenty or thirty after it — so the limit fired mid-generation
and the reader watched the text stop halfway:
MaxExecutionTimeError: Maximum execution time of 30 seconds exceeded
at CurlResponse.php line 370
The worst possible place for it to land, too: inside the HTTP client's read, so
the unwinding that cancels the model call and records the row never happened and
Ollama carried on generating for a reader who had already been shown an error.
The bound now comes from the pieces that can actually enforce it — the upstream
timeout, and connection_aborted() on the next write.
The answer went in at the end of the editor, which in a reply is UNDERNEATH the
quoted original: an answer nobody scrolls to, sitting below the message it was
answering. It goes above the quote now, which is where a person's cursor already
is when they start typing a reply. Plain text too, where the boundary is the
first `>` line less its attribution line — inserting between "On … wrote:" and
the quotation splits it in half.
And the wait had nothing moving in it. There is nothing honest to put in a
progress bar, because the host reports no progress at all while it loads, so a
still sentence was all there was and it read as a window that had hung. Three
dots that breathe say "working" and claim nothing about how far along it is,
which is exactly as much as we know. They stop at the first token, where
arriving text says it better, and they hold still but stay visible under
prefers-reduced-motion.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"I click the AI button in the composer and nothing happens" turned out not to be
about the model at all. #editor() queried data-compose--toolbar-target while the
composer renders data-compose--compose-toolbar-target, so run() returned at its
first guard: no request, no status, no error. The feature had never worked once.
#subject() was wrong the same way and quieter — every prompt went out with an
empty subject. And the status element lived INSIDE the dropdown that the same
click closes, so even once working it would have written into a hidden subtree.
The rest of this is what makes the next such thing findable.
RECORDING
One row per model call, tagged by the workload that caused it, and nothing about
what was in it — no prompt, no completion, no message id, no subject, no
address. error_kind is a category from a closed set for the same reason: an HTTP
client's exception message routinely quotes the request body back, and here that
is somebody's mail. OllamaClient was discarding Ollama's whole timing block; it
now returns it, and AiAssistant — the only place that knows both the feature and
the model — writes the row.
DBAL and deliberately no entity, following EmbeddingStore: BackfillEmbeddings
clears the EntityManager after every chunk, so an unflushed ORM row would be
detached and silently dropped, fifty at a time, for as long as a backfill runs —
which is exactly the window the panel exists to show.
AiFeature could not carry this. A four-character search query somebody is
waiting on and two thousand characters of mail body written unattended a hundred
thousand times both went through embed() with no discriminator, and averaged
together the second buries the first — the panel would report backfill
throughput under the heading "search". AiCallFeature splits them.
THE PANEL
Live state from /api/ps, which is passive and safe to poll. Six states that look
different, and the one that justifies the work is "partly on the CPU":
size_vram < size means layers spilled, throughput collapses, and the model still
answers correctly — invisible everywhere else, so everything is simply slow
forever.
THE COMPOSER
Tokens stream into a provisional surface and commit in one operation, because
hundreds of per-token insertions destroy the undo stack and Ctrl+Z is the first
thing anyone tries. Waiting, generating, done and failed are distinct states
with reasons: outside the keep-alive window the first thirteen seconds produce
nothing while 20 GiB loads, and a featureless spinner for that IS the bug.
Cancellation propagates to the host — an abandoned draft otherwise holds the GPU
and everything else queues behind it.
SEARCH
Silent degradation to keyword-only is replaced by the reason it degraded, and
rows found by meaning say so. The semantic arm and its join are now matched on
model AND width: vectors from different models are not comparable, and after
somebody changes the search model every existing vector is dead weight. On a
pgvector install comparing different widths raises an error — a 500 on
/mail/search caused by an admin dropdown — and on the shipped loop it silently
compares the overlap, which is worse. Coverage is stated where results are shown,
because semantic search over an 8%-indexed mailbox is a trap.
BACKFILL
Progress is counted from the database, never accumulated, so a killed worker
resumes. It stands aside while anyone is using the AI and comes back on its own:
both compete for one integrated GPU, and a batch in flight when somebody presses
the button puts them straight back to "nothing happens".
Also carries the cron line for app:jobs:reap from the commit before this one.
Verified against the real host: reachable, both configured models installed,
timings parsed, 1024-dimension embeddings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two faults in the same indicator, reported together as "it fades in and out but
clicking does nothing" plus three "Marking as read" jobs stuck at partial
progress that had stopped weeks earlier.
The menu was hidden by Tailwind's `hidden` CLASS while ui--dropdown opens a menu
by clearing the `hidden` PROPERTY and never touches classes. So the panel was
display:none for the whole life of the page — and because the attribute was
absent, the first click read the menu as already open and CLOSED it. No error,
no console warning, nothing to search for. Every other dropdown in the app uses
the attribute; this was the one that did not, and a static test now says so.
The stuck jobs were a different fault with the same face. findVisibleForUser
matched `state IN (queued, running)` with no time bound of any kind, so "still
going" and "abandoned" were literally the same row and no amount of waiting
would ever change it. A job that fails properly calls finish() and clears after
five minutes; these never reached that line — a killed worker, a restart
mid-deploy, a fatal that bypassed the failure path.
The entity had no progress timestamp, so a long legitimate job could not be told
from an abandoned one by time. It has one now, stamped by begin() and advance()
so the counter and the stamp cannot come apart, the query is bounded by it, and
app:jobs:reap marks the stragglers failed so they are seen once and then go.
The migration backfills last_progress_at from created_at, which is the only
value that gets both directions right: a job stranded weeks ago is instantly
outside the window and swept, while a bulk action started two minutes before the
deploy is inside it and left alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One slider drove every translucent surface, and translucency multiplies: a
composer at 0.7 over a pane at 0.7 left about a tenth of the wallpaper landing
in the middle of the text being read.
So the knob splits. Structural surfaces — sidebar, top bar, main and calendar
panes — keep --pane-alpha. Floating ones — the compose window, modals, confirm
dialogs, toasts and twelve menus — take --popover-alpha, and both floating
utilities re-point --pane-alpha to it for their own subtree, so an inset strip
inside the composer belongs to the composer rather than to the sidebar behind
it.
The guarantee is a floor, not just a default: 0.5, so a popover can never be
made as see-through as a pane. Boxed spends the gap at 0.9 over its panes'
0.7 — about 3% of the background survives both layers instead of 10%.
The old stored value is deliberately NOT copied across. Twelve of the sixteen
floating surfaces never honoured it: `popover` has been hardcoded opaque since
it was written, for exactly the illegibility reason being reported. Backfilling
0.7 would have introduced the fault in twelve places to preserve it in four.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reported live from the admin AI pane: clicking "Model host" pulled in the
operator's own mail address, and the Token field — type="password" — took their
mail password. Nothing was submitted, but a login credential was one press away
from being stored as an API token and sent to whatever host that field names.
PasswordManagerIgnore's existing attributes speak to extensions. This is the
browser, which is a separate problem with a separate opt-out: Chrome's built-in
autofill has no vendor attribute to respect and ignores autocomplete="off" on a
password field. It sees type="password" on the same origin as the login form
and offers the password saved for the site.
So a second constant, autocomplete="new-password", applied to the seven
credential INPUTS and deliberately not to the text fields around them — on a
non-password input that invites Chrome to offer to GENERATE one, which is a
worse interruption than the autofill it would prevent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both refusal paths returned the 1x1 placeholder and logged nothing at all. The
reader sees the same transparent GIF whichever way it failed — deliberately,
because an error that distinguishes "refused" from "timed out" hands back the
port-scan result the fetcher's rules exist to deny — but that opacity was
extended to the operator by accident.
The cost showed up chasing an Amazon QR code that would not render. The logs
had nine outbound fetches, all 200, and nothing whatsoever about the tenth: the
signature check had refused it before a request existed, so ImageProxyFetcher
never logged either, and the only symptom was one picture in one mail that
quietly did not appear.
The response is unchanged. Only the server now says why.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ten tests failed in full runs and passed in isolation, always as `the row has
no "Archive" action` — a row that was present and visible, whose buttons were
merely display:none.
The calendar pane's mode and width live on the USER, and the client persists
them whenever the pane is docked, including on arrival at mail from outside it
(ui--split's connect does that by itself, with nobody touching the switch). So
the state outlives the browser context, and the per-worker user is shared by
every spec file that worker runs. One spec that left the calendar docked
narrowed the list below the 36rem CONTAINER width the row's `hidden @xl:flex`
actions query — and a hidden element is not in the accessibility tree, so
getByRole found nothing and retrying re-read the same width until it timed out.
calendar-pane.spec.ts already had an afterEach putting the furniture back, but
a cleanup in one file cannot cover every producer. Seeding does it instead: a
deterministic inbox has to mean a deterministic LIST, not merely deterministic
mail. Every spec that uses rowAction seeds mail, so this covers all of them.
Proven rather than assumed: poisoning the worker's user to split/900 and
running app:test:seed-mail puts it back to mail/380.
The snooze menu's forced clicks go too. `force` skips the hit-target check, and
the trace of a 90s timeout shows exactly what that buys — no POST to /snooze at
all, and a POST to /status/thread/<id>/read, which is the conversation being
opened by a click the test believed it had aimed at a menu entry.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
feature/ai-assist — vector search, LLM categorisation, writing help, all
optional and all off by default. See docs/internals/ai-assist.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reported against the log level toggle added in v0.1.44, and it turned out
to be twelve older ones plus that.
Executing inline event handler violates the following Content Security
Policy directive 'script-src 'self' 'nonce-…'
`script-src` is enforced in production with a nonce, and the spec is
explicit that NEITHER a nonce NOR a hash authorises an inline event
handler without `unsafe-hashes`. So every `onchange="this.form.
requestSubmit()"` in this application was refused: the log browser's two
filters, the clock picker, the timezone picker, two compose defaults, two
compose-behaviour radios, the three push delivery filters, a copy field's
onclick, and the new log level select.
WHY IT LASTED
An inline handler violates nothing when the page LOADS. csp.spec.ts walks
every surface asserting a clean load and it was clean — the pages were
fine, the controls were fine, and nothing happened when you touched one.
Silent unless a console was open, and a control that does nothing reads
as "this setting is broken" rather than as a policy refusal.
Tom Select is what surfaced it: it re-dispatches a real `change` on the
hidden original element, which is exactly what fired the handler the
policy then refused — so the console names tom-select as the culprit and
it is nothing of the sort.
THE FIX, AND WHERE THE CONTROLLER GOES
`ui--auto-submit`, served from a file the policy already allows.
`data-controller` on the FORM and `data-action` on the input, because
Stimulus resolves an action against the element or an ancestor — on the
input alone it silently never fires, which is the same failure again.
requestSubmit(), not submit(): it fires the submit event so Turbo sees it
and validation runs, and on the two forms here that opt out with
data-turbo="false" it behaves exactly as submit() did.
`ui--broken-image` for the last one, which was doing something else
entirely: an attachment thumbnail's `onerror="this.remove()"`. Thumbnails
may legitimately not exist — a PDF, a format GD cannot read, one that has
been pruned — and the chip is fine without one, but a broken-image icon
in the middle of it is a defect rather than a fallback. That icon has
been there the whole time.
THE TEST IS STATIC, DELIBERATELY
A browser test would have to drive thirteen widgets, several of them Tom
Select with a hidden original, to state one rule. And the load-time walk
already exists and already passed. So the rule is asserted directly:
no template carries an inline event handler, with the message pointing at
the controller to use instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TESTS
The refusals are the half worth having. This produces text that goes into
somebody's outgoing mail and vectors that decide what a search finds, so
the failure that matters is not a clumsy sentence — it is a model
inventing a message because it was handed nothing to work from, or a
search quietly getting worse for an install that never switched any of
this on.
· WritingAssistant: the three rewriting tasks refuse an empty draft
without spending a request, a reply can still be drafted from the
message being answered alone, a code fence is stripped and prose that
merely mentions backticks is not, and the task list is closed —
asserted as a count, because it is a security property rather than
tidiness.
· SemanticQuery: every reason a vector might be missing — off,
unconfigured, host refusing the connection, query too short, a vector
that cannot be normalised — arrives at the same null, because search
must never be worse than it was before this feature existed.
DOCS
Admin -> AI, in both handbooks, written for somebody deciding whether
they want this rather than for somebody who has already decided. The
three switches are a table with a "when it runs" column, because that is
the difference that matters: sorting runs unattended on every arriving
message and the other two do not.
Says plainly that semantic search needs one pass over the existing
mailbox and that switching it on does nothing for mail already there —
which is the thing most likely to be reported as "it does not work".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
compose-send-optimistic.spec.ts:124 failed in the full suite twice
running and passed in isolation and in pairs — verified with the new
compose menu both present and absent, 24 passed either way, so it was
never the template.
Its first action clicked a message row immediately after goto with no
visibility wait, alone among the specs. The list animates in: rows carry
an entrance with a stagger delay and `animation-fill-mode: backwards`, so
a row is deliberately at opacity 0 until its turn.
The part worth writing down is why a 30-second auto-wait lost that race
when a 5-second explicit one wins it. `click()` resolves the locator once
and then waits for THAT element; a list re-render immediately after load
detaches it, and a detached node never becomes visible however long
anybody waits. Re-querying — which is what an `expect().toBeVisible()`
does on every attempt — gets the live one.
Full suite: 612 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three remaining pieces. Written to the seams the survey found, and
keeping the traps it named.
SEARCH — ANOTHER ARM OF THE SAME UNION
A vector hit is a candidate on exactly the same footing as a lexical one:
it lands in COUNT(*) OVER (), it pages correctly, and it needs no second
statement to reconcile with. Four things had to hold and each is written
down where it applies:
· The query is embedded ONCE, in the controller. buildSearchSql() runs
up to four times per search — the cheap pass, the body rescue, and
twice more when a page past the end recovers its total — and one of
those is inside a statement-timeout transaction, where a slow model
would be reported as a database fault.
· The candidate set is BOUNDED to the most recent 2,000 messages. The
distance function costs 0.11ms a row; over a whole mailbox it is
eleven seconds, worse than the pathology the UNION shape exists to
escape.
· A THRESHOLD, not a top-k. The total rides on the same statement, and
"lexical matches plus up to k" is a number that is only true on page
one.
· The inner LIMIT is parenthesised, because these arms are joined with
a bare ' UNION ' and an unparenthesised LIMIT binds to the whole
union rather than to one arm.
The rank join is 1:1, which is the property the label joins lost and had
to be given back in v0.1.40. Three tests: a message found by meaning that
no lexical arm can reach, one rejected by the threshold, and one with no
embedding still found by words — that last is what proves the LEFT JOIN
does not drop rows.
WRITING HELP — THE SERVER MAKES THE CALL
connect-src 'self' is enforced, so a browser could not reach the model
host even if it should — and it should not: that address is on the
operator's private network, and putting it in a page hands it to every
script the page loads.
The route carries NO draft id. A composer opens before one exists, which
is exactly when somebody wants a first draft; a URL built at render time
with an id in it is stale in that window. Nothing is saved as a side
effect either.
Four tasks, a closed set. A free-form instruction parameter would be a
prompt an attacker could write, with the user's own mail as context and
the answer pasted into the user's own draft. The message being replied to
is read from the DATABASE by id, ownership checked — never from the page.
Generated text is INSERTED, never substituted. A rewrite that overwrote
the original would be a model deleting somebody's words with no undo they
can see. It goes in as a plain text node at the end of the body: a styled
wrapper would be subtracted by the composer's typed-length calculation
and could stop the draft autosaving, and "at the caret" is a position the
writer did not choose, because opening the menu moved focus.
The menu is behind a Twig FUNCTION, not a variable — this window is
included from four places, two of them undo streams that include it
`only`, and strict_variables makes a variable threaded through three of
four a 500 on the undo path rather than a missing button.
EMBEDDING — ARRIVALS ON INGEST, THE REST ON DEMAND
A post-ingest step for new mail, and a command for everything already
there. The backfill re-dispatches itself with a cursor rather than
looping: a hundred thousand messages at a round trip each is hours, and a
single job holding that is killed by any worker restart with nothing to
show for it.
Ascending id, because it is the one ordering nothing can change under a
walk that long — mail arriving during it gets a higher id and is met by
the pass still coming. The vectors already stored ARE the progress
record, so there is no second one to keep in step, and the cursor
advances past everything LOOKED AT rather than everything stored, so one
message the model cannot answer for does not become a wall it restarts at
forever.
On the maintenance transport, not ingest: a backfill running for hours
would otherwise sit in front of every arriving message, and new mail
would stop appearing until an old mailbox had finished being catalogued.
KNOWN FAILING, AND NOT FROM THIS
compose-send-optimistic.spec.ts:124 fails in the FULL e2e run and passes
in isolation and in pairs — verified with the AI menu both present and
removed, 24 passed either way. It is the same spec that broke in the
earlier seed race, and the row it cannot click is present but still
mid-entrance. Cross-file interference, pre-existing, and left for the
debugging pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Written now rather than at the end because the two decisions most likely
to be argued with — the model can only break a tie, and pgvector is
refused on an upgrade path that silently corrupts indexes — are both
easier to overturn before more is built on them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Storage for semantic search. Nothing searches yet — this is the part the
search has to stand on, and most of its design was settled by measurement
rather than by preference.
NO pgvector, AND THAT IS THE RECOMMENDATION NOT THE COMPROMISE
The shipped image is postgres:18-alpine, which has no vector extension.
Moving to pgvector/pgvector:pg18 means musl -> glibc on an existing data
volume, and that was tested rather than reasoned about: the swap returns
WRONG ROWS from an index scan with an empty log. musl records no
collation version, so datcollversion is NULL, so Postgres's mismatch
check never fires at all. 44 collation-dependent indexes, 18 of them
UNIQUE, silently walked with the wrong comparator.
On a stack whose upgrade doc says "Starting the new image IS running it",
that is not a hazard to accept for a feature nobody has asked to be
mandatory.
ONE FUNCTION NAME, TWO POSSIBLE BODIES
The application only ever writes plmail_embed_distance(a, b). The
MIGRATION picks the body: a plpgsql loop normally, and a one-line SQL
cast to `vector` if the extension happens to exist — which Postgres
inlines, so even an HNSW expression index over the same real[] column is
reachable. Detection lives in SQL, once. No PHP branch, no second query
builder, and the same tests exercise whichever body is installed.
Measured: 0.11ms per 768-dimension row, so 56ms over 500 candidates —
cheaper than the full-text pass that produces them. Over 100,000 rows the
same loop is 11 seconds, which is worse than the 9-10s pathology the
search UNION was rewritten to escape. So it is only ever allowed to run
over a bounded candidate set, and that is a rule the search integration
has to keep rather than a hope.
ONE ROW PER MESSAGE, IN A TABLE OF ITS OWN
Per chunk would multiply every (thread, matching message) row before the
GROUP BY collapsed it again — precisely the row multiplication removed
from the label joins in v0.1.40, whose defining property is that the
answers stay correct and only the clock shows it.
A table rather than a column because a 768-dim float4 array is 3,092
bytes, past the TOAST threshold, so every vector is stored out of line —
391 MB for 100,000 messages, measured. That does not belong on `message`.
NORMALISED ON THE WAY IN
Unit length at write time is what lets the distance function be a dot
product with no square root per row, and the arithmetic dominates I/O
about six to one in that loop. The test reads the value back OUT of
Postgres to check the length, because a vector that came back
denormalised would make every distance subtly wrong in a way nothing
reports — the search would simply rank badly.
A zero-length or non-finite vector is refused rather than stored: one NaN
in that column poisons every ORDER BY that touches it, and it would be
blamed on the search rather than on the model that produced it.
DBAL, no entity, on purpose. real[] is not a type Doctrine can name, and
teaching it one means a mapping_types entry so the comparator stops
asking to rewrite the column on every diff — the workaround already
carried for tsrange. Nothing here needs an object.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Categorisation, built so that the model can only ever break a tie.
WHERE IT IS ALLOWED TO COUNT
At the very bottom of the cascade, after every rule has declined. At that
point the honest answer is "we do not know, so Primary", and that is the
one place another opinion helps without doing harm.
Anywhere above it, the model could only overrule something that actually
matched a header — which would make a tab's contents depend on which
model happened to be installed the week the mail arrived, with nothing on
the row to say so, decided unattended on every message that arrives.
MailClassificationTest asserts it cannot: a matched header, a
correspondent and a Gmail label each outrank a verdict that disagrees.
STORED BESIDE THE CATEGORY, NEVER IN IT
`category` is what the deterministic rules produced and what every list
query filters on, and it has to stay explicable without reaching for a
machine that may since have been switched off. So the verdict lives in
its own column and the cascade reads it only in the branch it would
otherwise have answered "default" from.
Which makes switching the feature off sufficient on its own: nothing
clears the column, nothing needs re-processing, and a stale verdict from
a model removed months ago is inert as long as a rule recognises the
mail. There is a test for exactly that.
NO CALL ON ANY READ PATH
MessageCategorizer never talks to anything — it reads a verdict stored
asynchronously. explain() is called from the details panel, from ingest
and behind list rendering; an HTTP round trip in there would put another
machine on the path of drawing a page.
The one request lives in a Messenger handler, on a worker, after the
transaction, over a batch, for a subset. The step that dispatches it
checks the feature is on before enqueueing anything, and the handler
checks again — not redundantly: almost every install has this off, and
without the first check every one of them would queue a job per batch for
a handler whose whole body is `return`; without the second, a job already
queued when somebody switches it off would still run.
WHAT IT REFUSES TO ASK ABOUT
Anything a rule recognised — the answer is known, explicable, and would
be discarded anyway. That test needs the correspondent set, which is a
query, so it happens in the handler rather than in the step that runs
inside a sync holding a mailbox connection.
The attempt is stamped even when the answer is unusable, so a backfill
can tell "asked, got nothing" from "never asked" and does not retry the
same unanswerable message forever. An ambiguous answer — two category
names in one reply — is discarded rather than resolved by first match:
the model has not answered the question, and leaving the mail where the
rules put it is the honest result.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The AI step, sitting with the other two admin steps because it configures
the install rather than a person's mailbox — and last of the three,
because it is the only one that can reasonably be answered with "no".
Mail credentials and integrations decide whether the app works at all.
SATISFIED MEANS ANSWERED, NOT ENABLED
"No thank you" is a complete answer and has to count as one, or the
wizard nags forever at every install that does not want this — which
will be most of them. So persist() writes the row even when every box is
empty, and isSatisfied() asks only whether a row exists.
That is deliberately weaker than the sibling steps, and it has to be: the
strongest available signal of "decided against" is indistinguishable from
"never opened" unless something is written down when they say no. The row
IS that record — the only way one appears is somebody pressing the button
here or in Admin -> AI.
FOUR PLACES, NOT ONE
Adding a step touches the enum case (the ordering), the icon match, the
requiresAdmin match, and OnboardingController::STEP_PATTERN. The last is
the one that does not update itself, and the comment above it is blunt
about the cost: a new case without it 404s AND takes the whole wizard
down with it, because the progress rail generates a URL for every
applicable step. The icon match is worse in a quieter way — non-exhaustive
is a fatal error, not a missing icon.
OnboardingStepCoverageTest exists to catch exactly this and passes.
The step renders the admin panel's own form, so there is one description
of these fields rather than two that drift. The aside spends half its
words on permission to skip, because that is the answer most installs
should give and a wizard that only explains how to say yes is a wizard
that pressures people into saying it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
First slice of the optional AI feature set. Nothing user-facing does
anything yet; this is the part everything else stands on, and it is
built so that an installation which never touches it cannot tell the
difference.
OFF IS A REAL STATE, NOT AN UNCONFIGURED ONE
Every column starts off or empty and plMail is a complete mail client in
that state: no degraded search, no empty panels, no nagging. That is the
state every existing install is in and the state most will stay in, so
the expensive thing to get wrong is not a bad completion — it is a
request being made at all by somebody who never asked for this. The tests
that matter most assert what did NOT happen: no HTTP call, no exception,
a null the caller already handles because the host can equally be off at
the wall.
THREE SWITCHES, NOT ONE
Writing help is asked for once, deliberately, by somebody looking at the
result. Categorisation runs on every message that arrives, unattended,
and a confidently wrong model quietly misfiles mail. Embedding runs over
the whole mailbox and then on everything new. Wanting the first and not
the third is reasonable, and a single master switch would make that
choice for people. The master switch overrules all three, so everything
can be stopped in one action without losing the configuration under it —
the difference FcmConfig draws between "not using this" and "lost the
key".
WHY OllamaClient IS NOT ImageProxyFetcher
That class exists to REFUSE private addresses: it takes a URL out of a
message somebody was sent, so a link to 10.0.0.5 is an attacker asking
this server to fetch something only this server can reach. Here the
private address IS the feature — it is typed by an administrator into a
form only an administrator can open and stored as configuration.
So OllamaClient performs no address validation at all, and the safety is
entirely in where the URL is allowed to come from. The two must never
share a policy and must never share a code path: a helper both called
would eventually be relaxed for one and quietly weaken the other. Said
in both docblocks, because that is the comment somebody will be tempted
to delete.
DIALECT
/api/embed with a fallback to /api/embeddings — a host set up a year ago
speaks the older one, and refusing to work against it would be reported
as broken rather than as out of date. Chat is unstreamed: everything
asked for here is short, wanted whole, and inserted into a form.
Components are cast to float because whole numbers decode from JSON as
int, and a mixed list is a valid embedding and an invalid list<float>.
The admin section is registered in all four of the hand-maintained lists
that make one exist — the allow-list, the nav, the refresh-hint
suppression and the section body. Three of four renders a broken page.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Asked for directly: "let the admin panel set the log level. i do not want
to mess with env vars."
APP_DB_LOG_LEVEL meant editing a file on the host and restarting the
stack, and the moment it is actually wanted is the moment that is most
awkward — something is wrong, the answer is one level further down, and
restarting throws away whatever was in flight. So the level gets turned
down, forgotten, and left down.
WHAT IT IS NOT
Not another control in the filter row. That row narrows what is SHOWN out
of what was stored; this decides what gets STORED. They look alike and
mean opposite things, and side by side they would invite an admin to
change the wrong one and conclude the log was broken. Its own row, its own
form, its own wording.
NULL IS NOT "TODAY'S VALUE"
A stored level of null means "follow the environment", which is
deliberately different from copying APP_DB_LOG_LEVEL's current value into
the row. An install that has never chosen keeps following its
configuration including later changes to it, and there is a way back to
that after a level has been set — which storing the resolved value would
have removed, silently, on the day the env var next changed.
PER RECORD, NOT PER PROCESS
Monolog fixes a handler's level at construction. That is right for one
configured in a file and useless for one an administrator can turn down
from the page they are reading, so DoctrineLogHandler decides in
isHandling() instead, and the constructor's level is now only the answer
when no resolver was given — which keeps directly-constructed handlers,
and every existing test, behaving as they did.
LogLevelResolver caches for ten seconds. A query per log line would put a
database round-trip inside the thing that runs when the database is in
trouble; a value cached for the life of the process would leave every
long-lived worker ignoring the change until the next deploy, and the
worker logs are usually the ones it was lowered for. It uses DBAL, not the
ORM, for the same reason the handler does, and every failure — no table
yet, no database, a name that is not a level — falls back to the
environment rather than throwing. A logger that throws while deciding
whether to log is worse than one briefly out of date.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reported as "the list loading animation gets interrupted in the middle
and restarts", and it took an instrumented browser to find because none
of it is visible from the code.
motion.js decides whether to play an entrance by asking remember()
whether it has shown this element before. remember() identifies elements
by their id, and answers "fresh" to anything with none:
function remember(element) {
if ("" === element.id) {
return true;
}
The rows region is <ul data-list-region="rows" data-enter-children>. No
id. So the cascade played every single time that element was inserted —
and playChildren() then handed the entrance to every child without asking
about any of them, though it did carefully remember them.
Turbo replaces <body> wholesale on every visit, so "every time it is
inserted" means every navigation. Coming back out of a conversation built
a fresh <ul> holding the same fifty conversations and cascaded all fifty
in again, on top of whatever was still playing.
The rows are the things with identity, so they are what gets asked now.
Seen before is a redraw and stays still; not seen is an arrival and drops
in. A folder change is all new ids and plays in full, one new mail among
fifty still announces itself, and returning to a list changes nothing.
Suppressed rows are still marked played, remembered and scoped: skipping
that would hand them back to the batch pass, which would give each one
the long single-arrival drop it carries for itself — fifty at once, which
is exactly what the container exists to prevent.
FOUND BY MEASURING, NOT BY READING
Three rounds of instrumenting the real browser. A MutationObserver showed
the list being re-inserted ~3s after every row update; patching innerHTML
and the stream hooks caught nothing; a stack trace finally landed on
Turbo's assignNewBody under onPopState — a whole-body replacement, which
none of the DOM paths I had been reading could have produced.
THE NEW TESTS GO IN AN EXISTING FILE ON PURPOSE
Their own spec file made ten unrelated tests fail. Playwright runs files
in parallel and each seeds its own fixture, so a second new seeding file
raced the others; the failures were all missing fixture rows and every
one passed in isolation. Same lesson the config comments already record
about per-worker users. Merged into list-back-navigation.spec.ts, which
seeds once and is about this exact journey anyway.
Verified both ways: with the fix off, four rows report data-enter on the
way back; with it on, none. The companion test — a first sight still
plays — guards against "fixing" this by simply never animating.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reported: open a mail, leave it a few seconds, press the browser's back
button, and the row is bold again — unread, still counted — until the
page is reloaded by hand.
Opening a conversation is a Turbo Drive visit, so Turbo snapshots the
list on the way out. It leaves BEFORE the read: mail--thread-read waits
two seconds and then posts, and the stream that post renders back targets
a row which is not on the thread page, so the snapshot is never corrected
either. Back restored a picture taken before the read, with no request
made that could have known better.
A poll repairs it a few seconds later. That is why the report says
"sometimes", why a reload always fixed it — and why this took three
attempts to test.
The mail lists opt out of the snapshot cache now. Back re-fetches, which
costs one request on a screen whose only purpose is to show current
state, and whose contents may have changed for a dozen other reasons
while the conversation was open. "It should never load a stale list."
TWO WAYS THE TEST WAS WORTHLESS FIRST, both written down in the spec
1. toHaveAttribute RETRIES for five seconds, so it waited for the poll
that repairs the list and then reported success about a list the
user never saw. Passed against the bug. The read is one-shot now.
2. CLICKING A ROW DOES NOT NAVIGATE at the test viewport — it swaps the
reading pane in and leaves the URL on /mail/inbox, so page.goBack()
went somewhere else entirely and the sequence proved nothing. The
visit has to be a real one.
Both were found the same way, and it is the only way that works here:
disable the fix and watch the test pass anyway. list-back-navigation.spec
next door carries the same warning about the same button, learned about a
different symptom. With the fix off this reports data-unread="true"; with
it on, "false".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reported from the live install, and reported WELL — the log entry carried
the whole previous chain down to the root, which is what v0.1.38 was for:
HandlerFailedException: Handling "SyncAccountMessage" failed
-> Mercure RuntimeException: Failed to send an update
-> TransportException: Failed to connect to mercure port 80
Mail fetched, threaded and committed. Then the handler failed and
Messenger sent it back for retry, because the update saying "your mailbox
changed" could not be published. The doorbell failing does not un-deliver
the parcel.
FIVE OF SEVEN PUBLISHERS HAD NO GUARD
CalendarNotifier and RuleRunNotifier already caught. SyncNotifier,
InsightNotifier, JobNotifier and SendOutcomeNotifier did not, and two of
those are worse than the one that was reported:
· JobNotifier is called from RunBulkStatusHandler's own catch block, so
an outage there would replace the real failure with this one.
· SendOutcomeNotifier announces the result of SENDING MAIL.
All of them record and continue now. RuleRunNotifier caught without
binding — `catch (\Throwable)` — so an outage and a bug in it were
equally invisible; it logs too.
Level comes from ThrowableSeverity, so an outage is a warning an operator
can act on and a programming error is still an error.
ALSO IN THIS COMMIT
The details panel gains a second category line showing what plMail's own
rules would have said, on messages Gmail classified. Gmail's CATEGORY_*
labels are authoritative while they arrive, which means the local cascade
never runs on that mailbox and takes over the moment they stop — untested
against real mail on the day it matters. Both answers together make it
tunable in advance. Suppressed where Gmail is not deciding, since it
would then repeat the line above.
And the blocked-images bar moves above the message, under the attachment
chips. Below the body it was past the fold on anything longer than a
screen: gaps everywhere, scroll to the end to learn why, scroll back. It
is the control for reading the message rather than a note about it. Still
outside the frame — that part is not negotiable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reported from the live install: one search for "amazon", 2.1 seconds mean
over two executions, from the slow-query panel.
The statement joins `thread_label` and `label` unconditionally. Exactly
two filters ever read them — `label:` and the mailbox role behind `in:` —
and a free-text search reads neither. Both joins are to-many, so every
(thread, matching message) row was multiplied by the number of labels the
conversation carries, and the GROUP BY then collapsed the duplicates back
down. A mailbox whose threads sit in an inbox, a category and a user
label did three times the work to build a set it discarded.
The answers were always right. That is why it survived: GROUP BY hides
the duplication perfectly, and the only symptom is the clock.
Joined now only when something asks about a label.
THE TEST IS ABOUT THE SQL, NOT THE RESULTS
Unusual here and worth stating: there is nothing observable to assert on.
The result set was identical before and after, so an assertion about rows
could not fail if the joins came back. The test reads the built statement
and requires the label tables to be absent for free text and present for
`label:` and `in:`.
ALSO: a flaky booking spec, fixed properly
calendar-sharing checked a weekday box with check({force:true}). The input
is `sr-only` — clipped to one pixel and out of flow — so there was never
anything to click, and once the modal grew tall enough to scroll it
failed with "Element is outside of the viewport": a message about
geometry for what was really a wrong target. It clicks the <label> that
wraps the input now, which is what a person clicks and what toggles the
box. Matching that label by text does not work either, because the
template indents and its textContent is "\n Sat\n " — so it goes from
the input to its ancestor label instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three commands in the handbook, one of which fails silently.
`docker compose pull` can run out of disk part-way through extracting a
layer. `up -d --wait` then brings the stack up on the image that is
ALREADY there and reports every container Healthy — so a deploy that
changed nothing is indistinguishable from one that worked, and the only
evidence is a line buried in the pull output. v0.1.36 was believed
deployed for twenty minutes while the demo went on serving v0.1.35.
Piping made it worse. Through a pipe the exit status belongs to whatever
came after it, so the `pull | tail -2` in the handbook's sequence
reported success no matter what the pull did. Nothing in the script is
piped.
And the disk fills on its own: every deploy leaves its predecessor behind
and nothing collects them. Eleven orphaned images is what filled it, on a
host that also runs Apache, MySQL and ClamAV and never has more than a
few GB spare.
So: check for room, prune if short, pull and STOP if that fails, up,
then clear the images the deploy replaced.
· `-a` on the prune is load-bearing. A replaced release is a fully
tagged image nothing references any more, and dangling-only leaves
every one of them.
· `--volumes` is never passed. The demo's database is in one.
· compose.proxy.yaml is included by existence rather than by name: it
belongs to a host that already has something on port 80 and is not in
this repository.
Verified on the real demo host, both ways — a normal run reports "image
unchanged" and exits 0, and DEMO_MIN_FREE_MB=999999 refuses, exits 1, and
leaves the running stack alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three changes, one theme: the log was recording that something failed
without recording anything you could act on.
1. NINETY-FOUR ENTRIES KEPT THE SENTENCE AND DROPPED THE EVIDENCE
`'error' => $e->getMessage()` is a string. No class, no file, no line, no
previous exception. Forty-eight of those sites log at error or critical,
which is precisely where a trace is the point of the entry.
Every one now passes the exception ITSELF alongside the message, which
DoctrineLogHandler expands into class, file, line, trace and the whole
`previous` chain. Purely additive — the existing `error` key stays, so
anything reading it still works.
2. THE IMAGE PROXY CALLED EVERY FAILURE ROUTINE
One catch(\Throwable) covered "the host did not answer" and "this code is
broken" and wrote both at info, below what is stored. That is not a
hypothetical: a typo in the commit before this one threw a fatal, was
caught as a fetch problem, and vanished.
Split. Transport failures keep info, because an unreachable host is not
an application fault and any mailbox with newsletters produces a steady
trickle. Everything else is error. Still not rethrown — this is one <img>
among dozens and a 500 per image would break the reading pane over
something now plainly visible.
3. THE SAME TRAP IN FIVE MORE PLACES
Listing an IMAP folder, testing a UID's presence, purging a remote
message, issuing a Mercure cookie, running a calendar mapper. All catch
everything and continue, all correctly — those failures are weather.
But \Throwable covers \Error, and an \Error is never weather. Undefined
method, wrong type, wrong argument count: only this codebase can cause
one. At info, under the stored threshold, it is invisible.
ThrowableSeverity::level() puts that reasoning in one documented place
and those sites now choose their level from what they actually caught.
Control flow is untouched — same failures tolerated, same values
returned.
It is deliberately a FLOOR, not an oracle. Library exceptions are all
\Exception subclasses so it cannot tell an outage from a bad header, and
it does not pretend to. A site that can name what it expects should name
it, the way the proxy above now names the HTTP client's interface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The reported bug, and the cause was one condition.
if (false === str_starts_with($contentType, 'image/')) {
return null;
}
Amazon's return-label QR codes live in an S3 bucket whose objects were
uploaded with no content type, so S3 answers `application/octet-stream`.
Fetched directly, the URL from the report is HTTP 200,
`application/octet-stream`, 8001 bytes of PNG — a perfectly good image
this proxy refused because the sender declined to name it. The reader got
a blank box where their barcode should have been.
That is not an Amazon quirk. Any object put in a bucket without a type
comes back this way, and a great deal of the web's images are in exactly
that state.
WHAT CHANGED
An unlabelled response — no content type, or one of the octet-stream
spellings — is now read and identified from its own magic bytes, and
served as what it actually is.
This is LESS trust in the sender, not more. The declared type is
discarded and the content decides. A type that is declared and is not an
image is still refused before a byte of body is read, because that is the
error page served with a cheerful 200 and there is no reason to download
it.
SVG
The sniffed type is held to InlineDisposition's list, which excludes SVG
on purpose: it is the one image format that can carry script, and serving
one would be serving script from plMail's own origin. "The sender did not
label it" must not become the way past that, so there is a test that
feeds the proxy an unlabelled SVG with a <script> in it and requires a
refusal.
FOUND BY ACCIDENT WHILE FIXING THIS
`fetch()` wraps `request()` in `catch (\Throwable)` and logs at info. A
typo'd method name in the new code was swallowed by it and surfaced as
"fetch failed" — an outright fatal reported as a routine fetch problem,
one level below what gets stored. Left alone here because narrowing it is
a change of its own, but it is worth knowing the net is that wide.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
