v0.2.31 could not mint a token at all on a TrueNAS install, and nothing an
operator set would have fixed it.
That release bound `iss` and `aud` to MERCURE_PUBLIC_URL. truenas.compose.yaml
leaves that BLANK ON PURPOSE — the setup screen asks for the public URL on first
boot and stores it in the database, which is long after the hub container has
had to be told what to trust. An empty issuer is not a weak issuer: the factory
refuses it outright with `The "iss" additional claim is required by RFC 9068
access tokens`, so there was no subscriber cookie, every publish threw, and the
symptom was a red dot indistinguishable from a hub that was simply down.
The mistake underneath was deriving a value that must exist at CONTAINER START
from one that only exists after SETUP. So it is a literal now, shared by the
application and every compose file, and no installation has to know it. An
identifier does not have to resolve — `.invalid` is reserved by RFC 2606 for
exactly this — and all that matters is that both halves agree on the same
string, which a literal guarantees and a derived value cannot. The same string
everywhere costs nothing: the signing secret is generated per installation, so
one install's token has never been usable against another's hub however it
identifies itself.
truenas.compose.yaml had been left behind twice more. It still carried the
"/healthz" check that v0.2.30 replaced in the other two files, so its hub would
have gone unhealthy the moment the image moved; and it had no pull_policy, so
the image never moved at all. That is the combination that produced the original
report: an application on the 1.0 protocol talking to a 0.x hub that nothing had
ever pulled.
The regression test empties MERCURE_PUBLIC_URL rather than reading the config,
because what broke was the derivation — a test that read the same placeholder
the code did would have passed throughout.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
v0.2.31 moved plMail to the Mercure 1.0 protocol and assumed the hub had moved
too. On a real install it had not, and the result is the mirror image of the
bug that release fixed: plMail speaking 1.0 to a 0.x hub, red dot, no live
updates, nothing in plMail's own logs to say why.
A TAG IS A POINTER, NOT A SUBSCRIPTION, which is the whole of it. Compose
defaults to pull_policy: missing — it pulls only when the image is absent, so
`dunglas/mercure` present on the box means "use it" and the registry is never
asked whether the tag has moved. Restarting pulls nothing either. The app image
updates because the app is the thing being updated; the hub is a sidecar nobody
mentions, and it sits on the digest it first resolved. Leaving the tag unpinned
therefore bought exactly nothing: the tag floated and the image did not.
pull_policy: always is the missing half of the decision not to pin. It costs one
registry round trip per `up` that answers "same digest" almost every time.
The changelog entry said "Nothing to configure" and was wrong three ways: the
hub image must be pulled, an install with its own compose needs
MERCURE_TRUSTED_ISSUERS and a pinned resource_identifier that no image pull
supplies, and a plain-HTTP install needs MERCURE_COOKIE_NAME. It now says all
three, and leads with the pull, because that is the one that bites an install
that changed nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Live updates were dead: every subscription came back 401 and the inbox sat on
"Live updates unavailable" while the hub answered its own healthcheck happily.
The hub had moved to the 1.0 protocol and now wants RFC 9068 access tokens — an
`iss` it has been told to trust, an `aud` naming the hub the token is for, and
`authorization_details` in place of the bare `mercure` claim. plMail was minting
the older shape.
NOT HAND-ROLLED CLAIMS. symfony/mercure already implements 1.0, so this is
configuration: `protocol_version: '1.0'` plus the three claims the bundle
requires (it refuses to build the container without `iss`, `sub` and
`client_id`, which is how the requirement was found). They are merged into the
token FACTORY as well as the publisher provider, so the subscriber cookie gets
them too — one block covers both halves.
The protocol switch also changes the subscription query parameter from `topic`
to `match` and the cookie's name. Both belong to the bundle — the `mercure()`
Twig helper writes the URL, Authorization writes the cookie — which is why no
JavaScript and no controller is touched here.
`resource_identifier` IS PINNED ON THE HUB, and that is not decoration. A 1.0
hub works its expected audience out from each request, and plMail reaches it by
two roads: the browser subscribes at MERCURE_PUBLIC_URL, the application
publishes to the internal MERCURE_URL. One token cannot name both, so a hub
left to derive its own answer would refuse every publish. Issuer and audience
are both read from MERCURE_PUBLIC_URL so the app and the hub cannot be
configured into disagreeing.
THE `__Secure-` COOKIE is the one thing an operator may have to know about. 1.0
names the cookie `__Secure-mercure_access_token`, and a browser drops a
`__Secure-` cookie that did not arrive over TLS. Right for every real install,
fatal for the test stack, which serves plain HTTP on 127.0.0.1 — so the name is
a parameter defaulting to the secure one, and only the test stack overrides it,
with the hub told the same name.
Verified in a browser against the current hub image: the indicator reads
"connected", and the subscription is GET /.well-known/mercure?match=mail%2Fuser
%2F1 → 200. Full suite green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follow-up to eca0f8b7, which pointed our healthcheck at the hub's new
readiness path. Correct, and still the wrong shape: it leaves plMail asserting
something about the internals of an image that is deliberately unpinned so it
can update itself. The next time the hub moves that path, CI goes red for
another three days.
The image has always shipped its own HEALTHCHECK — wget against
127.0.0.1:2019/mercure/health/ready, 60s grace, 5s timeout, 5 retries — which
is the same grace and retries compose.yaml was spelling out by hand. Inheriting
it costs nothing there and cannot drift, because it travels with the image it
describes.
The test stack wanted to keep its five-second interval and inherit the command,
and Compose refuses: a healthcheck block without a `test` is rejected with
"must define a healthcheck command for all healthchecks". So owning the cadence
means owning the command again, which is the thing that rotted. It asks every
30s now — a few more seconds before a job starts, against never again waiting
for ever.
Verified against the current image: the running container carries the image's
healthcheck verbatim and reports healthy, and the test stack comes up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every E2E job has failed since 18 September with "dependency failed to start:
container pl_mail_test-mercure-1 is unhealthy". The hub was fine. Our
healthcheck was not.
dunglas/mercure is pinned to a floating tag, and the image moved its readiness
probe. "/healthz" now answers 404; `curl -f` treats a 404 as failure; the
container is marked unhealthy for ever while serving every subscription put to
it. Verified against the current image: /healthz gives 404, and
127.0.0.1:2019/mercure/health/ready gives 200 — which is the path the image's
OWN healthcheck has always used, and a container left to use it reports healthy.
Overriding the image's healthcheck with a URL of our choosing is precisely what
let the two drift apart, so the fix is to ask where it asks.
compose.yaml carried the same line, and that half was shipped. Nothing there
depends on the hub being healthy, so live updates kept working and the symptom
was an install permanently showing "(unhealthy)" in docker ps and in the
TrueNAS view — about nothing at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RunBulkStatusHandler exists to take a mark-read or archive over a whole view
out of the request — its docblock quotes the thirty-second timeout that
produced it, and BulkStatusController answers with an indicator so nobody
waits. Neither was true: the message was never listed in messenger.yaml's
routing map, and Messenger handles an unrouted message inline in whichever
process dispatched it. Every whole-view action ran inside the web request,
under the limit it was written to escape, at request concurrency — which is
where the production deadlock came from.
It gets a queue of its own rather than joining maintenance, because somebody
is watching an indicator for it and queueing it behind an embedding backfill
would be slower than the inline version it replaces. That means one new
container, worker-bulk, in every compose file and in the handbook.
The deadlock itself is retried now instead of failing the job, three times per
chunk, and a collision that outlasts those hands the job back to the queue
rather than ending it. The failure path no longer writes through an
EntityManager Doctrine has closed, which is what it was doing for every
failure that came out of a flush.
And the list catches up: nothing ever refreshed it on the strength of the job,
only on the toolbar's own "the write landed" event, which is now the instant
the work was QUEUED rather than finished.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
docker compose resolves ${VAR} against the .env file beside compose.yaml,
and the .env beside ours is Symfony's — committed, full of development
defaults, and never written with compose in mind. So APP_ENV: ${APP_ENV:-prod}
was not a default: .env says APP_ENV=dev, and that is what every operator who
ran `git clone && docker compose up -d` actually got.
The published image is built --no-dev and has no DebugBundle, which
bundles.php loads in dev. Every bin/console call died on the missing class
before it could do anything — including the entrypoint's database probe, which
then reported "The database is not up or not reachable" while the database sat
there healthy. Six services in a restart loop, and the message pointing at the
cause was not the one printed.
APP_ENV is hardcoded to prod here now. Running this stack in dev is what
compose.override.yaml.dist is for, and it still merges over the top.
Verified the way it broke: a fresh clone, `docker compose up -d`, nothing else,
against the published image — all eight containers healthy, APP_ENV=prod
inside, and the setup screen served.
ComposeEnvironmentTest holds every ${VAR} in the shipped compose files to the
rule this turned on: interpolate only what .env leaves alone, or leaves at
exactly the value written in the compose file. DATABASE_URL is the one
documented exemption. Text parsing rather than `docker compose config`, so it
fails in CI without a Docker daemon.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
README says what plMail is and gets you running; CONTRIBUTING says how to
change it. Nothing covered the part in between — using it, operating it,
and registering the Google and Microsoft applications it talks to. That
is docs/: 34 pages across features, installation, providers and
internals, mirrored to the GitHub wiki.
The wiki is a mirror and never a source. bin/mirror-wiki.php flattens the
tree into the wiki's flat namespace, rewrites every relative link,
generates the sidebar from the index so the two cannot disagree, and
stamps each page with a footer saying where it came from. A browser edit
survives until the next push to main. That cost is deliberate:
documentation not versioned with the code drifts from it, and the drift
is invisible until somebody follows an instruction that stopped being
true.
Which is the whole problem, so the checkable half is checked.
DocumentationCoverageTest fails the build when a command is missing from
the reference or documented and gone, when a variable in .env is absent
from the configuration page, when a relative link dangles, when a page is
unreachable from the index or promised and missing, or when a page has no
"Things that bite" section. It cannot check whether a paragraph is TRUE,
and says so — this is the floor, so review attention goes to the part a
machine cannot have an opinion about. CONTRIBUTING has claimed since it
was written that the command table "is part of the definition of done";
nothing enforced it until now, and it was wrong in three places.
Writing it found things. They are fixed here rather than documented
around, because a handbook describing a bug accurately is still a
handbook nobody can follow.
compose.yaml mounted no volume for var/attachments, var/raw
or var/uploads. Workers write those during sync
and the web container serves them back, so each
kept its own copy in its own layer — downloads
404, and everything died on a recreate. The
override and the TrueNAS file both mount them;
the file the README tells an operator to run did
not. The same omission cost real data once
before. See the CHANGELOG for moving what is
currently stranded.
app:calendar:materialise new, nightly. Occurrences are drawn when an
event is saved and nothing moved that window
again, so a weekly standup ran out of dates in
two years and its reminders stopped with them —
silently. The query it runs had existed since
the materialiser did, documented as exactly this
sweep, and had never been called.
Microsoft wizard asked for three of the nine permissions plMail
requests, and a token already issued is not
upgraded when the rest are added later.
GmailWatchService told you to grant Pub/Sub Publisher to
gmail-api@system.gserviceaccount.com. The
account is gmail-api-push@; a topic granted
exactly what the docblock asked for never
delivered a notification.
IcsExporter said it wrote no VALARM. It has written them
since alerts landed.
CLIENT_DEVELOPMENT.md said there was no JMAP calendar API and that
SearchSnippet/get was absent. Both shipped.
CONTRIBUTING said only app:calendar:push registers a push
channel; subscribing has done it since the
subscribe flow learned to.
Three docblocks named getters removed long ago, or described a
method two below the one they sat on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seventeen message types shared one transport and one worker, so they were
ordered by arrival and nothing else. Pressing Send behind a Gmail batch meant
waiting for the batch. The queue was deciding by accident what should be
decided on purpose.
Three now, named for who is waiting. export is anything leaving plMail — the
send itself, the outgoing label and flag pushes, notifier traffic — and is the
only queue a person is watching. ingest is mail arriving and the work that
immediately follows it. maintenance is bulk and deliberate: backfills, a rule
applied to existing mail, admin sweeps.
Three worker processes rather than three transports on one, and that is the
part that matters. Priority ordering does not help: a worker already inside a
long handler cannot pick up a send however the queues are ranked. Only a
separate process makes sending independent of syncing. Measured before
choosing — the existing worker holds 76MB, so two more cost about 150MB
against the 1.1GB the IMAP supervisor already uses, and running three
processes in one container would have meant adding a supervisor the image does
not have, with multiplexed logs and hangs Docker cannot see.
export retries faster than the other two. Its failures are a relay refusing a
connection while somebody watches the outcome, not a rate limit that needs
minutes; five attempts over about ninety seconds rather than eight minutes.
async stays declared and consumed by the maintenance worker, because envelopes
queued before the upgrade still need somewhere to land. Nothing routes there.
Remove both once a release has drained.
Four tests asked the async transport what had been sent and now ask export.
That coupling is deliberate: which queue an outgoing push lands on is the
behaviour, and routing one onto ingest would put a user's archive back behind a
mailbox sync.
Verified by dispatching one of each against the dev stack: send landed on
export, sync on ingest, rule run on maintenance.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NTFY_BASE_URL was one more thing to get right, and it did not need to be: the
host a user sets at first boot is already the address phones can reach, which
is the only property this URL has to have. It now defaults to
http://$SERVER_NAME:8090, so enabling push is the compose command and nothing
else. An explicit NTFY_BASE_URL still wins, for a separate hostname or TLS
terminated in front.
The obvious tidier answer does not work. ntfy cannot be hosted under a
sub-path -- it refuses a base-url containing one at startup -- so it cannot be
folded in behind the app's own Caddy at /push the way the Mercure hub sits at
/.well-known/mercure. Hence a port of its own, and that is a limitation of ntfy
rather than a choice worth revisiting.
The derived URL is http, which is right for the LAN and Tailscale deployments
this is usually reached over and wrong for the open internet: the endpoint URL
is itself the secret, so anyone able to watch the traffic could learn a topic
and publish noise to it. They still could not read anything -- the payload is
encrypted to the device's key before it arrives -- but facing the internet you
want TLS in front and NTFY_BASE_URL set to the https address. The comment says
so where someone deploying it will read it.
Verified with nothing but SERVER_NAME set: the container comes up with the
derived URL, the distributor issues an endpoint on it, and the subscription
comes back from PushSubscription/get verified.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Web Push assumes a push service: something that owns the endpoint URL, holds
the connection to the device and receives our encrypted POST. Browsers ship
one. A native Android app does not, and Android's own is FCM, which speaks its
own protocol and cannot be reached by WebPushSender at all.
That left a self-hoster with a gap this repository was well placed to close and
did not: to get push without Firebase they had to find and run a push service
themselves, having already been asked to self-host their mail.
So ntfy is available here, behind a `push` profile and off by default:
NTFY_BASE_URL=https://push.example.com docker compose --profile push up -d ntfy
It adds no application code. ntfy is simply a push service speaking the
protocol WebPushSender already emits -- RFC 8030 transport, RFC 8291 payload
encryption, RFC 8292 VAPID -- so a UnifiedPush distributor on the phone can
receive what this server already sends today. Nobody who does not want push
pays for a container they never start.
Two settings are worth reading before exposing it. NTFY_BASE_URL is the URL
*phones* use, so it must be reachable from outside this machine, and it is
baked into every endpoint issued -- changing it later forces every device to
re-register. And access is open by default, which is a considered choice rather
than an oversight: UnifiedPush topics are random and unguessable, and payloads
are encrypted to the device's own key before they arrive, so this service
cannot read mail and someone who guessed a topic could only send noise the app
discards. The comment says how to lock it down for an internet-facing
deployment.
Verified end to end against the isolated Android test stack: the app registers
with the distributor, ntfy issues an endpoint, the app registers it through
PushSubscription/set, this server POSTs its encrypted PushVerification there,
the connector decrypts it and echoes the code back. PushSubscription/get then
returns the subscription with no verificationCode, which is what live means.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three faults, one story: the stream could die silently, nothing told the
user, and nothing tested any of it.
**It did not recover.** EventSource retries on its own only when a
connection drops cleanly. An HTTP error response is fatal per spec —
readyState goes to CLOSED and the browser never tries again — and our
Caddy answers 502 for the seconds a hub restart takes. Measured: an open
stream, a 20s outage, and the tab stayed dead after the hub was back.
Every deploy did this to every open tab. There is now a reconnect loop
with jittered backoff, plus an immediate retry when a tab is refocused
or the network returns.
**Nothing said so.** The mail list has no polling fallback, so a dead
stream looks exactly like a quiet mailbox — the inbox stops changing and
nothing on screen explains why. A glow behind the logo now carries the
state, quiet when healthy, and announced rather than colour-only. While
the stream is down the list falls back to a 60s poll: strictly the worse
mechanism, which is why it runs only when the good one is unavailable.
**The hub accepted anyone.** dev.Caddyfile was in use in every
environment including TrueNAS, and it sets `anonymous`. Since
/.well-known/mercure* is proxied publicly, anyone who could reach an
instance could subscribe to mail/user/<any id> and watch that account's
sync activity. Both stacks now run the image's default config, which
requires the subscriber JWT.
That JWT expires — an hour by default, well inside a remember-me
session — and the hub only checks it when a connection is made, so the
expiry is invisible until the next reconnect is refused. /mercure/auth
re-issues the cookie, and the client refreshes it before every retry.
The cookie is attached by a response subscriber rather than by
`mercure(topics, {subscribe: …})` in the layout, because that helper
throws for a hub on another second-level domain and a throw in a
template takes the whole page with it. PHPUnit caught it at once: its
client requests as localhost against a 127.0.0.1 hub, and every
page-rendering test turned 500. An external hub is a legitimate
deployment; it should cost live updates, not the page.
Tests: compose.test.yaml grew a real hub, running the same non-anonymous
config as production so the suite exercises the authorization path
instead of one that accepts anything. The recovery spec stops the
container — `setOffline` does not sever an established SSE connection
and route interception never fires without a request, so both looked
like passing tests of nothing. It lives in the exclusive project: there
is one hub, and taking it away mid-run breaks specs that never asked.
Verified by disabling the reconnect and watching the recovery test fail,
then restoring it and watching it pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
