jody/pl_mail
1
0 Fork 0
mirror of https://github.com/karatektus/pl_mail.git synced 2026-11-02 13:39:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

Leave the ingest worker to fetch mail, and nothing else

Browse Source 
Adding an account put new mail behind hours of follow-up work. What
plMail does to a message once it has it — a model call where the
assistant sorts, event extraction, insight cards, receipts — was queued
on `ingest`, beside the syncs, for every message of the import, oldest
first. One consumer serves that queue. 988 envelopes pending and the
oldest an hour old was a mailbox being asked about newsletters from
2019 while mail on the server waited to be fetched.

`ingest` is now what talks to a mail provider. The rest moved:

  enrich_live     worker-live      follow-up work for mail that has just
                                   arrived on an account whose import is
                                   over; never behind an import
  enrich          worker-enrich    the same for an import's recent mail,
                                   and search indexing
  enrich_backlog  worker-enrich    old mail for the assistant, consumed
                                   only when `enrich` is empty
  release         worker-release   ends a hold (below); nothing else

Which queue is decided per batch, not per message class: the pipeline
asks InitialImportState once, carries the answer on PostIngestResult,
and EnrichmentRouter turns it into a stamp.

Only mail from the last 30 days (ENRICH_RECENT_DAYS) gets follow-up
work on arrival. Older mail is stored, threaded and filed by the rules.
The assistant is asked about it afterwards — newest first, a bounded
handful per run, and not until every account of that person has
finished importing (app:ai:classify-backlog, ClassificationCatchUp).
For the other steps `app:backfill` remains the way to read history.

ExtractInsightsStep and ProposeEventsStep ran their work inside the
sync and argued for it; both dispatch now.

New mail no longer appears in one tab and moves to another. For
somebody sorting by assistant the rules' category is a guess about to
be replaced, so where the verdict could still change it the message is
held: stamped before threading, its new thread given no category, which
every tab query already reads as "in no tab". The answer releases it.
Failing that a timed release does, after AiSettings::holdMaxSeconds —
60 by default — and the mail is shown where the rules put it; the model
call carries on and moves the mail when it answers. A message is asked
about once. Admin → AI has the switch, the ceiling, and the measured
delay over the last week beside them.

Three things about the release that were each wrong once first:

  - It ran on the live worker, so it could not fire while that worker
    was inside the call that was taking too long. It was moved to
    export, which mixed mail appearing with mail leaving. It has a
    queue and a worker of its own.
  - The call was cut off at the ceiling and the message asked about
    again later. The hold and the question are separate; only the hold
    has a deadline.
  - A late verdict was stored and moved nothing. It moves the mail.

Both workers still end at one model host, so bulk classification looks
at the live queue before each call and stands aside (LiveMailPriority).

One migration: two columns on `message`, two on `ai_settings`.

Not verified against a real import or a live model host: the tests pin
the queues, the hold and the release against in-memory transports and a
mocked host. Known and left: IMAP still imports oldest first; a reply
into an existing thread cannot be hidden; a Gmail account counts as
importing for up to an hour after its last message lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Paul LütznerandClaude Opus 5.5 committed 2026-10-07 15:18:44 +02:00
1 parent 464df70473
commit 88b1744385
66 files changed
+3361 -192
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

CHANGELOG.md
+37 -1
View File
@@ -8,6 +8,34 @@ The published image tags: `latest` follows the most recent release below,
## Unreleased
**Adding an account no longer puts new mail behind hours of follow-up work.** Everything plMail does
to a message after fetching it — sorting by the assistant, event extraction, insight cards, read
receipts — was queued beside the fetching, on the one worker that fetches mail. A first import of a
large mailbox therefore queued that work for every message it brought in, oldest first, and new mail
waited behind it; with the assistant sorting mail that was a model call per message and could be
hours. Fetching mail is now the only thing that worker does.
- **Follow-up work has its own queues and workers.** `worker-live` handles mail that has just
arrived on an account whose import is over, and nothing else, so new mail is never behind an
import. `worker-enrich` handles an import's recent mail and then, only when that is done, old
mail. `worker-release` only ends the wait described below. See
[Mail ingest → What follows a message](docs/internals/mail-ingest.md#what-follows-a-message-and-where-it-runs).
- **Only mail from the last 30 days gets follow-up work as it is imported.** Older mail is stored,
threaded, filed by the ordinary rules and run through your own rules, as before — but no events,
insight cards or receipt handling are derived from it. `app:backfill insights` and
`app:backfill event-extraction` read history if you want it.
- **Old mail is sorted by the assistant after the import, not during it** — newest first, in small
batches, in the background, and only once every account of yours has finished importing.
- **New mail waits for the assistant before it appears in a tab.** If you let the assistant sort your
mail, a message used to appear where the rules put it and move a few seconds later when the
assistant answered. It is now kept out of the inbox tabs for those seconds and appears once, in
the right tab. The wait is capped — 60 seconds by default — and past the cap the mail is shown
where the rules put it, and moved if the assistant answers later. Mail from a first import is never held.
- **Admin → AI has a switch for that wait, and shows what it costs.** **Wait for the assistant
before showing new mail in a tab** turns it off, **Wait at most** sets the cap (2–300 seconds), and
the line underneath reports how long mail was actually held over the last 7 days and how often the
wait ran out. See [Administration](docs/features/admin.md).
**"Move to" files a conversation under a label in one step.** There was a label button, which adds a
label and leaves the mail where it is, and dragging onto a folder, which only works with a pointer
and a sidebar in reach. Filing mail the way Gmail does — label it and get it out of the list you are
@@ -30,7 +58,15 @@ back in the inbox. Everything else the conversation wears stays.
### Before you upgrade
- **Nothing has to change.** There is no migration and no new setting.
- **One migration, applied on boot.** It adds two columns to `message` and two to `ai_settings`, and
changes no existing row.
- **Holding new mail for the assistant is on by default**, and applies only to people who sort by
assistant. Switch it off under Admin → AI if your model host is slow.
- **The worker container runs three more processes**, `worker-live`, `worker-release` and `worker-enrich`. Nothing to
configure on the standard images. If you run your own consumers instead of `app:work`, add
`messenger:consume enrich_live`, `messenger:consume release` and `messenger:consume enrich enrich_backlog` — without them
nothing sorts mail, extracts events or indexes for search.
- **Jobs already queued at upgrade are worked through where they are**, on `ingest`, once.
## v0.2.58 — 2026-10-04
CONTRIBUTING.md
+57 -1
View File
@@ -199,7 +199,8 @@ the tag is then what fails. So run the suites locally before pushing anything yo
anyway. `workflow_dispatch` on the Actions tab is the other way to get a full run without tagging.
A tag starts the E2E workflow and the image build **in parallel**, so a failing suite does not stop
the image being published. Tag from a tree you have already run the suites against.
the image being published. Tag from a tree you have already run the suites against — see
[Releasing](#releasing) for what a tag sets in motion and what has to be true before one is pushed.
### How the browser suite stays fast
@@ -256,6 +257,59 @@ Two smaller things: `video` is off (the trace is the useful artifact and costs n
run), and nothing waits for a TOTP window to roll over — otphp accepts a code minted in window `W`
for any submission in `[30W-15, 30W+45)`, so the wait was superstition.
## Releasing
**A `vX.Y.Z` tag is the release, and nothing after it asks again.** Pushing one publishes the image
as `X.Y.Z`, `X.Y` and `latest`, and from there it travels without anybody here doing anything:
1. Every install on `latest` gets it on its next pull.
2. Renovate in [truenas/apps](https://github.com/truenas/apps) sees the new tag — it runs daily at
12:00 UTC and is also started by hand — rewrites the tag and digest in
`ix-dev/community/plmail/ix_values.yaml`, sets `app_version` and bumps the catalogue `version` by
a patch.
3. That lands in the shared "Update updates-patch-minor" pull request, which a TrueNAS maintainer
merges, usually within hours. A major version gets a pull request of its own.
4. The merge publishes the catalogue, and TrueNAS offers the update to everyone who installed plMail
from **Apps → Discover**.
So a tag reaches other people's NAS boxes within about a day, with migrations that run on boot, and
**it cannot be taken back**: there is no cool-down in that Renovate configuration and no way to
unpublish a catalogue version. A bad release is repaired by tagging the next one.
That makes the tag the only gate there is, and two things have to be true before it is pushed:
- **The suites are green on the exact commit being tagged.** Either locally —
`npm run test:unit:docker && npm run test:e2e:docker` — or by starting the E2E workflow on `main`
with `workflow_dispatch` and waiting for it. The run the tag itself starts does not count: the
image is published beside it, not after it.
- **That commit has been installed and used on a real installation.** A push to `main` publishes
`main` and `sha-<commit>` without touching `latest` or the catalogue, which is what those tags are
for: pull one onto the production install, let the migrations run against real data, and read
mail with it. The suites start from an empty database every time; an upgrade never does.
Then, and in this order: rename `## Unreleased` in `CHANGELOG.md` to `## vX.Y.Z — date`, commit it as
`Release vX.Y.Z`, tag that commit and push both. The release commit changes the changelog and
nothing else, so what was tested is what is tagged.
Things that steer what the catalogue receives:
- **No tag, no update.** `main` and `sha-…` are not versions and Renovate ignores them.
- **Two tags before Renovate's next run ship as one update**, the newer one. Nobody on TrueNAS ever
sees the first.
- **A tag of any other shape is not picked up** — `v0.3.0-rc.1` publishes an image but is not a
successor to `0.2.58` as far as Renovate is concerned. It does move `latest`, because
`docker.yml` moves `latest` on anything starting with `v`, so it is not a private tag.
**Renovate moves the image and nothing else.** The catalogue app has a template and an install form
of its own (`templates/docker-compose.yaml`, `questions.yaml`), and a release that needs either
changed — a new required variable, a container added or renamed, a different user — arrives on
TrueNAS as the new image inside the old template. Either keep the image working under the old
template, which is the better answer and the reason `0.2.56` and `0.2.57` exist, or open a pull
request to truenas/apps with the image bump, the template change and a `version` bump in `app.yaml`
together, straight after tagging, and say so on the open Renovate pull request so that one is not
merged first. truenas/apps is the source of that app now; the `pl_mail_truenas` repository is a
working copy to prepare such a change in, and is behind upstream after every release.
## README screenshots
`tests/e2e/screenshots.spec.ts` regenerates the images in `docs/screenshots/`. It asserts nothing and
@@ -790,6 +844,7 @@ php bin/mirror-wiki.php --check
| `app:calendar:push [calendar-id] [--force] [--stop]` | Register and renew Google/Microsoft calendar push channels, so changes arrive instead of being polled for |
| `app:calendar:alerts [--dry-run]` | Deliver the event reminders that have come due, and prune the records of ones long past. Runs every minute; `--dry-run` lists what is due without sending or recording anything |
| `app:mail:wake-snoozed` | Return snoozed conversations whose time has come. Runs every minute |
| `app:mail:release-held` | Show mail that has been held for the assistant longer than **Wait at most** allows. Runs every minute and finds nothing on an ordinary day — the live worker releases held mail itself. It is the backstop for that worker being down, and it warns when it has to act |
| `app:calendar:materialise [--dry-run]` | Redraw the occurrences of recurring events whose horizon no longer reaches far enough. Runs nightly; without it a long-untouched series eventually runs out of dates |
| `app:backfill [task]` | Run a one-off backfill over stored data; with no argument it lists the tasks and asks. `events` re-runs calendar extraction, `proposals` re-reads mail for dates written in prose, `safe-html-table-links` restores links that were wrapped around a table and came out empty |
| `app:imap:idle <mailbox-id>` | Hold an IMAP IDLE connection for a single mailbox |
@@ -818,6 +873,7 @@ php bin/mirror-wiki.php --check
| `app:db:migrate` | Run pending migrations under a lock, so several containers booting together cannot collide. This is what the entrypoint calls; run it by hand only when a boot was interrupted |
| `app:ai:embed-mailbox --email=… \| --all` | Queue a pass that embeds an existing mailbox for semantic search. Needed once after switching the feature on, and again after changing the search model. Runs on the maintenance worker a chunk at a time, takes hours on a large mailbox, and is safe to interrupt and safe to repeat — it skips whatever is already embedded under the current model, so changing the model is how a re-embed is asked for. This is the whole-mailbox job: it claims a state row, resumes where it stopped and can be paused from Admin → AI |
| `app:ai:index-new-mail [--limit=N] [--email=…]` | Queue a bounded catch-up over mail that arrived and has not been indexed for semantic search. Runs nightly at 03:20, newest mail first, at most `--limit` messages per mailbox (500 by default). It is the BACKSTOP: mail is normally indexed in the warm minutes after somebody searches, because search and the indexer share the same small embedding model. Deliberately bounded — an unbounded version would be `app:ai:embed-mailbox` with no state row and no pause button, so a mailbox that is genuinely far behind wants that command instead |
| `app:ai:classify-backlog [--limit=N] [--email=…]` | Queue a bounded pass of old mail for the assistant to sort: the newest mail it has never been asked about, at most `--limit` per mailbox (200 by default). Runs every quarter of an hour, and posts nothing while an account of that person is still importing, while the last pass is still queued, or while somebody is using the composer. This is how a mailbox imported before today gets sorted without the import waiting on a model |
| `app:ai:categorise-check` | Ask the configured model to sort ten messages whose right answers are known, and print the score. For judging a change to the categorisation prompt — from Admin → AI or in `PromptRules` — which can otherwise only be judged by reading it, and for finding out whether the model an installation actually runs is up to the job. Not a phpunit test on purpose: it needs a live host, and a fake would be a test of the fake. The fixtures are shaped exactly as `ClassifyMailHandler::describe()` shapes a real message, bulk-header line and all, because a benchmark against a different input format measures a prompt nobody sends |
| `app:ai:prune-metrics [--days=N]` | Prune recorded model-call timings older than the retention window (30 days by default). Runs nightly at 05:10. The table holds counts and durations only — no prompt, no completion, no message id — so pruning it forgets how fast the box was last month and nothing else. Its own command rather than a fourth window on `app:monitoring:prune`: this is the retention of a feature, empty on installations that never switched the AI on and a hundred thousand rows after an afternoon's backfill |
| `app:demo:reap [--dry-run]` | Delete demo visitors whose time is up, and everything they own. Only does anything when `APP_DEMO_MODE` is on; runs every 10 minutes there. See "Demo mode" |
compose.yaml
+1 -1
View File
@@ -119,7 +119,7 @@ services:
condition: service_healthy
# ── Everything that is not the web server ─────────────────────────────────
# One container, seven processes (app:work): the IMAP supervisor, one
# One container, ten processes (app:work): the IMAP supervisor, one
# consumer per queue, the scheduler, and the Mercure hub. They used to be
# seven containers of this same image and the hub's own; see
# App\Service\System\Work\BackgroundProcesses for what each is for.
config/packages/messenger.yaml
+136 -21
View File
@@ -37,7 +37,8 @@ framework:
max_retries: 5
max_delay: 60000
# Mail coming in, and the work that immediately follows it.
# Mail coming in, and nothing else. What follows it has queues of its
# own below — see enrich_live.
ingest:
dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
options:
@@ -57,6 +58,110 @@ framework:
max_retries: 5
max_delay: 300000
# What follows mail arriving, and none of it is fetching.
#
# THREE QUEUES (and `release` above them), BECAUSE `ingest` USED TO HOLD ALL OF THIS AND THE
# FETCHING WAITED ON IT. Classification, event extraction, receipts
# and embeddings were routed beside the syncs on the reasoning that
# they were "mail arriving". On a first import that put an hour of
# model calls in front of the next sync: 988 envelopes pending, the
# oldest an hour old, and mail that had reached the server sitting
# unfetched behind a newsletter from 2019 being asked what it was.
# `ingest` is now only what talks to a mail provider.
#
# They are separate from each other for the reason the first three
# were: who is waiting.
#
# enrich_live mail that has just arrived on an account whose
# import is over. Somebody may be looking at the
# inbox, and with sorting by assistant the mail is
# held out of its tab until this answers — so it
# has a worker of its own and nothing bulk is ever
# in front of it.
# enrich recent mail brought in by an import. Wanted, not
# watched.
# enrich_backlog old mail being classified after the import, a
# bounded handful at a time. Nobody's foreground.
#
# The last two share one worker, consumed in that order, which is
# what "recent first" means: Messenger drains an earlier transport
# before it looks at a later one. Two workers would only queue on the
# same model host. See BackgroundProcesses.
#
# enrich_live and enrich_backlog are reached by TransportNamesStamp
# and by nothing in the routing table below — the same message is
# live or bulk depending on the mail it carries, and only the code
# dispatching it knows which. See App\Service\Mail\PostIngest\EnrichmentRouter.
enrich_live:
dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
options:
queue_name: enrich_live
# The export ladder: this is the one enrichment queue somebody
# is waiting on. A held message is released by its own timeout
# long before a patient retry would come round, so the only
# retry worth having is a prompt one.
retry_strategy:
delay: 2000
multiplier: 3
max_retries: 5
max_delay: 60000
# Ending a hold, and nothing else.
#
# A message held out of the inbox tabs for the assistant is shown
# when its wait is over, whatever the model is doing — see
# ReleaseHeldMailMessage. That job is a flag and one UPDATE, and it
# has a queue and a worker to itself anyway, because every other
# place it could go has something long in it at exactly the wrong
# moment:
#
# enrich_live is where the mail is being classified. Its worker
# may be inside the very model call that is taking
# too long, and the release would wait for the call
# it is the limit on.
# export was where it went first. A release is quick enough
# not to hold up a send, but a send — or an
# attachment upload — can hold up a release, and
# mail leaving and mail appearing have no business
# sharing a worker.
#
# So this queue is empty nearly always and its worker idle nearly
# always, which is the point: nothing is ever in front of a release.
release:
dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
options:
queue_name: release
# The export ladder: somebody is waiting, and the only thing
# that fails here is a database blink.
retry_strategy:
delay: 2000
multiplier: 3
max_retries: 5
max_delay: 60000
enrich:
dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
options:
queue_name: enrich
# The ingest ladder, for the ingest reason: what fails here is a
# model host restarting or a provider rate limit on a raw fetch,
# and both take minutes to clear.
retry_strategy:
delay: 5000
multiplier: 3
max_retries: 5
max_delay: 300000
enrich_backlog:
dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
options:
queue_name: enrich_backlog
retry_strategy:
delay: 5000
multiplier: 3
max_retries: 5
max_delay: 300000
# A whole-view action: mark every unread read, archive the lot.
#
# ITS OWN QUEUE RATHER THAN maintenance, and the difference is who
@@ -198,25 +303,35 @@ framework:
App\Infrastructure\Messaging\Message\ApplyLabelStructureMessage: export
App\Infrastructure\Messaging\Message\ApplyMailRuleMessage: maintenance
App\Infrastructure\Messaging\Message\UploadAttachmentsMessage: export
App\Infrastructure\Messaging\Message\ExtractEventsMessage: ingest
# ingest too, and for the same reason — it is mail arriving. It is
# much slower than its neighbours (a round trip to a model host,
# seconds on a cold model), but it is also dispatched only when
# somebody switched the feature on, and it narrows to the mail no
# rule recognised. A transport of its own would buy isolation from
# a queue that is empty on almost every installation.
App\Infrastructure\Messaging\Message\ClassifyMailMessage: ingest
# ingest as well: it is mail that arrived, only caught up with
# later, and it is the same shape of work — one round trip per
# message to a box on the LAN. Nothing dispatches it on arrival any
# more; the two triggers are a nightly sweep and the warm window
# after somebody searched (see App\Service\Ai\EmbeddingCatchUp), and
# both post it in chunks a worker can finish rather than one
# envelope holding a whole mailbox. Maintenance would be the other
# enrich, with everything else that happens to mail once it is in.
# These were on ingest; see the transport's note for what that cost.
# A batch of mail that has only just arrived is stamped onto
# enrich_live instead, by the step that dispatches it.
App\Infrastructure\Messaging\Message\ExtractEventsMessage: enrich
# Both used to run INSIDE the sync, on the worker holding the IMAP
# connection — the one thing PostIngestStepInterface says a step
# must not do. They are deterministic and quick per message, and
# that was the argument for leaving them inline; it stops being one
# at fifty messages a batch for a whole mailbox.
App\Infrastructure\Messaging\Message\ExtractInsightsMessage: enrich
App\Infrastructure\Messaging\Message\ProposeEventsMessage: enrich
# A queue of its own; see the `release` transport for why.
App\Infrastructure\Messaging\Message\ReleaseHeldMailMessage: release
# enrich, and it is the reason the queue exists. It is the slowest
# thing here by an order of magnitude — a round trip to a model host
# per message — and on ingest one envelope of it held the only
# consumer that fetches mail for minutes.
App\Infrastructure\Messaging\Message\ClassifyMailMessage: enrich
# enrich as well, and no longer ingest: it is one round trip per
# message to a model host, which is exactly the work that must not
# sit in front of a sync. Nothing dispatches it on arrival; the two
# triggers are a nightly sweep and the warm window after somebody
# searched (see App\Service\Ai\EmbeddingCatchUp), and both post it in
# chunks a worker can finish. Maintenance would be the other
# candidate and is the wrong one: a batch queued while somebody is
# sitting at their search results should not be behind an hours-long
# backfill on the queue built to hold hours-long backfills.
App\Infrastructure\Messaging\Message\EmbedMessagesMessage: ingest
App\Infrastructure\Messaging\Message\EmbedMessagesMessage: enrich
# maintenance, NOT ingest. A backfill re-dispatches itself for hours
# and would otherwise sit in front of every arriving message on the
# ingest queue — new mail would stop appearing until an old mailbox
@@ -239,11 +354,11 @@ framework:
#
# export for the send: it is mail leaving plMail, and it wants that
# queue's tighter retry ladder, because the failures are a relay
# refusing a connection. ingest for the scan: it is work that
# immediately follows mail arriving, and its retries should be the
# patient ones.
# refusing a connection. enrich for the scan: it is work that
# follows mail arriving without being the fetching of it, and its
# retries should be the patient ones.
App\Infrastructure\Messaging\Message\SendReadReceiptMessage: export
App\Infrastructure\Messaging\Message\ProcessReadReceiptsMessage: ingest
App\Infrastructure\Messaging\Message\ProcessReadReceiptsMessage: enrich
# ingest rather than export, although a calendar sync writes
# outward as well as reading. Nobody waits on it: a local edit is
# saved and on screen before the push is dispatched, so the send
config/services.yaml
+12
View File
@@ -97,6 +97,15 @@ parameters:
app.ai_backfill_retry_seconds_default: 60
app.ai_backfill_retry_seconds: '%env(default:app.ai_backfill_retry_seconds_default:int:AI_BACKFILL_RETRY_SECONDS)%'
# How old a message may be and still have things done to it on arrival —
# event extraction, insight cards, receipts, and a model call where the
# assistant is on. Thirty days: the same window MessageThreader gives a
# subject match, and about what is plausibly still in somebody's view. See
# App\Service\Mail\PostIngest\RecentMailPolicy. Not a row in the
# installation reference, for the reason the block above is not.
app.enrich_recent_days_default: 30
app.enrich_recent_days: '%env(default:app.enrich_recent_days_default:int:ENRICH_RECENT_DAYS)%'
services:
# default configuration for services in *this* file
_defaults:
@@ -156,6 +165,9 @@ services:
$pauseMs: '%app.ai_backfill_pause_ms%'
$cooldownSeconds: '%app.ai_backfill_cooldown_seconds%'
$retrySeconds: '%app.ai_backfill_retry_seconds%'
App\Service\Mail\PostIngest\RecentMailPolicy:
arguments:
$recentDays: '%app.enrich_recent_days%'
App\Service\Mail\AttachmentThumbnailer:
arguments:
$projectDir: '%kernel.project_dir%'
docs/de/features/admin.md
+19 -1
View File
@@ -1,4 +1,4 @@
<!-- translated-from: features/admin.md sha1:68b7587fcf7aa419435e09dab064077c8e01c68f -->
<!-- translated-from: features/admin.md sha1:070a1c1c3254e1683e96281a9cef8e0d4571572b -->
# Administration
@@ -474,6 +474,24 @@ den Stichentscheid geben — wenn ein Header, ein Gmail-Label oder die Tatsache,
schon geschrieben hast, die Kategorie bereits bestimmt hat, wird das Modell gar nicht gefragt.
Ausschalten erfordert kein Aufräumen.
Zwei Dinge sorgen dafür, dass das Einsortieren ankommender Mail nicht im Weg steht:
- **Nur aktuelle Mail wird beim Eintreffen einsortiert.** Nach Mail aus den letzten 30 Tagen wird
sofort gefragt. Wenn du ein Konto hinzufügst, wird der Rest des Postfachs danach einsortiert, im
Hintergrund, die neueste zuerst, und erst, wenn der Import abgeschlossen ist — ein erster Import
besteht also nicht aus stundenlangen Modellaufrufen, hinter denen neue Mail wartet.
- **Auf den Assistenten warten, bevor neue Mail in einem Tab erscheint.** Für alle, die ihre Mail
vom Assistenten sortieren lassen, bleibt neue Mail für den Moment, den der Assistent braucht, aus
den Posteingangs-Tabs heraus — sie erscheint dann einmal, im richtigen Tab, statt in einem
aufzutauchen und in einen anderen zu wandern. **Höchstens warten** ist die Obergrenze, 60
Sekunden als Vorgabe: Danach wird die Mail dort angezeigt, wo die normalen Regeln sie einordnen,
und verschoben, falls der Assistent später antwortet. Es ist eine Obergrenze, nicht die
Wartezeit — ein bereits geladenes Modell antwortet in ein, zwei Sekunden.
Die Zeile darunter nennt, wie lange Mail in den letzten 7 Tagen tatsächlich zurückgehalten wurde
und wie oft die Wartezeit ablief — ist das meistens der Fall, ist dein Modell-Host zu langsam,
als dass sich das Warten lohnt, und du kannst es abschalten. Mail aus einem ersten Import wird
nie zurückgehalten.
Die Schreibhilfe ersetzt nie, was du geschrieben hast. Sie hängt an, dein Original bleibt stehen.
Zusammenfassungen sind die teuerste der vier. Sie nutzen dasselbe Schreibmodell, aber ein ganzes
docs/de/install/docker.md
+5 -2
View File
@@ -1,4 +1,4 @@
<!-- translated-from: install/docker.md sha1:73ae0d9a9b349c20c36dcfbc031c792cf1631649 -->
<!-- translated-from: install/docker.md sha1:c5abef947d51031887eb536d6c09b3784406aa96 -->
# Installation mit Docker Compose
Der unterstützte Weg, von Anfang bis Ende: was du brauchst, was `docker compose up` tatsächlich tut,
@@ -148,7 +148,10 @@ bis zu dreißig Sekunden, wenn er abgestürzt ist.
| `mercure` | Der Mercure-Hub, 1.x | Live-Aktualisierungen — die Mailliste, die sich von selbst auffrischt |
| `imap-supervisor` | `app:imap:supervise` | Startet und überwacht je einen `app:imap:idle`-Prozess pro IDLE-fähigem Postfach, damit gewöhnliche IMAP-Mail in dem Moment ankommt, in dem sie eintrifft |
| `worker-export` | `messenger:consume export` | Alles, was plMail verlässt, und die einzige Warteschlange, auf die jemand wartet. In einem eigenen Prozess, damit ein Versand nie hinter einem Abgleich steht |
| `worker-ingest` | `messenger:consume ingest` | Eingehende Mail und die Arbeit, die unmittelbar darauf folgt |
| `worker-ingest` | `messenger:consume ingest` | Mail abholen, und nur das |
| `worker-live` | `messenger:consume enrich_live` | Was auf Mail folgt, die gerade eben angekommen ist — Einsortieren, Terminerkennung, Insight-Karten, Lesebestätigungen. Eigener Prozess, damit neue Mail nie hinter einem Import steht |
| `worker-release` | `messenger:consume release` | Zeigt Mail an, die für den Assistenten zurückgehalten wurde, sobald ihre Wartezeit vorbei ist. Fast immer untätig, und in einem eigenen Prozess, damit nie etwas davor steht |
| `worker-enrich` | `messenger:consume enrich enrich_backlog` | Dieselbe Arbeit für die aktuelle Mail eines Imports, danach — nur wenn diese Warteschlange leer ist — alte Mail, die der Assistent einsortiert |
| `worker-maintenance` | `messenger:consume maintenance async` | Nachträgliche Verarbeitungen, Regelläufe über vorhandene Mail, administrative Durchläufe. Leert außerdem die stillgelegte Warteschlange `async` |
| `worker-bulk` | `messenger:consume bulk` | Aktionen über eine ganze Ansicht: alles Ungelesene als gelesen markieren, alles archivieren. Eigener Prozess, weil jemand dabei auf eine Fortschrittsanzeige schaut und das nicht hinter einer nachträglichen Verarbeitung warten darf |
| `scheduler` | `messenger:consume scheduler_default` | Löst alles Wiederkehrende aus. **Ohne diesen Prozess plant sich nichts von selbst** |
docs/de/install/troubleshooting.md
+7 -3
View File
@@ -1,4 +1,4 @@
<!-- translated-from: install/troubleshooting.md sha1:978f8c024df0873bd3f40023340d6b2667177efa -->
<!-- translated-from: install/troubleshooting.md sha1:6d423596fc9eca05c89352c7cb809a5557a18761 -->
# Fehlersuche
Was `/healthz` bedeutet, wie du eine hängende von einer leeren Warteschlange unterscheidest, wo die
@@ -54,13 +54,17 @@ HTTP-Status.
## Die Warteschlange
Vier Transports auf dem Doctrine-Transport, jeder mit einem eigenen Consumer-Prozess im
Acht Transports auf dem Doctrine-Transport, abgearbeitet von sieben Prozessen im
Worker-Container:
| Warteschlange | Prozess | Was darauf liegt |
|---|---|---|
| `export` | `worker-export` | Alles, was plMail verlässt — Versand, Flag-Pushes, Gmail-Label-Änderungen, Mail aus dem Notifier. Die einzige Warteschlange, auf die jemand wartet |
| `ingest` | `worker-ingest` | Eingehende Mail, Gmail- und Graph-Nachrichten-Batches, Kalenderabgleiche, Terminerkennung |
| `ingest` | `worker-ingest` | Mail abholen und sonst nichts: Konto- und Postfachabgleiche, Gmail- und Graph-Nachrichten-Batches, Kalenderabgleiche |
| `enrich_live` | `worker-live` | Was auf Mail folgt, die gerade eben in einem Konto angekommen ist, dessen erster Import abgeschlossen ist: Einsortieren durch den Assistenten, Terminerkennung, Insight-Karten, Lesebestätigungen. Eigener Prozess, damit neue Mail nie hinter einem Import steht |
| `release` | `worker-release` | Beendet das Warten für Mail, die für den Assistenten zurückgehalten wird. Fast immer leer; eigener Prozess, damit nie etwas davor steht |
| `enrich` | `worker-enrich` | Dieselbe Arbeit für aktuelle Mail aus einem Import, dazu die Suchindizierung |
| `enrich_backlog` | `worker-enrich` | Alte Mail, die der Assistent nach einem Import einsortiert. Wird nur angefasst, wenn `enrich` leer ist |
| `maintenance` | `worker-maintenance` | Nachträgliche Verarbeitungen, Regelläufe über vorhandene Mail, "Jetzt ausführen"-Knöpfe im Administrationsbereich, Registrierung von Kalender-Push |
| `bulk` | `worker-bulk` | Mark-Read- und Archiv-Läufe über eine ganze Ansicht. Jemand schaut dabei auf eine Fortschrittsanzeige, deshalb reihen sie sich nicht hinter einer nachträglichen Verarbeitung ein |
| `async` | `worker-maintenance` | Stillgelegt. Wird geleert, damit vor der Aufteilung eingestellte Envelopes noch einen Konsumenten haben; hierher wird nichts mehr geroutet |
docs/de/internals/architecture.md
+5 -3
View File
@@ -1,4 +1,4 @@
<!-- translated-from: internals/architecture.md sha1:ad354438d897072ff5d4a852b98d222881661e2b -->
<!-- translated-from: internals/architecture.md sha1:6dbabd36bda20eb3426c8d4237e129b868d776f6 -->
# Architektur
Die Schichten, was wo liegt, und die Regeln, die das so halten. Diese Seite beschreibt die
@@ -177,8 +177,10 @@ Workers, damit ein verzögerter Retry nie auf einen Neustart warten muss.
Getrennte Transports allein genügen nicht. Ein Worker, der bereits in einem langen Handler
steckt, kann eine Sendung nicht aufnehmen, wie hoch sie auch priorisiert ist — also hat jeder
Transport seinen eigenen Prozess: `worker-export`, `worker-ingest`, `worker-maintenance` und
`worker-bulk`, die `app:work` nebeneinander im Worker-Container betreibt. Ein fünfter Transport, `async`, wird ohne Routing
Transport seinen eigenen Prozess: `worker-export`, `worker-ingest`, `worker-live`, `worker-release`, `worker-enrich`,
`worker-maintenance` und `worker-bulk`, die `app:work` nebeneinander im Worker-Container betreibt.
`worker-enrich` ist der eine, der zwei Transports abarbeitet, `enrich` und `enrich_backlog`, in
dieser Reihenfolge — siehe [Mail-Ingest](mail-ingest.md#was-auf-eine-nachricht-folgt-und-wo-es-läuft). Ein fünfter Transport, `async`, wird ohne Routing
weitergeführt, damit Envelopes, die vor der Aufteilung eingereiht wurden, noch irgendwo landen
können; der Maintenance-Worker leert ihn.
docs/de/internals/mail-ingest.md
+96 -2
View File
@@ -1,4 +1,4 @@
<!-- translated-from: internals/mail-ingest.md sha1:0c095f9f207803c75806b4b3b7d645ae471f24db -->
<!-- translated-from: internals/mail-ingest.md sha1:38b6beb3ba071c7ee0ef05200adc575b44fdf9c7 -->
# Mail-Ingest
Vom Anbieter in die Datenbank: wie aus Bytes eine Zeile `Message` wird, was mit ihr geschieht,
@@ -114,7 +114,10 @@ Stellen, die die Regeln der Nutzerin nicht erklären.
**Steps stoßen an, sie arbeiten nicht.** `afterCommit()` läuft auf dem Worker, der eine
IMAP-Verbindung oder ein Graph-Rate-Limit-Budget hält; ein Parse, ein HTTP-Aufruf oder eine
Bilddekodierung gehören in einen eigenen Handler.
Bilddekodierung gehören in einen eigenen Handler — und ebenso Arbeit, die bloß schnell ist; so
kamen die Insight- und Vorschlags-Steps dazu, inline zu laufen, und so hörten sie damit auf.
`HarvestContactsStep` ist die eine verbliebene Ausnahme: Er liest Zeilen, die schon im Speicher
liegen, und speist die Kategorisierung.
`App\Service\Mail\PostIngest\ExtractEventsStep` ist die Form zum Abschauen — er sammelt Ids ein
und schickt `ExtractEventsMessage` ab, sonst nichts.
@@ -128,6 +131,87 @@ einmal hindurchgegangen ist — ein zweiter Lauf würde also ein Create für ein
JMAP-Clients längst halten, und Regeln erneut über Mail laufen lassen, die die Nutzerin
inzwischen von Hand eingeordnet haben kann.
## Was auf eine Nachricht folgt, und wo es läuft
`ingest` ist zum Abholen von Mail da und für sonst nichts. Alles, was ein Step einreiht, geht
auf eine von drei anderen Warteschlangen — und welche, hängt von der Mail ab, nicht von der Art
der Arbeit:
| Warteschlange | Prozess | Was darauf liegt |
|---|---|---|
| `enrich_live` | `worker-live` | Folgearbeit für Mail, die gerade eben angekommen ist |
| `enrich` | `worker-enrich` | Folgearbeit für aktuelle Mail aus einem Import; Suchindizierung |
| `enrich_backlog` | `worker-enrich`, nach `enrich` | Alte Mail, die der Assistent nach dem Import einsortiert |
Früher war das eine Warteschlange. Ein erster Import legte pro Stapel einen Modellaufruf, eine
Terminerkennung und einen Bestätigungs-Scan auf `ingest`, neben die Abgleiche — und der eine
Worker, der Mail abholt, fragte stundenlang ein Modell nach Newslettern von 2019, während neue
Mail dahinter wartete.
**Aktuell.** `App\Service\Mail\PostIngest\RecentMailPolicy` entscheidet das, nach dem Datum der
Mail selbst — `receivedAt`, dann `sentAt` — gegen ein Fenster von 30 Tagen
(`ENRICH_RECENT_DAYS`). Jeder Step reiht nur für aktuelle Mail Arbeit ein. Ältere Mail bekommt,
was die Pipeline selbst tut — eine Zeile, eine Konversation, die Kategorie der Regelkaskade, die
Regeln der Nutzerin, ihre Kontakte — und keinen Job. Die Backfill-Aufgaben (`app:backfill
insights`, `event-extraction`, …) sind der Weg, Vergangenes zu lesen, wenn jemand es will.
**Live.** `App\Service\Mail\InitialImportState` sagt, ob ein Konto sein Postfach noch zum ersten
Mal hereinholt: Gmail über `needsBackfill()`, Graph darüber, ob es überhaupt einen Delta-Link
gibt, IMAP darüber, ob jeder zum Abgleich aktivierte Ordner ein `syncedAt` hat. Die Pipeline
fragt einmal pro Stapel und trägt die Antwort in `PostIngestResult::$live`; `EnrichmentRouter`
macht daraus einen `TransportNamesStamp`. Mail eines fertigen Kontos geht auf `enrich_live`,
dessen Worker nie Import-Arbeit vor sich hat. Im Zweifel heißt es „importiert noch" — ein
Gmail-Backfill gilt bis zu eine Stunde zu spät als abgeschlossen —, weil der umgekehrte Fehler
ein ganzes Postfach als live einreihen würde.
**Alte Mail und der Assistent.** `App\Service\Ai\ClassificationCatchUp`, alle Viertelstunde von
`app:ai:classify-backlog` gestartet, reiht die neueste Mail, nach der der Assistent noch nie
gefragt wurde, auf `enrich_backlog` ein — aber erst, wenn jedes Konto dieser Person fertig
importiert ist, und nur, solange die Warteschlange leer ist; das ersetzt einen eigenen Zustand.
`Message::$aiCategorisedAt` ist die Fortschrittsmarke.
**Ein Modell-Host.** Getrennte Worker enden trotzdem an einem Host, der eine Anfrage nach der
anderen beantwortet. `App\Service\Ai\LiveMailPriority` lässt die Massen-Klassifizierung vor jedem
Aufruf auf `enrich_live` schauen und zurücktreten, solange dort etwas fällig ist — Live-Mail
wartet so höchstens hinter einem laufenden Aufruf.
### Mail für den Assistenten zurückhalten
Für jemanden, der vom Assistenten sortieren lässt, ist die Kategorie der Regeln eine Vermutung,
die gleich ersetzt wird. Sofort angezeigt, erschien die Mail unter „Allgemein" und wanderte
Sekunden später zu „Werbung". Wo das Urteil die Antwort also noch ändern kann, wird die
Nachricht **zurückgehalten**: `Message::$categoryHeldAt` wird vor der Konversationsbildung
gesetzt, ihre neue Konversation bekommt keine Kategorie — was jede Tab-Abfrage und
`EmailFilterCompiler::threadCategory()` schon als „in keinem Tab" lesen —, und eine Antwort in
eine bestehende Konversation lässt diese, wo sie ist.
`App\Service\Mail\ClassificationHold` ist die Entscheidung, und sie ist eine Liste von Gründen,
nicht zurückzuhalten: Das Zurückhalten ist unter Administration → KI abgeschaltet; die Mail
gehört zu einem Import; sie ist alt oder die eigene gesendete Kopie des Kontos; oder
`MessageCategorizer::verdictCouldDecide()` sagt, dass die Antwort schon feststeht — wer nach
Regeln sortiert, ein Gmail-Label, eine bekannte Korrespondenzadresse.
Drei Dinge beenden das Zurückhalten, und `App\Service\Mail\HeldMailReleaser` ist das eine Ende,
das sie teilen:
1. `ClassifyMailHandler`, wenn das Modell antwortet.
2. `ReleaseHeldMailMessage`, im Moment des Zurückhaltens mit einer Verzögerung von
`AiSettings::$holdMaxSeconds` (Vorgabe 60, 2–300) abgeschickt. Meist kommt sie an und findet
nichts Zurückgehaltenes mehr. Sie hat eine eigene Warteschlange, `release`, und einen Worker,
der sonst nichts tut (`worker-release`): Der Live-Worker steckt womöglich gerade in genau dem
Aufruf, der zu lange dauert, und die Freigabe darf nicht auf ihn warten — und auch nicht auf
einen Versand.
3. `app:mail:release-held`, jede Minute, falls keiner der beiden Jobs lief.
Dass das Warten endet, beendet nicht die Frage. Eine Freigabe ohne Urteil ordnet die Mail unter
der Kategorie der Regeln ein, die beim Ingest genau dafür geschrieben wurde; der Modellaufruf
läuft weiter, und wenn seine Antwort eintrifft, verschiebt sie die Mail und sagt es den offenen
Seiten. Nach der Nachricht wird einmal gefragt.
`category_held_at` und `category_released_at` bleiben erhalten, und ihre Differenz ist die Zahl,
die Administration → KI neben dem Schalter zeigt — Median, 95. Perzentil und wie viele
Nachrichten angezeigt wurden, bevor das Modell geantwortet hatte.
## Konversationsbildung
`App\Service\Imap\MessageThreader::assignThread()` ist eine Kaskade aus drei Stufen, und jede
@@ -374,3 +458,13 @@ jemandes Seitenleiste stehen oder eben nicht.
Interface dokumentiert „anstoßen, nicht arbeiten", und die Pipeline kann es nicht erzwingen; der
Fehler ist keine Fehlermeldung, sondern ein Postfach, das langsamer wird, je mehr Features an
einen Haken gehängt werden, der auf dem verbindungshaltenden Worker läuft.
**Ein Step, der ohne `EnrichmentRouter` einreiht, stellt Live-Mail hinter einen Import.** Die
Routing-Tabelle schickt jede Post-Ingest-Nachricht auf `enrich`; nur der Stamp bringt die Mail
eines fertigen Kontos auf `enrich_live`. Abschicken mit `$this->router->stampsFor($result)`, und
nur für `RecentMailPolicy::recentIds()`.
**Eine zurückgehaltene Nachricht, die nichts freigibt, ist Mail, die niemand im Posteingang
sieht.** Ihre Konversation hat keine Kategorie und steht damit in keinem Tab. Deshalb wird die
zeitgesteuerte Freigabe in `ClassifyMailStep` vor der Feature-Prüfung abgeschickt, und deshalb
sichert sie ein Durchlauf auf einem anderen Worker ab.
docs/features/admin.md
+16
View File
@@ -430,6 +430,22 @@ Sorting is the one to be careful with: it runs unattended. It can only ever act
if a header, a Gmail label or the fact that you have written to somebody already decided the
category, the model is not consulted and its opinion is not used. Switching it off needs no cleanup.
Two things keep sorting from getting in the way of mail arriving:
- **Only recent mail is sorted as it comes in.** Mail from the last 30 days is asked about straight
away. When you add an account, the rest of its mailbox is sorted afterwards, in the background,
newest first, and only once the import has finished — so a first import is not hours of model
calls with new mail waiting behind them.
- **Wait for the assistant before showing new mail in a tab.** For people who let the assistant sort
their mail, new mail is kept out of the inbox tabs for the moment the assistant needs, so it shows
up once, in the right tab, instead of appearing in one and moving to another. **Wait at most** is
the ceiling, 60 seconds by default: past it, the mail is shown where the ordinary rules put it,
and moved if the assistant answers later. It is a ceiling, not the wait — a model that is already
loaded answers in a second or two.
The line underneath reports how long mail was actually held over the last 7 days, and how often
the wait ran out — if that is most of the time, your model host is too slow for the wait to be
worth it, and you can switch it off. Mail brought in by a first import is never held.
Writing help never replaces what you wrote. It appends, so the original is always still there.
Summaries are the most expensive of the four. They use the same writing model, but a whole
docs/install/docker.md
+4 -1
View File
@@ -133,7 +133,10 @@ crashed.
| `mercure` | The Mercure hub, 1.x | Live updates — the mail list refreshing by itself |
| `imap-supervisor` | `app:imap:supervise` | Spawns and watches one `app:imap:idle` process per IDLE-enabled mailbox, so standard IMAP mail arrives the moment it lands |
| `worker-export` | `messenger:consume export` | Anything leaving plMail, and the only queue somebody is watching. On its own process so a send is never behind a sync |
| `worker-ingest` | `messenger:consume ingest` | Mail arriving and the work that immediately follows it |
| `worker-ingest` | `messenger:consume ingest` | Fetching mail, and only that |
| `worker-live` | `messenger:consume enrich_live` | What follows mail that has just arrived — sorting, event extraction, insight cards, receipts. On its own process so new mail is never behind an import |
| `worker-release` | `messenger:consume release` | Shows mail that was held for the assistant once its wait is over. Idle nearly always, and on its own process so that nothing is ever in front of it |
| `worker-enrich` | `messenger:consume enrich enrich_backlog` | The same work for an import's recent mail, then — only when that queue is empty — old mail being sorted by the assistant |
| `worker-maintenance` | `messenger:consume maintenance async` | Backfills, rule runs over existing mail, admin sweeps. Also drains the retired `async` queue |
| `worker-bulk` | `messenger:consume bulk` | Whole-view actions: mark every unread read, archive the lot. On its own process because somebody is watching an indicator for it and it must not wait behind a backfill |
| `scheduler` | `messenger:consume scheduler_default` | Fires everything recurring. **Nothing schedules itself without this process** |
docs/install/troubleshooting.md
+6 -3
View File
@@ -49,13 +49,16 @@ answer 200. Watch the `checks` object, not just the HTTP status.
## The queue
Four transports on the Doctrine transport, each with its own consumer process in the worker
container:
Eight transports on the Doctrine transport, consumed by seven processes in the worker container:
| Queue | Process | What is on it |
|---|---|---|
| `export` | `worker-export` | Anything leaving plMail — sends, flag pushes, Gmail label changes, mail from the notifier. The only queue somebody is waiting on |
| `ingest` | `worker-ingest` | Mail arriving, Gmail and Graph message batches, calendar syncs, event extraction |
| `ingest` | `worker-ingest` | Fetching mail and nothing else: account and mailbox syncs, Gmail and Graph message batches, calendar syncs |
| `enrich_live` | `worker-live` | What follows mail that has just arrived on an account whose first import is over: sorting by the assistant, event extraction, insight cards, read receipts. Its own process, so new mail is never behind an import |
| `release` | `worker-release` | Ends the wait for mail held for the assistant. Nearly always empty; its own process so nothing is ever in front of it |
| `enrich` | `worker-enrich` | The same work for recent mail brought in by an import, and search indexing |
| `enrich_backlog` | `worker-enrich` | Old mail being sorted by the assistant after an import. Only touched when `enrich` is empty |
| `maintenance` | `worker-maintenance` | Backfills, rule runs over existing mail, admin "run now" buttons, calendar push registration |
| `bulk` | `worker-bulk` | Whole-view mark-read and archive runs. Somebody is watching an indicator for these, so they do not queue behind a backfill |
| `async` | `worker-maintenance` | Retired. Drained so envelopes queued before the split still have a consumer; nothing routes here any more |
docs/internals/architecture.md
+4 -2
View File
@@ -163,8 +163,10 @@ delayed retry is never left waiting on a restart.
Separate transports are not enough on their own. A worker already inside a long handler
cannot pick up a send however it is prioritised, so each transport has its own process:
`worker-export`, `worker-ingest`, `worker-maintenance` and `worker-bulk`, which `app:work` runs side by side
in the worker container. A
`worker-export`, `worker-ingest`, `worker-live`, `worker-release`, `worker-enrich`, `worker-maintenance` and
`worker-bulk`, which `app:work` runs side by side in the worker container. `worker-enrich` is the one
that takes two transports, `enrich` and `enrich_backlog`, in that order — see
[Mail ingest](mail-ingest.md#what-follows-a-message-and-where-it-runs). A
fifth transport, `async`, is kept routing nothing, so envelopes queued before the split still
have somewhere to land; the maintenance worker drains it.
docs/internals/mail-ingest.md
+86 -1
View File
@@ -101,7 +101,9 @@ do not explain.
**Steps dispatch, they do not work.** `afterCommit()` runs on the worker holding an IMAP
connection or a Graph rate-limit budget; a parse, an HTTP call or an image decode belongs in
its own handler. `App\Service\Mail\PostIngest\ExtractEventsStep` is the shape to copy — it
its own handler — and so does work that is merely quick, which is how the insight and proposal
steps came to run inline and stopped. `HarvestContactsStep` is the one exception left: it reads
rows already in memory and feeds categorisation. `App\Service\Mail\PostIngest\ExtractEventsStep` is the shape to copy — it
collects ids and dispatches `ExtractEventsMessage`, and nothing else.
**Steps cannot fail a sync.** `notifySteps()` catches and logs whatever a step throws and
@@ -112,6 +114,80 @@ Two branches deliberately do **not** come through the pipeline: IMAP's Gmailify
through it once, so running it again would re-record a create for an id JMAP clients already
hold, and re-run rules over mail the user may since have filed by hand.
## What follows a message, and where it runs
`ingest` is for fetching mail and nothing else. Everything a step queues goes to one of three
other queues, and which one depends on the mail rather than on the kind of work:
| Queue | Process | What is on it |
|---|---|---|
| `enrich_live` | `worker-live` | Follow-up work for mail that has just arrived |
| `enrich` | `worker-enrich` | Follow-up work for recent mail brought in by an import; search indexing |
| `enrich_backlog` | `worker-enrich`, after `enrich` | Old mail being sorted by the assistant, after the import |
It used to be one queue. A first import put a model call, an event extraction and a receipt scan
per batch on `ingest`, beside the syncs, and the one worker that fetches mail spent hours asking
a model about newsletters from 2019 while new mail waited behind them.
**Recent.** `App\Service\Mail\PostIngest\RecentMailPolicy` decides, by the mail's own date —
`receivedAt`, then `sentAt` — against a window of 30 days (`ENRICH_RECENT_DAYS`). Every step
queues work only for recent mail. Older mail gets what the pipeline itself does — a row, a
thread, the rule cascade's category, the user's rules, its contacts — and no job. The backfill
tasks (`app:backfill insights`, `event-extraction`, …) are how history is read when somebody
wants it.
**Live.** `App\Service\Mail\InitialImportState` says whether an account is still bringing in its
mailbox for the first time: Gmail by `needsBackfill()`, Graph by having a delta link at all,
IMAP by every sync-enabled folder having a `syncedAt`. The pipeline asks once per batch and
carries the answer on `PostIngestResult::$live`; `EnrichmentRouter` turns it into a
`TransportNamesStamp`. Mail of a finished account goes to `enrich_live`, whose worker never has
import work in front of it. It errs towards "still importing" — a Gmail backfill settles up to
an hour late — because the other mistake would queue a mailbox as live.
**Old mail and the assistant.** `App\Service\Ai\ClassificationCatchUp`, run by
`app:ai:classify-backlog` every quarter of an hour, queues the newest mail the assistant has
never been asked about onto `enrich_backlog` — but only once every account of that person has
finished importing, and only while the queue is empty, which is what stands in for state.
`Message::$aiCategorisedAt` is the progress marker.
**One model host.** Separate workers still end at one host that answers one request at a time.
`App\Service\Ai\LiveMailPriority` makes bulk classification look at `enrich_live` before each
call and stand aside while anything is due there, so live mail waits behind one in-flight call
at most.
### Holding mail for the assistant
For somebody who sorts by assistant, the rules' category is a guess about to be replaced. Shown
at once, the mail appeared under Primary and moved to Promotions seconds later. So where the
verdict could still change the answer, the message is **held**: `Message::$categoryHeldAt` is
stamped before threading, its new thread is given no category — which every tab query, and
`EmailFilterCompiler::threadCategory()`, already reads as "in no tab" — and a reply into an
existing thread leaves that thread where it is.
`App\Service\Mail\ClassificationHold` is the decision, and it is a list of reasons not to hold:
holding is switched off in Admin → AI; the mail is part of an import; it is old, or the
account's own sent copy; or `MessageCategorizer::verdictCouldDecide()` says the answer is
already final — a reader sorting by rules, a Gmail label, a known correspondent.
Three things end a hold, and `App\Service\Mail\HeldMailReleaser` is the one ending they share:
1. `ClassifyMailHandler`, when the model answers.
2. `ReleaseHeldMailMessage`, dispatched at the moment of the hold with a delay of
`AiSettings::$holdMaxSeconds` (60 by default, 2–300). It usually arrives to find nothing
held. It has a queue of its own, `release`, and a worker that does nothing else
(`worker-release`): the live worker may be inside the very call that is taking too long, and
the release must not wait for it — nor for a send.
3. `app:mail:release-held`, every minute, for when neither job ran.
The wait ending does not end the question. A release without a verdict files the mail under
the rules' category, which was written at ingest for exactly this; the model call carries on,
and when its answer arrives it moves the mail and tells open pages. The message is asked about
once.
`category_held_at` and `category_released_at` are kept, and their difference is the figure
Admin → AI shows beside the switch — median, 95th percentile, and how many messages were shown
before the model had answered.
## Threading
`App\Service\Imap\MessageThreader::assignThread()` is a three-step cascade, and each step
@@ -334,3 +410,12 @@ somebody's sidebar.
documents "dispatch, do not work" and the pipeline cannot enforce it; the failure is not an
error but a mailbox that gets slower as features are added to a hook that runs on the
connection-holding worker.
**A step that queues without `EnrichmentRouter` puts live mail behind an import.** The routing
table sends every post-ingest message to `enrich`; only the stamp moves a finished account's
mail to `enrich_live`. Dispatch with `$this->router->stampsFor($result)`, and only for
`RecentMailPolicy::recentIds()`.
**A held message nothing releases is mail nobody can see in their inbox.** Its thread has no
category and so is in no tab. That is why the timed release is dispatched before the feature
check in `ClassifyMailStep`, and why a sweep on a different worker backs it up.
migrations/Version20261007120000.php
+53
View File
@@ -0,0 +1,53 @@
<?php
declare(strict_types=1);
namespace DoctrineMigrations;
use Doctrine\DBAL\Schema\Schema;
use Doctrine\Migrations\AbstractMigration;
/**
* Holding arriving mail out of the inbox tabs until the assistant has sorted
* it: the two timestamps on the message, and the switch and the ceiling on the
* AI settings.
*
* See Message::$categoryHeldAt for what the pair means and AiSettings for why
* the switch defaults to on.
*
* NOTHING IS BACKFILLED, and nothing should be. Every existing message has
* both timestamps null, which reads as "was never held" — true of all of them.
*
* The index is partial for the reason idx_message_ai_categorised is: the two
* queries that read these columns — the minute sweep for mail held too long,
* and the delay figures in Admin → AI — both start from "was held", and on an
* installation that never sorts by assistant that is no row at all. An
* unconditional index would be one more entry per message, on every install,
* for a feature most of them have switched off.
*/
final class Version20261007120000 extends AbstractMigration
{
public function getDescription(): string
{
return 'Hold arriving mail for the assistant: message.category_held_at/released_at, ai_settings.hold_*';
}
public function up(Schema $schema): void
{
$this->addSql('ALTER TABLE message ADD category_held_at TIMESTAMP(0) WITHOUT TIME ZONE DEFAULT NULL');
$this->addSql('ALTER TABLE message ADD category_released_at TIMESTAMP(0) WITHOUT TIME ZONE DEFAULT NULL');
$this->addSql('CREATE INDEX idx_message_category_held ON message (category_held_at) WHERE category_held_at IS NOT NULL');
$this->addSql('ALTER TABLE ai_settings ADD hold_until_classified BOOLEAN DEFAULT TRUE NOT NULL');
$this->addSql('ALTER TABLE ai_settings ADD hold_max_seconds INT DEFAULT 60 NOT NULL');
}
public function down(Schema $schema): void
{
$this->addSql('DROP INDEX idx_message_category_held');
$this->addSql('ALTER TABLE message DROP category_held_at');
$this->addSql('ALTER TABLE message DROP category_released_at');
$this->addSql('ALTER TABLE ai_settings DROP hold_until_classified');
$this->addSql('ALTER TABLE ai_settings DROP hold_max_seconds');
}
}
src/Command/Ai/ClassifyBacklogCommand.php
+125
View File
@@ -0,0 +1,125 @@
<?php
declare(strict_types=1);
namespace App\Command\Ai;
use App\Entity\Ai\AiFeature;
use App\Repository\User\UserRepository;
use App\Service\Ai\AiAssistant;
use App\Service\Ai\ClassificationCatchUp;
use Symfony\Component\Console\Attribute\AsCommand;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Console\Style\SymfonyStyle;
/**
* The schedule's end of ClassificationCatchUp: every quarter of an hour, top
* the backlog queue up with the newest mail the assistant has not seen.
*
* The shape of app:ai:index-new-mail, which does the same for the search
* index, and deliberately so — a bounded sweep per mailbox, a keyset walk over
* users, success when the feature is off.
*
* FREQUENT AND SMALL rather than nightly and large. The sweep posts nothing
* while the queue still holds the last run's work, so the cadence is a ceiling
* and the model host sets the pace; a quarter of an hour only decides how long
* the queue may stand empty before it is noticed.
*/
#[AsCommand(
name: 'app:ai:classify-backlog',
description: 'Queue a bounded pass of old mail for the assistant to sort, once imports have finished',
)]
final class ClassifyBacklogCommand extends Command
{
/**
* Messages per mailbox per run.
*
* At a second or two a call this is a few minutes of a warm host — less
* than the gap to the next run, so the queue drains and the host idles in
* between rather than being held for the whole quarter hour.
*/
private const string DEFAULT_LIMIT = '200';
private const int USER_PAGE = 50;
public function __construct(
private readonly UserRepository $users,
private readonly AiAssistant $ai,
private readonly ClassificationCatchUp $catchUp,
) {
parent::__construct();
}
protected function configure(): void
{
$this
->addOption('limit', null, InputOption::VALUE_REQUIRED, 'Messages per mailbox', self::DEFAULT_LIMIT)
->addOption('email', null, InputOption::VALUE_REQUIRED, 'One mailbox instead of every mailbox');
}
protected function execute(InputInterface $input, OutputInterface $output): int
{
$io = new SymfonyStyle($input, $output);
// Success, not failure, for the reason app:ai:index-new-mail gives:
// this runs on every installation and almost none have the assistant
// sorting mail.
if (false === $this->ai->isEnabledFor(AiFeature::Categorise)) {
$io->note('Sorting by assistant is off — nothing to classify.');
return Command::SUCCESS;
}
if (false === $this->catchUp->mayRun()) {
$io->note('The last pass is still being worked through, or the assistant is in use — nothing queued.');
return Command::SUCCESS;
}
$limit = max(1, (int) $input->getOption('limit'));
$email = $input->getOption('email');
if (null !== $email) {
$user = $this->users->findOneBy(['email' => (string) $email]);
if (null === $user || null === $user->id) {
$io->error('No such user.');
return Command::FAILURE;
}
$queued = $this->catchUp->sweep($user, $limit);
$io->success(sprintf('Queued %d message(s) from %s.', $queued, (string) $user->email));
return Command::SUCCESS;
}
$lastId = 0;
$mailboxes = 0;
$queued = 0;
// Keyset by id; see IndexNewMailCommand.
while (true) {
$page = $this->users->findBatchAfterId($lastId, self::USER_PAGE);
if ([] === $page) {
break;
}
foreach ($page as $user) {
$lastId = (int) $user->id;
$queued += $this->catchUp->sweep($user, $limit);
$mailboxes++;
}
}
$io->success(sprintf('Queued %d message(s) for the assistant, across %d mailbox(es).', $queued, $mailboxes));
return Command::SUCCESS;
}
}
src/Command/Maintenance/ReleaseHeldMailCommand.php
+73
View File
@@ -0,0 +1,73 @@
<?php
declare(strict_types=1);
namespace App\Command\Maintenance;
use App\Repository\Ai\AiSettingsRepository;
use App\Repository\Mail\MessageRepository;
use App\Service\Mail\HeldMailReleaser;
use DateTimeImmutable;
use Symfony\Component\Console\Attribute\AsCommand;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Console\Style\SymfonyStyle;
/**
* Shows any mail that has been held for the assistant longer than it may be.
*
* THE LAST OF THREE WAYS OUT OF A HOLD, and the only one that needs no queue
* at all. The assistant's answer releases a message; failing that, a timed
* message queued when it was held does. Both are Messenger jobs. If the
* transport is backed up or a worker is down, neither may happen in time — and
* a held message is one nobody can see in their inbox.
*
* So this runs every minute on the scheduler, on the maintenance worker, and
* finds nothing on every run of every ordinary day. It is here for the day it
* finds something: mail can be seconds late, and with this it can be a minute
* late, but it cannot be lost from view.
*
* A GRACE ON TOP OF THE CEILING, so this does not race the timed release it is
* the backstop for. The two would both be correct — a release is idempotent —
* but each would announce to open pages, and there is no reason to.
*/
#[AsCommand(
name: 'app:mail:release-held',
description: 'Show mail that has waited for the assistant longer than the configured maximum.',
)]
final class ReleaseHeldMailCommand extends Command
{
/** Seconds past the ceiling before this steps in. */
private const int GRACE_SECONDS = 15;
public function __construct(
private readonly MessageRepository $messages,
private readonly AiSettingsRepository $settings,
private readonly HeldMailReleaser $releaser,
) {
parent::__construct();
}
protected function execute(InputInterface $input, OutputInterface $output): int
{
$io = new SymfonyStyle($input, $output);
$ceiling = $this->settings->currentOrDefault()->holdSeconds() + self::GRACE_SECONDS;
$overdue = $this->messages->findHeldBefore(new DateTimeImmutable(sprintf('-%d seconds', $ceiling)));
if ([] === $overdue) {
return Command::SUCCESS;
}
$released = $this->releaser->release($overdue);
// A warning, because it means the live worker did not do its job.
$io->warning(sprintf(
'Released %d message(s) that were left held. Check that worker-release is running.',
$released,
));
return Command::SUCCESS;
}
}
src/Controller/Admin/AiSettingsController.php
+24
View File
@@ -10,6 +10,7 @@ use App\Domain\Enum\Ai\MetricWindow;
use App\Domain\Enum\Ai\PromptSlot;
use App\Form\Admin\AiSettingsType;
use App\Repository\Ai\AiSettingsRepository;
use App\Repository\Mail\MessageRepository;
use App\Service\Ai\AiAssistant;
use App\Service\Ai\AiPerformancePanel;
use App\Service\Ai\EmbeddingBackfill;
@@ -56,6 +57,7 @@ final class AiSettingsController extends AbstractController
public function __construct(
private readonly AiSettingsRepository $settings,
private readonly MessageRepository $messages,
private readonly AiAssistant $assistant,
private readonly EntityManagerInterface $entityManager,
private readonly AiPerformancePanel $panel,
@@ -113,6 +115,7 @@ final class AiSettingsController extends AbstractController
'saved' => $saved,
'probe' => null,
'prompts' => $this->promptRows(),
'hold' => $this->holdDelay(),
]);
}
@@ -151,6 +154,7 @@ final class AiSettingsController extends AbstractController
$submitted['embeddingModel'] ?? null,
])),
'prompts' => $this->promptRows(),
'hold' => $this->holdDelay(),
]);
}
@@ -359,10 +363,30 @@ final class AiSettingsController extends AbstractController
'saved' => false,
'probe' => null,
'prompts' => $this->promptRows(),
'hold' => $this->holdDelay(),
'promptsSaved' => true,
]);
}
/**
* What holding arriving mail for the assistant has cost, over the last week.
*
* Rendered beside the switch that turns holding off, because that switch
* is a trade — mail never moves tabs, in exchange for appearing a little
* later — and "a little" is a property of this installation's model host
* that nobody can know in advance. See MessageRepository::holdDelayStats().
*
* A week and not the panel's selectable window: this is one sentence under
* a checkbox, and trickle mail on a small installation is a few dozen
* messages a day — a shorter window is mostly empty.
*
* @return array{held: int, timedOut: int, median: float|null, p95: float|null}
*/
private function holdDelay(): array
{
return $this->messages->holdDelayStats(MetricWindow::Week->since(new \DateTimeImmutable()));
}
/**
* The seven prompts as the page needs them: what is stored, and what we
* ship.
src/Domain/DTO/Mail/PostIngestResult.php
+11
View File
@@ -27,6 +27,17 @@ final readonly class PostIngestResult
public array $messages,
public array $accounts,
public array $threadIdsByAccount,
/**
* Whether this is mail trickling in rather than a mailbox being
* imported: every owning account has finished its first import.
*
* Decided once by the pipeline and carried here, because every step
* that queues work needs the same answer to choose a queue and each
* asking for itself would be a query per step per batch. Defaulted to
* false, which is the safe reading — bulk work is never put in front
* of live mail by an omission. See EnrichmentRouter.
*/
public bool $live = false,
) {
}
src/Entity/Ai/AiSettings.php
+52
View File
@@ -67,6 +67,11 @@ class AiSettings
*/
public const float DEFAULT_MIN_SIMILARITY = 0.42;
/** See $holdMaxSeconds. */
public const int DEFAULT_HOLD_MAX_SECONDS = 60;
public const int MIN_HOLD_SECONDS = 2;
public const int MAX_HOLD_SECONDS = 300;
use TimestampableTrait;
#[ORM\Id]
@@ -216,6 +221,40 @@ class AiSettings
public bool $categorisationEnabled = false;
/** Drafting help in the composer. */
/**
* Keep arriving mail out of the inbox tabs until the assistant has sorted
* it, for people who let the assistant sort.
*
* ON BY DEFAULT, because the thing it prevents — mail appearing in one tab
* and moving to another seconds later — is a defect rather than a taste,
* and it applies only where somebody chose the assistant as the sorter.
* A SWITCH ANYWAY, because the cure is a delay and how long the delay is
* depends on hardware this application has never seen: on a quick host it
* is a second or two and nobody would turn it off, on a slow one it may be
* much longer, up to $holdMaxSeconds for a message. The panel beside the switch
* shows the measured figure, which is what makes it a decision.
*/
#[ORM\Column(name: 'hold_until_classified', options: ['default' => true])]
public bool $holdUntilClassified = true;
/**
* The longest a message may wait, in seconds.
*
* This is a promise about ARRIVING MAIL, not a tuning knob for the model:
* past it the message is filed where the rules put it and shown, whatever
* the host is doing.
*
* A MINUTE BY DEFAULT. Ten seconds was the first answer and it was too
* tight: a model that has to be loaded first takes longer than that on its
* own, so the wait ran out on exactly the mail it was for, and the mail
* then moved tabs when the answer came — the jump holding exists to
* prevent. A minute covers a cold model, and it is a ceiling rather than
* the wait: a warm host answers in a second or two and the mail is shown
* then. What a message actually waited is in the panel beside the field.
*/
#[ORM\Column(name: 'hold_max_seconds', options: ['default' => self::DEFAULT_HOLD_MAX_SECONDS])]
public int $holdMaxSeconds = self::DEFAULT_HOLD_MAX_SECONDS;
#[ORM\Column(name: 'writing_help_enabled', options: ['default' => false])]
public bool $writingHelpEnabled = false;
@@ -269,6 +308,19 @@ class AiSettings
$this->prompts = new AiPrompts();
}
/**
* $holdMaxSeconds as something safe to wait for.
*
* Clamped on the way out rather than on the way in: the column is written
* by a form, and a zero that reached it must not mean "never hold" by
* accident (that is what the switch is for) nor a typo of 6000 mean mail
* sitting unseen for an hour and forty minutes.
*/
public function holdSeconds(): int
{
return max(self::MIN_HOLD_SECONDS, min(self::MAX_HOLD_SECONDS, $this->holdMaxSeconds));
}
/**
* Whether a model host is configured at all — as opposed to switched on.
*
src/Entity/Mail/Message.php
+49
View File
@@ -482,6 +482,44 @@ class Message extends MessageModel
#[ORM\Column(name: 'ai_categorised_at', nullable: true)]
public ?DateTimeImmutable $aiCategorisedAt = null;
/**
* When this message was kept out of the inbox tabs to wait for the model.
*
* THE PROBLEM IT ANSWERS: mail arrived, was filed by the rules, appeared
* under Primary, and a few seconds later moved to Promotions when the
* assistant answered. Somebody watching their inbox saw mail jump. Where
* the assistant is the one deciding, the rules' answer is a guess that is
* about to be replaced, and showing the guess is worse than showing
* nothing for the seconds the real answer takes.
*
* So the message is HELD: its thread is given no category, which is how
* every tab query already says "not in any tab", until the verdict is in
* or the wait runs out. See App\Service\Mail\ClassificationHold for when
* this is set and ClassifyMailHandler for what clears it.
*
* `category` beside it is still written, with the rules' answer. That is
* the fallback a release without a verdict files the mail under, and it is
* why a held message is never one the tabs lose: whatever happens to the
* model, the answer the mail would have had anyway is already on the row.
*
* KEPT AFTER THE RELEASE, together with the column below. The pair is the
* measurement — how long holding actually delays mail on this installation
* — and Admin → AI shows it beside the switch that turns holding off, so
* the choice is made against a number rather than a feeling.
*/
#[ORM\Column(name: 'category_held_at', nullable: true)]
public ?DateTimeImmutable $categoryHeldAt = null;
/**
* When the hold ended, by a verdict or by running out of time.
*
* Which of the two is not stored and does not need to be: a release with
* no `aiCategorisedAt` beside it, or with a later one, is a message shown
* before the model had answered.
*/
#[ORM\Column(name: 'category_released_at', nullable: true)]
public ?DateTimeImmutable $categoryReleasedAt = null;
/**
* @var Collection<int, Label>
*/
@@ -619,4 +657,15 @@ class Message extends MessageModel
return $this;
}
/**
* Held and not yet released — see $categoryHeldAt.
*
* The one reading of the pair, so that the threader, the pipeline and the
* handler cannot each grow their own idea of what "held" means.
*/
public function isCategoryHeld(): bool
{
return null !== $this->categoryHeldAt && null === $this->categoryReleasedAt;
}
}
src/Form/Admin/AiSettingsType.php
+21
View File
@@ -10,6 +10,7 @@ use App\Entity\Ai\AiSettings;
use App\Form\PasswordManagerIgnore;
use Symfony\Component\Form\AbstractType;
use Symfony\Component\Form\Extension\Core\Type\CheckboxType;
use Symfony\Component\Form\Extension\Core\Type\IntegerType;
use Symfony\Component\Form\Extension\Core\Type\NumberType;
use Symfony\Component\Form\Extension\Core\Type\PasswordType;
use Symfony\Component\Form\Extension\Core\Type\TextareaType;
@@ -168,6 +169,26 @@ final class AiSettingsType extends AbstractType
'help' => 'admin.ai.field.categorisation_help',
'required' => false,
])
// The two that qualify the switch above rather than being features
// of their own: whether arriving mail waits for the assistant
// before it is shown in a tab, and for how long at most. See
// AiSettings::$holdUntilClassified.
->add('holdUntilClassified', CheckboxType::class, [
'label' => 'admin.ai.field.hold',
'help' => 'admin.ai.field.hold_help',
'required' => false,
])
->add('holdMaxSeconds', IntegerType::class, [
'label' => 'admin.ai.field.hold_max_seconds',
'help' => 'admin.ai.field.hold_max_seconds_help',
'constraints' => [
new Range(min: AiSettings::MIN_HOLD_SECONDS, max: AiSettings::MAX_HOLD_SECONDS),
],
'attr' => [
'min' => AiSettings::MIN_HOLD_SECONDS,
'max' => AiSettings::MAX_HOLD_SECONDS,
],
])
->add('writingHelpEnabled', CheckboxType::class, [
'label' => 'admin.ai.field.writing_help',
'help' => 'admin.ai.field.writing_help_help',
src/Infrastructure/Messaging/Handler/ClassifyMailHandler.php
+53 -1
View File
@@ -14,6 +14,8 @@ use App\Repository\Mail\MessageThreadRepository;
use App\Service\Mail\MessageCategorizer;
use App\Service\Ai\AiAssistant;
use App\Service\Ai\AiPermissions;
use App\Service\Ai\LiveMailPriority;
use App\Service\Mail\HeldMailReleaser;
use App\Service\Ai\PromptLibrary;
use DateTimeImmutable;
use Doctrine\ORM\EntityManagerInterface;
@@ -73,6 +75,8 @@ final readonly class ClassifyMailHandler
private AiAssistant $ai,
private AiPermissions $permissions,
private PromptLibrary $prompts,
private LiveMailPriority $priority,
private HeldMailReleaser $releaser,
private EntityManagerInterface $entityManager,
private LoggerInterface $logger,
) {
@@ -98,12 +102,42 @@ final readonly class ClassifyMailHandler
continue;
}
$held = null !== $mail->categoryHeldAt;
if (false === $held) {
// Bulk work, by elimination: an import, the catch-up after
// one, or somebody's "ask again" button. None of it may be in
// front of mail that has just arrived. See LiveMailPriority.
$this->priority->standAside();
}
// NO SHORTER CLOCK FOR HELD MAIL, though there was one. The call
// used to be cut off when the hold ran out, which threw away an
// answer that was seconds from arriving and left the message to be
// asked about a second time, later. The hold and the question are
// separate things: the timed release shows the mail when the wait
// is over — on a worker of its own, precisely so it can happen while
// this call is still running — and this call simply finishes, and
// moves the mail if its answer differs.
$verdict = $this->ask($mail);
if (true === $held) {
// Minutes may have passed inside ask(), and the timed release
// may have run in them. Re-read, or the stale copy held here
// would still say "held" and the release below would stamp a
// second, later release time over the real one.
$this->entityManager->refresh($mail);
}
// Stamped whether or not the answer was usable — see the docblock.
$mail->aiCategorisedAt = new DateTimeImmutable();
$mail->aiCategory = $verdict;
$touched[] = $mail;
// A VERDICT THAT COMES AFTER THE WAIT RAN OUT STILL MOVES THE MAIL.
// The reader chose the assistant as the sorter, and a late answer
// is still the answer they asked for. Holding makes the jump rare;
// it does not make the assistant's answer optional.
$touched[] = $mail;
}
if ([] === $touched) {
@@ -168,7 +202,25 @@ final readonly class ClassifyMailHandler
// raw SQL reading the very columns written above.
$this->entityManager->flush();
$late = array_values(array_filter(
$mail,
static fn (Message $row): bool => null !== $row->categoryReleasedAt,
));
// Whatever in this batch was being held is released here, with the
// category written above: the verdict's where there was one, the
// rules' where the model had nothing to say in time. It resolves those
// threads itself and tells open pages — which is the moment held mail
// first appears in a tab. A no-op for a batch that held nothing, and
// for mail the timed release has already shown.
$this->releaser->release($mail);
$this->threads->recomputeCategoriesForThreads(array_keys($threadIds));
// And the ones the timed release got to first. They were shown under
// the rules' answer and have just been moved, which an open inbox has
// no other way to hear about.
$this->releaser->announce($late);
}
/**
src/Infrastructure/Messaging/Handler/ExtractInsightsHandler.php
+83
View File
@@ -0,0 +1,83 @@
<?php
declare(strict_types=1);
namespace App\Infrastructure\Messaging\Handler;
use App\Entity\User\User;
use App\Infrastructure\Messaging\Message\ExtractInsightsMessage;
use App\Repository\Mail\MessageRepository;
use App\Service\Insight\InsightHarvester;
use App\Service\Insight\InsightNotifier;
use Doctrine\ORM\EntityManagerInterface;
use Symfony\Component\Messenger\Attribute\AsMessageHandler;
/**
* Offers a batch of messages to the insight extractors.
*
* The loop ExtractInsightsStep used to run inside the sync, moved here whole.
* The harvester already shields the batch per extractor and per message, so
* what is left is the flush — once, after the batch, and only when something
* was actually written.
*
* The Mercure announcement belongs here for the same reason the flush does:
* this is the one place that knows whether the batch wrote anything, and for
* whom. A publish anywhere else would either fire on a batch that changed
* nothing (waking every open tab to re-request a fragment that comes back
* identical) or fire once per insight, which for a catch-up of thirty parcels
* is thirty identical updates for one strip.
*/
#[AsMessageHandler]
final readonly class ExtractInsightsHandler
{
public function __construct(
private MessageRepository $messages,
private InsightHarvester $harvester,
private InsightNotifier $notifier,
private EntityManagerInterface $em,
) {
}
public function __invoke(ExtractInsightsMessage $message): void
{
if ([] === $message->messageIds) {
return;
}
$written = 0;
/** @var array<int, User> $touched users whose strip has something new, keyed by id */
$touched = [];
// findByIds() simply omits a message deleted between the dispatch and
// the run, which is normal: the batch is queued while the mailbox
// keeps moving.
foreach ($this->messages->findByIds($message->messageIds) as $mail) {
$count = $this->harvester->harvest($mail);
if (0 === $count) {
continue;
}
$written += $count;
// Keyed by id, so a batch is one update per user however many of
// their messages carried facts.
$user = $mail->account->usr;
if (true === $user instanceof User) {
$touched[(int) $user->id] = $user;
}
}
if (0 === $written) {
return;
}
$this->em->flush();
foreach ($touched as $user) {
$this->notifier->publishInsightsChanged($user);
}
}
}
src/Infrastructure/Messaging/Handler/ProposeEventsHandler.php
+67
View File
@@ -0,0 +1,67 @@
<?php
declare(strict_types=1);
namespace App\Infrastructure\Messaging\Handler;
use App\Infrastructure\Messaging\Message\ProposeEventsMessage;
use App\Repository\Mail\MessageRepository;
use App\Service\Calendar\Proposal\EventProposer;
use Doctrine\ORM\EntityManagerInterface;
use Psr\Log\LoggerInterface;
use Symfony\Component\Messenger\Attribute\AsMessageHandler;
/**
* Looks for a date in prose in each message of a batch.
*
* The loop ProposeEventsStep used to run inside the sync, moved here whole.
* It still performs no I/O beyond the rows it loads — the body is on the row,
* the gate is a regex, and most messages are refused by the category check
* before even that.
*
* Flushes its own writes: a proposal that is only persisted is a proposal
* nobody ever sees.
*/
#[AsMessageHandler]
final readonly class ProposeEventsHandler
{
public function __construct(
private MessageRepository $messages,
private EventProposer $proposer,
private EntityManagerInterface $em,
private LoggerInterface $logger,
) {
}
public function __invoke(ProposeEventsMessage $message): void
{
if ([] === $message->messageIds) {
return;
}
$proposed = 0;
foreach ($this->messages->findByIds($message->messageIds) as $mail) {
// Per message, because one message the parser chokes on must not
// cost the batch its other proposals — and the job is retried as a
// whole, for a message that will not parse the second time either.
try {
if (null !== $this->proposer->propose($mail)) {
$proposed++;
}
} catch (\Throwable $e) {
$this->logger->error('EventProposal: proposing failed', [
'messageId' => $mail->id,
'error' => $e->getMessage(),
'exception' => $e,
]);
}
}
if (0 === $proposed) {
return;
}
$this->em->flush();
}
}
src/Infrastructure/Messaging/Handler/ReleaseHeldMailHandler.php
+55
View File
@@ -0,0 +1,55 @@
<?php
declare(strict_types=1);
namespace App\Infrastructure\Messaging\Handler;
use App\Infrastructure\Messaging\Message\ReleaseHeldMailMessage;
use App\Repository\Mail\MessageRepository;
use App\Service\Mail\HeldMailReleaser;
use Psr\Log\LoggerInterface;
use Symfony\Component\Messenger\Attribute\AsMessageHandler;
/**
* The end of the wait: shows held mail where the rules put it.
*
* Delivered after AiSettings::$holdMaxSeconds, and in the ordinary case finds
* nothing to do — the assistant answered and ClassifyMailHandler released the
* message long before. What it does find is the case the delay was for: no
* answer yet. The question is still open and its answer will move the mail;
* this only stops making the reader wait for it. See ReleaseHeldMailMessage.
*
* The category is not touched. The rules' answer was written when the message
* arrived and is still on the row, which is the whole reason it was written;
* all that is missing is letting the thread adopt it.
*/
#[AsMessageHandler]
final readonly class ReleaseHeldMailHandler
{
public function __construct(
private MessageRepository $messages,
private HeldMailReleaser $releaser,
private LoggerInterface $logger,
) {
}
public function __invoke(ReleaseHeldMailMessage $message): void
{
if ([] === $message->messageIds) {
return;
}
$released = $this->releaser->release($this->messages->findByIds($message->messageIds));
if (0 === $released) {
return;
}
// Worth a line: each of these is a message the assistant did not
// answer for in time, and a steady stream of them is a host too slow
// for the wait it has been given.
$this->logger->info('Held mail shown before the assistant answered', [
'messages' => $released,
]);
}
}
src/Infrastructure/Messaging/Message/ExtractInsightsMessage.php
+23
View File
@@ -0,0 +1,23 @@
<?php
declare(strict_types=1);
namespace App\Infrastructure\Messaging\Message;
/**
* Read a batch of freshly ingested messages for the facts the radar shows —
* parcels, flights, tickets, code review.
*
* Ids and a batch, the shape of every other post-ingest job. This one used to
* be no job at all: ExtractInsightsStep ran the extractors inside the sync. See
* that class for why it stopped.
*/
final readonly class ExtractInsightsMessage
{
/**
* @param list<int> $messageIds
*/
public function __construct(
public array $messageIds,
) {}
}
src/Infrastructure/Messaging/Message/ProposeEventsMessage.php
+21
View File
@@ -0,0 +1,21 @@
<?php
declare(strict_types=1);
namespace App\Infrastructure\Messaging\Message;
/**
* Look for a date in prose in a batch of freshly ingested messages.
*
* Ids and a batch, the shape of every other post-ingest job. See
* ProposeEventsStep for why this is a job now and was not before.
*/
final readonly class ProposeEventsMessage
{
/**
* @param list<int> $messageIds
*/
public function __construct(
public array $messageIds,
) {}
}
src/Infrastructure/Messaging/Message/ReleaseHeldMailMessage.php
+39
View File
@@ -0,0 +1,39 @@
<?php
declare(strict_types=1);
namespace App\Infrastructure\Messaging\Message;
/**
* Stop waiting for the assistant and show this mail where the rules put it.
*
* Dispatched WITH A DELAY at the moment a message is held, for exactly as long
* as the hold is allowed to last — see Message::$categoryHeldAt and
* AiSettings::$holdMaxSeconds. It is the promise that holding is bounded: the
* classification job normally releases the message long before this is
* delivered, and then this finds nothing held and does nothing.
*
* A message of its own rather than a timeout inside the classification job,
* because the job is the thing that might be slow or might not happen. A model
* that is still loading, a host that is off, a job sitting out a retry delay —
* in every one of those the mail still has to appear, and this is what makes
* it.
*
* IT DOES NOT STOP THE QUESTION. The classification job carries on, and when
* the answer arrives it moves the mail if it differs from the rules'. Ending
* the wait and giving up on the answer are different decisions, and only the
* first is this message's.
*
* On a queue of its own, `release`, with a worker that does nothing else; see
* messenger.yaml for why it must not share one with the call it is the limit
* on, nor with anything else.
*/
final readonly class ReleaseHeldMailMessage
{
/**
* @param list<int> $messageIds
*/
public function __construct(
public array $messageIds,
) {}
}
src/Infrastructure/Scheduler/MaintenanceSchedule.php
+17
View File
@@ -58,6 +58,14 @@ final class MaintenanceSchedule implements ScheduleProviderInterface
// when nothing is due — which is almost always.
RecurringMessage::cron('* * * * *', new RunCommandMessage('app:mail:wake-snoozed')),
// The backstop for mail held out of the tabs for the assistant.
// Two things release it sooner and both live on the live
// worker; this is what releases it if that worker is not
// there. Every minute for the same reason as the line above:
// one indexed query that finds nothing, almost always, and the
// one time it finds something a minute is already too long.
RecurringMessage::cron('* * * * *', new RunCommandMessage('app:mail:release-held')),
// Polling is the mechanism for CalDAV and the backstop for the
// other two. Two of the three providers DO offer push and this
// comment used to say none did: Google Calendar has watch
@@ -150,6 +158,15 @@ final class MaintenanceSchedule implements ScheduleProviderInterface
// one only dispatches.
RecurringMessage::cron('20 3 * * *', new RunCommandMessage('app:ai:index-new-mail')),
// Old mail, asked about by the assistant after an import rather
// than during it — see ClassificationCatchUp. Often and small,
// unlike the line above: the command posts nothing while its
// queue still holds the last pass, so this is how soon an empty
// queue is noticed and not how hard the host is pushed. Offset
// from the quarter-hour mail sync so the two do not start
// together.
RecurringMessage::cron('5-59/15 * * * *', new RunCommandMessage('app:ai:classify-backlog')),
// Counts and durations of model calls. Its own line rather than
// a fourth window on app:monitoring:prune above, because it is
// the retention of a FEATURE rather than of the deployment: the
src/Repository/Mail/MailboxRepository.php
+25
View File
@@ -136,4 +136,29 @@ class MailboxRepository extends ServiceEntityRepository
['backdated' => $backdated->format('Y-m-d H:i:s'), 'id' => $mailboxId],
);
}
/**
* How many of an account's folders have never finished a sync.
*
* What "the first import is still running" means for IMAP, which has no
* account-level cursor to ask: every folder that is synced at all gets a
* syncedAt at the end of MessageSyncer::syncMailbox(), so a folder without
* one is a folder still being read for the first time — or one discovered
* a moment ago, which is the same thing as far as its mail is concerned.
*
* A count rather than a boolean so the caller can tell "none outstanding"
* from "no folders at all"; see InitialImportState.
*/
public function countAwaitingFirstSync(Account $account): int
{
return (int) $this->createQueryBuilder('mailbox')
->select('COUNT(mailbox.id)')
->andWhere('mailbox.account = :account')
->andWhere('mailbox.isSyncEnabled = :isSyncEnabled')
->andWhere('mailbox.syncedAt IS NULL')
->setParameter('account', $account)
->setParameter('isSyncEnabled', true)
->getQuery()
->getSingleScalarResult();
}
}
src/Repository/Mail/MessageRepository.php
+107
View File
@@ -2389,4 +2389,111 @@ class MessageRepository extends ServiceEntityRepository
return $bodies;
}
/**
* The newest mail of one person's that the assistant has never been asked
* about — what ClassificationCatchUp works through after an import.
*
* NEWEST FIRST, the opposite of a backfill's walk and for the reason
* EmbeddingCatchUp gives: this is a bounded handful per run, and the mail
* worth spending it on is the mail somebody is most likely to scroll to.
* A mailbox is classified backwards from today, so stopping at any point
* leaves the useful end done.
*
* `ai_categorised_at IS NULL` is "never asked", not "no answer" — the
* stamp is written even when the model said nothing usable — so a message
* the model cannot answer is offered once and not on every run for ever.
*
* @return list<int>
*/
public function unclassifiedIdsForUser(int $userId, int $limit): array
{
/** @var list<int|string> $ids */
$ids = $this->getEntityManager()->getConnection()->fetchFirstColumn(
<<<'SQL'
SELECT m.id
FROM message m
JOIN account a ON a.id = m.account_id
WHERE a.usr_id = :userId
AND a.is_active = true
AND m.ai_categorised_at IS NULL
ORDER BY m.received_at DESC NULLS LAST, m.id DESC
LIMIT :limit
SQL,
['userId' => $userId, 'limit' => $limit],
['userId' => ParameterType::INTEGER, 'limit' => ParameterType::INTEGER],
);
return array_map(intval(...), $ids);
}
/**
* Mail still held for the assistant that was held before $before.
*
* For the sweep that is the last line of the promise that holding is
* bounded — see ReleaseHeldMailCommand. Limited, because the caller
* hydrates what this returns and the case where there is a great deal of
* it is the case where something is already wrong.
*
* @return list<Message>
*/
public function findHeldBefore(DateTimeImmutable $before, int $limit = 200): array
{
/** @var list<Message> */
return $this->createQueryBuilder('m')
->where('m.categoryHeldAt IS NOT NULL')
->andWhere('m.categoryReleasedAt IS NULL')
->andWhere('m.categoryHeldAt < :before')
->setParameter('before', $before)
->orderBy('m.categoryHeldAt', 'ASC')
->setMaxResults($limit)
->getQuery()
->getResult();
}
/**
* How long holding mail for the assistant has actually delayed it.
*
* The number Admin → AI puts beside the switch, so that "is the wait worth
* it" is answered by this installation's own host rather than by a guess.
*
* `timedOut` is a message shown before the model had answered — no
* ai_categorised_at at all, or one later than the release. It went to the
* rules' tab first and may have moved when the answer came. A high share
* of those means the wait is being paid and the jump is happening anyway.
*
* percentile_cont is Postgres-only, which this application already is.
* Both come back null over an empty window, and are passed on as null —
* a median of zero would read as "instant" where the truth is "no data".
*
* @return array{held: int, timedOut: int, median: float|null, p95: float|null}
*/
public function holdDelayStats(DateTimeImmutable $since): array
{
/** @var array{held: int|string, timed_out: int|string, median: float|string|null, p95: float|string|null}|false $row */
$row = $this->getEntityManager()->getConnection()->fetchAssociative(
<<<'SQL'
SELECT COUNT(*) AS held,
COUNT(*) FILTER (WHERE m.ai_categorised_at IS NULL OR m.ai_categorised_at > m.category_released_at) AS timed_out,
percentile_cont(0.5) WITHIN GROUP (ORDER BY EXTRACT(EPOCH FROM (m.category_released_at - m.category_held_at))) AS median,
percentile_cont(0.95) WITHIN GROUP (ORDER BY EXTRACT(EPOCH FROM (m.category_released_at - m.category_held_at))) AS p95
FROM message m
WHERE m.category_held_at >= :since
AND m.category_released_at IS NOT NULL
SQL,
['since' => $since],
['since' => Types::DATETIME_IMMUTABLE],
);
if (false === $row) {
return ['held' => 0, 'timedOut' => 0, 'median' => null, 'p95' => null];
}
return [
'held' => (int) $row['held'],
'timedOut' => (int) $row['timed_out'],
'median' => null === $row['median'] ? null : (float) $row['median'],
'p95' => null === $row['p95'] ? null : (float) $row['p95'],
];
}
}
src/Repository/Mail/MessageThreadRepository.php
+5
View File
@@ -2426,6 +2426,10 @@ class MessageThreadRepository extends ServiceEntityRepository
SELECT DISTINCT ON (m.thread_id) m.thread_id, m.category
FROM message m
WHERE m.category IS NOT NULL
-- A message held for the assistant has no say yet: its
-- category is the rules' guess, and adopting it is the
-- flicker holding exists to prevent. Message::$categoryHeldAt.
AND (m.category_held_at IS NULL OR m.category_released_at IS NOT NULL)
AND m.thread_id IN (:threadIds)
ORDER BY m.thread_id, m.received_at DESC NULLS LAST
) sub
@@ -2446,6 +2450,7 @@ class MessageThreadRepository extends ServiceEntityRepository
SELECT DISTINCT ON (m.thread_id) m.thread_id, m.category
FROM message m
WHERE m.category IS NOT NULL
AND (m.category_held_at IS NULL OR m.category_released_at IS NOT NULL)
ORDER BY m.thread_id, m.received_at DESC NULLS LAST
) sub
WHERE t.id = sub.thread_id
src/Repository/Monitoring/MessengerQueueRepository.php
+49
View File
@@ -98,6 +98,55 @@ final readonly class MessengerQueueRepository
);
}
/**
* Whether one queue has work that is due or in hand right now.
*
* What bulk classification asks about the live queue before each model
* call, so that mail which has just arrived is never behind more than the
* one call already in flight — see LiveMailPriority.
*
* TWO THINGS ARE DELIBERATELY NOT COUNTED. A row whose available_at is
* still in the future is a delayed message — the live queue always holds
* the timed release of whatever is currently held — and counting it would
* make "busy" permanent. And a row delivered more than $freshSeconds ago is
* not work in hand but a worker that died holding it; the transport will
* redeliver it in its own time, and bulk work must not stand aside for an
* hour waiting for that.
*
* The transport writes its timestamps in UTC, so "now" is passed in as UTC
* rather than left to the database's session time zone.
*/
public function countActiveOn(string $queue, int $freshSeconds = 120): int
{
$now = new \DateTimeImmutable('now', new \DateTimeZone('UTC'));
return (int) $this->connection->fetchOne(
'SELECT COUNT(*) FROM messenger_messages
WHERE queue_name = :queue
AND ((delivered_at IS NULL AND available_at <= :now)
OR (delivered_at IS NOT NULL AND delivered_at >= :fresh))',
[
'queue' => $queue,
'now' => $now->format('Y-m-d H:i:s'),
'fresh' => $now->modify(sprintf('-%d seconds', $freshSeconds))->format('Y-m-d H:i:s'),
],
);
}
/**
* How many messages are waiting on one queue, delayed ones included.
*
* What a catch-up asks before topping a queue up: anything still there,
* due or not, is work it already posted.
*/
public function countWaitingOn(string $queue): int
{
return (int) $this->connection->fetchOne(
'SELECT COUNT(*) FROM messenger_messages WHERE queue_name = :queue',
['queue' => $queue],
);
}
/**
* Sync work still queued for one user's accounts — what the topbar's sync
* button polls so it can stop spinning.
src/Service/Ai/ClassificationCatchUp.php
+146
View File
@@ -0,0 +1,146 @@
<?php
declare(strict_types=1);
namespace App\Service\Ai;
use App\Entity\Ai\AiFeature;
use App\Entity\User\User;
use App\Infrastructure\Messaging\Message\ClassifyMailMessage;
use App\Repository\Mail\AccountRepository;
use App\Repository\Mail\MessageRepository;
use App\Repository\Monitoring\MessengerQueueRepository;
use App\Service\Mail\InitialImportState;
use App\Service\Mail\PostIngest\EnrichmentRouter;
use Psr\Log\LoggerInterface;
use Symfony\Component\Messenger\MessageBusInterface;
use Throwable;
/**
* Asks the assistant about old mail, a bounded handful at a time, once an
* import is over.
*
* WHY OLD MAIL IS NOT ASKED ABOUT AS IT IS IMPORTED
* ─────────────────────────────────────────────────
* It used to be: every message of a fifty-thousand-message mailbox was queued
* for a model call the moment it was ingested, oldest first, on the queue that
* fetches mail. Hours of somebody's GPU spent on newsletters from 2019 before
* last week's mail had been looked at, with new mail waiting behind all of it.
*
* So the import classifies only what is recent (RecentMailPolicy), and this
* does the rest afterwards: newest first, so the mail somebody is most likely
* to scroll to is done soonest and stopping anywhere leaves the useful end
* finished; in the background, on the queue behind every other; and not at all
* until the import has finished, because a mailbox still arriving is a moving
* target and its own recent mail has first call on the model.
*
* WHAT THIS IS, AND IS MODELLED ON: EmbeddingCatchUp::sweep(). A budget per
* call, no state of its own, nothing to resume — the message's own
* `aiCategorisedAt` stamp is the progress marker, written by the handler
* whether or not the model said anything usable.
*
* WHAT STANDS IN FOR STATE is the queue itself. A run posts nothing while its
* queue still holds what the last run posted, which is all that keeps two
* sweeps from queueing the same ids twice — and also makes the pace
* self-adjusting: a slow host is simply topped up less often.
*
* IT DISPATCHES, IT DOES NOT ASK. ClassifyMailHandler already checks the
* feature and the owner again, already skips what is stamped, already re-files
* what it touches and already stands aside for live mail. A second path that
* called the model itself would re-implement all four.
*/
final readonly class ClassificationCatchUp
{
public function __construct(
private MessageRepository $messages,
private AccountRepository $accounts,
private AiPermissions $permissions,
private InitialImportState $importState,
private InteractiveAiActivity $interactive,
private BackfillPolicy $policy,
private MessengerQueueRepository $queues,
private MessageBusInterface $bus,
private LoggerInterface $logger,
) {
}
/**
* Whether a sweep may post anything at all right now.
*
* Asked once per run by the command rather than once per mailbox: neither
* answer depends on whose mail it is.
*/
public function mayRun(): bool
{
// Somebody is using the composer or a summary. Those share the chat
// model with classification and are what a person is actually waiting
// on; the backlog is nobody's foreground. Same signal, same cooldown,
// as the embedding backfill.
if (true === $this->interactive->shouldYield($this->policy->cooldownSeconds)) {
return false;
}
try {
return 0 === $this->queues->countWaitingOn(EnrichmentRouter::BACKLOG);
} catch (Throwable $exception) {
// A transport whose table this cannot read — anything but the
// doctrine one. Without the depth there is nothing to stop a run
// re-posting what the last one posted, so it does not run; the
// feature degrades to "old mail keeps the rules' answer", which is
// what it had before this class existed.
$this->logger->warning('ClassificationCatchUp: could not read the backlog queue, not sweeping', [
'error' => $exception->getMessage(),
'exception' => $exception,
]);
return false;
}
}
/**
* Queue the newest unasked mail of one person's.
*
* @return int how many messages were queued
*/
public function sweep(User $user, int $limit): int
{
$userId = (int) $user->id;
if ($userId <= 0 || $limit <= 0) {
return 0;
}
if (false === $this->permissions->allows($user, AiFeature::Categorise)) {
return 0;
}
// ALL of this person's accounts, not each on its own. Their mail is
// one inbox to them and one queue to the model, and starting on the
// finished account's history while another is still importing would
// put old mail in front of that import's recent mail.
$accounts = $this->accounts->findBy(['usr' => $user, 'isActive' => true]);
if (false === $this->importState->allComplete($accounts)) {
return 0;
}
$ids = $this->messages->unclassifiedIdsForUser($userId, $limit);
if ([] === $ids) {
return 0;
}
// CHUNKED BY BackfillPolicy::$batchSize, the tuned answer to "how long
// may one job hold the host before something else gets a turn".
foreach (array_chunk($ids, $this->policy->batchSize) as $chunk) {
$this->bus->dispatch(new ClassifyMailMessage(array_values($chunk)), EnrichmentRouter::backlog());
}
$this->logger->info('ClassificationCatchUp: queued old mail for the assistant', [
'userId' => $userId,
'messages' => count($ids),
]);
return count($ids);
}
}
src/Service/Ai/LiveMailPriority.php
+83
View File
@@ -0,0 +1,83 @@
<?php
declare(strict_types=1);
namespace App\Service\Ai;
use App\Repository\Monitoring\MessengerQueueRepository;
use App\Service\Mail\PostIngest\EnrichmentRouter;
use Psr\Log\LoggerInterface;
use Throwable;
/**
* Lets mail that has just arrived go ahead of bulk classification at the model
* host.
*
* WHY A QUEUE OF ITS OWN WAS NOT ENOUGH
* ─────────────────────────────────────
* Live mail has its own transport and its own worker, so nothing bulk is ever
* in front of it IN THE QUEUE. But the two workers end at the same model host,
* and that host answers one request at a time: a worker that is free to pick a
* live message up at once still waits for whatever the bulk worker has in
* flight, and then for the next one, because the bulk worker's following
* request is already on its way.
*
* So bulk classification asks before each call whether the live queue has
* anything due, and stands aside while it does. The cost to a held message is
* then at most the one call that was already in flight when it arrived — which
* is the difference between a hold measured in seconds and one that runs out.
*
* BOUNDED, AND FAILS OPEN. The wait has a ceiling so a live queue that never
* drains — a dead worker — delays an import rather than stopping it. And a
* queue that cannot be read answers "go ahead": this is a courtesy between
* two workers, and it must never be the reason no mail is classified at all.
* That includes every transport that is not the doctrine one, where the table
* this reads does not exist.
*/
final readonly class LiveMailPriority
{
/** The longest one bulk call is put off, in seconds. */
private const int MAX_WAIT_SECONDS = 20;
/** How often to look again. Short: the thing waited for takes a second or two. */
private const int POLL_MICROSECONDS = 250_000;
public function __construct(
private MessengerQueueRepository $queues,
private LoggerInterface $logger,
) {
}
/**
* Returns once the live queue is quiet, or once it has been waited on for
* long enough.
*/
public function standAside(): void
{
$waited = 0;
while ($waited < self::MAX_WAIT_SECONDS * 1_000_000) {
if (false === $this->liveMailWaiting()) {
return;
}
usleep(self::POLL_MICROSECONDS);
$waited += self::POLL_MICROSECONDS;
}
}
private function liveMailWaiting(): bool
{
try {
return $this->queues->countActiveOn(EnrichmentRouter::LIVE) > 0;
} catch (Throwable $exception) {
$this->logger->debug('LiveMailPriority: could not read the live queue, not waiting', [
'error' => $exception->getMessage(),
'exception' => $exception,
]);
return false;
}
}
}
src/Service/Backup/ConfigBackupDatabase.php
+7
View File
@@ -257,6 +257,8 @@ final readonly class ConfigBackupDatabase
'embeddingDimensions' => $settings->embeddingDimensions,
'searchEnabled' => $settings->searchEnabled,
'categorisationEnabled' => $settings->categorisationEnabled,
'holdUntilClassified' => $settings->holdUntilClassified,
'holdMaxSeconds' => $settings->holdMaxSeconds,
'writingHelpEnabled' => $settings->writingHelpEnabled,
'summaryEnabled' => $settings->summaryEnabled,
'prompts' => $prompts,
@@ -309,6 +311,11 @@ final readonly class ConfigBackupDatabase
$settings->embeddingDimensions = is_int($values['embeddingDimensions'] ?? null) ? $values['embeddingDimensions'] : null;
$settings->searchEnabled = true === ($values['searchEnabled'] ?? false);
$settings->categorisationEnabled = true === ($values['categorisationEnabled'] ?? false);
// Absent from every backup taken before these existed, and absent has
// to mean the default rather than "off" — a restore must not be the
// thing that quietly switches a behaviour off.
$settings->holdUntilClassified = false !== ($values['holdUntilClassified'] ?? true);
$settings->holdMaxSeconds = is_int($values['holdMaxSeconds'] ?? null) ? $values['holdMaxSeconds'] : AiSettings::DEFAULT_HOLD_MAX_SECONDS;
$settings->writingHelpEnabled = true === ($values['writingHelpEnabled'] ?? false);
$settings->summaryEnabled = true === ($values['summaryEnabled'] ?? false);
src/Service/Calendar/Proposal/ProposeEventsStep.php
+22 -43
View File
@@ -6,8 +6,10 @@ namespace App\Service\Calendar\Proposal;
use App\Domain\DTO\Mail\PostIngestResult;
use App\Domain\Interface\PostIngestStepInterface;
use Doctrine\ORM\EntityManagerInterface;
use Psr\Log\LoggerInterface;
use App\Infrastructure\Messaging\Message\ProposeEventsMessage;
use App\Service\Mail\PostIngest\EnrichmentRouter;
use App\Service\Mail\PostIngest\RecentMailPolicy;
use Symfony\Component\Messenger\MessageBusInterface;
/**
* Looks for a date in prose in each freshly ingested message.
@@ -19,60 +21,37 @@ use Psr\Log\LoggerInterface;
* is assigned in that loop, and refusing bulk mail is this feature's first and
* most important rule.
*
* It does its work inline instead of dispatching a job, which is the one place
* this bends PostIngestStepInterface's stated contract, so here is the case for
* it and the line it must not cross. That rule is about cost: a step runs on a
* worker holding an IMAP connection or a Graph rate-limit budget, and a fetch,
* an image decode or a raw-MIME parse there delays every message behind it.
* This path performs no I/O at all. The body is already hydrated on the row,
* the gate is a single regex over it, and the overwhelming majority of messages
* are refused by the category check before even that — the ones that get as far
* as being parsed are a few per cent, and the parse is regular expressions over
* a few kilobytes. Queuing that would cost a Messenger round trip per batch to
* move microseconds off the ingest worker.
* IT USED TO DO ITS WORK INLINE, and argued for it at length here: the path
* performs no I/O, the gate is a single regex, and queuing it would cost a
* Messenger round trip to move microseconds off the ingest worker. All true of
* one batch. What it left out is the mailbox import, where it is fifty messages
* a batch for tens of thousands of messages on the one worker that fetches
* mail — and the rule in PostIngestStepInterface has no exception for work
* that is merely quick, because what it protects is that fetching mail is the
* only thing the fetching worker does. So this dispatches like every other
* step, and ProposeEventsHandler holds the loop.
*
* The line: a detector that calls anything — a model, a service, a network —
* must NOT run here. When one is added, it goes behind a job and this step
* dispatches for the messages the deterministic detector could not read.
* ProposalDetectorInterface is where that boundary will be drawn.
*
* Flushes its own writes. The pipeline's flushes are done by the time steps
* run, and a proposal that is only persisted is a proposal nobody ever sees.
* RECENT MAIL ONLY, which costs nothing: EventProposer already refuses a date
* in the past, so an old message could only ever have been parsed in order to
* be discarded. See RecentMailPolicy.
*/
final readonly class ProposeEventsStep implements PostIngestStepInterface
{
public function __construct(
private EventProposer $proposer,
private EntityManagerInterface $em,
private LoggerInterface $logger,
private MessageBusInterface $bus,
private RecentMailPolicy $recent,
private EnrichmentRouter $router,
) {
}
public function afterCommit(PostIngestResult $result): void
{
$proposed = 0;
$ids = $this->recent->recentIds($result);
foreach ($result->messages as $message) {
// Per message, because one message the parser chokes on must not
// cost the batch its other proposals. The pipeline's own guard is
// one level coarser than that: it catches for the whole step.
try {
if (null !== $this->proposer->propose($message)) {
$proposed++;
}
} catch (\Throwable $e) {
$this->logger->error('EventProposal: proposing failed', [
'messageId' => $message->id,
'error' => $e->getMessage(),
'exception' => $e,
]);
}
}
if (0 === $proposed) {
if ([] === $ids) {
return;
}
$this->em->flush();
$this->bus->dispatch(new ProposeEventsMessage($ids), $this->router->stampsFor($result));
}
}
src/Service/Imap/MessageThreader.php
+16 -1
View File
@@ -282,7 +282,14 @@ final class MessageThreader
// Seed from the message that opens the thread; attachMessageToThread()
// takes over from here. Primary only when the message is uncategorised,
// which is the case for locally-composed drafts.
$thread->category = $message->category ?? MessageCategory::Primary;
//
// NO CATEGORY AT ALL for a message being held for the assistant. Null
// is how a thread says it is in no tab — every tab query filters on an
// exact category — and that is the whole of what holding means; see
// Message::$categoryHeldAt. ClassifyMailHandler fills it in on release.
$thread->category = true === $message->isCategoryHeld()
? null
: $message->category ?? MessageCategory::Primary;
$thread->attachmentCount = 0;
// The new-mail marker starts null — nobody has been shown this row yet
@@ -458,6 +465,14 @@ final class MessageThreader
return;
}
// A held message has an answer that is about to be replaced, and
// adopting it would move the conversation twice — which is the thing
// holding exists to stop. The thread stays where it is until the
// release re-resolves it.
if (true === $message->isCategoryHeld()) {
return;
}
$category = $message->category;
if (null !== $category) {
src/Service/Insight/ExtractInsightsStep.php
+25 -57
View File
@@ -6,79 +6,47 @@ namespace App\Service\Insight;
use App\Domain\DTO\Mail\PostIngestResult;
use App\Domain\Interface\PostIngestStepInterface;
use App\Entity\User\User;
use Doctrine\ORM\EntityManagerInterface;
use App\Infrastructure\Messaging\Message\ExtractInsightsMessage;
use App\Service\Mail\PostIngest\EnrichmentRouter;
use App\Service\Mail\PostIngest\RecentMailPolicy;
use Symfony\Component\Messenger\MessageBusInterface;
/**
* The insight pipeline's foothold in the sync: every freshly ingested message
* is offered to the extractors, exactly the way ProposeEventsStep offers them
* to the proposal detectors.
* The insight pipeline's foothold in the sync: every recent, freshly ingested
* message is queued for the extractors.
*
* The harvester already shields the batch per extractor and per message, so
* the only job left here is the flush — once, after the batch, and only when
* something was actually written. The pipeline's own catch shields the sync
* from this step entirely; see PostIngestStepInterface.
* IT USED TO RUN THEM HERE, inside the sync, and the case for that was that
* the extractors read only what is already on the row. True, and beside the
* point on the day a mailbox is imported: fifty messages a batch, every batch,
* each offered to every extractor on the one worker that fetches mail. The
* rule in PostIngestStepInterface — dispatch, do not work — has no exception
* for work that is merely quick, because what it protects is that fetching
* mail is the only thing the fetching worker does.
*
* The Mercure announcement belongs here for the same reason the flush does:
* this is the one place that knows whether the batch wrote anything, and for
* whom. The harvester returns a count per message but has no notion of a
* batch, and the step above has no notion of what was written — so a publish
* anywhere else would either fire on a sync that changed nothing (waking every
* open tab to re-request a fragment that comes back identical) or fire once
* per insight, which for a nightly catch-up of thirty parcels is thirty
* identical updates for one strip.
* RECENT MAIL ONLY. A parcel delivered in 2021 is not something to put on
* somebody's radar, and that is what offering an imported mailbox to the
* extractors produced. `app:backfill insights` reads history when somebody
* wants it. See RecentMailPolicy.
*
* The work, the flush and the Mercure announcement are ExtractInsightsHandler's.
*/
final readonly class ExtractInsightsStep implements PostIngestStepInterface
{
public function __construct(
private InsightHarvester $harvester,
private EntityManagerInterface $em,
private InsightNotifier $notifier,
private MessageBusInterface $bus,
private RecentMailPolicy $recent,
private EnrichmentRouter $router,
) {
}
public function afterCommit(PostIngestResult $result): void
{
$written = 0;
$ids = $this->recent->recentIds($result);
/** @var array<int, User> $touched users whose strip has something new, keyed by id */
$touched = [];
foreach ($result->messages as $message) {
$count = $this->harvester->harvest($message);
if (0 === $count) {
continue;
}
$written += $count;
// Keyed by id, so a batch is one update per user however many of
// their messages carried facts. A batch is usually one account and
// therefore one user, but Gmail's handler ingests per account
// inside a loop and nothing here should depend on that staying
// true.
$user = $message->account->usr;
if ($user instanceof User) {
$touched[$user->id] = $user;
}
}
if (0 === $written) {
if ([] === $ids) {
return;
}
$this->em->flush();
// After the flush, never before: the update tells tabs to come and
// read, and a tab that reads before the rows are committed sees the
// strip it had. The publish itself is not worth failing the sync over,
// but it does not need its own catch — PostIngestStepInterface's
// contract is that the pipeline swallows whatever this step throws,
// and the rows are already durable by the time it can throw.
foreach ($touched as $user) {
$this->notifier->publishInsightsChanged($user);
}
$this->bus->dispatch(new ExtractInsightsMessage($ids), $this->router->stampsFor($result));
}
}
src/Service/Mail/ClassificationHold.php
+114
View File
@@ -0,0 +1,114 @@
<?php
declare(strict_types=1);
namespace App\Service\Mail;
use App\Entity\Ai\AiFeature;
use App\Entity\Embeddable\CategorySorting;
use App\Entity\Mail\Account;
use App\Entity\Mail\Message;
use App\Entity\User\User;
use App\Repository\Ai\AiSettingsRepository;
use App\Service\Ai\AiPermissions;
use App\Service\Mail\PostIngest\RecentMailPolicy;
/**
* Decides whether an arriving message waits for the assistant before it is
* shown in a tab.
*
* What holding IS lives on Message::$categoryHeldAt, and what ends it lives in
* ClassifyMailHandler. This is only the decision, and it is its own class
* because the decision is a list of reasons NOT to hold — every one of them a
* case where waiting would cost seconds and change nothing.
*
* NARROW ON PURPOSE. A held message is mail somebody cannot see yet, which is
* the most expensive thing a mail client can do to its user, so it is done
* only where all of these are true:
*
* · the installation has holding switched on, and sorting by assistant at all;
* · this person lets the assistant sort their mail;
* · the mail is trickling in rather than being imported — the caller's
* $live, from InitialImportState. An import that held its mail would put a
* mailbox behind a model;
* · the mail is recent, and not this account's own outgoing copy;
* · and the verdict could actually move it. See
* MessageCategorizer::verdictCouldDecide() — a reader sorting by rules, a
* Gmail label and a known correspondent are all answers that are already
* final.
*
* NOT CHECKED: WHETHER THE MAIL IS IN THE INBOX. The user's own rules run
* after threading and may archive it, and the thread's category has to be
* decided before threading. A held message that a rule then archives is held
* for nothing — it was never going to be in a tab — and that costs one
* prioritised model call, which it would have had anyway.
*/
final readonly class ClassificationHold
{
public function __construct(
private AiSettingsRepository $settings,
private AiPermissions $permissions,
private MessageCategorizer $categorizer,
private RecentMailPolicy $recent,
) {
}
/**
* The half of the decision that is the same for every message in a batch.
*
* Split off so the pipeline asks it once: it is a settings read, and the
* other half is called per message.
*/
public function isOnFor(?User $user): bool
{
if (null === $user) {
return false;
}
if (false === $this->settings->currentOrDefault()->holdUntilClassified) {
return false;
}
return $this->permissions->allows($user, AiFeature::Categorise);
}
/**
* @param array<string,true> $correspondentEmails
*/
public function shouldHold(
Message $message,
Account $account,
array $correspondentEmails,
?CategorySorting $sorting,
): bool {
// Asked about already — a row re-ingested after a move, say. The answer
// is on it and there is nothing to wait for.
if (null !== $message->aiCategorisedAt) {
return false;
}
if (false === $this->recent->isRecent($message)) {
return false;
}
if (true === $this->isOwnMail($message, $account)) {
return false;
}
return $this->categorizer->verdictCouldDecide($message, $correspondentEmails, $sorting);
}
/**
* The Sent copy of something this account wrote.
*
* It is ingested like any other message and is never in an inbox tab, so
* holding it would be a model call jumping the queue to sort mail nobody
* is going to look for there.
*/
private function isOwnMail(Message $message, Account $account): bool
{
$from = mb_strtolower(trim((string) $message->fromAddress));
return '' !== $from && $from === mb_strtolower(trim((string) $account->email));
}
}
src/Service/Mail/HeldMailReleaser.php
+142
View File
@@ -0,0 +1,142 @@
<?php
declare(strict_types=1);
namespace App\Service\Mail;
use App\Entity\Mail\Account;
use App\Entity\Mail\Message;
use App\Repository\Mail\MessageThreadRepository;
use DateTimeImmutable;
use Doctrine\ORM\EntityManagerInterface;
/**
* Ends a hold: puts mail that was waiting for the assistant into its tab and
* tells everybody looking.
*
* THREE CALLERS, ONE ENDING. ClassifyMailHandler releases a message when the
* model has answered. ReleaseHeldMailHandler releases it when the wait ran out
* first — the model is then still being asked, and its answer moves the mail
* when it comes. The minute sweep releases whatever both of them missed. They differ
* only in what category is on the message by the time they get here — the
* verdict's, or the rules' — and everything after that is the same and is
* this class, so that a release cannot be done three slightly different ways.
*
* WHAT A RELEASE IS, in order, and the order is load-bearing:
*
* 1. Stamp categoryReleasedAt, and flush. The thread resolution below is raw
* SQL that excludes held messages, so it has to be able to see that this
* one no longer is.
* 2. Re-resolve the threads' categories — the step that was skipped when the
* message arrived. This is the moment the conversation appears in a tab.
* 3. Record the change for JMAP clients, and flush that.
* 4. Tell open pages to look again. After the flush, never before: a page
* that re-reads before the rows are committed draws the list it had.
*
* IDEMPOTENT, because the callers race by design: the timed release is queued
* the moment a message is held and usually arrives to find it already
* released. A message that is not held is skipped, and a call that releases
* nothing announces nothing.
*/
final readonly class HeldMailReleaser
{
public function __construct(
private MessageThreadRepository $threads,
private MailChangeRecorder $changes,
private SyncNotifier $notifier,
private EntityManagerInterface $em,
) {
}
/**
* @param iterable<Message> $messages
*
* @return int how many were actually released
*/
public function release(iterable $messages, ?DateTimeImmutable $now = null): int
{
$now ??= new DateTimeImmutable();
$released = [];
foreach ($messages as $message) {
if (false === $message->isCategoryHeld()) {
continue;
}
$message->categoryReleasedAt = $now;
$released[] = $message;
}
if ([] === $released) {
return 0;
}
$this->em->flush();
$threadIds = [];
foreach ($released as $message) {
$threadId = $message->thread?->id;
if (null !== $threadId) {
$threadIds[(int) $threadId] = true;
}
}
$this->threads->recomputeCategoriesForThreads(array_keys($threadIds));
$this->announce($released);
return count($released);
}
/**
* Steps 3 and 4 on their own: record the change and tell open pages.
*
* Public for the one caller that moves held mail WITHOUT releasing it —
* ClassifyMailHandler, when the assistant's answer arrives after the timed
* release has already shown the message. The thread has just changed tab
* under somebody's eyes, and nothing else would say so.
*
* The caller has already flushed and resolved the threads.
*
* @param iterable<Message> $messages
*/
public function announce(iterable $messages): void
{
/** @var array<int, list<int>> $threadIdsByAccount */
$threadIdsByAccount = [];
/** @var array<int, Account> $accounts */
$accounts = [];
foreach ($messages as $message) {
$account = $message->account;
$accountId = (int) $account->id;
$accounts[$accountId] = $account;
$this->changes->emailChanged($accountId, (string) $message->id, created: false, thread: null);
$threadId = $message->thread?->id;
if (null !== $threadId) {
$threadIdsByAccount[$accountId][] = (int) $threadId;
}
}
if ([] === $accounts) {
return;
}
foreach ($threadIdsByAccount as $accountId => $ids) {
$this->changes->threadsTouched($accountId, array_values(array_unique($ids)));
}
$this->em->flush();
foreach ($accounts as $account) {
$this->notifier->publishAccountSynced($account);
}
}
}
src/Service/Mail/InitialImportState.php
+89
View File
@@ -0,0 +1,89 @@
<?php
declare(strict_types=1);
namespace App\Service\Mail;
use App\Entity\Mail\Account;
use App\Repository\Mail\MailboxRepository;
/**
* Whether an account is still bringing in its mailbox for the first time.
*
* WHY ANYTHING NEEDS TO KNOW
* ──────────────────────────
* An import and an ordinary day put the same messages through the same
* pipeline, and until now nothing downstream could tell them apart. They want
* opposite things. Mail trickling in is mail somebody may be watching arrive:
* it should be sorted and scraped within seconds, and it is worth holding back
* for the moment that takes. A mailbox being imported is tens of thousands of
* messages nobody is watching individually: it should never be in front of the
* first kind, and holding it back would hide a mailbox behind a model.
*
* Two things read this — EnrichmentRouter, to choose a queue, and
* ClassificationHold, to decide whether to hold — and one more waits on it:
* ClassificationCatchUp will not start on old mail until the import is over.
*
* ONE QUESTION, THREE ANSWERS, because the three providers keep their progress
* in three places and none of them was written to be asked this:
*
* Gmail has real backfill state. needsBackfill() is false only once a
* listing has found nothing left to fetch — see GmailApiSyncer.
* Graph keeps a delta link per folder, written once that folder has been
* enumerated. None at all means no folder has been.
* IMAP has neither, but every folder gets a syncedAt when its first pass
* ends, so a sync-enabled folder without one is still importing.
*
* IT ERRS TOWARDS "STILL IMPORTING", AND THAT IS THE SAFE SIDE. Gmail settles
* its backfill up to an hour after the last message lands, and for that hour
* new mail is treated as import mail: sorted promptly on the shared queue but
* not held, so it may move tabs once. The opposite mistake — calling an import
* finished early — would hold a mailbox's worth of mail and queue it as live.
*
* A folder discovered later makes an IMAP account "importing" again until that
* folder has been read, which is correct rather than a quirk: what is about to
* arrive from it is a folder's history, not new mail.
*/
final readonly class InitialImportState
{
public function __construct(private MailboxRepository $mailboxes)
{
}
public function isComplete(Account $account): bool
{
if (true === $account->isGmail()) {
return false === $account->needsBackfill();
}
if (true === $account->isMicrosoft()) {
return [] !== $account->graphDeltaLinks;
}
// Read from the database rather than off $account->mailboxes: the
// callers are long-running workers whose entity manager is cleared
// between messages, and a collection hydrated before a clear answers
// for folders as they were then.
return 0 === $this->mailboxes->countAwaitingFirstSync($account);
}
/**
* @param iterable<Account> $accounts
*/
public function allComplete(iterable $accounts): bool
{
$any = false;
foreach ($accounts as $account) {
$any = true;
if (false === $this->isComplete($account)) {
return false;
}
}
// No accounts is not "finished" — it is nothing to say anything about,
// and the callers treat true as licence to do more.
return $any;
}
}
src/Service/Mail/MessageCategorizer.php
+28
View File
@@ -70,6 +70,34 @@ final class MessageCategorizer
)['category'];
}
/**
* Whether a verdict that has not arrived yet could still change where this
* message is filed.
*
* The question ClassificationHold asks before keeping mail out of the tabs:
* there is no point holding a message whose answer is already final. Three
* things make it final, and they are read off the cascade below rather
* than restated here — the reader sorts by rules, so the verdict is never
* consulted; the provider's own label decided; or the sender is somebody
* the reader writes to. Everywhere else, an assistant-first reader's mail
* goes wherever the model says.
*
* Asked with $ignoreAi so the answer describes the cascade and not a
* verdict that happens to be on the row already.
*
* @param array<string,true> $correspondentEmails
*/
public function verdictCouldDecide(Message $message, array $correspondentEmails, ?CategorySorting $sorting): bool
{
if (null === $sorting || false === $sorting->assistantFirst()) {
return false;
}
$reason = $this->explain($message, $correspondentEmails, $sorting->overrideProvider, true)['reason'];
return 'gmail' !== $reason && 'correspondent' !== $reason;
}
/**
* The same decision, with the step that made it and the thing it matched
* on.
src/Service/Mail/PostIngest/ClassifyMailStep.php
+65 -17
View File
@@ -8,8 +8,12 @@ use App\Domain\DTO\Mail\PostIngestResult;
use App\Domain\Interface\PostIngestStepInterface;
use App\Entity\Ai\AiFeature;
use App\Infrastructure\Messaging\Message\ClassifyMailMessage;
use App\Infrastructure\Messaging\Message\ReleaseHeldMailMessage;
use App\Repository\Ai\AiSettingsRepository;
use App\Service\Ai\AiAssistant;
use App\Service\Ai\BackfillPolicy;
use Symfony\Component\Messenger\MessageBusInterface;
use Symfony\Component\Messenger\Stamp\DelayStamp;
/**
* Queues a second opinion for a freshly ingested batch — when anybody asked
@@ -19,6 +23,25 @@ use Symfony\Component\Messenger\MessageBusInterface;
* holding an IMAP connection or a Graph rate-limit budget, and a language model
* on another machine is the last thing that belongs inside a sync.
*
* THREE KINDS OF MAIL LEAVE HERE DIFFERENTLY
* ──────────────────────────────────────────
* HELD mail — kept out of the inbox tabs until the assistant answers, see
* Message::$categoryHeldAt — goes out one message to an envelope on the
* live queue, with a delayed release queued elsewhere beside it. One each, because somebody
* is waiting on every one of them separately and a batch would make the
* first wait for the last.
*
* RECENT mail goes out in small chunks, on the live queue if it is
* trickling in and the shared one if it is part of an import. Small because
* a chunk is how long the worker is deaf to anything else: this used to
* post the whole batch as one envelope, fifty or a hundred model calls, and
* that envelope was minutes long.
*
* OLD mail is not queued at all. It keeps the rules' answer, which is the
* one it would have been shown under anyway, and is asked about later, in
* the background, once the import is over. See RecentMailPolicy and
* App\Service\Ai\ClassificationCatchUp.
*
* THE GUARD IS HERE AND NOT ONLY IN THE HANDLER
* ─────────────────────────────────────────────
* Checking twice looks redundant and is not. Almost every installation has this
@@ -34,38 +57,63 @@ use Symfony\Component\Messenger\MessageBusInterface;
final readonly class ClassifyMailStep implements PostIngestStepInterface
{
public function __construct(
private MessageBusInterface $bus,
private AiAssistant $ai,
private MessageBusInterface $bus,
private AiAssistant $ai,
private AiSettingsRepository $settings,
private RecentMailPolicy $recent,
private BackfillPolicy $policy,
private EnrichmentRouter $router,
) {
}
public function afterCommit(PostIngestResult $result): void
{
if (false === $this->ai->isEnabledFor(AiFeature::Categorise)) {
return;
}
$ids = [];
$held = [];
$recent = [];
foreach ($result->messages as $message) {
$id = $message->id;
if (null !== $id) {
$ids[] = (int) $id;
if (null === $id) {
continue;
}
if (true === $message->isCategoryHeld()) {
$held[] = (int) $id;
continue;
}
if (true === $this->recent->isRecent($message)) {
$recent[] = (int) $id;
}
}
// WHICH of these are worth asking about is the handler's decision, not
// this one. The test is "did the deterministic cascade fall through to
// its default", and answering it properly needs the correspondent set —
// which is a query, and a query does not belong in a step that runs
// inside a sync. Dispatching the batch and narrowing on the other side
// costs one job for a batch that turns out to need nothing.
// BEFORE THE FEATURE CHECK, and unconditionally. A held message is
// invisible in the tabs until something releases it, so the release is
// queued whatever else is true — including the feature having been
// switched off in the moment between the pipeline holding the message
// and this line. Holding mail and then declining to queue its way out
// is the one outcome here that loses mail from view.
if ([] !== $held) {
$this->bus->dispatch(new ReleaseHeldMailMessage($held), [
new DelayStamp($this->settings->currentOrDefault()->holdSeconds() * 1000),
]);
}
if ([] === $ids) {
if (false === $this->ai->isEnabledFor(AiFeature::Categorise)) {
return;
}
$this->bus->dispatch(new ClassifyMailMessage($ids));
foreach ($held as $id) {
$this->bus->dispatch(new ClassifyMailMessage([$id]), EnrichmentRouter::live());
}
// WHICH of these are worth asking about is the handler's decision, not
// this one — it needs the owner's own preference, which is a query, and
// a query does not belong in a step that runs inside a sync.
foreach (array_chunk($recent, $this->policy->batchSize) as $chunk) {
$this->bus->dispatch(new ClassifyMailMessage(array_values($chunk)), $this->router->stampsFor($result));
}
}
}
src/Service/Mail/PostIngest/EnrichmentRouter.php
+58
View File
@@ -0,0 +1,58 @@
<?php
declare(strict_types=1);
namespace App\Service\Mail\PostIngest;
use App\Domain\DTO\Mail\PostIngestResult;
use Symfony\Component\Messenger\Stamp\StampInterface;
use Symfony\Component\Messenger\Stamp\TransportNamesStamp;
/**
* Which queue a batch's follow-up work goes on.
*
* The routing table in messenger.yaml sends every post-ingest message to
* `enrich`, and that is right for a batch that is part of an import. It is
* wrong for mail that has just arrived on an account whose import is over —
* that mail may have somebody waiting on it, and `enrich` may be an hour deep
* in another account's first sync.
*
* The message CLASS cannot tell the two apart; the same ClassifyMailMessage is
* either, depending on the mail it carries. So the choice is made here, where
* the batch is known, and travels as a stamp.
*
* One class rather than a line in each step, because there are five steps and
* the day a sixth is written it should not be possible to forget.
*/
final readonly class EnrichmentRouter
{
/** Mail that has just arrived; a worker of its own. */
public const string LIVE = 'enrich_live';
/** Old mail being classified after an import; behind everything else. */
public const string BACKLOG = 'enrich_backlog';
/**
* The stamps to dispatch a batch's follow-up work with.
*
* Empty for an import, which leaves the routing table to decide.
*
* @return list<StampInterface>
*/
public function stampsFor(PostIngestResult $result): array
{
return true === $result->live ? self::live() : [];
}
/** @return list<StampInterface> */
public static function live(): array
{
return [new TransportNamesStamp([self::LIVE])];
}
/** @return list<StampInterface> */
public static function backlog(): array
{
return [new TransportNamesStamp([self::BACKLOG])];
}
}
src/Service/Mail/PostIngest/ExtractEventsStep.php
+13 -12
View File
@@ -20,29 +20,30 @@ use Symfony\Component\Messenger\MessageBusInterface;
* The first implementation of PostIngestStepInterface, and the reason it
* exists — before it, this would have had to be wired into all three sync
* paths by hand.
*
* RECENT MAIL ONLY. An invitation to a meeting held four years ago is not an
* event anybody needs on a calendar, and for Gmail and Graph reading it means
* fetching raw MIME from the provider — an API call per old invite during the
* one sync where quota matters most. `app:backfill event-extraction` reads
* history when somebody wants it. See RecentMailPolicy.
*/
final readonly class ExtractEventsStep implements PostIngestStepInterface
{
public function __construct(private MessageBusInterface $bus)
{
public function __construct(
private MessageBusInterface $bus,
private RecentMailPolicy $recent,
private EnrichmentRouter $router,
) {
}
public function afterCommit(PostIngestResult $result): void
{
$ids = [];
foreach ($result->messages as $message) {
$id = $message->id;
if (null !== $id) {
$ids[] = (int) $id;
}
}
$ids = $this->recent->recentIds($result);
if ([] === $ids) {
return;
}
$this->bus->dispatch(new ExtractEventsMessage($ids));
$this->bus->dispatch(new ExtractEventsMessage($ids), $this->router->stampsFor($result));
}
}
src/Service/Mail/PostIngest/ReadReceiptStep.php
+11 -12
View File
@@ -26,29 +26,28 @@ use Symfony\Component\Messenger\MessageBusInterface;
* Dispatches and returns, per the interface contract. The real work reads
* message bodies and, for an inbound MDN, hunts the original message by
* Message-ID — none of which belongs on a worker holding an IMAP connection.
*
* RECENT MAIL ONLY. A receipt request on mail that predates the import is one
* nobody is going to honour, and a bounce from years ago describes a send this
* installation never made. See RecentMailPolicy.
*/
final readonly class ReadReceiptStep implements PostIngestStepInterface
{
public function __construct(private MessageBusInterface $bus)
{
public function __construct(
private MessageBusInterface $bus,
private RecentMailPolicy $recent,
private EnrichmentRouter $router,
) {
}
public function afterCommit(PostIngestResult $result): void
{
$ids = [];
foreach ($result->messages as $message) {
$id = $message->id;
if (null !== $id) {
$ids[] = (int) $id;
}
}
$ids = $this->recent->recentIds($result);
if ([] === $ids) {
return;
}
$this->bus->dispatch(new ProcessReadReceiptsMessage($ids));
$this->bus->dispatch(new ProcessReadReceiptsMessage($ids), $this->router->stampsFor($result));
}
}
src/Service/Mail/PostIngest/RecentMailPolicy.php
+109
View File
@@ -0,0 +1,109 @@
<?php
declare(strict_types=1);
namespace App\Service\Mail\PostIngest;
use App\Domain\DTO\Mail\PostIngestResult;
use App\Entity\Mail\Message;
use DateTimeImmutable;
/**
* How old a message may be and still be worth doing things to on arrival.
*
* WHY ARRIVING AND BEING NEW STOPPED BEING THE SAME THING
* ──────────────────────────────────────────────────────
* Every post-ingest step used to treat "this message has just been ingested"
* as "this message has just arrived", and on an ordinary day the two are the
* same. On the day an account is added they are not: a mailbox of fifty
* thousand messages is ingested in an afternoon, and each one was queued for a
* model call, an event extraction and a receipt scan as though somebody were
* waiting to read it. A newsletter from 2019 does not need a second opinion on
* its tab, an invitation to a meeting held four years ago is not an event, and
* nobody is owed a read receipt for mail that predates the installation.
*
* So the question a step asks is no longer "is this in my batch" but "is this
* recent", and this is the one place that says what recent means.
*
* BY THE MAIL'S OWN DATE, NOT BY WHEN IT WAS IMPORTED. The alternative — a
* flag on the account saying "the first import is still running" — answers a
* different question and answers it worse: it would skip last week's mail
* during an import and process a ten-year-old folder somebody subscribed to
* afterwards. Age is the property that actually decides whether the work is
* worth doing, and it is already on the row.
*
* WHAT OLD MAIL STILL GETS: a row, a thread, the rule cascade's category, the
* user's own rules and its contacts harvested — everything PostIngestPipeline
* does itself. What it does not get is anything a step queues. The backfill
* tasks (`app:backfill insights`, `event-extraction`, …) are how history is
* read when somebody wants it, and old mail is classified by the assistant
* later and in the background — see App\Service\Ai\ClassificationCatchUp.
*
* Read through `%env(default:…)%` in config/services.yaml, like BackfillPolicy
* and for its reason: the answer for almost everybody is "leave it alone".
*/
final readonly class RecentMailPolicy
{
/** Days back from now that still count as recent. */
public int $recentDays;
public function __construct(int $recentDays)
{
// Clamped rather than validated: it comes from the environment, and a
// typo must not be able to switch every step off (zero) or quietly
// restore the behaviour this exists to remove (a million).
$this->recentDays = max(1, min(3_650, $recentDays));
}
public function cutoff(?DateTimeImmutable $now = null): DateTimeImmutable
{
return ($now ?? new DateTimeImmutable())->modify(sprintf('-%d days', $this->recentDays));
}
/**
* receivedAt first and sentAt as the fallback, the order EventProposer
* anchors on.
*
* A MESSAGE WITH NO DATE AT ALL IS RECENT. It is rare — a builder that
* could not read either — and of the two mistakes available, doing a little
* unnecessary work is the one nobody notices. Treating it as old would
* silently drop a real arrival out of everything the steps do.
*/
public function isRecent(Message $message, ?DateTimeImmutable $now = null): bool
{
$date = $message->receivedAt ?? $message->sentAt;
if (null === $date) {
return true;
}
return $date >= $this->cutoff($now);
}
/**
* The ids in a batch that a step should queue work for.
*
* Here rather than in each step because five of them were about to carry
* the same loop, and the loop is where the cutoff has to be applied the
* same way every time.
*
* @return list<int>
*/
public function recentIds(PostIngestResult $result, ?DateTimeImmutable $now = null): array
{
$now ??= new DateTimeImmutable();
$ids = [];
foreach ($result->messages as $message) {
$id = $message->id;
if (null === $id || false === $this->isRecent($message, $now)) {
continue;
}
$ids[] = (int) $id;
}
return $ids;
}
}
src/Service/Mail/PostIngestPipeline.php
+29 -1
View File
@@ -11,6 +11,7 @@ use App\Entity\Mail\Account;
use App\Repository\Mail\ContactRepository;
use App\Service\Imap\MessageThreader;
use App\Service\Rule\MailRuleEngine;
use DateTimeImmutable;
use Doctrine\ORM\EntityManagerInterface;
use Psr\Log\LoggerInterface;
use Symfony\Component\DependencyInjection\Attribute\AutowireIterator;
@@ -46,6 +47,8 @@ final readonly class PostIngestPipeline
private MailBodySanitizer $sanitizer,
private RawMessageResolver $rawResolver,
private MessageCategorizer $categorizer,
private InitialImportState $importState,
private ClassificationHold $hold,
private MessageThreader $messageThreader,
private MailRuleEngine $ruleEngine,
private MailChangeRecorder $changes,
@@ -85,6 +88,20 @@ final readonly class PostIngestPipeline
$accounts = [];
$ruleTargets = [];
// Is this mail trickling in, or a mailbox being imported? Asked once,
// before the loop, because two things below turn on it and both have
// to agree: whether a message is held back for the assistant, and
// which queue the steps put its follow-up work on. See
// InitialImportState for what "imported" means per provider.
$owners = [];
foreach ($ingested as $item) {
$owners[(int) $item->account->id] = $item->account;
}
$live = $this->importState->allComplete($owners);
$holding = true === $live && true === $this->hold->isOnFor($user);
foreach ($ingested as $item) {
$message = $item->message;
$accountId = (int) $item->account->id;
@@ -113,6 +130,17 @@ final readonly class PostIngestPipeline
$message->category = $this->categorizer->categorize($message, $correspondents, $user?->categorySorting);
// BEFORE THREADING, because the thread is where a hold takes
// effect: assignThread() gives a held message's new thread no
// category, and every tab query already reads that as "in no tab".
// The category written one line up stays — it is the rules' answer,
// and it is what the message is filed under if the model never
// replies. See ClassificationHold and Message::$categoryHeldAt.
if (true === $holding
&& true === $this->hold->shouldHold($message, $item->account, $correspondents, $user?->categorySorting)) {
$message->categoryHeldAt = new DateTimeImmutable();
}
try {
$this->messageThreader->assignThread($message, $item->account);
} catch (\Throwable $e) {
@@ -155,7 +183,7 @@ final readonly class PostIngestPipeline
// The change-log rows recorded just now.
$this->em->flush();
$result = new PostIngestResult($messages, $accounts, $threadIdsByAccount);
$result = new PostIngestResult($messages, $accounts, $threadIdsByAccount, $live);
$this->notifySteps($result);
src/Service/System/Work/BackgroundProcesses.php
+9 -1
View File
@@ -16,7 +16,7 @@ use Symfony\Component\DependencyInjection\Attribute\Autowire;
* consumer already inside a long handler cannot pick up anything else, however
* the queues are prioritised, so a send would wait behind a sync. What changed
* is only where the processes live. They used to be seven containers of one
* image; now they are seven children of one supervisor.
* image; now they are children of one supervisor, and there are ten of them.
*
* The names are the containers' old ones, deliberately. Each child gets its
* name as APP_CONTAINER_NAME, which is what the heartbeats are keyed by and
@@ -73,6 +73,14 @@ final readonly class BackgroundProcesses
new BackgroundProcess('imap-supervisor', ['php', 'bin/console', 'app:imap:supervise']),
$this->consumer('worker-export', 'export'),
$this->consumer('worker-ingest', 'ingest'),
// Mail that has just arrived, and nothing else, so that it is never
// behind an import. See messenger.yaml on the three enrich queues.
$this->consumer('worker-live', 'enrich_live'),
// Ends holds, and is otherwise idle on purpose: see messenger.yaml.
$this->consumer('worker-release', 'release'),
// IN THIS ORDER, which is the priority: Messenger drains `enrich`
// before it looks at `enrich_backlog`.
$this->consumer('worker-enrich', 'enrich', 'enrich_backlog'),
// Also drains `async`, the pre-split queue; see messenger.yaml.
$this->consumer('worker-maintenance', 'maintenance', 'async'),
$this->consumer('worker-bulk', 'bulk'),
templates/admin/ai/_frame.html.twig
+24
View File
@@ -262,6 +262,30 @@ poll would empty a field mid-word.
{{ form_row(form.searchEnabled) }}
{{ form_row(form.categorisationEnabled) }}
{# The two that qualify the switch above, set in from
it so they read as belonging to it, and the measured
cost directly underneath: holding is a trade of a
short wait for mail that never moves tabs, and how
short is a property of this host that only a
measurement can state. #}
<div class="flex flex-col gap-4 pl-4 border-l border-subtle">
{{ form_row(form.holdUntilClassified) }}
{{ form_row(form.holdMaxSeconds) }}
<p class="text-[12px] leading-snug text-muted" data-testid="ai-hold-delay">
{% if hold.held > 0 %}
{{ 'admin.ai.hold.measured'|trans({
'%count%': hold.held,
'%median%': hold.median|number_format(1),
'%p95%': hold.p95|number_format(1),
'%timeout%': hold.timedOut,
}) }}
{% else %}
{{ 'admin.ai.hold.no_data'|trans }}
{% endif %}
</p>
</div>
{{ form_row(form.writingHelpEnabled) }}
{{ form_row(form.summaryEnabled) }}
tests/Command/Ai/IndexNewMailCommandTest.php
+1 -1
View File
@@ -185,7 +185,7 @@ final class IndexNewMailCommandTest extends KernelTestCase
private function queue(): InMemoryTransport
{
$transport = self::getContainer()->get('messenger.transport.ingest');
$transport = self::getContainer()->get('messenger.transport.enrich');
self::assertInstanceOf(InMemoryTransport::class, $transport);
tests/Jmap/State/EmailChangesTest.php
+4
View File
@@ -24,6 +24,8 @@ use App\Service\Imap\MessageSendService;
use App\Service\Imap\MessageThreader;
use App\Service\Label\LabelResolver;
use App\Service\Label\ThreadLabelSynchronizer;
use App\Service\Mail\InitialImportState;
use App\Service\Mail\ClassificationHold;
use App\Service\Mail\AttachmentResolver;
use App\Service\Mail\DraftPersister;
use App\Service\Mail\MailBodySanitizer;
@@ -422,6 +424,8 @@ final class EmailChangesTest extends KernelTestCase
$container->get(MailBodySanitizer::class),
$container->get(RawMessageResolver::class),
$container->get(MessageCategorizer::class),
$container->get(InitialImportState::class),
$container->get(ClassificationHold::class),
$container->get(MessageThreader::class),
$container->get(MailRuleEngine::class),
$container->get(MailChangeRecorder::class),
tests/Service/Ai/ClassificationCatchUpTest.php
+269
View File
@@ -0,0 +1,269 @@
<?php
declare(strict_types=1);
namespace App\Tests\Service\Ai;
use App\Entity\Ai\AiSettings;
use App\Entity\Mail\Account;
use App\Entity\Mail\Mailbox;
use App\Entity\Mail\Message;
use App\Entity\User\User;
use App\Infrastructure\Messaging\Message\ClassifyMailMessage;
use App\Service\Ai\BackfillPolicy;
use App\Service\Ai\ClassificationCatchUp;
use DateTimeImmutable;
use Doctrine\DBAL\Connection;
use Doctrine\ORM\EntityManagerInterface;
use Symfony\Bundle\FrameworkBundle\Test\KernelTestCase;
use Symfony\Component\Messenger\Transport\InMemory\InMemoryTransport;
/**
* Old mail is asked about after an import, not during it — and then newest
* first, a bounded handful at a time, on the queue behind every other.
*
* Each of those is a separate way the first import used to go wrong, so each
* has a test.
*/
final class ClassificationCatchUpTest extends KernelTestCase
{
private EntityManagerInterface $em;
private Connection $connection;
private User $user;
private Mailbox $mailbox;
protected function setUp(): void
{
self::bootKernel();
$container = self::getContainer();
$this->em = $container->get(EntityManagerInterface::class);
$this->connection = $container->get(Connection::class);
$this->connection->beginTransaction();
$this->seed();
$this->assistantSorts();
}
protected function tearDown(): void
{
if (true === $this->connection->isTransactionActive()) {
$this->connection->rollBack();
}
parent::tearDown();
}
/**
* A mailbox still arriving is a moving target, and its own recent mail has
* first call on the model.
*/
public function testNothingIsQueuedWhileTheImportIsStillRunning(): void
{
$this->mail(5);
self::assertSame(0, $this->catchUp()->sweep($this->user, 100));
self::assertSame([], $this->backlog()->getSent());
}
public function testTheNewestUnaskedMailIsQueuedFirstAndOnlyUpToTheLimit(): void
{
$ids = $this->mail(6);
$this->importFinished();
self::assertSame(4, $this->catchUp()->sweep($this->user, 4));
// mail() dates them oldest first, so the newest four are the last four
// and they come out newest first.
self::assertSame(array_reverse(array_slice($ids, 2)), $this->queuedIds());
}
/**
* The stamp is the progress marker: asked once, whatever the answer was.
*/
public function testMailAlreadyAskedAboutIsNotQueuedAgain(): void
{
$ids = $this->mail(3);
$this->importFinished();
$this->connection->executeStatement(
'UPDATE message SET ai_categorised_at = NOW() WHERE id = :id',
['id' => $ids[2]],
);
$this->catchUp()->sweep($this->user, 100);
self::assertSame([$ids[1], $ids[0]], $this->queuedIds());
}
public function testTheWorkIsChunkedIntoJobsThatAreShort(): void
{
$this->mail(5);
$this->importFinished();
$this->catchUp()->sweep($this->user, 100);
$batchSize = self::getContainer()->get(BackfillPolicy::class)->batchSize;
self::assertCount((int) ceil(5 / $batchSize), $this->backlog()->getSent());
}
public function testNothingIsQueuedWhileSortingByAssistantIsOff(): void
{
$this->mail(3);
$this->importFinished();
$this->assistantSorts(enabled: false);
self::assertSame(0, $this->catchUp()->sweep($this->user, 100));
}
/**
* The queue is what stands in for state: a run that posted on top of the
* last run's unfinished work would queue the same ids twice.
*/
public function testARunWaitsForTheLastRunToBeWorkedThrough(): void
{
self::assertTrue($this->catchUp()->mayRun());
$this->connection->insert('messenger_messages', [
'body' => 'x',
'headers' => '{}',
'queue_name' => 'enrich_backlog',
'created_at' => new DateTimeImmutable(),
'available_at' => new DateTimeImmutable(),
], [
'created_at' => 'datetime_immutable',
'available_at' => 'datetime_immutable',
]);
self::assertFalse($this->catchUp()->mayRun());
}
private function catchUp(): ClassificationCatchUp
{
return self::getContainer()->get(ClassificationCatchUp::class);
}
/** @return list<int> */
private function queuedIds(): array
{
$ids = [];
foreach ($this->backlog()->getSent() as $envelope) {
$message = $envelope->getMessage();
self::assertInstanceOf(ClassifyMailMessage::class, $message);
self::assertFalse($message->force);
$ids = [...$ids, ...$message->messageIds];
}
return $ids;
}
/**
* enrich_backlog and no other: the routing table sends this message to
* `enrich`, where it would be level with an import's recent mail.
*/
private function backlog(): InMemoryTransport
{
self::assertSame([], $this->transport('enrich')->getSent());
self::assertSame([], $this->transport('ingest')->getSent());
return $this->transport('enrich_backlog');
}
private function transport(string $name): InMemoryTransport
{
$transport = self::getContainer()->get('messenger.transport.' . $name);
self::assertInstanceOf(InMemoryTransport::class, $transport);
return $transport;
}
private function assistantSorts(bool $enabled = true): void
{
$settings = $this->em->getRepository(AiSettings::class)->findOneBy([]) ?? new AiSettings();
$settings->isEnabled = true;
$settings->baseUrl = 'http://model-host.invalid:11434';
$settings->chatModel = 'qwen3:4b-instruct';
$settings->categorisationEnabled = $enabled;
$this->em->persist($settings);
$this->em->flush();
}
private function importFinished(): void
{
$this->mailbox->syncedAt = new DateTimeImmutable('-1 day');
$this->em->flush();
}
/**
* Old mail, oldest first: the i-th is a day newer than the one before it.
*
* @return list<int>
*/
private function mail(int $count): array
{
$ids = [];
for ($i = 0; $i < $count; $i++) {
$message = new Message();
$message->account = $this->mailbox->account;
$message->mailbox = $this->mailbox;
$message->subject = 'Backlog fixture ' . $i;
$message->fromAddress = 'somebody@elsewhere.test';
$message->receivedAt = new DateTimeImmutable(sprintf('-%d days', 400 - $i));
$message->hasAttachments = false;
$message->messageId = sprintf('<backlog-%d-%s@example.test>', $i, uniqid('', true));
$this->em->persist($message);
$this->em->flush();
$ids[] = (int) $message->id;
}
return $ids;
}
private function seed(): void
{
$this->user = new User();
$this->user->email = 'backlog-' . uniqid('', true) . '@example.test';
$this->user->nameFirst = 'Back';
$this->user->nameLast = 'Log';
$this->user->roles = ['ROLE_USER'];
$this->user->password = 'x';
$this->em->persist($this->user);
$account = new Account();
$account->usr = $this->user;
$account->email = 'backlog-fixture@example.test';
$account->username = 'backlog-fixture@example.test';
$account->imapHost = 'localhost';
$account->imapPort = 993;
$account->imapEncryption = 'ssl';
$account->smtpHost = 'localhost';
$account->smtpPort = 587;
$account->smtpEncryption = 'starttls';
$account->password = 'x';
$account->authType = 'password';
$account->isActive = true;
$this->em->persist($account);
$this->mailbox = new Mailbox();
$this->mailbox->account = $account;
$this->mailbox->name = 'INBOX';
$this->mailbox->fullPath = 'INBOX';
$this->mailbox->isSyncEnabled = true;
$this->mailbox->isIdleEnabled = false;
$this->em->persist($this->mailbox);
$this->em->flush();
}
}
tests/Service/Ai/EmbeddingCatchUpTest.php
+2 -2
View File
@@ -329,11 +329,11 @@ final class EmbeddingCatchUpTest extends KernelTestCase
private function queue(): InMemoryTransport
{
// ingest, because catching up IS mail arriving, only later. The routing
// enrich, with the rest of what happens to mail once it is in. The routing
// is part of what is being asserted: unrouted, a Messenger message is
// handled in the process that dispatched it, and the search would wait
// on the model after all.
$transport = self::getContainer()->get('messenger.transport.ingest');
$transport = self::getContainer()->get('messenger.transport.enrich');
self::assertInstanceOf(InMemoryTransport::class, $transport);
tests/Service/Mail/ClassificationHoldFlowTest.php
+514
View File
@@ -0,0 +1,514 @@
<?php
declare(strict_types=1);
namespace App\Tests\Service\Mail;
use App\Domain\DTO\Mail\IngestedMessage;
use App\Domain\Enum\Mail\CategorySource;
use App\Domain\Enum\Mail\MessageCategory;
use App\Entity\Ai\AiSettings;
use App\Entity\Mail\Account;
use App\Entity\Mail\Mailbox;
use App\Entity\Mail\Message;
use App\Entity\User\User;
use App\Infrastructure\Messaging\Handler\ClassifyMailHandler;
use App\Infrastructure\Messaging\Message\ClassifyMailMessage;
use App\Infrastructure\Messaging\Message\ExtractEventsMessage;
use App\Infrastructure\Messaging\Message\ExtractInsightsMessage;
use App\Infrastructure\Messaging\Message\ProcessReadReceiptsMessage;
use App\Infrastructure\Messaging\Message\ProposeEventsMessage;
use App\Infrastructure\Messaging\Message\ReleaseHeldMailMessage;
use App\Repository\Ai\AiSettingsRepository;
use App\Repository\Mail\ContactRepository;
use App\Repository\Mail\MessageRepository;
use App\Repository\Mail\MessageThreadRepository;
use App\Service\Ai\AiAssistant;
use App\Service\Ai\AiCallRecorder;
use App\Service\Ai\AiPermissions;
use App\Service\Ai\LiveMailPriority;
use App\Service\Ai\PromptLibrary;
use App\Service\Mail\MessageCategorizer;
use App\Service\Ai\OllamaClient;
use App\Service\Mail\HeldMailReleaser;
use App\Service\Mail\PostIngestPipeline;
use DateTimeImmutable;
use Doctrine\DBAL\Connection;
use Doctrine\ORM\EntityManagerInterface;
use Psr\Log\NullLogger;
use Symfony\Bundle\FrameworkBundle\Test\KernelTestCase;
use Symfony\Component\HttpClient\MockHttpClient;
use Symfony\Component\HttpClient\Response\MockResponse;
use Symfony\Component\Messenger\Stamp\DelayStamp;
use Symfony\Component\Messenger\Transport\InMemory\InMemoryTransport;
/**
* What happens to a message between being fetched and being seen, end to end
* through the real pipeline and its real steps.
*
* TWO PROMISES ARE PINNED HERE, and they pull in opposite directions:
*
* · Fetching mail is all the fetching worker does. Everything that follows
* a message is queued somewhere other than `ingest`, old mail is not
* queued at all, and mail that has just arrived goes to a queue no import
* can be in front of.
*
* · Mail that the assistant is going to sort is not shown in a tab until it
* has been — and is shown regardless once the wait runs out.
*
* The real container's pipeline rather than one built by hand, unlike
* PostIngestPipelineTest: the steps are the subject.
*/
final class ClassificationHoldFlowTest extends KernelTestCase
{
private EntityManagerInterface $em;
private Connection $connection;
private User $user;
private Account $account;
private Mailbox $mailbox;
protected function setUp(): void
{
self::bootKernel();
$container = self::getContainer();
$this->em = $container->get(EntityManagerInterface::class);
$this->connection = $container->get(Connection::class);
// Never committed, so the suite leaves nothing behind.
$this->connection->beginTransaction();
$this->seed();
}
protected function tearDown(): void
{
if (true === $this->connection->isTransactionActive()) {
$this->connection->rollBack();
}
parent::tearDown();
}
public function testNothingThatFollowsAMessageIsQueuedBesideTheFetching(): void
{
$this->assistantSorts();
$this->importFinished();
$this->ingest($this->message('fresh', '-1 hour'));
self::assertSame([], $this->transport('ingest')->getSent(), 'ingest is for talking to a mail provider');
}
public function testMailThatHasJustArrivedGoesToTheLiveQueue(): void
{
$this->importFinished();
$message = $this->message('fresh', '-1 hour');
$this->ingest($message);
$live = $this->classesOn('enrich_live');
self::assertContains(ExtractEventsMessage::class, $live);
self::assertContains(ExtractInsightsMessage::class, $live);
self::assertContains(ProposeEventsMessage::class, $live);
self::assertContains(ProcessReadReceiptsMessage::class, $live);
self::assertSame([], $this->classesOn('enrich'), 'nothing of a finished account is behind an import');
}
/**
* The same recent message, on an account still being imported: wanted,
* not watched, and so on the shared queue rather than the live one.
*/
public function testRecentMailOfAnImportGoesToTheSharedQueue(): void
{
$this->assistantSorts();
// No importFinished(): the folder has never completed a sync.
$message = $this->message('fresh', '-1 hour');
$this->ingest($message);
self::assertSame([], $this->classesOn('enrich_live'));
self::assertContains(ClassifyMailMessage::class, $this->classesOn('enrich'));
self::assertNull($message->categoryHeldAt, 'an import that held its mail would hide a mailbox behind a model');
self::assertNotNull($message->thread?->category);
}
/**
* The newsletter from 2019. It gets a row, a thread and the rules' answer
* — and no job of any kind.
*/
public function testOldMailIsStoredAndNothingElse(): void
{
$this->assistantSorts();
$message = $this->message('ancient', '-3 years');
$this->ingest($message);
self::assertNotNull($message->thread);
self::assertNotNull($message->category);
foreach (['ingest', 'enrich_live', 'enrich', 'enrich_backlog'] as $queue) {
self::assertSame([], $this->classesOn($queue), $queue . ' holds nothing for mail this old');
}
}
public function testArrivingMailIsKeptOutOfTheTabsWhileTheAssistantSorts(): void
{
$this->assistantSorts();
$this->importFinished();
$message = $this->message('fresh', '-1 minute');
$this->ingest($message);
self::assertTrue($message->isCategoryHeld());
self::assertNotNull($message->category, 'the rules\' answer is kept: it is what a release without a verdict files under');
self::assertNull($message->thread?->category, 'no category is how a thread says it is in no tab');
// One message to an envelope, on the live queue, with the timed way
// out queued behind it.
$classify = $this->sentOn('enrich_live', ClassifyMailMessage::class);
self::assertCount(1, $classify);
self::assertSame([(int) $message->id], $classify[0]->getMessage()->messageIds);
// On its own queue. NOT the live one, whose worker may be inside the
// very model call that is taking too long when this falls due, and not
// export, where it would share a worker with mail leaving.
self::assertSame([], $this->sentOn('enrich_live', ReleaseHeldMailMessage::class));
self::assertSame([], $this->sentOn('export', ReleaseHeldMailMessage::class));
$release = $this->sentOn('release', ReleaseHeldMailMessage::class);
self::assertCount(1, $release);
self::assertSame([(int) $message->id], $release[0]->getMessage()->messageIds);
$delay = $release[0]->last(DelayStamp::class);
self::assertNotNull($delay);
self::assertSame(AiSettings::DEFAULT_HOLD_MAX_SECONDS * 1000, $delay->getDelay(), 'the hold ends when its ceiling does');
}
public function testAReleaseFilesTheMailWhereTheRulesPutItAndOnlyOnce(): void
{
$this->assistantSorts();
$this->importFinished();
$message = $this->message('fresh', '-1 minute');
$this->ingest($message);
$releaser = self::getContainer()->get(HeldMailReleaser::class);
self::assertSame(1, $releaser->release([$message]));
$this->em->refresh($message->thread);
self::assertFalse($message->isCategoryHeld());
self::assertNotNull($message->categoryReleasedAt);
self::assertSame($message->category, $message->thread->category);
// The timed release arriving after the verdict has already released
// the message is the ordinary case, and must be a no-op.
self::assertSame(0, $releaser->release([$message]));
}
public function testMailHeldTooLongIsFoundByTheSweep(): void
{
$this->assistantSorts();
$this->importFinished();
$message = $this->message('fresh', '-1 minute');
$this->ingest($message);
$repository = self::getContainer()->get(MessageRepository::class);
self::assertSame([], $repository->findHeldBefore(new DateTimeImmutable('-1 hour')));
$found = $repository->findHeldBefore(new DateTimeImmutable('+1 minute'));
self::assertCount(1, $found);
self::assertSame($message->id, $found[0]->id);
}
/**
* The model host is a box on somebody's network and may simply be off.
* Held mail must come out anyway, at once, under the rules' answer. It
* is stamped as asked, like any other unanswered message: the question was
* put once and is not put again.
*/
public function testAHostThatDoesNotAnswerReleasesTheMailUnderTheRules(): void
{
$this->assistantSorts();
$this->importFinished();
$message = $this->message('fresh', '-1 minute');
$this->ingest($message);
self::getContainer()->get(ClassifyMailHandler::class)(new ClassifyMailMessage([(int) $message->id]));
$this->em->refresh($message->thread);
self::assertFalse($message->isCategoryHeld());
self::assertNotNull($message->aiCategorisedAt);
self::assertSame(MessageCategory::Primary, $message->thread->category);
}
/**
* What Admin → AI shows beside the switch: how long holding delayed mail,
* and how often the wait ran out with nothing to show for it.
*/
public function testTheDelayIsMeasuredFromTheHoldToTheRelease(): void
{
$this->assistantSorts();
$this->importFinished();
$answered = $this->message('answered', '-1 minute');
$timedOut = $this->message('timed-out', '-1 minute');
$this->ingest($answered);
$this->ingest($timedOut);
$heldAt = new DateTimeImmutable('-1 minute');
$answered->categoryHeldAt = $heldAt;
$answered->aiCategorisedAt = $heldAt->modify('+2 seconds');
$answered->categoryReleasedAt = $heldAt->modify('+2 seconds');
$timedOut->categoryHeldAt = $heldAt;
$timedOut->categoryReleasedAt = $heldAt->modify('+10 seconds');
$this->em->flush();
$repository = self::getContainer()->get(MessageRepository::class);
$stats = $repository->holdDelayStats(new DateTimeImmutable('-1 hour'));
self::assertSame(2, $stats['held']);
self::assertSame(1, $stats['timedOut']);
self::assertEqualsWithDelta(6.0, $stats['median'], 0.01);
self::assertGreaterThan(9.0, $stats['p95']);
self::assertSame(
['held' => 0, 'timedOut' => 0, 'median' => null, 'p95' => null],
$repository->holdDelayStats(new DateTimeImmutable('+1 hour')),
'no data is null, not zero: a median of zero would read as instant',
);
}
/**
* The wait ran out, the mail was shown under the rules' answer, and then
* the assistant's answer arrived. It still decides — the same question,
* asked once, moves the mail — and the release time is left alone, because
* that pair of timestamps is the measurement.
*/
public function testAnAnswerThatArrivesAfterTheWaitStillMovesTheMail(): void
{
$this->assistantSorts();
$this->importFinished();
$message = $this->message('fresh', '-1 minute');
$this->ingest($message);
// The timed release got there first.
self::getContainer()->get(HeldMailReleaser::class)->release([$message], new DateTimeImmutable('-30 seconds'));
$this->em->refresh($message->thread);
self::assertSame(MessageCategory::Primary, $message->thread->category);
$releasedAt = $message->categoryReleasedAt;
$this->handlerWithAModelThatSays('promotions')(new ClassifyMailMessage([(int) $message->id]));
$this->em->refresh($message->thread);
self::assertSame(MessageCategory::Promotions, $message->aiCategory);
self::assertSame(MessageCategory::Promotions, $message->thread->category);
// To the second: the column keeps no fraction, and the handler re-reads the row.
self::assertSame($releasedAt?->getTimestamp(), $message->categoryReleasedAt?->getTimestamp());
}
public function testNothingIsHeldWhenAnAdministratorSwitchedHoldingOff(): void
{
$this->assistantSorts(hold: false);
$this->importFinished();
$message = $this->message('fresh', '-1 minute');
$this->ingest($message);
self::assertNull($message->categoryHeldAt);
self::assertNotNull($message->thread?->category);
self::assertSame([], $this->sentOn('release', ReleaseHeldMailMessage::class));
}
/**
* Somebody sorting by rules has an answer that is already final — no
* verdict is ever consulted — so there is nothing to wait for.
*/
public function testNothingIsHeldForSomebodyWhoSortsByRules(): void
{
$this->assistantSorts(source: CategorySource::Rules);
$this->importFinished();
$message = $this->message('fresh', '-1 minute');
$this->ingest($message);
self::assertNull($message->categoryHeldAt);
self::assertSame(MessageCategory::Primary, $message->thread?->category);
}
/**
* The real handler around a model host that answers.
*
* Built by hand because the container's OllamaClient is already in use by
* the time a test body runs and cannot be swapped; everything but the
* assistant is the container's own.
*/
private function handlerWithAModelThatSays(string $answer): ClassifyMailHandler
{
$container = self::getContainer();
$assistant = new AiAssistant(
$container->get(AiSettingsRepository::class),
new OllamaClient(
new MockHttpClient(new MockResponse((string) json_encode(['message' => ['content' => $answer]]))),
new NullLogger(),
),
$container->get(AiCallRecorder::class),
new NullLogger(),
);
return new ClassifyMailHandler(
$container->get(MessageRepository::class),
$container->get(MessageThreadRepository::class),
$container->get(ContactRepository::class),
$container->get(MessageCategorizer::class),
$assistant,
$container->get(AiPermissions::class),
$container->get(PromptLibrary::class),
$container->get(LiveMailPriority::class),
$container->get(HeldMailReleaser::class),
$this->em,
new NullLogger(),
);
}
private function ingest(Message $message): void
{
self::getContainer()->get(PostIngestPipeline::class)->run(
$this->account,
[new IngestedMessage($message, $this->account, null)],
);
}
/** @return list<class-string> */
private function classesOn(string $queue): array
{
return array_map(
static fn ($envelope): string => $envelope->getMessage()::class,
$this->transport($queue)->getSent(),
);
}
/**
* @param class-string $class
*
* @return list<\Symfony\Component\Messenger\Envelope>
*/
private function sentOn(string $queue, string $class): array
{
return array_values(array_filter(
$this->transport($queue)->getSent(),
static fn ($envelope): bool => $envelope->getMessage() instanceof $class,
));
}
private function transport(string $name): InMemoryTransport
{
$transport = self::getContainer()->get('messenger.transport.' . $name);
self::assertInstanceOf(InMemoryTransport::class, $transport);
return $transport;
}
/** The installation has the assistant sorting mail, and so does this person. */
private function assistantSorts(CategorySource $source = CategorySource::Assistant, bool $hold = true): void
{
$settings = $this->em->getRepository(AiSettings::class)->findOneBy([]) ?? new AiSettings();
$settings->isEnabled = true;
$settings->baseUrl = 'http://model-host.invalid:11434';
$settings->chatModel = 'qwen3:4b-instruct';
$settings->categorisationEnabled = true;
$settings->holdUntilClassified = $hold;
$settings->holdMaxSeconds = AiSettings::DEFAULT_HOLD_MAX_SECONDS;
$this->em->persist($settings);
$this->user->categorySorting->source = $source->value;
$this->em->flush();
}
/** What ends an IMAP import: every folder has completed a sync. */
private function importFinished(): void
{
$this->mailbox->syncedAt = new DateTimeImmutable('-1 day');
$this->em->flush();
}
/** Persisted and flushed, which is the pipeline's stated precondition. */
private function message(string $slug, string $received): Message
{
$message = new Message();
$message->account = $this->account;
$message->mailbox = $this->mailbox;
$message->subject = 'Hold fixture ' . $slug;
$message->fromAddress = 'somebody@elsewhere.test';
$message->receivedAt = new DateTimeImmutable($received);
$message->hasAttachments = false;
$message->bodyHtml = '<p>hello</p>';
$message->messageId = sprintf('<hold-%s-%s@example.test>', $slug, uniqid('', true));
$this->em->persist($message);
$this->em->flush();
return $message;
}
private function seed(): void
{
$this->user = new User();
$this->user->email = 'hold-' . uniqid('', true) . '@example.test';
$this->user->nameFirst = 'Hold';
$this->user->nameLast = 'Flow';
$this->user->roles = ['ROLE_USER'];
$this->user->password = 'x';
$this->em->persist($this->user);
$this->account = new Account();
$this->account->usr = $this->user;
$this->account->email = 'hold-fixture@example.test';
$this->account->username = 'hold-fixture@example.test';
$this->account->imapHost = 'localhost';
$this->account->imapPort = 993;
$this->account->imapEncryption = 'ssl';
$this->account->smtpHost = 'localhost';
$this->account->smtpPort = 587;
$this->account->smtpEncryption = 'starttls';
$this->account->password = 'x';
$this->account->authType = 'password';
$this->account->isActive = true;
$this->em->persist($this->account);
$this->mailbox = new Mailbox();
$this->mailbox->account = $this->account;
$this->mailbox->name = 'INBOX';
$this->mailbox->fullPath = 'INBOX';
$this->mailbox->isSyncEnabled = true;
$this->mailbox->isIdleEnabled = false;
$this->em->persist($this->mailbox);
$this->em->flush();
}
}
tests/Service/Mail/PostIngest/RecentMailPolicyTest.php
+76
View File
@@ -0,0 +1,76 @@
<?php
declare(strict_types=1);
namespace App\Tests\Service\Mail\PostIngest;
use App\Entity\Mail\Message;
use App\Service\Mail\PostIngest\RecentMailPolicy;
use DateTimeImmutable;
use PHPUnit\Framework\TestCase;
/**
* The one definition of "recent" that every post-ingest step shares.
*
* What is pinned here is the edge and the two fallbacks, because those are
* where five steps would each have chosen slightly differently.
*/
final class RecentMailPolicyTest extends TestCase
{
private const string NOW = '2026-10-07 12:00:00';
public function testMailInsideTheWindowIsRecentAndMailOutsideItIsNot(): void
{
$policy = new RecentMailPolicy(30);
$now = new DateTimeImmutable(self::NOW);
self::assertTrue($policy->isRecent($this->received('2026-10-06 12:00:00'), $now));
self::assertTrue($policy->isRecent($this->received('2026-09-07 12:00:00'), $now), 'the cutoff itself is inside');
self::assertFalse($policy->isRecent($this->received('2026-09-07 11:59:59'), $now));
self::assertFalse($policy->isRecent($this->received('2019-03-01 08:00:00'), $now));
}
/**
* A message is dated by when it was received and, failing that, by when it
* was sent — the order EventProposer anchors on.
*/
public function testTheSentDateStandsInForAMissingReceivedDate(): void
{
$policy = new RecentMailPolicy(30);
$now = new DateTimeImmutable(self::NOW);
$old = new Message();
$old->sentAt = new DateTimeImmutable('2020-01-01 00:00:00');
self::assertFalse($policy->isRecent($old, $now));
}
/**
* Of the two mistakes available for a message with no date, doing a little
* unnecessary work is the one nobody notices.
*/
public function testAMessageWithNoDateAtAllIsRecent(): void
{
self::assertTrue(new RecentMailPolicy(30)->isRecent(new Message(), new DateTimeImmutable(self::NOW)));
}
/**
* It comes from the environment: a typo must not switch every step off,
* nor quietly restore the behaviour this exists to remove.
*/
public function testTheWindowIsClampedRatherThanTrusted(): void
{
self::assertSame(1, new RecentMailPolicy(0)->recentDays);
self::assertSame(1, new RecentMailPolicy(-5)->recentDays);
self::assertSame(3650, new RecentMailPolicy(1_000_000)->recentDays);
self::assertSame(90, new RecentMailPolicy(90)->recentDays);
}
private function received(string $at): Message
{
$message = new Message();
$message->receivedAt = new DateTimeImmutable($at);
return $message;
}
}
tests/Service/Mail/PostIngestPipelineTest.php
+4
View File
@@ -13,6 +13,8 @@ use App\Entity\Mail\Message;
use App\Entity\User\User;
use App\Repository\Mail\ContactRepository;
use App\Service\Imap\MessageThreader;
use App\Service\Mail\InitialImportState;
use App\Service\Mail\ClassificationHold;
use App\Service\Mail\MailBodySanitizer;
use App\Service\Mail\MailChangeRecorder;
use App\Service\Mail\MessageCategorizer;
@@ -239,6 +241,8 @@ final class PostIngestPipelineTest extends KernelTestCase
$container->get(MailBodySanitizer::class),
$container->get(RawMessageResolver::class),
$container->get(MessageCategorizer::class),
$container->get(InitialImportState::class),
$container->get(ClassificationHold::class),
$container->get(MessageThreader::class),
$container->get(MailRuleEngine::class),
$container->get(MailChangeRecorder::class),
tests/Service/Monitoring/QueueMonitorTest.php
+35
View File
@@ -210,6 +210,41 @@ final class QueueMonitorTest extends KernelTestCase
self::fail(sprintf('No stats for queue "%s".', $queue));
}
/**
* What bulk classification asks about the live queue before each model
* call. Two rows must not count: a delayed message — the live queue always
* holds the timed release of whatever is currently held — and a row some
* worker took long ago and died holding.
*/
public function testOnlyWorkThatIsDueOrFreshlyInHandMakesAQueueActive(): void
{
$queues = self::getContainer()->get(\App\Repository\Monitoring\MessengerQueueRepository::class);
$utc = new \DateTimeZone('UTC');
$insert = function (string $available, ?string $delivered = null) use ($utc): void {
$this->connection->insert('messenger_messages', [
'body' => 'x',
'headers' => '{}',
'queue_name' => 'enrich_live',
'created_at' => (new DateTimeImmutable('now', $utc))->format('Y-m-d H:i:s'),
'available_at' => (new DateTimeImmutable($available, $utc))->format('Y-m-d H:i:s'),
'delivered_at' => null === $delivered ? null : (new DateTimeImmutable($delivered, $utc))->format('Y-m-d H:i:s'),
]);
};
$insert('+10 seconds');
$insert('-1 hour', '-50 minutes');
self::assertSame(0, $queues->countActiveOn('enrich_live'));
self::assertSame(2, $queues->countWaitingOn('enrich_live'));
$insert('-1 second');
$insert('-5 seconds', '-2 seconds');
self::assertSame(2, $queues->countActiveOn('enrich_live'));
self::assertSame(0, $queues->countActiveOn('enrich'));
}
/**
* Written the way the doctrine transport writes: a PHP-serialised envelope
* in `body`, which is what makes the decoding under test real.
tests/Service/System/Work/BackgroundProcessesTest.php
+1 -1
View File
@@ -19,7 +19,7 @@ final class BackgroundProcessesTest extends TestCase
public function testEveryProcessKeepsTheNameItsContainerHad(): void
{
self::assertSame(
['mercure', 'imap-supervisor', 'worker-export', 'worker-ingest', 'worker-maintenance', 'worker-bulk', 'scheduler'],
['mercure', 'imap-supervisor', 'worker-export', 'worker-ingest', 'worker-live', 'worker-release', 'worker-enrich', 'worker-maintenance', 'worker-bulk', 'scheduler'],
new BackgroundProcesses(self::IDENTIFIER)->names(),
);
}
translations/messages.de.yaml
+7
View File
@@ -159,6 +159,10 @@ admin:
embedding_keep_alive_help: 'Gleiches Format wie oben. Dieses Modell ist klein und wird ständig gebraucht, deshalb kostet -1 — dauerhaft geladen — meist kaum etwas und spart bei jeder Suche eine Wartezeit; wenn der Host den Speicher übrig hat, lohnt es sich. Leer lassen, um das zu nehmen, was am Host selbst eingestellt ist.'
embedding_model: 'Such-Modell'
embedding_model_help: 'Wandelt Mail in Vektoren um, damit die Suche Bedeutung statt Wörter findet.'
hold: 'Auf den Assistenten warten, bevor neue Mail in einem Tab erscheint'
hold_help: 'Für alle, die ihre Mail vom Assistenten sortieren lassen. Neue Mail bleibt für den Moment, den der Assistent braucht, aus den Posteingangs-Tabs heraus — sie erscheint dann einmal, im richtigen Tab, statt in einem aufzutauchen und in einen anderen zu wandern. Mail aus dem ersten Import wird nie zurückgehalten.'
hold_max_seconds: 'Höchstens warten (Sekunden)'
hold_max_seconds_help: 'Danach wird die Mail dort angezeigt, wo die normalen Regeln sie einordnen — egal, was der Assistent gerade tut — und verschoben, falls der Assistent später antwortet. Zwischen 2 und 300.'
search_query_instruction: 'Anweisung für die Suchanfrage'
search_query_instruction_help: 'Wird vor den Suchtext gesetzt, bevor er in einen Vektor umgewandelt wird. Die meisten Modelle brauchen eine und antworten ohne sie schlecht; für Modelle ohne lässt du das Feld leer. Betrifft nur Suchanfragen — du musst deine Mail dafür nie neu indexieren.'
semantic_min_similarity: 'Mindest-Ähnlichkeit'
@@ -175,6 +179,9 @@ admin:
writing_help: 'Beim Schreiben helfen'
writing_help_help: 'Bietet im Editor Entwürfe und Umformulierungen an, wenn du danach fragst.'
heading: 'Sprachmodell'
hold:
measured: 'Letzte 7 Tage: %count% Nachrichten zurückgehalten, typischerweise %median% s (95 % innerhalb von %p95% s). %timeout% wurden ohne Antwort angezeigt, weil die Wartezeit abgelaufen war.'
no_data: 'In den letzten 7 Tagen wurde keine Mail zurückgehalten — es gibt noch keine Verzögerung zu berichten.'
intro: 'plMail kann ein Sprachmodell benutzen, das du selbst betreibst. Es geht nichts nach außen, nichts wird eingeschaltet, bis du es sagst, und solange das aus ist, funktioniert der Mail-Client genau wie bisher.'
perf:
by_model: 'Nach Modell'
translations/messages.en.yaml
+7
View File
@@ -155,6 +155,10 @@ admin:
embedding_keep_alive_help: 'Same format as above. This model is small and asked for constantly, so -1 — loaded for good — usually costs little and saves a wait on every search; it is worth setting if the host has the memory to spare. Leave empty to use whatever the host itself is set to.'
embedding_model: 'Search model'
embedding_model_help: 'Turns mail into vectors so search can find meaning rather than words.'
hold: 'Wait for the assistant before showing new mail in a tab'
hold_help: 'For people who let the assistant sort their mail. New mail is kept out of the inbox tabs for the moment the assistant needs, so it appears once, in the right tab, instead of appearing in one and moving to another. Mail brought in by a first import is never held.'
hold_max_seconds: 'Wait at most (seconds)'
hold_max_seconds_help: 'Past this, the mail is shown where the ordinary rules put it, whatever the assistant is doing — and moved if the assistant answers later. Between 2 and 300.'
search_query_instruction: 'Query instruction'
search_query_instruction_help: 'Put in front of the search text before it is embedded. Most models need one and answer badly without it; leave it empty for models that do not. Only searches are affected, so changing it never means indexing your mail again.'
semantic_min_similarity: 'Minimum similarity'
@@ -171,6 +175,9 @@ admin:
writing_help: 'Help me write'
writing_help_help: 'Offers drafts and rewrites in the composer, when you ask for them.'
heading: 'Language model'
hold:
measured: 'Last 7 days: %count% messages held, typically for %median% s (95% within %p95% s). %timeout% were shown without an answer because the wait ran out.'
no_data: 'No mail has been held in the last 7 days, so there is no delay to report yet.'
intro: 'plMail can use a language model you run yourself. Nothing is sent anywhere else, nothing is switched on until you say so, and the mail client works exactly as it does now while this is off.'
perf:
by_model: 'Per model'
translations/messages.en_PI.yaml
+7
View File
@@ -156,6 +156,10 @@ admin:
embedding_keep_alive_help: "Same shape as above. This one be small an' called for constantly, so -1 — aboard for good — mostly costs barely a crumb an' spares a wait on ev'ry search; worth settin' if the host has the hold to spare. Leave it empty to sail by whatever the host itself says."
embedding_model: "Divinin' model"
embedding_model_help: "Turns mail into vectors so the search finds meanin', not words."
hold: "Wait fer the oracle afore showin' new mail in a hold"
hold_help: "Fer hands who let the oracle sort their mail. New mail stays out o' the holds fer the moment the oracle needs, so it turns up once, in the right hold, 'stead o' landin' in one an' driftin' to another. Mail hauled in by a first import be never held back."
hold_max_seconds: 'Wait at most (seconds)'
hold_max_seconds_help: "Past this, the mail be shown where the ordinary rules stow it, whatever the oracle be doin' — an' shifted if the oracle answers later. Betwixt 2 an' 300."
search_query_instruction: "Orders for the query"
search_query_instruction_help: "Set afore the search text ere it be turned into a vector. Most models want one an' answer poorly without; leave it empty for them as don't. Only searches be touched, so changin' it never means readin' all yer mail again."
semantic_min_similarity: "Least likeness"
@@ -172,6 +176,9 @@ admin:
writing_help: 'Help me write'
writing_help_help: "Offers drafts an' rewrites at the writin' desk, when ye ask."
heading: 'Oracle'
hold:
measured: "Last 7 days: %count% messages held, most fer %median% s (95% within %p95% s). %timeout% were shown with no answer, the wait havin' run out."
no_data: "No mail's been held these 7 days, so there be no delay to report yet."
intro: "plMail can consult an oracle ye run yerself. Nowt leaves yer waters, nowt starts until ye say so, an' while this be off the mail works exactly as it does now."
perf:
by_model: 'By model'
truenas.compose.yaml
+1 -1
View File
@@ -217,7 +217,7 @@ services:
# Everything that is not the web server, in one container: the IMAP
# supervisor, one consumer per queue, the scheduler and the Mercure hub —
# seven processes, kept running by `app:work`. They used to be seven
# ten processes, kept running by `app:work`. They used to be seven
# containers. Still one PROCESS per queue, so a send never waits behind a
# sync; only the containers merged.
#