Let a backfill that re-reads mail be told how far back to read

`app:backfill events` and `app:backfill insights` re-read stored mail for
what an extractor can now find in it. The usual occasion is a reader that
was just fixed and a mail from this week; run over a whole mailbox they
also dig up every calendar file and parcel since it began — and since
calendar files that name nobody here are now offered rather than added,
that is a card on every one of them.

Both take --since and default to the last 24 hours: a count with h, d or
w, a date, or "all". Anything else is refused and nothing runs. The
command prints the window before it starts.

A separate interface, WindowedBackfillTaskInterface, rather than a
parameter on run(): the other tasks repair a table as a whole, and one
that accepted a window and ignored it would look as though it had done
what it was asked. Given --since, they say they take none.

The insights walk is keyset on the id now instead of OFFSET.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Paul LütznerandClaude Opus 5.5 committed 2026-10-08 16:55:30 +02:00
1 parent 1fc6adb146
commit 4ca055a724
13 files changed
+415 -33

No files matched your search

+4 -2
View File
@@ -17,7 +17,8 @@ empty.
- A date announced as the day the shop hands the parcel to the carrier ("Voraussichtliche Übergabe
an den Versanddienstleister") is no longer taken for the delivery date.
- Mail already stored is not re-read on its own; `app:backfill insights` reads it again.
- Mail already stored is not re-read on its own; `app:backfill insights` reads it again
(the last 24 hours unless `--since` says otherwise).
**A calendar file somebody mails you is offered, not added.** Two changes that belong together:
@@ -31,7 +32,8 @@ empty.
calendar** or **No thanks**. No reply is sent to anyone. Invitations addressed to you, bookings
and your own events are unchanged, and events of this kind already on your calendar stay there.
`app:backfill events` picks up mail that arrived before this.
`app:backfill events` picks up mail that arrived before this (the last 24 hours unless `--since`
says otherwise).
**An Amazon order is on the radar from the moment it is placed.** The parcel card appeared with the
dispatch mail; the order confirmation before it ("Bestellt: …", "Ordered: …") was not read at all,
+1 -1
View File
@@ -846,7 +846,7 @@ php bin/mirror-wiki.php --check
| `app:mail:wake-snoozed` | Return snoozed conversations whose time has come. Runs every minute |
| `app:mail:release-held` | Show mail that has been held for the assistant longer than **Wait at most** allows. Runs every minute and finds nothing on an ordinary day — the live worker releases held mail itself. It is the backstop for that worker being down, and it warns when it has to act |
| `app:calendar:materialise [--dry-run]` | Redraw the occurrences of recurring events whose horizon no longer reaches far enough. Runs nightly; without it a long-untouched series eventually runs out of dates |
| `app:backfill [task]` | Run a one-off backfill over stored data; with no argument it lists the tasks and asks. `events` re-runs calendar extraction, `proposals` re-reads mail for dates written in prose, `safe-html-table-links` restores links that were wrapped around a table and came out empty |
| `app:backfill [task]` | Run a one-off backfill over stored data; with no argument it lists the tasks and asks. The tasks that re-read mail (`events`, `insights`) cover the last 24 hours unless `--since` says otherwise (`7d`, `2026-09-01`, `all`). `events` re-runs calendar extraction, `proposals` re-reads mail for dates written in prose, `safe-html-table-links` restores links that were wrapped around a table and came out empty |
| `app:imap:idle <mailbox-id>` | Hold an IMAP IDLE connection for a single mailbox |
| `app:imap:supervise` | Spawn and watch one `app:imap:idle` process per IDLE-enabled mailbox |
| `app:imap:test [--account=ID]` | Test an IMAP connection and folder listing |
+12 -2
View File
@@ -1,4 +1,4 @@
<!-- translated-from: features/calendar-invitations.md sha1:4f37cd5044b46127ecf9f09948935b2a475a38cc -->
<!-- translated-from: features/calendar-invitations.md sha1:2831c23dd958dfe411f1eb0b87c79d2d9ccb8719 -->
# Einladungen und Termine aus E-Mails
@@ -260,6 +260,16 @@ docker compose exec php php bin/console app:backfill events
docker compose exec php php bin/console app:backfill proposals
```
`events` liest **die Mail der letzten 24 Stunden**, solange du nichts anderes sagst — der übliche
Anlass ist eine Mail von dieser Woche, nicht jede Kalenderdatei der letzten zehn Jahre. `--since`
weitet das aus: eine Zahl mit `h`, `d` oder `w`, ein Datum oder `all`.
```bash
docker compose exec php php bin/console app:backfill events --since=7d
docker compose exec php php bin/console app:backfill events --since=2026-09-01
docker compose exec php php bin/console app:backfill events --since=all
```
Ohne Argument aufgerufen, listet `app:backfill` die verfügbaren Aufgaben auf und fragt nach. Ein
erneuter Lauf ist ungefährlich: Was du verworfen hast, bleibt verworfen, und was du selbst bearbeitet
hast, wird nicht überschrieben.
@@ -274,7 +284,7 @@ oder **Vielleicht**, und sie erscheinen.
**Eine Einladung, die vor dieser Änderung ankam, bleibt im Kalender stehen, ganz gleich, was du
geantwortet hast.** Beim Upgrade wurde nichts umgeschrieben, und zwar mit Absicht: jemandem während
eines `docker compose up` den Kalender zu leeren, ist keine annehmbare Art, eine Funktion
auszuliefern. `app:backfill events` liest die Mail neu und bringt die alten in Ordnung.
auszuliefern. `app:backfill events --since=all` liest die Mail neu und bringt die alten in Ordnung.
**Deine Antwort kann gespeichert werden, ohne dass die Organisation davon erfährt.** Die beiden
Hälften sind mit Absicht unabhängig. Kann die Antwort nicht gesendet werden, sagt die Meldung das —
+5 -4
View File
@@ -1,4 +1,4 @@
<!-- translated-from: features/mail.md sha1:e4c2f01bebced62a2b1f4fcc67d4c72d5e8869f4 -->
<!-- translated-from: features/mail.md sha1:4d527f6e86ae3478d94e7a628b559a798f22d239 -->
# Mail
@@ -278,9 +278,10 @@ ausblendest. Die Menge der Quellen ist absichtlich erweiterbar — ein Extraktor
hinzukommt, trägt sich selbst auf dieser Einstellungsseite ein und läuft los, ohne dass sich sonst
etwas ändert.
Neue Extraktoren lesen alte Mails nicht von allein noch einmal. `app:backfill insights` geht die
Mails, die schon in der Datenbank liegen, einmal durch und reicht sie jedem eingeschalteten
Extraktor — derselbe Durchlauf, den auch die anderen Backfill-Aufgaben machen.
Neue Extraktoren lesen alte Mails nicht von allein noch einmal. `app:backfill insights` reicht
Mails, die schon in der Datenbank liegen, jedem eingeschalteten Extraktor. Gelesen werden die
letzten 24 Stunden, solange du nichts anderes sagst: `--since=7d`, `--since=2026-09-01` oder
`--since=all` gehen weiter zurück.
Einmalcodes sind die einzige Quelle, deren Zeitpunkt ein Ablauf ist und kein Termin. Die Karte
trägt den Moment, an dem der Code laut Mail nicht mehr gilt — "gültig für 10 Minuten", "gültig bis
+12 -2
View File
@@ -239,6 +239,16 @@ docker compose exec php php bin/console app:backfill proposals
Run with no argument, `app:backfill` lists the available tasks and asks. Re-running is safe:
anything you dismissed stays dismissed, and anything you edited yourself is not overwritten.
`events` reads **the last 24 hours of mail** unless you say otherwise — the usual reason to run it is
a mail from this week, not every calendar file of the last ten years. `--since` widens it: a count
with `h`, `d` or `w`, a date, or `all`.
```bash
docker compose exec php php bin/console app:backfill events --since=7d
docker compose exec php php bin/console app:backfill events --since=2026-09-01
docker compose exec php php bin/console app:backfill events --since=all
```
## Things that bite
**An invitation you have not answered is not in your calendar.** That is deliberate — see above —
@@ -247,8 +257,8 @@ is where they are; press **Yes** or **Maybe** and they appear.
**An invitation that arrived before this behaviour existed stays on the calendar whatever you
answered.** Nothing was rewritten on upgrade, on purpose: emptying somebody's calendar during a
`docker compose up` is not an acceptable way to ship a feature. `app:backfill events` re-reads the
mail and brings the old ones in line.
`docker compose up` is not an acceptable way to ship a feature. `app:backfill events --since=all`
re-reads the mail and brings the old ones in line.
**Your RSVP can be saved without the organiser hearing it.** The two halves are independent on
purpose. When the reply cannot be sent, the toast says so — and that is the only notice you get.
+3 -3
View File
@@ -252,9 +252,9 @@ from that source; what it already found stays until dismissed. The set of source
design — an extractor added to a build lists itself on that settings page and starts running,
without anything else changing.
New extractors do not re-read old mail on their own. `app:backfill insights` walks the mail already
in the database once and hands it to every enabled extractor, the same sweep the other backfill
tasks run.
New extractors do not re-read old mail on their own. `app:backfill insights` hands mail already in
the database to every enabled extractor. It reads the last 24 hours unless told otherwise:
`--since=7d`, `--since=2026-09-01` or `--since=all` go further back.
One-time codes are the one source with an expiry rather than an occurrence. A code's card carries
the moment the mail says the code stops working — "valid for 10 minutes", "gültig bis 09:30" — so
+96 -2
View File
@@ -5,10 +5,13 @@ declare(strict_types=1);
namespace App\Command\Backfill;
use App\Command\Backfill\BackfillTaskInterface;
use DateTimeImmutable;
use Symfony\Component\Clock\Clock;
use Symfony\Component\Console\Attribute\AsCommand;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Console\Style\SymfonyStyle;
use Symfony\Component\DependencyInjection\Attribute\AutowireIterator;
@@ -26,6 +29,10 @@ use Symfony\Component\DependencyInjection\Attribute\AutowireIterator;
* Run `app:backfill` with no argument to pick a task from a list; pass the task
* name to run it directly. Under --no-interaction (CI, cron) the argument is
* required, since there is no one to answer the prompt.
*
* --since applies to the tasks that re-read mail (WindowedBackfillTaskInterface)
* and defaults to the last 24 hours for them; see that interface for why. The
* table repairs take no window, and say so if they are given one.
*/
#[AsCommand(
name: 'app:backfill',
@@ -33,6 +40,12 @@ use Symfony\Component\DependencyInjection\Attribute\AutowireIterator;
)]
final class BackfillCommand extends Command
{
/** How far back a task that re-reads mail goes when nobody says. */
public const string DEFAULT_SINCE = '24h';
/** The --since value that lifts the limit. */
public const string SINCE_ALL = 'all';
/** @var array<string, BackfillTaskInterface> */
private array $tasks = [];
@@ -56,7 +69,18 @@ final class BackfillCommand extends Command
{
$this
->addArgument('task', InputArgument::OPTIONAL, 'The backfill task to run')
->setHelp('Run "app:backfill" with no argument to choose a task interactively.');
->addOption(
'since',
null,
InputOption::VALUE_REQUIRED,
'For tasks that re-read mail: how far back to go — 36h, 7d, 4w, a date (2026-09-01), or "all"',
self::DEFAULT_SINCE,
)
->setHelp(
'Run "app:backfill" with no argument to choose a task interactively.' . "\n\n"
. 'Tasks that re-read mail (events, insights) look at the last 24 hours unless --since says otherwise: '
. '--since=7d, --since=2026-09-01, --since=all.',
);
}
protected function execute(InputInterface $input, OutputInterface $output): int
@@ -87,11 +111,81 @@ final class BackfillCommand extends Command
$task = $this->tasks[$name];
if (false === $task instanceof WindowedBackfillTaskInterface) {
$io->title(sprintf('Backfill: %s', $task->getName()));
$io->text($task->getDescription());
$io->newLine();
// Said, not swallowed: somebody who typed --since=7d and watched a
// whole table being rewritten should have been told beforehand.
if (true === $input->hasParameterOption('--since')) {
$io->note('This task repairs stored data as a whole and takes no --since; it covers everything.');
}
return $task->run($io);
}
try {
$since = $this->since((string) $input->getOption('since'));
} catch (\InvalidArgumentException $e) {
$io->error($e->getMessage());
return Command::FAILURE;
}
$io->title(sprintf('Backfill: %s', $task->getName()));
$io->text($task->getDescription());
$io->text(null === $since
? 'Reading all stored mail.'
: sprintf(
'Reading mail received since %s. Pass --since=7d, a date, or --since=all to go further back.',
$since->format('Y-m-d H:i T'),
));
$io->newLine();
return $task->run($io);
return $task->runSince($io, $since);
}
/**
* The moment --since names, or null for "all".
*
* A small vocabulary on purpose — a count and h, d or w, or a date. What
* strtotime() would additionally accept ("last tuesday", "-1 fortnight")
* is also what it silently misreads, and this decides how much of a
* mailbox gets rewritten.
*
* @throws \InvalidArgumentException when the value is none of those
*/
private function since(string $value): ?DateTimeImmutable
{
$value = mb_strtolower(trim($value));
if (self::SINCE_ALL === $value) {
return null;
}
$now = Clock::get()->now();
if (1 === preg_match('~^(\d+)\s*([hdw])$~', $value, $m) && (int) $m[1] > 0) {
$unit = ['h' => 'hours', 'd' => 'days', 'w' => 'weeks'][$m[2]];
return $now->modify(sprintf('-%d %s', (int) $m[1], $unit));
}
if (1 === preg_match('~^\d{4}-\d{2}-\d{2}$~', $value)) {
$date = DateTimeImmutable::createFromFormat('!Y-m-d', $value, $now->getTimezone());
// createFromFormat rolls 2026-02-31 over into March rather than
// refusing it; the round trip is what catches that.
if (false !== $date && $date->format('Y-m-d') === $value) {
return $date;
}
}
throw new \InvalidArgumentException(sprintf(
'--since=%s is not understood. Use a count with h, d or w (36h, 7d, 4w), a date (2026-09-01), or "all".',
$value,
));
}
/**
@@ -8,6 +8,7 @@ use App\Entity\Mail\Message;
use App\Repository\Mail\MessageRepository;
use App\Service\Calendar\EventReconciler;
use App\Service\Calendar\Extraction\EventExtractionRunner;
use DateTimeImmutable;
use Doctrine\ORM\EntityManagerInterface;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Style\SymfonyStyle;
@@ -30,7 +31,7 @@ use Symfony\Component\Console\Style\SymfonyStyle;
* mostly newsletters, and parsing all of it to find the few per cent that are
* bookings is work nobody gets back.
*/
final readonly class EventExtractionBackfillTask implements BackfillTaskInterface
final readonly class EventExtractionBackfillTask implements WindowedBackfillTaskInterface
{
private const int BATCH_SIZE = 200;
@@ -54,7 +55,12 @@ final readonly class EventExtractionBackfillTask implements BackfillTaskInterfac
public function run(SymfonyStyle $io): int
{
$total = $this->messages->countExtractionCandidates();
return $this->runSince($io, null);
}
public function runSince(SymfonyStyle $io, ?DateTimeImmutable $since): int
{
$total = $this->messages->countExtractionCandidates($since);
if (0 === $total) {
$io->success('No messages carry anything to extract.');
@@ -72,7 +78,7 @@ final readonly class EventExtractionBackfillTask implements BackfillTaskInterfac
while (true) {
/** @var list<Message> $batch */
$batch = $this->messages->extractionCandidates($lastId, self::BATCH_SIZE);
$batch = $this->messages->extractionCandidates($lastId, self::BATCH_SIZE, $since);
if (0 === count($batch)) {
break;
+11 -5
View File
@@ -7,6 +7,7 @@ namespace App\Command\Backfill;
use App\Entity\Mail\Message;
use App\Repository\Mail\MessageRepository;
use App\Service\Insight\InsightHarvester;
use DateTimeImmutable;
use Doctrine\ORM\EntityManagerInterface;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Style\SymfonyStyle;
@@ -31,7 +32,7 @@ use Symfony\Component\Console\Style\SymfonyStyle;
* writes into mail_insight and never touches `message`, so the set it pages
* over cannot shift under it.
*/
final readonly class InsightsBackfillTask implements BackfillTaskInterface
final readonly class InsightsBackfillTask implements WindowedBackfillTaskInterface
{
private const int BATCH_SIZE = 200;
@@ -54,7 +55,12 @@ final readonly class InsightsBackfillTask implements BackfillTaskInterface
public function run(SymfonyStyle $io): int
{
$total = $this->messages->count([]);
return $this->runSince($io, null);
}
public function runSince(SymfonyStyle $io, ?DateTimeImmutable $since): int
{
$total = $this->messages->countReceivedSince($since);
if (0 === $total) {
$io->success('No mail to read.');
@@ -64,19 +70,19 @@ final readonly class InsightsBackfillTask implements BackfillTaskInterface
$io->progressStart($total);
$offset = 0;
$lastId = 0;
$written = 0;
while (true) {
/** @var list<Message> $batch */
$batch = $this->messages->findBy([], ['id' => 'ASC'], self::BATCH_SIZE, $offset);
$batch = $this->messages->receivedSince($since, $lastId, self::BATCH_SIZE);
if (0 === count($batch)) {
break;
}
foreach ($batch as $message) {
$offset++;
$lastId = (int) $message->id;
// The harvester shields per extractor; a failed flush is the
// one fault left, and the manager-closed guard below is what
@@ -0,0 +1,34 @@
<?php
declare(strict_types=1);
namespace App\Command\Backfill;
use DateTimeImmutable;
use Symfony\Component\Console\Style\SymfonyStyle;
/**
* A backfill that reads MAIL, and can therefore be told how far back to read.
*
* Most tasks repair a table: a column that was computed wrongly is wrong in
* every row, and half a repair is a table in two states. Those implement
* BackfillTaskInterface and always walk everything.
*
* A task that re-reads mail for what an extractor can now find in it is a
* different kind of work. Its usual occasion is "a reader was fixed, and the
* mail I am thinking of arrived this week" — and run over a whole mailbox it
* answers that by also digging up every calendar file and parcel of the last
* ten years. So these take a starting point, and BackfillCommand gives them
* the last 24 hours unless it is told otherwise.
*
* A separate interface rather than a parameter on run(): a table repair that
* accepted a window and ignored it would be a command that looks as though it
* did what it was asked.
*/
interface WindowedBackfillTaskInterface extends BackfillTaskInterface
{
/**
* @param DateTimeImmutable|null $since only mail received at or after this; null for all of it
*/
public function runSince(SymfonyStyle $io, ?DateTimeImmutable $since): int;
}
+63 -9
View File
@@ -62,25 +62,31 @@ class MessageRepository extends ServiceEntityRepository
)
SQL;
/** Raw DBAL for the reasons EXTRACTION_CANDIDATE_WHERE gives above. */
public function countExtractionCandidates(): int
/**
* Raw DBAL for the reasons EXTRACTION_CANDIDATE_WHERE gives above.
*
* @param DateTimeImmutable|null $since only mail received at or after this; null for all of it
*/
public function countExtractionCandidates(?DateTimeImmutable $since = null): int
{
return (int) $this->getEntityManager()->getConnection()->fetchOne(
'SELECT COUNT(*) FROM message m WHERE ' . self::EXTRACTION_CANDIDATE_WHERE,
$this->candidateParameters(0),
'SELECT COUNT(*) FROM message m WHERE ' . self::EXTRACTION_CANDIDATE_WHERE . $this->candidateSince($since),
$this->candidateParameters(0, $since),
$this->candidateTypes(),
);
}
/**
* @param DateTimeImmutable|null $since only mail received at or after this; null for all of it
*
* @return list<Message>
*/
public function extractionCandidates(int $afterId, int $limit): array
public function extractionCandidates(int $afterId, int $limit, ?DateTimeImmutable $since = null): array
{
$ids = $this->getEntityManager()->getConnection()->fetchFirstColumn(
'SELECT m.id FROM message m WHERE ' . self::EXTRACTION_CANDIDATE_WHERE
'SELECT m.id FROM message m WHERE ' . self::EXTRACTION_CANDIDATE_WHERE . $this->candidateSince($since)
. ' ORDER BY m.id ASC LIMIT ' . max(1, $limit),
$this->candidateParameters($afterId),
$this->candidateParameters($afterId, $since),
$this->candidateTypes(),
);
@@ -94,9 +100,9 @@ class MessageRepository extends ServiceEntityRepository
/**
* @return array<string, mixed>
*/
private function candidateParameters(int $afterId): array
private function candidateParameters(int $afterId, ?DateTimeImmutable $since = null): array
{
return [
return (null === $since ? [] : ['extSince' => $since]) + [
'extAfterId' => $afterId,
'extCalendarTypes' => CalendarAttachment::CONTENT_TYPES,
'extCalendarName' => '%' . CalendarAttachment::EXTENSION,
@@ -105,6 +111,53 @@ class MessageRepository extends ServiceEntityRepository
];
}
/**
* The clause that narrows candidates to recent mail, or nothing.
*
* Appended rather than written into the constant as "IS NULL OR": Postgres
* cannot infer the type of a parameter that is only ever compared to NULL,
* and a clause that is absent needs no parameter at all.
*/
private function candidateSince(?DateTimeImmutable $since): string
{
return null === $since ? '' : ' AND m.received_at >= :extSince';
}
/**
* Mail in id order, a batch at a time, optionally only what arrived since
* a given moment — the walk a backfill that re-reads mail makes.
*
* Keyset on the id rather than OFFSET, so the cost of a batch does not
* grow with how far the walk has got.
*
* @return list<Message>
*/
public function receivedSince(?DateTimeImmutable $since, int $afterId, int $limit): array
{
$qb = $this->createQueryBuilder('m')
->where('m.id > :afterId')
->setParameter('afterId', $afterId)
->orderBy('m.id', 'ASC')
->setMaxResults(max(1, $limit));
if (null !== $since) {
$qb->andWhere('m.receivedAt >= :since')->setParameter('since', $since, Types::DATETIME_IMMUTABLE);
}
return $qb->getQuery()->getResult();
}
public function countReceivedSince(?DateTimeImmutable $since): int
{
$qb = $this->createQueryBuilder('m')->select('COUNT(m.id)');
if (null !== $since) {
$qb->where('m.receivedAt >= :since')->setParameter('since', $since, Types::DATETIME_IMMUTABLE);
}
return (int) $qb->getQuery()->getSingleScalarResult();
}
/**
* @return array<string, mixed>
*/
@@ -114,6 +167,7 @@ class MessageRepository extends ServiceEntityRepository
'extAfterId' => ParameterType::INTEGER,
'extCalendarTypes' => ArrayParameterType::STRING,
'extCalendarName' => ParameterType::STRING,
'extSince' => Types::DATETIME_IMMUTABLE,
'extMeetingHeader' => ParameterType::STRING,
'extJsonLd' => ParameterType::STRING,
];
@@ -6,7 +6,12 @@ namespace App\Tests\Command\Backfill;
use App\Command\Backfill\BackfillCommand;
use App\Command\Backfill\BackfillTaskInterface;
use App\Command\Backfill\WindowedBackfillTaskInterface;
use PHPUnit\Framework\Attributes\DataProvider;
use PHPUnit\Framework\TestCase;
use Symfony\Component\Clock\Clock;
use Symfony\Component\Clock\MockClock;
use Symfony\Component\Clock\NativeClock;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Style\SymfonyStyle;
use Symfony\Component\Console\Tester\CommandTester;
@@ -26,6 +31,12 @@ use Symfony\Component\Console\Tester\CommandTester;
*/
final class BackfillCommandTest extends TestCase
{
protected function tearDown(): void
{
// --since is read against the global clock, which two tests pin.
Clock::set(new NativeClock());
}
public function testItRunsTheTaskNamedOnTheCommandLine(): void
{
$wanted = $this->task('wanted');
@@ -134,6 +145,134 @@ final class BackfillCommandTest extends TestCase
return new CommandTester(new BackfillCommand($tasks));
}
// ── --since ───────────────────────────────────────────────────────────
/**
* The default is the point of the option: a task that re-reads mail and is
* told nothing reads a day, not a mailbox.
*/
public function testATaskThatReadsMailGetsTheLastDayUnlessToldOtherwise(): void
{
Clock::set(new MockClock('2026-10-08 16:00:00 UTC'));
$events = $this->windowedTask('events');
$tester = $this->tester($events);
self::assertSame(Command::SUCCESS, $tester->execute(['task' => 'events']));
self::assertSame(['2026-10-07 16:00'], $events->windows);
self::assertStringContainsString('2026-10-07 16:00', $tester->getDisplay());
self::assertStringContainsString('--since=all', $tester->getDisplay(), 'How to widen it is said up front.');
}
/**
* @return iterable<string, array{0: string, 1: ?string}>
*/
public static function windows(): iterable
{
yield 'hours' => ['36h', '2026-10-07 04:00'];
yield 'days' => ['7d', '2026-10-01 16:00'];
yield 'weeks' => ['2w', '2026-09-24 16:00'];
yield 'a date, from its start' => ['2026-09-01', '2026-09-01 00:00'];
yield 'everything' => ['all', null];
yield 'case and spaces' => [' ALL ', null];
}
#[DataProvider('windows')]
public function testSinceIsReadAsAnAmountADateOrEverything(string $since, ?string $expected): void
{
Clock::set(new MockClock('2026-10-08 16:00:00 UTC'));
$events = $this->windowedTask('events');
self::assertSame(Command::SUCCESS, $this->tester($events)->execute(['task' => 'events', '--since' => $since]));
self::assertSame([$expected], $events->windows);
}
/**
* @return iterable<string, array{0: string}>
*/
public static function nonsense(): iterable
{
yield 'words' => ['yesterday'];
yield 'no unit' => ['7'];
yield 'zero' => ['0d'];
yield 'a unit not offered' => ['3m'];
yield 'a day that does not exist' => ['2026-02-31'];
yield 'empty' => [''];
}
/**
* Refused, and nothing run. Guessing here would decide how much of a
* mailbox is re-read on the strength of a typo.
*/
#[DataProvider('nonsense')]
public function testAWindowItCannotReadRunsNothing(string $since): void
{
$events = $this->windowedTask('events');
$tester = $this->tester($events);
self::assertSame(Command::FAILURE, $tester->execute(['task' => 'events', '--since' => $since]));
self::assertSame([], $events->windows);
self::assertStringContainsString('not understood', $tester->getDisplay());
}
/**
* A table repair has no window to honour. It still runs — over everything,
* as it always did — and says that it ignored the option.
*/
public function testATableRepairSaysItTakesNoWindow(): void
{
$repair = $this->task('recipients');
$tester = $this->tester($repair);
self::assertSame(Command::SUCCESS, $tester->execute(['task' => 'recipients', '--since' => '7d']));
self::assertSame(1, $repair->runs);
self::assertStringContainsString('takes no --since', $tester->getDisplay());
}
public function testATableRepairGivenNoWindowSaysNothingAboutOne(): void
{
$tester = $this->tester($this->task('recipients'));
$tester->execute(['task' => 'recipients']);
self::assertStringNotContainsString('--since', $tester->getDisplay());
}
/**
* @return WindowedBackfillTaskInterface&object{windows: list<?string>}
*/
private function windowedTask(string $name): object
{
return new class ($name) implements WindowedBackfillTaskInterface {
/** @var list<?string> what each run was told to read from, in UTC; null for everything */
public array $windows = [];
public function __construct(private readonly string $name) {}
public function getName(): string
{
return $this->name;
}
public function getDescription(): string
{
return sprintf('Description of %s', $this->name);
}
public function run(SymfonyStyle $io): int
{
throw new \LogicException('A windowed task is run through runSince().');
}
public function runSince(SymfonyStyle $io, ?\DateTimeImmutable $since): int
{
$this->windows[] = $since?->setTimezone(new \DateTimeZone('UTC'))->format('Y-m-d H:i');
return Command::SUCCESS;
}
};
}
/**
* @return BackfillTaskInterface&object{runs: int}
*/
@@ -578,6 +578,32 @@ final class IcsExtractionTest extends KernelTestCase
self::assertContains($message->id, array_map(static fn (Message $m): ?int => $m->id, $candidates));
}
/**
* `app:backfill events --since=…`: mail from before the window is not a
* candidate, however much calendar data it carries.
*/
public function testTheCandidateQueryCanBeNarrowedToRecentMail(): void
{
$old = $this->messageWithInvite('BEGIN:VCALENDAR');
$old->receivedAt = new DateTimeImmutable('-3 days');
$new = $this->messageWithInvite('BEGIN:VCALENDAR');
$this->em->flush();
$messages = $this->em->getRepository(Message::class);
$after = (int) $old->id - 1;
$ids = static fn (array $found): array => array_map(static fn (Message $m): ?int => $m->id, $found);
$recent = $ids($messages->extractionCandidates($after, 10, new DateTimeImmutable('-24 hours')));
self::assertContains($new->id, $recent);
self::assertNotContains($old->id, $recent);
self::assertContains($old->id, $ids($messages->extractionCandidates($after, 10)), 'no window, no limit');
self::assertLessThan(
$messages->countExtractionCandidates(),
$messages->countExtractionCandidates(new DateTimeImmutable('-24 hours')),
);
}
// ── Fixtures ──────────────────────────────────────────────────────────
/**