mirror of
https://github.com/karatektus/pl_mail.git
synced 2026-10-25 10:04:18 +00:00
Let a backfill that re-reads mail be told how far back to read
`app:backfill events` and `app:backfill insights` re-read stored mail for what an extractor can now find in it. The usual occasion is a reader that was just fixed and a mail from this week; run over a whole mailbox they also dig up every calendar file and parcel since it began — and since calendar files that name nobody here are now offered rather than added, that is a card on every one of them. Both take --since and default to the last 24 hours: a count with h, d or w, a date, or "all". Anything else is refused and nothing runs. The command prints the window before it starts. A separate interface, WindowedBackfillTaskInterface, rather than a parameter on run(): the other tasks repair a table as a whole, and one that accepted a window and ignored it would look as though it had done what it was asked. Given --since, they say they take none. The insights walk is keyset on the id now instead of OFFSET. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
1fc6adb146
commit
4ca055a724
13 files changed
+415
-33
No files matched your search
+4
-2
@@ -17,7 +17,8 @@ empty.
|
||||
|
||||
- A date announced as the day the shop hands the parcel to the carrier ("Voraussichtliche Übergabe
|
||||
an den Versanddienstleister") is no longer taken for the delivery date.
|
||||
- Mail already stored is not re-read on its own; `app:backfill insights` reads it again.
|
||||
- Mail already stored is not re-read on its own; `app:backfill insights` reads it again
|
||||
(the last 24 hours unless `--since` says otherwise).
|
||||
|
||||
**A calendar file somebody mails you is offered, not added.** Two changes that belong together:
|
||||
|
||||
@@ -31,7 +32,8 @@ empty.
|
||||
calendar** or **No thanks**. No reply is sent to anyone. Invitations addressed to you, bookings
|
||||
and your own events are unchanged, and events of this kind already on your calendar stay there.
|
||||
|
||||
`app:backfill events` picks up mail that arrived before this.
|
||||
`app:backfill events` picks up mail that arrived before this (the last 24 hours unless `--since`
|
||||
says otherwise).
|
||||
|
||||
**An Amazon order is on the radar from the moment it is placed.** The parcel card appeared with the
|
||||
dispatch mail; the order confirmation before it ("Bestellt: …", "Ordered: …") was not read at all,
|
||||
|
||||
+1
-1
@@ -846,7 +846,7 @@ php bin/mirror-wiki.php --check
|
||||
| `app:mail:wake-snoozed` | Return snoozed conversations whose time has come. Runs every minute |
|
||||
| `app:mail:release-held` | Show mail that has been held for the assistant longer than **Wait at most** allows. Runs every minute and finds nothing on an ordinary day — the live worker releases held mail itself. It is the backstop for that worker being down, and it warns when it has to act |
|
||||
| `app:calendar:materialise [--dry-run]` | Redraw the occurrences of recurring events whose horizon no longer reaches far enough. Runs nightly; without it a long-untouched series eventually runs out of dates |
|
||||
| `app:backfill [task]` | Run a one-off backfill over stored data; with no argument it lists the tasks and asks. `events` re-runs calendar extraction, `proposals` re-reads mail for dates written in prose, `safe-html-table-links` restores links that were wrapped around a table and came out empty |
|
||||
| `app:backfill [task]` | Run a one-off backfill over stored data; with no argument it lists the tasks and asks. The tasks that re-read mail (`events`, `insights`) cover the last 24 hours unless `--since` says otherwise (`7d`, `2026-09-01`, `all`). `events` re-runs calendar extraction, `proposals` re-reads mail for dates written in prose, `safe-html-table-links` restores links that were wrapped around a table and came out empty |
|
||||
| `app:imap:idle <mailbox-id>` | Hold an IMAP IDLE connection for a single mailbox |
|
||||
| `app:imap:supervise` | Spawn and watch one `app:imap:idle` process per IDLE-enabled mailbox |
|
||||
| `app:imap:test [--account=ID]` | Test an IMAP connection and folder listing |
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
<!-- translated-from: features/calendar-invitations.md sha1:4f37cd5044b46127ecf9f09948935b2a475a38cc -->
|
||||
<!-- translated-from: features/calendar-invitations.md sha1:2831c23dd958dfe411f1eb0b87c79d2d9ccb8719 -->
|
||||
|
||||
# Einladungen und Termine aus E-Mails
|
||||
|
||||
@@ -260,6 +260,16 @@ docker compose exec php php bin/console app:backfill events
|
||||
docker compose exec php php bin/console app:backfill proposals
|
||||
```
|
||||
|
||||
`events` liest **die Mail der letzten 24 Stunden**, solange du nichts anderes sagst — der übliche
|
||||
Anlass ist eine Mail von dieser Woche, nicht jede Kalenderdatei der letzten zehn Jahre. `--since`
|
||||
weitet das aus: eine Zahl mit `h`, `d` oder `w`, ein Datum oder `all`.
|
||||
|
||||
```bash
|
||||
docker compose exec php php bin/console app:backfill events --since=7d
|
||||
docker compose exec php php bin/console app:backfill events --since=2026-09-01
|
||||
docker compose exec php php bin/console app:backfill events --since=all
|
||||
```
|
||||
|
||||
Ohne Argument aufgerufen, listet `app:backfill` die verfügbaren Aufgaben auf und fragt nach. Ein
|
||||
erneuter Lauf ist ungefährlich: Was du verworfen hast, bleibt verworfen, und was du selbst bearbeitet
|
||||
hast, wird nicht überschrieben.
|
||||
@@ -274,7 +284,7 @@ oder **Vielleicht**, und sie erscheinen.
|
||||
**Eine Einladung, die vor dieser Änderung ankam, bleibt im Kalender stehen, ganz gleich, was du
|
||||
geantwortet hast.** Beim Upgrade wurde nichts umgeschrieben, und zwar mit Absicht: jemandem während
|
||||
eines `docker compose up` den Kalender zu leeren, ist keine annehmbare Art, eine Funktion
|
||||
auszuliefern. `app:backfill events` liest die Mail neu und bringt die alten in Ordnung.
|
||||
auszuliefern. `app:backfill events --since=all` liest die Mail neu und bringt die alten in Ordnung.
|
||||
|
||||
**Deine Antwort kann gespeichert werden, ohne dass die Organisation davon erfährt.** Die beiden
|
||||
Hälften sind mit Absicht unabhängig. Kann die Antwort nicht gesendet werden, sagt die Meldung das —
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
<!-- translated-from: features/mail.md sha1:e4c2f01bebced62a2b1f4fcc67d4c72d5e8869f4 -->
|
||||
<!-- translated-from: features/mail.md sha1:4d527f6e86ae3478d94e7a628b559a798f22d239 -->
|
||||
|
||||
# Mail
|
||||
|
||||
@@ -278,9 +278,10 @@ ausblendest. Die Menge der Quellen ist absichtlich erweiterbar — ein Extraktor
|
||||
hinzukommt, trägt sich selbst auf dieser Einstellungsseite ein und läuft los, ohne dass sich sonst
|
||||
etwas ändert.
|
||||
|
||||
Neue Extraktoren lesen alte Mails nicht von allein noch einmal. `app:backfill insights` geht die
|
||||
Mails, die schon in der Datenbank liegen, einmal durch und reicht sie jedem eingeschalteten
|
||||
Extraktor — derselbe Durchlauf, den auch die anderen Backfill-Aufgaben machen.
|
||||
Neue Extraktoren lesen alte Mails nicht von allein noch einmal. `app:backfill insights` reicht
|
||||
Mails, die schon in der Datenbank liegen, jedem eingeschalteten Extraktor. Gelesen werden die
|
||||
letzten 24 Stunden, solange du nichts anderes sagst: `--since=7d`, `--since=2026-09-01` oder
|
||||
`--since=all` gehen weiter zurück.
|
||||
|
||||
Einmalcodes sind die einzige Quelle, deren Zeitpunkt ein Ablauf ist und kein Termin. Die Karte
|
||||
trägt den Moment, an dem der Code laut Mail nicht mehr gilt — "gültig für 10 Minuten", "gültig bis
|
||||
|
||||
@@ -239,6 +239,16 @@ docker compose exec php php bin/console app:backfill proposals
|
||||
Run with no argument, `app:backfill` lists the available tasks and asks. Re-running is safe:
|
||||
anything you dismissed stays dismissed, and anything you edited yourself is not overwritten.
|
||||
|
||||
`events` reads **the last 24 hours of mail** unless you say otherwise — the usual reason to run it is
|
||||
a mail from this week, not every calendar file of the last ten years. `--since` widens it: a count
|
||||
with `h`, `d` or `w`, a date, or `all`.
|
||||
|
||||
```bash
|
||||
docker compose exec php php bin/console app:backfill events --since=7d
|
||||
docker compose exec php php bin/console app:backfill events --since=2026-09-01
|
||||
docker compose exec php php bin/console app:backfill events --since=all
|
||||
```
|
||||
|
||||
## Things that bite
|
||||
|
||||
**An invitation you have not answered is not in your calendar.** That is deliberate — see above —
|
||||
@@ -247,8 +257,8 @@ is where they are; press **Yes** or **Maybe** and they appear.
|
||||
|
||||
**An invitation that arrived before this behaviour existed stays on the calendar whatever you
|
||||
answered.** Nothing was rewritten on upgrade, on purpose: emptying somebody's calendar during a
|
||||
`docker compose up` is not an acceptable way to ship a feature. `app:backfill events` re-reads the
|
||||
mail and brings the old ones in line.
|
||||
`docker compose up` is not an acceptable way to ship a feature. `app:backfill events --since=all`
|
||||
re-reads the mail and brings the old ones in line.
|
||||
|
||||
**Your RSVP can be saved without the organiser hearing it.** The two halves are independent on
|
||||
purpose. When the reply cannot be sent, the toast says so — and that is the only notice you get.
|
||||
|
||||
@@ -252,9 +252,9 @@ from that source; what it already found stays until dismissed. The set of source
|
||||
design — an extractor added to a build lists itself on that settings page and starts running,
|
||||
without anything else changing.
|
||||
|
||||
New extractors do not re-read old mail on their own. `app:backfill insights` walks the mail already
|
||||
in the database once and hands it to every enabled extractor, the same sweep the other backfill
|
||||
tasks run.
|
||||
New extractors do not re-read old mail on their own. `app:backfill insights` hands mail already in
|
||||
the database to every enabled extractor. It reads the last 24 hours unless told otherwise:
|
||||
`--since=7d`, `--since=2026-09-01` or `--since=all` go further back.
|
||||
|
||||
One-time codes are the one source with an expiry rather than an occurrence. A code's card carries
|
||||
the moment the mail says the code stops working — "valid for 10 minutes", "gültig bis 09:30" — so
|
||||
|
||||
@@ -5,10 +5,13 @@ declare(strict_types=1);
|
||||
namespace App\Command\Backfill;
|
||||
|
||||
use App\Command\Backfill\BackfillTaskInterface;
|
||||
use DateTimeImmutable;
|
||||
use Symfony\Component\Clock\Clock;
|
||||
use Symfony\Component\Console\Attribute\AsCommand;
|
||||
use Symfony\Component\Console\Command\Command;
|
||||
use Symfony\Component\Console\Input\InputArgument;
|
||||
use Symfony\Component\Console\Input\InputInterface;
|
||||
use Symfony\Component\Console\Input\InputOption;
|
||||
use Symfony\Component\Console\Output\OutputInterface;
|
||||
use Symfony\Component\Console\Style\SymfonyStyle;
|
||||
use Symfony\Component\DependencyInjection\Attribute\AutowireIterator;
|
||||
@@ -26,6 +29,10 @@ use Symfony\Component\DependencyInjection\Attribute\AutowireIterator;
|
||||
* Run `app:backfill` with no argument to pick a task from a list; pass the task
|
||||
* name to run it directly. Under --no-interaction (CI, cron) the argument is
|
||||
* required, since there is no one to answer the prompt.
|
||||
*
|
||||
* --since applies to the tasks that re-read mail (WindowedBackfillTaskInterface)
|
||||
* and defaults to the last 24 hours for them; see that interface for why. The
|
||||
* table repairs take no window, and say so if they are given one.
|
||||
*/
|
||||
#[AsCommand(
|
||||
name: 'app:backfill',
|
||||
@@ -33,6 +40,12 @@ use Symfony\Component\DependencyInjection\Attribute\AutowireIterator;
|
||||
)]
|
||||
final class BackfillCommand extends Command
|
||||
{
|
||||
/** How far back a task that re-reads mail goes when nobody says. */
|
||||
public const string DEFAULT_SINCE = '24h';
|
||||
|
||||
/** The --since value that lifts the limit. */
|
||||
public const string SINCE_ALL = 'all';
|
||||
|
||||
/** @var array<string, BackfillTaskInterface> */
|
||||
private array $tasks = [];
|
||||
|
||||
@@ -56,7 +69,18 @@ final class BackfillCommand extends Command
|
||||
{
|
||||
$this
|
||||
->addArgument('task', InputArgument::OPTIONAL, 'The backfill task to run')
|
||||
->setHelp('Run "app:backfill" with no argument to choose a task interactively.');
|
||||
->addOption(
|
||||
'since',
|
||||
null,
|
||||
InputOption::VALUE_REQUIRED,
|
||||
'For tasks that re-read mail: how far back to go — 36h, 7d, 4w, a date (2026-09-01), or "all"',
|
||||
self::DEFAULT_SINCE,
|
||||
)
|
||||
->setHelp(
|
||||
'Run "app:backfill" with no argument to choose a task interactively.' . "\n\n"
|
||||
. 'Tasks that re-read mail (events, insights) look at the last 24 hours unless --since says otherwise: '
|
||||
. '--since=7d, --since=2026-09-01, --since=all.',
|
||||
);
|
||||
}
|
||||
|
||||
protected function execute(InputInterface $input, OutputInterface $output): int
|
||||
@@ -87,11 +111,81 @@ final class BackfillCommand extends Command
|
||||
|
||||
$task = $this->tasks[$name];
|
||||
|
||||
if (false === $task instanceof WindowedBackfillTaskInterface) {
|
||||
$io->title(sprintf('Backfill: %s', $task->getName()));
|
||||
$io->text($task->getDescription());
|
||||
$io->newLine();
|
||||
|
||||
// Said, not swallowed: somebody who typed --since=7d and watched a
|
||||
// whole table being rewritten should have been told beforehand.
|
||||
if (true === $input->hasParameterOption('--since')) {
|
||||
$io->note('This task repairs stored data as a whole and takes no --since; it covers everything.');
|
||||
}
|
||||
|
||||
return $task->run($io);
|
||||
}
|
||||
|
||||
try {
|
||||
$since = $this->since((string) $input->getOption('since'));
|
||||
} catch (\InvalidArgumentException $e) {
|
||||
$io->error($e->getMessage());
|
||||
|
||||
return Command::FAILURE;
|
||||
}
|
||||
|
||||
$io->title(sprintf('Backfill: %s', $task->getName()));
|
||||
$io->text($task->getDescription());
|
||||
$io->text(null === $since
|
||||
? 'Reading all stored mail.'
|
||||
: sprintf(
|
||||
'Reading mail received since %s. Pass --since=7d, a date, or --since=all to go further back.',
|
||||
$since->format('Y-m-d H:i T'),
|
||||
));
|
||||
$io->newLine();
|
||||
|
||||
return $task->run($io);
|
||||
return $task->runSince($io, $since);
|
||||
}
|
||||
|
||||
/**
|
||||
* The moment --since names, or null for "all".
|
||||
*
|
||||
* A small vocabulary on purpose — a count and h, d or w, or a date. What
|
||||
* strtotime() would additionally accept ("last tuesday", "-1 fortnight")
|
||||
* is also what it silently misreads, and this decides how much of a
|
||||
* mailbox gets rewritten.
|
||||
*
|
||||
* @throws \InvalidArgumentException when the value is none of those
|
||||
*/
|
||||
private function since(string $value): ?DateTimeImmutable
|
||||
{
|
||||
$value = mb_strtolower(trim($value));
|
||||
|
||||
if (self::SINCE_ALL === $value) {
|
||||
return null;
|
||||
}
|
||||
|
||||
$now = Clock::get()->now();
|
||||
|
||||
if (1 === preg_match('~^(\d+)\s*([hdw])$~', $value, $m) && (int) $m[1] > 0) {
|
||||
$unit = ['h' => 'hours', 'd' => 'days', 'w' => 'weeks'][$m[2]];
|
||||
|
||||
return $now->modify(sprintf('-%d %s', (int) $m[1], $unit));
|
||||
}
|
||||
|
||||
if (1 === preg_match('~^\d{4}-\d{2}-\d{2}$~', $value)) {
|
||||
$date = DateTimeImmutable::createFromFormat('!Y-m-d', $value, $now->getTimezone());
|
||||
|
||||
// createFromFormat rolls 2026-02-31 over into March rather than
|
||||
// refusing it; the round trip is what catches that.
|
||||
if (false !== $date && $date->format('Y-m-d') === $value) {
|
||||
return $date;
|
||||
}
|
||||
}
|
||||
|
||||
throw new \InvalidArgumentException(sprintf(
|
||||
'--since=%s is not understood. Use a count with h, d or w (36h, 7d, 4w), a date (2026-09-01), or "all".',
|
||||
$value,
|
||||
));
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -8,6 +8,7 @@ use App\Entity\Mail\Message;
|
||||
use App\Repository\Mail\MessageRepository;
|
||||
use App\Service\Calendar\EventReconciler;
|
||||
use App\Service\Calendar\Extraction\EventExtractionRunner;
|
||||
use DateTimeImmutable;
|
||||
use Doctrine\ORM\EntityManagerInterface;
|
||||
use Symfony\Component\Console\Command\Command;
|
||||
use Symfony\Component\Console\Style\SymfonyStyle;
|
||||
@@ -30,7 +31,7 @@ use Symfony\Component\Console\Style\SymfonyStyle;
|
||||
* mostly newsletters, and parsing all of it to find the few per cent that are
|
||||
* bookings is work nobody gets back.
|
||||
*/
|
||||
final readonly class EventExtractionBackfillTask implements BackfillTaskInterface
|
||||
final readonly class EventExtractionBackfillTask implements WindowedBackfillTaskInterface
|
||||
{
|
||||
private const int BATCH_SIZE = 200;
|
||||
|
||||
@@ -54,7 +55,12 @@ final readonly class EventExtractionBackfillTask implements BackfillTaskInterfac
|
||||
|
||||
public function run(SymfonyStyle $io): int
|
||||
{
|
||||
$total = $this->messages->countExtractionCandidates();
|
||||
return $this->runSince($io, null);
|
||||
}
|
||||
|
||||
public function runSince(SymfonyStyle $io, ?DateTimeImmutable $since): int
|
||||
{
|
||||
$total = $this->messages->countExtractionCandidates($since);
|
||||
|
||||
if (0 === $total) {
|
||||
$io->success('No messages carry anything to extract.');
|
||||
@@ -72,7 +78,7 @@ final readonly class EventExtractionBackfillTask implements BackfillTaskInterfac
|
||||
|
||||
while (true) {
|
||||
/** @var list<Message> $batch */
|
||||
$batch = $this->messages->extractionCandidates($lastId, self::BATCH_SIZE);
|
||||
$batch = $this->messages->extractionCandidates($lastId, self::BATCH_SIZE, $since);
|
||||
|
||||
if (0 === count($batch)) {
|
||||
break;
|
||||
|
||||
@@ -7,6 +7,7 @@ namespace App\Command\Backfill;
|
||||
use App\Entity\Mail\Message;
|
||||
use App\Repository\Mail\MessageRepository;
|
||||
use App\Service\Insight\InsightHarvester;
|
||||
use DateTimeImmutable;
|
||||
use Doctrine\ORM\EntityManagerInterface;
|
||||
use Symfony\Component\Console\Command\Command;
|
||||
use Symfony\Component\Console\Style\SymfonyStyle;
|
||||
@@ -31,7 +32,7 @@ use Symfony\Component\Console\Style\SymfonyStyle;
|
||||
* writes into mail_insight and never touches `message`, so the set it pages
|
||||
* over cannot shift under it.
|
||||
*/
|
||||
final readonly class InsightsBackfillTask implements BackfillTaskInterface
|
||||
final readonly class InsightsBackfillTask implements WindowedBackfillTaskInterface
|
||||
{
|
||||
private const int BATCH_SIZE = 200;
|
||||
|
||||
@@ -54,7 +55,12 @@ final readonly class InsightsBackfillTask implements BackfillTaskInterface
|
||||
|
||||
public function run(SymfonyStyle $io): int
|
||||
{
|
||||
$total = $this->messages->count([]);
|
||||
return $this->runSince($io, null);
|
||||
}
|
||||
|
||||
public function runSince(SymfonyStyle $io, ?DateTimeImmutable $since): int
|
||||
{
|
||||
$total = $this->messages->countReceivedSince($since);
|
||||
|
||||
if (0 === $total) {
|
||||
$io->success('No mail to read.');
|
||||
@@ -64,19 +70,19 @@ final readonly class InsightsBackfillTask implements BackfillTaskInterface
|
||||
|
||||
$io->progressStart($total);
|
||||
|
||||
$offset = 0;
|
||||
$lastId = 0;
|
||||
$written = 0;
|
||||
|
||||
while (true) {
|
||||
/** @var list<Message> $batch */
|
||||
$batch = $this->messages->findBy([], ['id' => 'ASC'], self::BATCH_SIZE, $offset);
|
||||
$batch = $this->messages->receivedSince($since, $lastId, self::BATCH_SIZE);
|
||||
|
||||
if (0 === count($batch)) {
|
||||
break;
|
||||
}
|
||||
|
||||
foreach ($batch as $message) {
|
||||
$offset++;
|
||||
$lastId = (int) $message->id;
|
||||
|
||||
// The harvester shields per extractor; a failed flush is the
|
||||
// one fault left, and the manager-closed guard below is what
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace App\Command\Backfill;
|
||||
|
||||
use DateTimeImmutable;
|
||||
use Symfony\Component\Console\Style\SymfonyStyle;
|
||||
|
||||
/**
|
||||
* A backfill that reads MAIL, and can therefore be told how far back to read.
|
||||
*
|
||||
* Most tasks repair a table: a column that was computed wrongly is wrong in
|
||||
* every row, and half a repair is a table in two states. Those implement
|
||||
* BackfillTaskInterface and always walk everything.
|
||||
*
|
||||
* A task that re-reads mail for what an extractor can now find in it is a
|
||||
* different kind of work. Its usual occasion is "a reader was fixed, and the
|
||||
* mail I am thinking of arrived this week" — and run over a whole mailbox it
|
||||
* answers that by also digging up every calendar file and parcel of the last
|
||||
* ten years. So these take a starting point, and BackfillCommand gives them
|
||||
* the last 24 hours unless it is told otherwise.
|
||||
*
|
||||
* A separate interface rather than a parameter on run(): a table repair that
|
||||
* accepted a window and ignored it would be a command that looks as though it
|
||||
* did what it was asked.
|
||||
*/
|
||||
interface WindowedBackfillTaskInterface extends BackfillTaskInterface
|
||||
{
|
||||
/**
|
||||
* @param DateTimeImmutable|null $since only mail received at or after this; null for all of it
|
||||
*/
|
||||
public function runSince(SymfonyStyle $io, ?DateTimeImmutable $since): int;
|
||||
}
|
||||
@@ -62,25 +62,31 @@ class MessageRepository extends ServiceEntityRepository
|
||||
)
|
||||
SQL;
|
||||
|
||||
/** Raw DBAL for the reasons EXTRACTION_CANDIDATE_WHERE gives above. */
|
||||
public function countExtractionCandidates(): int
|
||||
/**
|
||||
* Raw DBAL for the reasons EXTRACTION_CANDIDATE_WHERE gives above.
|
||||
*
|
||||
* @param DateTimeImmutable|null $since only mail received at or after this; null for all of it
|
||||
*/
|
||||
public function countExtractionCandidates(?DateTimeImmutable $since = null): int
|
||||
{
|
||||
return (int) $this->getEntityManager()->getConnection()->fetchOne(
|
||||
'SELECT COUNT(*) FROM message m WHERE ' . self::EXTRACTION_CANDIDATE_WHERE,
|
||||
$this->candidateParameters(0),
|
||||
'SELECT COUNT(*) FROM message m WHERE ' . self::EXTRACTION_CANDIDATE_WHERE . $this->candidateSince($since),
|
||||
$this->candidateParameters(0, $since),
|
||||
$this->candidateTypes(),
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param DateTimeImmutable|null $since only mail received at or after this; null for all of it
|
||||
*
|
||||
* @return list<Message>
|
||||
*/
|
||||
public function extractionCandidates(int $afterId, int $limit): array
|
||||
public function extractionCandidates(int $afterId, int $limit, ?DateTimeImmutable $since = null): array
|
||||
{
|
||||
$ids = $this->getEntityManager()->getConnection()->fetchFirstColumn(
|
||||
'SELECT m.id FROM message m WHERE ' . self::EXTRACTION_CANDIDATE_WHERE
|
||||
'SELECT m.id FROM message m WHERE ' . self::EXTRACTION_CANDIDATE_WHERE . $this->candidateSince($since)
|
||||
. ' ORDER BY m.id ASC LIMIT ' . max(1, $limit),
|
||||
$this->candidateParameters($afterId),
|
||||
$this->candidateParameters($afterId, $since),
|
||||
$this->candidateTypes(),
|
||||
);
|
||||
|
||||
@@ -94,9 +100,9 @@ class MessageRepository extends ServiceEntityRepository
|
||||
/**
|
||||
* @return array<string, mixed>
|
||||
*/
|
||||
private function candidateParameters(int $afterId): array
|
||||
private function candidateParameters(int $afterId, ?DateTimeImmutable $since = null): array
|
||||
{
|
||||
return [
|
||||
return (null === $since ? [] : ['extSince' => $since]) + [
|
||||
'extAfterId' => $afterId,
|
||||
'extCalendarTypes' => CalendarAttachment::CONTENT_TYPES,
|
||||
'extCalendarName' => '%' . CalendarAttachment::EXTENSION,
|
||||
@@ -105,6 +111,53 @@ class MessageRepository extends ServiceEntityRepository
|
||||
];
|
||||
}
|
||||
|
||||
/**
|
||||
* The clause that narrows candidates to recent mail, or nothing.
|
||||
*
|
||||
* Appended rather than written into the constant as "IS NULL OR": Postgres
|
||||
* cannot infer the type of a parameter that is only ever compared to NULL,
|
||||
* and a clause that is absent needs no parameter at all.
|
||||
*/
|
||||
private function candidateSince(?DateTimeImmutable $since): string
|
||||
{
|
||||
return null === $since ? '' : ' AND m.received_at >= :extSince';
|
||||
}
|
||||
|
||||
/**
|
||||
* Mail in id order, a batch at a time, optionally only what arrived since
|
||||
* a given moment — the walk a backfill that re-reads mail makes.
|
||||
*
|
||||
* Keyset on the id rather than OFFSET, so the cost of a batch does not
|
||||
* grow with how far the walk has got.
|
||||
*
|
||||
* @return list<Message>
|
||||
*/
|
||||
public function receivedSince(?DateTimeImmutable $since, int $afterId, int $limit): array
|
||||
{
|
||||
$qb = $this->createQueryBuilder('m')
|
||||
->where('m.id > :afterId')
|
||||
->setParameter('afterId', $afterId)
|
||||
->orderBy('m.id', 'ASC')
|
||||
->setMaxResults(max(1, $limit));
|
||||
|
||||
if (null !== $since) {
|
||||
$qb->andWhere('m.receivedAt >= :since')->setParameter('since', $since, Types::DATETIME_IMMUTABLE);
|
||||
}
|
||||
|
||||
return $qb->getQuery()->getResult();
|
||||
}
|
||||
|
||||
public function countReceivedSince(?DateTimeImmutable $since): int
|
||||
{
|
||||
$qb = $this->createQueryBuilder('m')->select('COUNT(m.id)');
|
||||
|
||||
if (null !== $since) {
|
||||
$qb->where('m.receivedAt >= :since')->setParameter('since', $since, Types::DATETIME_IMMUTABLE);
|
||||
}
|
||||
|
||||
return (int) $qb->getQuery()->getSingleScalarResult();
|
||||
}
|
||||
|
||||
/**
|
||||
* @return array<string, mixed>
|
||||
*/
|
||||
@@ -114,6 +167,7 @@ class MessageRepository extends ServiceEntityRepository
|
||||
'extAfterId' => ParameterType::INTEGER,
|
||||
'extCalendarTypes' => ArrayParameterType::STRING,
|
||||
'extCalendarName' => ParameterType::STRING,
|
||||
'extSince' => Types::DATETIME_IMMUTABLE,
|
||||
'extMeetingHeader' => ParameterType::STRING,
|
||||
'extJsonLd' => ParameterType::STRING,
|
||||
];
|
||||
|
||||
@@ -6,7 +6,12 @@ namespace App\Tests\Command\Backfill;
|
||||
|
||||
use App\Command\Backfill\BackfillCommand;
|
||||
use App\Command\Backfill\BackfillTaskInterface;
|
||||
use App\Command\Backfill\WindowedBackfillTaskInterface;
|
||||
use PHPUnit\Framework\Attributes\DataProvider;
|
||||
use PHPUnit\Framework\TestCase;
|
||||
use Symfony\Component\Clock\Clock;
|
||||
use Symfony\Component\Clock\MockClock;
|
||||
use Symfony\Component\Clock\NativeClock;
|
||||
use Symfony\Component\Console\Command\Command;
|
||||
use Symfony\Component\Console\Style\SymfonyStyle;
|
||||
use Symfony\Component\Console\Tester\CommandTester;
|
||||
@@ -26,6 +31,12 @@ use Symfony\Component\Console\Tester\CommandTester;
|
||||
*/
|
||||
final class BackfillCommandTest extends TestCase
|
||||
{
|
||||
protected function tearDown(): void
|
||||
{
|
||||
// --since is read against the global clock, which two tests pin.
|
||||
Clock::set(new NativeClock());
|
||||
}
|
||||
|
||||
public function testItRunsTheTaskNamedOnTheCommandLine(): void
|
||||
{
|
||||
$wanted = $this->task('wanted');
|
||||
@@ -134,6 +145,134 @@ final class BackfillCommandTest extends TestCase
|
||||
return new CommandTester(new BackfillCommand($tasks));
|
||||
}
|
||||
|
||||
// ── --since ───────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* The default is the point of the option: a task that re-reads mail and is
|
||||
* told nothing reads a day, not a mailbox.
|
||||
*/
|
||||
public function testATaskThatReadsMailGetsTheLastDayUnlessToldOtherwise(): void
|
||||
{
|
||||
Clock::set(new MockClock('2026-10-08 16:00:00 UTC'));
|
||||
|
||||
$events = $this->windowedTask('events');
|
||||
$tester = $this->tester($events);
|
||||
|
||||
self::assertSame(Command::SUCCESS, $tester->execute(['task' => 'events']));
|
||||
self::assertSame(['2026-10-07 16:00'], $events->windows);
|
||||
self::assertStringContainsString('2026-10-07 16:00', $tester->getDisplay());
|
||||
self::assertStringContainsString('--since=all', $tester->getDisplay(), 'How to widen it is said up front.');
|
||||
}
|
||||
|
||||
/**
|
||||
* @return iterable<string, array{0: string, 1: ?string}>
|
||||
*/
|
||||
public static function windows(): iterable
|
||||
{
|
||||
yield 'hours' => ['36h', '2026-10-07 04:00'];
|
||||
yield 'days' => ['7d', '2026-10-01 16:00'];
|
||||
yield 'weeks' => ['2w', '2026-09-24 16:00'];
|
||||
yield 'a date, from its start' => ['2026-09-01', '2026-09-01 00:00'];
|
||||
yield 'everything' => ['all', null];
|
||||
yield 'case and spaces' => [' ALL ', null];
|
||||
}
|
||||
|
||||
#[DataProvider('windows')]
|
||||
public function testSinceIsReadAsAnAmountADateOrEverything(string $since, ?string $expected): void
|
||||
{
|
||||
Clock::set(new MockClock('2026-10-08 16:00:00 UTC'));
|
||||
|
||||
$events = $this->windowedTask('events');
|
||||
|
||||
self::assertSame(Command::SUCCESS, $this->tester($events)->execute(['task' => 'events', '--since' => $since]));
|
||||
self::assertSame([$expected], $events->windows);
|
||||
}
|
||||
|
||||
/**
|
||||
* @return iterable<string, array{0: string}>
|
||||
*/
|
||||
public static function nonsense(): iterable
|
||||
{
|
||||
yield 'words' => ['yesterday'];
|
||||
yield 'no unit' => ['7'];
|
||||
yield 'zero' => ['0d'];
|
||||
yield 'a unit not offered' => ['3m'];
|
||||
yield 'a day that does not exist' => ['2026-02-31'];
|
||||
yield 'empty' => [''];
|
||||
}
|
||||
|
||||
/**
|
||||
* Refused, and nothing run. Guessing here would decide how much of a
|
||||
* mailbox is re-read on the strength of a typo.
|
||||
*/
|
||||
#[DataProvider('nonsense')]
|
||||
public function testAWindowItCannotReadRunsNothing(string $since): void
|
||||
{
|
||||
$events = $this->windowedTask('events');
|
||||
$tester = $this->tester($events);
|
||||
|
||||
self::assertSame(Command::FAILURE, $tester->execute(['task' => 'events', '--since' => $since]));
|
||||
self::assertSame([], $events->windows);
|
||||
self::assertStringContainsString('not understood', $tester->getDisplay());
|
||||
}
|
||||
|
||||
/**
|
||||
* A table repair has no window to honour. It still runs — over everything,
|
||||
* as it always did — and says that it ignored the option.
|
||||
*/
|
||||
public function testATableRepairSaysItTakesNoWindow(): void
|
||||
{
|
||||
$repair = $this->task('recipients');
|
||||
$tester = $this->tester($repair);
|
||||
|
||||
self::assertSame(Command::SUCCESS, $tester->execute(['task' => 'recipients', '--since' => '7d']));
|
||||
self::assertSame(1, $repair->runs);
|
||||
self::assertStringContainsString('takes no --since', $tester->getDisplay());
|
||||
}
|
||||
|
||||
public function testATableRepairGivenNoWindowSaysNothingAboutOne(): void
|
||||
{
|
||||
$tester = $this->tester($this->task('recipients'));
|
||||
$tester->execute(['task' => 'recipients']);
|
||||
|
||||
self::assertStringNotContainsString('--since', $tester->getDisplay());
|
||||
}
|
||||
|
||||
/**
|
||||
* @return WindowedBackfillTaskInterface&object{windows: list<?string>}
|
||||
*/
|
||||
private function windowedTask(string $name): object
|
||||
{
|
||||
return new class ($name) implements WindowedBackfillTaskInterface {
|
||||
/** @var list<?string> what each run was told to read from, in UTC; null for everything */
|
||||
public array $windows = [];
|
||||
|
||||
public function __construct(private readonly string $name) {}
|
||||
|
||||
public function getName(): string
|
||||
{
|
||||
return $this->name;
|
||||
}
|
||||
|
||||
public function getDescription(): string
|
||||
{
|
||||
return sprintf('Description of %s', $this->name);
|
||||
}
|
||||
|
||||
public function run(SymfonyStyle $io): int
|
||||
{
|
||||
throw new \LogicException('A windowed task is run through runSince().');
|
||||
}
|
||||
|
||||
public function runSince(SymfonyStyle $io, ?\DateTimeImmutable $since): int
|
||||
{
|
||||
$this->windows[] = $since?->setTimezone(new \DateTimeZone('UTC'))->format('Y-m-d H:i');
|
||||
|
||||
return Command::SUCCESS;
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* @return BackfillTaskInterface&object{runs: int}
|
||||
*/
|
||||
|
||||
@@ -578,6 +578,32 @@ final class IcsExtractionTest extends KernelTestCase
|
||||
self::assertContains($message->id, array_map(static fn (Message $m): ?int => $m->id, $candidates));
|
||||
}
|
||||
|
||||
/**
|
||||
* `app:backfill events --since=…`: mail from before the window is not a
|
||||
* candidate, however much calendar data it carries.
|
||||
*/
|
||||
public function testTheCandidateQueryCanBeNarrowedToRecentMail(): void
|
||||
{
|
||||
$old = $this->messageWithInvite('BEGIN:VCALENDAR');
|
||||
$old->receivedAt = new DateTimeImmutable('-3 days');
|
||||
$new = $this->messageWithInvite('BEGIN:VCALENDAR');
|
||||
$this->em->flush();
|
||||
|
||||
$messages = $this->em->getRepository(Message::class);
|
||||
$after = (int) $old->id - 1;
|
||||
$ids = static fn (array $found): array => array_map(static fn (Message $m): ?int => $m->id, $found);
|
||||
|
||||
$recent = $ids($messages->extractionCandidates($after, 10, new DateTimeImmutable('-24 hours')));
|
||||
|
||||
self::assertContains($new->id, $recent);
|
||||
self::assertNotContains($old->id, $recent);
|
||||
self::assertContains($old->id, $ids($messages->extractionCandidates($after, 10)), 'no window, no limit');
|
||||
self::assertLessThan(
|
||||
$messages->countExtractionCandidates(),
|
||||
$messages->countExtractionCandidates(new DateTimeImmutable('-24 hours')),
|
||||
);
|
||||
}
|
||||
|
||||
// ── Fixtures ──────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
|
||||
Reference in new issue
Block a user