mirror of
https://github.com/karatektus/pl_mail.git
synced 2026-10-25 01:54:10 +00:00
The blob prune follows APP_STORAGE_DIR and clears orphaned thumbnails (#234)
app:prune:blobs looked under a hardcoded var/, so installs with another APP_STORAGE_DIR reclaimed nothing; it and the thumbnail cache now use APP_STORAGE_DIR. A third pass deletes attachment thumbnails whose part no longer exists. Closes #232, closes #233. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
cdd0cd4fd8
commit
9c76bbc636
10 files changed
+343
-33
No files matched your search
@@ -8,6 +8,16 @@ The published image tags: `latest` follows the most recent release below,
|
||||
|
||||
## Unreleased
|
||||
|
||||
**The weekly cleanup works on installs that moved their storage, and clears old thumbnails.**
|
||||
`app:prune:blobs` looked for leftover files under a fixed `var/`, so on an install that sets
|
||||
`APP_STORAGE_DIR` to another directory it found nothing and reclaimed nothing. It now looks where
|
||||
the files are written. The preview thumbnails of attachments are kept there too now, beside the
|
||||
attachments, instead of under a fixed `var/`; on such an install the old `var/attachments/thumbs`
|
||||
directory can be deleted, and the thumbnails are drawn again when they are next shown. The cleanup
|
||||
also deletes the thumbnails of attachments that no longer exist, which nothing removed before. No
|
||||
schema change. [#232](https://github.com/karatektus/pl_mail/issues/232),
|
||||
[#233](https://github.com/karatektus/pl_mail/issues/233).
|
||||
|
||||
**The installed app has the icon you picked, on iPhone and Android too.** The tab icon followed
|
||||
Settings → Appearance → Logo, but an app installed from the browser or added to a home screen kept
|
||||
the wax seal, because its icons came from fixed files. The install manifest and the iOS home-screen
|
||||
|
||||
+1
-1
@@ -929,7 +929,7 @@ php bin/mirror-wiki.php --check
|
||||
| `app:gmail:quota-probe [account] [--mode=all] [--count=30]` | Without an account, list the Gmail accounts and ask which. Fetch a fixed number of stored Gmail messages one way per minute (batched as an import does, or singly in each format), holding the account's import meanwhile, so the cost Google charges for each can be read off the Cloud console's quota chart. Stores and changes nothing |
|
||||
| `app:gmail:skipped [account] [--list=N] [--forget]` | Show the Gmail messages an import looked at and did not store. Without an account, one line per Gmail account: stored, skipped, Gmail's own total. With one, the skipped messages grouped by kind (chat, draft, sent from another address, delivered to another address, unfetchable), by Gmail label and by address, and with `--list` the messages themselves. Reads only what was recorded when each message was looked at; asks Gmail nothing. `--forget` deletes the record for the account so the next listing asks for those messages again |
|
||||
| `app:attachments:reclassify` | Recompute inline/attachment classification for stored parts |
|
||||
| `app:prune:blobs [--days=N] [--dry-run]` | Expire staged JMAP uploads and delete files orphaned by deleted rows. Only looks in the per-account directories of `var/uploads`, `var/attachments` and `var/raw`; everything else on those volumes belongs to someone else |
|
||||
| `app:prune:blobs [--days=N] [--dry-run]` | Expire staged JMAP uploads, delete files orphaned by deleted rows and the thumbnails of attachments that no longer exist. Only looks in the per-account directories of `uploads`, `attachments` and `raw` under `APP_STORAGE_DIR`, and in `attachments/thumbs`; everything else on those volumes belongs to someone else |
|
||||
| `app:user:promote <email> [--revoke]` | Grant or revoke `ROLE_ADMIN` |
|
||||
| `app:user:2fa-disable <email> [--force]` | Turn off 2FA for someone locked out — see "Two-factor authentication" |
|
||||
| `app:user:password <email> [--password=…]` | Set a new password for a user who can no longer sign in. Prompted for, hidden and twice, unless `--password` is given. Console-only for the same reason as `app:user:2fa-disable`: an administrator who could do it from a browser would be a second way into that person's mailbox |
|
||||
|
||||
@@ -168,6 +168,7 @@ services:
|
||||
App\Command\Maintenance\PruneBlobsCommand:
|
||||
arguments:
|
||||
$projectDir: '%kernel.project_dir%'
|
||||
$storageDir: '%env(APP_STORAGE_DIR)%'
|
||||
|
||||
App\Service\Ai\BackfillPolicy:
|
||||
arguments:
|
||||
@@ -181,6 +182,7 @@ services:
|
||||
App\Service\Mail\AttachmentThumbnailer:
|
||||
arguments:
|
||||
$projectDir: '%kernel.project_dir%'
|
||||
$storageDir: '%env(APP_STORAGE_DIR)%'
|
||||
App\Service\Mail\MessageFrameScript:
|
||||
arguments:
|
||||
$projectDir: '%kernel.project_dir%'
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
<!-- translated-from: internals/architecture.md sha1:ec72660311a2cdee4cc506e88e54e70e89f571c7 -->
|
||||
<!-- translated-from: internals/architecture.md sha1:776308c1fc128a2474116af3c889c58aa5c5e49a -->
|
||||
# Architektur
|
||||
|
||||
Die Schichten, was wo liegt, und die Regeln, die das so halten. Diese Seite beschreibt die
|
||||
@@ -237,7 +237,7 @@ war, in dem das Projekt vorher steckte: Logs und verwaiste Blobs wuchsen ohne Gr
|
||||
| `50 3 * * *` | `app:calendar:materialise` | Rollt den Horizont der Termininstanzen vorwärts, damit einem Serientermin nicht klammheimlich die Termine ausgehen |
|
||||
| `30 4 * * *` | `app:monitoring:prune` | Log-Einträge, Push-Zustellungen, tote Heartbeats, fertige Hintergrundjobs (30 Tage) und abgelaufene vertraute Geräte |
|
||||
| `37 * * * *` | `app:updates:check` | Stündlich, zu einer krummen Minute: Die Registry und GitHubs anonyme API teilen sich alle Installationen. Jeder neue Build wird einmal angekündigt, wie oft er auch gefunden wird |
|
||||
| `0 5 * * 0` | `app:prune:blobs` | Wöchentlich; er läuft die Verzeichnisse pro Konto in drei Ablagen ab, und eine Woche Waisen ist ein Rundungsfehler. Nur diese: Profilbilder, Signaturbilder, Hintergründe und alles andere auf denselben Volumes bleiben unangetastet |
|
||||
| `0 5 * * 0` | `app:prune:blobs` | Wöchentlich; er läuft die Verzeichnisse pro Konto in drei Ablagen unter `APP_STORAGE_DIR` ab, dazu die Vorschaubilder der Anhänge, und eine Woche Waisen ist ein Rundungsfehler. Nur diese: Profilbilder, Signaturbilder, Hintergründe und alles andere auf denselben Volumes bleiben unangetastet |
|
||||
|
||||
Der Zeitplan ist `stateful()` gegen den Cache und `processOnlyLastMissedRun(true)`. Ein Worker,
|
||||
der über einen geplanten Lauf hinweg unten war, holt also nach, statt den Tag stillschweigend
|
||||
|
||||
@@ -219,7 +219,7 @@ in before, with logs and orphaned blobs growing without bound.
|
||||
| `50 3 * * *` | `app:calendar:materialise` | Rolls the occurrence horizon forward, so a repeating event does not quietly run out of dates |
|
||||
| `30 4 * * *` | `app:monitoring:prune` | Log entries, push deliveries, dead heartbeats, finished background jobs (30 days) and expired trusted devices |
|
||||
| `37 * * * *` | `app:updates:check` | Hourly, at an odd minute: the registry and GitHub's anonymous API are shared with everybody else's installations. Each new build is announced once, however often it is found |
|
||||
| `0 5 * * 0` | `app:prune:blobs` | Weekly; it walks the per-account trees of three stores and a week of orphans is a rounding error. Only those trees: avatars, signature images, backgrounds and anything else kept under the same volumes are left alone |
|
||||
| `0 5 * * 0` | `app:prune:blobs` | Weekly; it walks the per-account trees of three stores under `APP_STORAGE_DIR`, and the attachment thumbnails, and a week of orphans is a rounding error. Only those: avatars, signature images, backgrounds and anything else kept under the same volumes are left alone |
|
||||
|
||||
The schedule is `stateful()` against the cache and `processOnlyLastMissedRun(true)`, so a
|
||||
worker that was down over a scheduled run catches up rather than silently skipping the day —
|
||||
|
||||
@@ -8,6 +8,7 @@ use App\Domain\Helper\UploadStorage;
|
||||
use App\Repository\Mail\MessagePartRepository;
|
||||
use App\Repository\Mail\MessageRepository;
|
||||
use App\Repository\Mail\UploadedBlobRepository;
|
||||
use App\Service\Mail\AttachmentThumbnailer;
|
||||
use Doctrine\ORM\EntityManagerInterface;
|
||||
use Symfony\Component\Console\Attribute\AsCommand;
|
||||
use Symfony\Component\Console\Command\Command;
|
||||
@@ -48,13 +49,24 @@ use Symfony\Component\Console\Style\SymfonyStyle;
|
||||
* A week of slack is many orders of magnitude more than that gap.
|
||||
*
|
||||
* The second pass only looks where the three writers write: the per-account
|
||||
* trees, var/uploads/<account>/, var/attachments/<account>/ and
|
||||
* var/raw/<account>/, and in them only files laid out the way the writer lays
|
||||
* them out. The roots are shared volumes, and other owners keep files there
|
||||
* that no column of ours names: profile pictures (uploads/avatars), signature
|
||||
* images (uploads/signatures), appearance backgrounds (uploads/backgrounds),
|
||||
* attachment thumbnails (attachments/thumbs). Until #214 the sweep walked the
|
||||
* whole root, took those for orphans and deleted them every Sunday.
|
||||
* trees, uploads/<account>/, attachments/<account>/ and raw/<account>/, and
|
||||
* in them only files laid out the way the writer lays them out. The roots are
|
||||
* shared volumes, and other owners keep files there that no column of ours
|
||||
* names: profile pictures (uploads/avatars), signature images
|
||||
* (uploads/signatures), appearance backgrounds (uploads/backgrounds). Until
|
||||
* #214 the sweep walked the whole root, took those for orphans and deleted
|
||||
* them every Sunday, along with every attachment thumbnail (attachments/thumbs),
|
||||
* which the third pass below now handles by its own rule.
|
||||
*
|
||||
* The roots are under APP_STORAGE_DIR, the same directory the writers use
|
||||
* and the prefix of every path the three columns hold. They were hardcoded
|
||||
* as var/… until #233, so an install with another storage dir swept
|
||||
* directories nothing wrote to and reclaimed nothing.
|
||||
*
|
||||
* A third pass clears attachment thumbnails (attachments/thumbs/<part>.webp,
|
||||
* AttachmentThumbnailer) whose part no longer exists. Nothing else deletes
|
||||
* them, and until #228 they were only cleared by accident, along with the
|
||||
* thumbnails of parts that still existed (#232).
|
||||
*
|
||||
* The road not taken is a deny-list of those directories. It would have been
|
||||
* a smaller diff, and it fails the same way the original did: the next owner
|
||||
@@ -72,8 +84,8 @@ final class PruneBlobsCommand extends Command
|
||||
private const int DEFAULT_DAYS = 7;
|
||||
|
||||
/**
|
||||
* The roots this command sweeps, each with the shape of a path its writer
|
||||
* produces, relative to the root. Which column references files in each is
|
||||
* The stores this command sweeps, by their directory under APP_STORAGE_DIR,
|
||||
* each with the shape of a path its writer produces, relative to it. Which column references files in each is
|
||||
* the repositories' business — see referencedPaths().
|
||||
*
|
||||
* The first segment is always the account id, so a directory under a root
|
||||
@@ -91,11 +103,17 @@ final class PruneBlobsCommand extends Command
|
||||
* @var array<string, non-empty-string>
|
||||
*/
|
||||
private const array STORES = [
|
||||
'var/uploads' => '#^\d+/[0-9a-f]{32}\.blob$#',
|
||||
'var/attachments' => '#^\d+/\d+/\d+/[^/]+$#',
|
||||
'var/raw' => '#^\d+/\d+/\d+\.eml$#',
|
||||
'uploads' => '#^\d+/[0-9a-f]{32}\.blob$#',
|
||||
'attachments' => '#^\d+/\d+/\d+/[^/]+$#',
|
||||
'raw' => '#^\d+/\d+/\d+\.eml$#',
|
||||
];
|
||||
|
||||
/** How many part ids one existence query asks about. */
|
||||
private const int ID_BATCH = 1000;
|
||||
|
||||
/** The largest message_part.id there can be (a Postgres integer). */
|
||||
private const int MAX_PART_ID = 2_147_483_647;
|
||||
|
||||
public function __construct(
|
||||
private readonly UploadedBlobRepository $repository,
|
||||
private readonly MessagePartRepository $messageParts,
|
||||
@@ -103,6 +121,7 @@ final class PruneBlobsCommand extends Command
|
||||
private readonly UploadStorage $storage,
|
||||
private readonly EntityManagerInterface $em,
|
||||
private readonly string $projectDir,
|
||||
private readonly string $storageDir,
|
||||
) {
|
||||
parent::__construct();
|
||||
}
|
||||
@@ -145,11 +164,15 @@ final class PruneBlobsCommand extends Command
|
||||
}
|
||||
|
||||
if (false === $input->getOption('skip-orphans')) {
|
||||
foreach (self::STORES as $root => $layout) {
|
||||
[$count, $size] = $this->sweepOrphans($io, $root, $layout, $cutoff, $dryRun);
|
||||
foreach (self::STORES as $store => $layout) {
|
||||
[$count, $size] = $this->sweepOrphans($io, $store, $layout, $cutoff, $dryRun);
|
||||
$files += $count;
|
||||
$bytes += $size;
|
||||
}
|
||||
|
||||
[$count, $size] = $this->sweepThumbnails($io, $cutoff, $dryRun);
|
||||
$files += $count;
|
||||
$bytes += $size;
|
||||
}
|
||||
|
||||
$io->success(sprintf(
|
||||
@@ -202,14 +225,15 @@ final class PruneBlobsCommand extends Command
|
||||
}
|
||||
|
||||
/**
|
||||
* Pass 2: files in $root's account trees that no row points at any more.
|
||||
* Pass 2: files in a store's account trees that no row points at any more.
|
||||
*
|
||||
* @param non-empty-string $layout
|
||||
*
|
||||
* @return array{int, int} files, bytes
|
||||
*/
|
||||
private function sweepOrphans(SymfonyStyle $io, string $root, string $layout, \DateTimeImmutable $cutoff, bool $dryRun): array
|
||||
private function sweepOrphans(SymfonyStyle $io, string $store, string $layout, \DateTimeImmutable $cutoff, bool $dryRun): array
|
||||
{
|
||||
$root = $this->storageDir.'/'.$store;
|
||||
$absoluteRoot = $this->projectDir.'/'.$root;
|
||||
|
||||
if (false === is_dir($absoluteRoot)) {
|
||||
@@ -218,7 +242,7 @@ final class PruneBlobsCommand extends Command
|
||||
return [0, 0];
|
||||
}
|
||||
|
||||
$referenced = $this->referencedPaths($root);
|
||||
$referenced = $this->referencedPaths($store, $root);
|
||||
|
||||
$files = 0;
|
||||
$bytes = 0;
|
||||
@@ -274,19 +298,83 @@ final class PruneBlobsCommand extends Command
|
||||
*
|
||||
* @return array<string, true>
|
||||
*/
|
||||
private function referencedPaths(string $root): array
|
||||
private function referencedPaths(string $store, string $root): array
|
||||
{
|
||||
return match ($root) {
|
||||
'var/uploads' => $this->repository->findReferencedPaths($root),
|
||||
'var/attachments' => $this->messageParts->findReferencedStoragePaths($root),
|
||||
'var/raw' => $this->messages->findReferencedRawPaths($root),
|
||||
// Reached only by adding a root to STORES without saying who owns
|
||||
return match ($store) {
|
||||
'uploads' => $this->repository->findReferencedPaths($root),
|
||||
'attachments' => $this->messageParts->findReferencedStoragePaths($root),
|
||||
'raw' => $this->messages->findReferencedRawPaths($root),
|
||||
// Reached only by adding a store to STORES without saying who owns
|
||||
// it. Loud, because the silent answer would be an empty keep-list,
|
||||
// and an empty keep-list deletes the store.
|
||||
default => throw new \LogicException(sprintf('No repository owns the blob store "%s".', $root)),
|
||||
default => throw new \LogicException(sprintf('No repository owns the blob store "%s".', $store)),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Pass 3: thumbnails of attachments that no longer exist (#232).
|
||||
*
|
||||
* A thumbnail is named by its part's id and nothing deletes it with the
|
||||
* part, so it is kept exactly while a message_part row with that id
|
||||
* exists. Only files named `<id>.webp` are considered; anything else in
|
||||
* the directory is left alone. The retention window applies as in the
|
||||
* other passes, although a thumbnail is only ever written for a part that
|
||||
* is already committed.
|
||||
*
|
||||
* @return array{int, int} files, bytes
|
||||
*/
|
||||
private function sweepThumbnails(SymfonyStyle $io, \DateTimeImmutable $cutoff, bool $dryRun): array
|
||||
{
|
||||
$directory = $this->projectDir.'/'.$this->storageDir.'/'.AttachmentThumbnailer::DIRECTORY;
|
||||
|
||||
if (false === is_dir($directory)) {
|
||||
$io->text('Orphaned thumbnails: none yet.');
|
||||
|
||||
return [0, 0];
|
||||
}
|
||||
|
||||
/** @var array<int, \SplFileInfo> $candidates part id => file */
|
||||
$candidates = [];
|
||||
|
||||
foreach (new \FilesystemIterator($directory) as $file) {
|
||||
if (
|
||||
$file instanceof \SplFileInfo
|
||||
&& true === $file->isFile()
|
||||
&& 1 === preg_match('/^(\d{1,10})\.webp$/', $file->getFilename(), $match)
|
||||
// message_part.id is a Postgres integer: a larger number is no
|
||||
// thumbnail of ours, and asking about it would fail the query.
|
||||
&& (int) $match[1] <= self::MAX_PART_ID
|
||||
&& $file->getMTime() < $cutoff->getTimestamp()
|
||||
) {
|
||||
$candidates[(int) $match[1]] = $file;
|
||||
}
|
||||
}
|
||||
|
||||
$files = 0;
|
||||
$bytes = 0;
|
||||
|
||||
foreach (array_chunk(array_keys($candidates), self::ID_BATCH) as $ids) {
|
||||
$existing = $this->messageParts->findExistingIds($ids);
|
||||
|
||||
foreach ($ids as $id) {
|
||||
if (true === isset($existing[$id])) {
|
||||
continue;
|
||||
}
|
||||
|
||||
++$files;
|
||||
$bytes += $candidates[$id]->getSize();
|
||||
|
||||
if (false === $dryRun) {
|
||||
@unlink($candidates[$id]->getPathname());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
$io->text(sprintf('Orphaned thumbnails: %d (%s).', $files, $this->humanBytes($bytes)));
|
||||
|
||||
return [$files, $bytes];
|
||||
}
|
||||
|
||||
/**
|
||||
* The per-account directories directly under a root: the ones named by a
|
||||
* number. Everything else at that level belongs to someone else.
|
||||
|
||||
@@ -4,6 +4,7 @@ namespace App\Repository\Mail;
|
||||
|
||||
use App\Entity\Mail\MessagePart;
|
||||
use Doctrine\Bundle\DoctrineBundle\Repository\ServiceEntityRepository;
|
||||
use Doctrine\DBAL\ArrayParameterType;
|
||||
use Doctrine\Persistence\ManagerRegistry;
|
||||
|
||||
/**
|
||||
@@ -62,4 +63,33 @@ class MessagePartRepository extends ServiceEntityRepository
|
||||
|
||||
return $referenced;
|
||||
}
|
||||
|
||||
/**
|
||||
* Which of these part ids still exist, as a lookup set. For the thumbnail
|
||||
* sweep in PruneBlobsCommand, which names its files by part id.
|
||||
*
|
||||
* @param list<int> $ids
|
||||
*
|
||||
* @return array<int, true>
|
||||
*/
|
||||
public function findExistingIds(array $ids): array
|
||||
{
|
||||
if ([] === $ids) {
|
||||
return [];
|
||||
}
|
||||
|
||||
$existing = [];
|
||||
|
||||
$result = $this->getEntityManager()->getConnection()->executeQuery(
|
||||
'SELECT id FROM message_part WHERE id IN (:ids)',
|
||||
['ids' => $ids],
|
||||
['ids' => ArrayParameterType::INTEGER],
|
||||
);
|
||||
|
||||
foreach ($result->iterateColumn() as $id) {
|
||||
$existing[(int) $id] = true;
|
||||
}
|
||||
|
||||
return $existing;
|
||||
}
|
||||
}
|
||||
@@ -24,6 +24,10 @@ use Psr\Log\LoggerInterface;
|
||||
* Thumbnails are written next to nothing — a flat cache directory keyed by part
|
||||
* id — and generated once. The part's bytes are immutable, so there is no
|
||||
* invalidation to get wrong; deleting the directory is a full, safe reset.
|
||||
*
|
||||
* The directory is under APP_STORAGE_DIR, beside the attachments themselves
|
||||
* (it was a hardcoded var/ until #233). A thumbnail outlives its part until
|
||||
* the weekly app:prune:blobs clears it (#232).
|
||||
*/
|
||||
final readonly class AttachmentThumbnailer
|
||||
{
|
||||
@@ -37,10 +41,14 @@ final readonly class AttachmentThumbnailer
|
||||
|
||||
private const int EDGE = 160;
|
||||
|
||||
/** The cache directory, relative to APP_STORAGE_DIR. PruneBlobsCommand sweeps it. */
|
||||
public const string DIRECTORY = 'attachments/thumbs';
|
||||
|
||||
public function __construct(
|
||||
private AttachmentResolver $attachments,
|
||||
private LoggerInterface $logger,
|
||||
private string $projectDir,
|
||||
private string $storageDir,
|
||||
) {
|
||||
}
|
||||
|
||||
@@ -75,7 +83,7 @@ final readonly class AttachmentThumbnailer
|
||||
return null;
|
||||
}
|
||||
|
||||
$cachePath = sprintf('%s/var/attachments/thumbs/%d.webp', $this->projectDir, $part->id);
|
||||
$cachePath = sprintf('%s/%s/%s/%d.webp', $this->projectDir, $this->storageDir, self::DIRECTORY, $part->id);
|
||||
|
||||
if (true === is_file($cachePath)) {
|
||||
return $cachePath;
|
||||
|
||||
@@ -53,6 +53,7 @@ final class PruneBlobsCommandTest extends TestCase
|
||||
'var/uploads/avatars/3/3f9c1e.png',
|
||||
'var/uploads/signatures/3/logo-a1b2c3.png',
|
||||
'var/uploads/backgrounds/7d2e.jpg',
|
||||
// A thumbnail of a part that exists: the thumbnail pass's, and kept.
|
||||
'var/attachments/thumbs/42.webp',
|
||||
// A store nobody has written yet, in the shape the sealed messages
|
||||
// of v0.4.0 take: it is left alone without anyone listing it here.
|
||||
@@ -68,7 +69,7 @@ final class PruneBlobsCommandTest extends TestCase
|
||||
$this->file($path, self::OLD);
|
||||
}
|
||||
|
||||
$exit = $this->command()->execute([]);
|
||||
$exit = $this->command(existingParts: [42])->execute([]);
|
||||
|
||||
self::assertSame(Command::SUCCESS, $exit);
|
||||
|
||||
@@ -147,6 +148,69 @@ final class PruneBlobsCommandTest extends TestCase
|
||||
self::assertStringContainsString('Orphans in var/raw: 1', $tester->getDisplay());
|
||||
}
|
||||
|
||||
/**
|
||||
* #233: the stores are under APP_STORAGE_DIR, not a hardcoded var/. On an
|
||||
* install that moved them, the sweep looked where nothing was written and
|
||||
* reclaimed nothing.
|
||||
*/
|
||||
public function testTheStoresAreUnderTheStorageDirectory(): void
|
||||
{
|
||||
$orphan = 'var/data/raw/3/1/1042.eml';
|
||||
$referenced = 'var/data/attachments/3/12/4711/kept.pdf';
|
||||
$elsewhere = 'var/raw/3/1/1042.eml';
|
||||
|
||||
foreach ([$orphan, $referenced, $elsewhere] as $path) {
|
||||
$this->file($path, self::OLD);
|
||||
}
|
||||
|
||||
$this->command(parts: [$referenced => true], storageDir: 'var/data')->execute([]);
|
||||
|
||||
self::assertFileDoesNotExist($this->projectDir.'/'.$orphan, 'swept where the writers write');
|
||||
self::assertFileExists($this->projectDir.'/'.$referenced, 'the keep-list is asked with the storage dir\'s prefix');
|
||||
self::assertFileExists($this->projectDir.'/'.$elsewhere, 'and var/ is not this install\'s store');
|
||||
}
|
||||
|
||||
/**
|
||||
* #232: a thumbnail is named by its part id, and nothing deletes it with
|
||||
* the part. It goes once the part is gone, and not before.
|
||||
*/
|
||||
public function testThumbnailsOfPartsThatAreGoneAreDeleted(): void
|
||||
{
|
||||
$gone = 'var/attachments/thumbs/41.webp';
|
||||
$existing = 'var/attachments/thumbs/42.webp';
|
||||
$recent = 'var/attachments/thumbs/43.webp';
|
||||
$foreign = 'var/attachments/thumbs/notes.txt';
|
||||
// Larger than any message_part.id can be: not ours, and it must not
|
||||
// turn the existence query into an error that stops the run.
|
||||
$tooLarge = 'var/attachments/thumbs/99999999999.webp';
|
||||
|
||||
foreach ([$gone, $existing, $foreign, $tooLarge] as $path) {
|
||||
$this->file($path, self::OLD);
|
||||
}
|
||||
|
||||
$this->file($recent, '-1 day');
|
||||
|
||||
$tester = $this->command(existingParts: [42]);
|
||||
$tester->execute([]);
|
||||
|
||||
self::assertFileDoesNotExist($this->projectDir.'/'.$gone);
|
||||
self::assertFileExists($this->projectDir.'/'.$existing, 'its part still exists');
|
||||
self::assertFileExists($this->projectDir.'/'.$recent, 'inside the window');
|
||||
self::assertFileExists($this->projectDir.'/'.$foreign, 'not a thumbnail');
|
||||
self::assertFileExists($this->projectDir.'/'.$tooLarge, 'not a part id');
|
||||
self::assertStringContainsString('Orphaned thumbnails: 1', $tester->getDisplay());
|
||||
}
|
||||
|
||||
public function testADryRunKeepsThumbnailsToo(): void
|
||||
{
|
||||
$gone = 'var/attachments/thumbs/41.webp';
|
||||
$this->file($gone, self::OLD);
|
||||
|
||||
$this->command()->execute(['--dry-run' => true]);
|
||||
|
||||
self::assertFileExists($this->projectDir.'/'.$gone);
|
||||
}
|
||||
|
||||
private function file(string $relativePath, string $age): void
|
||||
{
|
||||
$path = $this->projectDir.'/'.$relativePath;
|
||||
@@ -163,15 +227,24 @@ final class PruneBlobsCommandTest extends TestCase
|
||||
* @param array<string, true> $uploads
|
||||
* @param array<string, true> $parts
|
||||
* @param array<string, true> $raw
|
||||
* @param list<int> $existingParts part ids that still exist
|
||||
*/
|
||||
private function command(array $uploads = [], array $parts = [], array $raw = []): CommandTester
|
||||
{
|
||||
private function command(
|
||||
array $uploads = [],
|
||||
array $parts = [],
|
||||
array $raw = [],
|
||||
array $existingParts = [],
|
||||
string $storageDir = 'var',
|
||||
): CommandTester {
|
||||
$blobs = $this->createStub(UploadedBlobRepository::class);
|
||||
$blobs->method('findOlderThan')->willReturn([]);
|
||||
$blobs->method('findReferencedPaths')->willReturn($uploads);
|
||||
|
||||
$messageParts = $this->createStub(MessagePartRepository::class);
|
||||
$messageParts->method('findReferencedStoragePaths')->willReturn($parts);
|
||||
$messageParts->method('findExistingIds')->willReturnCallback(
|
||||
static fn (array $ids): array => array_fill_keys(array_values(array_intersect($ids, $existingParts)), true),
|
||||
);
|
||||
|
||||
$messages = $this->createStub(MessageRepository::class);
|
||||
$messages->method('findReferencedRawPaths')->willReturn($raw);
|
||||
@@ -180,9 +253,10 @@ final class PruneBlobsCommandTest extends TestCase
|
||||
$blobs,
|
||||
$messageParts,
|
||||
$messages,
|
||||
new UploadStorage($this->projectDir, 'var'),
|
||||
new UploadStorage($this->projectDir, $storageDir),
|
||||
$this->createStub(EntityManagerInterface::class),
|
||||
$this->projectDir,
|
||||
$storageDir,
|
||||
));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace App\Tests\Repository\Mail;
|
||||
|
||||
use App\Domain\Enum\Mail\LabelRole;
|
||||
use App\Entity\Mail\Message;
|
||||
use App\Entity\Mail\MessagePart;
|
||||
use App\Repository\Mail\MessagePartRepository;
|
||||
use App\Tests\Support\Mail\SeedsMarkerFixtures;
|
||||
use Doctrine\ORM\EntityManagerInterface;
|
||||
use Symfony\Bundle\FrameworkBundle\Test\KernelTestCase;
|
||||
|
||||
/**
|
||||
* Which part ids still exist: the thumbnail sweep's keep-list (#232).
|
||||
*
|
||||
* Against the real database, because the answer decides which files the
|
||||
* weekly prune deletes: an id it wrongly calls gone costs a thumbnail, and a
|
||||
* query that comes back empty would cost all of them.
|
||||
*/
|
||||
final class MessagePartExistingIdsTest extends KernelTestCase
|
||||
{
|
||||
use SeedsMarkerFixtures;
|
||||
|
||||
protected function setUp(): void
|
||||
{
|
||||
self::bootKernel();
|
||||
|
||||
$this->em = static::getContainer()->get(EntityManagerInterface::class);
|
||||
$this->em->getConnection()->beginTransaction();
|
||||
|
||||
$this->user = $this->seedUser();
|
||||
$this->account = $this->seedAccount();
|
||||
$this->inbox = $this->seedLabel('Inbox', LabelRole::Inbox);
|
||||
}
|
||||
|
||||
protected function tearDown(): void
|
||||
{
|
||||
if (true === $this->em->getConnection()->isTransactionActive()) {
|
||||
$this->em->getConnection()->rollBack();
|
||||
}
|
||||
|
||||
parent::tearDown();
|
||||
}
|
||||
|
||||
public function testOnlyTheIdsOfPartsThatExistComeBack(): void
|
||||
{
|
||||
$first = $this->part();
|
||||
$second = $this->part();
|
||||
$gone = $this->part();
|
||||
|
||||
$goneId = (int) $gone->id;
|
||||
$this->em->remove($gone);
|
||||
$this->em->flush();
|
||||
|
||||
$existing = static::getContainer()->get(MessagePartRepository::class)
|
||||
->findExistingIds([(int) $first->id, (int) $second->id, $goneId, 2_147_483_647]);
|
||||
|
||||
self::assertSame([(int) $first->id => true, (int) $second->id => true], $this->sorted($existing));
|
||||
}
|
||||
|
||||
public function testNoIdsAskNothing(): void
|
||||
{
|
||||
self::assertSame([], static::getContainer()->get(MessagePartRepository::class)->findExistingIds([]));
|
||||
}
|
||||
|
||||
private function part(): MessagePart
|
||||
{
|
||||
/** @var Message $message */
|
||||
$message = $this->thread('Teile')->messages->first();
|
||||
|
||||
$part = new MessagePart();
|
||||
$part->contentType = 'image/png';
|
||||
$part->filename = 'p.png';
|
||||
$part->disposition = 'attachment';
|
||||
$part->size = 1;
|
||||
$part->storagePath = 'var/attachments/1/1/1/p.png';
|
||||
|
||||
$message->addMessagePart($part);
|
||||
$this->em->persist($part);
|
||||
$this->em->flush();
|
||||
|
||||
return $part;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param array<int, true> $set
|
||||
*
|
||||
* @return array<int, true>
|
||||
*/
|
||||
private function sorted(array $set): array
|
||||
{
|
||||
ksort($set);
|
||||
|
||||
return $set;
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user