Put any job Gmail refuses on quota back quietly, not only the import's

b44283cb made a quota refusal "not now" for a batch, an import page and
a sync. Everything else that calls Gmail still threw it to Messenger: a
label change pushed while an import had the quota was retried with a
WARNING and two stack traces per attempt (#90).

GiveUpQuietlyOnThrottling now takes a Gmail quota refusal on the first
attempt, for every message alike: the same message goes back onto the
queue it came from, delayed until the account's minute is over, with one
line at info. The job's own retries are not spent on it. After sixty
such waits in a row the refusal is a failure again and goes to the
retry ladder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CHANGELOG.md

@@ -15,7 +15,8 @@ a new project most of what an import asked for was refused, the log filled with
 warnings and an import could stop without resuming. plMail now reads the limit out of Google's
 first refusal and keeps to it — nothing to configure — counts a message at its real cost,
 fetches fifty at a time, and shares one allowance between its workers. A refusal is no longer an
-error: the batch, import page or sync that met it waits out the minute and carries on. The
+error: whatever met it — a batch, an import page, a sync, a label change being pushed — waits
+out the minute and carries on. The
 import line in the background-work indicator now shows the quota, roughly how much of it is in
 use, how long the rest of the import takes, and says so when Google has paused it. No
 migration. [#90](https://github.com/karatektus/pl_mail/issues/90), reported by @karamanliev.

src/Infrastructure/Messaging/Middleware/GiveUpQuietlyOnThrottling.php

@@ -10,6 +10,7 @@ use App\Infrastructure\Messaging\Message\SyncAccountMessage;
 use App\Infrastructure\Messaging\Message\SyncGmailMessageBatchMessage;
 use App\Infrastructure\Messaging\Message\SyncGraphMessageBatchMessage;
 use App\Infrastructure\Messaging\Stamp\ThrottleDeferralStamp;
+use Psr\Container\ContainerInterface;
 use Psr\Log\LoggerInterface;
 use Symfony\Component\DependencyInjection\Attribute\Autowire;
 use Symfony\Component\Messenger\Envelope;
@@ -59,6 +60,25 @@ use Symfony\Component\Messenger\Transport\Sender\SenderInterface;
  * The sender is the ingest transport itself rather than the bus, which is
  * where both batch messages are routed: this class is PART of the bus, and
  * asking for the bus here would be asking for itself.
+ *
+ * ## Gmail's quota is not a failure at all
+ *
+ * Everything above is about the LAST attempt. A Gmail quota refusal is dealt
+ * with on the first, and for every kind of job alike: the same message goes
+ * back onto the queue it came from, to be handled when the account's minute
+ * is over, and one line is logged at `info`.
+ *
+ * Being refused is how plMail learns what its quota is (see GmailQuotaPacer),
+ * so on a project with a low limit it is certain to happen during an import
+ * — and to whatever else is using the same mailbox's quota at that moment. A
+ * label change pushed to Gmail while an import was running was refused, and
+ * Messenger's own retry did the right thing in the loudest way it has: a
+ * WARNING carrying two stack traces, per attempt (#90).
+ *
+ * The retry that replaces it is not counted against the job's own ladder,
+ * because nothing about the job went wrong. It is bounded all the same:
+ * after QUOTA_WAITS of them in a row the refusal is handed to the ladder like
+ * any other failure, and from there to everything above.
  */
 final readonly class GiveUpQuietlyOnThrottling implements MiddlewareInterface
 {
@@ -79,10 +99,23 @@ final readonly class GiveUpQuietlyOnThrottling implements MiddlewareInterface
     /** How many times, before it is allowed to fail in earnest. */
     public const int MAX_DEFERRALS = 10;
 
+    /**
+     * How many times in a row a job waits out Gmail's quota before the
+     * refusal counts as a failure. An hour of minutes: a quota that is still
+     * exhausted then is being spent by something this application cannot see.
+     */
+    public const int QUOTA_WAITS = 60;
+
+    /**
+     * @param ContainerInterface $transports every transport by name, to put a
+     *                                       job back where it came from
+     */
     public function __construct(
         #[Autowire(service: 'messenger.transport.ingest')]
         private SenderInterface $ingest,
         private LoggerInterface $logger,
+        #[Autowire(service: 'messenger.receiver_locator')]
+        private ContainerInterface $transports,
     ) {
     }
 
@@ -91,6 +124,10 @@ final readonly class GiveUpQuietlyOnThrottling implements MiddlewareInterface
         try {
             return $stack->next()->handle($envelope, $stack);
         } catch (HandlerFailedException $failure) {
+            if (true === $this->waitsOutGmailQuota($envelope, $failure)) {
+                return $envelope;
+            }
+
             if (false === $this->isLastAttempt($envelope) || false === $this->isThrottling($failure)) {
                 throw $failure;
             }
@@ -138,6 +175,72 @@ final readonly class GiveUpQuietlyOnThrottling implements MiddlewareInterface
         }
     }
 
+    /**
+     * Put a job Gmail refused on quota back onto its own queue, to run when
+     * the refusal has cleared.
+     *
+     * @return bool whether that was done; false leaves the failure to
+     *              everything else this class and Messenger do with one
+     */
+    private function waitsOutGmailQuota(Envelope $envelope, HandlerFailedException $failure): bool
+    {
+        $received = $envelope->last(ReceivedStamp::class);
+        $wrapped  = $failure->getWrappedExceptions();
+
+        // One handled inline has no queue to go back to and fails to its
+        // caller, as before.
+        if (null === $received || [] === $wrapped) {
+            return false;
+        }
+
+        $delay = 0;
+
+        foreach ($wrapped as $exception) {
+            if (false === $exception instanceof GmailThrottledException) {
+                return false;
+            }
+
+            $delay = max($delay, $exception->getRetryDelay());
+        }
+
+        $stamp = $envelope->last(ThrottleDeferralStamp::class);
+        $waits = null === $stamp ? 0 : $stamp->count;
+
+        if ($waits >= self::QUOTA_WAITS || false === $this->transports->has($received->getTransportName())) {
+            return false;
+        }
+
+        $transport = $this->transports->get($received->getTransportName());
+
+        if (false === $transport instanceof SenderInterface) {
+            return false;
+        }
+
+        $message = $envelope->getMessage();
+
+        // A few seconds past the refusal's end, and not the same few for
+        // every job: the ones refused together would otherwise all come due
+        // in the same second and be refused together again.
+        $delay += 1000 * random_int(1, 10);
+
+        // A NEW envelope around the same message. The job's own retries are
+        // for its own failures and this was not one.
+        $transport->send(new Envelope($message, [
+            new DelayStamp($delay),
+            new ThrottleDeferralStamp($waits + 1),
+        ]));
+
+        $this->logger->info('Put back: waiting for Gmail quota', [
+            'class'        => $message::class,
+            'queue'        => $received->getTransportName(),
+            'delaySeconds' => intdiv($delay, 1000),
+            'wait'         => $waits + 1,
+            'of'           => self::QUOTA_WAITS,
+        ]);
+
+        return true;
+    }
+
     /**
      * Only a job a worker took off a transport has a retry ladder to be at the
      * end of. One handled inline, in the process that dispatched it, has no

tests/Infrastructure/Messaging/GiveUpQuietlyOnThrottlingTest.php

@@ -5,6 +5,8 @@ declare(strict_types=1);
 namespace App\Tests\Infrastructure\Messaging;
 
 use App\Domain\Exception\GmailThrottledException;
+use App\Domain\Exception\GraphThrottledException;
+use App\Infrastructure\Messaging\Message\ApplyGmailLabelsMessage;
 use App\Infrastructure\Messaging\Message\SyncAccountMessage;
 use App\Infrastructure\Messaging\Message\SyncGmailMessageBatchMessage;
 use App\Infrastructure\Messaging\Middleware\GiveUpQuietlyOnThrottling;
@@ -12,6 +14,7 @@ use App\Infrastructure\Messaging\Stamp\ThrottleDeferralStamp;
 use PHPUnit\Framework\TestCase;
 use Psr\Log\NullLogger;
 use RuntimeException;
+use Symfony\Component\DependencyInjection\ServiceLocator;
 use Symfony\Component\Messenger\Envelope;
 use Symfony\Component\Messenger\Exception\HandlerFailedException;
 use Symfony\Component\Messenger\Middleware\MiddlewareInterface;
@@ -33,10 +36,12 @@ use Throwable;
 final class GiveUpQuietlyOnThrottlingTest extends TestCase
 {
     private InMemoryTransport $ingest;
+    private InMemoryTransport $export;
 
     protected function setUp(): void
     {
         $this->ingest = new InMemoryTransport();
+        $this->export = new InMemoryTransport();
     }
 
     /**
@@ -91,6 +96,54 @@ final class GiveUpQuietlyOnThrottlingTest extends TestCase
         }
     }
 
+    /**
+     * #90: a label change pushed to Gmail while an import had the quota was
+     * refused, and Messenger retried it — correctly, and with a WARNING and
+     * two stack traces each time. A quota refusal is not a failure: the job
+     * goes back where it came from, on its first attempt, and says so once.
+     */
+    public function testAJobGmailRefusedOnQuotaGoesBackOntoItsOwnQueueAtOnce(): void
+    {
+        $message  = new ApplyGmailLabelsMessage(7, ['m1'], ['STARRED'], []);
+        $envelope = new Envelope($message, [new ReceivedStamp('export')]);
+
+        $returned = $this->handle($envelope, $this->refusedByGmail());
+
+        self::assertSame($envelope, $returned, 'returned means acknowledged: no retry, no warning');
+        self::assertSame([], $this->ingest->getSent(), 'back where it came from, not onto the queue mail arrives on');
+
+        $sent = $this->export->getSent();
+
+        self::assertCount(1, $sent);
+        self::assertSame($message, $sent[0]->getMessage());
+        self::assertSame(0, RedeliveryStamp::getRetryCountFromEnvelope($sent[0]), 'its own retries are untouched');
+        self::assertSame(1, $sent[0]->last(ThrottleDeferralStamp::class)?->count);
+
+        $delay = $sent[0]->last(DelayStamp::class)?->getDelay();
+
+        self::assertGreaterThan(45_000, $delay);
+        self::assertLessThanOrEqual(55_000, $delay);
+    }
+
+    public function testAQuotaThatNeverClearsIsAFailureInTheEnd(): void
+    {
+        $envelope = (new Envelope(new ApplyGmailLabelsMessage(7, ['m1'], ['STARRED'], []), [new ReceivedStamp('export')]))
+            ->with(new ThrottleDeferralStamp(GiveUpQuietlyOnThrottling::QUOTA_WAITS));
+
+        $this->expectException(HandlerFailedException::class);
+
+        $this->handle($envelope, $this->refusedByGmail());
+    }
+
+    public function testAGmailRefusalBesideAnotherFailureIsStillAFailure(): void
+    {
+        $envelope = new Envelope(new ApplyGmailLabelsMessage(7, ['m1'], ['STARRED'], []), [new ReceivedStamp('export')]);
+
+        $this->expectException(HandlerFailedException::class);
+
+        $this->through($envelope, new HandlerFailedException($envelope, [$this->refusedByGmail(), new RuntimeException('and this')]));
+    }
+
     public function testAnEarlierAttemptIsLeftToTheRetryLadder(): void
     {
         $this->expectException(HandlerFailedException::class);
@@ -149,7 +202,12 @@ final class GiveUpQuietlyOnThrottlingTest extends TestCase
             }
         };
 
-        return new GiveUpQuietlyOnThrottling($this->ingest, new NullLogger())
+        $transports = new ServiceLocator([
+            'ingest' => fn (): InMemoryTransport => $this->ingest,
+            'export' => fn (): InMemoryTransport => $this->export,
+        ]);
+
+        return new GiveUpQuietlyOnThrottling($this->ingest, new NullLogger(), $transports)
             ->handle($envelope, new StackMiddleware($handler));
     }
 
@@ -158,8 +216,17 @@ final class GiveUpQuietlyOnThrottlingTest extends TestCase
         return new Envelope($message, [new ReceivedStamp('ingest'), new RedeliveryStamp($retries)]);
     }
 
-    private function throttled(): GmailThrottledException
+    /**
+     * Microsoft's, in the tests of the last attempt: a Gmail refusal never
+     * gets that far any more.
+     */
+    private function throttled(): GraphThrottledException
     {
-        return new GmailThrottledException('Gmail labels.list failed with 403 (rateLimitExceeded)', 403, 'rateLimitExceeded');
+        return new GraphThrottledException('Graph answered 429 (TooManyRequests)');
+    }
+
+    private function refusedByGmail(): GmailThrottledException
+    {
+        return new GmailThrottledException('Gmail messages.batchModify failed with 403 (rateLimitExceeded)', 403, 'rateLimitExceeded', 45);
     }
 }