Fetch Gmail messages ten to a request, not a minute's worth in a second

On a project allowed 6,000 units a minute an import fetched one batch,
was refused, waited a minute and was refused on the next batch: the
indicator read "paused" beside "about 1,005 of 6,000 units". Fifty
messages in one batch request are 1,000 units in one second, and Google
does not let a per-minute limit be spent that way (#90).

  requests     ten messages each, the pacer asked before every one;
               a refused request stops the rest of its batch, which is
               handed back unfetched
  the reserve  five seconds of allowance instead of twenty, so the
               pacer no longer lets a burst through that Google will
               refuse
  the line     while paused it shows what had been counted when Google
               refused, not a running count that has fallen to nothing

That a burst is what Google refused is inferred from the figures, not
documented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CHANGELOG.md

@@ -14,7 +14,8 @@ of fetching a message from 5 units to 20; plMail paced imports for 15,000 at the
 a new project most of what an import asked for was refused, the log filled with rate-limit
 warnings and an import could stop without resuming. plMail now reads the limit out of Google's
 first refusal and keeps to it — nothing to configure — counts a message at its real cost,
-fetches fifty at a time, and shares one allowance between its workers. A refusal is no longer an
+fetches ten to a request so a minute's worth is never asked for in one second, and shares one
+allowance between its workers. A refusal is no longer an
 error: whatever met it — a batch, an import page, a sync, a label change being pushed — waits
 out the minute and carries on. The
 import line in the background-work indicator now shows the quota, roughly how much of it is in

docs/de/internals/mail-ingest.md

@@ -1,4 +1,4 @@
-<!-- translated-from: internals/mail-ingest.md sha1:7329297e0b7553b6e0fb3e23948afdcff155a609 -->
+<!-- translated-from: internals/mail-ingest.md sha1:e6630b7c2f371b937d8dd3a6ff7318bbfb74a57d -->
 # Mail-Ingest
 
 Vom Anbieter in die Datenbank: wie aus Bytes eine Zeile `Message` wird, was mit ihr geschieht,
@@ -96,8 +96,10 @@ die Vorgeschichte gehört `App\Service\Mail\MailImporter`, auf der Warteschlange
 - **Gmails Kontingent wird gelernt, nicht eingestellt.** Google erlaubt einem Nutzer eine Zahl
   von Kontingent-Einheiten pro Minute und sagt erst, wie viele, wenn es eine Anfrage ablehnt.
   `GmailQuotaPacer` beginnt bei 15.000, liest das Limit aus der ersten Ablehnung (ein seit Mai
-  2026 angelegtes Projekt hat 6.000) und taktet von da an jedes Konto auf vier Fünftel davon;
-  ein gelerntes Limit wird nach einem Tag vergessen, damit ein angehobenes wiedergefunden wird.
+  2026 angelegtes Projekt hat 6.000) und taktet von da an jedes Konto auf drei Fünftel davon,
+  zehn Nachrichten pro Anfrage — Google lehnt ab, was den Vorrat einer Minute in einer Sekunde
+  verbraucht; ein gelerntes Limit wird nach einem Tag vergessen, damit ein angehobenes
+  wiedergefunden wird.
   Was ein Konto verbraucht hat und wann es zuletzt abgelehnt wurde, liegt in einem Cache-Pool
   in der Datenbank, sodass alle Worker aus einem Kontingent schöpfen. Eine Ablehnung ist auf
   diesem Weg nirgends ein Fehler: Ein Stapel, eine Import-Seite oder ein Sync, der auf eine

docs/internals/mail-ingest.md

@@ -85,8 +85,9 @@ The two are separate now. A sync arranges for new mail to be seen and stops; the
 - **Gmail's quota is learnt, not configured.** Google allows one user a number of quota units
   a minute and does not say how many until it refuses a request. `GmailQuotaPacer` starts from
   15,000, reads the limit out of the first refusal (a project created since May 2026 has
-  6,000) and paces every account to four fifths of it from then on; a learnt limit is forgotten
-  after a day so a raised one is found again. What an account has spent and when it was last
+  6,000) and paces every account to three fifths of it from then on, ten messages to a request
+  — Google refuses a minute's worth spent in one second; a learnt limit is forgotten after a
+  day so a raised one is found again. What an account has spent and when it was last
   refused are in a database-backed cache pool, so every worker draws on one allowance. A
   refusal is not an error anywhere on this path: a batch, an import page or a sync that meets
   one is put back to run when the account's minute is over.

src/Domain/DTO/Mail/GmailQuotaStatus.php

@@ -25,7 +25,8 @@ final readonly class GmailQuotaStatus
      * @param bool     $learnt        whether Google named that limit, or it is
      *                                the one assumed until it does
      * @param int      $used          roughly what the bulk fetching spent in
-     *                                the minute just gone
+     *                                the minute just gone — or, while paused,
+     *                                in the minute before Google refused
      * @param int      $pausedSeconds how long Google's last refusal still
      *                                holds the account; 0 when it does not
      * @param int|null $minutesLeft   how long the rest of the import takes at

src/Service/Gmail/GmailQuotaPacer.php

@@ -51,7 +51,7 @@ use Symfony\Component\DependencyInjection\Attribute\Autowire;
  * it. Starting from the top again costs one refused batch a day on a project
  * that has less, and that batch is put back like any other.
  *
- * WELL UNDER the limit — the worst minute here is four fifths of it. The rest
+ * WELL UNDER the limit — the worst minute here is about two thirds of it. The rest
  * belongs to what is not counted: the person using the mailbox (a send is
  * 100, a label change on a selection 50), push-driven syncs, and whatever
  * else is signed in to the same Google account.
@@ -95,8 +95,17 @@ final class GmailQuotaPacer
     /** The share of the limit that is the steady allowance. */
     private const float STEADY_SHARE = 0.6;
 
-    /** How far ahead of the allowance a burst may run: a fifth of the limit. */
-    private const float RESERVE_SECONDS = 20.0;
+    /**
+     * How far ahead of the allowance a burst may run: five seconds of it,
+     * which is one request of ten messages and a listing page beside it.
+     *
+     * It was twenty, a fifth of the limit, on the reading that a limit of so
+     * much a minute may be spent however the minute likes. Google does not
+     * see it that way — a thousand units in one second was refused on a
+     * project allowed six thousand a minute — so the reserve is only what
+     * lets a request go out without waiting when the account has been idle.
+     */
+    private const float RESERVE_SECONDS = 5.0;
 
     /**
      * How long an account is left alone after a refusal that came with no
@@ -191,6 +200,14 @@ final class GmailQuotaPacer
         $state = $this->read($item->get());
 
         $state['refusedUntil'] = max($state['refusedUntil'], $now + $cooldown);
+        // What had been spent when Google said no. Kept because the running
+        // count falls to nothing during the wait that follows, and "paused,
+        // 0 of 6,000 used" explains nothing; see usedWhenRefused().
+        $state['usedWhenRefused'] = array_sum(array_filter(
+            $this->sinceAMinuteAgo($state['spent'], $now),
+            static fn (int $slot): bool => $slot <= $now,
+            ARRAY_FILTER_USE_KEY,
+        ));
         // The reserve is gone as well: whatever was thought to be left of this
         // minute, Google has just said there is none.
         $state['paidOffAt'] = max($state['paidOffAt'], $now + self::RESERVE_SECONDS);
@@ -252,6 +269,19 @@ final class GmailQuotaPacer
         ));
     }
 
+    /**
+     * What this application had counted for the account in the minute before
+     * Google last refused it.
+     *
+     * Shown while the account is paused, and the honest measure of how far
+     * the count is from Google's: a refusal at a figure well under the limit
+     * is quota spent by something this pacer does not see.
+     */
+    public function usedWhenRefused(Account $account): int
+    {
+        return $this->read($this->state->getItem($this->accountKey($account))->get())['usedWhenRefused'];
+    }
+
     /** Whether the limit is one Google named, rather than the one assumed until it does. */
     public function isLimitLearnt(): bool
     {
@@ -280,7 +310,7 @@ final class GmailQuotaPacer
     }
 
     /**
-     * @return array{paidOffAt: float, refusedUntil: float, spent: array<int, int>}
+     * @return array{paidOffAt: float, refusedUntil: float, usedWhenRefused: int, spent: array<int, int>}
      *         the moment the account's spending so far is paid off at the
      *         steady rate, and the moment a refusal stops applying — both in
      *         the past for an account that has been idle — and what was spent
@@ -291,9 +321,10 @@ final class GmailQuotaPacer
         $spent = true === is_array($stored) && true === is_array($stored['spent'] ?? null) ? $stored['spent'] : [];
 
         return [
-            'paidOffAt'    => (float) (true === is_array($stored) ? ($stored['paidOffAt'] ?? 0.0) : 0.0),
-            'refusedUntil' => (float) (true === is_array($stored) ? ($stored['refusedUntil'] ?? 0.0) : 0.0),
-            'spent'        => array_map(intval(...), $spent),
+            'paidOffAt'       => (float) (true === is_array($stored) ? ($stored['paidOffAt'] ?? 0.0) : 0.0),
+            'refusedUntil'    => (float) (true === is_array($stored) ? ($stored['refusedUntil'] ?? 0.0) : 0.0),
+            'usedWhenRefused' => (int) (true === is_array($stored) ? ($stored['usedWhenRefused'] ?? 0) : 0),
+            'spent'           => array_map(intval(...), $spent),
         ];
     }
 

src/Service/Mail/GmailApiClient.php

@@ -41,6 +41,12 @@ final class GmailApiClient
      */
     private const int IDS_PER_BATCH = 1000;
 
+    /**
+     * How many messages.get go into one batch request. See getMessages() for
+     * why it is not the fifty a caller hands over.
+     */
+    private const int MESSAGES_PER_REQUEST = 10;
+
     private const array TRANSIENT_REASONS = [
         'rateLimitExceeded',
         'userRateLimitExceeded',
@@ -136,8 +142,8 @@ final class GmailApiClient
     /**
      * Fetch multiple messages (format=full) using the Gmail Batch API.
      *
-     * Packs up to 50 individual messages.get sub-requests into a single
-     * multipart/mixed HTTP POST. This avoids hammering the per-user per-second
+     * Packs individual messages.get sub-requests into multipart/mixed HTTP
+     * POSTs, a few to each. This avoids hammering the per-user per-second
      * quota with individual concurrent requests and dramatically reduces
      * round-trips for large initial syncs.
      *
@@ -161,16 +167,52 @@ final class GmailApiClient
      */
     public function getMessages(Account $account, array $messageIds): array
     {
-        if (count($messageIds) === 0) {
-            return [
-                'payloads'  => [],
-                'retryable' => [],
-                'gone'      => [],
-            ];
+        $fetched = [
+            'payloads'  => [],
+            'retryable' => [],
+            'gone'      => [],
+        ];
+
+        // A FEW AT A TIME, with the pacer asked before each few.
+        //
+        // The limit is "units a minute", but Google does not let a minute's
+        // worth be spent in a second: fifty messages in one request are a
+        // thousand units at once, ten seconds' worth on a project allowed
+        // 6,000 a minute, and Google refused part of every such batch — one
+        // batch, a refusal, a minute's wait, and the next batch refused the
+        // same way (#90). Ten messages are two seconds' worth there, which is
+        // a burst Google lets through, and the pacer's wait between requests
+        // is what keeps the rate under the limit.
+        $chunks = array_chunk(array_values($messageIds), self::MESSAGES_PER_REQUEST);
+
+        foreach ($chunks as $index => $chunk) {
+            $part = $this->getMessagesInOneRequest($account, $chunk);
+
+            $fetched['payloads'] += $part['payloads'];
+            $fetched['retryable'] = [...$fetched['retryable'], ...$part['retryable']];
+            $fetched['gone']      = [...$fetched['gone'], ...$part['gone']];
+
+            if (true === $part['refused']) {
+                // Not asked for at all: what is left would only be refused
+                // too, and counted against the minute all the same.
+                foreach (array_slice($chunks, $index + 1) as $rest) {
+                    $fetched['retryable'] = [...$fetched['retryable'], ...$rest];
+                }
+
+                break;
+            }
         }
 
-        // The one call here that can spend a minute's quota on its own, so it
-        // is the one that is paced. See GmailQuotaPacer.
+        return $fetched;
+    }
+
+    /**
+     * @param list<string> $messageIds at most MESSAGES_PER_REQUEST
+     *
+     * @return array{payloads: array<string,array<string,mixed>>, retryable: list<string>, gone: list<string>, refused: bool}
+     */
+    private function getMessagesInOneRequest(Account $account, array $messageIds): array
+    {
         $this->pacer->spend($account, GmailQuotaPacer::UNITS_PER_MESSAGE * count($messageIds));
 
         $token    = $this->tokenManager->getValidAccessToken($account);
@@ -196,6 +238,7 @@ final class GmailApiClient
                 'payloads'  => [],
                 'retryable' => array_values($messageIds),
                 'gone'      => [],
+                'refused'   => true,
             ];
         } catch (HttpException $e) {
             throw new \RuntimeException(
@@ -240,10 +283,10 @@ final class GmailApiClient
             'payloads'  => $payloads,
             'retryable' => $retryable,
             'gone'      => $gone,
+            'refused'   => null !== $parsed['refusal'],
         ];
     }
 
-
     /**
      * Fetch a single message in full format.
      *

src/Service/Mail/ImportProgress.php

@@ -129,11 +129,16 @@ final readonly class ImportProgress
         $total = $account->importTotal;
         $left  = null === $total ? null : max(0, $total - $done);
 
+        $paused = $this->pacer->waitSeconds($account);
+
         return new GmailQuotaStatus(
             $this->pacer->limit(),
             $this->pacer->isLimitLearnt(),
-            $this->pacer->usedThisMinute($account),
-            $this->pacer->waitSeconds($account),
+            // While Google holds the account nothing is being spent, and the
+            // running count falls to nothing; what was spent when it said no
+            // is the figure that explains the pause.
+            $paused > 0 ? $this->pacer->usedWhenRefused($account) : $this->pacer->usedThisMinute($account),
+            $paused,
             null === $left || 0 === $left ? null : (int) ceil($left / $this->pacer->messagesPerMinute()),
         );
     }

src/Service/Mail/MailImporter.php

@@ -245,6 +245,7 @@ final readonly class MailImporter
             'accountId'    => $account->id,
             'delaySeconds' => $seconds,
             'limit'        => $this->pacer->limit(),
+            'countedWhenRefused' => $this->pacer->usedWhenRefused($account),
         ]);
     }
 

templates/mail/_jobs_indicator.html.twig

@@ -165,7 +165,7 @@ the record and a missed nudge costs one stale render, never a wrong answer.
                                    the limit is only assumed is the end of the
                                    sentence, and the part that was cut off. #}
                                 <p class="mt-0.5 text-[11px] text-ink-faint" data-import-quota>
-                                    {{ (line.quota.learnt ? 'jobs.import.quota' : 'jobs.import.quota_assumed')|trans({
+                                    {{ (line.quota.isPaused() ? 'jobs.import.quota_refused' : (line.quota.learnt ? 'jobs.import.quota' : 'jobs.import.quota_assumed'))|trans({
                                         '%used%': line.quota.used|number_format,
                                         '%limit%': line.quota.limit|number_format,
                                     }) }}

tests/Service/Gmail/GmailQuotaPacerTest.php

@@ -22,8 +22,8 @@ use Symfony\Component\Clock\MockClock;
  */
 final class GmailQuotaPacerTest extends TestCase
 {
-    /** A full batch: fifty messages.get at twenty units. */
-    private const int BATCH = 1000;
+    /** One request: ten messages.get at twenty units. */
+    private const int BATCH = 200;
 
     public function testAFewBatchesInARowAreNotHeld(): void
     {
@@ -46,7 +46,7 @@ final class GmailQuotaPacerTest extends TestCase
 
         $worst = $this->worstMinute($pacer, $clock, $this->account(1));
 
-        self::assertLessThanOrEqual(13_000, $worst);
+        self::assertLessThanOrEqual(10_000, $worst);
         // And not so cautious that an import crawls: the steady rate is used.
         self::assertGreaterThanOrEqual(9_000, $worst);
     }
@@ -71,7 +71,7 @@ final class GmailQuotaPacerTest extends TestCase
 
         $worst = $this->worstMinute($pacer, $clock, $account);
 
-        self::assertLessThanOrEqual(6000, $worst);
+        self::assertLessThanOrEqual(4200, $worst);
         self::assertGreaterThanOrEqual(3600, $worst);
     }
 
@@ -187,17 +187,17 @@ final class GmailQuotaPacerTest extends TestCase
         $pacer->spend($account, self::BATCH);
         $pacer->spend($account, self::BATCH);
 
-        self::assertSame(2000, $pacer->usedThisMinute($account));
+        self::assertSame(400, $pacer->usedThisMinute($account));
         self::assertSame(0, $pacer->usedThisMinute($this->account(2)));
 
         $clock->sleep(30);
         $pacer->spend($account, self::BATCH);
 
-        self::assertSame(3000, $pacer->usedThisMinute($account));
+        self::assertSame(600, $pacer->usedThisMinute($account));
 
         $clock->sleep(45);
 
-        self::assertSame(1000, $pacer->usedThisMinute($account));
+        self::assertSame(200, $pacer->usedThisMinute($account));
 
         $clock->sleep(60);
 
@@ -214,16 +214,37 @@ final class GmailQuotaPacerTest extends TestCase
         $pacer->refused($account, 'defaultPerMinutePerUser', 6000);
         $clock->sleep(120);
 
-        for ($i = 0; $i < 40; $i++) {
+        for ($i = 0; $i < 100; $i++) {
             $pacer->spend($account, self::BATCH);
         }
 
-        self::assertLessThanOrEqual(6000, $pacer->usedThisMinute($account));
+        self::assertLessThanOrEqual(4200, $pacer->usedThisMinute($account));
         self::assertGreaterThanOrEqual(3000, $pacer->usedThisMinute($account));
         self::assertTrue($pacer->isLimitLearnt());
         self::assertSame(180, $pacer->messagesPerMinute());
     }
 
+    /**
+     * During the wait after a refusal nothing is spent and the running count
+     * falls to nothing. What had been counted when Google said no is kept,
+     * because that is the figure the pause is explained by.
+     */
+    public function testWhatHadBeenSpentWhenGoogleRefusedIsRemembered(): void
+    {
+        $clock   = new MockClock('2026-10-08 14:30:00');
+        $pacer   = new GmailQuotaPacer($clock);
+        $account = $this->account(1);
+
+        $pacer->spend($account, self::BATCH);
+        $pacer->spend($account, self::BATCH);
+        $pacer->refused($account, 'defaultPerMinutePerUser', 6000);
+
+        $clock->sleep(70);
+
+        self::assertSame(0, $pacer->usedThisMinute($account));
+        self::assertSame(400, $pacer->usedWhenRefused($account));
+    }
+
     public function testOneAccountImportingDoesNotHoldAnother(): void
     {
         $clock = new MockClock('2026-10-08 14:30:00');
@@ -268,7 +289,7 @@ final class GmailQuotaPacerTest extends TestCase
         /** @var list<float> $spentAt */
         $spentAt = [];
 
-        for ($i = 0; $i < 100; $i++) {
+        for ($i = 0; $i < 300; $i++) {
             $pacer->spend($account, self::BATCH);
             $spentAt[] = (float) $clock->now()->format('U.u');
         }

tests/Service/Mail/GmailApiClientFailureTest.php

@@ -445,6 +445,35 @@ final class GmailApiClientFailureTest extends TestCase
         self::assertGreaterThan(0, $pacer->waitSeconds($this->account()));
     }
 
+    /**
+     * Fifty messages in one request are a thousand units in one second, which
+     * Google refuses on a project allowed 6,000 a minute. They go out ten to
+     * a request, and when one request is refused the rest are not sent.
+     */
+    public function testABatchGoesOutAFewMessagesAtATimeAndStopsAtARefusal(): void
+    {
+        $requests = 0;
+        $refusal  = json_encode($this->perUserQuotaBody(6000), JSON_THROW_ON_ERROR);
+
+        $http = new MockHttpClient(function () use (&$requests, $refusal): MockResponse {
+            ++$requests;
+
+            return 2 === $requests
+                ? new MockResponse($refusal, ['http_code' => 403])
+                : new MockResponse("--batch_x\r\n--batch_x--", ['http_code' => 200]);
+        });
+
+        $tokenManager = $this->createStub(OAuthTokenManager::class);
+        $tokenManager->method('getValidAccessToken')->willReturn('test-token');
+
+        $ids     = array_map(static fn (int $n): string => 'm' . $n, range(1, 35));
+        $fetched = new GmailApiClient($http, $tokenManager, new GmailQuotaPacer(new MockClock()))
+            ->getMessages($this->account(), $ids);
+
+        self::assertSame(2, $requests, 'the third and fourth requests are never made');
+        self::assertSame($ids, $fetched['retryable'], 'every id is handed back: none fetched, none lost');
+    }
+
     /** The whole request refused is the same answer as every part refused. */
     public function testAWhollyRefusedBatchIsHandedBackRatherThanThrown(): void
     {

translations/messages.de.yaml

@@ -2190,6 +2190,7 @@ jobs:
         quota: "Gmail-Kontingent: etwa %used% von %limit% Einheiten in dieser Minute"
         quota_assumed: "Gmail-Kontingent: etwa %used% von %limit% Einheiten in dieser Minute (Limit angenommen, bis Google es nennt)"
         quota_paused: "Pausiert durch Gmails Kontingent · weiter in %seconds% s"
+        quota_refused: "Gmail hat bei etwa %used% von %limit% Einheiten pro Minute abgelehnt"
         remaining_hours: "Noch etwa %hours% Std. %minutes% Min."
         remaining_minutes: "Noch etwa %minutes% Min."
         waiting: "Wartet darauf, weiterzumachen"

translations/messages.en.yaml

@@ -2189,6 +2189,7 @@ jobs:
         quota: "Gmail quota: about %used% of %limit% units this minute"
         quota_assumed: "Gmail quota: about %used% of %limit% units this minute (limit assumed until Google names it)"
         quota_paused: "Paused by Gmail's quota · continues in %seconds% s"
+        quota_refused: "Gmail said no at about %used% of %limit% units a minute"
         remaining_hours: "About %hours% h %minutes% min left"
         remaining_minutes: "About %minutes% min left"
         waiting: "Waiting to continue"

translations/messages.en_PI.yaml

@@ -2069,6 +2069,7 @@ jobs:
         quota: "Gmail's ration: about %used% o' %limit% units this minute"
         quota_assumed: "Gmail's ration: about %used% o' %limit% units this minute (a guess, till Google names the limit)"
         quota_paused: "Becalmed by Gmail's ration · under way in %seconds% s"
+        quota_refused: "Gmail said nay at about %used% o' %limit% units a minute"
         remaining_hours: "About %hours% h %minutes% min o' haulin' left"
         remaining_minutes: "About %minutes% min o' haulin' left"
         waiting: "Waitin' fer the tide"