Keeping this issue open so that we can track on production.

Canary pod attempted to exceed 2GB of memory.

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
<--- Last few GCs --->
[1:0x5609758d6ca0] 99708641 ms: Mark-sweep 2034.7 (2068.3) -> 2032.3 (2053.8) MB, 742.3 / 0.0 ms  (+ 11.7 ms in 2 steps since start of marking, biggest step 6.0 ms, walltime since start of marking 765 ms) (average mu = 0.221, current mu = 0.261) allocatio[1:0x5609758d6ca0] 99710309 ms: Mark-sweep 2032.3 (2053.8) -> 2031.9 (2053.8) MB, 1473.2 / 0.0 ms  (average mu = 0.160, current mu = 0.117) allocation failure GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
    0: ExitFrame [pc: 0x560972624eb9]
    1: StubFrame [pc: 0x5609726970c6]
Security context: 0x040fa0d40921 <JSObject>
    2: updateDirectives [0x1d97732ecb89] [/dist/server.js:161185] [bytecode=0x13c1796bdeb1 offset=1058](this=0x37a09e8afc01 <Object map = 0x35e98440cb1>,0x0ad3f2cc1d21 <JSFunction prodCheckAndUpdateNode (sfi = 0x133ce79d0941)>,0x08990b0ca311 <Object map = 0x2277f5ff8161>)
    3: updateDirectives [0x3f8c1dcb6fd9] [/dist/s...

added scoped labels

mentioned in commit status@10704bcf

Trying to reduce restart memory to 1GB (a3bb24b4) to see if that helps. (currently 2GB)

mentioned in commit 902e149b