Keeping this issue open so that we can track on production.

Canary pod attempted to exceed 2GB of memory.

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
<--- Last few GCs --->
[1:0x5609758d6ca0] 99708641 ms: Mark-sweep 2034.7 (2068.3) -> 2032.3 (2053.8) MB, 742.3 / 0.0 ms  (+ 11.7 ms in 2 steps since start of marking, biggest step 6.0 ms, walltime since start of marking 765 ms) (average mu = 0.221, current mu = 0.261) allocatio[1:0x5609758d6ca0] 99710309 ms: Mark-sweep 2032.3 (2053.8) -> 2031.9 (2053.8) MB, 1473.2 / 0.0 ms  (average mu = 0.160, current mu = 0.117) allocation failure GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
    0: ExitFrame [pc: 0x560972624eb9]
    1: StubFrame [pc: 0x5609726970c6]
Security context: 0x040fa0d40921 <JSObject>
    2: updateDirectives [0x1d97732ecb89] [/dist/server.js:161185] [bytecode=0x13c1796bdeb1 offset=1058](this=0x37a09e8afc01 <Object map = 0x35e98440cb1>,0x0ad3f2cc1d21 <JSFunction prodCheckAndUpdateNode (sfi = 0x133ce79d0941)>,0x08990b0ca311 <Object map = 0x2277f5ff8161>)
    3: updateDirectives [0x3f8c1dcb6fd9] [/dist/s...

added scoped labels

mentioned in commit status@10704bcf

Trying to reduce restart memory to 1GB (a3bb24b4) to see if that helps. (currently 2GB)

mentioned in commit 902e149b

300M looks to be working so far. Will keep an eye on it.

mentioned in commit eff963de

closed