Record per-job peak memory and route jobs to learned size tiers

[Mearman]

Runner pods are sized by one request (2 CPU, 7 Gi) regardless of the job, and slot count is limited by that request, not by what jobs use. Observed use across running pods was a fraction of the request, so many more jobs would fit if each pod were sized to its job. Nothing records what a job actually needs, so the request cannot be lowered safely, and a smaller default would kill the jobs that do need it.

What is known. ARC creates the runner pod before a job is assigned to it, so a pod cannot be resized for the job it receives. Sizing therefore means several scale sets (small, default, large) with the job choosing one through its runs-on label; the role's scale-set profiles already take a sizing and a runs_on_label. The controller already knows which job each pod got: EphemeralRunner status carries jobRepositoryName, jobWorkflowRef, jobDisplayName and workflowRunId.

What is needed.

1. Measure a job's peak memory when it ends. Polling kubectl top (the autoscaler's current signal) misses spikes, so the better source is the cgroup's memory.peak read from a job-completed hook (ACTIONS_RUNNER_HOOK_JOB_COMPLETED) inside the runner pod. Unverified: that the runner image's cgroup exposes memory.peak and the hook runs with the permissions to read it.
2. Store it keyed by repository, workflow and job name. The runner pod must not hold a gist or cluster write credential, so the hook should emit the figure somewhere the platform already reads (the pod's log or the EphemeralRunner), and a platform component should aggregate it. That storage choice is the open design question.
3. Route a job to a tier. runner-fallback-action already picks the runner label per job, so it is the natural place to read the learned size and choose the label.
4. Handle under-sizing: a job killed for memory is retried one tier up and that tier is recorded, so a wrong guess costs one rerun.

Start with step 1 alone, and read the data for a representative period before building steps 2 to 4. Lowering the default request is a separate, smaller change that this data would justify.

Related: the pool is currently bounded by memory requests, see #30 for why replicated storage was deferred.

[Mearman]

Step 1 is now partly delivered without the in-pod hook: the autoscaler keeps each job's highest observed pod memory (sampled once per poll, so a lower bound) in its status ConfigMap under job-memory, keyed by repository, workflow ref and job name. Read it with kubectl get configmap autoscaler-status -n github-runner-platform -o jsonpath='{.data.job-memory}'. It is deployed once the autoscaler change releases and the fleet is redeployed, and it needs time to fill before it says anything.

What it decides. If observed peaks sit far below the 7 Gi request, lowering the default request is a small change that frees most of the capacity on its own, with no hook. If they sit close to it, or the sampling gaps matter, the accurate measurement is worth building, and that needs a decision that is not mine to make: the job-completed hook can only hand its figure to the platform through something the runner pod is allowed to write, and runner pods run untrusted CI code. The options are a narrowly scoped Kubernetes permission for runner pods (for example create on events in their own namespace, with a collector reading them), or no permission and a longer sampling record. Steps 2 to 4 follow whichever is chosen.

[Mearman]

Step 1 is done and running on the fleet in its accurate form. The runner image's job-completed hook reads the container cgroup's memory.peak and posts it as a JobMemoryPeak Event in the pod's namespace; the autoscaler reads the Events and keeps the maximum per repository, workflow ref and job id under the status ConfigMap's job-peak key, in MiB. The runner service account holds create on events in its own namespace and nothing else, checked with kubectl auth can-i. Read it with kubectl get configmap autoscaler-status -n github-runner-platform -o jsonpath='{.data.job-peak}'.

Verified on the live fleet: a real job from another repository ran on the new image and was recorded. The sampled record (job-memory) stays as a cross-check and can go once job-peak has a representative period of data.

Steps 3 and 4 (routing a job to a size tier, and retrying one tier up after an out-of-memory kill) need that data first: lowering the default request is the small change it is most likely to justify, and tiers only if the spread between jobs is wide. Read job-peak after a few days of CI traffic, compare each job's peak against the 7 Gi request, and decide from that.