Deploy runner with multiple label with gha-runner-scale-set?

[tbirk-bhs]

Hello,

I'm trying to migrate from the legacy ARC to the new gha-runner-scale-set. With the legacy ARC I had a RunnerDeployment and added multiple labels. e.g.

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
...
spec:
  template:
    spec:
      ...
      labels:
        - ubuntu-22.04
        - ubuntu-latest

Is it also possible to add multiple runner labels to a gha-runner-scale-set?

Thanks for helping me!

Timon

[Link-]

Labels are not supported and there are no plans to support them. It's already discussed in our documentation:

Here: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/deploying-runner-scale-sets-with-actions-runner-controller#about-runner-scale-sets

and here: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/about-actions-runner-controller#using-arc-runners-in-a-workflow

Runner scale sets cannot be comprised of heterogeneous types of runners (different OSs, specs etc.). As such, each deployment should have a specific runner configuration and a unique name.

If you want different types of runners, you need to configure different runner scale sets.

[thesuperzapper]

I have raised an issue saying that all self-hosted runners should be getting the self-hosted label in addition to whatever the custom label is:

• [CRITICAL] self-hosted label is missing from all scale-set (AutoscalingRunnerSet) runners actions/actions-runner-controller#3330

Please promote this issue within your teams and to your github account managers!

---

Just so we are clear, for scaling purposes we can just ignore the self-hosted label, so this should not be technically challenging, because EVERY runner should just be assumed to have that label.

This is a critical security issue, as selecting self-hosted in addition to your custom label(s) prevents accidentally running on public runners (if GitHub decides to create a public runner label with the same name as your internal ones). In fact, the official GitHub docs suggest selecting self-hosted runners using runs_on: [self-hosted, MY_LABEL] for this very reason:

All self-hosted runners have the self-hosted label. Using only this label will select any self-hosted runner. To select runners that meet certain criteria, such as operating system or architecture, we recommend providing an array of labels that begins with self-hosted (this must be listed first) and then includes additional labels as needed. When you specify an array of labels, jobs will be queued on runners that have all the labels that you specify.

[danmanners]

I'll second everything @thesuperzapper called out here. Cross referencing this discussion topic from @tmehlinger.

[thesuperzapper]

I have tagged the GitHub leadership in actions/actions-runner-controller#3330 (comment).

Please show your support by liking that comment, if you agree that the GHA team needs to resolve this.

[dave-schmitz1]

As Runner Scale Sets appear to require unique names, we can't use the same names in both blue and green clusters. And by not being able to assign both of them the same label, we would have to change runs-on for all workflows every time we spin up a new Runner Scale Set. This is of course not viable.

For this specific use case, you can. Check High availability and automatic failover in our docs.

@Link- If I'm following your guidance right and I have a blue / green cluster for dev and blue / green cluster for prod you saying I would make this operational by having a 4 specific runners and runners called runner-prod or runner-dev along with removing runs-on group for all workflows and only use the label to direct workflows?

[naikrovek]

Actions Runner Controller just wasn't thought out well enough to be used in production.

This limitation is silly. If you want people to know how long their job has to wait in the queue before a runner takes it, then show that. I know that info is in the application somewhere, because jobs go into a queue and get worked on mostly in order. What you should not have done is to come up with an entirely new runner system which moves the queue out of GHES entirely and limits users in so many other ways. The tradeoffs made take so much away and offer almost nothing in return.

I am still salty today after reading about this months ago. There's not even a security boundary between ARC runner containers in DinD mode. They all have the host's Docker socket mapped in. What is the point of this mode (and KinD mode, I assume) at all if any running container can easily reach into any other container on the same host?

ARC just isn't production ready. GitHub themselves don't even use it, to my knowledge.

What kinds of other goofy decisions are GitHub going to make in the future?

You wrote GHES and GitHub.com. If you want to show queue depth, then show it. If you can't, then that's a hole you dug yourself, so fill it in and dig the correct hole this time. "lol nah you only get one label and it must be unique and it can't be descriptive. You're welcome."

[nebuk89]

The reason we opted for single labels on ARC (and are doing the same thing on the larger hosted runners as well) is to get us ready to improve the Dev UX about 'why isn't my job starting'.

When we had multiple labels, doing deterministic assignment of jobs to a 'scale set' wasn't possible. The ven diagram of runners that matched labels vs. the labels I was looking for meant that jobs always just waited until 'something was free' and 'where' something would start was less clear. This conflicted with a reasonably high request we had to 'see the queue of jobs and when mine will start'.
I note that some people mitigated this by adding 'lots and lots' of labels to make a run on target very specific – which also isn't great for new users.

To mitigate this, we want to move to a world of more deterministic assignment of jobs to a scale set that takes jobs in a FIFO fashion.

We made the decision that the way to approach this was to move to single named scale sets/queues as a target. That way when a job is assigned it 'goes to its queue and waits' rather than sitting in some meta queue of all jobs for all potential machines.

I do appreciate that this results in a change in workflow files :( this is why we haven't “done away” with multiple labels at a system level yet.

What we don't really want to do, if we can help it, is to create an experience that is good for our hosted machines to review their queue but sucks for self-hosted as no one moves off multi label.

Which lastly led to the decision – what if we used new ARC as a bit of a forcing factor for this to help gently nudge people to the future if people want to come over.

[thesuperzapper]

I have tagged the GitHub leadership in actions/actions-runner-controller#3330 (comment).

Please show your support by liking that comment, if you agree that the GHA team needs to resolve this.

[jeffmccune]

Note

Summary request: previously, actions supported highly available runners. Scale sets support HA only for enterprise plan customers. Please consider restoring the ability to configure highly available runners without an enterprise feature gate.

"Highly Available" in this context is a workflow executing on any one of multiple available regional clusters globally.

Cross posting from #3351 as requested.

Thanks for taking the time to answer this. You're correct about my two questions. The only comment I have is that I'm a bit frustrated because with scale sets, groups are required to achieve the same behavior that was previously supported with labels. My frustration is that groups are a paid enterprise feature but labels are not, so this feels like a step backwards.

I'm a paying customer, but some of the github orgs I work with are not, so they cannot use groups and as a result cannot deploy highly available scale sets. Please consider adding back some mechanism to have a workflow execute on any available cluster without requiring a paid feature like was possible previously. Thanks again for your time responding to the question.

[crohr]

What an insane decision.

This.

We are now looking (admittedly not very intensively) for an alternative solution. ARC with no labels sucks, it's that plain simple. I'm sure there has to be some self-hosted software that'll allow to launch build runner pods with github used only to fire up webhooks on certain actions on repositories. It'll be a little more hassle to deal with repository secrets etc., though.

Anyone knows of a good alternative to GHA that can be self-hosted and is k8s-friendly?

@shapirus I'm not sure it has to be k8s friendly, since efficiently launching bare ephemeral VMs is all that's needed, but I believe you might be well served by RunsOn, which is entirely self-hosted in your AWS account.

[shapirus]

Anyone knows of a good alternative to GHA that can be self-hosted and is k8s-friendly?

@shapirus I'm not sure it has to be k8s friendly, since efficiently launching bare ephemeral VMs is all that's needed, but I believe you might be well served by RunsOn, which is entirely self-hosted in your AWS account.

Thanks. This looks interesting.

[duaneking]

This was an insane decision made by people that are clearly not technical enough to be making these kind of decisions.

[nebuk89]

👋 I don't want to dupe out too much and @tmehlinger had put quite a full and detailed single breakdown here https://github.com/actions/actions-runner-controller/discussions/3340 so I have added some updates there following more discussions with @juliandunn and other folks :)

[mbarany]

I am trying to use CodeQL with self-hosted runners. The documentation over here on GitHub's help page says to add a label to the runner, but it seems like ARC does not support adding additional labels.

Would I have to create a scale-set just for CodeQL with a runnerScaleSetName value of code-scanning?
If so, that seems a bit silly.

[luis-lf]

Thats the only way I was able to make it work for dependabot and code-scanning

[stasostrovskyi]

Alternatively, you can have your own workflows for codeql and dependabot and run them on the same runners as regular builds. And having them as shared workflows can simplify rollout across multiple repositories.

[lifeofmoo]

Gah - only 2 weeks ago I finally got over the hurdle that the git cli was finally added to the runner container and now this 🤯

Not being able to even use the self-hosted label is painful

[naikrovek]

Yeah. I still don't understand this decision. If GitHub want to make it so that people can have an idea of when their job will run, which is why they say they made this decision, then show the queue depth of the queue the job is in. It seems very simple to me. Instead they are changing how all runners operate, so that they each get only one label so that queues are simpler and have names... this seems like a lot of work for them when they could simply show a job's position in a queue and let the jobs have labels as we're used to them. I'm going to have to get a job at GitHub in order to truly understand why this decision was made, I think.

[pawelros]

There is not even an option in GitHub org settings to forbid using public runners. With such option, going live with default runner group and this no-label absurd would be easier to swallow.

[ChrisOwen273]

The decision here seems to contradict guidance GitHub give for other parts of their product suite.

See Configuring self-hosted runners for code scanning in your enterprise

Enabling GHAS Code scanning on self hosted runners requires you to set a label called "self-hosted".

Between this thread and that document am I to take it that you cannot use Code scanning on self hosted runners when using ARC?

This seems a bit short sighted. And before anyone suggests it we are moving to EMU and removing access to public runners through that.

[stasostrovskyi]

You can run "advanced" code scanning workflow for codeql and choose runners like you would for any other workflow. Yes, it requires adding one more file to a repo, which is a downside, but it's still doable.

[hicksjacobp]

Enabling GHAS Code scanning on self hosted runners requires you to set a label called "self-hosted".

This doesn't apply to ARC runners.

You can register your runners with the code-scanning label, and it should work just fine, which is what your linked documentation states:

If you are using default setup for code scanning, assign the code-scanning label to your self-hosted runner.

The same goes for running Dependabot on self-hosted runners with the dependabot label (and you've opted-in to running Dependabot on self-hosted runners in your repository's Code security settings).

[mumoshu]

Hey everyone!
FWIW, from the perspective of the legacy ARC maintainer, I still don't understand the real issue.

I'm not a GitHubber but the new ARC looks designed well with a proper constraint on the number of runner labels being 1, so that the scheduling is easy and the runners are more scalable, thanks to the out-of-the-box support for "sub-controller sharding" via the listener per runnerscaleset architecture.

In terms of infrastructure cost(I saw this is mentioned in relation to lack of multiple runner labels), listener pods cannot be scaled to zero, but are relatively small, and you can still scale runner pods to zero with the new ARC.

Honestly speaking, I guess there could be a "potential" enhancement to the new ARC where you can "consolidate" listener processes into one pod so that you can run multiple RunnerScaleSets with less footprint(mainly memory usage across the cluster), so that it looks like there's a controller pod and a listener pod in the cluster and nothing else, where they look like the equivalents to the legacy ARC's controller and webhookserver, respectively. Still, no need to have multiple labels per runnerscaleset.

Put another way, listener-per-runnerscaleset is actually a killer feature to me, because you no longer need to deploy multiple ARC instances to scale horizontally.
If two or more runnerscalesets were able to have multiple overlapping runner labels, you no longer know which listener received/processed the job without looking into the logs. That's unnecessary nondeterminism to me.

Could everyone please read my lengthy comment at actions/actions-runner-controller#3598 (comment) and share your thoughts?

I see requests for more queue visibility mentioned in various threads related to the runner label count limitation, so let me leave my opinion for that too- I would love it if GitHub Actions added more visibility to queue statuses and lengths. But again, multiple labels to runnersets- that looks unnecessary to me.

[0x416e746f6e]

for me the problem is not so much about heterogeneous labels for runner sets, but rather about not having no control whatsoever on what labels are assigned to them.

the new ARC is rigidly opinionated about choosing what single label would it add to the runner, which is not really user friendly.

I don't want dozens of weird labels on my runner. I want just: self-hosted, and runner-set-abcd - just be be able to clearly indicate in the workflow file that it is: (a) a self-hosted workflow, and (b) going to be run in the env "abcd".

whatever random label the new ARC is giving to the set is not good enough.

[0x416e746f6e]

so, to get this straight:

• the issue was raised some 20 months ago
• rejected on the premise of "we are not going to do this b/c we want to improve the devex w.r.t. job queue management, predictability and visibility"
• the premise is still hanging and the issue is still blocked

so, we are basically having the worst of the both worlds, right? or am I missing something?

[daversomethingsomething]

125 👎 and counting on the "answer" at this point..

[duaneking]

I just want to call out that there's an absolute conflict In both the documentation and the UI because what's happening here is the docs are saying that label management is supported and then it just flat out acts like it's not the case for large runners but you only ever find that one single line saying that in the wrong spot deeply hidden; And the reasoning is insane Because there is no reasoning given It's just something critical that doesn't work.

It's absolutely insane that we can't manage our labels because the group of runners is defined by the query and perspective that we look at them; using labels allows us to consider the same pool of resources to be larger and allows us to get our work done faster because now a set of 3 runners can be multiple distinct groups that depend on the labels used.

If a CPU has multiple chipset support, or a runner can do different things, and I need specific things as part of the build for testing , builds, etc, then that allowed me to set up the same runner to be Part of multiple different pools for actions that responds more flexibly, so it honestly just feels to me like GitHub is nerfing runners to intentionally make them less useful because you want us to spend more money on more runners instead of actually letting us just get our work done with what we currently have, Which has incidentally also been the result of all of this because now people are required to spit up more runners and throw them explicitly into groups that can't be shared ... so it feels like technical debt that's creating technical inflation and creating bigger invoices but not actually helping developers, imho.

[daversomethingsomething]

Ok, only one label. But why does that label have to be locked to the name of the resources in K8s? K8s resources have different syntax constraints than GitHub labels.

I'm trying to use a perfectly valid GitHub Runner label 'linux-x86_64', for instance, same as I did for my non-ARC self-hosted runner.

Error: UPGRADE FAILED: failed to create resource: ServiceAccount "linux-x86_64-gha-rs-no-permission" is invalid: metadata.name: Invalid value: "linux-x86_64-gha-rs-no-permission": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

This is especially problematic in the context of multi-platform reusable workflows. All consumers are required to name their scale sets identically or provide workflow input overrides. Previously all we needed was "just tag your runners with this generic/free label". Instead they need to design their infrastructure around this label. Which leads directly into the next arbitrary limitation of only 10 precious inputs on workflow_dispatch events...

There's very good value in using labels that are distinct from the name of the runner for non-ARC self-hosted runners, and ARC nukes that value. Enterprise use requires a higher level of sophistication and flexibility, not just lowest common denominator.

GitHub is missing the mark here.