Github Actions: Dispatching a job matrix from an environment variable

Sometimes you want to dynamically generate some jobs in a Github Action. The matrix pattern is a powerful tool for that. But the way you can interact with variables are limited, you can not use environment variables directly to define the job matrix.

Motivation ¶

A use case where this can be relevant is building or deploying services with different configurations, for different tenants etc. For example, imagine you have an application that polls events from different queues. For a proper isolation you want to deploy one polling service for each queue. In different environments (e.g. dev/staging/prod) there exist different queues (different names, properties and number of queues). We can define the job matrix for the list of services (for simplicity we just pass the service names here, in reality we might want to use more complex objects with other information and use more than one matrix dimension):

name: Continuous Delivery

on:
  push:
    branches:
      - main

jobs:
  deploy-service:
    runs-on: ubuntu-latest
    name: "Update service"
    strategy:
      matrix:
        service: [abc-service, def-service, xyz-service]
    steps:
      - name: Deploy service
        run: echo "Deploying service ${{ matrix.service }} ..."
      # Deployment code ...

But how can we pass different services for different environments?

Github lets us define environments in the repository settings and set environment variables that we can use in the actions. We create an environment “dev” and define a variable SERVICES (with properly escaped strings):

[\"abc-service\", \"def-service\", \"xyz-service\"]

We reference the environment: dev and can use the env var in a job:

...
jobs:
  echo-services:
    runs-on: ubuntu-latest
    environment: dev
    steps:
      - name: Echo SERVICES
        id: echo-services
        run: echo "services=${{ vars.SERVICES }}"

Problem ¶

We cannot use the env var to define the matrix! E.g. trying

...
jobs:
  deploy-service:
    runs-on: ubuntu-latest
    environment: dev
    name: "Update service"
    strategy:
      matrix:
        service: ${{ fromJson(vars.SERVICES) }}
    steps:
      - name: Deploy service
        run: echo "Deploying service ${{ matrix.service }} ..."
      # Deployment code ...

… yields an error:

Error when evaluating 'strategy' for job 'deploy-service'. .github/workflows/deploy.yml (Line: 23, Col: 18): Error from function 'fromJson': empty input, .github/workflows/deploy.yml (Line: 23, Col: 18): Unexpected value ''

This is a Github Action limitation: Env vars are “available to the steps of all jobs in the workflow” (source), but not outside of steps.

Solution ¶

Here is a pattern that I found works well, inspired by this stackoverflow answer.

In the Github yaml file, define a helper job dispatch-deployments that copies the environment variable to the job output. While the matrix strategy can not reference environment variables, it can reference job outputs! We reference the job another job depends on with needs.

name: Continuous Delivery

on:
  push:
    branches:
      - main

jobs:
  dispatch-deployments:
    runs-on: ubuntu-latest
    environment: dev
    outputs:
      services: ${{ steps.set-services.outputs.services }}
    steps:
      - name: Extract SERVICES
        id: set-services
        run: echo "services=${{ vars.SERVICES }}" >> $GITHUB_OUTPUT

  deploy-service:
    runs-on: ubuntu-latest
    environment: dev
    name: "Update service"
    needs: dispatch-deployments
    strategy:
      matrix:
        service: ${{ fromJson(needs.dispatch-deployments.outputs.services) }}
    steps:
      - name: Deploy service
        run: echo "Deploying service ${{ matrix.service }} ..."
      # Deployment code ...

Voilà, the jobs are spawned based on the values in the environment variable:

Thinking beyond env vars, the dispatch helper job could be expanded to perform service discovery and more.