How We're Preventing Breaking Changes in GraphQL APIs at Buffer — and Why It's Essential for Our Customers

We’ve all experienced it at some point — a change is deployed to an API and suddenly, clients stop working.

User experience deteriorates, negative reviews start coming in, customer advocacy starts dealing with requests, multiple engineers start digging into the issue, and it quickly becomes all hands on deck.

Not only does this lose the trust of our users and interrupt their workflows, but it also costs an organization a lot of time and money to resolve these issues.

One way to prevent all of this from ever happening is to detect these breaking changes before they are merged into our repository at a Pull Request stage.

This way, we can prevent such changes from ever being merged, avoiding a breaking experience for our clients and reducing downtime for our users. 

As part of our commitment to transparency and building in public, I’m going to share how we’re doing this for our own GraphQL API via the use of GitHub Actions.

When it comes to detecting breaking changes, we can detect these by taking the schema representation on the branch of the pull request and comparing it with the schema representation on the main branch. We can then use the result of this diff to determine whether breaking changes exist in our schema changes.

When it comes to this workflow, we can break this down into several steps:

• Generate schema for the current branch: This will give us a schema that represents the changes we have made in our pull request
• Generate schema for the main branch: This will give us a schema that represents our current production API
• Perform verification of the current branch schema against the main branch schema: This will tell us what changes exist in our schema comparison and if any of them will break clients
• Post the result to the Pull Request: This will allow us to ‘fail’ the pull request to prevent it from being merged, along with alerting the author of the breaking changes

With this in mind, we’re going to build out an automated workflow that will run these operations for any Pull Request in our repository. For these pull request checks, we’re using GitHub Actions, but most of the following code will work for whatever CI setup you are using.

Note: We won’t be diving too much into the concepts of GitHub Actions here. If you are not familiar with Actions, I suggest following the quickstart tutorial.

Setting up the workflow

We’re going to start by setting up a new GitHub Action, we’ll create a new file named breaking_change_check.yml and start by giving our Action a name.

name: Schema Change Verification

Next, we’ll want to specify when this action is going to be run — this will essentially allow us to define when we want to perform the checks in the PR. 

We’ll not only want to do this on opened events (when the PR is initially opened), but also if it is reopened or synchronized — which will allow the checks the re-run if there are additional commits pushed to the pull request. 

This ensures that we are always checking for breaking changes on the latest commits pushed to the branch.

name: Schema Change Verification:  
on:
  pull_request:    
    types:      
      - opened      
      - reopened      
      - synchronize

We’ll also only want to run these checks when .graphql schema files are changed, so we’ll specify this rule using the paths property.

name: Schema Change Verification
on:
  pull_request:
    types:
      - opened
      - reopened
      - synchronize
    paths:
      - 'graphql/*/**.graphql'

graphql/*/**.graphql is the path where our GraphQl schema files are located; you will need to adjust this according to your project.

Generating the current branch schema

Now that we have the foundations of our action configured, we can move on to defining the jobs that will be responsible for generating and verifying our schema.

We’ll start here by defining a new job generateChangedSchema and defining the use of the ubuntu-latest runner.

name: Schema Change Verification
...

jobs:
  generateChangedSchema:
    runs-on: [ubuntu-latest]

Next, we’ll need to perform a couple of setup operations for our job. We’ll want to start by checking out the repository at the branch of our PR, for which we’ll use the checkout action.

Our action is also going to utilize node, so we’ll need to install this using the setup-node action.

name: Schema Change Verification
...

jobs:
  generateChangedSchema:
    runs-on: [ubuntu-latest]
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v3
        with:
          node-version: '18'

At this point, we’re now ready to move on to the generation of our schema. For this, we’re going to need to load our schema files and then merge them into a single schema file. This makes the verification process much simpler as we only need to work with a single file instead of multiple.

To make this process easier, we’re going to utilize a couple of dependencies from graphql-tools. We can see that these are named according to what we need them for, we just need to add them to our package.json file as a dev-dependency.

"@graphql-tools/load-files": "7.0.0",
"@graphql-tools/merge": "9.0.1"

With these dependencies in place, we’re now going to write a small script that will load and merge all of the .graphql files in a given directory.

import * as fs from 'fs'
import { loadFilesSync } from '@graphql-tools/load-files'
import { mergeTypeDefs } from '@graphql-tools/merge'
import { print } from 'graphql/language/printer'

const mergeFiles = async (): Promise<void> => {
  try {
    // using the provided path, load the types from the schema files
    const typesArray = loadFilesSync(`../../graphql/src`, {
      extensions: ['graphql'],
    })
 
    // merge all of the types from the recieved schemas, compressing all of the types     // into a single place
    const result = mergeTypeDefs(typesArray)
    // write the schema to a single file to be used for diffing
    await fs.promises.writeFile('generated/schema.graphql', print(result))
  } catch (e) {
    console.error("We've thrown! Whoops!", e)
  }
}

;(async (): Promise<void> => {
  try {
    // the merged schema file will be created in the generated directory, so create
    // the directory if it does not yet exist
    if (!fs.existsSync('generated')) {
      fs.mkdirSync('generated')
    }
    await mergeFiles()
  } catch (e) {
    console.error("We've thrown an error! Whoops!", e)
  }
})()

We’ll then add a new [task] to our package.json file so that we can easily execute this with the required arguments. This script takes a single argument when being executed, which is the path for the merged schema to be saved. 

If you need to provide paths for the location of schema files, this can be done through additional arguments. Our schemas are located in a single directory, so we hardcode this inside of the script itself.

"graph:generateSchema": "ts-node scripts/generateSchema.ts"

With this in place, we can now execute this command from our generateChangedSchema job. For this, we’ll use a bash script step where we’ll need to navigate to the directory where the generateSchema command exists and then use the node command to execute it.

name: Schema Change Verification
...

jobs:
  generateChangedSchema:
    runs-on: [ubuntu-latest]
    steps:
      ...
      - name: Generate Schema
        run: |
          cd services/api-gateway
          node graph:generateSchema

At this point, we will have a merged schema file that contains the contents of all our schema files. To wrap up this job we’re going to attach this schema file to our workflow run, this is so that we can download that file for use within the next job in our workflow. 

While this could all be done in a single job, your actions can be kept far more organized if work is broken down into smaller chunks. Here, we’ll use the upload-artifact action and attach the schema file from the path that we saved it to, assigning it a name of branch-schema for referencing it later.

name: Schema Change Verification
...

jobs:
  generateChangedSchema:
    runs-on: [ubuntu-latest]
    steps:
      ...
      - name: Attach schema
        uses: actions/upload-artifact@v1
        with:
          name: branch-schema
          path: services/api-gateway/generated/schema.graphql

At this point, we have created a merged schema file that represents all of the schemas in our project and attached this single file to our workflow, meaning that the step for generating the schema representation for the current branch is now complete.

Verify schema changes

Now that we have the schema for our branch generated, we’re going to want to verify this against the schema representation on the main branch. 

We’ll start here by setting up the second job in our workflow, performSchemaVerification. We’ll use the needs property to declare that this job will wait for the generateChangedSchema to complete successfully before running.

name: Schema Change Verification
...

jobs:
  generateChangedSchema:
    ...
  performSchemaVerification:
    runs-on: [ubuntu-latest]
    needs: generateChangedSchema

Similar to the previous job, we’ll configure some foundations for this job by checking out the repository and configuring node. The only difference here is that when triggering the checkout action we’ll pass a ref property with the value of main. 

This is because we need to generate the schema for the main branch for verification, so we need the main branch to be the current branch when checking out the repository.

name: Schema Change Verification
...

jobs:
  generateChangedSchema:
    ...
  performSchemaVerification:
    runs-on: [ubuntu-latest]
    needs: generateChangedSchema
    steps:
      - uses: actions/checkout@v2
        with:
          ref: main
      - uses: actions/setup-node@v3
        with:
          node-version: '18'

Before we can verify schemas, we’ll need to go ahead and download the schema that we generated in the last job. 

For this we can use the download-artifact action, providing the name reference for the file that we want to download (which we previously defined as branch-schema), followed by the path that the schema should be downloaded to.

name: Schema Change Verification
...

jobs:
  generateChangedSchema:
    ...
  performSchemaVerification:
    runs-on: [ubuntu-latest]
    needs: generateChangedSchema
    steps:
      ...
      - name: Download branch schema
        uses: actions/download-artifact@v1
        with:
          name: branch-schema
          path: services/api-gateway/branch-schema

And then so that we have a schema representation for our main branch to compare this to, we’ll go ahead and execute our generateSchema command. This will do the same as before, except this time, we will have a single schema file that represents our main branch instead of the branch for our pull request.

name: Schema Change Verification
...

jobs:
  generateChangedSchema:
    ...
  performSchemaVerification:
    runs-on: [ubuntu-latest]
    needs: generateChangedSchema
    steps:
      ...
      - name: Generate main schema
        run: |
          cd services/api-gateway
          node  graph:generateSchema

At this point, we have the two schema files that we need for the verification step. When it comes to the verification step, we’re going to utilize graphql-inspector. 

This tool contains a verification process that allows you to diff two schemas and returns you a result of the changes in that diff. We’ll start here by adding these as dev-dependencies to our project.

"@graphql-inspector/ci": "^4.0.2",
"@graphql-inspector/diff-command": "^4.0.2"

Next, we’ll add another command to our package.json file that we will use to execute this verification process. For this command we’ll use graphql-inspector and its diff command, for which we’ll need to provide two arguments. 

The first is the schema path for the main branch, which we just generated in this step. The second is the schema path for the PR branch, which we previously downloaded and saved to the branch-schema path. 

It’s important here that your main schema is passed as the first argument, as this represents your base schema, while the second represents the changes your branch is introducing.

"graph:verifySchema": "graphql-inspector diff generated/schema.graphql branch-schema/schema.graphql",

With this command in place, we’re now going to execute it in our step. We’ll need the result of this command so that we can depict the breaking change state, so we’ll save its output to a variable reference.

name: Schema Change Verification
...

jobs:
  generateChangedSchema:
    ...
  performSchemaVerification:
    runs-on: [ubuntu-latest]
    needs: generateChangedSchema
    steps:
      ...
      - name: Generate main schema
        run: |
          cd services/api-gateway
          node graph:generateSchema
          OUTPUT=$(node graph:verifySchema || true)

When it comes to the result of the diff operation, there is a lot of content that is output to the console. In the context of a PR comment, the author is only going to care about the information in the context of their changes. For this reason, we’re going to extract this information from the output so that we can use it for the PR comment.

The content that we wish to extract starts from “Detected N breaking changes,” so we’ll use this to extract the text from the diff output.

name: Schema Change Verification
...

jobs:
  generateChangedSchema:
    ...
  performSchemaVerification:
    runs-on: [ubuntu-latest]
    needs: generateChangedSchema
    steps:
      ...
      - name: Generate main schema
        run: |
          cd services/api-gateway
          node graph:generateSchema
          OUTPUT=$(node graph:verifySchema || true)
          // grab everything after the Detected string
          CONTENT=${OUTPUT#*Detected}
          // reapply the Detected string at the start of the contnet
          FORMATTED="Detected"$CONTENT
          // write the extracted content to an environment variable
          echo "PR_COMMENT<<EOF" >> $GITHUB_ENV
          echo "$FORMATTED" >> $GITHUB_ENV
          echo "EOF" >> $GITHUB_ENV

At this point we now have the message for the comment stored in an environment variable, this will look something like the following, depending on the result of the diff.

Detected the following changes (1) between schemas:
[log] ✖ Input field aNewType of type String! was added to input object type OrganizationIdInput
[error] Detected 1 breaking change

Now that we have this content, we’re going to want to publish it as a comment on the Pull Request. For this we’ll use the create-or-update-comment action, posting the content of the previously created environment variable.

name: Schema Change Verification
...

jobs:
  generateChangedSchema:
    ...
  performSchemaVerification:
    runs-on: [ubuntu-latest]
    needs: generateChangedSchema
    steps:
      ...
      - name: Create comment
        uses: peter-evans/create-or-update-comment@v1
        with:
          issue-number: ${{ github.event.pull_request.number }}
          body: ${{ env.PR_COMMENT }}
        if: ${{ !contains(env.PR_COMMENT, 'No changes detected') }}

Now that we have the status of our breaking change, we’re going to want to set the status of our PR check based on this. Here we’re going to add a new step to our job, but only run this step if the generated comment does not contain the ‘success’ label. 

This is because by default, the Action will have the success status, it will only be otherwise if we manually mark it as failed (or it fails for some other reason). When this is the case, we’ll utilise the github-script action to set the failed status for our check, along with a failure reason. 

This way, the check will fail within the Pull Request and the author will be unable to merge the Pull Request until the issue is resolved.

name: Schema Change Verification
...

jobs:
  generateChangedSchema:
    ...
  performSchemaVerification:
    steps:
      ...
      - name: Set Breaking Change status
        if: ${{ !contains(env.PR_COMMENT, 'success') }}
        uses: actions/github-script@v3
        with:
          script: |
            core.setFailed('Schema Breaking Changes detected')

Wrapping up

With all of the above in place, we will now be able to see the breaking change states published as comments on our Pull Request. For success states, engineers will be made aware of their schema changes, highlighting that no breaking changes were detected.

On the other hand, any breaking changes will be highlighted in the published comment.

When there is a breaking change, the check will be marked as a failure and the pull request will be unable to be merged.

Now that we have breaking change checks in place, engineers will be unable to merge changes that will break the client experience. This allows to reduce any downtime for our users, increasing the trust in our product and reducing business cost from incident management.

If you’re not already using breaking change checks in your CI, now is the time to get started! I’d love to hear about any learnings you have along the way to making this a part of your development workflow. Comment below or find me on LinkedIn.