Run a Test Suite in GitHub CI

A Diffie test suite is a collection of tests you've already authored in your workspace. This guide wires that suite into GitHub Actions so it runs on every pull request, posts a pass/fail check, and links to the dashboard for screenshots and traces when something breaks.

If you're looking for on-demand tests generated from a PR diff (reviewer comments /diffie test, Diffie writes a new test for the change), use Testing with a PR comment instead. The two are complementary: most teams run both.

How it works

The integration is two API calls wrapped in a shell script:

1. Start the run. Tell Diffie which suite to execute and what base URL to test against.
2. Wait for results. Poll the run status every 10 seconds until it finishes.

If all tests pass, the workflow exits with code 0 and the PR gets a green check. If any test fails, the workflow exits with code 1 and the check turns red. The script prints a link to the Diffie dashboard run so you can jump straight to screenshots, traces, and network logs for the failure.

Before you start

You need two things from your Diffie workspace:

• API token. Generate one at app.diffie.ai/settings/api-tokens. This becomes DIFFIE_TOKEN.
• Suite ID. Open the suite you want to run from the Diffie dashboard and copy the ID from the URL or the settings panel. This becomes DIFFIE_SUITE_ID.

Step 1: Add secrets to GitHub

Don't hardcode tokens in workflow files. Store them as repo secrets instead:

1. Open the repository on GitHub.
2. Go to Settings → Secrets and variables → Actions.
3. Click New repository secret and add each of:

• DIFFIE_TOKEN: paste your API token.
• DIFFIE_SUITE_ID: paste the suite ID.

Step 2: Add the workflow file

Create .github/workflows/diffie.yml:

name: Diffie Tests

on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  diffie-tests:
    name: diffie-e2e-tests
    runs-on: ubuntu-latest

    steps:
      - name: Run Diffie Test Suite
        env:
          DIFFIE_TOKEN: ${{ secrets.DIFFIE_TOKEN }}
          DIFFIE_SUITE_ID: ${{ secrets.DIFFIE_SUITE_ID }}
          PREVIEW_URL: https://your-app.example.com
        run: |
          # Start the suite run
          RESPONSE=$(curl -s -X POST \
            "https://api.diffie.ai/ci/suites/$DIFFIE_SUITE_ID/execute" \
            -H "Authorization: Bearer $DIFFIE_TOKEN" \
            -H "Content-Type: application/json" \
            -d "{\"baseUrl\": \"$PREVIEW_URL\"}")

          RUN_ID=$(echo $RESPONSE | jq -r '.suiteRunId')
          RUN_URL=$(echo $RESPONSE | jq -r '.url')

          if [ "$RUN_ID" = "null" ] || [ -z "$RUN_ID" ]; then
            echo "Failed to start suite run"
            echo "$RESPONSE"
            exit 1
          fi

          echo "Suite run started: $RUN_ID"
          echo "View results: $RUN_URL"

          # Poll until the run finishes
          while true; do
            STATUS_RESPONSE=$(curl -s \
              "https://api.diffie.ai/ci/suite-runs/$RUN_ID" \
              -H "Authorization: Bearer $DIFFIE_TOKEN")

            STATUS=$(echo $STATUS_RESPONSE | jq -r '.status')
            PASSED=$(echo $STATUS_RESPONSE | jq -r '.passed_tests')
            TOTAL=$(echo $STATUS_RESPONSE | jq -r '.total_tests')

            echo "Status: $STATUS ($PASSED/$TOTAL passed)"

            if [ "$STATUS" = "passed" ]; then
              echo "All tests passed!"
              exit 0
            elif [ "$STATUS" = "failed" ]; then
              echo "Tests failed! Details: $RUN_URL"
              exit 1
            elif [ "$STATUS" = "cancelled" ]; then
              echo "Run was cancelled"
              exit 1
            fi

            sleep 10
          done

Commit and push. On the next pull request, GitHub Actions runs this workflow and the suite executes.

What the script does

1. Sends a POST to Diffie with the suite ID and the base URL under test. Diffie replies with a suiteRunId and a link to the live run page.
2. Enters a polling loop, checking status every 10 seconds. Diffie returns one of four statuses: running, passed, failed, or cancelled.
3. Exits with code 0 (success) or 1 (failure) when the run finishes. GitHub uses the exit code to mark the PR check.

Using dynamic preview URLs

The example above hardcodes PREVIEW_URL, which works fine if every PR tests against the same environment. If your host gives each PR its own preview deployment (Vercel, Netlify, Cloudflare Pages, Fly Preview), grab that URL dynamically so tests always hit the actual code in the PR.

Vercel example:

steps:
  - name: Wait for Vercel preview
    uses: patrickedqvist/[email protected]
    id: vercel
    with:
      token: ${{ secrets.GITHUB_TOKEN }}
      max_timeout: 300

  - name: Run Diffie Test Suite
    env:
      DIFFIE_TOKEN: ${{ secrets.DIFFIE_TOKEN }}
      DIFFIE_SUITE_ID: ${{ secrets.DIFFIE_SUITE_ID }}
      PREVIEW_URL: ${{ steps.vercel.outputs.url }}
    run: |
      # Same script as Step 2

The wait step blocks until Vercel finishes deploying and exposes the preview URL, which the Diffie step then consumes as PREVIEW_URL. Netlify and Cloudflare have equivalent actions; the pattern is identical.

API quick reference

Start a run

POST https://api.diffie.ai/ci/suites/{suiteId}/execute
Authorization: Bearer <token>

Body: { "baseUrl": "https://your-app.example.com" }

Response: { "suiteRunId": "abc-123", "url": "https://app.diffie.ai/runs/abc-123" }

Check run status

GET https://api.diffie.ai/ci/suite-runs/{runId}
Authorization: Bearer <token>

Response: { "status": "passed", "passed_tests": 12, "failed_tests": 0, "total_tests": 12 }

That's the full API surface for the suite-in-CI flow: two endpoints, one token.

Running on other CI systems

Because the integration is plain curl and jq, it runs on any CI that executes a shell step: GitLab CI, CircleCI, Jenkins, Bitbucket Pipelines, Buildkite, Drone. Paste the same script into the platform's equivalent of a run: step, map the two secrets to the platform's secret store, and the setup is identical.

See the integration guides for platform-specific snippets.

What the workflow looks like after this

1. A teammate opens a pull request.
2. GitHub Actions triggers the Diffie workflow.
3. Diffie runs your suite against the preview URL in a real cloud browser.
4. Results appear as a PR check: green for pass, red for fail.
5. On failure, click through to the Diffie dashboard for screenshots, traces, and error details for each failing test.

No local browser setup, no flaky infrastructure to maintain, and no test glue code in your repo. Tests live in Diffie; the workflow just points at them.