Trigger AI E2E Tests with a PR Comment

Most CI testing setups run a fixed suite of tests on every PR. That works until you ship a new feature that nobody has tests for yet, and reviewers have to spin it up locally to sanity-check it. A comment-triggered workflow closes that gap: anyone on the team can ask for a test on demand, without writing one first.

This guide sets up Diffie QA so /diffie test comments work on every PR in your repo. Setup is three repo secrets and one workflow file.

If you use Claude Code, the Diffie plugin can generate this workflow for you. Just ask it to “set up Diffie PR testing for this repo” and skip straight to Step 4. The steps below cover the manual path if you'd rather wire it up by hand.

How it works

1. A reviewer comments /diffie test (optionally followed by a URL) on the pull request.
2. GitHub Actions picks up the comment and kicks off the Diffie QA workflow.
3. The workflow checks out the PR branch, reads the diff, generates a test for the user-facing change, and runs it against your preview URL in a real browser.
4. Diffie posts a PR comment with pass/fail status and a public link to the session recording, viewable by anyone with the link and no Diffie account needed.

Before you start

• A Diffie account and API token. Generate one at app.diffie.ai/settings/api-tokens.
• An Anthropic API key (from console.anthropic.com) or a Claude Max/Pro OAuth token (run claude setup-token locally). You only need one.
• A preview deployment URL for pull requests (Vercel, Netlify, Cloudflare Pages, Fly Preview, whatever your host calls it).

Step 1: Install the Diffie QA Bot app

Install the Diffie QA Bot GitHub App on the repository you want to test. This is what makes PR comments post under the Diffie identity instead of a generic github-actions[bot].

Step 2: Add your repo secrets

Go to Settings → Secrets and variables → Actions and add:

• DIFFIE_API_TOKEN: your Diffie API token.
• ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN: set exactly one. The workflow uses whichever is non-empty.

Step 3: Add the workflow file

Create .github/workflows/diffie-qa.yml:

name: Diffie QA
on:
  issue_comment:
    types: [created]

jobs:
  diffie-qa:
    if: |
      github.event.issue.pull_request &&
      contains(github.event.comment.body, '/diffie test')
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
      issues: write
      id-token: write
    steps:
      - name: Mint Diffie OIDC token
        id: oidc
        uses: actions/github-script@v7
        with:
          script: |
            const token = await core.getIDToken('diffie.ai')
            core.setSecret(token)
            core.setOutput('token', token)

      - name: Exchange OIDC for Diffie bot token
        id: diffie-bot
        run: |
          RESPONSE=$(curl -sS -X POST https://api.diffie.ai/gh/token \
            -H "Authorization: Bearer ${{ steps.oidc.outputs.token }}")
          TOKEN=$(echo "$RESPONSE" | jq -r '.token // empty')
          if [ -z "$TOKEN" ]; then
            echo "Failed to mint Diffie bot token: $RESPONSE"
            exit 1
          fi
          echo "::add-mask::$TOKEN"
          echo "token=$TOKEN" >> $GITHUB_OUTPUT

      - name: Get PR branch
        id: pr
        run: |
          PR_DATA=$(gh api repos/${{ github.repository }}/pulls/${{ github.event.issue.number }})
          echo "ref=$(echo $PR_DATA | jq -r .head.ref)" >> $GITHUB_OUTPUT
          echo "number=${{ github.event.issue.number }}" >> $GITHUB_OUTPUT
        env:
          GH_TOKEN: ${{ steps.diffie-bot.outputs.token }}

      - uses: actions/checkout@v4
        with:
          ref: ${{ steps.pr.outputs.ref }}
          fetch-depth: 0

      - name: Extract URL from comment
        id: comment
        run: |
          COMMENT="${{ github.event.comment.body }}"
          URL=$(echo "$COMMENT" | grep -oP '/diffie test \K(https?://\S+)' || echo "")
          echo "url=$URL" >> $GITHUB_OUTPUT

      # Add your preview deployment wait step here.
      # Vercel:  patrickedqvist/[email protected]
      # Netlify: jakepartusch/wait-for-netlify-action@v1

      - name: Set preview URL
        id: preview
        run: |
          if [ -n "${{ steps.comment.outputs.url }}" ]; then
            echo "url=${{ steps.comment.outputs.url }}" >> $GITHUB_OUTPUT
          else
            echo "url=" >> $GITHUB_OUTPUT
          fi

      - name: Setup Diffie credentials
        run: |
          mkdir -p ~/.diffie
          echo '{"apiToken":"${{ secrets.DIFFIE_API_TOKEN }}","apiUrl":"https://api.diffie.ai"}' > ~/.diffie/credentials.json

      - name: Run Diffie QA
        uses: anthropics/claude-code-action@v1
        with:
          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
          github_token: ${{ steps.diffie-bot.outputs.token }}
          prompt: |
            You are in PR testing mode (CI=true). Use the diffie-qa skill.
            PR number: ${{ steps.pr.outputs.number }}
            Preview URL: ${{ steps.preview.outputs.url }}
            Repo: ${{ github.repository }}
            Follow the "PR Testing Mode (Autonomous)" workflow in the skill.

Commit and push. The next time someone comments /diffie test on a PR, the workflow fires.

Step 4: Try it

Open any PR and comment /diffie test. Within a minute or two you'll see a reply from the Diffie QA Bot with a pass/fail status and a Watch test run link that works for anyone on the PR, logged in to Diffie or not.

Want to point at a specific URL? Pass it inline: /diffie test https://staging.example.com. This is handy when your preview URL doesn't match the default pattern the workflow resolves.

What Diffie actually does on your PR

When the workflow fires, Diffie's agent:

1. Reads the PR diff to understand what changed.
2. Decides what the user-visible change is (a new form field, a renamed button, a new page) and writes a natural-language test spec targeting it.
3. Runs the spec through Diffie's test generator, which explores the preview URL in a real browser and produces Playwright code.
4. Executes the generated test against your preview URL, records the session, and grades pass/fail.
5. Posts a PR comment summarizing the result with a public link to the recording.

Tests created by the PR workflow stay in your Diffie workspace, so you can re-run them from the dashboard or promote a particularly good one into a regression suite later.

When to use this vs. a regression suite

The /diffie test workflow is for per-PR, on-demand sanity checks of whatever just changed. It's fast to set up and generates tests no one has written yet. Use it for code-review acceleration, since reviewers get a recorded walk-through of the feature without leaving GitHub.

For every-PR regression coverage (running your existing suite of critical paths on every change) use the suite-based CI integration instead. See Run a Test Suite in GitHub CI for that setup. Most teams run both.