Testing with Drape in GitHub Actions¶
This guide shows the canonical workflow pattern for running tests with Drape's suppression-aware CI gating and per-shard job attribution.
The key ideas:
- Uploads live inside the test job, not in a separate fan-in job. This makes the "Job" column in the Drape dashboard reflect the job that actually ran the tests.
- Your test runner runs with
continue-on-error: true, letting drape-cli be the authoritative exit code. This is how suppressed-only failures keep the shard check green. - Coverage fans in server-side via a
total-shardscount: each shard uploads independently, and the server merges them into a single snapshot. No artifact round-tripping.
Minimum viable pattern¶
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run tests
id: pytest
continue-on-error: true
run: pytest --junitxml=junit.xml --cov --cov-report=xml:coverage.xml
- name: Upload test results to Drape
if: always() && hashFiles('junit.xml') != ''
uses: drape-io/drape-action@v2
with:
command: tests
file: junit.xml
api-key: ${{ secrets.DRAPE_API_KEY }}
- name: Upload coverage to Drape
if: always() && hashFiles('coverage.xml') != ''
uses: drape-io/drape-action@v2
with:
command: coverage
file: coverage.xml
format: cobertura
api-key: ${{ secrets.DRAPE_API_KEY }}
- name: Fail job if pytest crashed without producing JUnit
if: steps.pytest.outcome == 'failure'
run: |
test -f junit.xml || { echo "pytest crashed — failing the job."; exit 1; }
Four steps: run, upload tests, upload coverage, crash-guard. The shape is the same for any test runner that writes JUnit XML.
Why continue-on-error: true?¶
Without it, pytest (and most test runners) exit non-zero on any failure — even failures that are already marked as suppressed in Drape. GitHub then marks the step red before drape-cli runs, so the PR check doesn't reflect suppression.
With continue-on-error: true, the test runner's exit is swallowed. drape-cli runs, reads the failure list from Drape, and exits:
- Exit 0 when every failure matches a suppression rule (shard check stays green).
- Non-zero when any unsuppressed failures exist (shard check goes red).
The crash-guard step at the end fails the job if the runner crashed before producing a JUnit — otherwise a segfault during test collection would silently pass.
Matrix / sharded test jobs¶
Each shard uploads its own results. Set job-name explicitly so the Drape dashboard shows per-shard attribution (otherwise all shards collapse under the matrix's base GITHUB_JOB name):
env:
SHARDS: 3
jobs:
test:
name: Test Python (${{ matrix.group }}/${{ env.SHARDS }})
strategy:
fail-fast: false
matrix:
group: [1, 2, 3]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run tests (shard ${{ matrix.group }}/${{ env.SHARDS }})
id: pytest
continue-on-error: true
env:
SPLIT_GROUP: ${{ matrix.group }}
run: |
pytest \
--splits "$SHARDS" --group "$SPLIT_GROUP" \
--junitxml=junit-${SPLIT_GROUP}.xml \
--cov --cov-report=xml:coverage-${SPLIT_GROUP}.xml
- name: Upload test results to Drape
if: always() && hashFiles(format('junit-{0}.xml', matrix.group)) != ''
uses: drape-io/drape-action@v2
with:
command: tests
file: junit-${{ matrix.group }}.xml
job-name: Test Python (${{ matrix.group }}/${{ env.SHARDS }})
api-key: ${{ secrets.DRAPE_API_KEY }}
- name: Upload coverage to Drape (batched)
if: always() && hashFiles(format('coverage-{0}.xml', matrix.group)) != ''
uses: drape-io/drape-action@v2
with:
command: coverage
file: coverage-${{ matrix.group }}.xml
format: cobertura
total-shards: ${{ env.SHARDS }} # tells server to wait for 3 shards
api-key: ${{ secrets.DRAPE_API_KEY }}
- name: Fail job if pytest crashed without producing JUnit
if: steps.pytest.outcome == 'failure'
env:
JUNIT_FILE: junit-${{ matrix.group }}.xml
run: |
test -f "$JUNIT_FILE" || { echo "pytest crashed — failing the job."; exit 1; }
total-shards: 3 tells the server: "wait for 3 shards before merging coverage into a single snapshot." The first 2 uploads sit pending; the 3rd triggers finalization. If a shard crashes, a 5-minute reaper finalizes with whatever arrived and flags the snapshot as partial.
Multiple independent test jobs sharing coverage¶
If you have multiple jobs producing coverage for the same logical snapshot (e.g., a test matrix and a separate test-github-api job), they share coverage via the same group. Compute total-shards dynamically in a changes or setup job so conditional skips don't force the reaper timeout:
jobs:
changes:
runs-on: ubuntu-latest
outputs:
python: ${{ steps.filter.outputs.python_any_changed }}
total_shards: ${{ steps.shards.outputs.count }}
steps:
- uses: tj-actions/changed-files@v47
id: filter
with:
files_yaml: |
python:
- 'src/**/*.py'
- id: shards
env:
PYTHON_CHANGED: ${{ steps.filter.outputs.python_any_changed }}
ACTOR: ${{ github.actor }}
run: |
count=0
if [ "$PYTHON_CHANGED" = "true" ]; then
count=$((count + 3)) # matrix shards
if [ "$ACTOR" != "dependabot[bot]" ]; then
count=$((count + 1)) # test-github-api
fi
fi
echo "count=$count" >> "$GITHUB_OUTPUT"
test:
needs: changes
# ... as above, with:
# total-shards: ${{ needs.changes.outputs.total_shards }}
test-github-api:
needs: changes
if: needs.changes.outputs.python == 'true' && github.actor != 'dependabot[bot]'
# ... same 4-step pattern, same group, same `total-shards` ...
All shards referencing ${{ needs.changes.outputs.total_shards }} agree on the count, so the server finalizes immediately after the last one arrives. No 5-minute wait.
Footguns¶
- Forgetting
continue-on-error: truesilently turns off suppression-gating. Every failure (including suppressed ones) will fail the shard check. Symptom: your suppression rules "don't work" in CI. - Globbing the crash-guard (
test -f test-results/*.xml) lets a stale XML from a prior workflow run bypass the guard. Use the exact filename withmatrix.groupinterpolated. - Hardcoding
total-shards: 3when some jobs are conditional. Every skipped-shard run will wait 5 minutes for the reaper before coverage appears on the dashboard. Computetotal-shardsdynamically. - Default
timeout: 120on batched coverage is too short when sibling shards have wildly different runtimes. The first shard to upload polls the batch until all siblings arrive AND the server merges them. If your slowest shard takes 5 minutes, settimeout: 600on the coverage step so the early shard's poll has enough headroom. - Omitting
job-namein matrix jobs. All shards collapse under the same name in the dashboard. Set it explicitly. - Running drape-cli in a fan-in job (downloading artifacts from test jobs). The uploading job's name ends up in the dashboard, not the test-running job's. The whole point of the in-job pattern is to avoid this.
Verifying it works¶
After your next PR run, the shard check (e.g. Test Python (1/3)) should:
- Go green when all failures are covered by suppression rules.
- Go red when any unsuppressed failure exists.
- Go red when the test runner crashed before writing JUnit.
In the Drape dashboard ("Top Slowest Tests"), the "Job" column should show Test Python (1/3), Test Python (2/3), Test Python (3/3) — not the upload step's name.
Other test runners¶
The same 4-step shape works for any runner with a JUnit-compatible output. Swap pytest for your runner; everything else stays the same:
- Go:
go test -json ./... | go-junit-report > junit.xml(and coverage via-coverprofile=cov.out; gocov convert cov.out | gocov-xml > coverage.xml). - Jest:
jest --ci --reporters=default --reporters=jest-junit+jest-junitconfig writing tojunit.xml. - RSpec:
rspec --format RspecJunitFormatter --out junit.xml. - Go / Rust / .NET:
ctrfformat is also supported where JUnit isn't natural.
If you wire up a runner with a twist worth writing down, open a PR with a recipe under this page.