Gemini CLI's strength in this workflow isn't case authoring — it's analysis over large volumes of structured data, thanks to Gemini's long context window handling large get_cases/get_results_for_run dumps without the aggressive summarization some other tools apply. If your TestRail instance has thousands of cases and dozens of historical runs, Gemini CLI is often the better tool for "what's actually going on with our coverage" questions, even if you're using Claude Code or Cursor for the day-to-day case writing. This topic covers setup, coverage-gap querying, a release-readiness report example, and an honest comparison against Claude Code's output on the same tasks.
Installing and Connecting TestRail MCP to Gemini CLI
Gemini CLI reads MCP config from .gemini/settings.json at the project level or ~/.gemini/settings.json globally. The config shape is close to Claude Code's, with mcpServers as the top-level key:
{
"mcpServers": {
"testrail": {
"command": "npx",
"args": ["-y", "@testrail/mcp-server"],
"env": {
"TESTRAIL_URL": "https://yourcompany.testrail.io",
"TESTRAIL_USERNAME": "qa-bot@yourcompany.com",
"TESTRAIL_API_KEY": "$TESTRAIL_API_KEY",
"TESTRAIL_PROJECT_ID": "14"
},
"trust": false
}
}
}
trust: false is the safer default — it means Gemini CLI will prompt for confirmation on tool calls rather than auto-executing them. Flip it to true only for read-only analysis sessions where you're confident no write tool (add_case, add_run, add_result_for_case) will ever fire; I keep it false on any config that has TestRail write scopes attached to the API key, full stop.
Verify with:
gemini
> /mcp list
This prints connected servers and tool counts. A known rough edge as of the current Gemini CLI release line: environment variable expansion in .gemini/settings.json uses plain $VAR syntax (no braces), and a config copied from a Claude Code .mcp.json using ${VAR} will pass the literal string through unexpanded — the server starts, but every TestRail call 401s. This is the exact same class of mistake as the OpenCode {env:VAR} gotcha, just a different syntax to watch for; if you're standardizing MCP config across three or four agent tools on the same team, write down each tool's substitution syntax in one internal doc rather than relying on memory.
Tips
- Keeptrust: falseon any TestRail MCP config where the API key has write scopes; only trust read-only analysis configs.
- Confirm$VAR(no braces) is Gemini CLI's expansion syntax before copying config from another agent tool — mismatched syntax fails silently as a 401, not a config error.
- Run/mcp listafter every settings change — Gemini CLI, like most of these tools, doesn't hot-reload MCP config mid-session.
Querying Runs, Results, and Coverage Gaps from Gemini CLI
Where Gemini CLI earns its keep is chewing through result history across many runs without losing the thread. A representative query:
Pull get_results_for_run for the last 8 runs in suite 12 (get_runs to find
them). For each case_id, compute: pass rate, number of distinct runs it
failed in, and whether any failure comment mentions "flaky" or "timeout".
Output a table sorted by pass rate ascending.
On a suite with roughly 340 cases across 8 runs (2,700+ individual result records), this is exactly the kind of bulk-pull-then-analyze task that benefits from a large context window — Gemini CLI pulls the full result set in a small number of paginated get_results_for_run calls and does the aggregation in one pass, rather than needing to summarize-and-discard intermediate data the way a smaller-context tool sometimes has to.
For coverage-gap analysis specifically — cases with no corresponding automated test — the query pattern combines TestRail data with a local codebase grep, same convention as covered for other tools:
Get all cases in suite 12 with type_id for "Regression" (check get_case_types
for the exact ID). Grep the codebase at tests/e2e/ and tests/api/ for
"TestRail: C\d+" annotations. List cases with no matching annotation,
grouped by section. Then list any annotation referencing a case_id that
get_case returns a 400 for (meaning the case was deleted or archived).
That second list — dangling annotations pointing at deleted cases — is a genuinely useful signal that's easy to let rot. It means someone archived a TestRail case (maybe the feature was cut) but nobody removed the corresponding automated test's TestRail reference, or worse, nobody removed the automated test itself, and it's now asserting behavior nobody is tracking as a requirement anymore.
A realistic coverage-gap output for a checkout suite:
Section: Payment Failures (14 cases)
No automation reference (6): C1044, C1046, C1052, C1058, C1061, C1063
Dangling reference (1): tests/e2e/checkout.spec.ts:142 -> C1039 (case not found, likely archived)
Section: Address Validation (9 cases)
No automation reference (2): C1071, C1073
Dangling reference (0)
That's an actionable, specific list a QA lead can triage in minutes — decide which of the 6 uncovered payment-failure cases justify new Playwright tests versus which are genuinely manual-only (some payment failure modes, like actual card issuer timeouts, are hard to automate reliably and stay manual by design).
Tips
- Lean on Gemini CLI for the "pull everything, then aggregate" style query — it handles large result-history dumps without needing to chunk the analysis across multiple turns.
- Always run the dangling-reference check (annotation points to a deleted/archived case) alongside the missing-reference check — it catches a different, easily overlooked failure mode.
- Ask for output grouped by section, not a flat list — a flat list of 40 uncovered case IDs is much harder for a human reviewer to prioritize than the same 40 grouped by feature area.
Practical Example: Producing a Release Readiness Report
Here's a full release-readiness prompt and a realistic output, the kind of thing you'd run the morning of a release-candidate cutoff.
Generate a release readiness report for run 187 ("Release 2.14.0 - Full
Regression") in project 14. Include: overall pass rate, list of failed
cases with their comments, list of blocked cases, any case not yet
executed, and a coverage note comparing this run's case_ids against
all P1/P2 cases in suite 12 to flag anything P1/P2 that was excluded
from this run entirely.
RELEASE READINESS REPORT — Release 2.14.0 - Full Regression (Run 187)
Generated: 2026-08-21
Overall: 218/231 executed (13 not run), 201 passed, 12 failed, 5 blocked
FAILED (12):
C1050 - Checkout declines card, network timeout — "Retry banner not
shown, generic 500 rendered instead" — P1
C1067 - Address autocomplete handles diacritics — "Autocomplete drops
accented characters in suggestion list" — P3
[10 more, full list in appendix]
BLOCKED (5):
C1102, C1103, C1104 - blocked by "staging environment payment gateway
sandbox down" (all reference the same blocking comment — likely one
environment issue, not three separate product bugs)
C1140, C1141 - blocked by "test data account locked, pending reset"
NOT EXECUTED (13):
C1200-C1206 (7 cases) - new cases added to suite after this run was
created; not included in run scope
C1210-C1215 (6 cases) - marked "Retest" from previous cycle, not
re-run yet
COVERAGE GAP CHECK:
3 P1 cases exist in suite 12 that are NOT included in this run at all:
C1301 (added 3 days ago, likely missed at run creation), C1250, C1251
(both tagged "mobile checkout" — possible scope decision, verify with
mobile lead before sign-off)
RECOMMENDATION: Do not sign off without resolving C1050 (P1, real product
defect, not environment) and confirming C1301/C1250/C1251 P1 exclusion
is intentional.
The genuinely useful part of this output is the grouping insight on the three blocked cases sharing one root cause comment — that's the kind of pattern a tired human skimming a 231-row spreadsheet at 6pm before a release is likely to miss, and it changes the triage conversation from "we have 5 blocked cases" to "we have one environment issue affecting 3 cases and one test-data issue affecting 2."
Tips
- Always ask explicitly for a coverage-gap check against P1/P2 cases excluded from the run — a run can look "healthy" at 95% pass rate while quietly missing critical cases that were never scoped in.
- Have the agent cluster blocked/failed cases by shared root-cause comment text — it turns a flat failure list into an actionable triage plan.
- Treat the "recommendation" section as a draft opinion, not a gate — the final sign-off call belongs to a release manager who knows business context the report can't see (e.g., whether C1250/C1251's mobile scope exclusion was a deliberate release-scope decision).
Comparing TestRail MCP Output Between Gemini CLI and Claude Code
Running the same three task types on both tools over several weeks surfaced consistent, specific differences worth knowing before you pick a default.
Case authoring: Claude Code's drafts tend to have slightly tighter, more consistent expected field phrasing out of the box — Gemini CLI's drafts are functionally equivalent but vary more in verbosity between cases in the same batch, occasionally writing a two-sentence expected result for one case and a ten-word one for the next case in the same batch. Neither is wrong; Claude Code's output needs less normalization pass.
Bulk analysis (coverage gaps, multi-run aggregation): Gemini CLI is faster and more thorough here, specifically on large pulls — the "compute pass rate across 8 runs and 2,700 results" query completed in one pass on Gemini CLI, while the equivalent Claude Code session on the same data occasionally needed a follow-up prompt to extend analysis it had implicitly truncated partway through when the intermediate data got large.
CI-driven result posting: Roughly equivalent — both correctly parse JUnit/JSON test reports and map to add_result_for_case calls. This is a simpler, more mechanical task where the underlying model differences matter less than the MCP tool wrapper's reliability, which is shared code in most setups.
Tool-call transparency: Claude Code's VS Code integration has a better visual review experience (collapsible JSON per call); Gemini CLI's terminal output is plain but complete — a matter of taste more than capability.
The practical takeaway from running both in parallel on the same TestRail instance for a quarter: use Claude Code (or Cursor) for daily case-authoring and result-posting work where consistency of format matters, and reach for Gemini CLI specifically when a task is "read a lot of TestRail history and tell me what's really going on" — release readiness reports, quarterly flakiness audits, multi-suite coverage sweeps. Don't treat this as a permanent tool choice; both ecosystems update fast enough that a comparison run six months from now could look different.
Tips
- Route large historical-analysis tasks (multi-run aggregation, quarterly coverage audits) to Gemini CLI specifically for its context-window advantage on bulk data pulls.
- Keep case-authoring conventions (title format, priority mapping) documented in a shared file both tools can read — it closes most of the consistency gap between them regardless of which one drafts a given batch.
- Re-run this kind of side-by-side comparison periodically — the relative strengths described here are a snapshot, not a permanent ranking, and both tools' MCP handling improves across releases.
Tips
Tips
- Match$VAR(no braces) expansion syntax in.gemini/settings.json— a config copied from another tool's${VAR}syntax fails silently as an auth error, not a config error.
- Default TestRail MCP config totrust: falsein Gemini CLI whenever the API key carries write scopes.
- Prefer Gemini CLI for large-scale historical analysis (coverage sweeps, multi-run aggregation, release readiness reports) and a case-authoring tool for the daily write-heavy work — pick per task, not per team-wide default.