How to Generate E2E and API Tests from a Feature Spec and API Documentation?
This hands-on topic walks through building a complete, production-quality test suite for a real feature from scratch — starting with nothing but a feature specification and API documentation. The example feature is a user invitation system: an admin invites a team member by email, the invitee receives an email, clicks the link, sets a password, and lands on the dashboard.
This is an end-to-end workflow that spans API tests (the invitation creation endpoint), E2E tests (the acceptance flow), and email verification (the notification).
Step 1: Analyze the Feature Spec
Before writing a single prompt, read the feature spec carefully and extract the testable behaviors:
Feature Spec (abridged):
User Invitation Feature
=======================
Admin Flow:
1. Admin navigates to Settings > Team > Invite User
2. Fills: email (required), role (required: viewer/editor/admin),
first name (optional), personal message (optional, max 500 chars)
3. Submits → POST /api/v1/invitations
4. System sends invitation email with a unique link (expires in 72 hours)
5. Admin sees invited user in team list with status "pending"
6. Admin can resend invitation (if not accepted after 24 hours)
7. Admin can revoke invitation
Invitee Flow:
1. Invitee clicks link in email → /accept-invitation?token={uuid}
2. If token valid: shows "Set Your Password" form
3. If token expired: shows "Invitation Expired" page with re-request option
4. Fills: password (min 8, max 128, uppercase + number required),
confirm password (must match)
5. Submits → PATCH /api/v1/invitations/{id}/accept
6. On success: logged in automatically, redirected to /dashboard
7. First login shows "Welcome to the team!" onboarding modal
Business Rules:
- Cannot invite an email already in the system
- Cannot invite an email already pending an invitation
- Invitation link is single-use (cannot be used again after acceptance)
- Admin cannot invite someone at a higher role than themselves
From this spec, extract the test inventory:
Ask AI to derive the test inventory:
"Given the following feature specification, generate a complete test inventory.
For each test case, specify:
- Test ID (T-001, T-002, etc.)
- Test type (API / E2E / Email)
- Description (what behavior is being tested)
- Priority (P1-Critical, P2-High, P3-Medium)
- Pre-conditions
- Test data requirements
Feature spec: [PASTE SPEC]"
This produces a structured list of 20-30 test cases that becomes your implementation checklist.
Step 2: Generate API Tests from the API Documentation
With the test inventory, generate the API layer first. The API docs define the endpoints:
Generate a complete pytest API test suite for the User Invitation API.
API DOCUMENTATION:
=================
POST /api/v1/invitations
Auth: Bearer token (admin or owner role required)
Request body:
email: string, required, valid email format
role: string, required, enum: [viewer, editor, admin]
first_name: string, optional, max 50 chars
message: string, optional, max 500 chars
Response 201:
{ id: uuid, email: string, role: string, status: "pending",
expires_at: ISO8601, created_at: ISO8601 }
Response 400: email already registered in system
Response 409: email already has pending invitation
Response 422: validation errors
Response 403: attempting to invite at higher role than requester
GET /api/v1/invitations
Auth: Bearer token (admin or owner)
Query params: status (optional: pending/accepted/expired/revoked), page, limit
Response 200: { data: [invitation], pagination: {...} }
DELETE /api/v1/invitations/{id}
Auth: Bearer token (admin or owner, must be invitation creator or owner)
Response 204: success
Response 404: invitation not found
Response 409: cannot revoke already-accepted invitation
PATCH /api/v1/invitations/{id}/resend
Auth: Bearer token (admin or owner)
Conditions: Only if invitation is pending and created_at > 24 hours ago
Response 200: { ...invitation, expires_at: (new 72h expiry) }
Response 409: too early to resend (< 24 hours since last send)
PATCH /api/v1/invitations/{id}/accept
Auth: None (public endpoint, authenticated by token in URL)
Request body: token: string (from URL), password: string, confirm_password: string
Response 200: { access_token: string, user: {...} }
Response 400: passwords don't match
Response 410: token expired
Response 422: password validation error
FRAMEWORK:
- Pytest, Python 3.11
- requests library
- Fixtures: base_url, admin_api_client, owner_api_client, viewer_api_client
- Factories: UserFactory, InvitationFactory (available in tests/factories/)
- File location: tests/api/test_invitations.py
GENERATE all test cases for the POST /api/v1/invitations endpoint first,
covering all response codes documented above plus edge cases.
Then continue with the remaining endpoints.
Step 3: Generate E2E Tests from the UI Flow
After the API tests, generate the E2E layer:
Generate Playwright TypeScript E2E tests for the user invitation acceptance flow.
FRAMEWORK CONTEXT:
- Playwright 1.44, TypeScript
- Import test and expect from '../../fixtures/auth.fixture'
- Page Objects available:
InvitationPage:
- goto(token: string): navigate to /accept-invitation?token=token
- fillPassword(password: string): fills password field
- fillConfirmPassword(password: string): fills confirm field
- submit(): clicks submit button
- getErrorMessage(): returns error text or null
- isExpiredTokenPageVisible(): returns boolean
- isWelcomeModalVisible(): returns boolean
LoginPage: login(email, password), isLoggedIn()
- Seeding: use InvitationFactory.createPendingInvitation({ email, role })
to create an invitation in DB and return its token
- Cleanup: use InvitationFactory.deleteInvitation(id) after each test
GENERATE E2E tests for:
1. Happy path: valid token → set password → auto-login → dashboard + welcome modal
2. Expired token: expired token → shows expired page
3. Already-used token: token already accepted → shows "already used" error
4. Password mismatch: submit with non-matching passwords → error message
5. Weak password: password without uppercase → validation error
6. After acceptance, token is invalidated (using same token again → error)
Each test must:
- Seed its own invitation with unique email (use timestamp)
- Clean up in afterEach regardless of test outcome
- Assert both URL and visible UI state at the end of the happy path
Learning Tip: Always generate the API test layer before the E2E layer. API tests run in milliseconds, cover all the edge cases (error codes, validation rules, business logic) more efficiently than E2E, and form the foundation that tells you whether the backend is reliable before you invest in the UI layer. A test suite that has 80% API coverage and 20% E2E coverage is more valuable and faster than one that is 80% E2E.
How to Provide Codebase Context So the Output Is Immediately Runnable?
The difference between a "needs heavy editing" AI output and an "runs on first try" AI output is almost entirely the context you provide. This section shows the exact context dump structure that produces immediately runnable tests for the invitation feature.
The Complete Context Package
Build a context package document for your project. Copy this template and fill it in once; reuse it in every generation session:
## Project Identity
Name: [Your Project Name]
Framework: Playwright 1.44 / TypeScript 5.4
Test runner: @playwright/test
Node version: 20 LTS
Package manager: npm
## Directory Structure
tests/
e2e/ # Playwright E2E tests, organized by feature
api/ # Pytest API tests (Python)
fixtures/ # Playwright fixture extensions
pages/ # Page Object Models
factories/ # Test data factories
helpers/ # Shared utilities
## Critical Import Rule
For ALL Playwright tests:
import { test, expect } from '../../fixtures/auth.fixture';
NOT from '@playwright/test'
Exception: unauthenticated tests (login, signup, public pages) use:
import { test, expect } from '@playwright/test';
## Playwright Config Summary
baseURL: http://localhost:3000 (set via process.env.BASE_URL)
testDir: ./tests/e2e
trace: on-first-retry
workers: process.env.CI ? 2 : 4
## Fixture Interfaces
[paste auth.fixture.ts exports]
[paste db.fixture.ts exports]
## Page Object Interfaces
[paste public method signatures for all relevant POMs]
## Factory Interfaces
[paste factory function signatures]
## Environment Variables
E2E tests:
BASE_URL, TEST_USER_EMAIL, TEST_USER_PASSWORD, TEST_ADMIN_EMAIL, TEST_ADMIN_PASSWORD
API tests (Python):
API_BASE_URL, ADMIN_TOKEN, OWNER_TOKEN, TEST_DB_URL
## Selector Strategy (Priority Order)
1. data-testid="..." → page.getByTestId('...')
2. ARIA role + name → page.getByRole('button', { name: 'Submit' })
3. Label text → page.getByLabel('Email address')
4. NEVER: CSS class, XPath, nth()
## Test File Naming
E2E: tests/e2e/{module}/{feature}.spec.ts
API: tests/api/test_{resource}.py
## Known DOM Structure (from current application)
Invitation acceptance page (/accept-invitation):
data-testid="accept-invitation-form"
data-testid="invitation-password-input"
data-testid="invitation-confirm-password-input"
data-testid="invitation-submit-btn"
data-testid="invitation-expired-message"
data-testid="invitation-error-message"
data-testid="welcome-onboarding-modal"
Providing DOM Snapshots for New Features
For a brand-new feature that isn't in production yet, ask the frontend developer for the component's data-testid map as part of the feature development ticket. If that's not available, inspect the staging environment:
npx playwright eval "
const { chromium } = require('playwright');
(async () => {
const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('http://localhost:3000/accept-invitation?token=test-token');
const testIds = await page.evaluate(() => {
return Array.from(document.querySelectorAll('[data-testid]'))
.map(el => el.getAttribute('data-testid'));
});
console.log(testIds.join('\n'));
await browser.close();
})();
"
Paste this output into your AI prompt as "AVAILABLE DATA-TESTID ATTRIBUTES."
The Immediate Runability Checklist
Before submitting any generation prompt, verify:
[ ] Have I specified the exact framework version?
[ ] Have I included the correct import path for test fixtures?
[ ] Have I provided the public interfaces of any POM classes the test should use?
[ ] Have I listed the available factory functions with their signatures?
[ ] Have I provided the known data-testid attributes for the page?
[ ] Have I specified the environment variable names for test credentials?
[ ] Have I stated the test file's intended location (for correct relative import paths)?
[ ] Have I included a representative example of an existing passing test?
If you check all 8 boxes, the probability of "runs on first try" is very high. If you skip any, expect to spend 5–15 minutes on fixes that AI could have avoided with the context.
Learning Tip: Build an alias or shell function that dumps your context package into your clipboard, ready to paste:
alias testctx="cat tests/CONTEXT.md | pbcopy && echo 'Context copied'". Running this before opening an AI session takes 2 seconds. Hunting down the right import path because you forgot to include it takes 10 minutes. The context file is your highest-leverage investment in AI-assisted test generation.
How to Debug and Fix AI-Generated Test Failures?
Even with excellent context, AI-generated tests will fail on the first run. Knowing how to diagnose and fix these failures efficiently is a core skill for AI-assisted QA.
The Failure Triage Framework
When an AI-generated test fails, classify the failure in under 2 minutes using this decision tree:
Is the error a TypeScript/syntax compilation error?
YES → Paste the TS error into the same AI conversation: "Fix this TypeScript error: [error]"
Is the error "element not found" or "locator resolved to 0 elements"?
YES → Selector issue. Check:
1. Is the data-testid correct? Inspect the real DOM.
2. Is the element inside an iframe? (Playwright requires frame.locator() for iframes)
3. Is the element visible only after an action/wait? Add a waitForSelector.
Is the error "expected X but received Y" in an assertion?
YES → Assertion value issue. Check:
1. Does the real page actually show the expected text/value?
2. Is the assertion too strict? (exact match vs. partial match)
3. Is there a timing issue? (asserting before a data load completes)
Is the error a timeout (waiting for navigation, response, or element)?
YES → Wait strategy issue. Check:
1. Does the flow actually complete in the test environment?
2. Is there a network request that needs to complete first?
3. Is the timeout value appropriate for CI environments (which are slower than local)?
Is the error a fixture or import resolution error?
YES → Context issue. Check:
1. Import paths are relative — is the test file in the expected location?
2. Is the fixture exported from the expected file?
3. Does the factory function exist and accept those parameters?
The Iterative Fix Prompt
When a test fails, paste the full failure output into your AI conversation with precise context:
The test I generated is failing. Help me diagnose and fix it.
FAILURE OUTPUT (complete CI/terminal output):
1) [chromium] › e2e/invitations/acceptance.spec.ts:23:7 › acceptance flow ›
happy path: valid token → auto-login → dashboard
Error: locator.fill: Error: strict mode violation:
getByTestId('invitation-password-input') resolved to 2 elements
Call log:
- waiting for getByTestId("invitation-password-input")
- locator resolved to 2 elements:
<input data-testid="invitation-password-input" type="password" name="password">
<input data-testid="invitation-password-input" type="password" name="confirm">
FAILING TEST CODE:
[paste the specific failing lines]
WHAT I FOUND IN THE DOM (I inspected the real page):
There are two input fields with the same data-testid. This seems like a bug in the
application — the developer should have used different testids. But for now, I need
the test to work.
The password field has: name="password", placeholder="Enter password"
The confirm field has: name="confirm", placeholder="Confirm password"
FIX THE TEST to use the most resilient selector that still uniquely identifies each field,
and add a comment explaining the workaround until the application uses distinct testids.
Debugging Timing and Flakiness Issues
The most insidious failures in AI-generated tests are timing-related — they pass most of the time but fail intermittently. Debug these systematically:
Step 1: Reproduce locally with trace:
npx playwright test tests/e2e/invitations/acceptance.spec.ts --trace on
npx playwright show-trace test-results/.../trace.zip
The trace viewer shows every action, network request, and DOM state change with timestamps. Share the trace screenshot with AI:
This Playwright test is flaky — it fails about 15% of the time on this step:
await checkout.placeOrder();
await expect(page).toHaveURL('/confirmation');
The Playwright trace shows that placeOrder() clicks the button, an API call is made to
POST /api/v1/orders, and the response comes back 201. But sometimes the redirect to
/confirmation happens AFTER the assertion already ran and failed.
CURRENT CODE:
await checkout.placeOrder();
await expect(page).toHaveURL('/confirmation');
FIX: Update the test to explicitly wait for the order API response before asserting the URL.
The endpoint is POST /api/v1/orders and returns 201 on success.
The fix:
// Wait for the order creation API call to complete before asserting navigation
const orderResponse = page.waitForResponse(
response => response.url().includes('/api/v1/orders') && response.status() === 201
);
await checkout.placeOrder();
await orderResponse;
await expect(page).toHaveURL('/confirmation');
Debugging Data/State Issues
A test is failing because the database state is not what the test expects.
The test creates an invitation and then tries to accept it, but gets a 404.
FAILING TEST:
[paste]
CI OUTPUT:
Error: expected status 200, received 404
Response body: { "error": { "code": "INVITATION_NOT_FOUND", "message": "..." } }
WHAT I KNOW:
- The invitation factory creates the invitation successfully (I verified with console.log)
- The token used in the accept step is coming from the factory return value
- This failure only happens in CI, not locally
LIKELY ISSUE: The factory creates the invitation in one database, but the test app
reads from a different database in our CI environment.
Help me add debugging to the test to:
1. Log the created invitation ID and token
2. Immediately after creation, verify the invitation exists via API call GET /api/v1/invitations/{id}
3. Only proceed to the accept step if the verification GET returns 200
This will help determine if the issue is in creation, retrieval, or cross-DB isolation.
Learning Tip: Keep a "failure log" for every AI-generated test failure you encounter — record the test name, the error type, the root cause, and the fix. After 20-30 entries, you'll see clear patterns: your three most common failure types will account for 70% of all failures. Use those patterns to improve your generation prompts so AI stops making those same mistakes. The failure log is your personal feedback loop for getting better at AI test generation.
How to Commit and Document AI-Generated Test Suites?
AI-generated tests deserve the same engineering hygiene as manually written tests — proper commits, meaningful descriptions, and documentation that future maintainers can rely on. But AI-generated code also benefits from specific provenance documentation that helps the team understand how to maintain it.
The Commit Strategy
Structure your commits so AI-generated tests are reviewable and traceable:
git add tests/api/test_invitations.py
git commit -m "feat(tests): add API test suite for user invitation system
Covers: POST/GET/DELETE /api/v1/invitations, PATCH /accept, PATCH /resend
Test types: positive, negative, boundary, authentication, error structure
Generated with Claude Code (claude-sonnet-4-5), reviewed and validated manually.
Test count: 34 tests across 5 endpoint test classes
Coverage: all documented response codes + edge cases from feature spec
See tests/api/INVITATION_TESTS.md for test inventory and maintenance notes."
git add tests/e2e/invitations/
git commit -m "feat(tests): add E2E test suite for invitation acceptance flow
Covers: happy path, expired token, duplicate use, password validation
Playwright TypeScript, using InvitationPage POM and InvitationFactory.
Generated with Claude Code, reviewed against staging environment.
All 6 tests passing locally and in CI (GitHub Actions, chromium only)."
git add tests/factories/invitation.factory.ts
git commit -m "feat(test-infra): add InvitationFactory for test data seeding
Factory methods: createPendingInvitation(), createExpiredInvitation(),
createAcceptedInvitation(), deleteInvitation()
Used by both API and E2E test suites."
PR Description Template for AI-Generated Tests
## Summary
Adds a complete test suite for the User Invitation feature (see [ticket link]).
**Test Coverage Added:**
- 34 API tests (`tests/api/test_invitations.py`)
- 6 E2E tests (`tests/e2e/invitations/acceptance.spec.ts`)
- InvitationFactory (`tests/factories/invitation.factory.ts`)
**Test Types:**
- API: positive, negative, boundary, authentication error, error response structure
- E2E: happy path, expired token, duplicate use, password validation (4 error cases)
**AI Generation Notes:**
Tests were generated with Claude Code using the project context package in
`tests/CONTEXT.md`. All generated tests were:
- [ ] Run locally and confirmed passing
- [ ] Reviewed against the application's actual DOM (selectors verified)
- [ ] Reviewed against the API spec (response assertions verified)
- [ ] Checked for hardcoded credentials, URLs, and magic values
- [ ] Checked for proper teardown in afterEach
**Known Gaps (follow-up tickets):**
- Email delivery testing not covered (requires mailhog integration — see [ticket])
- Bulk invitation (future feature) not yet tested
## Test Plan
- Run `npx playwright test tests/e2e/invitations/` — all 6 tests should pass
- Run `pytest tests/api/test_invitations.py -v` — all 34 tests should pass
- Review selector strategy: confirm all use data-testid or ARIA role
Inline Documentation for AI-Generated Tests
Add comments that help maintainers update the tests when the application changes:
/**
* E2E test suite: Invitation Acceptance Flow
*
* Generated: 2025-01-15 using Claude Code (claude-sonnet-4-5)
* Feature spec: https://jira.company.com/browse/FEAT-441
*
* MAINTENANCE NOTES:
* - Selectors use data-testid attributes — see InvitationPage POM for full list
* - InvitationFactory.createPendingInvitation() seeds DB directly via API
* - If invitation expiry time changes (currently 72h), update EXPIRY_HOURS constant
* - Welcome modal (data-testid="welcome-onboarding-modal") is shown only on first login;
* if the modal condition changes, update the onboarding modal assertion
*
* KNOWN BEHAVIOR:
* - The invitation token is single-use; the duplicate-use test must run AFTER the
* happy path test if run in serial. In parallel mode, each creates its own token.
*/
test.describe('Invitation Acceptance Flow', () => {
const EXPIRY_HOURS = 72; // Update if business rule changes
// ...
});
Generating Test Documentation with AI
Once the test suite is complete, use AI to generate a test inventory document:
Generate a test inventory document for the following test files.
For each test, extract:
- Test ID (use file path + test name as ID)
- Test description (from the test name)
- Test type (E2E / API)
- Priority (infer from test name and coverage — P1 if it tests a critical path)
- Pre-conditions (from beforeEach)
- Steps (infer from test body)
- Expected result (from assertions)
Output as a markdown table, grouped by test file.
FILES:
[paste or reference test files]
This produces a human-readable test inventory that can be imported into Jira, TestRail, or Xray for traceability between tests and requirements.
Learning Tip: AI-generated tests that aren't documented are AI-generated tests that will be deleted. When a future engineer sees a test failing after an application change and can't quickly understand what it was testing or why — they'll delete it instead of fixing it. The 10 minutes you spend adding maintenance notes and a PR description is what converts an AI-generated test from a disposable artifact into a durable asset. Document for the person who doesn't know AI generated it.