# Agent V2 E2E

This file scopes Agent v2 E2E conventions for `e2e/features/agent-v2/` and its step definitions under `e2e/features/step-definitions/agent-v2/`. Keep package-wide runner, lifecycle, hook, fixture, locator, assertion, and cleanup rules in `e2e/AGENTS.md`.

Do not add deeper `AGENTS.md` files unless an Agent v2 submodule becomes independently owned.

## Scope

Agent v2 scenarios live under `features/agent-v2/` and use the `@agent-v2` capability tag.

The E2E web environment enables Agent v2 through `NEXT_PUBLIC_ENABLE_AGENT_V2=true` in `scripts/common.ts`, because `/roster` routes are guarded by that feature flag.

Preview/Test Run scenarios are not part of the current build-mode slice unless explicitly requested. Current Agent v2 coverage should prioritize Configure, Build draft, saved configuration display, publish state, Access Point, preflight, files, advanced settings, and other build-mode behavior. Published Web app runtime is not Builder Preview; keep it as a separate `@web-app-runtime` slice because it exercises the public app surface and real model-backed responses after publish.

Use API setup for prerequisite state, then use Playwright only for user-observable navigation, editing, and assertions. Do not make assertions pass by mirroring the current implementation blindly; if a failure exposes a product ambiguity, resource gap, or test-quality problem, identify the owner before changing the test.

## Tags

Use tags in three layers:

- Capability tags describe the product area: `@build`, `@files`, `@advanced-settings`, `@agent-edit`, `@publish`, `@access-point`, `@output-variables`, and similar tags.
- Execution-scope tags describe how the scenario should be selected: `@core`, `@infra`, `@web-app-runtime`, `@service-api-runtime`, `@agent-backend-runtime`, `@preview`, `@feature-gated`, `@external-model`, and `@external-tool`.
- Narrow fixture or sub-surface tags describe a specific dependency or slice: `@stable-model`, `@agent-decision-model`, `@skill-fixture`, `@web-app-access`, `@workflow-reference`, `@files-limits`, and similar tags.

- `@agent-v2` — required capability tag for all Agent v2 scenarios.
- `@core` — stable non-runtime scenario expected to run in the regular Agent v2 suite when its explicit preconditions are met. Do not apply `@core` to Preview/Test Run, Web app chat runtime, or Backend service API chat runtime scenarios.
- `@infra` — infrastructure or readiness checks.
- `@build` — Build mode and Build draft behavior.
- `@build-unavailable-resources` — feature-gated Build chat recovery when the user requests unavailable Skills or Tools.
- `@files` — Files section upload, display, and fixture behavior.
- `@files-limits` — file limit behavior. Multiple-file drop is stable core coverage; format and size rejection remain feature-gated until their product contracts are stable.
- `@knowledge` — Knowledge Retrieval configuration display, persistence, and reference cleanup.
- `@advanced-settings` — Env Editor, Content Moderation, and related Advanced Settings behavior.
- `@agent-create` — Agent Roster creation and initial Configure navigation.
- `@agent-edit` — saved Agent detail/configuration display surfaces.
- `@publish` — publish and publish-bar state.
- `@access-point` — Web app, Backend service API, and Workflow access surfaces.
- `@stable-model` — active model fixture dependency. Apply this to every scenario that includes `the Agent Builder stable chat model is available` or otherwise requires an active model configured in the workspace.
- `@agent-decision-model` — stronger active model fixture dependency for scenarios that specifically validate Agent autonomous planning or resource-selection behavior, such as generated-query Knowledge Retrieval. Do not use it for scenarios that only need a model to exist or answer a deterministic prompt.
- `@tool-fixture` — preseeded Tool dependency such as `JSON Process / JSON Replace` or `Tavily / Tavily Search`.
- `@skill-fixture` — checked-in or preseeded Skill dependency such as `e2e-summary-skill`.
- `@knowledge-fixture` — preseeded dataset dependency such as `E2E Agent Knowledge Base`.
- `@full-config-agent` — fixed `E2E New Agent Builder Full Config` Agent dependency.
- `@tool-states-agent` — fixed `E2E New Agent Builder Tool States` Agent dependency.
- `@oauth-tool-agent` — fixed `E2E Agent With OAuth Tool` Agent dependency for OAuth2 tool credential preservation.
- `@dual-retrieval-fixture` — fixed dual Knowledge Retrieval Agent dependency.
- `@backend-api-access` — fixed or scenario-owned Backend service API access dependency.
- `@published-web-app` — fixed or scenario-owned published Web app access dependency.
- `@web-app-runtime` — published public Web app runtime behavior. Use it for scenarios that open the public Web app and assert real chat responses. Access Point URL, launch, customization, and settings surfaces remain `@access-point` behavior unless they send messages through the public Web app.
- `@service-api-runtime` — Backend service API runtime behavior. Use it for scenarios that call the published service API and assert real chat responses. Endpoint display, copy, API key, and API reference surfaces remain `@access-point` behavior.
- `@agent-backend-runtime` — standalone `dify-agent` run-server dependency. Apply it only when the scenario goes through Agent v2 runtime execution that requires `AGENT_BACKEND_BASE_URL`, such as Build chat/finalize, published Web app chat, or Backend service API chat. This is separate from `@external-model`; a scenario can call a real model without using `dify-agent`, and a future Agent v2 runtime scenario might exercise `dify-agent` without a paid model provider.
- `@external-model` — package-wide external runtime tag. In Agent v2, apply it to Build chat generation/finalization, published Web app chat, Backend service API chat, and other Agent runtime requests that can invoke a real model provider. Do not use this for scenarios that only require a configured active model.
- `@external-tool` — package-wide external runtime tag. In Agent v2, apply it only when the scenario executes a real external tool provider at runtime. Do not use this for plugin installation, tool discovery, tool configuration persistence, or local deterministic tools.
- `@feature-gated` — product capability is optional. This tag alone does not skip execution; the scenario must include an explicit step that returns `skipped` with a blocked-precondition reason when the feature is unavailable.

Use feature-level `@core` only when every scenario in the file is stable, non-runtime, and not feature-gated. If a feature file mixes stable scenarios with runtime, Preview, or feature-gated scenarios, put `@core` only on the stable scenarios. Keep runtime tags scenario-level so the regular core suite cannot inherit them accidentally.

## Step Organization

Keep Agent v2 step definitions grouped by user capability, not by DOM component or Cucumber keyword:

- `configure.steps.ts` — common configure navigation, refresh, autosave, and normal draft assertions.
- `build-draft.steps.ts` — Build mode checkout, apply, discard, supported writeback, and Build draft isolation.
- `files.steps.ts` — Files upload, display, and fixture-list assertions.
- `knowledge.steps.ts` — Knowledge Retrieval configuration persistence and reference cleanup.
- `tools.steps.ts` — Tools selector, search, and configuration-boundary behavior.
- `advanced-settings.steps.ts` — common Advanced Settings shell and supported-entry assertions.
- `env-editor.steps.ts` — Env Editor add, import, delete, persistence, and restored-display behavior.
- `content-moderation.steps.ts` — Content Moderation availability, keyword settings, and feature-gated assertions.
- `agent-roster.steps.ts` — Agent Roster creation and Roster-level user actions.
- `agent-edit.steps.ts` — saved Agent detail display assertions.
- `publish.steps.ts` — publish and publish-bar assertions.
- `access-point.steps.ts` — common Access Point navigation and overview.
- `access-point-web-app.steps.ts` — Web app access entrypoints and public Web app assertions.
- `access-point-service-api.steps.ts` — Backend service API entrypoints, keys, API reference, and service requests.
- `access-point-workflow.steps.ts` — Workflow access references.
- `preflight.steps.ts` — explicit `Given` entrypoints for Agent Builder preflight resources.

Cucumber step definitions are globally registered. Do not duplicate the same step text across files, even if one is written as `Given` and another as `Then`.

## World State

`DifyWorld` owns generic scenario state such as `page`, `context`, errors, downloads, cleanup queues, and created resource IDs.

Agent v2 business state belongs under `world.agentBuilder`; do not keep adding Agent v2-specific fields to the top level of `DifyWorld`.

Use the existing namespace shape:

- `world.agentBuilder.preflight.stableModel`
- `world.agentBuilder.preflight.brokenModel`
- `world.agentBuilder.preflight.preseededResources`
- `world.agentBuilder.accessPoint.serviceApiBaseURL`
- `world.agentBuilder.accessPoint.generatedApiKey`
- `world.agentBuilder.accessPoint.serviceApiResponse`
- `world.agentBuilder.accessPoint.apiReferencePage`
- `world.agentBuilder.accessPoint.webAppPage`
- `world.agentBuilder.accessPoint.webAppURL`
- `world.agentBuilder.accessPoint.workflowReferencePage`
- `world.agentBuilder.accessPoint.composerDraftSnapshot`
- `world.agentBuilder.configure.concurrentPage`
- `world.agentBuilder.workflow.agentConsolePage`
- `world.agentBuilder.workflow.outputVariables`

Use `features/agent-v2/support/agent.ts` for Agent v2 core API fixtures. It owns roster-shaped Agent IDs, configure/access route helpers, composer draft sync, version details, workflow references, publish, and Agent cleanup. Use `features/agent-v2/support/agent-soul.ts` for reusable Agent Soul fixture configuration, prompts, and model/dataset config builders. Use `features/agent-v2/support/agent-build-draft.ts` for Build draft checkout/save/discard API helpers. Use `features/agent-v2/support/agent-drive.ts` for Agent drive/config file and Skill upload plus cleanup helpers. Use `features/agent-v2/support/access-point.ts` for Web app access, Backend service API access, API keys, and service API request helpers. Store created roster Agent IDs in `DifyWorld.createdAgentIds`; the shared `After` hook deletes them after each scenario.

Use `DifyWorld.createdAgentDriveFiles` for Agent drive files committed during a scenario and `DifyWorld.createdBuiltinToolCredentials` for built-in tool credentials created during a scenario. The shared `After` hook deletes Agent drive files first so cleanup also works for scenarios that upload into a preseeded Agent.

## Setup Boundary

Use `a basic configured Agent v2 test agent has been created via API` when a scenario only needs a created Agent with a composer draft. Do not use that basic shell for runtime, model, tool, skill, knowledge, environment variable, moderation, or output-variable coverage until those resources have explicit seed helpers and readiness checks.

Use `a runnable Agent v2 test agent has been created via API` after `the Agent Builder stable chat model is available` when a scenario needs a real model-backed Agent. The step writes the preflight model into the Agent Soul model config through `features/agent-v2/support/agent-soul.ts` with deterministic E2E model settings; do not duplicate provider/model payload construction in individual steps.

Use `a runnable Agent v2 test agent using the agent-decision model has been created via API` after `the Agent Builder agent-decision chat model is available` only when the scenario is asserting Agent autonomous decision quality rather than generic runtime availability. Keep this scoped to scenarios tagged `@agent-decision-model`.

Use `the Agent v2 configuration should be saved automatically` after UI edits that rely on Configure autosave. It waits for the user-visible publish bar saved state; do not replace it with network-idle waits or internal store checks.

API setup is acceptable for creating scenario-owned Agents, enabling Backend service API, writing composer drafts, seeding Build drafts, and preparing fixed state. The scenario must still assert user-visible behavior or a real persisted product contract through the public Console API. Do not assert only that a setup API call succeeded.

Do not use scenario API setup to repair an environment-owned Agent Builder seed. If a scenario depends on a fixed Agent, dataset, workflow, Skill, Tool, credential, published Web app, or active model, use the matching preflight step to verify it and block when it is missing or drifted. Create or mutate resources only when they are scenario-owned and registered for cleanup.

## API Contract Types

Agent v2 support helpers consume Console API contracts from `@dify/contracts/api/console/.../types.gen`. When a generated request, response, or payload type exists, import and use that exact type name at the helper boundary. Do not keep an old local response type name as an alias for the generated type.

Keep local types for Agent v2 E2E-owned state only, such as `DifyWorld.agentBuilder` state, scenario preflight resource records, fixture registry entries, helper input options, and deliberately narrowed test view models. If an endpoint response needs a field that the generated contract does not expose yet, fix the backend schema and regenerate contracts before broadening E2E types.

## Build Mode And Preview

Build mode scope means:

- Configure page behavior.
- Build draft checkout, pending state, apply, discard, and route isolation.
- Supported Build draft writeback for files, skills, and env.
- Saved configuration display after apply/discard/refresh.

Preview/Test Run scope means:

- Model-backed runtime responses.
- Duplicate run prevention.
- Runtime failure recovery.
- Tool/knowledge hit behavior proven through Agent replies.

Keep Preview/Test Run scenarios out of the current build-mode slice unless the task explicitly reintroduces them.

## Preflight Resources

Agent Builder resource checks live under `features/agent-v2/support/preflight/`. Import from the specific module that owns the resource contract; do not add a preflight barrel file:

- `models.ts`
- `agents.ts`
- `datasets.ts`
- `tools.ts`
- `access.ts`

`preflight.steps.ts` should remain the explicit `Given` entrypoint. Do not move preflight into hidden hooks.

Agent Builder preflight is read-only. It checks long-lived seed resources and records their IDs or normalized metadata for later steps, but it must not create missing resources, toggle fixed access settings, upload missing Skills/files, publish fixed Agents, or patch model/provider credentials. Seed creation and repair belong to the environment setup process, not to Cucumber scenarios.

Treat preseeded Agent Builder resources as environment contracts. Preflight can report that a stable model, dataset, Skill, Tool credential, fixed Agent, published Web app, or workflow reference is missing, inactive, not indexed, or drifted, but it must not repair that drift during a scenario. Seed scripts, CI bootstrap, or the documented environment maintenance flow own creating and keeping those resources valid.

Use `the Agent v2 runtime backend is available` before scenarios tagged `@agent-backend-runtime`. The step checks the standalone `dify-agent` run server through `E2E_AGENT_BACKEND_URL`, `AGENT_BACKEND_BASE_URL`, or the default `http://127.0.0.1:5050` when `E2E_START_AGENT_BACKEND=1`. The E2E runner starts this service with `uv run --project dify-agent --extra server uvicorn dify_agent.server.app:app --host 127.0.0.1 --port 5050` only when `E2E_START_AGENT_BACKEND=1` is explicit. Because Agent App runtime always carries the `dify.config` layer and that layer depends on `dify.shell`, the same runner path also starts the `dify-agent/docker/local-sandbox` shellctl container on `E2E_SHELLCTL_PORT` (default `5004`) and injects `DIFY_AGENT_SHELLCTL_ENTRYPOINT` into the Agent backend. Missing or unreachable runtime backend or shellctl sandbox should be reported as a blocked precondition, not discovered later as a `/build-chat/finalize` 400 or Web app response timeout.

Use `the Agent Builder stable chat model is available` before scenarios that need a real Agent Soul model configuration. This includes true runtime scenarios, model-backed build-mode assertions, and Workflow Agent v2 node setup because the backend rejects Agent nodes without model config. Do not add the model preflight to pure navigation or identity checks unless the setup API itself requires model config. `E2E_STABLE_MODEL_PROVIDER`, `E2E_STABLE_MODEL_NAME`, and optional `E2E_STABLE_MODEL_TYPE` are selectors for a model already configured in the workspace; they are not provider credentials. The step defaults to `openai` / `gpt-5-nano` / `llm`, verifies the selected model is present and `active` through `/console/api/workspaces/current/models/model-types/{type}`, then stores it on `DifyWorld.agentBuilder.preflight.stableModel`.

Use `the Agent Builder agent-decision chat model is available` before scenarios that need a stronger model to exercise Agent autonomous planning, generated query selection, or tool/resource choice. `E2E_AGENT_DECISION_MODEL_PROVIDER`, `E2E_AGENT_DECISION_MODEL_NAME`, and optional `E2E_AGENT_DECISION_MODEL_TYPE` are selectors for a second active model fixture, defaulting to `openai` / `gpt-5.5` / `llm`. The step stores the model on `DifyWorld.agentBuilder.preflight.agentDecisionModel`. Do not use this fixture as a broad replacement for `@stable-model`; it is intentionally narrower and costlier.

Keep `@stable-model` on Build draft apply scenarios that click `Apply`. The current product path calls `/build-chat/finalize` before applying the draft, and the backend returns `model is required` when the Agent Soul has no model config. Discard-only and pending-draft isolation scenarios can stay model-free when they do not finalize the Build draft.

Do not pass model provider API keys through Cucumber or Playwright env vars. Provider credentials belong to the Dify environment seed/admin setup. If the selected provider/model is missing or inactive, the scenario must be blocked by preflight instead of trying to create or patch provider credentials during the test.

Override the default selector only when a scenario or environment explicitly needs a different stable model:

```bash
E2E_STABLE_MODEL_PROVIDER=openai
E2E_STABLE_MODEL_NAME=gpt-5-nano
E2E_STABLE_MODEL_TYPE=llm

E2E_AGENT_DECISION_MODEL_PROVIDER=openai
E2E_AGENT_DECISION_MODEL_NAME=gpt-5.5
E2E_AGENT_DECISION_MODEL_TYPE=llm
```

Dify may expose OpenAI as either `openai` or a plugin provider ID such as `langgenius/openai/openai`. The preflight accepts both forms for selection and stores the actual Console API provider ID for Agent Soul setup.

Use `the Agent Builder broken chat model is available` before model-recovery scenarios that intentionally start from an invalid model. The step requires `E2E_BROKEN_MODEL_PROVIDER`, defaults `E2E_BROKEN_MODEL_NAME` to `e2e-broken-model`, defaults `E2E_BROKEN_MODEL_TYPE` to `llm`, and only verifies that the model entry exists. The scenario must still assert the user-visible failure and recovery behavior.

Use `the Agent Builder preseeded Agent "{name}" is available`, `the Agent Builder preseeded workflow "{name}" is available`, `the Agent Builder preseeded dataset "{name}" is available`, and `the Agent Builder preseeded tool "{provider} / {tool}" is available` when a scenario depends on a fixed environment resource. These steps verify the resource through Console APIs, store the result in `DifyWorld.agentBuilder.preflight.preseededResources`, and return `skipped` when the resource is missing.

Use `the Agent Builder preseeded dataset "{name}" is indexed and ready` for knowledge retrieval scenarios that require a completed knowledge base. It verifies that the dataset exists, has documents, all listed documents are available, and every document indexing status is `completed`. For `E2E Agent Knowledge Base`, it also verifies through the Console segment list API that at least one enabled segment contains `AGENT_KNOWLEDGE_PASS`.

Use `the Agent Builder preseeded dataset "{name}" is indexing` for failure-recovery scenarios that require an indexing or queued knowledge base. It verifies at least one document is in `waiting`, `parsing`, `cleaning`, `splitting`, or `indexing`.

`e2e-summary-skill` has two separate E2E contracts. The checked-in package under `e2e/fixtures/test-materials/e2e-summary-skill/SKILL.md` is used by scenario-owned Agents to verify that the Skill package can be uploaded to an Agent drive. Fixed display/configuration scenarios use `e2e-summary-skill` as a preseeded resource: the environment-owned `E2E New Agent Builder Full Config` Agent must already include that drive-backed Skill before the scenario starts. Do not mutate fixed preseeded Agents during a scenario to add missing Skills.

Use `the Agent Builder preseeded Agent "{agent}" includes drive skill "{skill}"` to verify that a fixed Agent has a drive-backed Skill attached. If it is missing, return a blocked precondition owned by seed/product instead of uploading the Skill into the fixed Agent.

Use `the Agent Builder preseeded Agent "{agent}" has Backend service API access with an API key` to verify that a fixed Agent has Backend service API enabled and at least one key. The API key step does not validate a human-readable key name because the Console API key response does not expose one.

Use `the Agent Builder preseeded Agent "{agent}" includes the core fixture configuration` for the fixed Full Config Agent prerequisite. It composes the stable model, Summary Skill, JSON Replace tool, and indexed knowledge-base preflights, then reads `/console/api/agent/{agent_id}/composer` to verify the Agent Soul contains the selected model, prompt success token, required file fixtures, JSON Replace tool entry, and knowledge dataset reference. Do not use this step for Agent node output variables; those live in workflow node-job `declared_outputs`, not the roster Agent App composer response.

Use `the Agent Builder preseeded Agent "{agent}" includes the tool state fixture configuration` for the fixed Tool States Agent prerequisite. It composes the Summary Skill, JSON Replace tool, and Tavily Search tool preflights, then reads `/console/api/agent/{agent_id}/composer` to verify the Agent Soul includes JSON Replace, Tavily Search, and a Tavily credential reference. This proves the seed is configured to exercise tool status UI; keep actual invalid-credential errors in dependent user-visible configuration or runtime scenarios.

Use `the Agent Builder preseeded Agent "{agent}" includes an OAuth2 tool credential` for the fixed OAuth Tool Agent prerequisite. It reads the Agent composer and verifies at least one Dify tool has `credential_type: oauth2` with a `credential_ref.id`. Scenarios that need to save or mutate this configuration should copy that tool config into a scenario-owned Agent instead of mutating the fixed preseeded Agent.

Use `the Agent Builder preseeded Agent "{agent}" includes the dual retrieval fixture configuration` for the fixed Dual Retrieval Agent prerequisite. It composes the indexed knowledge-base preflight, then reads `/console/api/agent/{agent_id}/composer` to verify `agent_soul.knowledge.sets` includes both an Agent-decide generated query set and a custom user-query set using the fixed custom query.

Use `the Agent Builder preseeded Agent "{agent}" has published Web app access` to verify that a fixed Agent is published, Web app access is enabled, and the Agent detail response includes the site token and base URL needed to open the Web app.

Use `the Agent Builder preseeded Agent "{agent}" is referenced by workflow "{workflow}"` to verify Workflow access prerequisites. It checks both fixed resources exist, then uses `/console/api/agent/{agent_id}/referencing-workflows`, the same Console API used by the Access Point Workflow references table, to verify the workflow references the Agent through at least one published Agent node.

Run `pnpm -C e2e e2e -- --tags @agent-v2-preflight` against a seeded environment to verify Agent Builder preseeded resource readiness before running dependent scenarios. Keep each resource as a separate preflight scenario so a missing resource marks only its dependent precondition as blocked instead of hiding the rest of the readiness report.

## Blocked And Partial Policy

Use explicit skipped steps for missing resources, disabled feature flags, and product capabilities that are not currently implemented. `@feature-gated` is only a label; it is not execution semantics.

Blocked messages should be specific enough to route ownership:

```text
Blocked precondition: <resource> missing <capability>. Owner: seed/product. Remediation: <what to seed or decide>.
```

Order blocked steps by the real owner of the first unresolved condition. If a scenario is not automatable yet because the product behavior or test fixture contract is undefined, put that explicit availability step before model/tool/dataset preflights so the report is not masked by an unrelated missing seed. If the scenario is otherwise automatable and only depends on an external seed resource, run the matching resource preflight before creating scenario-owned state. When an availability check must inspect a real Configure UI surface, create only scenario-owned disposable state first and rely on the shared cleanup path after the skipped result.

Use partial coverage only when current product behavior is intentionally narrower than the written requirement and the test still asserts a real user-visible behavior. Example: Files are currently flat in Agent config files, so the flat Files list can be asserted while tree display remains blocked until product support exists.

Multiple-file drop is already covered as stable `@core @files-limits` behavior. File format and size rejection remain feature-gated until the product exposes stable Agent config file restrictions and user-visible error states. Do not convert those gated `@files-limits` scenarios to passing tests by relying on default environment behavior; first align the product contract or seed configuration.

Do not mark a scenario as complete if it only proves setup state and does not assert the user-visible behavior or persisted product contract required by the case.
