Concepts
A pipeline has six parts: a source (where data comes from), a trigger (when it runs), a transform (the agent that processes each record), an optional verification (how to confirm success), a sync mode (incremental vs full refresh), and optional processing options (per-record timeout and parallelism overrides).
Source
The source defines where data comes from. Each record in the source is sent as a separate file upload to a new agent session.
See sources for details on each supported source type and configuration.
Source credentials are encrypted at rest using a customer-specific encryption key and are never returned in API responses.
Trigger
The trigger defines when the pipeline runs.
Cron
Run on a cron schedule. The expression is a standard 5-field cron expression (minute, hour, day-of-month, month, day-of-week) evaluated in UTC.
TRIGGER FIELD (CRON)
Code example with json syntax.1
Interval
Run at a fixed interval. The duration is an ISO-8601 duration string.
TRIGGER FIELD (INTERVAL)
Code example with json syntax.1
Manual
The pipeline never runs automatically — it only runs when explicitly
triggered via the /trigger endpoint.
TRIGGER FIELD (MANUAL)
Code example with json syntax.1
Overlap policy
Only one run can be active per pipeline at a time. If a scheduled trigger
fires while a previous run is still in progress, the new run is skipped.
Manual triggers return 409 Conflict if a run is already in progress.
Transform
The transform defines how source records are processed. Currently only agent transforms are supported.
Each source record the run processes gets a fresh agent session. The session receives a
record processing input: a JSON object with the record's source_record_id
and an operation of upsert or delete. For upsert records, the session
also receives the source file as an uploaded input. New operation values
may be added; treat unrecognized values as opaque. The agent then acts
on the operation according to its configured instructions and tools: it
transforms and indexes an upserted record, and handles a delete with
whatever tools it is configured with. A record whose agent run_condition
evaluated to false gets no session, and a record already processed at the same watermark
reuses its prior session.
A delete record follows the same lifecycle as an upsert. The agent's
run_condition runs when the session is created, the input is sent, verification
runs if configured, and the record succeeds when the session completes. The
one difference is the input: a delete has no file, so the session receives
only the record processing input. What the agent does with it is up to the
agent's configuration. A run_condition that evaluated to false skips the record. A full-refresh
run ignores deletions and processes only the records the source lists. A failed
session sends the record to the dead letter queue,
and the dead letter remembers that it was a delete, so a retry runs it as a
delete again. A completed session counts in records_processed and emits a
record_processing event whose operation is delete.
The default indexing agent deletes with the delete_documents tool.
Deletion works for documents that carry the record's source_record_id and
the pipeline's pipeline_key in their metadata, which the agent stamps at
write time, and requires the target corpus to declare both fields as
document-level filter attributes. A delete for a record indexed without the
stamps removes nothing: delete_documents returns
documents_deleted: 0 in the session's tool output. Deleting the same
record again is safe. An agent with no delete handling completes each delete
session without deleting anything, and the record still counts in
records_processed. Add delete_documents to its configuration, or
configure a judge to verify delete sessions. A custom agent that deletes through its own destination tools
configures delete_documents alongside them, bound to the metadata
fields the agent stamps on the documents it writes.
TRANSFORM FIELD
Code example with json syntax.1
Verification
By default, any agent session that completes without throwing an exception
is treated as success. You can configure stricter success criteria by
adding a verification field to the transform — either a UserFn condition
expression or a separate judge agent. Records that fail verification are
added to the dead letter queue.
See Verification for configuration, available context fields, and examples.
Sync mode
| Mode | Behavior |
|---|---|
incremental | Only processes records that are new, changed, or deleted since the last successful run. Uses an internal watermark to track progress. |
full_refresh | Processes all records from the source on every run. Source-reported deletions are not processed. |
Incremental mode is the default and recommended for most use cases. The pipeline tracks a watermark (source-specific, e.g. a timestamp for S3) after each successful run. The next run only fetches records whose watermark is after the stored value.
Use full_refresh when you need to reprocess the entire source, such as
after changing the agent's instructions or to recover from a data
corruption.
Processing options
The optional processing_options object overrides the service defaults for
this pipeline only:
| Field | Behavior |
|---|---|
record_timeout_minutes | Maximum time in minutes (1–180) for one processing attempt of a record, including the agent session work. Overrides the 30-minute default. The total retry budget for the record is twice this value. |
max_concurrent_records | Maximum number (1–64) of the pipeline's records processed in parallel within a run. Parallelism never exceeds this value; the service may run fewer in parallel, and values above the service's own limit are reduced to that limit. |
On update, a provided processing_options object replaces the stored one
wholesale; send an empty object to return to the service defaults.
Set these when a pipeline's records are unusually heavy — for example large documents whose conversion exceeds the default per-record window — or when the source system should see fewer concurrent requests.
Pipeline vs pipeline run
A pipeline is the persistent configuration — source, trigger, transform, sync mode, and the optional verification and processing options. It has a stable key and can be enabled, disabled, updated, or deleted.
A pipeline run is a single execution. Each run fetches records from the source and creates one agent session per record it processes. Runs have their own status (running, completed, failed, cancelled) and track how many records were fetched, processed, skipped, and failed.
Comparison to other features
| Feature | Purpose | Unit of work |
|---|---|---|
| Pipeline | Automated flow of all source data through an agent | One session per processed source record |
| Agent schedule | Recurring single execution of an agent with a fixed message | One session per trigger |
| Agent connector (e.g. Slack) | Bidirectional chat integration | One session per conversation |