Confluence

The Confluence source ingests pages from a Confluence instance. Each page becomes a record in the pipeline — its contents are uploaded to a new agent session for processing. Each page inherits its own and its ancestors' read restrictions, or the space audience, as document-level ACL. The source works against both Confluence Cloud and Data Center.

This is different from the Confluence tools for agents, which read and edit Confluence Cloud at conversation time rather than ingesting it into a corpus.

note

Confluence is configured through the API, not the Console source picker. Create or edit the pipeline with the source JSON below.

Authentication

Before you create the pipeline, set up your Confluence credential. Omit auth only if the instance allows anonymous read access. auth is a RemoteAuth credential. The credential depends on your deployment:

Deployment	auth
Cloud	{ "type": "header", "header": "Authorization", "value": "Basic <base64(email:api_token)>" }
Data Center	{ "type": "bearer", "token": "<personal_access_token>" }

For Cloud, create an API token at id.atlassian.com/manage-profile/security/api-tokens, then base64-encode email:api_token and send it as the value.

For Data Center, create a personal access token in your Confluence profile settings and supply it as token.

The account behind the credential needs read access to the spaces and pages you want to ingest.

caution

For Cloud, the value is sent verbatim as the Authorization header, so the literal Basic prefix must be included before the base64 string. A raw email:api_token value will not authenticate.

Configuration

SOURCE FIELD (CONFLUENCE CLOUD)

Code example with json syntax.json

1

Fields

Field	Required	Description
type	Yes	confluence.
base_url	Yes	Base URL of your Confluence instance, e.g. https://your-domain.atlassian.net for Cloud or https://confluence.example.com for Data Center. The Cloud /wiki context path is added automatically, so it does not need to be included.
deployment	No	cloud for Atlassian-hosted Confluence Cloud, data_center for a self-hosted Data Center or legacy Server instance. Defaults to cloud.
space_keys	No	Space keys to ingest. When omitted, every global space the authenticated account can read is ingested.
auth	No	Authentication for the instance. Omit only if the instance allows anonymous read access. See Authentication.
attachments	No	How page attachments are ingested. enabled (default true) emits each attachment as its own record; max_bytes (default 25 MiB, between 1 KiB and 100 MiB) skips larger attachments; pos_regex keeps only attachments whose file name matches one of its patterns, and neg_regex skips matching file names. See Attachments.
embedded_images	No	How images embedded in page bodies are handled. handling is inline (default) or omit; max_bytes (default 5 MiB, between 1 KiB and 15 MiB) leaves larger images as-is. See Embedded images.

Omit attachments or embedded_images to take the defaults. Set attachments.enabled to false to skip attachment records, or embedded_images.handling to omit to leave embedded images as Confluence URLs.

How records are fetched

Each run fans out one task per space for pages and, when attachments are enabled, one task per space for attachments. Each task streams its content type using CQL ordered by last-modified time.

Each page becomes one record; its body.export_view HTML is the record content. Each attachment becomes one record; its bytes are the record content, with the attachment's file name and media type, so the transform routes it like an uploaded file. Content with no parseable last-modified version is skipped, since it cannot take part in incremental sync.

Attachments

Every attachment is ingested unless a filter excludes it. pos_regex and neg_regex are Java regular expressions matched against the whole file name: a non-empty pos_regex keeps only attachments whose name matches at least one pattern, and neg_regex skips attachments whose name matches any pattern. max_bytes applies in every case.

Set pos_regex to the file-name patterns of the types your transform converts. The default transform converts PDF (.pdf), Word (.docx, .doc), PowerPoint (.pptx, .ppt), Excel (.xlsx, .xlsm), HTML (.html, .htm, .xhtml), Markdown (.md, .markdown, .mdx), CSV (.csv), TSV (.tsv), log (.log), plain-text (.txt, .text), RTF (.rtf), ODT (.odt) and EPUB (.epub) attachments. It recognizes an attachment by the media type Confluence reports or by its file name; a match on either is enough. It does not convert legacy binary Excel (.xls) or macro-enabled PowerPoint (.pptm); leave those names out of pos_regex, or use a custom transform that sends them to the spreadsheet and presentation tools, which accept both. This pattern lists the extensions the default transform converts and leaves image attachments to the page they are embedded in:

ATTACHMENTS FOR THE DEFAULT TRANSFORM

Code example with json syntax.json

1

Without pos_regex, an attachment of a type the transform does not convert is still downloaded and handed to an agent session. The session ends without indexing anything and, by default, the record counts as processed. Configure verification to catch this: a record whose verification fails is recorded in the dead-letter queue. Image attachments are separate records from the images inlined into their page, so a pasted screenshot becomes both an image part of its page and its own document unless neg_regex excludes image file names.

Attachments the filter excludes, by pos_regex, neg_regex, or as larger than max_bytes, are skipped before download and never reach the dead-letter queue. The same filter applies when a dead-letter entry is retried: an entry whose attachment the filter excludes is not downloaded, and the retry fails with "excluded by the attachment filter" as its reason. When Confluence does not report an attachment's size, the attachment is downloaded and, if it exceeds max_bytes, is discarded without retry and recorded in the dead-letter queue as too large. An attachment's record id is its Confluence content id, so re-uploading an attachment reprocesses it on the next run without a page edit.

Embedded images

With handling: inline, every image in a page body that is attached to the page itself or to one of its ancestors is downloaded with the source's credentials and embedded in the page, so it is indexed as an image part of the page. Full-resolution images are preferred over thumbnails when both are available. Images hosted elsewhere, images attached to any other page, images larger than max_bytes, and images that fail to download are left as Confluence rendered them; a page is never failed because of one image. At most 20 MiB of base64-encoded image data (roughly 15 MiB of original image bytes) is inlined per page; images beyond that total are left as Confluence URLs.

With handling: omit, page HTML is passed through as Confluence renders it. Images then reference Confluence URLs, which are not fetched and so are not indexed; only their alt text reaches the page content.

Source metadata

Each record carries source metadata that the connector resolves at fetch time.

system_metadata on a page record:

Key	Description
kind	page.
title	The page title. Absent when Confluence reports none.
page_id	The Confluence content id of the page.
space_key	The key of the space the page belongs to.
url	The page's web URL.
updated_time	The page's last-modified timestamp.
parent_id	The content id of the parent page. Absent for a top-level page.
ancestor_ids	The content ids of the page's ancestors, root first. Empty for a top-level page.
ancestor_titles	The titles of the page's ancestors, root first. Empty for a top-level page. An ancestor without a title is listed by its id.
path	The page's title path: the ancestor titles, root first, then the page title, joined with /. Absent when the page or any ancestor has no title. For structure, use ancestor_titles plus title, since a title may itself contain a slash.
labels	The page's labels. Empty when the page has none.

system_metadata on an attachment record:

Key	Description
kind	attachment.
title	The attachment's file name. Absent when Confluence reports none.
space_key	The key of the space the attachment belongs to.
url	The attachment's web URL when Confluence reports one, else its download URL.
updated_time	The attachment's last-modified timestamp.
media_type	The attachment's media type as reported by Confluence.
file_size	The attachment's size in bytes as reported by Confluence.
parent_id	The content id of the page the attachment belongs to. Absent when Confluence reports none.
parent_title	The title of that page. Absent when Confluence reports none.
parent_url	The web URL of that page. Absent when Confluence reports none.
ancestor_ids	The content ids of the containing page's ancestors, root first, then that page's own id. Empty when the containing page is not available.
ancestor_titles	The titles of the pages in ancestor_ids, in the same order. A page without a title is listed by its id.
path	The attachment's title path: the titles in ancestor_titles, then the attachment's file name (title), joined with /. Absent when the attachment or any page in ancestor_ids has no title. For structure, use ancestor_titles plus title, since a title may itself contain a slash.

user_metadata is empty for Confluence.

acl_metadata holds each page's effective read access in the source-independent ACL metadata shape. Resolution is conservative — it never grants more access than Confluence does:

• A page restricted by its own or an ancestor's read restriction emits the intersection of the named readers (readers) and named groups (group_readers) across every restricted level, and is never public or org-wide.
• An unrestricted page inherits the space audience: named users and groups map to readers and group_readers, and anonymous read maps to public_access.
• An attachment inherits the effective read access of the page it belongs to, resolved the same way. When the attachment reports no container page, its readers and group_readers are empty and its public_access and org_wide_access are none, so no principal is granted read access.

Confluence has no notion of comment or edit access in this shape, so owners, editors, commenters, and the group edit/comment buckets are left null. Unreadable permission data degrades to fewer grants, never more.

Incremental sync

When sync_mode is incremental (the default), the pipeline tracks a watermark based on each record's last-modified time: a page's own version time for pages, and the attachment's own version time for attachments. See Sync mode. On the next run, only pages and attachments modified since the stored watermark are reprocessed. Attachments modified before the stored watermark are not picked up by an incremental run; run the pipeline once with sync_mode set to full_refresh to ingest them. Replacing an image that a page embeds does not change the page's version, so the inlined image refreshes when the page itself is next edited.

Deletes are not propagated. A page the connector can no longer retrieve stops appearing in new runs. The pipeline emits no delete signal, so anything a previous run already produced downstream — for example, a document indexed into a corpus — is left in place.

Related

• Sources overview
• Pipeline concepts
• Session metadata