Fluid Topics pipeline source

The Fluid Topics pipeline source ingests content from a Fluid Topics tenant through the standard Knowledge Hub REST API and sends each record to the pipeline's transform agent for indexing.

Content scope

A pipeline ingests one content layer, selected by content_scope:

• documents — standalone documents (PDFs, community articles, web pages, courses). When no query or filters are set, enumeration uses the list endpoint (GET /api/khub/documents, paginated by page/per_page), which returns the whole corpus with inline metadata. When a query or filters are set, it uses the search endpoint (POST /api/khub/documents/search), which honors them but returns at most 1000 results. The body is the post-OCR text from /api/khub/documents/{id}/content/text; documents whose body lives outside the tenant are indexed from their title and description.
• topics — the individual topics inside publications. Enumeration walks map by map: each map's topic list (/api/khub/maps/{mapId}/topics) is read once, so every topic carries its parent map's identity (map_id, map_title) and its inherited classification, and each map is read exactly once. The body is the topic's reader content.
• maps — the publications themselves. Enumeration emits one record per map rather than walking the topics inside it. query is ignored in this scope.

Run a separate pipeline per scope to cover more than one layer.

Authentication

The source authenticates with a Fluid Topics API key, sent as an HTTP Bearer authorization token. Create the key in the Fluid Topics administration interface under Integrations → API keys, then provide it as api_key in the source configuration. Vectara encrypts the key at rest and never returns it in responses.

Example source configuration

{
  "type": "fluidtopics",
  "base_url": "https://example.fluidtopics.net",
  "api_key": "<your Fluid Topics API key>",
  "content_scope": "documents",
  "query": "*",
  "locale": "en-US",
  "filters": {
    "Disclosure": ["Public"]
  }
}

Metadata

Every record carries its full Fluid Topics metadata, flattened into user_metadata:

• Each Fluid Topics metadata key is preserved with its value (a string for single-valued keys, a list for multi-valued keys).
• Hierarchical taxonomy values are preserved as a parallel <key>_hierarchy entry, with each path joined by > (for example Device Families > Agilex Families > Agilex 5).

system_metadata carries title, url (the ft:prettyUrl), source (fluidtopics), and document_type. Topic records additionally carry map_id and map_title for their parent publication. In every scope the full metadata and body are fetched together at content-fetch time (not during enumeration), so per-record listing stays lightweight.

The connector marks every record org-wide readable and applies no tenant access policy of its own. Classification and access control (for example deriving an access tier from Disclosure / content_classification / entitlement_groups, and routing to the appropriate corpus) are the responsibility of the pipeline transform.

Filtering

filters applies server-side metadata filters during enumeration, as a map of Fluid Topics metadata key to accepted values — for example {"Disclosure": ["Public"]} to ingest only public content.

Under content_scope: maps the filters are matched against each map's own metadata, and the reserved key id matches the map's identifier — {"id": ["<map_id>"]} restricts a run to a single publication. include_sources and exclude_sources apply in every scope.

Incremental sync

The Knowledge Hub search API exposes no server-side modified-since bound, so incremental sync re-enumerates and filters by last-edition date client-side. islatest is not a filterable field; latest-version selection is left to the pipeline using the ft:clusterId and ft:lastEdition metadata carried on every record.

Document result limit

Fluid Topics returns at most 1000 documents per tenant from both the list and the search endpoints. Topics are unaffected — they are enumerated map by map, not by search. To ingest more than 1000 documents, ask Fluid Topics to raise the per-tenant limit; the list-endpoint enumeration then pages through the full corpus with no further change.