DataGEMS Cross-Dataset Discovery Service v1¶
The Cross-Dataset Discovery Service provides a secure API for performing natural language search queries across a pre-indexed collection of datasets, and for adding datasets to or removing them from that index.
Terms of service: https://datagems.eu/terms
Contact: DataGEMS helpdesk
License: EUPL-1.2 license
Servers¶
| Description | URL |
|---|---|
| Default server path when running behind a reverse proxy | /cdd |
Monitoring¶
GET /¶
Liveness
Description
Returns a static OK response without checking dependencies. Used as the Kubernetes liveness probe. No authentication required.
Responses
{
"message": "string",
"status": "ok"
}
Schema of the response body
{
"properties": {
"message": {
"type": "string"
},
"status": {
"example": "ok",
"type": "string"
}
},
"type": "object"
}
GET /health¶
Health Check
Description
Checks that the search component is initialized and that Qdrant and the TEI embedding service are reachable. Used as the Kubernetes readiness probe. No authentication required.
Responses
{
"message": "string",
"status": "ok"
}
Schema of the response body
{
"properties": {
"message": {
"type": "string"
},
"status": {
"example": "ok",
"type": "string"
}
},
"type": "object"
}
{
"code": 104,
"error": "error communicating with underpinning service",
"message": {
"correlationId": "string",
"payload": {},
"source": "string",
"statusCode": 0
}
}
Schema of the response body
{
"properties": {
"code": {
"example": 104,
"type": "integer"
},
"error": {
"example": "error communicating with underpinning service",
"type": "string"
},
"message": {
"$ref": "#/components/schemas/FailedDependencyMessage"
}
},
"type": "object"
}
Search¶
POST /corpus-analysis-search/¶
Corpus Analysis Search
Description
Search for text passages answering a natural language question, with LLM assistance. An LLM first extracts the text request from the question (time-period, location and language constraints are extracted but not yet applied). The index is searched with that request, and the LLM then scores each candidate and drops those scoring below 3, so fewer than k results may be returned. If either LLM step fails, the original question and the unreranked results are used instead. Dataset access filtering is the same as POST /search/. Results also carry the item metadata.
Input parameters
| Parameter | In | Type | Default | Nullable | Description |
|---|---|---|---|---|---|
OAuth2Bearer |
header | string | N/A | No | JWT token for authentication, obtained from the OIDC provider. |
Request body
{
"dataset_ids": [
"string"
],
"k": 0,
"query": "string",
"search_mode": "sparse"
}
Schema of the request body
{
"properties": {
"dataset_ids": {
"description": "Dataset UUIDs to restrict the search to. Intersected with the datasets the caller may browse. Must not be an empty list.",
"items": {
"type": "string"
},
"nullable": true,
"type": "array"
},
"k": {
"default": 50,
"deprecated": true,
"description": "Deprecated. Number of candidates to retrieve before LLM reranking; fewer results may be returned.",
"maximum": 100,
"minimum": 1,
"type": "integer"
},
"query": {
"description": "The natural language question. An LLM extracts the text request from it before searching.",
"type": "string"
},
"search_mode": {
"default": "dense",
"deprecated": true,
"description": "Deprecated. The retrieval strategy; the API will choose it automatically in a future version.",
"enum": [
"sparse",
"dense",
"hybrid"
],
"type": "string"
}
},
"required": [
"query"
],
"type": "object"
}
Responses
{
"query_time": 10.12,
"results": [
{
"content": "string",
"dataset_id": "string",
"metadata": {},
"object_id": "string",
"similarity": 10.12
}
]
}
Schema of the response body
{
"properties": {
"query_time": {
"description": "The time taken to process the query, in milliseconds.",
"format": "float",
"type": "number"
},
"results": {
"items": {
"$ref": "#/components/schemas/API_CorpusSearchResult"
},
"type": "array"
}
},
"required": [
"query_time",
"results"
],
"type": "object"
}
{
"code": 102,
"error": "Validation Error",
"message": [
{
"Key": "string",
"Value": [
"string"
]
}
]
}
Schema of the response body
{
"properties": {
"code": {
"example": 102,
"type": "integer"
},
"error": {
"example": "Validation Error",
"type": "string"
},
"message": {
"items": {
"$ref": "#/components/schemas/ValidationErrorDetail"
},
"type": "array"
}
},
"type": "object"
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
{
"code": 104,
"error": "error communicating with underpinning service",
"message": {
"correlationId": "string",
"payload": {},
"source": "string",
"statusCode": 0
}
}
Schema of the response body
{
"properties": {
"code": {
"example": 104,
"type": "integer"
},
"error": {
"example": "error communicating with underpinning service",
"type": "string"
},
"message": {
"$ref": "#/components/schemas/FailedDependencyMessage"
}
},
"type": "object"
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
POST /search-ap/¶
Search via an Analytical Pattern
Description
Same search as POST /search/, but the request and response are DataGEMS analytical patterns. The search arguments are read from the properties of the node labelled CDD_Operator. In the response, any nodes previously connected as outputs of the operator are removed; one sc:Dataset node is added per result dataset and one cr:FileObject node per result object, with an output edge from the operator and a distribution edge from the dataset to each object. The full search response is merged into the pattern's metadata. A pattern without a CDD_Operator node, without query, or with invalid optional arguments results in a 500 error.
Input parameters
| Parameter | In | Type | Default | Nullable | Description |
|---|---|---|---|---|---|
OAuth2Bearer |
header | string | N/A | No | JWT token for authentication, obtained from the OIDC provider. |
Request body
{
"ap": {
"edges": [
{
"from": "string",
"labels": [
"string"
],
"to": "string"
}
],
"nodes": [
{
"id": "string",
"labels": [
"string"
],
"properties": {}
}
]
},
"metadata": {}
}
Schema of the request body
{
"additionalProperties": true,
"description": "A DataGEMS analytical pattern. It must contain a node labelled `CDD_Operator` whose `properties` hold the search arguments (`query` required; `k`, `dataset_ids`, `search_mode` optional, same meaning as in `SearchRequest`).",
"properties": {
"ap": {
"properties": {
"edges": {
"items": {
"properties": {
"from": {
"type": "string"
},
"labels": {
"items": {
"type": "string"
},
"type": "array"
},
"to": {
"type": "string"
}
},
"type": "object"
},
"type": "array"
},
"nodes": {
"items": {
"properties": {
"id": {
"type": "string"
},
"labels": {
"items": {
"type": "string"
},
"type": "array"
},
"properties": {
"additionalProperties": true,
"type": "object"
}
},
"type": "object"
},
"type": "array"
}
},
"required": [
"nodes",
"edges"
],
"type": "object"
},
"metadata": {
"additionalProperties": true,
"type": "object"
}
},
"required": [
"ap"
],
"type": "object"
}
Responses
{
"ap": {
"edges": [
{
"from": "string",
"labels": [
"string"
],
"to": "string"
}
],
"nodes": [
{
"id": "string",
"labels": [
"string"
],
"properties": {}
}
]
},
"metadata": {}
}
Schema of the response body
{
"additionalProperties": true,
"description": "A DataGEMS analytical pattern. It must contain a node labelled `CDD_Operator` whose `properties` hold the search arguments (`query` required; `k`, `dataset_ids`, `search_mode` optional, same meaning as in `SearchRequest`).",
"properties": {
"ap": {
"properties": {
"edges": {
"items": {
"properties": {
"from": {
"type": "string"
},
"labels": {
"items": {
"type": "string"
},
"type": "array"
},
"to": {
"type": "string"
}
},
"type": "object"
},
"type": "array"
},
"nodes": {
"items": {
"properties": {
"id": {
"type": "string"
},
"labels": {
"items": {
"type": "string"
},
"type": "array"
},
"properties": {
"additionalProperties": true,
"type": "object"
}
},
"type": "object"
},
"type": "array"
}
},
"required": [
"nodes",
"edges"
],
"type": "object"
},
"metadata": {
"additionalProperties": true,
"type": "object"
}
},
"required": [
"ap"
],
"type": "object"
}
{
"code": 102,
"error": "Validation Error",
"message": [
{
"Key": "string",
"Value": [
"string"
]
}
]
}
Schema of the response body
{
"properties": {
"code": {
"example": 102,
"type": "integer"
},
"error": {
"example": "Validation Error",
"type": "string"
},
"message": {
"items": {
"$ref": "#/components/schemas/ValidationErrorDetail"
},
"type": "array"
}
},
"type": "object"
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
{
"code": 104,
"error": "error communicating with underpinning service",
"message": {
"correlationId": "string",
"payload": {},
"source": "string",
"statusCode": 0
}
}
Schema of the response body
{
"properties": {
"code": {
"example": 104,
"type": "integer"
},
"error": {
"example": "error communicating with underpinning service",
"type": "string"
},
"message": {
"$ref": "#/components/schemas/FailedDependencyMessage"
}
},
"type": "object"
}
POST /search/¶
Perform a Search Query
Description
Submits a natural language query and returns a ranked list of relevant results. The search can be restricted to dataset_ids; this list is intersected with the datasets the caller holds a dg_ds-browse grant on, and results from any other dataset are always dropped. Callers with a privileged role (dg_admin, dg_system, dg_dataset-curator) are not filtered. If none of the requested datasets are permitted, an empty result list is returned.
Input parameters
| Parameter | In | Type | Default | Nullable | Description |
|---|---|---|---|---|---|
OAuth2Bearer |
header | string | N/A | No | JWT token for authentication, obtained from the OIDC provider. |
Request body
{
"dataset_ids": [
"string"
],
"k": 0,
"query": "string",
"search_mode": "sparse"
}
Schema of the request body
{
"properties": {
"dataset_ids": {
"description": "Dataset UUIDs to restrict the search to. Intersected with the datasets the caller may browse. Must not be an empty list.",
"items": {
"type": "string"
},
"nullable": true,
"type": "array"
},
"k": {
"default": 5,
"description": "The number of results to return.",
"maximum": 100,
"minimum": 1,
"type": "integer"
},
"query": {
"description": "The natural language query string.",
"type": "string"
},
"search_mode": {
"default": "sparse",
"description": "The retrieval strategy: `sparse` (BM25), `dense` (bge-m3 vectors) or `hybrid` (both, fused with Reciprocal Rank Fusion).",
"enum": [
"sparse",
"dense",
"hybrid"
],
"type": "string"
}
},
"required": [
"query"
],
"type": "object"
}
Responses
{
"query_time": 10.12,
"results": [
{
"content": "string",
"dataset_id": "string",
"object_id": "string",
"similarity": 10.12
}
]
}
Schema of the response body
{
"properties": {
"query_time": {
"description": "The time taken to process the query, in milliseconds.",
"format": "float",
"type": "number"
},
"results": {
"items": {
"$ref": "#/components/schemas/API_SearchResult"
},
"type": "array"
}
},
"required": [
"query_time",
"results"
],
"type": "object"
}
{
"code": 102,
"error": "Validation Error",
"message": [
{
"Key": "string",
"Value": [
"string"
]
}
]
}
Schema of the response body
{
"properties": {
"code": {
"example": 102,
"type": "integer"
},
"error": {
"example": "Validation Error",
"type": "string"
},
"message": {
"items": {
"$ref": "#/components/schemas/ValidationErrorDetail"
},
"type": "array"
}
},
"type": "object"
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
{
"code": 104,
"error": "error communicating with underpinning service",
"message": {
"correlationId": "string",
"payload": {},
"source": "string",
"statusCode": 0
}
}
Schema of the response body
{
"properties": {
"code": {
"example": 104,
"type": "integer"
},
"error": {
"example": "error communicating with underpinning service",
"type": "string"
},
"message": {
"$ref": "#/components/schemas/FailedDependencyMessage"
}
},
"type": "object"
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
Ingestion¶
DELETE /dataset/{dataset_id}¶
Delete a Dataset from the Index
Description
Removes all indexed points of a dataset. The caller needs a dg_ds-delete grant on the dataset or a privileged role. The source data is not touched.
Input parameters
| Parameter | In | Type | Default | Nullable | Description |
|---|---|---|---|---|---|
OAuth2Bearer |
header | string | N/A | No | JWT token for authentication, obtained from the OIDC provider. |
dataset_id |
path | string | No | The dataset UUID. |
Responses
{
"dataset_id": "string",
"deleted_count": 0
}
Schema of the response body
{
"properties": {
"dataset_id": {
"description": "The deleted dataset.",
"type": "string"
},
"deleted_count": {
"description": "Number of indexed points removed.",
"type": "integer"
}
},
"required": [
"dataset_id",
"deleted_count"
],
"type": "object"
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
{
"code": 104,
"error": "error communicating with underpinning service",
"message": {
"correlationId": "string",
"payload": {},
"source": "string",
"statusCode": 0
}
}
Schema of the response body
{
"properties": {
"code": {
"example": 104,
"type": "integer"
},
"error": {
"example": "error communicating with underpinning service",
"type": "string"
},
"message": {
"$ref": "#/components/schemas/FailedDependencyMessage"
}
},
"type": "object"
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
GET /ingest/¶
List Ingestion Jobs
Description
Lists ingestion jobs, newest first. Callers with a privileged role (dg_admin, dg_system, dg_dataset-curator) can list all jobs. Other callers must pass dataset_id and hold a dg_ds-edit grant on that dataset.
Input parameters
| Parameter | In | Type | Default | Nullable | Description |
|---|---|---|---|---|---|
OAuth2Bearer |
header | string | N/A | No | JWT token for authentication, obtained from the OIDC provider. |
dataset_id |
query | string | No | Only return jobs for this dataset. Required for callers without a privileged role. | |
limit |
query | integer | 10 | No | Maximum number of jobs to return. |
status |
query | string | No | Only return jobs with this status. |
Responses
{
"jobs": [
{
"dataset_id": "string",
"deleted_count": 0,
"error": "string",
"finished_at": "2022-04-13T15:42:05.901Z",
"items_indexed": 0,
"items_submitted": 0,
"job_id": "string",
"points_in_qdrant": 0,
"profile_path": "string",
"replace": true,
"requested_by": "string",
"started_at": "2022-04-13T15:42:05.901Z",
"status": "pending"
}
]
}
Schema of the response body
{
"properties": {
"jobs": {
"description": "The matching jobs, newest first.",
"items": {
"$ref": "#/components/schemas/IngestionJob"
},
"type": "array"
}
},
"required": [
"jobs"
],
"type": "object"
}
{
"code": 102,
"error": "Validation Error",
"message": [
{
"Key": "string",
"Value": [
"string"
]
}
]
}
Schema of the response body
{
"properties": {
"code": {
"example": 102,
"type": "integer"
},
"error": {
"example": "Validation Error",
"type": "string"
},
"message": {
"items": {
"$ref": "#/components/schemas/ValidationErrorDetail"
},
"type": "array"
}
},
"type": "object"
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
{
"code": 104,
"error": "error communicating with underpinning service",
"message": {
"correlationId": "string",
"payload": {},
"source": "string",
"statusCode": 0
}
}
Schema of the response body
{
"properties": {
"code": {
"example": 104,
"type": "integer"
},
"error": {
"example": "error communicating with underpinning service",
"type": "string"
},
"message": {
"$ref": "#/components/schemas/FailedDependencyMessage"
}
},
"type": "object"
}
POST /ingest/¶
Ingest a Dataset
Description
Indexes a dataset described by a dataset profile (JSON) that the service reads from path_to_profile (a path on the service's mounted storage, e.g. /s3/...). The caller needs a dg_ds-edit grant on the profile's @id or a privileged role. Ingestion runs as a background job: the endpoint returns 202 with a job_id to poll via GET /ingest/{job_id}. The job record is saved before the job starts; if it cannot be saved, the request fails with 424. If the dataset is already indexed and replace is false, nothing is done and 200 with status skipped is returned.
Input parameters
| Parameter | In | Type | Default | Nullable | Description |
|---|---|---|---|---|---|
OAuth2Bearer |
header | string | N/A | No | JWT token for authentication, obtained from the OIDC provider. |
path_to_profile |
query | string | No | Path to the dataset profile JSON, as seen by the service. | |
replace |
query | boolean | False | No | Delete the dataset's existing indexed points and rebuild it from scratch. |
Responses
{
"dataset_id": "string",
"job_id": "string",
"status": "pending"
}
Schema of the response body
{
"properties": {
"dataset_id": {
"description": "The `@id` of the dataset in the profile.",
"type": "string"
},
"job_id": {
"description": "The ingestion job id to poll, or `null` when the ingestion was skipped.",
"nullable": true,
"type": "string"
},
"status": {
"description": "`pending` when a job was scheduled, `skipped` when the dataset is already indexed and `replace` is false.",
"enum": [
"pending",
"skipped"
],
"type": "string"
}
},
"required": [
"job_id",
"status",
"dataset_id"
],
"type": "object"
}
{
"dataset_id": "string",
"job_id": "string",
"status": "pending"
}
Schema of the response body
{
"properties": {
"dataset_id": {
"description": "The `@id` of the dataset in the profile.",
"type": "string"
},
"job_id": {
"description": "The ingestion job id to poll, or `null` when the ingestion was skipped.",
"nullable": true,
"type": "string"
},
"status": {
"description": "`pending` when a job was scheduled, `skipped` when the dataset is already indexed and `replace` is false.",
"enum": [
"pending",
"skipped"
],
"type": "string"
}
},
"required": [
"job_id",
"status",
"dataset_id"
],
"type": "object"
}
Schema of the response body
{
"oneOf": [
{
"$ref": "#/components/schemas/ErrorResponse"
},
{
"$ref": "#/components/schemas/ValidationErrorResponse"
}
]
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
{
"code": 104,
"error": "error communicating with underpinning service",
"message": {
"correlationId": "string",
"payload": {},
"source": "string",
"statusCode": 0
}
}
Schema of the response body
{
"properties": {
"code": {
"example": 104,
"type": "integer"
},
"error": {
"example": "error communicating with underpinning service",
"type": "string"
},
"message": {
"$ref": "#/components/schemas/FailedDependencyMessage"
}
},
"type": "object"
}
GET /ingest/{job_id}¶
Get Ingestion Job Status
Description
Returns the state of an ingestion job. Requires the same dg_ds-edit grant (or privileged role) as starting it. Job records are stored in Qdrant and survive service restarts.
Input parameters
| Parameter | In | Type | Default | Nullable | Description |
|---|---|---|---|---|---|
OAuth2Bearer |
header | string | N/A | No | JWT token for authentication, obtained from the OIDC provider. |
job_id |
path | string | No | The `job_id` returned by `POST /ingest/`. |
Responses
{
"dataset_id": "string",
"deleted_count": 0,
"error": "string",
"finished_at": "2022-04-13T15:42:05.901Z",
"items_indexed": 0,
"items_submitted": 0,
"job_id": "string",
"points_in_qdrant": 0,
"profile_path": "string",
"replace": true,
"requested_by": "string",
"started_at": "2022-04-13T15:42:05.901Z",
"status": "pending"
}
Schema of the response body
{
"properties": {
"dataset_id": {
"description": "The dataset being ingested.",
"type": "string"
},
"deleted_count": {
"description": "Number of existing points deleted (set when `replace` is true).",
"nullable": true,
"type": "integer"
},
"error": {
"description": "The failure reason (set when the job fails or is interrupted).",
"nullable": true,
"type": "string"
},
"finished_at": {
"description": "UTC timestamp when the job completed, failed or was interrupted.",
"format": "date-time",
"nullable": true,
"type": "string"
},
"items_indexed": {
"description": "Number of points this job wrote to Qdrant (set on completion). Normally equal to `items_submitted`.",
"nullable": true,
"type": "integer"
},
"items_submitted": {
"description": "Number of items sent for embedding (set on completion).",
"nullable": true,
"type": "integer"
},
"job_id": {
"description": "The ingestion job id.",
"type": "string"
},
"points_in_qdrant": {
"description": "Total number of the dataset's points in Qdrant after the job (set on completion). After a fresh or `replace=true` ingestion it should equal `items_indexed`.",
"nullable": true,
"type": "integer"
},
"profile_path": {
"description": "The profile path the job was started with.",
"type": "string"
},
"replace": {
"description": "Whether existing points for the dataset were deleted first.",
"type": "boolean"
},
"requested_by": {
"description": "The `sub` claim of the caller that started the job.",
"nullable": true,
"type": "string"
},
"started_at": {
"description": "UTC timestamp when the job was created.",
"format": "date-time",
"type": "string"
},
"status": {
"description": "The job state. `interrupted` means the service restarted before the job finished; the dataset may be partially indexed and should be re-ingested with `replace=true`.",
"enum": [
"pending",
"running",
"completed",
"failed",
"interrupted"
],
"type": "string"
}
},
"type": "object"
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
{
"code": 403,
"error": "User does not have permission to ingest this dataset."
}
Schema of the response body
{
"properties": {
"code": {
"description": "The HTTP status code.",
"example": 403,
"type": "integer"
},
"error": {
"description": "A human-readable error message.",
"example": "User does not have permission to ingest this dataset.",
"type": "string"
}
},
"type": "object"
}
{
"code": 104,
"error": "error communicating with underpinning service",
"message": {
"correlationId": "string",
"payload": {},
"source": "string",
"statusCode": 0
}
}
Schema of the response body
{
"properties": {
"code": {
"example": 104,
"type": "integer"
},
"error": {
"example": "error communicating with underpinning service",
"type": "string"
},
"message": {
"$ref": "#/components/schemas/FailedDependencyMessage"
}
},
"type": "object"
}
Schemas¶
AnalyticalPattern¶
| Name | Type | Description |
|---|---|---|
ap |
Properties: edges, nodes |
|
metadata |
API_CorpusSearchResult¶
| Name | Type | Description |
|---|---|---|
content |
string | The text of the matched item. |
dataset_id |
string | The UUID of the source dataset. |
metadata |
The indexed payload of the item (e.g. `dataset_id`, `file_object_id`, `original_filename`, plus loader-specific fields). | |
object_id |
string | The UUID of the source file object within the dataset, or `dataset_id` for dataset-level items. |
similarity |
number(float) | The relevance score from retrieval. |
API_SearchResult¶
| Name | Type | Description |
|---|---|---|
content |
string | The text of the matched item. |
dataset_id |
string | The UUID of the source dataset. |
object_id |
string | The UUID of the source file object within the dataset. For dataset-level items (e.g. the dataset description) this equals `dataset_id`. |
similarity |
number(float) | The relevance score (BM25 score, cosine similarity or RRF score, depending on `search_mode`). |
CorpusSearchRequest¶
| Name | Type | Description |
|---|---|---|
dataset_ids |
Array<string> | Dataset UUIDs to restrict the search to. Intersected with the datasets the caller may browse. Must not be an empty list. |
k |
integer | Deprecated. Number of candidates to retrieve before LLM reranking; fewer results may be returned. |
query |
string | The natural language question. An LLM extracts the text request from it before searching. |
search_mode |
string | Deprecated. The retrieval strategy; the API will choose it automatically in a future version. |
CorpusSearchResponse¶
| Name | Type | Description |
|---|---|---|
query_time |
number(float) | The time taken to process the query, in milliseconds. |
results |
Array<API_CorpusSearchResult> |
DeleteDatasetResponse¶
| Name | Type | Description |
|---|---|---|
dataset_id |
string | The deleted dataset. |
deleted_count |
integer | Number of indexed points removed. |
ErrorResponse¶
| Name | Type | Description |
|---|---|---|
code |
integer | The HTTP status code. |
error |
string | A human-readable error message. |
FailedDependencyMessage¶
| Name | Type | Description |
|---|---|---|
correlationId |
string | null | |
payload |
||
source |
string | The failing component: `SearchComponent`, `Qdrant`, `TEI` or `IngestionJobStore`. |
statusCode |
integer | The status code reported by (or for) the downstream service. |
FailedDependencyResponse¶
| Name | Type | Description |
|---|---|---|
code |
integer | |
error |
string | |
message |
FailedDependencyMessage |
IngestionJob¶
| Name | Type | Description |
|---|---|---|
dataset_id |
string | The dataset being ingested. |
deleted_count |
integer | null | Number of existing points deleted (set when `replace` is true). |
error |
string | null | The failure reason (set when the job fails or is interrupted). |
finished_at |
string(date-time) | null | UTC timestamp when the job completed, failed or was interrupted. |
items_indexed |
integer | null | Number of points this job wrote to Qdrant (set on completion). Normally equal to `items_submitted`. |
items_submitted |
integer | null | Number of items sent for embedding (set on completion). |
job_id |
string | The ingestion job id. |
points_in_qdrant |
integer | null | Total number of the dataset's points in Qdrant after the job (set on completion). After a fresh or `replace=true` ingestion it should equal `items_indexed`. |
profile_path |
string | The profile path the job was started with. |
replace |
boolean | Whether existing points for the dataset were deleted first. |
requested_by |
string | null | The `sub` claim of the caller that started the job. |
started_at |
string(date-time) | UTC timestamp when the job was created. |
status |
string | The job state. `interrupted` means the service restarted before the job finished; the dataset may be partially indexed and should be re-ingested with `replace=true`. |
IngestionJobList¶
| Name | Type | Description |
|---|---|---|
jobs |
Array<IngestionJob> | The matching jobs, newest first. |
IngestResponse¶
| Name | Type | Description |
|---|---|---|
dataset_id |
string | The `@id` of the dataset in the profile. |
job_id |
string | null | The ingestion job id to poll, or `null` when the ingestion was skipped. |
status |
string | `pending` when a job was scheduled, `skipped` when the dataset is already indexed and `replace` is false. |
SearchRequest¶
| Name | Type | Description |
|---|---|---|
dataset_ids |
Array<string> | Dataset UUIDs to restrict the search to. Intersected with the datasets the caller may browse. Must not be an empty list. |
k |
integer | The number of results to return. |
query |
string | The natural language query string. |
search_mode |
string | The retrieval strategy: `sparse` (BM25), `dense` (bge-m3 vectors) or `hybrid` (both, fused with Reciprocal Rank Fusion). |
SearchResponse¶
| Name | Type | Description |
|---|---|---|
query_time |
number(float) | The time taken to process the query, in milliseconds. |
results |
Array<API_SearchResult> |
StatusResponse¶
| Name | Type | Description |
|---|---|---|
message |
string | |
status |
string |
ValidationErrorDetail¶
| Name | Type | Description |
|---|---|---|
Key |
string | Location of the invalid field, e.g. `body.dataset_ids`. |
Value |
Array<string> |
ValidationErrorResponse¶
| Name | Type | Description |
|---|---|---|
code |
integer | |
error |
string | |
message |
Array<ValidationErrorDetail> |
Security schemes¶
| Name | Type | Scheme | Description |
|---|---|---|---|
| OAuth2Bearer | http | bearer | JWT token for authentication, obtained from the OIDC provider. |
Tags¶
| Name | Description |
|---|---|
| Search | Endpoints for performing search operations. |
| Ingestion | Endpoints for adding datasets to and removing them from the search index. |
| Monitoring | Endpoints for monitoring the service health. |