Overview

Welcome to the documentation for the Cross-Dataset Discovery Service.

This service lets users search across DataGEMS datasets with natural language. It provides an API to:

• Search the indexed content of all datasets a user has access to, using keyword (BM25), semantic, or hybrid search, optionally assisted by an LLM.
• Ingest datasets into the search index from their dataset profiles, and remove them again.

How to Use This Documentation

• System: Learn about the high-level Architecture, the Security model, and the Datastores.
• API: Find detailed information on the API endpoints, the status and error codes, and the OpenAPI specification.
• Setup & Monitoring: Get instructions on Deployment, Configuration, Logging, and Maintenance, and see the Automations that build, scan, test and publish the service.
• Developer Guides: Extend the retrieval framework with the Onboarding guide, and see how the service is checked in Quality Assurance.