Datasources

​

Overview

A datasource is a container that makes your data available to agents. It groups related content, whether files you upload or storage you connect, and prepares that content so an agent can retrieve from it while producing its outputs, whether responding in a conversation or extracting structured data to a schema. A datasource does the preparatory work up front. It parses, indexes, and organizes its contents into a queryable form that an agent reaches through tools. A datasource holds two shapes of data, and it serves each one differently. Unstructured documents such as PDFs and contracts become searchable by meaning. Structured data such as tables and spreadsheets stays queryable by its columns and values. A single datasource can hold both, and an agent bound to it draws on either depending on the task in front of it. This page explains what a datasource contains, how an agent works with it, and how its metadata connects to the execution policies that govern access. For the pipeline that turns raw files into this queryable form, see context engineering.

​

Two shapes of data

The two shapes are queried in different ways, each suited to a different kind of retrieval. Unstructured documents carry meaning in prose: reports, contracts, manuals, correspondence. During ingestion the platform parses each file and breaks it into data elements, the atomic units of content that each carry their own text, source, and metadata. An agent searches these data elements by semantic similarity, so a query about “termination clauses” surfaces the relevant passages even where they never use that exact phrase. This is retrieval over unstructured data, often called RAG. Structured data lives in tables with columns and typed values: order records, financial figures, inventory. Semantic search suits it poorly, because the questions it answers are quantitative. How many orders shipped to the Northeast? What was the average discount last quarter? For these an agent generates a query against the tables and reads the rows that return, which yields the exact aggregates and arithmetic that similarity search does not. The shape decides how an agent gets what it needs. Context engineering covers how raw files become data elements and tables on the way in.

​

Structure-Augmented Generation

Meibel’s approach to retrieval rests on one idea: most data that looks unstructured is in fact structurable. A contract has a clause hierarchy. A report nests sections within sections. A product manual carries specifications buried in its prose. The platform recovers this latent structure during ingestion and uses it to retrieve with more precision than similarity alone reaches. Meibel calls the approach Structure-Augmented Generation, or STAG. Structure appears in both shapes of data. In tables it is explicit, already present in the schema. In documents it is implicit, held in the headings, sections, and ordering that parsing recovers. Treating structure as first-class in both is what lets an agent move between a value in a table and the passage that explains it. The Structure-Augmented Generation post develops the idea in full.

​

How agents work with a datasource

An agent does not query a datasource on a fixed path. It reaches a datasource through tools and decides when and how to use them as it works through a task. Binding a datasource to an agent gives that agent a document-search tool for the datasource’s unstructured content and a structured-query tool for its tables. The agent chooses which to call for a given task, reads what returns, and can call again to refine a search or to bring results from both together. This makes retrieval adapt to the task. Work that needs a figure and the reasoning behind it can lead an agent to query a table, then search the documents for the surrounding context. An agent can bind several datasources at once, each exposed as its own tools, and direct a question to whichever holds the relevant data. That routing across shapes and sources, driven by the agent’s own reasoning, is what STAG’s structure extraction exists to serve. Each of these tool calls runs under the agent’s execution policy. The agent’s freedom in how to retrieve never widens what the session is permitted to see. See agents for how tools and datasource bindings are configured.

​

Metadata

Alongside its content, a datasource carries metadata: typed fields attached to its data, each with a name, a type, and a description. Metadata reaches a datasource in three ways, and one datasource can use all three together. Automatic metadata. Every ingested document carries structural fields the platform records as it parses, including the source filename, page numbers, the section hierarchy, and the kinds of content found on each page. For documents with a page layout, it also records the bounding box of each piece of content, its position on the page, so a retrieved passage can be traced back to the exact region it came from. The platform can generate descriptive fields as well, such as a title and a summary for a document. For structured data, it writes a description for each column from the column’s own values, reading the column’s type, its distinct values, and how often it is empty. Metadata shapes. When a domain has a familiar set of fields, you apply a prebuilt shape instead of defining fields by hand. Academic sources can take a bibliography shape of title, authors, publication date, and DOI; other shapes cover legal, medical, and insurance documents. Where no prebuilt shape fits, you define your own fields. In either case the platform runs an extraction pass over each document and fills the fields from what it finds. Manual annotation. You can set or correct a field on an individual data element by hand, which helps where an extracted value is uncertain or where a value is known only to you. An annotation can be locked, so a later re-ingestion keeps your value rather than overwriting it. Metadata does more than describe its data. Marking a field as indexed turns it into a dimension you can filter on, both in search and in the access rules described next. A fixed number of indexed fields is available for each data type, so indexing is a decision about which fields you most need to query and govern by. An unindexed field still describes its data, but cannot be filtered on.

​

Datasources and execution policies

The fields a datasource exposes are the fields an execution policy can constrain. A policy scopes a session by writing filters over those axes: table columns and the built-in table name for structured data, indexed metadata fields and document identifiers for documents. However a field arrived, through automatic extraction, an applied shape, or your own annotation, once it is indexed it becomes an axis a policy can filter on. A policy can only reference dimensions that the datasource actually exposes, which binds the two together. Structured data brings its columns with it, so a filter on orders.region works as long as the orders table has a region column. Documents differ, because a field has to be extracted and indexed before a policy can use it. Say you add a product_line field to a datasource of product documents and mark it indexed. A session policy can then hold one user to product_line of commercial, so that user’s searches never surface consumer-line material. Leave product_line uncaptured and no policy can draw that boundary, because the field is not there to filter on. This is the practical consequence of the coupling: you can only govern the dimensions you chose to capture. When a session starts, the platform validates the policy against the datasource’s real schema and rejects any filter that references a column or field that does not exist.

​

The datasource lifecycle

A datasource moves through a few stages. It begins empty, created with a name and a description. Content enters it, uploaded directly or pulled from connected storage. Ingestion processes that content into data elements and tables and extracts metadata along the way. Once ingested, the datasource answers queries, both from agents bound to it and directly through the data elements API. Binding it to an agent is what places its tools in the agent’s hands. For the steps to create a datasource, add content, and trigger ingestion, see Managing datasources.

​

Related concepts