> ## Documentation Index
> Fetch the complete documentation index at: https://docs.meibel.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Managing Datasource Metadata
> Apply a metadata shape, define custom fields, annotate data elements, and index the fields you want to search and scope by
Metadata is the set of typed fields attached to a datasource's content. Good metadata sharpens retrieval, and once a field is indexed it becomes a dimension an [execution policy](/concepts/execution-policies) can scope a session by. This guide covers the ways to populate metadata and how to index a field so you can filter and govern by it. For the concepts behind metadata and its link to access control, see [Datasources](/concepts/datasources).
## Prerequisites
* A datasource. See [Managing datasources](/guides/datasources) to create one.
* Your API key in the `MEIBEL_API_KEY` environment variable.
## Apply a metadata shape
When your documents fit a familiar domain, a prebuilt shape saves you from defining fields by hand. The catalog ships shapes for common document types, including bibliography, legal, medical, and insurance. Browse it to find a shape and note its `model_id`.
```python Python theme={null}
import os
from meibel import MeibelClient
client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])
catalog = client.metadata_model_catalog.list()
for entry in catalog.models:
print(entry.model_id, entry.name)
```
```bash CLI theme={null}
meibel metadata-model-catalog list
```
Set the datasource's metadata configuration to that shape. On the next ingestion the platform extracts the shape's fields from every document.
```python Python theme={null}
from meibel.models import UpdateDatasourceRequest, MetadataConfigRequest
client.datasources.update(
datasource_id="ds_123",
body=UpdateDatasourceRequest(
metadata_config=MetadataConfigRequest(type="catalog", model_id="bibliography"),
),
)
```
```bash CLI theme={null}
meibel metadata-configuration update-config ds_123 --data '{
"type": "catalog", "model_id": "bibliography"
}'
```
## Define custom fields
When no prebuilt shape fits, define the fields yourself. Each field carries a name, a type, and a description that tells the extractor what to capture. Set `index` to `true` on a field to make it filterable, which is what lets you search and scope by it later.
```python Python theme={null}
from meibel.models import UpdateDatasourceRequest, MetadataConfigRequest, MetadataField
client.datasources.update(
datasource_id="ds_123",
body=UpdateDatasourceRequest(
metadata_config=MetadataConfigRequest(
type="custom",
fields=[
MetadataField(
name="product_line",
type="string",
description="The product line a document covers, e.g. commercial or consumer",
index=True,
),
MetadataField(
name="effective_date",
type="datetime",
description="The date the document takes effect",
index=True,
),
],
),
),
)
```
```bash CLI theme={null}
meibel metadata-configuration update-config ds_123 --data '{
"type": "custom",
"fields": [
{ "name": "product_line", "type": "string", "description": "The product line a document covers, e.g. commercial or consumer", "index": true },
{ "name": "effective_date", "type": "datetime", "description": "The date the document takes effect", "index": true }
]
}'
```
A field's type is one of `string`, `integer`, `float`, `boolean`, `datetime`, `uuid`, `geo`, or `list[string]`.
A fixed number of fields can be indexed per data type, so index the fields you most need to filter and scope by. An unindexed field is still extracted and still describes its data, but cannot be filtered on.
## Annotate a data element
Automatic extraction covers most values, but sometimes you know a value the extractor cannot infer, or you need to correct one it got wrong. Set metadata directly on a single data element.
```python Python theme={null}
from meibel.models import UpdateDataElementRequest
client.datasources.data_elements.update(
data_element_id="de_456",
datasource_id="ds_123",
body=UpdateDataElementRequest(
metadata={"product_line": "commercial", "reviewed": True},
),
)
```
```bash CLI theme={null}
meibel data-element-metadata update ds_123 de_456 --data '{
"metadata": { "product_line": "commercial", "reviewed": true }
}'
```
A value you set by hand is preserved when the datasource is re-ingested, so a later ingest keeps your annotation rather than overwriting it with a fresh extraction.
## Scope access with an indexed field
Indexing is what connects metadata to access control. Once a field is indexed, an [execution policy](/concepts/execution-policies) can filter on it, so you can hold each session to the slice of data a given user should see. With `product_line` indexed, a policy can restrict a session to a single line:
```json theme={null}
{
"datasources": {
"ds_123": {
"documents": {
"filter": { "product_line": "commercial" }
}
}
}
}
```
A session created with this policy retrieves only documents whose `product_line` is `commercial`. A field that was never indexed cannot appear in a filter like this, which is why the fields you choose to index set the boundary of what you can govern. For the steps to create and apply policies, see [Managing execution policies](/guides/execution-policies).
## Related guides
How metadata, retrieval, and access control fit together.
Create datasources, add content, and trigger ingestion.
List, search, and inspect the data elements metadata attaches to.
Scope sessions by filtering on indexed metadata fields.