Managing Datasource Metadata

Metadata is the set of typed fields attached to a datasource’s content. Good metadata sharpens retrieval, and once a field is indexed it becomes a dimension an execution policy can scope a session by. This guide covers the ways to populate metadata and how to index a field so you can filter and govern by it. For the concepts behind metadata and its link to access control, see Datasources.

Prerequisites

A datasource. See Managing datasources to create one.
Your API key in the MEIBEL_API_KEY environment variable.

Apply a metadata shape

When your documents fit a familiar domain, a prebuilt shape saves you from defining fields by hand. The catalog ships shapes for common document types, including bibliography, legal, medical, and insurance. Browse it to find a shape and note its model_id.

import os
from meibel import MeibelClient

client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

catalog = client.metadata_model_catalog.list()
for entry in catalog.models:
    print(entry.model_id, entry.name)

meibel metadata-model-catalog list

Set the datasource’s metadata configuration to that shape. On the next ingestion the platform extracts the shape’s fields from every document.

from meibel.models import UpdateDatasourceRequest, MetadataConfigRequest

client.datasources.update(
    datasource_id="ds_123",
    body=UpdateDatasourceRequest(
        metadata_config=MetadataConfigRequest(type="catalog", model_id="bibliography"),
    ),
)

meibel metadata-configuration update-config ds_123 --data '{
  "type": "catalog", "model_id": "bibliography"
}'

Define custom fields

When no prebuilt shape fits, define the fields yourself. Each field carries a name, a type, and a description that tells the extractor what to capture. Set index to true on a field to make it filterable, which is what lets you search and scope by it later.

from meibel.models import UpdateDatasourceRequest, MetadataConfigRequest, MetadataField

client.datasources.update(
    datasource_id="ds_123",
    body=UpdateDatasourceRequest(
        metadata_config=MetadataConfigRequest(
            type="custom",
            fields=[
                MetadataField(
                    name="product_line",
                    type="string",
                    description="The product line a document covers, e.g. commercial or consumer",
                    index=True,
                ),
                MetadataField(
                    name="effective_date",
                    type="datetime",
                    description="The date the document takes effect",
                    index=True,
                ),
            ],
        ),
    ),
)

meibel metadata-configuration update-config ds_123 --data '{
  "type": "custom",
  "fields": [
    { "name": "product_line", "type": "string", "description": "The product line a document covers, e.g. commercial or consumer", "index": true },
    { "name": "effective_date", "type": "datetime", "description": "The date the document takes effect", "index": true }
  ]
}'

A field’s type is one of string, integer, float, boolean, datetime, uuid, geo, or list[string].

A fixed number of fields can be indexed per data type, so index the fields you most need to filter and scope by. An unindexed field is still extracted and still describes its data, but cannot be filtered on.

Annotate a data element

Automatic extraction covers most values, but sometimes you know a value the extractor cannot infer, or you need to correct one it got wrong. Set metadata directly on a single data element.

from meibel.models import UpdateDataElementRequest

client.datasources.data_elements.update(
    data_element_id="de_456",
    datasource_id="ds_123",
    body=UpdateDataElementRequest(
        metadata={"product_line": "commercial", "reviewed": True},
    ),
)

meibel data-element-metadata update ds_123 de_456 --data '{
  "metadata": { "product_line": "commercial", "reviewed": true }
}'

A value you set by hand is preserved when the datasource is re-ingested, so a later ingest keeps your annotation rather than overwriting it with a fresh extraction.

Scope access with an indexed field

Indexing is what connects metadata to access control. Once a field is indexed, an execution policy can filter on it, so you can hold each session to the slice of data a given user should see. With product_line indexed, a policy can restrict a session to a single line:

{
  "datasources": {
    "ds_123": {
      "documents": {
        "filter": { "product_line": "commercial" }
      }
    }
  }
}

A session created with this policy retrieves only documents whose product_line is commercial. A field that was never indexed cannot appear in a filter like this, which is why the fields you choose to index set the boundary of what you can govern. For the steps to create and apply policies, see Managing execution policies.

Datasources

How metadata, retrieval, and access control fit together.

Managing Datasources

Create datasources, add content, and trigger ingestion.

Data Elements

List, search, and inspect the data elements metadata attaches to.

Managing Execution Policies

Scope sessions by filtering on indexed metadata fields.

Get started

Tutorials

Core concepts

Agents and sessions

Data and documents

Operations

Support

Managing Datasource Metadata

Prerequisites

Apply a metadata shape

Define custom fields

Annotate a data element

Scope access with an indexed field

Datasources

Managing Datasources

Data Elements

Managing Execution Policies

​Prerequisites

​Apply a metadata shape

​Define custom fields

​Annotate a data element

​Scope access with an indexed field

​Related guides

Datasources

Managing Datasources

Data Elements

Managing Execution Policies

Prerequisites

Apply a metadata shape

Define custom fields

Annotate a data element

Scope access with an indexed field

Related guides