Managing Datasources

A datasource is where Meibel keeps the data your agents draw on. It holds two shapes of content: unstructured documents and structured tables. Uploading a file only begins the process: when a datasource ingests your files, it parses each one, recovers the structure inside it, and extracts metadata as it goes. Your documents become data elements an agent can search by meaning, and your tables, along with other data suited to tabular representation, become queryable by their columns and values. By the time ingestion finishes, an agent bound to the datasource can retrieve from your files directly as it reasons through a task. This guide walks through the full datasource lifecycle: creating a datasource, uploading files, triggering ingestion, retrieving its details and status, updating it, and deleting it. By the end you will know how to take a datasource from empty to queryable and how to manage it as your content changes. For detailed explanations of what a datasource is and how agents query it, see the Datasources concept. To learn how to configure the metadata you can search and scope by, see Managing datasource metadata. The examples work on one datasource throughout: Q4 Financial Reports, which holds quarterly earnings reports and analyst briefings. It starts with a single uploaded PDF, earnings-q4.pdf, and each step below acts on that same datasource, so the snippets follow in order as one walkthrough. Set your API key in the MEIBEL_API_KEY environment variable before you begin.

Create a datasource

Every datasource starts empty. You create one with a name and a description to organize it by. You supply its content yourself by uploading files, which the next step covers.

import os
from meibel import MeibelClient

client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

datasource = client.datasources.create(
    name="Q4 Financial Reports",
    description="Quarterly earnings reports and analyst briefings",
)

print(datasource.id, datasource.name)

import { MeibelClient } from 'meibel';

const client = new MeibelClient({ apiKey: process.env.MEIBEL_API_KEY });

const datasource = await client.datasources.create({
  name: 'Q4 Financial Reports',
  description: 'Quarterly earnings reports and analyst briefings',
});

console.log(datasource.id, datasource.name);

The response includes the datasource id, name, description, and timestamps. Every operation that follows needs this id. The snippets reuse the datasource object returned here, so keep it in scope as you work through the steps.

Upload files

With a datasource in place, add the files whose contents you want agents to reach. Each call uploads one file, so repeat it for every file you want to add. The SDK streams the file to the server in chunks rather than loading it into memory first, so a large document uploads without exhausting memory. Uploading stores a file but does not process it. Its contents become searchable only after ingestion, which the next step triggers.

with open("earnings-q4.pdf", "rb") as f:
    upload = client.datasources.file_uploads.upload_content(
        datasource_id=datasource.id,
        files=f,
        files_name="earnings-q4.pdf",
    )

print(upload.upload_id)

import { readFile } from "node:fs/promises";

const fileBlob = new Blob([await readFile("earnings-q4.pdf")]);
const upload = await client.datasources.fileUploads.uploadContent(datasource.id, fileBlob, "earnings-q4.pdf");

console.log(upload.uploadId);

You can upload multiple files to the same datasource. Supported formats include PDF, DOCX, XLSX, CSV, TXT, and JSON.

Trigger ingestion

Ingestion is the step that turns uploaded files into queryable content. The pipeline parses each file, breaks documents into data elements, reads structured files into tables, and extracts metadata along the way. Trigger it once your files are in place. You can trigger it again later after adding more files or changing the datasource’s metadata configuration, and the pipeline reprocesses the content accordingly.

result = client.datasources.ingest.trigger(datasource_id=datasource.id)

print(result.message)

const result = await client.datasources.ingest.trigger(datasource.id);

console.log(result.message);

Ingestion runs asynchronously, so the call returns before processing finishes. Track its progress by polling the datasource’s status, described next, or by subscribing to streaming events.

Get datasource details

Fetching a datasource returns its current state, which is how you check where it stands. The last_sync_status field reflects the most recent ingest run, and total_ingested_files reports how many files have been ingested. Read the status here to confirm ingestion has finished before you rely on the datasource in an agent.

current = client.datasources.get(datasource_id=datasource.id)

print(current.name)
print(current.last_sync_status)
print(current.total_ingested_files)

const current = await client.datasources.get(datasource.id);

console.log(current.name);
console.log(current.lastSyncStatus);
console.log(current.totalIngestedFiles);

Update a datasource

Updating changes a datasource’s name, description, or configuration after you create it. Here you rename Q4 Financial Reports to add the year and record that it now reflects the final audited numbers. The update is partial: only the fields you include in the request body change, and anything you leave out keeps its current value. Changing configuration that affects how content is processed, such as the metadata shape, takes effect on the next ingestion rather than immediately.

from meibel.models import UpdateDatasourceRequest

updated = client.datasources.update(
    datasource_id=datasource.id,
    body=UpdateDatasourceRequest(
        name="Q4 2025 Financial Reports",
        description="Updated with final audited numbers",
    ),
)

print(updated.name)

const updated = await client.datasources.update(datasource.id, {
  name: 'Q4 2025 Financial Reports',
  description: 'Updated with final audited numbers',
});

console.log(updated.name);

Delete a datasource

Deleting removes a datasource along with every file and data element it holds. The removal is permanent, so reserve it for datasources you are sure you no longer need, and check what depends on the datasource before you delete it.

client.datasources.delete(datasource_id=datasource.id)

await client.datasources.delete(datasource.id);

Deleting a datasource removes all uploaded files and extracted data elements. Agents that reference this datasource will lose access to its content.

Get started

Tutorials

Core concepts

Agents and sessions

Data and documents

Operations

Document parsing reference

Support

Managing Datasources

Create a datasource

Upload files

Trigger ingestion

Get datasource details

Update a datasource

Delete a datasource

​Create a datasource

​Upload files

​Trigger ingestion

​Get datasource details

​Update a datasource

​Delete a datasource

Create a datasource

Upload files

Trigger ingestion

Get datasource details

Update a datasource

Delete a datasource