> ## Documentation Index
> Fetch the complete documentation index at: https://docs.meibel.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Managing Datasources

> Create datasources, upload files, trigger ingestion, and manage the data pipeline

A datasource is where Meibel keeps the data your agents draw on. It holds two shapes of content: unstructured documents and structured tables. Uploading a file only begins the process: when a datasource ingests your files, it parses each one, recovers the structure inside it, and extracts metadata as it goes. Your documents become data elements an agent can search by meaning, and your tables, along with other data suited to tabular representation, become queryable by their columns and values. By the time ingestion finishes, an agent bound to the datasource can retrieve from your files directly as it reasons through a task.

This guide walks through the full datasource lifecycle: creating a datasource, uploading files, triggering ingestion, retrieving its details and status, updating it, and deleting it. By the end you will know how to take a datasource from empty to queryable and how to manage it as your content changes. For detailed explanations of what a datasource is and how agents query it, see the [Datasources concept](/concepts/datasources). To learn how to configure the metadata you can search and scope by, see [Managing datasource metadata](/guides/datasource-metadata).

The examples work on one datasource throughout: **Q4 Financial Reports**, which holds quarterly earnings reports and analyst briefings. It starts with a single uploaded PDF, `earnings-q4.pdf`, and each step below acts on that same datasource, so the snippets follow in order as one walkthrough. Set your API key in the `MEIBEL_API_KEY` environment variable before you begin.

## Create a datasource

Every datasource starts empty. You create one with a name and a description to organize it by. You supply its content yourself by uploading files, which the next step covers.

<CodeGroup>
  ```python Python theme={null}
  import os
  from meibel import MeibelClient

  client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

  datasource = client.datasources.create(
      name="Q4 Financial Reports",
      description="Quarterly earnings reports and analyst briefings",
  )

  print(datasource.id, datasource.name)
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient } from 'meibel';

  const client = new MeibelClient({ apiKey: process.env.MEIBEL_API_KEY });

  const datasource = await client.datasources.create({
    name: 'Q4 Financial Reports',
    description: 'Quarterly earnings reports and analyst briefings',
  });

  console.log(datasource.id, datasource.name);
  ```
</CodeGroup>

The response includes the datasource `id`, `name`, `description`, and timestamps. Every operation that follows needs this `id`. The snippets reuse the `datasource` object returned here, so keep it in scope as you work through the steps.

## Upload files

With a datasource in place, add the files whose contents you want agents to reach. Each call uploads one file, so repeat it for every file you want to add. The SDK streams the file to the server in chunks rather than loading it into memory first, so a large document uploads without exhausting memory. Uploading stores a file but does not process it. Its contents become searchable only after ingestion, which the next step triggers.

<CodeGroup>
  ```python Python theme={null}
  with open("earnings-q4.pdf", "rb") as f:
      upload = client.datasources.file_uploads.upload_content(
          datasource_id=datasource.id,
          files=f,
          files_name="earnings-q4.pdf",
      )

  print(upload.upload_id)
  ```

  ```typescript TypeScript theme={null}
  import { readFile } from "node:fs/promises";

  const fileBlob = new Blob([await readFile("earnings-q4.pdf")]);
  const upload = await client.datasources.fileUploads.uploadContent(datasource.id, fileBlob, "earnings-q4.pdf");

  console.log(upload.uploadId);
  ```
</CodeGroup>

<Note>
  You can upload multiple files to the same datasource. Supported formats include PDF, DOCX, XLSX, CSV, TXT, and JSON.
</Note>

## Trigger ingestion

Ingestion is the step that turns uploaded files into queryable content. The pipeline parses each file, breaks documents into data elements, reads structured files into tables, and extracts metadata along the way. Trigger it once your files are in place. You can trigger it again later after adding more files or changing the datasource's metadata configuration, and the pipeline reprocesses the content accordingly.

<CodeGroup>
  ```python Python theme={null}
  result = client.datasources.ingest.trigger(datasource_id=datasource.id)

  print(result.message)
  ```

  ```typescript TypeScript theme={null}
  const result = await client.datasources.ingest.trigger(datasource.id);

  console.log(result.message);
  ```
</CodeGroup>

Ingestion runs asynchronously, so the call returns before processing finishes. Track its progress by polling the datasource's status, described next, or by subscribing to streaming events.

## Get datasource details

Fetching a datasource returns its current state, which is how you check where it stands. The `last_sync_status` field reflects the most recent ingest run, and `total_ingested_files` reports how many files have been ingested. Read the status here to confirm ingestion has finished before you rely on the datasource in an agent.

<CodeGroup>
  ```python Python theme={null}
  current = client.datasources.get(datasource_id=datasource.id)

  print(current.name)
  print(current.last_sync_status)
  print(current.total_ingested_files)
  ```

  ```typescript TypeScript theme={null}
  const current = await client.datasources.get(datasource.id);

  console.log(current.name);
  console.log(current.lastSyncStatus);
  console.log(current.totalIngestedFiles);
  ```
</CodeGroup>

## Update a datasource

Updating changes a datasource's name, description, or configuration after you create it. Here you rename **Q4 Financial Reports** to add the year and record that it now reflects the final audited numbers. The update is partial: only the fields you include in the request body change, and anything you leave out keeps its current value. Changing configuration that affects how content is processed, such as the metadata shape, takes effect on the next ingestion rather than immediately.

<CodeGroup>
  ```python Python theme={null}
  from meibel.models import UpdateDatasourceRequest

  updated = client.datasources.update(
      datasource_id=datasource.id,
      body=UpdateDatasourceRequest(
          name="Q4 2025 Financial Reports",
          description="Updated with final audited numbers",
      ),
  )

  print(updated.name)
  ```

  ```typescript TypeScript theme={null}
  const updated = await client.datasources.update(datasource.id, {
    name: 'Q4 2025 Financial Reports',
    description: 'Updated with final audited numbers',
  });

  console.log(updated.name);
  ```
</CodeGroup>

## Delete a datasource

Deleting removes a datasource along with every file and data element it holds. The removal is permanent, so reserve it for datasources you are sure you no longer need, and check what depends on the datasource before you delete it.

<CodeGroup>
  ```python Python theme={null}
  client.datasources.delete(datasource_id=datasource.id)
  ```

  ```typescript TypeScript theme={null}
  await client.datasources.delete(datasource.id);
  ```
</CodeGroup>

<Warning>
  Deleting a datasource removes all uploaded files and extracted data elements. Agents that reference this datasource will lose access to its content.
</Warning>