> ## Documentation Index
> Fetch the complete documentation index at: https://docs.meibel.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Managing Datasources

> Create datasources, upload files, trigger ingestion, and manage the data pipeline

# Managing Datasources

Datasources are containers for the knowledge that powers your agents. A datasource holds uploaded files, manages the ingestion pipeline, and serves as the searchable knowledge base for agent conversations. This guide covers the full datasource lifecycle.

## Create a datasource

Create a new, empty datasource to begin uploading content into.

<CodeGroup>
  ```python Python theme={null}
  import os
  from meibel import MeibelClient
  from meibel.models import CreateDatasourceRequest, ConnectorConfig

  client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

  datasource = client.datasources.create_datasource(body=CreateDatasourceRequest(
      name="Q4 Financial Reports",
      description="Quarterly earnings reports and analyst briefings",
      connector=ConnectorConfig(type="managed"),
  ))

  print(datasource.id, datasource.name)
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient } from 'meibel';

  const client = new MeibelClient({ apiKey: process.env.MEIBEL_API_KEY });

  const datasource = await client.datasources.createDatasource({
    name: 'Q4 Financial Reports',
    description: 'Quarterly earnings reports and analyst briefings',
    connector: { type: 'managed' },
  });

  console.log(datasource.id, datasource.name);
  ```

  ```go Go theme={null}
  import (
      "context"
      "fmt"
      "os"

      meibel "github.com/meibel-ai/meibel-go"
  )

  client := meibel.NewClient(meibel.WithAPIKey(os.Getenv("MEIBEL_API_KEY")))
  ctx := context.Background()

  datasource, err := client.Datasources.CreateDatasource(ctx, meibel.CreateDatasourceRequest{
      Name:        "Q4 Financial Reports",
      Description: "Quarterly earnings reports and analyst briefings",
      Connector:   meibel.ConnectorConfig{Type: "managed"},
  })
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(datasource.ID, datasource.Name)
  ```

  ```bash CLI theme={null}
  meibel datasources create --data '{
    "name": "Q4 Financial Reports",
    "description": "Quarterly earnings reports and analyst briefings",
    "connector": { "type": "managed" }
  }'
  ```
</CodeGroup>

The response includes the datasource `id`, `name`, `description`, and timestamps. Use the `id` for all subsequent operations.

## Upload files

Upload files to a datasource. The SDK streams the file directly to the server in chunks, so files of any size can be uploaded without being fully buffered in memory.

<CodeGroup>
  ```python Python theme={null}
  with open("earnings-q4.pdf", "rb") as f:
      upload = client.content.upload_content(
          file=f,
          file_name="earnings-q4.pdf",
      )

  print(upload.id)
  ```

  ```typescript TypeScript theme={null}
  import { readFile } from "node:fs/promises";

  const fileBlob = new Blob([await readFile("earnings-q4.pdf")]);
  const upload = await client.fileUpload.uploadContent(fileBlob, "earnings-q4.pdf", { datasourceId: datasource.id });

  console.log(upload.id);
  ```

  ```go Go theme={null}
  f, err := os.Open("earnings-q4.pdf")
  if err != nil {
      log.Fatal(err)
  }
  defer f.Close()

  upload, err := client.Content.UploadContent(ctx, f, "earnings-q4.pdf")
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(upload.ID)
  ```

  ```bash CLI theme={null}
  meibel content upload --file earnings-q4.pdf
  ```
</CodeGroup>

<Note>
  You can upload multiple files to the same datasource. Supported formats include PDF, DOCX, XLSX, CSV, TXT, and JSON.
</Note>

## Trigger ingestion

After uploading files, trigger the ingestion pipeline to parse, extract, and index the content into searchable data elements.

<CodeGroup>
  ```python Python theme={null}
  result = client.content.trigger_ingest(datasource_id=datasource.id)

  print(result.status)
  ```

  ```typescript TypeScript theme={null}
  const result = await client.ingest.triggerIngest(datasource.id);

  console.log(result.status);
  ```

  ```go Go theme={null}
  result, err := client.Content.TriggerIngest(ctx, datasource.ID)
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(result.Status)
  ```

  ```bash CLI theme={null}
  meibel content trigger-ingest "$DATASOURCE_ID"
  ```
</CodeGroup>

Ingestion runs asynchronously. You can poll the datasource status or use streaming events to track progress.

## Get datasource details

Retrieve the full metadata and status for a datasource, including ingestion state and file counts.

<CodeGroup>
  ```python Python theme={null}
  datasource = client.datasources.get_datasource(datasource_id="ds_123")

  print(datasource.name)
  print(datasource.status)
  print(datasource.file_count)
  ```

  ```typescript TypeScript theme={null}
  const datasource = await client.datasources.getDatasource('ds_123');

  console.log(datasource.name);
  console.log(datasource.status);
  console.log(datasource.fileCount);
  ```

  ```go Go theme={null}
  datasource, err := client.Datasources.GetDatasource(ctx, "ds_123")
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(datasource.Name)
  fmt.Println(datasource.Status)
  fmt.Println(datasource.FileCount)
  ```

  ```bash CLI theme={null}
  meibel datasources get ds_123
  ```
</CodeGroup>

## Update a datasource

Modify a datasource's name, description, or configuration. Only the fields you include in the request body are changed.

<CodeGroup>
  ```python Python theme={null}
  from meibel.models import UpdateDatasourceRequest

  updated = client.datasources.update_datasource(
      datasource_id="ds_123",
      body=UpdateDatasourceRequest(
          name="Q4 2025 Financial Reports",
          description="Updated with final audited numbers",
      ),
  )

  print(updated.name)
  ```

  ```typescript TypeScript theme={null}
  const updated = await client.datasources.updateDatasource('ds_123', {
    name: 'Q4 2025 Financial Reports',
    description: 'Updated with final audited numbers',
  });

  console.log(updated.name);
  ```

  ```go Go theme={null}
  updated, err := client.Datasources.UpdateDatasource(ctx, "ds_123", meibel.UpdateDatasourceRequest{
      Name:        "Q4 2025 Financial Reports",
      Description: "Updated with final audited numbers",
  })
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(updated.Name)
  ```

  ```bash CLI theme={null}
  meibel datasources update ds_123 --data '{
    "name": "Q4 2025 Financial Reports",
    "description": "Updated with final audited numbers"
  }'
  ```
</CodeGroup>

## Delete a datasource

Permanently remove a datasource and all its uploaded content and data elements. This action cannot be undone.

<CodeGroup>
  ```python Python theme={null}
  client.datasources.delete_datasource(datasource_id="ds_123")
  ```

  ```typescript TypeScript theme={null}
  await client.datasources.deleteDatasource('ds_123');
  ```

  ```go Go theme={null}
  err := client.Datasources.DeleteDatasource(ctx, "ds_123")
  if err != nil {
      log.Fatal(err)
  }
  ```

  ```bash CLI theme={null}
  meibel datasources delete ds_123
  ```
</CodeGroup>

<Warning>
  Deleting a datasource removes all uploaded files and extracted data elements. Agents that reference this datasource will lose access to its content.
</Warning>