# Error Handling
Source: https://docs.meibel.ai/api-ref-guides/error-handling

Handle Meibel API errors: interpret HTTP status codes, parse error responses, and implement retry and recovery patterns.

The API returns standard HTTP status codes to indicate success or failure of requests.

## Error Response Format

```json theme={null}
{
  "code": "not_found",
  "message": "The requested resource was not found"
}
```

## Common Error Codes

| Status | Error Type       | Description                       |
| ------ | ---------------- | --------------------------------- |
| 400    | Bad Request      | Invalid request parameters        |
| 401    | Unauthorized     | Missing or invalid authentication |
| 403    | Forbidden        | Insufficient permissions          |
| 404    | Not Found        | Resource does not exist           |
| 422    | Validation Error | Request validation failed         |
| 429    | Rate Limited     | Too many requests                 |
| 500    | Server Error     | Internal server error             |

## SDK Error Handling

<CodeGroup>
  ```python Python theme={null}
  from meibel import MeibelClient
  from meibel.exceptions import ApiError, NotFoundError, RateLimitError

  client = MeibelClient(api_key="your-api-key")

  try:
      result = client.items.get_item(id="item-123")
  except NotFoundError as e:
      print(f"Item not found: {e.message}")
  except RateLimitError as e:
      print(f"Rate limited. Retry after {e.retry_after} seconds")
  except ApiError as e:
      print(f"API error: {e.status_code} - {e.message}")
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient, ApiError, NotFoundError, RateLimitError } from 'meibel';

  const client = new MeibelClient({ apiKey: 'your-api-key' });

  try {
    const result = await client.items.getItem('item-123');
  } catch (error) {
    if (error instanceof NotFoundError) {
      console.log('Item not found:', error.message);
    } else if (error instanceof RateLimitError) {
      console.log('Rate limited. Retry after:', error.retryAfter, 'seconds');
    } else if (error instanceof ApiError) {
      console.log('API error:', error.statusCode, error.message);
    }
  }
  ```

  ```go Go theme={null}
  import (
      "errors"
      "github.com/meibel-ai/meibel-go/v2"
  )

  client := v2.NewClient(v2.WithAPIKey("your-api-key"))

  result, err := client.Items.GetItem(ctx, "item-123")
  if err != nil {
      var notFoundErr *v2.NotFoundError
      var rateLimitErr *v2.RateLimitError
      var apiErr *v2.APIError

      switch {
      case errors.As(err, &notFoundErr):
          fmt.Println("Item not found:", notFoundErr.Message)
      case errors.As(err, &rateLimitErr):
          fmt.Println("Rate limited. Retry after:", rateLimitErr.RetryAfter)
      case errors.As(err, &apiErr):
          fmt.Println("API error:", apiErr.StatusCode, apiErr.Message)
      default:
          fmt.Println("Unknown error:", err)
      }
  }
  ```
</CodeGroup>

## Retry Strategy

For transient errors (429, 5xx), we recommend implementing exponential backoff:

1. Wait 1 second, then retry
2. If still failing, wait 2 seconds, then retry
3. If still failing, wait 4 seconds, then retry
4. Maximum of 3 retries

<Note>
  The SDKs implement automatic retry with exponential backoff for transient errors.
</Note>


# File Uploads
Source: https://docs.meibel.ai/api-ref-guides/file-uploads

Upload files to the Meibel API with streamed 64 KB chunks, resumable transfers, and endpoints for progress tracking.

The SDKs stream file uploads directly to the server in 64 KB chunks, so even large files are not fully buffered in memory.

## Endpoints

| Endpoint                            | Behaviour                                                                                               |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------- |
| `POST /datasources/uploads`         | Returns immediately after upload. The server processes the file asynchronously. Track progress via SSE. |
| `POST /datasources/uploads/process` | Blocks until the server finishes processing and returns the result in one response.                     |

<Note>
  For files larger than 10 MB, use the async endpoint with progress streaming so your application stays responsive during server-side processing.
</Note>

## Basic Upload

<CodeGroup>
  ```python Python theme={null}
  from meibel import MeibelClient

  client = MeibelClient(api_key="your-api-key")

  # Upload a file to a datasource
  with open("document.pdf", "rb") as f:
      result = client.datasources.file_uploads.upload_content(
          datasource_id="ds_123",
          files=f,
          files_name="document.pdf",
      )

  print(result.upload_id)
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient } from 'meibel';
  import { readFileSync } from 'fs';

  const client = new MeibelClient({ apiKey: 'your-api-key' });

  // Upload a file to a datasource
  const file = new Blob([readFileSync('document.pdf')]);
  const result = await client.datasources.fileUploads.uploadContent('ds_123', file, 'document.pdf');
  console.log(result.uploadId);
  ```

  ```go Go theme={null}
  import v2 "github.com/meibel-ai/meibel-go/v2"

  client := v2.NewClient(v2.WithAPIKey("your-api-key"))
  ctx := context.Background()

  // Upload a file
  f, err := os.Open("document.pdf")
  if err != nil {
      log.Fatal(err)
  }
  defer f.Close()

  result, err := client.Agents.Sessions.SendChatMessageStream(ctx, "session_id_value", f, "document.pdf", nil)
  if err != nil {
      log.Fatal(err)
  }
  fmt.Println(result)
  ```
</CodeGroup>

## Upload with Processing Progress

After starting an upload, stream its progress in real time as the file is transferred and finalized:

<CodeGroup>
  ```python Python theme={null}
  from meibel import MeibelClient

  client = MeibelClient(api_key="your-api-key")

  # Upload the file. It streams directly to the server, never buffered in memory.
  with open("report.pdf", "rb") as f:
      upload = client.datasources.file_uploads.upload_content(
          datasource_id="ds_123",
          files=f,
          files_name="report.pdf",
      )

  print(f"Upload started: {upload.upload_id}")

  # Stream upload progress. Each event is an SSEEvent; parse its JSON payload.
  for event in client.datasources.file_uploads.stream_upload_progress(upload.upload_id):
      payload = event.json()
      if payload["type"] == "file_progress":
          data = payload["data"]
          print(f"{data['filename']}: {data['uploaded_bytes']}/{data['total_bytes']} bytes")
      elif payload["type"] == "file_complete" and payload["data"].get("error"):
          print("Error:", payload["data"]["error"])
      elif payload["type"] == "upload_complete":
          print(payload["data"]["message"])
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient } from 'meibel';
  import { readFileSync } from 'fs';

  const client = new MeibelClient({ apiKey: 'your-api-key' });

  // Upload the file. It streams directly to the server.
  const file = new Blob([readFileSync('report.pdf')]);
  const upload = await client.datasources.fileUploads.uploadContent('ds_123', file, 'report.pdf');

  console.log('Upload started:', upload.uploadId);

  // Stream upload progress. Each event is the parsed payload object.
  for await (const event of client.datasources.fileUploads.streamUploadProgress(upload.uploadId)) {
    if (event.type === 'file_progress') {
      console.log(`${event.data.filename}: ${event.data.uploaded_bytes}/${event.data.total_bytes} bytes`);
    } else if (event.type === 'file_complete' && event.data.error) {
      console.log('Error:', event.data.error);
    } else if (event.type === 'upload_complete') {
      console.log(event.data.message);
    }
  }
  ```

  ```go Go theme={null}
  import v2 "github.com/meibel-ai/meibel-go/v2"

  client := v2.NewClient(v2.WithAPIKey("your-api-key"))
  ctx := context.Background()

  // Upload the file. It streams directly to the server.
  f, err := os.Open("report.pdf")
  if err != nil {
      log.Fatal(err)
  }
  defer f.Close()

  upload, err := client.Agents.Sessions.SendChatMessageStream(ctx, f, "report.pdf")
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println("Upload started:", upload.UploadID)

  // Track server-side processing progress
  stream, err := client.Datasources.FileUploads.StreamUploadProgress(ctx, upload.UploadID)
  if err != nil {
      log.Fatal(err)
  }
  for event := range stream.Events {
      fmt.Println(event)
  }
  ```
</CodeGroup>

## Upload with Synchronous Processing

For smaller files where you don't need progress tracking, use the synchronous endpoint. The file is still streamed to the server, but the request blocks until processing completes and returns the result directly.

<CodeGroup>
  ```python Python theme={null}
  from meibel import MeibelClient

  client = MeibelClient(api_key="your-api-key")

  # Upload a file
  with open("document.pdf", "rb") as f:
      result = client.datasources.file_uploads.upload_and_list_content("datasource_id_value", files=f, files_name="document.pdf")
      print(result)
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient } from 'meibel';

  const client = new MeibelClient({ apiKey: 'your-api-key' });

  import fs from 'fs';

  // Upload a file
  const file = fs.createReadStream('document.pdf');
  const result = await client.datasources.fileUploads.uploadAndListContent('datasource_id_value', file, 'document.pdf');
  console.log(result);
  ```

  ```go Go theme={null}
  import v2 "github.com/meibel-ai/meibel-go/v2"

  client := v2.NewClient(v2.WithAPIKey("your-api-key"))
  ctx := context.Background()

  // Upload a file
  f, err := os.Open("document.pdf")
  if err != nil {
      log.Fatal(err)
  }
  defer f.Close()

  result, err := client.Datasources.FileUploads.UploadAndListContent(ctx, "datasource_id_value", f, "document.pdf", nil)
  if err != nil {
      log.Fatal(err)
  }
  fmt.Println(result)
  ```
</CodeGroup>

## Supported File Types

The API accepts common document formats including PDF, DOCX, XLSX, CSV, TXT, and JSON files. Maximum file size is 100 MB.

## Error Handling

Upload errors are returned as standard API errors. When using progress streaming, errors are delivered as events:

```json theme={null}
{
  "type": "error",
  "message": "Unsupported file format"
}
```


# Pagination
Source: https://docs.meibel.ai/api-ref-guides/pagination

Work with cursor-based pagination in the Meibel API: read next_cursor values, request additional pages, and iterate over long result sets.

Many API endpoints that return lists of items use cursor-based pagination.

## How It Works

Paginated responses include:

* `data` - Array of items for the current page
* `next_cursor` - Cursor for the next page (null if no more pages)

## Manual Pagination

```bash theme={null}
# First request
curl -X GET "https://api.example.com/items?limit=20"

# Response includes next_cursor
# Use it for the next request
curl -X GET "https://api.example.com/items?limit=20&cursor=abc123"
```

## SDK Examples

The SDKs handle pagination automatically using iterators:

<CodeGroup>
  ```python Python theme={null}
  from meibel import MeibelClient

  client = MeibelClient(api_key="your-api-key")

  # Pagination - iterate over all items
  for item in client.agents.list():
      print(item)
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient } from 'meibel';

  const client = new MeibelClient({ apiKey: 'your-api-key' });

  // Pagination - iterate over all items
  for await (const item of client.agents.list()) {
    console.log(item);
  }
  ```

  ```go Go theme={null}
  import v2 "github.com/meibel-ai/meibel-go/v2"

  client := v2.NewClient(v2.WithAPIKey("your-api-key"))
  ctx := context.Background()

  // Pagination - iterate over all items
  iter := client.Agents.List(ctx)
  for iter.Next(ctx) {
      item := iter.Item()
      fmt.Println(item)
  }
  if err := iter.Err(); err != nil {
      log.Fatal(err)
  }
  ```
</CodeGroup>


# Streaming
Source: https://docs.meibel.ai/api-ref-guides/streaming

Consume Server-Sent Events from the Meibel API to stream agent responses, job progress, and long-running results in real time.

Some API endpoints support Server-Sent Events (SSE) for real-time streaming responses.

## How It Works

Streaming endpoints return a stream of events rather than a single response. Each event is a JSON object prefixed with `data:`.

## Event Format

```text theme={null}
data: {"type": "message", "content": "Hello"}

data: {"type": "done"}
```

## SDK Examples

<CodeGroup>
  ```python Python theme={null}
  from meibel import MeibelClient

  client = MeibelClient(api_key="your-api-key")

  # Upload a file
  with open("document.pdf", "rb") as f:
      result = client.agents.sessions.send_chat_message_stream("session_id_value", file=f, file_name="document.pdf")
      print(result)
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient } from 'meibel';

  const client = new MeibelClient({ apiKey: 'your-api-key' });

  import fs from 'fs';

  // Upload a file
  const file = fs.createReadStream('document.pdf');
  const result = await client.agents.sessions.sendChatMessageStream('session_id_value', file, 'document.pdf');
  console.log(result);
  ```

  ```go Go theme={null}
  import v2 "github.com/meibel-ai/meibel-go/v2"

  client := v2.NewClient(v2.WithAPIKey("your-api-key"))
  ctx := context.Background()

  // Upload a file
  f, err := os.Open("document.pdf")
  if err != nil {
      log.Fatal(err)
  }
  defer f.Close()

  result, err := client.Agents.Sessions.SendChatMessageStream(ctx, "session_id_value", f, "document.pdf", nil)
  if err != nil {
      log.Fatal(err)
  }
  fmt.Println(result)
  ```
</CodeGroup>


# Authentication
Source: https://docs.meibel.ai/api-reference/authentication

Authenticate requests to the Meibel API with an API key passed in the Meibel-API-Key header, and manage keys securely in production.

The Meibel API uses API keys to authenticate requests. All API requests must include your API key in the `Meibel-API-Key` header.

## API Keys

Your API key carries many privileges, so be sure to keep it secure. Do not share your API key in publicly accessible areas such as GitHub, client-side code, or in your applications.

### Obtaining an API Key

1. Sign in to your [Meibel Dashboard](https://app.meibel.ai)
2. Navigate to Settings > API Keys
3. Click "Create New API Key"
4. Give your key a descriptive name
5. Copy the key immediately -- you won't be able to see it again

<Warning>
  API keys should be stored securely in environment variables or a secrets management solution, never in your codebase.
</Warning>

## Making Authenticated Requests

All API requests must be made over HTTPS and include a `Meibel-API-Key` header with your API key.

### Using cURL

```bash theme={null}
curl -X GET https://api.meibel.ai/v2/datasources \
  -H "Meibel-API-Key: YOUR_API_KEY"
```

### Using the SDKs

The official SDKs handle authentication for you when provided with an API key:

<CodeGroup>
  ```python Python theme={null}
  from meibel import MeibelClient
  import os

  client = MeibelClient(api_key=os.getenv("MEIBEL_API_KEY"))

  datasources = client.datasources.list()
  ```

  ```python Python (Async) theme={null}
  from meibel import AsyncMeibelClient
  import asyncio
  import os

  async def main():
      client = AsyncMeibelClient(api_key=os.getenv("MEIBEL_API_KEY"))
      datasources = client.datasources.list()
      await client.close()

  asyncio.run(main())
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient } from 'meibel';

  const client = new MeibelClient({
    apiKey: process.env.MEIBEL_API_KEY,
  });

  const datasources = client.datasources.list();
  ```

  ```go Go theme={null}
  import meibelgo "github.com/meibel-ai/meibel-go"

  client := meibelgo.NewClient(
      meibelgo.WithAPIKey(os.Getenv("MEIBEL_API_KEY")),
  )
  ```

  ```bash CLI theme={null}
  export MEIBEL_API_KEY="your-api-key"
  meibel datasources list
  ```
</CodeGroup>

## API Key Security Best Practices

To keep your API keys secure:

<Steps>
  <Step title="Use environment variables">
    Store API keys in environment variables, not in your code
  </Step>

  <Step title="Implement key rotation">
    Rotate your API keys periodically
  </Step>

  <Step title="Set appropriate permissions">
    Create keys with the minimum necessary permissions
  </Step>

  <Step title="Monitor usage">
    Regularly review API key usage in the dashboard
  </Step>

  <Step title="Revoke compromised keys">
    Immediately revoke any keys that may have been compromised
  </Step>
</Steps>

## Troubleshooting Authentication

If you're experiencing authentication issues:

* Ensure your API key is valid and active in the dashboard
* Check that you're using the `Meibel-API-Key` header (not `Authorization`)
* Verify there are no extra spaces or characters in your key
* Confirm your API key has the necessary permissions

If problems persist, contact [support@meibel.ai](mailto:support@meibel.ai) for assistance.


# Errors
Source: https://docs.meibel.ai/api-reference/errors

Understand Meibel API error codes: HTTP status ranges, structured error payloads, common failure modes, and how to recover from them.

The Meibel API uses conventional HTTP response codes to indicate the success or failure of an API request. In general, codes in the `2xx` range indicate success, codes in the `4xx` range indicate an error due to the information provided (e.g., a required parameter was missing), and codes in the `5xx` range indicate an error with our servers.

## HTTP Status Codes

| Code                               | Description                                                                                      |
| ---------------------------------- | ------------------------------------------------------------------------------------------------ |
| 200 - OK                           | The request was successful.                                                                      |
| 201 - Created                      | The resource was successfully created.                                                           |
| 400 - Bad Request                  | The request was unacceptable, often due to missing a required parameter.                         |
| 401 - Unauthorized                 | No valid API key provided.                                                                       |
| 403 - Forbidden                    | The API key doesn't have permissions to perform the request.                                     |
| 404 - Not Found                    | The requested resource doesn't exist.                                                            |
| 422 - Validation Error             | The request was well-formed but was unable to be processed due to semantic errors.               |
| 429 - Too Many Requests            | Too many requests hit the API too quickly. We recommend an exponential backoff of your requests. |
| 500, 502, 503, 504 - Server Errors | Something went wrong on our end. (These are rare.)                                               |

## Error Response Format

All API errors include a JSON response body:

```json theme={null}
{
  "message": "A human-readable error message",
  "detail": [
    {
      "loc": ["body", "field_name"],
      "msg": "Field required",
      "type": "value_error.missing"
    }
  ]
}
```

## Error Handling in SDKs

<CodeGroup>
  ```python Python theme={null}
  from meibel import MeibelClient
  from meibel.exceptions import ApiError, AuthenticationError, NotFoundError
  import os

  client = MeibelClient(api_key=os.getenv("MEIBEL_API_KEY"))

  try:
      result = client.documents.get_result("nonexistent-id")
  except AuthenticationError as e:
      print(f"Invalid API key: {e}")
  except NotFoundError as e:
      print(f"Not found: {e}")
  except ApiError as e:
      print(f"API error ({e.status_code}): {e}")
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient, ApiError, AuthenticationError, NotFoundError } from 'meibel';

  const client = new MeibelClient({ apiKey: process.env.MEIBEL_API_KEY });

  try {
    const result = await client.documents.getResult('nonexistent-id');
  } catch (e) {
    if (e instanceof AuthenticationError) {
      console.error('Invalid API key:', e.message);
    } else if (e instanceof NotFoundError) {
      console.error('Not found:', e.message);
    } else if (e instanceof ApiError) {
      console.error(`API error (${e.statusCode}):`, e.message);
    }
  }
  ```
</CodeGroup>

## Tips for Error Handling

<Note>
  Always implement proper error handling in your application to provide a good user experience and facilitate debugging.
</Note>

<Steps>
  <Step title="Handle expected errors">
    Handle common error cases (401, 404, 422, 429) gracefully in your application
  </Step>

  <Step title="Implement retry logic">
    Use exponential backoff for retrying failed requests, especially for rate limit errors (429)
  </Step>

  <Step title="Log errors">
    Log error details to help with troubleshooting
  </Step>

  <Step title="Provide user feedback">
    Translate API errors into helpful messages for your users
  </Step>
</Steps>

## Getting Help

If you're experiencing persistent errors:

1. Ensure you have the complete error message
2. Contact our support team at [support@meibel.ai](mailto:support@meibel.ai) with details
3. For SDK-specific issues, open an issue on GitHub


# API Reference
Source: https://docs.meibel.ai/api-reference/introduction

Introduction to the Meibel REST API for parsing documents, managing datasources, running agents, and extracting structured data.

Welcome to the Meibel API reference documentation. Our API allows you to interact with all our services programmatically.

## Overview

The Meibel API enables developers to:

* Parse documents into structured markdown with confidence scoring
* Create and manage datasources and data elements for contextual retrieval
* Search and query across your data
* Manage content, metadata, and tag descriptions

## Authentication

All API requests require authentication using an API key sent in the `Meibel-API-Key` header.

```bash theme={null}
curl -X GET https://api.meibel.ai/v2/datasources \
  -H "Meibel-API-Key: YOUR_API_KEY"
```

## Rate Limits

The Meibel API implements rate limiting to ensure fair usage across all users. Rate limits vary by endpoint and are specified in the HTTP headers of API responses:

* `X-RateLimit-Limit`: The maximum number of requests you're permitted to make per time period
* `X-RateLimit-Remaining`: The number of requests remaining in the current rate limit window
* `X-RateLimit-Reset`: The time at which the current rate limit window resets (UTC epoch seconds)

If you exceed the rate limit, an error response returns with status code 429 (Too Many Requests).

## SDK Support

We provide official SDKs to make integration easier:

<CardGroup>
  <Card title="Python" icon="python" href="/sdk/python">
    Sync and async support
  </Card>

  <Card title="TypeScript" icon="js" href="/sdk/nodejs">
    Full type safety with Zod
  </Card>

  <Card title="Go" icon="golang" href="/sdk/go">
    Idiomatic Go interface
  </Card>

  <Card title="CLI" icon="terminal" href="/sdk/cli">
    Command-line interface
  </Card>
</CardGroup>

## Getting Started

To start using the Meibel API:

<Steps>
  <Step title="Sign up for an account">
    Create an account at [app.meibel.ai](https://app.meibel.ai)
  </Step>

  <Step title="Generate an API key">
    Create an API key in the dashboard under Settings > API Keys
  </Step>

  <Step title="Make your first API call">
    Follow the [installation guide](/installation) to make your first API call
  </Step>
</Steps>

<Note>
  All API requests must be made over HTTPS. Calls made over plain HTTP will fail. API requests without authentication will also fail.
</Note>

## Need help?

If you encounter any issues, you can:

* Check the [installation guide](/installation)
* Email our support team at [support@meibel.ai](mailto:support@meibel.ai)
* Open an issue on [GitHub](https://github.com/meibel-ai)


# Rate Limits
Source: https://docs.meibel.ai/api-reference/rate-limits

Meibel API rate limits: per-key thresholds, rate-limit response headers, error codes for throttling, and backoff strategies for clients.

To ensure the stability and availability of our API for all users, Meibel implements rate limiting. This page explains our rate limiting system and how to handle rate limit errors.

## Understanding Rate Limits

Rate limits are applied on a per-API-key basis and vary depending on your subscription plan. Rate limits are calculated based on rolling time windows, typically per minute and per day.

## Rate Limit Headers

Every API response includes headers that provide information about your current rate limit status:

| Header                  | Description                                                                |
| ----------------------- | -------------------------------------------------------------------------- |
| `X-RateLimit-Limit`     | The maximum number of requests allowed in the current time window          |
| `X-RateLimit-Remaining` | The number of requests remaining in the current time window                |
| `X-RateLimit-Reset`     | The time at which the current rate limit window resets (UTC epoch seconds) |

Example headers:

```text theme={null}
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Reset: 1620000000
```

## Rate Limit Errors

When you exceed your rate limit, the API returns a `429 Too Many Requests` error:

```json theme={null}
{
  "message": "Rate limit exceeded. Please retry after 45 seconds."
}
```

The response will include a `Retry-After` header indicating how many seconds to wait.

## Handling Rate Limits

<CodeGroup>
  ```python Python theme={null}
  from meibel import MeibelClient
  from meibel.exceptions import RateLimitError
  import time
  import os

  client = MeibelClient(api_key=os.getenv("MEIBEL_API_KEY"))

  def request_with_retry(fn, *args, max_retries=5, **kwargs):
      for attempt in range(max_retries):
          try:
              return fn(*args, **kwargs)
          except RateLimitError as e:
              if attempt == max_retries - 1:
                  raise
              wait = e.retry_after or (2 ** attempt)
              print(f"Rate limited. Retrying in {wait}s...")
              time.sleep(wait)
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient, RateLimitError } from 'meibel';

  const client = new MeibelClient({ apiKey: process.env.MEIBEL_API_KEY });

  async function requestWithRetry<T>(fn: () => Promise<T>, maxRetries = 5): Promise<T> {
    for (let attempt = 0; attempt < maxRetries; attempt++) {
      try {
        return await fn();
      } catch (e) {
        if (!(e instanceof RateLimitError) || attempt === maxRetries - 1) throw e;
        const wait = e.retryAfter ?? 2 ** attempt;
        console.log(`Rate limited. Retrying in ${wait}s...`);
        await new Promise(r => setTimeout(r, wait * 1000));
      }
    }
    throw new Error('Max retries exceeded');
  }
  ```
</CodeGroup>

## Best Practices

<Steps>
  <Step title="Implement backoff and retry logic">
    When you receive a 429 response, use the `Retry-After` header to determine when to retry. Use exponential backoff as a fallback.
  </Step>

  <Step title="Monitor your usage">
    Check rate limit headers in API responses to stay within your limits.
  </Step>

  <Step title="Batch requests when possible">
    Combine operations to reduce the number of requests.
  </Step>

  <Step title="Cache responses">
    Cache API responses that don't change frequently to reduce request volume.
  </Step>
</Steps>

## Increasing Your Rate Limits

If you need higher rate limits:

1. **Contact sales**: Enterprise customers can request custom rate limits by contacting [sales@meibel.ai](mailto:sales@meibel.ai)
2. **Optimize your usage**: Review our best practices to ensure efficient API usage

<Warning>
  Repeatedly exceeding your rate limits may result in temporary restrictions on your API key. Always implement proper rate limit handling in production applications.
</Warning>


# Create Agent
Source: https://docs.meibel.ai/api-reference/v2/endpoints/agents/create-agent

/api-v2.json post /agents



# Create Session
Source: https://docs.meibel.ai/api-reference/v2/endpoints/agents/create-session

/api-v2.json post /agents/{agent_id}/sessions



# Create Session By Name
Source: https://docs.meibel.ai/api-reference/v2/endpoints/agents/create-session-by-name

/api-v2.json post /agents/name/{name}/sessions
Start a session against the latest published version of an agent by name.

Resolves the current latest published version at runtime — callers do not
need to track a specific agent ID or version. Returns 404 if no published
version exists for the given agent name.



# Delete Agent
Source: https://docs.meibel.ai/api-reference/v2/endpoints/agents/delete-agent

/api-v2.json delete /agents/{agent_id}



# Get Agent
Source: https://docs.meibel.ai/api-reference/v2/endpoints/agents/get-agent

/api-v2.json get /agents/{agent_id}



# List Agent Versions
Source: https://docs.meibel.ai/api-reference/v2/endpoints/agents/list-agent-versions

/api-v2.json get /agents/{agent_id}/versions



# List Agents
Source: https://docs.meibel.ai/api-reference/v2/endpoints/agents/list-agents

/api-v2.json get /agents



# List Sessions
Source: https://docs.meibel.ai/api-reference/v2/endpoints/agents/list-sessions

/api-v2.json get /agents/{agent_id}/sessions



# Publish Agent
Source: https://docs.meibel.ai/api-reference/v2/endpoints/agents/publish-agent

/api-v2.json post /agents/{agent_id}/publish



# Update Agent
Source: https://docs.meibel.ai/api-reference/v2/endpoints/agents/update-agent

/api-v2.json put /agents/{agent_id}



# Create Artifact Schema
Source: https://docs.meibel.ai/api-reference/v2/endpoints/artifact-schemas/create-artifact-schema

/api-v2.json post /artifact-schemas



# Delete Artifact Schema
Source: https://docs.meibel.ai/api-reference/v2/endpoints/artifact-schemas/delete-artifact-schema

/api-v2.json delete /artifact-schemas/{artifact_id}



# Get Artifact Schema
Source: https://docs.meibel.ai/api-reference/v2/endpoints/artifact-schemas/get-artifact-schema

/api-v2.json get /artifact-schemas/{artifact_id}



# List Artifact Schemas
Source: https://docs.meibel.ai/api-reference/v2/endpoints/artifact-schemas/list-artifact-schemas

/api-v2.json get /artifact-schemas



# Update Artifact Schema
Source: https://docs.meibel.ai/api-reference/v2/endpoints/artifact-schemas/update-artifact-schema

/api-v2.json put /artifact-schemas/{artifact_id}



# Create Batch Definition
Source: https://docs.meibel.ai/api-reference/v2/endpoints/batch-definitions/create-batch-definition

/api-v2.json post /batch-definitions



# Delete Batch Definition By Id
Source: https://docs.meibel.ai/api-reference/v2/endpoints/batch-definitions/delete-batch-definition-by-id

/api-v2.json delete /batch-definitions/id/{definition_id}



# Execute Batch Definition
Source: https://docs.meibel.ai/api-reference/v2/endpoints/batch-definitions/execute-batch-definition

/api-v2.json post /batch-definitions/id/{definition_id}/execute



# Get Batch Definition By Catalog Urn
Source: https://docs.meibel.ai/api-reference/v2/endpoints/batch-definitions/get-batch-definition-by-catalog-urn

/api-v2.json get /batch-definitions/catalog-urn



# Get Batch Definition By Id
Source: https://docs.meibel.ai/api-reference/v2/endpoints/batch-definitions/get-batch-definition-by-id

/api-v2.json get /batch-definitions/id/{definition_id}



# List Batch Definition Versions
Source: https://docs.meibel.ai/api-reference/v2/endpoints/batch-definitions/list-batch-definition-versions

/api-v2.json get /batch-definitions/id/{definition_id}/versions



# List Batch Definitions
Source: https://docs.meibel.ai/api-reference/v2/endpoints/batch-definitions/list-batch-definitions

/api-v2.json get /batch-definitions



# Update Batch Definition By Id
Source: https://docs.meibel.ai/api-reference/v2/endpoints/batch-definitions/update-batch-definition-by-id

/api-v2.json put /batch-definitions/id/{definition_id}



# Cancel Batch Execution
Source: https://docs.meibel.ai/api-reference/v2/endpoints/batch-executions/cancel-batch-execution

/api-v2.json post /batch-executions/id/{execution_id}/cancel



# Create Batch Execution
Source: https://docs.meibel.ai/api-reference/v2/endpoints/batch-executions/create-batch-execution

/api-v2.json post /batch-executions



# Get Batch Execution By Id
Source: https://docs.meibel.ai/api-reference/v2/endpoints/batch-executions/get-batch-execution-by-id

/api-v2.json get /batch-executions/id/{execution_id}



# Get Batch Realtime Progress
Source: https://docs.meibel.ai/api-reference/v2/endpoints/batch-executions/get-batch-realtime-progress

/api-v2.json get /batch-executions/id/{execution_id}/realtime-progress



# List Batch Executions
Source: https://docs.meibel.ai/api-reference/v2/endpoints/batch-executions/list-batch-executions

/api-v2.json get /batch-executions



# Retry Failed Items
Source: https://docs.meibel.ai/api-reference/v2/endpoints/batch-executions/retry-failed-items

/api-v2.json post /batch-executions/id/{execution_id}/retry-failed



# Update Batch Execution By Id
Source: https://docs.meibel.ai/api-reference/v2/endpoints/batch-executions/update-batch-execution-by-id

/api-v2.json put /batch-executions/id/{execution_id}



# Get a scoring job
Source: https://docs.meibel.ai/api-reference/v2/endpoints/confidence-scoring/get-a-scoring-job

/api-v2.json get /confidence-scoring/job/{job_id}
Retrieve a single confidence scoring job by its ID, including its current status and score if completed.



# Get agent scoring summary
Source: https://docs.meibel.ai/api-reference/v2/endpoints/confidence-scoring/get-agent-scoring-summary

/api-v2.json get /confidence-scoring/summary/agent/{agent_name}
Get an aggregated summary of confidence scores for a specific agent.



# Get agent session scoring summary
Source: https://docs.meibel.ai/api-reference/v2/endpoints/confidence-scoring/get-agent-session-scoring-summary

/api-v2.json get /confidence-scoring/summary/agent/{agent_name}/session/{session_id}
Get an aggregated summary of confidence scores for a specific agent session.



# List scoring jobs
Source: https://docs.meibel.ai/api-reference/v2/endpoints/confidence-scoring/list-scoring-jobs

/api-v2.json get /confidence-scoring/jobs
List confidence scoring jobs, optionally filtered by identity context fields. All filters are combined with AND logic.



# Get Data Element
Source: https://docs.meibel.ai/api-reference/v2/endpoints/data-elements/get-data-element

/api-v2.json get /datasources/{datasource_id}/data-elements/{data_element_id}



# List Data Elements
Source: https://docs.meibel.ai/api-reference/v2/endpoints/data-elements/list-data-elements

/api-v2.json get /datasources/{datasource_id}/data-elements



# Search Data Elements
Source: https://docs.meibel.ai/api-reference/v2/endpoints/data-elements/search-data-elements

/api-v2.json post /datasources/{datasource_id}/data-elements/search



# Update Data Element
Source: https://docs.meibel.ai/api-reference/v2/endpoints/data-elements/update-data-element

/api-v2.json put /datasources/{datasource_id}/data-elements/{data_element_id}



# Create Download Job (async)
Source: https://docs.meibel.ai/api-reference/v2/endpoints/datasource-downloads/create-download-job-async

/api-v2.json post /datasources/{datasource_id}/downloads



# Download File
Source: https://docs.meibel.ai/api-reference/v2/endpoints/datasource-downloads/download-file

/api-v2.json get /datasources/{datasource_id}/downloads/{job_id}/file



# Process Download (sync)
Source: https://docs.meibel.ai/api-reference/v2/endpoints/datasource-downloads/process-download-sync

/api-v2.json post /datasources/{datasource_id}/downloads/process



# Stream Download Progress
Source: https://docs.meibel.ai/api-reference/v2/endpoints/datasource-downloads/stream-download-progress

/api-v2.json get /datasources/{datasource_id}/downloads/{job_id}/progress



# Create Datasource
Source: https://docs.meibel.ai/api-reference/v2/endpoints/datasources/create-datasource

/api-v2.json post /datasources



# Delete Datasource
Source: https://docs.meibel.ai/api-reference/v2/endpoints/datasources/delete-datasource

/api-v2.json delete /datasources/{datasource_id}



# Get Datasource
Source: https://docs.meibel.ai/api-reference/v2/endpoints/datasources/get-datasource

/api-v2.json get /datasources/{datasource_id}



# List Datasources
Source: https://docs.meibel.ai/api-reference/v2/endpoints/datasources/list-datasources

/api-v2.json get /datasources



# Update Datasource
Source: https://docs.meibel.ai/api-reference/v2/endpoints/datasources/update-datasource

/api-v2.json put /datasources/{datasource_id}



# Download a deep-transform artifact
Source: https://docs.meibel.ai/api-reference/v2/endpoints/documents/download-a-deep-transform-artifact

/api-v2.json get /documents/deep-transform/{job_id}/artifact/{name}
Download a named artifact (e.g. output.json) produced by a succeeded job. Ownership is verified against the customer header before any bytes are returned.



# Get deep-transform job status
Source: https://docs.meibel.ai/api-reference/v2/endpoints/documents/get-deep-transform-job-status

/api-v2.json get /documents/deep-transform/{job_id}
Check status and, once succeeded, the list of downloadable artifacts.



# Get document parsing status
Source: https://docs.meibel.ai/api-reference/v2/endpoints/documents/get-document-parsing-status

/api-v2.json get /documents/{job_id}
Check the status of a document parsing job, including progress statistics.



# Get parsed document result
Source: https://docs.meibel.ai/api-reference/v2/endpoints/documents/get-parsed-document-result

/api-v2.json get /documents/{job_id}/result
Download the parsed result of a completed document parsing job.



# Get structured parse result
Source: https://docs.meibel.ai/api-reference/v2/endpoints/documents/get-structured-parse-result

/api-v2.json get /documents/{job_id}/structured
Download the fully structured parse result (the json format): pages, typed elements, tables, chart data, chart OCR text, and bounding boxes. The response schema (StructuredDocument) is defined by the parsing engine and hoisted into this spec by the OpenAPI generator.



# List child documents
Source: https://docs.meibel.ai/api-reference/v2/endpoints/documents/list-child-documents

/api-v2.json get /documents/{job_id}/children
For container files (ZIP, TAR, EML), list the child documents extracted from the container.



# List deep-transform jobs
Source: https://docs.meibel.ai/api-reference/v2/endpoints/documents/list-deep-transform-jobs

/api-v2.json get /documents/deep-transform
List the calling customer's deep-transform jobs, newest first. Scoped to the customer (and project, when a project header is set). Paginated via `offset`/`limit`.



# Move documents into a datasource (async)
Source: https://docs.meibel.ai/api-reference/v2/endpoints/documents/move-documents-into-a-datasource-async

/api-v2.json post /documents/move
Move documents (identified by their parse job IDs, e.g. the job_id returned by parseDocument) into an existing datasource or a newly created one. Returns a workflow_id to poll for completion.



# Parse a document (async)
Source: https://docs.meibel.ai/api-reference/v2/endpoints/documents/parse-a-document-async

/api-v2.json post /documents
Upload a document for asynchronous parsing. Returns a job ID to track progress.



# Parse a document (sync)
Source: https://docs.meibel.ai/api-reference/v2/endpoints/documents/parse-a-document-sync

/api-v2.json post /documents/process
Upload a document and block until parsing is complete. Returns the full parsed result.



# Stream document parsing trace
Source: https://docs.meibel.ai/api-reference/v2/endpoints/documents/stream-document-parsing-trace

/api-v2.json get /documents/{job_id}/trace
Subscribe to real-time parsing progress via Server-Sent Events.



# Submit a deep-transform extraction from a file upload (async)
Source: https://docs.meibel.ai/api-reference/v2/endpoints/documents/submit-a-deep-transform-extraction-from-a-file-upload-async

/api-v2.json post /documents/deep-transform
Upload a document and submit an extraction against a JSON schema, returning immediately with a job id. To reuse an already-parsed document instead of uploading, use POST /documents/deep-transform/from-document. Poll status via GET /documents/deep-transform/{job_id} and download artifacts once it succeeds. Submission is idempotent on the (document, schema) pair.



# Submit a deep-transform extraction reusing a parsed document (async)
Source: https://docs.meibel.ai/api-reference/v2/endpoints/documents/submit-a-deep-transform-extraction-reusing-a-parsed-document-async

/api-v2.json post /documents/deep-transform/from-document
Submit an extraction that reuses an already-parsed document (by `document_job_id` from POST /documents) instead of re-parsing an upload. Returns immediately with a job id. Poll status via GET /documents/deep-transform/{job_id} and download artifacts once it succeeds. Submission is idempotent on the (document, schema) pair.



# Submit a document transform (async)
Source: https://docs.meibel.ai/api-reference/v2/endpoints/documents/submit-a-document-transform-async

/api-v2.json post /documents/transform/submit
Upload a document for AI-powered extraction and return immediately. Poll for completion via client.sessions.get(execution_id).



# Transform a document using AI extraction (sync)
Source: https://docs.meibel.ai/api-reference/v2/endpoints/documents/transform-a-document-using-ai-extraction-sync

/api-v2.json post /documents/transform
Upload a document for AI-powered structured extraction and block until complete. The file is uploaded to cloud storage and processed by a system agent.



# Create Execution Policy
Source: https://docs.meibel.ai/api-reference/v2/endpoints/execution-policies/create-execution-policy

/api-v2.json post /execution-policies



# Delete Execution Policy
Source: https://docs.meibel.ai/api-reference/v2/endpoints/execution-policies/delete-execution-policy

/api-v2.json delete /execution-policies/{policy_id}



# Get Execution Policy
Source: https://docs.meibel.ai/api-reference/v2/endpoints/execution-policies/get-execution-policy

/api-v2.json get /execution-policies/{policy_id}



# List Execution Policies
Source: https://docs.meibel.ai/api-reference/v2/endpoints/execution-policies/list-execution-policies

/api-v2.json get /execution-policies



# Update Execution Policy
Source: https://docs.meibel.ai/api-reference/v2/endpoints/execution-policies/update-execution-policy

/api-v2.json put /execution-policies/{policy_id}



# List Content
Source: https://docs.meibel.ai/api-reference/v2/endpoints/file-upload/list-content

/api-v2.json get /datasources/{datasource_id}/content



# Stream Upload Progress
Source: https://docs.meibel.ai/api-reference/v2/endpoints/file-upload/stream-upload-progress

/api-v2.json get /datasources/uploads/{upload_id}/progress



# Upload Content (async)
Source: https://docs.meibel.ai/api-reference/v2/endpoints/file-upload/upload-content-async

/api-v2.json post /datasources/{datasource_id}/content



# Upload Content (sync)
Source: https://docs.meibel.ai/api-reference/v2/endpoints/file-upload/upload-content-sync

/api-v2.json post /datasources/{datasource_id}/content/process



# Get Ingest Status
Source: https://docs.meibel.ai/api-reference/v2/endpoints/ingest/get-ingest-status

/api-v2.json get /datasources/{datasource_id}/ingest-status



# Trigger Ingest
Source: https://docs.meibel.ai/api-reference/v2/endpoints/ingest/trigger-ingest

/api-v2.json post /datasources/{datasource_id}/trigger-ingest



# Get Metadata Model Catalog Entry
Source: https://docs.meibel.ai/api-reference/v2/endpoints/metadata-model-catalog/get-metadata-model-catalog-entry

/api-v2.json get /metadata-model-catalog/{model_id}



# List Metadata Model Catalog
Source: https://docs.meibel.ai/api-reference/v2/endpoints/metadata-model-catalog/list-metadata-model-catalog

/api-v2.json get /metadata-model-catalog



# Get Session
Source: https://docs.meibel.ai/api-reference/v2/endpoints/sessions/get-session

/api-v2.json get /sessions/{session_id}



# Get Session Messages
Source: https://docs.meibel.ai/api-reference/v2/endpoints/sessions/get-session-messages

/api-v2.json get /sessions/{session_id}/messages



# Send a chat message with file attachments and stream the response via SSE
Source: https://docs.meibel.ai/api-reference/v2/endpoints/sessions/send-a-chat-message-with-file-attachments-and-stream-the-response-via-sse

/api-v2.json post /sessions/{session_id}/chat/stream



# Send Chat Message
Source: https://docs.meibel.ai/api-reference/v2/endpoints/sessions/send-chat-message

/api-v2.json post /sessions/{session_id}/chat



# List Columns
Source: https://docs.meibel.ai/api-reference/v2/endpoints/table-descriptions/list-columns

/api-v2.json get /datasources/{datasource_id}/tables/{table_name}/columns



# List Tables
Source: https://docs.meibel.ai/api-reference/v2/endpoints/table-descriptions/list-tables

/api-v2.json get /datasources/{datasource_id}/tables



# Update Column Descriptions
Source: https://docs.meibel.ai/api-reference/v2/endpoints/table-descriptions/update-column-descriptions

/api-v2.json put /datasources/{datasource_id}/tables/{table_name}/columns



# Update Table Descriptions
Source: https://docs.meibel.ai/api-reference/v2/endpoints/table-descriptions/update-table-descriptions

/api-v2.json put /datasources/{datasource_id}/tables



# Agents
Source: https://docs.meibel.ai/concepts/agents

How Meibel agents use your context to reason, generate artifacts, and chat with users

## Overview

Agents are the primary interface for AI-powered interactions on Meibel. An agent combines three things: **what to know** (datasource bindings), **how to think** (system prompt and configuration), and **what to produce** (artifact schemas).

When a user sends a message to an agent, the agent searches its bound datasources for relevant context, applies its system prompt to frame its reasoning, and generates a response. If an artifact schema is attached, the agent also produces structured output conforming to that schema.

Agents are not stateless functions — they operate within sessions that maintain conversation history, and they follow a versioning model that gives you reproducibility and safe rollback.

## Agent Configuration

An agent is defined by:

* **Name and description** — human-readable identifiers for the agent
* **System prompt** — instructions that define the agent's behavior, tone, and constraints. This is the most important configuration: it tells the agent what role it plays and how it should respond.
* **Datasource bindings** — which datasources the agent can search for context. An agent with no datasources has no domain knowledge; it can only use its base model capabilities. An agent with well-curated datasources can answer domain-specific questions accurately.
* **Artifact schemas** — optional structured output definitions (see below)
* **Prompt templates** — optional reusable prompt configurations

The system prompt is where you encode domain expertise. A well-written system prompt for a legal review agent might specify: "You are a contract analyst. When asked about a contract, cite specific clauses by section number. If a clause is ambiguous, flag it explicitly. Never fabricate clause numbers."

## Versioning

Agents follow a **draft-publish** workflow:

1. **Create or edit a draft** — make changes to the agent's configuration, system prompt, or datasource bindings
2. **Publish** — freeze the current draft as an immutable version with a version number

Published versions are immutable. The same version number always produces the same behavior (given the same inputs and datasource state). This gives you:

* **Reproducibility** — you can point to a specific version and know exactly how it was configured
* **Safe rollback** — if a new version performs poorly, revert to a previous one
* **Audit trail** — each version captures the full configuration at the time of publication

```python theme={null}
from meibel.models import PublishAgentDefinitionRequest

# Publish the current draft
version = client.agents.publish(
    agent_id="agent_abc123",
    body=PublishAgentDefinitionRequest(commit_message="Publish current draft"),
)
print(f"Published version: {version.version}")

# List all versions
versions = client.agents.list_versions(agent_id="agent_abc123")
for v in versions:
    print(f"  v{v.version}: published {v.created_at}")
```

## Sessions

A session is a conversation container bound to an agent. When you create a session, you get a unique session ID that you use for all subsequent messages in that conversation.

Sessions maintain message history, so the agent has conversational context. The third message in a session can reference something discussed in the first message — the agent sees the full history when generating each response.

```python theme={null}
# Create a session
session = client.agents.sessions.create(agent_id="agent_abc123")

# Send messages within the session
response = client.agents.sessions.send_chat_message(
    session_id=session.session_id,
    body=ChatMessageRequest(user_message="What are the payment terms in the contract?"),
)
print(response.assistant_response)

# Follow up — the agent remembers the previous exchange
response = client.agents.sessions.send_chat_message(
    session_id=session.session_id,
    body=ChatMessageRequest(user_message="Are there any penalties for late payment?"),
)
```

Sessions can also be listed and inspected after the fact, which is useful for auditing and debugging.

## Chat

Agent chat supports two modes:

<Tabs>
  <Tab title="Synchronous">
    Send a message and wait for the full response. Simple and straightforward — best for backend integrations where latency tolerance is higher.

    ```python theme={null}
    from meibel.models import ChatMessageRequest

    response = client.agents.sessions.send_chat_message(
        session_id=session.session_id,
        body=ChatMessageRequest(user_message="Summarize the key risks."),
    )
    print(response.assistant_response)
    print(f"Tokens used: {response.token_usage}")
    ```
  </Tab>

  <Tab title="Streaming">
    Receive response tokens as they are generated via Server-Sent Events (SSE). Best for user-facing interfaces where you want to display the response progressively.

    ```python theme={null}
    stream = client.agents.sessions.send_chat_message_stream(
        session_id=session.session_id,
        body=ChatMessageRequest(user_message="Summarize the key risks."),
    )
    for event in stream:
        print(event, end="", flush=True)
    ```
  </Tab>
</Tabs>

Both modes return the same information — the assistant's message, token usage, suggested actions, and tool activity. Streaming just delivers it incrementally.

## Artifacts and Schemas

Agents can produce **structured outputs** defined by artifact schemas. An artifact schema specifies the JSON structure the agent should fill when generating a response.

This is particularly useful for extraction tasks. Instead of asking an agent "what are the key terms in this contract?" and parsing the free-text response, you define a schema:

```json theme={null}
{
  "type": "object",
  "properties": {
    "effective_date": { "type": "string", "description": "Contract effective date" },
    "termination_date": { "type": "string", "description": "Contract end date" },
    "total_value": { "type": "number", "description": "Total contract value" },
    "key_obligations": {
      "type": "array",
      "items": { "type": "string" },
      "description": "List of key obligations"
    }
  }
}
```

The agent fills this schema from the document context, giving you structured data you can store, compare, or feed into downstream systems without parsing natural language.

Artifact schemas are managed independently from agents — you create a schema once and attach it to any agent that needs it.

## Prompt Templates

Prompt templates are reusable prompt configurations that can be attached to agents. They are useful when multiple agents need similar instructions — for example, a set of agents that all need to follow the same citation format, or a common preamble about your organization's policies.

Instead of duplicating the same text across multiple agent system prompts, define it once as a template and reference it. When you update the template, all agents using it pick up the change.

```python theme={null}
# List available prompt templates
templates = client.prompts.list_prompts()
for t in templates.data:
    print(f"  {t.name}: {t.prompt_id}")
```

## Observability

Understanding how an agent arrived at its response is critical for debugging, quality assurance, and building trust in AI outputs. Meibel provides several layers of observability:

**Session message history** — the full conversation log for any session. You can review exactly what the user asked and what the agent responded, including intermediate exchanges.

**Token usage** — every response includes token counts, so you can track cost and monitor for unexpectedly long responses that might indicate the agent is struggling.

**Tool activity** — when an agent uses tools (searching datasources, running retrievals), the tool activity log shows what it did and why. This tells you which datasources were searched, what queries were used, and what results came back.

**Trace events** — for document processing operations, trace events provide a step-by-step record of the processing pipeline. You can see how a document was parsed, what structures were detected, and how data elements were extracted.

<Tip>
  Include `include_tool_activity=True` in your chat request to get tool activity logs alongside the response. This is the fastest way to debug cases where the agent gives an unexpected answer — check what context it actually retrieved.
</Tip>

```python theme={null}
response = client.agents.sessions.send_chat_message(
    session_id=session.session_id,
    body=ChatMessageRequest(
        user_message="What is the contract value?",
        include_tool_activity=True,
    ),
)

# Inspect what the agent did
print(f"Answer: {response.assistant_response}")
print(f"Tokens: {response.token_usage}")
if response.tool_activity:
    for activity in response.tool_activity:
        print(f"  Tool: {activity}")
```


# Batch Jobs
Source: https://docs.meibel.ai/concepts/batch-jobs

How Meibel runs an agent across many inputs in parallel, using the same configuration developed on a single input

## Overview

A batch job runs a single agent across many inputs at once. The inputs come from a datasource: the agent processes every data element in it independently, and each result is written to an output datasource.

The common case is applying one agent to a large set of documents: extracting fields from every invoice in a folder, or classifying a backlog of support tickets. Running an agent over that many inputs by hand means building the surrounding machinery: a loop over the inputs, a limit on how many run at once, retries for failures, collection of the results, and storage for later use. A batch job provides that machinery, leaving the agent and the datasource as the only things to specify.

The agent runs in batch exactly as it was configured for a single input. The configuration you developed and tuned on one document runs unchanged across the entire set.

## Running the same agent

When a batch definition is created, Meibel resolves the named agent and pins a snapshot of its full configuration into the definition: its system prompt, model, tools, artifact schemas, and confidence scoring modules. Every execution of that definition runs against that snapshot.

This is useful because of how agents are developed. An agent is tuned on individual inputs, its output checked and its configuration adjusted until it performs well. A batch job runs that same configuration, so the agent behaves in batch as it did during development. A single agent definition serves both interactive runs and batches.

Pinning also makes a run reproducible. A definition retains its snapshot even after the underlying agent changes, so a past execution reflects the agent as it was configured at the time. When a batch should use a newer version of the agent, updating the definition re-pins the snapshot and records a new version of the definition.

## The batch definition

A batch job has two parts. A **batch definition** is the reusable configuration: a catalog object with an ID, a name, and a version history. An **execution** is one run of a definition against its inputs. A single definition can have many executions, and each execution links back to the definition it ran, so any result can be traced to the configuration that produced it.

A definition records:

* **The agent** to run on each input, pinned at creation as described above.
* **The input datasource** whose data elements become the run's inputs.
* **Filters** to narrow that set, by name pattern, by content type (for example, only PDFs), or by an explicit list of element IDs. Without filters, every data element is processed.
* **The output datasource** where results are written. If it is unset, each execution creates a new one.
* **A user message**, an optional instruction sent to the agent alongside each input, the same way a message accompanies a document in an interactive run.
* **Concurrency and a retry limit**: how many inputs run at once, and how many times a failing input is retried before the failure is treated as final. These default to 10 and 2.

Editing a definition does not overwrite it. The edit creates a new version linked to the previous one, and earlier versions stay readable. A definition cannot be deleted while an execution still references it, so the configuration behind a past result remains available.

## How an execution runs

An execution proceeds in three stages: it reads the inputs, runs the agent on them in parallel, and writes the results.

**Reading the inputs.** The execution lists the data elements in the input datasource, applies the definition's filters, and confirms that both datasources belong to the same project before any work starts.

<Note>
  A batch job reads the parsed form of each document, which a datasource produces during ingest. The input datasource must have completed ingest at least once before a batch can run against it. If it has not, the execution stops at this stage and reports the missing ingest as a single error, rather than failing every item in turn.
</Note>

**Running in parallel.** The platform processes the inputs concurrently, up to the limit set on the definition, each one a separate run of the pinned agent with its data element attached as the document to work on. The runs are independent, so a failure on one input does not affect the others.

**Writing the results.** When a run finishes, the platform writes its artifacts to the output datasource, tagged with the source document. An input is counted as succeeded only after its artifacts are written, so a run that produces no artifacts is recorded as a failure. The platform then ingests the output datasource, so the results become data elements like any other. They can be searched, fed to another agent, or used as the input to a further batch job.

## Progress and failure handling

A batch job runs asynchronously. Rather than returning a result inline, an execution reports its progress while it runs: the total number of inputs, how many are in progress, how many have succeeded, how many are waiting for a retry, and how many have failed after their retries are exhausted. A running total of retry attempts is also reported.

Failures are handled at two levels. Within an execution, the platform retries a failing input automatically, up to the definition's retry limit. This absorbs transient failures, such as a model being briefly unavailable or a tool call timing out, where a later attempt is likely to succeed. An input still failing after its retries are exhausted is recorded as a failure for that execution.

After an execution finishes, its failed items can be rerun. This starts a new execution against the same definition, restricted to the inputs that failed, and leaves the successful inputs unchanged. It applies when the failures were transient and have since passed, or when the configuration has been corrected.

A running execution can also be canceled. Inputs that have already finished keep their results, and inputs that have not started do not begin.

## Related concepts

Batch jobs build on other parts of the platform:

<CardGroup>
  <Card title="Agents" icon="robot" href="/concepts/agents">
    The agent a batch job runs, including its versioning model and artifact schemas.
  </Card>

  <Card title="Confidence Scoring" icon="chart-bar" href="/concepts/confidence-scoring">
    The scoring that runs as the agent works, in batch as in any other run.
  </Card>
</CardGroup>


# Confidence Scoring
Source: https://docs.meibel.ai/concepts/confidence-scoring

How Meibel evaluates the quality of an agent's work at each step using configurable judged and statistical modules

## Overview

Meibel's confidence scoring system gives clear evaluations of the quality of an AI workflow at each step of the pipeline. It works by running configurable combinations of 15 individual modules, each built and tuned by Meibel to evaluate a particular type of performance or quality of output. Once enabled on an agent, these modules run asynchronously as the agent performs its work. Confidence Scoring evaluates the agent's own outputs, both its direct responses and its intermediate reasoning, as well as the results of supporting steps, including tool calls, data retrieval, and extraction.

Each module produces an independent score for the step it evaluates. Those scores can be read on their own or aggregated and summarized across steps and over time. Both views matter. An individual score speaks to one specific step, while aggregates reveal broader patterns in an agent's performance. Because the modules measure different things, a faithfulness result stays separate from an OCR result, which makes it possible to locate where and how quality changed.

## Confidence Scoring Modules

Confidence Scoring modules fall into two families that differ in how they arrive at a score: judged modules and statistical modules.

Judged modules use an internal AI agent as an expert evaluator. Each one receives the relevant input and output and assesses a specific quality dimension, working through a structured rubric before assigning a rating on a Likert scale from 0 to 10. The rubric sorts an output into one of a handful of discrete quality bands, and an integer from 0 to 10 captures that granularity without implying false precision. Because an AI agent performs the evaluation, every judged score comes with a written explanation of why it was assigned, so the reasoning behind the number travels with it. Judged modules are best suited to qualitative dimensions that require interpretation, such as whether an answer is helpful, coherent, or faithful to its sources.

Statistical modules compute a continuous score from the data itself using statistical methods, instead of producing a rubric-based rating. They report on a scale from 0.0 to 1.0, derived from calculations such as the consistency of repeated generations, the confidence of an OCR engine, or the overlap between an answer and its retrieved context, and that continuous value preserves the underlying measurement directly. Statistical modules are deterministic in nature and grounded in observable quantities, which makes them well-suited to dimensions where a measurement is more applicable than a judgment.

<Note>
  The two scales answer different questions, so a judged 7 and a statistical 0.7 are not interchangeable, and the platform never averages across them.
</Note>

The following modules are available.

<Tabs>
  <Tab title="Judged modules">
    | Module                | What it evaluates                                                                                  |
    | --------------------- | -------------------------------------------------------------------------------------------------- |
    | Coherence             | Logical flow, clear transitions, internal consistency, and structural organization of the output   |
    | Completeness          | Whether the output addresses all parts of the request, with sufficient depth on each part          |
    | Correctness           | Factual accuracy and logical soundness of the output's claims relative to its input                |
    | Custom Judgement      | A user-defined judge that evaluates against your own criteria and prompt                           |
    | Faithfulness          | Whether the output stays grounded in the input and provided context, without fabricating beyond it |
    | Helpfulness           | Whether the output is actionable and calibrated to the user's underlying need                      |
    | Instruction Following | How well the output adheres to the explicit instructions given in the input                        |
    | Readability           | Structure, language, conciseness, and accessibility of the output for its intended reader          |
    | Relevance             | Whether the output addresses the user's intent and stays on topic                                  |
    | Tool Selection        | Whether the agent chose appropriate tools from those available, independent of the tools' results  |
    | Tractability          | Whether the task's reasoning demands are a good fit for what an LLM and its tools can reliably do  |
  </Tab>

  <Tab title="Statistical modules">
    | Module                      | What it evaluates                                                                                                                  |
    | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
    | Artifact Extraction Quality | Per-field evaluation of structured-data extraction against the source documents, checking that each field's value is supported     |
    | Data Grounding              | How well the output's content is grounded in the retrieved source context, measured by token-frequency overlap                     |
    | Observed Consistency        | Self-consistency across multiple sampled completions, measured with a natural-language-inference model that detects contradictions |
    | OCR Confidence              | Confidence that the source document's text was read correctly, calibrated from the OCR engine's per-token confidence               |
  </Tab>
</Tabs>

## Scorable Steps and Outputs

Confidence Scoring evaluates the individual outputs an agent produces as it works, and different kinds of output support different modules. Those outputs include:

* **agent responses**: the responses an agent generates, including its final answers and intermediate reasoning steps.
* **Tool calls**: an agent's choice of which tool to invoke and the result it returns, along with the performance of internal tools and their intermediate inputs and outputs, such as a generated SQL query and the rows that query returns.
* **Data retrieval**: the data an agent retrieves to work with, whether context gathered from its datasources to ground a response or input data fetched for the task itself.
* **Structured extraction**: the values a model extracts from source documents into structured artifacts according to a defined schema, together with the document parsing those values are built on.

The pairing of outputs to modules follows from what each output is. Scoring a SQL query for readability or a chat response for OCR confidence would measure nothing meaningful, so the platform applies only the modules that fit a given output.

## Configuration

Confidence Scoring is configured declaratively. An agent definition lists the modules enabled for an agent, and the platform treats that list as the set of modules to run. At execution time it narrows the list to those that apply to the step the agent is performing, skipping any that are not applicable at each step. The platform already knows which modules fit which steps, for example that OCR Confidence applies to retrieval over parsed documents while Tool Selection applies to an agent's tool-using turns, so the mapping of modules to steps is not something an author maintains by hand. The result is configuration that stays declarative and scoring that runs only where it produces a meaningful result.

The logic behind this sits entirely within the platform. It determines which scores are supported by which processes and tool calls, what counts as an intermediate output that can be scored, and when and how a given output is scored. The platform also maintains the tuning behind each module, including its evaluator prompts, model choices, sampling parameters, and calibration, so enabling a module brings a well-tuned evaluator into play with no further setup. See the [agents concept guide](/concepts/agents) for more information on agents and agent definitions.

## How Scoring Runs

Confidence Scoring runs asynchronously, in independent background jobs that sit outside an agent's execution path. When an agent completes a scorable step, the platform dispatches a scoring job for each applicable module. The agent does not wait for these jobs, so enabling Confidence Scoring does not slow down the responses an agent returns. Because the jobs run independently, they can outlive the turn that triggered them and post each score back as it finishes, including for modules that take longer to compute.

## Reading the Results

As individual confidence scores are computed, they become available for retrieval and inspection. Each score carries a numerical value on its module's scale, and judged scores also carry the written explanation of why that score was assigned. A scoring job records the input and output it evaluated along with its status, so the full provenance of a score is traceable by the platform.

The platform makes individual and aggregated scores available for analysis. An individual score helps explain a specific step or account for a particular result. Aggregated and summarized scores are just as important, whether across the steps of a single agent execution or across an agent's performance over time. Because judged and statistical scores measure different things on different scales, the platform aggregates them separately rather than collapsing them into one figure, which preserves the meaning of each category.

The platform provides built-in aggregations and summaries of results for common needs:

* **Per-execution summaries**: the scores produced across all the steps of a single agent run, which show how quality held up turn by turn within one conversation.
* **Per-module breakdowns**: aggregated scores for each module, which surface the dimension, such as faithfulness or relevance, where performance is weakest.

Confidence scores also support longer-range analysis, including how a module's scores move over time and how agents or workflows compare against one another, or against their own past behavior across different periods. Together these views form a foundation for understanding typical performance over many runs, A/B testing different configurations or inputs, and detecting and correcting performance drift.

## Using Confidence Scoring in Practice

Confidence Scoring supports several distinct uses, and each is suited to a different situation. Reading individual scores alongside their written explanations is most useful while diagnosing a specific result or developing an agent, when the question is why a particular step scored the way it did. Aggregated and summarized views are better suited to questions about an agent as a whole, such as establishing what typical performance looks like, comparing two configurations or inputs through A/B testing, and watching for drift once an agent is running in production. Which modules to enable is also worth revisiting over time. As an agent runs, it becomes clearer which quality dimensions matter most for it, and the enabled set can be narrowed to focus on them.


# Datasources
Source: https://docs.meibel.ai/concepts/datasources

How a datasource turns your structured and unstructured data into knowledge that agents search and query while they reason

## Overview

A datasource is a container that makes your data available to agents. It groups related content, whether files you upload or storage you connect, and prepares that content so an agent can retrieve from it while producing its outputs, whether responding in a conversation or extracting structured data to a schema. A datasource does the preparatory work up front. It parses, indexes, and organizes its contents into a queryable form that an agent reaches through tools.

A datasource holds two shapes of data, and it serves each one differently. Unstructured documents such as PDFs and contracts become searchable by meaning. Structured data such as tables and spreadsheets stays queryable by its columns and values. A single datasource can hold both, and an agent bound to it draws on either depending on the task in front of it.

This page explains what a datasource contains, how an agent works with it, and how its metadata connects to the [execution policies](/concepts/execution-policies) that govern access.

## Two shapes of data

The two shapes are queried in different ways, each suited to a different kind of retrieval.

**Unstructured documents** carry meaning in prose: reports, contracts, manuals, correspondence. During ingestion the platform parses each file and breaks it into data elements, the atomic units of content that each carry their own text, source, and metadata. An agent searches these data elements by semantic similarity, so a query about "termination clauses" surfaces the relevant passages even where they never use that exact phrase. This is retrieval over unstructured data, often called RAG.

**Structured data** lives in tables with columns and typed values: order records, financial figures, inventory. Semantic search suits it poorly, because the questions it answers are quantitative. How many orders shipped to the Northeast? What was the average discount last quarter? For these an agent generates a query against the tables and reads the rows that return, which yields the exact aggregates and arithmetic that similarity search does not.

The shape decides how an agent gets what it needs.

## Structure-Augmented Generation

Meibel's approach to retrieval rests on one idea: most data that looks unstructured is in fact structurable. A contract has a clause hierarchy. A report nests sections within sections. A product manual carries specifications buried in its prose. The platform recovers this latent structure during ingestion and uses it to retrieve with more precision than similarity alone reaches. Meibel calls the approach Structure-Augmented Generation, or STAG.

Structure appears in both shapes of data. In tables it is explicit, already present in the schema. In documents it is implicit, held in the headings, sections, and ordering that parsing recovers. Treating structure as first-class in both is what lets an agent move between a value in a table and the passage that explains it. The [Structure-Augmented Generation post](https://www.meibel.ai/post/structure-augmented-generation-bridging-structured-and-unstructured-data-for-enhanced-rag-systems) develops the idea in full.

## How agents work with a datasource

An agent does not query a datasource on a fixed path. It reaches a datasource through tools and decides when and how to use them as it works through a task. Binding a datasource to an agent gives that agent a document-search tool for the datasource's unstructured content and a structured-query tool for its tables. The agent chooses which to call for a given task, reads what returns, and can call again to refine a search or to bring results from both together.

This makes retrieval adapt to the task. Work that needs a figure and the reasoning behind it can lead an agent to query a table, then search the documents for the surrounding context. An agent can bind several datasources at once, each exposed as its own tools, and direct a question to whichever holds the relevant data. That routing across shapes and sources, driven by the agent's own reasoning, is what STAG's structure extraction exists to serve.

Each of these tool calls runs under the agent's [execution policy](/concepts/execution-policies). The agent's freedom in how to retrieve never widens what the session is permitted to see. See [agents](/concepts/agents) for how tools and datasource bindings are configured.

## Metadata

Alongside its content, a datasource carries metadata: typed fields attached to its data, each with a name, a type, and a description. Metadata reaches a datasource in three ways, and one datasource can use all three together.

**Automatic metadata.** Every ingested document carries structural fields the platform records as it parses, including the source filename, page numbers, the section hierarchy, and the kinds of content found on each page. For documents with a page layout, it also records the bounding box of each piece of content, its position on the page, so a retrieved passage can be traced back to the exact region it came from. The platform can generate descriptive fields as well, such as a title and a summary for a document. For structured data, it writes a description for each column from the column's own values, reading the column's type, its distinct values, and how often it is empty.

**Metadata shapes.** When a domain has a familiar set of fields, you apply a prebuilt shape instead of defining fields by hand. Academic sources can take a bibliography shape of title, authors, publication date, and DOI; other shapes cover legal, medical, and insurance documents. Where no prebuilt shape fits, you define your own fields. In either case the platform runs an extraction pass over each document and fills the fields from what it finds.

**Manual annotation.** You can set or correct a field on an individual data element by hand, which helps where an extracted value is uncertain or where a value is known only to you. An annotation can be locked, so a later re-ingestion keeps your value rather than overwriting it.

Metadata does more than describe its data. Marking a field as indexed turns it into a dimension you can filter on, both in search and in the access rules described next. A fixed number of indexed fields is available for each data type, so indexing is a decision about which fields you most need to query and govern by. An unindexed field still describes its data, but cannot be filtered on.

## Datasources and execution policies

The fields a datasource exposes are the fields an [execution policy](/concepts/execution-policies) can constrain. A policy scopes a session by writing filters over those axes: table columns and the built-in table name for structured data, indexed metadata fields and document identifiers for documents. However a field arrived, through automatic extraction, an applied shape, or your own annotation, once it is indexed it becomes an axis a policy can filter on.

A policy can only reference dimensions that the datasource actually exposes, which binds the two together. Structured data brings its columns with it, so a filter on `orders.region` works as long as the `orders` table has a `region` column. Documents differ, because a field has to be extracted and indexed before a policy can use it. Say you add a `product_line` field to a datasource of product documents and mark it indexed. A session policy can then hold one user to `product_line` of `commercial`, so that user's searches never surface consumer-line material. Leave `product_line` uncaptured and no policy can draw that boundary, because the field is not there to filter on. This is the practical consequence of the coupling: you can only govern the dimensions you chose to capture. When a session starts, the platform validates the policy against the datasource's real schema and rejects any filter that references a column or field that does not exist.

<Note>
  Enriching a datasource with more indexed metadata widens what an execution policy can express. Planning the metadata you extract is part of planning how you will scope access to it.
</Note>

## The datasource lifecycle

A datasource moves through a few stages. It begins empty, created with a name and a description. Content enters it, uploaded directly or pulled from connected storage. Ingestion processes that content into data elements and tables and extracts metadata along the way. Once ingested, the datasource answers queries, both from agents bound to it and directly through the data elements API. Binding it to an agent is what places its tools in the agent's hands.

For the steps to create a datasource, add content, and trigger ingestion, see [Managing datasources](/guides/datasources).

## Related concepts

<CardGroup>
  <Card title="Execution Policies" icon="shield-halved" href="/concepts/execution-policies">
    How the platform scopes what data an agent session reaches, filtering on datasource metadata.
  </Card>

  <Card title="Agents" icon="robot" href="/concepts/agents">
    How agents bind datasources as tools and query them while reasoning through a task.
  </Card>

  <Card title="Managing Datasources" icon="database" href="/guides/datasources">
    Create datasources, add content, and trigger ingestion.
  </Card>
</CardGroup>


# Execution Policies
Source: https://docs.meibel.ai/concepts/execution-policies

How Meibel controls what data and tools an agent session can access at runtime through composable, platform-enforced constraints

## Overview

Conventional LLM guardrails operate through prompting, which means they can be bypassed through prompt injection or jailbreaking. Execution policies are **hardrails**: constraints enforced by the platform infrastructure itself, outside the LLM's execution path. The LLM cannot see, modify, or circumvent the authorization logic that drives an execution policy. Where a guardrail asks the model to comply, a hardrail makes non-compliance structurally impossible.

An execution policy narrows what data and tools are available to an agent session. It cannot make new datasources or tools available; it can only restrict access within what the agent is already configured to use. An execution policy has two sections: **datasource constraints**, which control access to the data the agent queries and retrieves, and **tool constraints**, which control which tools are available and what parameter values they accept.

The central use case is per-user data scoping in a multi-tenant environment. A single agent can serve many users, each with different data access permissions. Rather than building separate agents for each user, you define one agent and apply different execution policies when creating each user's session. One policy might restrict a session to European customer data; another might restrict it to a single client's documents. The agent configuration stays the same, and the policy controls what each session can see.

```json theme={null}
{
  "datasources": {
    "ds_abc123": {
      "tables": {
        "filter": {
          "table.__name__": { "$in": ["orders", "customers"] },
          "orders.region": "EU"
        },
        "hidden_columns": {
          "customers": ["ssn", "credit_card"]
        }
      },
      "documents": {
        "filter": {
          "data_element.__id__": { "$in": ["de_abc", "de_def"] }
        }
      }
    }
  },
  "tools": {
    "send_email": {
      "variables": {
        "to": "support@acme.com",
        "max_attachments": { "$lte": 3 }
      }
    },
    "web_search": { "disabled": true }
  }
}
```

This policy narrows the session to two tables in the EU region, hides sensitive columns from query results, limits document retrieval to two specific data elements, locks the email recipient, caps attachments, and removes web search entirely.

## Datasource constraints

Each entry in the `datasources` section is keyed by datasource ID and controls what the session can access within that datasource. A datasource can be disabled entirely by setting `disabled` to `true`, which blocks all queries against it.

For datasources that remain enabled, constraints are split into two categories that match the two ways an agent accesses data: table queries and document retrieval.

### Table constraints

Table constraints scope the SQL queries an agent can run against a datasource's structured data. They have three parts.

**Filters** narrow which tables and rows are accessible. Filter keys follow the format `table.column` for row-level scoping or use the built-in `table.__name__` field for table-level scoping. Values use constraint operators: comparison operators (`$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`), set operators (`$in`, `$nin`), and logical operators (`$and`, `$or`, `$not`).

```json theme={null}
{
  "filter": {
    "table.__name__": { "$in": ["orders", "customers"] },
    "orders.region": { "$eq": "EU" },
    "orders.amount": { "$gte": 100 }
  }
}
```

This filter restricts the session to the `orders` and `customers` tables, and within `orders`, only rows where `region` is `EU` and `amount` is at least 100. The platform enforces these constraints transparently, so the agent only ever sees rows that match.

Plain values are a shorthand for `$eq`: writing `"orders.region": "EU"` is equivalent to `"orders.region": { "$eq": "EU" }`.

**Hidden columns** remove specific column values from query results. The columns remain usable in WHERE and JOIN clauses, so the agent can filter on them and use them in joins, but their values are stripped from the output.

```json theme={null}
{
  "hidden_columns": {
    "customers": ["ssn", "credit_card"]
  }
}
```

**Hidden tables** work the same way at the table level: the table's columns are fully redacted from query output, but the table remains available for use in joins.

### Document constraints

Document constraints scope which documents the agent can retrieve through vector search. Their filter uses the same operator syntax as table constraints, with built-in fields for document identity:

* `data_element.__id__` scopes by data element ID
* `data_element.__name__` scopes by original filename

Custom [indexed metadata fields](/guides/datasource-metadata) configured on the datasource are also supported. In the example below, `product_line` is a custom metadata field:

```json theme={null}
{
  "filter": {
    "data_element.__id__": { "$in": ["de_abc", "de_def"] },
    "product_line": "commercial"
  }
}
```

The platform applies document filters to every vector search query. Documents that do not match are invisible to the agent.

## Tool constraints

Each entry in the `tools` section is keyed by tool instance name and controls how the session can use that tool. A tool can be disabled entirely by setting `disabled` to `true`, which removes it from the agent's available toolkit.

For tools that remain enabled, the `variables` field constrains parameter values. Constraints come in two forms that behave differently.

**Fixed values** are plain values (or `$eq` constraints). They constrain a parameter to a single allowed value. The platform annotates the tool's parameter description so the agent sees the constraint (e.g., "must equal '[support@acme.com](mailto:support@acme.com)'"), and validates at execution time that the agent provides the correct value.

```json theme={null}
{
  "send_email": {
    "variables": {
      "to": "support@acme.com"
    }
  }
}
```

Here, the agent sees that the `to` parameter must equal `support@acme.com`. If it provides any other value, the platform rejects the call.

**Range constraints** use operators like `$lte`, `$in`, or `$contains`. They work the same way: the parameter description is annotated with the constraint, and the platform validates the agent's value at execution time.

```json theme={null}
{
  "web_search": {
    "variables": {
      "query": { "$contains": "site:acme.com" },
      "count": { "$lte": 10 }
    }
  }
}
```

The agent can set `query` and `count`, but the platform rejects any search query that does not include `site:acme.com` and any count above 10.

Tool constraints support the same comparison, set, and logical operators as datasource filters, plus additional string operators (`$contains`, `$starts_with`, `$ends_with`, `$matches_regex`) and length operators (`$max_length`, `$min_length`).

## How enforcement works

Execution policies are hardrails, enforced at three points in a defense-in-depth model that operates around the LLM rather than relying on it.

**Before the LLM call.** The platform builds the agent's tool schemas and datasource schemas according to the policy. Disabled tools and inaccessible tables are removed from the schemas the agent sees. Constraint descriptions are annotated into parameter descriptions so the agent understands the boundaries it must operate within.

**During data access.** When the agent queries a datasource, the platform applies the policy's filters to the query itself. For table queries, filter conditions are enforced so only matching rows are returned. For document retrieval, filter conditions are added to the vector search query. The agent receives only data that passes the policy's filters.

**At tool execution.** When the agent invokes a tool, the platform validates every argument against the policy's constraints before dispatching the call. If any argument violates a constraint, the call is rejected and the agent receives an error.

This model means policies cannot be circumvented through prompt injection or creative instructions. The LLM never sees the authorization data that drives datasource scoping. Tool constraints are validated outside the LLM's execution path.

<Note>
  Execution policies default to denying access and narrowing scope rather than expanding it. If a datasource referenced in a policy does not exist, the platform rejects the policy. If a table referenced in a filter is not in the datasource, initialization fails. If a document does not match a filter, it is invisible.
</Note>

## Composing policies

Multiple execution policies can be composed together, and the platform merges them structurally rather than choosing one over another. This is useful when different concerns are managed by different policies: one policy might handle data region scoping, another might handle tool restrictions, and a third might handle column redaction.

When policies are composed:

* **Overlapping entries** for the same datasource or tool have their constraints merged with `$and`, meaning the data or parameter must satisfy all policies to be accessible.
* **Disabled is sticky.** If any policy disables a datasource or tool, it stays disabled in the composed result regardless of other policies.

Policies can be set at multiple levels that compose together in order:

1. **Agent definition**: default policies baked into the agent, applied to every session.
2. **Session creation**: policies passed when creating a session, layered on top of the agent defaults.

Because composition only narrows access (filters intersect, disabled is sticky), a session-level policy cannot grant access beyond what the agent definition allows. Each layer can only further restrict the previous one.

## Related concepts

Execution policies interact with other parts of the platform:

<CardGroup>
  <Card title="Datasources" icon="database" href="/concepts/datasources">
    The datasources whose documents and tables a policy scopes, and the metadata it filters on.
  </Card>

  <Card title="Agents" icon="robot" href="/concepts/agents">
    The agents that execution policies constrain, including their datasource bindings and tool configurations.
  </Card>

  <Card title="Batch Jobs" icon="layer-group" href="/concepts/batch-jobs">
    Running agents at scale with execution policies applied to each batch execution.
  </Card>
</CardGroup>


# Charts, formulas, and vision models
Source: https://docs.meibel.ai/document-parsing/concepts/charts-and-vision

How Meibel recognizes charts, formulas, seals, and pictures with vision models, and digitizes charts back into data series

## Overview

Some regions of a document resist ordinary text recognition: a formula is mathematical notation, not a line of words; a chart encodes its meaning in geometry and labels; a seal is stylized text arranged in a circle; a photograph carries information no character reader can extract. For these, character-level OCR returns fragments that lose the region's meaning.

Meibel handles them by routing each such region to a vision-language model, a model that reads an image and returns structured content. An initial layout stage labels a region as a formula, a chart, a seal, or a picture. Regions with those labels are then cropped from the page and sent to a model suited to the task, and the result is merged back into the element. Charts get a second treatment on top of this: a chart drawn as vector graphics is digitized from its geometry, and the recognized values are reconciled with what was drawn.

This page explains which regions are recognized this way, what each returns, and how charts are turned back into data.

## Which regions are recognized

Layout analysis gives every region a label naming its content role. That label determines whether a vision model reads the region. Four kinds of region are read this way, each with its own task.

| Region  | What the model returns                                                           |
| ------- | -------------------------------------------------------------------------------- |
| Formula | The mathematical notation of the expression                                      |
| Chart   | The chart's values as a structured table, reconciled with the digitized geometry |
| Seal    | The text contained in the seal or stamp                                          |
| Picture | A written description of the image                                               |

A page with none of these regions passes straight through without contacting a vision model, so a document of plain text and tables pays nothing for this stage.

## What each region contributes

**Formulas.** A region labeled as a formula returns its notation as the element's text. In the Markdown output, the formula renders as display math between `$$` delimiters so a reader or renderer treats it as an equation rather than as broken prose.

**Charts.** A chart returns its values as a structured table, so the numbers behind the plot come back as data rather than as an image. Those values are then reconciled with the chart's drawn geometry, a second analysis of the region that the sections below work through.

**Seals.** A seal or stamp returns its recognized text, which recovers content that would otherwise be lost inside a graphic.

**Pictures.** An image returns a written description of what it shows. The description travels with the figure, so an index or a language model has something meaningful to work with in place of an opaque image reference.

## Charts: two sources of truth, reconciled

A chart is a picture of data. The values that produced it are not stored in the file; they became lines, points, and axes when the chart was drawn. Recovering them means reading the chart the way a person does, and that can be done two ways, each covering the other's weakness.

**Geometry** gives the true shape of the chart. Reading the plotted paths and inverting them through the calibrated axes recovers where each plotted value sits and how the values move across the axes, faithful to what was drawn. What geometry alone cannot always pin down is the exact number, because a value read off an axis is only as precise as the spatial calibration.

**The vision model** gives exact values. It reads the chart and returns the values as a structured table, capturing numbers that a data label states outright. A table on its own carries no position for each value and no check against what the chart draws.

Meibel reconciles the two. It starts from the geometry, then matches each recognized value to the nearest geometric point within a tolerance. A value that matches replaces the geometric estimate and is marked as adjudicated by the model, while the point keeps its position on the page. A recognized value that disagrees with the geometry by more than the tolerance is treated as suspect: the geometry is kept and a warning is recorded, which guards against a model that misreads a number. When a chart has no usable vector geometry, such as an image of a chart in a scan, the values come from the vision model alone, and any text on the chart is recovered by OCR on the crop.

## What chart data contains

A digitized chart is a structured object carrying the series and everything needed to interpret them.

* **Series.** Each series has its points, and each point carries its `x` and `y` in data units, a confidence, a position on the page, and a note of where its value came from. A series also records its drawn color and dash pattern, which is how two monochrome lines are told apart.
* **Axis calibration.** Each axis records its scale, its detected tick marks, the fit from pixels to data values, and any title and unit.
* **Categories.** A chart with a categorical x-axis lists its category labels, such as a sequence of years or quarters.
* **Chart metadata.** The chart records its type, whether it came from vector geometry or an image, an overall confidence, and any warnings raised during digitization.

The exact fields are enumerated in the [output schema](/document-parsing/reference/output-schema).

## What is recovered

Chart digitization is strongest on the charts whose geometry is unambiguous.

* **Line and scatter charts** drawn as vector graphics are digitized from their geometry, including their series, points, and axis calibration.
* **Linear, logarithmic, and categorical axes** are supported, with the scale selected from the axis's own tick marks and labels.
* **Dual-axis charts** are detected. When a series cannot be assigned confidently to the left or right axis, it is marked ambiguous and a warning is recorded rather than guessing.
* **Values** come from the vision model reconciled with the geometry, so exact numbers stated as data labels are captured even where the calibration alone would only approximate them.

<Note>
  Bar, area, and pie charts are recognized and located as chart regions, and a vision model can still read their values, but their vector geometry is not digitized into series the way line and scatter charts are.
</Note>

## How the results appear

Vision model results reach you in the output, though where depends on the region and the format.

Formula notation and seal text sit in the element's text, so they appear in every rendering that carries text. In the structured result a chart element carries its digitized `chart_data` and its recognized labels on `ocr_text`. The Markdown output renders a chart's series as a table, either a category-by-series matrix when the series share an x-axis or a long series-x-value listing when they do not. [Choosing an output format](/document-parsing/guides/choosing-an-output-format) sets out which rendering carries which, and [extracting chart data](/document-parsing/guides/extracting-chart-data) works through reading these in code.

## Resilience

Recognizing a region depends on a model responding, so the pipeline is built to degrade rather than fail. A request that returns a transient error is retried with a widening delay between attempts. A region that still does not come back is logged and skipped, and the page continues with whatever other regions succeeded. One region's failure never stops the rest of the document, and a page where every vision model call fails still returns its text, its tables, and its layout.

## Related concepts

<CardGroup>
  <Card title="How parsing works" icon="gears" href="/document-parsing/concepts/how-parsing-works">
    Where recognition and digitization sit in the pipeline.
  </Card>

  <Card title="Extracting chart data" icon="chart-simple" href="/document-parsing/guides/extracting-chart-data">
    Read a chart's series and labels from the output.
  </Card>

  <Card title="The parsed document" icon="diagram-project" href="/document-parsing/concepts/the-parsed-document">
    Where chart data sits in the element model.
  </Card>

  <Card title="Output schema" icon="code" href="/document-parsing/reference/output-schema">
    The chart data fields in full.
  </Card>
</CardGroup>


# How parsing works
Source: https://docs.meibel.ai/document-parsing/concepts/how-parsing-works

What Meibel recovers from a document, and the stages a file moves through from raw bytes to ordered structural elements

## Overview

Document parsing reads a file such as a digital PDF or a scan of a paper document, and returns its content as structured elements: titles, headings, paragraphs, lists, tables, figures, formulas, and more. Each element is placed in reading order and tied to its position on the page. Where a raw text dump loses the layout that gives a document its meaning, Meibel recovers that layout and returns content a program can act on. The output has the same shape whether the source was a clean digital PDF, a skewed scan, or a phone photo of a document out in the world.

## What parsing recovers

Meibel reads a document the way its layout intends, recovering the structure a flat stream of characters would lose.

**Structure by content role.** A layout model labels every region by its role, spanning text, tables, figures, and page furniture. A `Title` or `SectionHeader` carries a level from H1 to H6, so the section hierarchy survives into the output. The full catalog is available in [element types](/document-parsing/reference/element-types).

**Tables as addressable grids.** A table returns as a grid of cells, each with its row and column position and any spans, so a program reads it by coordinate rather than from whitespace.

**Charts as data, and recognition of formulas and seals.** Charts, formulas, seals, and pictures are all read by vision-language models, and line and scatter charts are also digitized back into their series. See [charts, formulas, and vision models](/document-parsing/concepts/charts-and-vision).

**Text from scans.** Pages without an extractable text layer go through OCR automatically, decided page by page, with orientation corrected first and a large multilingual character set covered.

**Position and confidence on every element.** Each element carries a bounding box you can trace to the source, and a confidence score a downstream step can gate on.

## The pipeline

The stages below run connected by channels, so pages flow from one stage to the next and the work parallelizes across pages. A page that needs neither recognition nor a vision model passes through the stages that do not apply to it without waiting. This is what keeps large documents fast while still handling the hard pages thoroughly.

### Text extraction

The first stage reads the document's own contents. For a digital PDF this means walking the file's structure and its content streams to recover every character, its font, and its position. The output is a set of positioned characters, because a PDF stores text as placed glyphs rather than as sentences. From those glyphs the stage assembles words and lines by their geometry, so characters sitting together become a word and words on a shared baseline become a line.

This path is pure work over the file itself, with no model involved, which is why a text-based PDF parses quickly. Encrypted PDFs are decrypted where possible, and content nested inside reusable form objects is followed so its text is not missed.

### Orientation and optical character recognition

A page can arrive with no readable text layer, as scans and photographs do, or rotated, as faxed pages often are. A page that needs recognition has its orientation assessed and corrected: a page turned ninety, one hundred eighty, or two hundred seventy degrees is set upright along with the coordinates of anything already extracted from it. It then goes through OCR, which reads characters directly from the image and covers a large multilingual character set. Skew finer than a quarter turn is handled at this stage rather than by rotating the page.

Whether a page needs OCR is decided page by page. A scorer weighs several signals from the page: the quality of any embedded text encoding, whether text is present but invisible, how rich the fonts are, how much of the page is image, and whether the text reads coherently. A page with a clean text layer skips OCR and stays fast, while a scanned or garbled page is recognized from its image. A recognized page rejoins the pipeline in the same form as extracted text.

### Layout analysis

Positioned lines alone do not say what a line is. A layout model looks at the rendered page and divides it into regions, labeling each with its content role. This labeling is what lets later stages and your own code treat a heading differently from body text, a table apart from prose, and a chart apart from a photograph. Layout analysis reads the visual page, so it uses the same signal a person does: size, position, spacing, and emphasis. The text lines are then matched into the regions that contain them. Pages are analyzed in batches for efficiency.

When a page is detected as a line-numbered legal transcript, the numbers running down its gutter are lifted out of the body text and kept separately, each with its own position. The prose then reads continuously, and page and line citations into the transcript stay resolvable.

### Recognizing tables

A region labeled as a table still needs its internal grid rebuilt, because a PDF does not record which text belongs to which cell. A table model recovers its rows, its columns, and the cells within them, including cells that span more than one row or column, and the text lines are matched into those cells by position. A sanity check guards against grids produced from content that is not really tabular, treating a degenerate grid as ordinary text instead.

### Charts, formulas, and other recognized regions

Regions that ordinary recognition handles poorly are read by vision-language models. A formula, a chart, a seal, or a picture is cropped and sent to a model that returns structured content, and a chart drawn as vector graphics is digitized from its geometry in parallel. This stage fans out the eligible regions on a page concurrently, and a page with none of them passes through at once. The details of this stage are in [charts, formulas, and vision models](/document-parsing/concepts/charts-and-vision).

### Reading order

The regions on a page are found by their position, which may not match the order a person reads them in. For example, a two-column article, a page with a sidebar, or a layout with footnotes would read incoherently if taken strictly top to bottom, left to right. The ordering stage sequences the regions into human reading order, keeping page headers and footers apart from the body and associating captions and footnotes with what they belong to.

### Rendering the result

The stages above produce one internal representation: ordered, typed, positioned elements, with tables as grids and charts as data. The final stage renders that representation into the rendering you requested. Because every rendering derives from the same representation, they agree with each other, and requesting a second one re-renders rather than re-parses. See [the parsed document](/document-parsing/concepts/the-parsed-document) and [choosing an output format](/document-parsing/guides/choosing-an-output-format).

## Start here

<CardGroup>
  <Card title="Parse your first document" icon="play" href="/document-parsing/tutorials/parse-your-first-document">
    A guided walk from a PDF on disk to structured output.
  </Card>

  <Card title="The parsed document" icon="diagram-project" href="/document-parsing/concepts/the-parsed-document">
    The element model the pipeline produces.
  </Card>

  <Card title="Charts, formulas, and vision models" icon="eye" href="/document-parsing/concepts/charts-and-vision">
    How charts, formulas, seals, and images are recognized.
  </Card>

  <Card title="Output schema" icon="code" href="/document-parsing/reference/output-schema">
    Every field in the structured result.
  </Card>
</CardGroup>


# The parsed document
Source: https://docs.meibel.ai/document-parsing/concepts/the-parsed-document

The element model behind every output format: typed content, table grids, chart data, recognized labels, positions, and confidence

## Overview

A parsed document is a set of pages, each holding its elements in reading order. Each element is one piece of content the pipeline recovered, carrying what it is, what it says, where it sits, and how certain the extraction is. Some elements carry more: a table carries its grid, a chart carries its data and its recognized labels. The renderings you can request are different views of this one model, so understanding the model explains all of them at once.

This page describes the model conceptually. For the exact field names and types, see the [output schema](/document-parsing/reference/output-schema), and for the full set of content types, see [element types](/document-parsing/reference/element-types).

## Elements are typed content

The pipeline classifies every region of a page and turns it into a typed element. The type is what lets user code treat content by its role rather than by its appearance: pull the headings to build an outline, keep the tables for data, route charts to a chart handler, drop page headers and footers from a body-text index. The roles range across text, tables, figures, and page furniture, each cataloged in [element types](/document-parsing/reference/element-types).

A heading also carries a level from 1 to 6, derived from the document's own typography by clustering font sizes and weights. This level records the section hierarchy the document expressed through its layout, so the nesting of sections within sections survives into the output. An outline built from the headings and their levels reconstructs the document's structure without a separate pass.

## Different elements carry different content

What an element carries depends on its type. Most elements hold their content as text, and those whose content has an internal structure, such as a table's grid or a chart's series, carry that structure as well.

**Text-bearing elements** hold their content as a string. A paragraph holds its prose, a formula holds its mathematical notation, a seal holds its recognized text, a code block holds its code.

**Tables** hold a grid. A table element carries a set of cells and the grid's dimensions, and each cell knows its row and column position and can span more than one row or column. This grid form is what lets a program read a table by coordinate, addressing the value at a given row and column, rather than reconstructing structure from the spacing of rendered text. [Extracting tables](/document-parsing/guides/extracting-tables) works through reading this grid in code.

**Charts** hold data and recognized labels. A chart element can carry the series digitized from its geometry and reconciled with vision model values, and separately the individual text labels recovered from the chart, each with its own position and a note of whether it came from the document's text or from OCR. [Extracting chart data](/document-parsing/guides/extracting-chart-data) covers reading these.

## Every element knows where it came from

Each element carries a bounding box locating it on the page that holds it. This provenance is what connects extracted content back to the source. A value pulled from a table can be traced to the region it was read from, a passage can be highlighted on a rendered page, and a reviewer can see a low-confidence extraction in its original context. Chart data carries provenance too, with each point and each recognized label holding its own position.

Position also underlies reading order. The pipeline uses each region's geometry to sequence the elements the way a person reads the page, so the element list arrives already ordered rather than sorted by raw coordinates. The order is part of the model.

## Confidence travels with the content

Each element carries a confidence score, and the document carries an overall summary of scores. These express how certain the extraction is, and they vary with the source: a clean digital table scores higher than a scanned one, and text recovered by OCR scores lower than text read from the document's own text layer. Because the score sits on the element, a downstream step can act at the level of a single table or paragraph, flagging and responding to what needs review while trusting the rest. Confidence is the hook for a human-in-the-loop step over parsed output, and it connects to Meibel's broader [confidence scoring](/concepts/confidence-scoring).

## One model, several renderings

Each rendering carries this same model, and they differ in how much of it they keep. Markdown keeps what a reader or a language model needs, rendering elements as text and leaving out positions and scores. The structured result keeps all of it, exposing elements with their labels, cells, chart data, boxes, and scores. Annotated Markdown sits between the two, holding the readable text with each piece's position attached. Because all three come from one parse they agree with each other, so the choice comes down to how much of the model the consumer needs. [Choosing an output format](/document-parsing/guides/choosing-an-output-format) matches each one to its job.

## Related concepts

<CardGroup>
  <Card title="Element types" icon="tags" href="/document-parsing/reference/element-types">
    The content roles an element can take.
  </Card>

  <Card title="Output schema" icon="code" href="/document-parsing/reference/output-schema">
    The concrete fields and types of every element.
  </Card>

  <Card title="Charts, formulas, and vision models" icon="chart-line" href="/document-parsing/concepts/charts-and-vision">
    The chart data an element can carry.
  </Card>

  <Card title="Choosing an output format" icon="list-check" href="/document-parsing/guides/choosing-an-output-format">
    Which rendering of the model to request.
  </Card>
</CardGroup>


# Choosing an output format
Source: https://docs.meibel.ai/document-parsing/guides/choosing-an-output-format

Pick between Markdown, the structured result, and annotated output based on what your application does with the result

A single parse can be read several ways. The readable and interchange renderings come from the result endpoint, where you pass a `format`; the strongly-typed structured result has its own endpoint and method. The underlying parse is generated once and is the same each time; how you fetch it decides how the content is shaped for you. This guide matches each option to the job it suits.

## Choose by what you do next

<CardGroup>
  <Card title="Markdown" icon="markdown">
    Feeding a language model, or showing the document to a person.
  </Card>

  <Card title="Structured" icon="brackets-curly">
    Reading elements, tables, chart data, or positions in your own code.
  </Card>

  <Card title="Annotated" icon="location-dot">
    Needing the text together with where it sits on the page.
  </Card>
</CardGroup>

## Markdown: text for people and models

Reach for Markdown when the consumer of the output reads prose. It renders headings as headings, lists as lists, and tables as Markdown grids, which is the shape a language model handles most reliably and a person reads most easily. Use it for retrieval-augmented generation, for summaries, and for any view a human will look at. The result endpoint returns Markdown as a string.

<CodeGroup>
  ```python Python theme={null}
  result = client.documents.get_result(job_id=job_id, format="markdown")
  print(result)
  ```

  ```bash curl theme={null}
  curl "https://api.meibel.ai/v2/documents/$JOB_ID/result?format=markdown" \
    -H "Meibel-API-Key: $MEIBEL_API_KEY"
  ```
</CodeGroup>

## Structured: typed elements for your code

Choose the structured result when your program acts on the content rather than displaying it. It comes back as pages of typed elements, each carrying a `label`, its `text`, a `bbox`, a `reading_order`, and a `confidence` score, with tables exposing their cells by row and column and charts carrying their digitized data. This is the result to read when you route content by label, pull values out of tables, or read a chart's series.

<CodeGroup>
  ```python Python theme={null}
  from meibel import ParseLayoutLabel

  result = client.documents.get_structured_result(job_id=job_id)

  headings = [
      el
      for page in result.pages
      for el in page.elements
      if el.label in (ParseLayoutLabel.TITLE, ParseLayoutLabel.SECTIONHEADER)
  ]
  ```

  ```bash curl theme={null}
  curl "https://api.meibel.ai/v2/documents/$JOB_ID/structured" \
    -H "Meibel-API-Key: $MEIBEL_API_KEY"
  ```
</CodeGroup>

The full field list is in the [output schema](/document-parsing/reference/output-schema).

## Annotated: Markdown with position

Use annotated output when you want readable text and still need to know where each piece came from. It is Markdown with bounding-box provenance comments attached to the content, so you can render the text and, when needed, trace a passage back to its region on the page.

<CodeGroup>
  ```python Python theme={null}
  result = client.documents.get_result(job_id=job_id, format="annotated")
  ```

  ```bash curl theme={null}
  curl "https://api.meibel.ai/v2/documents/$JOB_ID/result?format=annotated" \
    -H "Meibel-API-Key: $MEIBEL_API_KEY"
  ```
</CodeGroup>

<Note>
  If you need both a readable view and full structured data, fetch each in turn. One parse serves every rendering, so a second fetch does not re-run the work.
</Note>

## Related

<CardGroup>
  <Card title="Output schema" icon="code" href="/document-parsing/reference/output-schema">
    Every field in the structured result.
  </Card>

  <Card title="The parsed document" icon="diagram-project" href="/document-parsing/concepts/the-parsed-document">
    The element model that every rendering expresses.
  </Card>
</CardGroup>


# Extracting chart data
Source: https://docs.meibel.ai/document-parsing/guides/extracting-chart-data

Read a chart's digitized series, values, and recognized labels from the structured result

When a document contains charts, you often want the numbers behind them rather than an image. Meibel digitizes line and scatter charts into their series, reconciles those values with a vision model, and recovers the text labels drawn on the chart. All of it arrives on the chart element in the structured result. This guide reads it in code.

For how chart digitization works and what it recovers, see [charts, formulas, and vision models](/document-parsing/concepts/charts-and-vision).

## Prerequisites

* A completed parse job for a document that contains charts.
* The structured result, fetched with `get_structured_result`, which is where chart data is exposed.

## Find the charts

A chart is an element with the `Chart` label. When its geometry was recovered, the element carries the digitized plot on `chart_data`. Walk the pages and collect the chart elements that have it.

<CodeGroup>
  ```python Python theme={null}
  from meibel import ParseLayoutLabel

  result = client.documents.get_structured_result(job_id=job_id)

  charts = [
      el
      for page in result.pages
      for el in page.elements
      if el.label == ParseLayoutLabel.CHART and el.chart_data
  ]
  print(f"Found {len(charts)} charts with data")
  ```

  ```typescript TypeScript theme={null}
  const result = await client.documents.getStructuredResult(jobId);

  const charts = result.pages
    .flatMap((page) => page.elements)
    .filter((el) => el.label === 'Chart' && el.chartData);
  console.log(`Found ${charts.length} charts with data`);
  ```
</CodeGroup>

## Read the series and points

A chart's `chart_data` holds its series, and each series holds its points. Each point carries an `x` and a `y` in data units. When the x-axis is categorical, the chart also lists its `categories`, and a point's `x` is the index into that list.

<CodeGroup>
  ```python Python theme={null}
  for element in charts:
      chart = element.chart_data
      for series in chart.series:
          name = series.name or "series"
          values = [(p.x, p.y) for p in series.points]
          print(name, values)
      if chart.categories:
          print("categories:", chart.categories)
  ```

  ```typescript TypeScript theme={null}
  for (const element of charts) {
    const chart = element.chartData;
    for (const series of chart.series) {
      const name = series.name ?? 'series';
      const values = series.points.map((p) => [p.x, p.y]);
      console.log(name, values);
    }
    if (chart.categories.length) console.log('categories:', chart.categories);
  }
  ```
</CodeGroup>

## Read the recognized labels

The text on a chart, such as axis titles and data labels, is recovered onto the element's `ocr_text`. Each entry carries the recognized `text`, its `confidence`, and a `source` of `PdfText` when it came from the document's own text or `Ocr` when it was read from the image. Because `ocr_text` is a sibling of `chart_data`, it is present even on a chart whose geometry could not be digitized.

```python Python theme={null}
for page in result.pages:
    for element in page.elements:
        if element.label == ParseLayoutLabel.CHART:
            for label in element.ocr_text or []:
                print(label.source, label.confidence, label.text)
```

## Check confidence and warnings

Digitization is an estimate, and the chart records how much to trust it. Each chart carries an `overall_confidence` and a `warnings` list. A warning is raised when a recognized value disagrees with the geometry, or when a series cannot be assigned to a left or right axis. Read both before treating the values as exact.

```python Python theme={null}
for element in charts:
    chart = element.chart_data
    print("confidence:", chart.overall_confidence)
    for w in chart.warnings:
        print("warning:", w)
```

<Warning>
  Chart values are recovered from geometry and vision model recognition, so they are estimates rather than the source data. Gate on `overall_confidence` and surface `warnings` before using the numbers in anything that assumes exact figures.
</Warning>

## Related

<CardGroup>
  <Card title="Charts, formulas, and vision models" icon="chart-line" href="/document-parsing/concepts/charts-and-vision">
    How chart data is digitized and reconciled.
  </Card>

  <Card title="Output schema" icon="code" href="/document-parsing/reference/output-schema">
    The full chart data field definitions.
  </Card>
</CardGroup>


# Extracting tables
Source: https://docs.meibel.ai/document-parsing/guides/extracting-tables

Read table cells by row and column from the structured result, including tables with merged and spanning cells

When a document carries data in tables, you usually want that data as rows and columns rather than as rendered text. The structured result returns each table as a grid of cells, each cell placed by its row and column index and carrying any spans. This guide turns that grid into a structure your code can iterate.

## Prerequisites

* A completed parse job. See [Parse your first document](/document-parsing/tutorials/parse-your-first-document) if you need one.
* The structured result, fetched with `get_structured_result`, which is where the cell grid is exposed.

## Find the tables

The structured result groups elements by page. Walk the pages and filter their elements by `label` to pull out the tables. Each table element holds a `table` object with its cell grid, its `num_rows` and `num_cols` counts, the page it sits on, and a `confidence` score.

<CodeGroup>
  ```python Python theme={null}
  from meibel import ParseLayoutLabel

  result = client.documents.get_structured_result(job_id=job_id)

  tables = [
      el for page in result.pages for el in page.elements if el.label == ParseLayoutLabel.TABLE
  ]
  print(f"Found {len(tables)} tables")
  ```

  ```typescript TypeScript theme={null}
  const result = await client.documents.getStructuredResult(jobId);

  const tables = result.pages
    .flatMap((page) => page.elements)
    .filter((el) => el.label === 'Table');
  console.log(`Found ${tables.length} tables`);
  ```
</CodeGroup>

## Read cells into a grid

Each cell reports its `row`, its `col`, and its `text`. Because the grid dimensions are known from `num_rows` and `num_cols`, you can allocate a 2D array and place every cell at its coordinate. This gives you the table as nested lists, ready to write to a CSV, load into a dataframe, or compare against expected values.

<CodeGroup>
  ```python Python theme={null}
  def to_grid(table):
      grid = [["" for _ in range(table.num_cols)] for _ in range(table.num_rows)]
      for cell in table.cells:
          grid[cell.row][cell.col] = cell.text
      return grid

  for element in tables:
      grid = to_grid(element.table)
      for row in grid:
          print(row)
  ```

  ```typescript TypeScript theme={null}
  function toGrid(table) {
    const grid = Array.from({ length: table.numRows }, () =>
      Array.from({ length: table.numCols }, () => ''),
    );
    for (const cell of table.cells) {
      grid[cell.row][cell.col] = cell.text;
    }
    return grid;
  }

  for (const element of tables) {
    const grid = toGrid(element.table);
    grid.forEach((row) => console.log(row));
  }
  ```
</CodeGroup>

## Handle merged and spanning cells

Real tables sometimes merge cells, most often in headers. A cell that spans more than one column or row reports `col_span` or `row_span` greater than 1. The cell's `text` belongs at its starting `row` and `col`; the positions it covers hold no separate cell of their own. To keep the grid rectangular, write the text across every position the cell spans.

<CodeGroup>
  ```python Python theme={null}
  def to_grid(table):
      grid = [["" for _ in range(table.num_cols)] for _ in range(table.num_rows)]
      for cell in table.cells:
          for r in range(cell.row, cell.row + cell.row_span):
              for c in range(cell.col, cell.col + cell.col_span):
                  grid[r][c] = cell.text
      return grid
  ```

  ```typescript TypeScript theme={null}
  function toGrid(table) {
    const grid = Array.from({ length: table.numRows }, () =>
      Array.from({ length: table.numCols }, () => ''),
    );
    for (const cell of table.cells) {
      for (let r = cell.row; r < cell.row + cell.rowSpan; r++) {
        for (let c = cell.col; c < cell.col + cell.colSpan; c++) {
          grid[r][c] = cell.text;
        }
      }
    }
    return grid;
  }
  ```
</CodeGroup>

<Note>
  Each cell carries `is_header`, so you can separate header cells from data cells directly rather than assuming the first row. Group the header cells to build column names, and treat the rest as the body.
</Note>

## Check confidence before trusting a table

A table element carries a `confidence` score. Complex or scanned tables score lower than clean digital ones. When you extract tables at scale, gate on this score and route low-confidence tables to review rather than into a system of record.

```python Python theme={null}
LOW = 0.7

for page in result.pages:
    for element in page.elements:
        if element.label == ParseLayoutLabel.TABLE and element.confidence < LOW:
            print(f"Low-confidence table on page {page.page_number}, flag for review")
```

## Related

<CardGroup>
  <Card title="Output schema" icon="code" href="/document-parsing/reference/output-schema">
    The full cell and table field definitions.
  </Card>

  <Card title="Confidence Scoring" icon="chart-bar" href="/concepts/confidence-scoring">
    How Meibel scores the quality of extracted content.
  </Card>
</CardGroup>


# Parsing scanned documents
Source: https://docs.meibel.ai/document-parsing/guides/parsing-scanned-documents

Parse scans and image-only PDFs, where OCR and orientation correction run automatically, and read the result with confidence in mind

Scanned pages, faxes, and photographed documents carry their text as pixels rather than as an extractable text layer. Meibel handles these through the same endpoints as digital PDFs. OCR runs automatically on the pages that need it, with page orientation corrected, and the structured result comes back in the same shape it has for a digital PDF. This guide covers what to submit, what happens to a scan on the way through, and how to read a result you can trust.

## Submit a scan the same way

There is no separate endpoint or flag for scanned input. Submit the file exactly as you would a digital PDF, and parsing decides per page whether OCR is needed.

<CodeGroup>
  ```python Python theme={null}
  with open("scanned-contract.pdf", "rb") as f:
      job = client.documents.parse(file=f, file_name="scanned-contract.pdf")
  ```

  ```bash curl theme={null}
  curl -X POST https://api.meibel.ai/v2/documents \
    -H "Meibel-API-Key: $MEIBEL_API_KEY" \
    -F "file=@scanned-contract.pdf"
  ```
</CodeGroup>

## What happens to a scanned page

A scanned page cannot yield text by reading a text layer, because there is none. Parsing detects this per page and switches that page to OCR. Three things happen before you see a result:

* **OCR runs only where it is needed.** A scorer weighs signals from each page, including the quality of any embedded text, how much of the page is image, and whether the text reads coherently. A page with a clean text layer skips OCR and stays fast, while a scanned or garbled page is recognized from its image.
* **Orientation is corrected.** A page rotated sideways or upside down, common in scans and faxes, is set upright before recognition, so its text reads correctly.
* **Text is recognized, in many languages.** Recognition reads the characters off the image and covers a large multilingual character set, so documents in non-Latin scripts are handled through the same path.

The result is that a scanned document returns the same typed, positioned elements as a digital one. Your code does not branch on whether the source was scanned.

<Note>
  A document can be mixed: some pages with a clean text layer, some scanned. Parsing decides page by page, so a single file with both kinds is handled efficiently in one pass.
</Note>

## Watch the trace to see OCR run

For a long scan you can watch the work in progress rather than only polling for the final status. The trace stream emits events as pages are extracted and recognition steps complete, which is useful for surfacing progress in a UI or for confirming that OCR ran on the pages you expected.

<CodeGroup>
  ```python Python theme={null}
  for event in client.documents.stream_trace(job_id=job.job_id):
      print(event)
  ```

  ```bash curl theme={null}
  curl "https://api.meibel.ai/v2/documents/$JOB_ID/trace" \
    -H "Meibel-API-Key: $MEIBEL_API_KEY"
  ```
</CodeGroup>

Trace events arrive as Server-Sent Events. Each one is a small object describing a parsing step as it completes.

## Read the result with confidence in mind

OCR is less certain than reading a digital text layer, and the confidence scores reflect that. When you process scans at scale, gate on the document and element confidence and route low-scoring pages to review. This keeps recognition error out of anything downstream that assumes clean text.

```python Python theme={null}
status = client.documents.get_status(job_id=job.job_id)

if (status.confidence or 1.0) < 0.75:
    print("Low overall confidence, review before ingesting")

result = client.documents.get_structured_result(job_id=job.job_id)
for page in result.pages:
    for element in page.elements:
        if element.confidence < 0.6:
            print(f"Review page {page.page_number}: {element.text[:60]}")
```

<Warning>
  A poor original limits what OCR can recover. Low-resolution faxes, heavy skew, and handwriting reduce accuracy. The confidence scores are the signal for when a page needs a human to check it.
</Warning>

## Related

<CardGroup>
  <Card title="How parsing works" icon="gears" href="/document-parsing/concepts/how-parsing-works">
    Where OCR and orientation correction sit in the pipeline.
  </Card>

  <Card title="Confidence Scoring" icon="chart-bar" href="/concepts/confidence-scoring">
    How Meibel evaluates recognition and extraction quality.
  </Card>
</CardGroup>


# Preparing parsed content for RAG
Source: https://docs.meibel.ai/document-parsing/guides/structure-for-rag

Use headings, elements, and provenance from a parse to build well-structured chunks for retrieval-augmented generation

Retrieval-augmented generation (RAG) answers a question by retrieving passages from your documents and handing them to a language model. The quality of those answers depends on the shape of what you indexed. Chunks split on a fixed character count cut across sentences and merge unrelated sections, which weakens both retrieval and the answers built on it. A parse gives you the document's real structure, so you can chunk on section boundaries, keep tables and their captions whole, and carry each chunk's page and position for citation. This guide turns a parse into retrieval-ready chunks.

For how the structure is produced, see [how parsing works](/document-parsing/concepts/how-parsing-works).

## Prerequisites

* A completed parse job.
* The structured result for element structure, or the `markdown` format when you want ready-to-index text.

## Chunk on section boundaries

Headings mark where one topic ends and the next begins, and their level records the section hierarchy. Walking the elements and starting a new chunk at each `Title` or `SectionHeader` keeps a section's content together and splits where the document itself splits. Carrying the current heading path onto each chunk gives every chunk a breadcrumb of where it sits.

<CodeGroup>
  ```python Python theme={null}
  from meibel import ParseLayoutLabel

  result = client.documents.get_structured_result(job_id=job_id)

  chunks, current, heading_path = [], [], []

  def flush():
      if current:
          chunks.append({"heading_path": list(heading_path), "text": "\n".join(current)})
          current.clear()

  for page in result.pages:
      for el in page.elements:
          if el.label in (ParseLayoutLabel.TITLE, ParseLayoutLabel.SECTIONHEADER):
              flush()
              level = el.heading_level or 1
              heading_path[:] = heading_path[: level - 1] + [el.text]
          elif el.text:
              current.append(el.text)
  flush()

  print(f"{len(chunks)} chunks")
  ```

  ```typescript TypeScript theme={null}
  const result = await client.documents.getStructuredResult(jobId);

  const chunks = [];
  let current = [];
  let headingPath = [];
  const flush = () => {
    if (current.length) {
      chunks.push({ headingPath: [...headingPath], text: current.join('\n') });
      current = [];
    }
  };

  for (const page of result.pages) {
    for (const el of page.elements) {
      if (el.label === 'Title' || el.label === 'SectionHeader') {
        flush();
        const level = el.headingLevel ?? 1;
        headingPath = [...headingPath.slice(0, level - 1), el.text];
      } else if (el.text) {
        current.push(el.text);
      }
    }
  }
  flush();
  ```
</CodeGroup>

## Keep tables whole

A table loses its meaning when a fixed-size splitter cuts it in half. Because a table is a single element, you can keep it intact as its own chunk, and pair it with a nearby caption for context. Serializing the grid to Markdown or to rows keeps the structure a language model can read.

```python Python theme={null}
for page in result.pages:
    for el in page.elements:
        if el.label == ParseLayoutLabel.TABLE and el.table:
            t = el.table
            rows = [["" for _ in range(t.num_cols)] for _ in range(t.num_rows)]
            for c in t.cells:
                rows[c.row][c.col] = c.text
            table_text = "\n".join(" | ".join(r) for r in rows)
            chunks.append({"heading_path": list(heading_path), "text": table_text, "kind": "table"})
```

## Carry provenance for citation

Each element carries its `bbox`, and the page it sits on is the page you are walking. Keeping these on a chunk lets an answer cite the page it came from and lets a reviewer find the exact region on the source. Attach them as metadata when you index a chunk.

```python Python theme={null}
def provenance(page, el):
    return {"page": page.page_number, "bbox": el.bbox.model_dump()}
```

<Note>
  For chunks you feed straight to a language model, the `markdown` format is often enough on its own, since it already renders headings, lists, and tables. Reach for the structured result when you need per-element control over chunk boundaries and metadata.
</Note>

## Related

<CardGroup>
  <Card title="The parsed document" icon="diagram-project" href="/document-parsing/concepts/the-parsed-document">
    The element model these chunks are built from.
  </Card>

  <Card title="Datasources" icon="database" href="/concepts/datasources">
    How Meibel indexes and retrieves over prepared content.
  </Card>
</CardGroup>


# Element types
Source: https://docs.meibel.ai/document-parsing/reference/element-types

The content roles an element can take, what each represents, and how it renders

Layout analysis labels every region of a page with a content role. The role is what lets you treat content by its meaning: build an outline from headings, keep tables for data, route charts and formulas to their own handling, and drop page furniture from a body index. This page catalogs the roles.

In the structured result, each element names its role with a `label` from a fixed set. The roles below group those labels by the kind of content they mark.

## Text roles

| Label           | Represents                                                | Rendered in Markdown as  |
| --------------- | --------------------------------------------------------- | ------------------------ |
| `Title`         | The document's main title                                 | `#` heading              |
| `SectionHeader` | A section or subsection heading, with a level from 1 to 6 | `#` to `######` by level |
| `Text`          | A block of body text                                      | Plain paragraph          |
| `Caption`       | A caption for a figure, table, or chart                   | Italic text              |
| `Footnote`      | A footnote                                                | Footnote definition      |
| `Code`          | A block of code                                           | Fenced code block        |
| `Formula`       | A mathematical expression, recognized as notation         | `$$` display math        |
| `DocumentIndex` | A table of contents or index                              | Plain text               |

A `Title` or `SectionHeader` carries a `heading_level` derived from the document's typography, so the section hierarchy is preserved. See [the parsed document](/document-parsing/concepts/the-parsed-document) for how levels are assigned.

## Figure roles

These roles cover regions that are visual rather than textual. Charts, formulas, seals, and pictures are recognized by vision-language models, described in [charts, formulas, and vision models](/document-parsing/concepts/charts-and-vision).

| Label     | Represents         | Notes                                                              |
| --------- | ------------------ | ------------------------------------------------------------------ |
| `Chart`   | A plotted chart    | Carries digitized `chart_data` and recognized labels on `ocr_text` |
| `Seal`    | A seal or stamp    | Carries recognized text                                            |
| `Picture` | An image or figure | Recognized by a vision model                                       |

## Tables

| Label   | Represents       | Notes                                                                                                                                |
| ------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `Table` | A tabular region | Carries a grid of cells with row and column positions and spans; see [extracting tables](/document-parsing/guides/extracting-tables) |

## Page furniture

Page headers and footers are the repeating content at the margins of a page rather than part of the document body. Filtering them out by label keeps them out of a body-text index.

| Label        | Represents                                                     |
| ------------ | -------------------------------------------------------------- |
| `PageHeader` | Repeating content at the top margin                            |
| `PageFooter` | Repeating content at the bottom margin, including page numbers |

## Legacy labels

A few labels remain for backward compatibility but are not produced by the current layout model: `ListItem`, `Form`, `KeyValueRegion`, `CheckboxSelected`, and `CheckboxUnselected`. A list item in a document parsed by the current model comes through as `Text`.

## Handling unknown roles

Treat the set of labels as open. The vocabulary can grow, so branch on the labels your application handles and fall through gracefully on the rest rather than assuming a fixed, exhaustive list.

## Related

<CardGroup>
  <Card title="The parsed document" icon="diagram-project" href="/document-parsing/concepts/the-parsed-document">
    How typed elements form the document model.
  </Card>

  <Card title="Output schema" icon="code" href="/document-parsing/reference/output-schema">
    The fields each element carries.
  </Card>
</CardGroup>


# Formats and capabilities
Source: https://docs.meibel.ai/document-parsing/reference/formats-and-capabilities

Supported inputs, output formats, the capability matrix, container documents, and the parsing endpoints

This page catalogs the renderings parsing returns, the capabilities it recovers, the inputs it accepts, and the endpoints that drive it. For step-by-step usage, see the [tutorial](/document-parsing/tutorials/parse-your-first-document) and the how-to guides.

## Renderings

Markdown and annotated Markdown come from the result endpoint, `GET /documents/{job_id}/result`, selected with the `format` query parameter, which defaults to `markdown`. The strongly-typed structured result has its own endpoint, `GET /documents/{job_id}/structured`.

| Rendering  | How to fetch                         | Shape                                                                       | Best for                          |
| ---------- | ------------------------------------ | --------------------------------------------------------------------------- | --------------------------------- |
| Markdown   | `format=markdown`                    | Readable text with headings, lists, tables, formulas, and chart tables      | Language models and human readers |
| Structured | `GET /documents/{job_id}/structured` | Typed pages of elements, with tables, chart data, positions, and confidence | Programmatic use                  |
| Annotated  | `format=annotated`                   | Markdown with bounding-box provenance comments                              | Text together with position       |

The `format=json` value returns an earlier flat JSON rendering; the structured result is the typed, richer replacement for programmatic use. Guidance on choosing is in [choosing an output format](/document-parsing/guides/choosing-an-output-format). The fields are documented in the [output schema](/document-parsing/reference/output-schema).

## Capability matrix

Each row names a capability parsing recovers, along with the part of the result where it surfaces.

| Capability               | What it does                                             | Where it appears                                                                |
| ------------------------ | -------------------------------------------------------- | ------------------------------------------------------------------------------- |
| Layout analysis          | Labels regions by content role                           | Element `label`; see [element types](/document-parsing/reference/element-types) |
| Heading hierarchy        | Assigns H1 to H6 from the document's typography          | Element `heading_level`                                                         |
| Table structure          | Recovers rows, columns, and spanning cells               | Element `table`                                                                 |
| Reading order            | Sequences content the way a person reads it              | Element `reading_order`                                                         |
| OCR                      | Recognizes text on scanned and image pages, multilingual | Element `text`                                                                  |
| Orientation correction   | Rotates pages upright by 90, 180, or 270 degrees         | Applied before recognition                                                      |
| Chart digitization       | Recovers line and scatter series from vector geometry    | Element `chart_data`                                                            |
| Vision model recognition | Reads formulas, charts, seals, and pictures              | Element `text` and `chart_data`                                                 |
| Confidence               | Scores each element and the document                     | Element `confidence`, document `confidence`                                     |
| Provenance               | Positions every element and chart point                  | Element `bbox`                                                                  |

## Inputs

| Input                     | Handling                                                                                                              |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| Digital PDF               | Text, layout, and tables read directly from the file. Encrypted PDFs are decrypted where possible.                    |
| Scanned or image-only PDF | OCR runs per page, with orientation correction first.                                                                 |
| Image files               | Recognized through the same OCR path as scanned pages.                                                                |
| Office documents          | Word, Excel, and PowerPoint files are parsed into the same element model.                                             |
| Container files           | ZIP, TAR, and EML files expand into child documents, each parsed on its own. See [child documents](#child-documents). |

Mixed PDFs, where some pages carry a text layer and others are scanned, are decided page by page in a single pass. See [parsing scanned documents](/document-parsing/guides/parsing-scanned-documents).

## Job statuses

A job reported by `GET /documents/{job_id}` moves through these statuses.

| Status       | Meaning                            |
| ------------ | ---------------------------------- |
| `queued`     | Submitted and waiting to start.    |
| `processing` | Being parsed.                      |
| `completed`  | Finished; the result is available. |
| `failed`     | Parsing did not complete.          |

Poll until the status reaches `completed` or `failed`. A 2-second interval suits most documents.

## Child documents

A container file expands into one child document per file it holds, and each child is parsed independently. `GET /documents/{job_id}/children` lists them. Each entry reports:

| Field        | Type   | Description                           |
| ------------ | ------ | ------------------------------------- |
| `job_id`     | string | The child's own job identifier.       |
| `filename`   | string | The file's name within the container. |
| `status`     | string | The child's job status.               |
| `media_type` | string | The child's detected media type.      |

Fetch a child's result with its own `job_id`, the same way as any other job.

<Note>
  Container extraction applies safety limits against malicious archives, including caps on the number of files, the total decompressed size, and the nesting depth.
</Note>

## Endpoints

| Method and path                      | Purpose                                                              |
| ------------------------------------ | -------------------------------------------------------------------- |
| `POST /documents`                    | Submit a file for asynchronous parsing. Returns a `job_id`.          |
| `POST /documents/process`            | Parse synchronously and return the result in one call.               |
| `GET /documents/{job_id}`            | Get job status and, when complete, a summary.                        |
| `GET /documents/{job_id}/result`     | Fetch a readable or interchange rendering in the requested `format`. |
| `GET /documents/{job_id}/structured` | Fetch the strongly-typed structured result.                          |
| `GET /documents/{job_id}/children`   | List child documents from a container.                               |
| `GET /documents/{job_id}/trace`      | Stream progress as Server-Sent Events.                               |

<Note>
  The synchronous endpoint suits small files, roughly under 10 MB. For larger documents, submit asynchronously and poll, or stream the trace for progress.
</Note>

Full request and response detail for each endpoint is in the [Documents API reference](/guides/documents).

## Related

<CardGroup>
  <Card title="Output schema" icon="code" href="/document-parsing/reference/output-schema">
    Field-by-field detail of the structured result.
  </Card>

  <Card title="How parsing works" icon="gears" href="/document-parsing/concepts/how-parsing-works">
    The stages behind these capabilities.
  </Card>
</CardGroup>


# Output schema
Source: https://docs.meibel.ai/document-parsing/reference/output-schema

Field-by-field detail of the strongly-typed structured result: the document, pages, elements, tables, cells, bounding boxes, and chart data

This page documents the strongly-typed structured result returned by `GET /documents/{job_id}/structured`, exposed in the SDKs as `get_structured_result`. The result is a `ParseStructuredDocument`: a set of pages, each holding its elements in reading order, with tables, chart data, recognized labels, positions, and confidence carried on the elements themselves. For the readable and interchange formats and how to choose among them, see [choosing an output format](/document-parsing/guides/choosing-an-output-format).

## Document

The top-level object of a structured parse.

| Field               | Type                                    | Description                                                                   |
| ------------------- | --------------------------------------- | ----------------------------------------------------------------------------- |
| `pages`             | array of [Page](#page)                  | The pages, each with its elements in reading order.                           |
| `num_pages`         | integer                                 | Number of pages in the source document.                                       |
| `confidence`        | [Confidence scores](#confidence-scores) | Aggregate confidence across all pages.                                        |
| `format`            | string \| null                          | Detected input format, such as `pdf`, `docx`, or `markdown`.                  |
| `gpu_ms`            | integer \| null                         | GPU inference time across all stages, in milliseconds.                        |
| `ocr_pages`         | integer \| null                         | Number of pages that required OCR.                                            |
| `orientation_pages` | integer \| null                         | Number of pages whose orientation was corrected.                              |
| `remote_regions`    | integer \| null                         | Number of regions sent to vision models, such as formulas, charts, and seals. |

```json theme={null}
{
  "num_pages": 12,
  "format": "pdf",
  "confidence": { "mean_layout_confidence": 0.96, "min_layout_confidence": 0.71, "num_elements": 214, "num_tables": 3 },
  "pages": [
    {
      "page_number": 0,
      "page_bbox": { "x0": 0, "y0": 0, "x1": 612, "y1": 792 },
      "elements": [
        { "label": "Title", "text": "Quarterly Report", "heading_level": 1, "reading_order": 0, "confidence": 0.98, "bbox": { "x0": 72, "y0": 60, "x1": 540, "y1": 96 } }
      ]
    }
  ]
}
```

## Confidence scores

The document-level summary carried on `confidence`.

| Field                    | Type    | Description                                           |
| ------------------------ | ------- | ----------------------------------------------------- |
| `mean_layout_confidence` | number  | Mean layout-detection confidence across all elements. |
| `min_layout_confidence`  | number  | Lowest layout-detection confidence of any element.    |
| `num_elements`           | integer | Total number of elements detected.                    |
| `num_tables`             | integer | Number of tables recognized.                          |

## Page

One page of the document. Elements sit in `elements`, already in reading order.

| Field                 | Type                                   | Description                                                                                                                                                  |
| --------------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `page_number`         | integer                                | The page index, zero-based.                                                                                                                                  |
| `elements`            | array of [Element](#element)           | The page's elements, in reading order.                                                                                                                       |
| `page_bbox`           | [BoundingBox](#bounding-box)           | Page dimensions in PDF points, with a bottom-left origin.                                                                                                    |
| `ocr_applied`         | boolean \| null                        | Whether OCR ran on this page.                                                                                                                                |
| `ocr_score`           | number \| null                         | The OCR-need score, from 0 (has text) to 1 (needs OCR). Present even when OCR did not run.                                                                   |
| `orientation_degrees` | integer \| null                        | Rotation applied to set the page upright: 0, 90, 180, or 270.                                                                                                |
| `image_size`          | array of integer \| null               | Page image dimensions in pixels, when a page image was provided.                                                                                             |
| `transcript_lines`    | array of `ParseTranscriptLine` \| null | Gutter line numbers lifted out of the body text, on a page detected as a line-numbered legal transcript. Each carries its printed `number` and its position. |

## Element

One piece of content. `label`, `text`, `bbox`, `reading_order`, and `confidence` are always present; the rest appear when they apply.

| Field           | Type                                      | Description                                                                                             |
| --------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| `label`         | string                                    | The element's role. See [element types](/document-parsing/reference/element-types).                     |
| `text`          | string                                    | The text content, assembled from the region. Empty for a purely visual element.                         |
| `bbox`          | [BoundingBox](#bounding-box)              | Position on the page, in pixel coordinates with a top-left origin.                                      |
| `reading_order` | integer                                   | Position in reading order, zero-based within the page.                                                  |
| `confidence`    | number                                    | Layout-detection confidence, from 0 to 1.                                                               |
| `heading_level` | integer \| null                           | Heading level from 1 to 6, on `Title` and `SectionHeader` elements.                                     |
| `table`         | [Table](#table) \| null                   | The cell grid, present when `label` is `Table`.                                                         |
| `chart_data`    | [ChartData](#chart-data) \| null          | The digitized plot, present on a `Chart` element when its geometry was recovered.                       |
| `ocr_text`      | array of [ChartText](#chart-text) \| null | Text recognized on a `Chart` region. Sibling to `chart_data`, so it survives when `chart_data` is null. |

<Note>
  Treat `label` as an open set. New roles can appear, so branch on the values you handle and fall through gracefully on the rest.
</Note>

## Table

The grid held by a `Table` element.

| Field         | Type                              | Description                             |
| ------------- | --------------------------------- | --------------------------------------- |
| `cells`       | array of [TableCell](#table-cell) | The cells of the table.                 |
| `num_rows`    | integer                           | Number of rows in the grid.             |
| `num_cols`    | integer                           | Number of columns in the grid.          |
| `page_number` | integer                           | The page the table sits on, zero-based. |
| `bbox`        | [BoundingBox](#bounding-box)      | Position of the whole table.            |

## Table cell

One cell within a table.

| Field       | Type                         | Description                                              |
| ----------- | ---------------------------- | -------------------------------------------------------- |
| `text`      | string                       | The cell's text.                                         |
| `row`       | integer                      | Zero-based row index of the cell's top-left position.    |
| `col`       | integer                      | Zero-based column index of the cell's top-left position. |
| `row_span`  | integer                      | Rows the cell spans; 1 when it does not span.            |
| `col_span`  | integer                      | Columns the cell spans; 1 when it does not span.         |
| `is_header` | boolean                      | Whether the cell is a header cell.                       |
| `bbox`      | [BoundingBox](#bounding-box) | Position of the cell.                                    |

```json theme={null}
{
  "num_rows": 2,
  "num_cols": 3,
  "page_number": 0,
  "cells": [
    { "text": "Region", "row": 0, "col": 0, "row_span": 1, "col_span": 1, "is_header": true },
    { "text": "Q1", "row": 0, "col": 1, "row_span": 1, "col_span": 1, "is_header": true },
    { "text": "Q2", "row": 0, "col": 2, "row_span": 1, "col_span": 1, "is_header": true },
    { "text": "West", "row": 1, "col": 0, "row_span": 1, "col_span": 1, "is_header": false },
    { "text": "120", "row": 1, "col": 1, "row_span": 1, "col_span": 1, "is_header": false },
    { "text": "140", "row": 1, "col": 2, "row_span": 1, "col_span": 1, "is_header": false }
  ]
}
```

Reading this grid, including spans, is covered in [extracting tables](/document-parsing/guides/extracting-tables).

## Bounding box

A rectangle on a page. An element's `bbox` is in pixel coordinates with a top-left origin; a page's `page_bbox` is in PDF points with a bottom-left origin.

| Field | Type   | Description                                         |
| ----- | ------ | --------------------------------------------------- |
| `x0`  | number | Left edge.                                          |
| `y0`  | number | Top or bottom edge, per the box's coordinate space. |
| `x1`  | number | Right edge.                                         |
| `y1`  | number | Bottom or top edge, per the box's coordinate space. |

## Chart data

The digitized plot carried by a `Chart` element's `chart_data`.

| Field                | Type                       | Description                                                                       |
| -------------------- | -------------------------- | --------------------------------------------------------------------------------- |
| `chart_type`         | string                     | `Line`, `Scatter`, `Bar`, `Area`, `Pie`, `Mixed`, or `Unknown`.                   |
| `series`             | array of [Series](#series) | The plotted series.                                                               |
| `x_axis`             | [Axis](#axis)              | Calibration of the x-axis.                                                        |
| `y_axis_left`        | [Axis](#axis) \| null      | Calibration of the left y-axis.                                                   |
| `y_axis_right`       | [Axis](#axis) \| null      | Calibration of the right y-axis, for dual-axis charts.                            |
| `categories`         | array of string            | Category labels, for a categorical x-axis.                                        |
| `plot_area`          | `ParseDualBBox`            | The plotting area inside the axes, given in both PDF points and pixels.           |
| `modality`           | string                     | `Vector` or `Raster`, whether the data came from drawn geometry or from an image. |
| `title`              | string \| null             | The chart title, when detected.                                                   |
| `overall_confidence` | number                     | Confidence in the digitization, from 0 to 1.                                      |
| `warnings`           | array of string            | Notes raised during digitization, such as a value disagreement.                   |

### Series

One plotted series within a chart.

| Field          | Type                              | Description                                                         |
| -------------- | --------------------------------- | ------------------------------------------------------------------- |
| `name`         | string \| null                    | The series name, when detected.                                     |
| `style`        | string                            | `Line`, `Scatter`, `Bar`, `Area`, or `PieSlice`.                    |
| `y_axis`       | string                            | `Left`, `Right`, or `Ambiguous`, the axis the series reads against. |
| `color`        | array of integer \| null          | The drawn RGB color.                                                |
| `dash_pattern` | array of number \| null           | The drawn dash pattern, which tells monochrome series apart.        |
| `points`       | array of [DataPoint](#data-point) | The series' points.                                                 |

### Data point

One digitized value on a series.

| Field           | Type            | Description                                                                                                                                               |
| --------------- | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `x`             | number          | The x value in data units, or the category index when `x_is_category` is true.                                                                            |
| `y`             | number          | The y value in data units.                                                                                                                                |
| `x_is_category` | boolean         | Whether `x` indexes `categories` rather than being a data value.                                                                                          |
| `confidence`    | number          | Confidence in the point.                                                                                                                                  |
| `source`        | string          | Where the value came from: `VectorPath`, `RasterMask`, `RasterMarker`, or `VlmAdjudicated` when a vision model's reading replaced the geometric estimate. |
| `bbox`          | `ParseDualBBox` | Position of the point, given in both PDF points and pixels.                                                                                               |

### Axis

Calibration of one axis, including the fit from pixels to data values.

| Field           | Type                     | Description                                                                               |
| --------------- | ------------------------ | ----------------------------------------------------------------------------------------- |
| `scale`         | string                   | `Linear`, `Log10`, `Categorical`, or `DateTime`.                                          |
| `data_range`    | array of number          | The axis's low and high data values.                                                      |
| `ticks`         | array of `ParseTickMark` | Detected tick marks and their values.                                                     |
| `pixel_to_data` | `ParseAffineFit`         | Linear fit from pixel position to data value, with `slope`, `intercept`, and `r_squared`. |
| `title`         | string \| null           | The axis title.                                                                           |
| `unit`          | string \| null           | The axis unit.                                                                            |

## Chart text

Each entry in an element's `ocr_text` array is one recognized label from a chart, such as an axis title or a data label.

| Field        | Type            | Description                                                                    |
| ------------ | --------------- | ------------------------------------------------------------------------------ |
| `text`       | string          | The recognized text.                                                           |
| `confidence` | number          | Confidence in the recognition.                                                 |
| `source`     | string          | `PdfText` when taken from the document's text, `Ocr` when read from the image. |
| `bbox`       | `ParseDualBBox` | Position, given in both PDF points and pixels.                                 |

Reading these in code is covered in [extracting chart data](/document-parsing/guides/extracting-chart-data).

## Job status

Returned by `GET /documents/{job_id}`. Reports where a job is and, once complete, a summary of what was found.

| Field                | Type            | Description                                       |
| -------------------- | --------------- | ------------------------------------------------- |
| `job_id`             | string          | The job identifier.                               |
| `status`             | string          | `queued`, `processing`, `completed`, or `failed`. |
| `format`             | string          | The result format the job produced.               |
| `pages`              | integer \| null | Page count. Populated when complete.              |
| `elements`           | integer \| null | Element count. Populated when complete.           |
| `tables`             | integer \| null | Table count. Populated when complete.             |
| `confidence`         | number \| null  | Overall confidence. Populated when complete.      |
| `processing_time_ms` | integer \| null | Time spent parsing, in milliseconds.              |
| `error`              | string \| null  | The failure reason, when `status` is `failed`.    |

## Related

<CardGroup>
  <Card title="The parsed document" icon="diagram-project" href="/document-parsing/concepts/the-parsed-document">
    The model these fields express.
  </Card>

  <Card title="Formats and capabilities" icon="table-list" href="/document-parsing/reference/formats-and-capabilities">
    Supported inputs, formats, and job statuses.
  </Card>
</CardGroup>


# Parse your first document
Source: https://docs.meibel.ai/document-parsing/tutorials/parse-your-first-document

Submit a PDF, wait for the job to finish, and read the structured result, all through the Meibel API

A PDF carries its content in a visual layout: headings, tables, and a reading order a person takes in at a glance but that code cannot act on directly. Parsing turns that document into structured content you can search, index, or feed to an agent.

By the end of this tutorial you will have taken a PDF from your disk and read it back two ways: clean Markdown for a person or a model to read, and the strongly-typed structured result for your code to work with. The core flow is three calls: submit the file, poll until it finishes, then fetch the result. You fetch twice here, once as Markdown and once as structured data, to see both renderings. You will work on one document throughout, a public jobs report from the U.S. Bureau of Labor Statistics (BLS), so each step builds on the last.

## Prerequisites

* A Meibel API key. Set it as an environment variable so the examples can read it.
* One of the Meibel SDKs installed, or `curl` for the raw HTTP examples.
* A PDF to parse. The first step downloads a sample; any PDF of your own works too, and a report or an invoice with a table in it shows off the structure best.

<CodeGroup>
  ```bash Environment theme={null}
  export MEIBEL_API_KEY="your-api-key"
  ```

  ```bash Install (Python) theme={null}
  pip install meibel
  ```

  ```bash Install (TypeScript) theme={null}
  npm install meibel
  ```
</CodeGroup>

## 1. Get the sample document

This tutorial works on a public jobs report from the U.S. Bureau of Labor Statistics. It is a good document to parse because it mixes the structure parsing recovers: a title and section headings, a summary table, and a couple of charts. Download the copy Meibel hosts for this tutorial.

```bash theme={null}
curl -fsSL https://storage.googleapis.com/meibel-examples/tutorials/employment-situation.pdf \
  -o employment-situation.pdf
```

## 2. Submit the document

Parsing runs as a job so your program stays responsive while a large file is processed. Submitting a file returns a job ID right away, before the work finishes. You hold onto that ID to check progress and collect the result.

<CodeGroup>
  ```python Python theme={null}
  import os
  from meibel import MeibelClient

  client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

  with open("employment-situation.pdf", "rb") as f:
      job = client.documents.parse(file=f, file_name="employment-situation.pdf")

  print(f"Submitted. Job ID: {job.job_id}")
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient } from 'meibel';
  import { readFile } from 'node:fs/promises';

  const client = new MeibelClient({ apiKey: process.env.MEIBEL_API_KEY });

  const file = new Blob([await readFile('employment-situation.pdf')]);
  const job = await client.documents.parse(file, 'employment-situation.pdf');

  console.log(`Submitted. Job ID: ${job.jobId}`);
  ```

  ```bash curl theme={null}
  curl -X POST https://api.meibel.ai/v2/documents \
    -H "Meibel-API-Key: $MEIBEL_API_KEY" \
    -F "file=@employment-situation.pdf"
  ```
</CodeGroup>

The response carries the `job_id` and an initial `status` of `queued`. The job ID is the handle for everything that follows, so store it.

## 3. Wait for it to finish

A successful job moves through three statuses: `queued`, then `processing`, then `completed`. Polling the status endpoint tells you where it is, and once it reaches `completed` the status also reports what parsing found: the page count, how many elements and tables were extracted, and an overall confidence score. Those numbers are a quick sanity check before you read the full result.

<CodeGroup>
  ```python Python theme={null}
  import time

  while True:
      status = client.documents.get_status(job_id=job.job_id)
      print(f"Status: {status.status}")

      if status.status == "completed":
          print(f"{status.pages} pages, {status.elements} elements, {status.tables} tables")
          print(f"Confidence: {status.confidence}")
          break
      if status.status == "failed":
          raise RuntimeError("Parsing failed")

      time.sleep(2)
  ```

  ```typescript TypeScript theme={null}
  let status;
  do {
    status = await client.documents.getStatus(job.jobId);
    console.log(`Status: ${status.status}`);

    if (status.status === 'failed') {
      throw new Error('Parsing failed');
    }
    if (status.status !== 'completed') {
      await new Promise((r) => setTimeout(r, 2000));
    }
  } while (status.status !== 'completed');

  console.log(`${status.pages} pages, ${status.elements} elements, ${status.tables} tables`);
  ```

  ```bash curl theme={null}
  curl https://api.meibel.ai/v2/documents/$JOB_ID \
    -H "Meibel-API-Key: $MEIBEL_API_KEY"
  ```
</CodeGroup>

<Note>
  A 2-second polling interval works well for most documents. Larger files take longer, so the loop simply runs a few more times.
</Note>

## 4. Read the result as Markdown

With the job complete, you can fetch the result. Markdown is the format to start with: it is the document as readable text, with headings kept as headings, lists as lists, and tables rendered as Markdown tables. This is what you would hand to a language model or drop into a page for a person to read.

<CodeGroup>
  ```python Python theme={null}
  markdown = client.documents.get_result(job_id=job.job_id, format="markdown")
  print(markdown)
  ```

  ```typescript TypeScript theme={null}
  const markdown = await client.documents.getResult(job.jobId, {
    format: 'markdown',
  });
  console.log(markdown);
  ```

  ```bash curl theme={null}
  curl "https://api.meibel.ai/v2/documents/$JOB_ID/result?format=markdown" \
    -H "Meibel-API-Key: $MEIBEL_API_KEY"
  ```
</CodeGroup>

Read through the output. The section headings from your PDF appear as Markdown headings, and any table has become a grid of pipes and dashes. The reading order matches how you would read the page, even if the source had columns.

## 5. Read the same result as structured data

Markdown is for reading. When your code needs to act on the content, fetch the strongly-typed structured result instead. It comes back organized by page, each holding its elements in reading order, and every element is a typed object with a `label`, its `text`, a `bbox` giving its position, a `reading_order`, and a `confidence` score. A `Title` or `SectionHeader` carries a `heading_level`, and a `Table` carries a grid of cells you can address by row and column.

<CodeGroup>
  ```python Python theme={null}
  from meibel import ParseLayoutLabel

  result = client.documents.get_structured_result(job_id=job.job_id)

  for page in result.pages:
      for element in page.elements:
          if element.label in (ParseLayoutLabel.TITLE, ParseLayoutLabel.SECTIONHEADER):
              print("#" * (element.heading_level or 1), element.text)
          elif element.label == ParseLayoutLabel.TABLE and element.table:
              t = element.table
              print(f"[table: {t.num_rows}x{t.num_cols} on page {page.page_number}]")
  ```

  ```typescript TypeScript theme={null}
  const result = await client.documents.getStructuredResult(job.jobId);

  for (const page of result.pages) {
    for (const element of page.elements) {
      if (element.label === 'Title' || element.label === 'SectionHeader') {
        console.log('#'.repeat(element.headingLevel ?? 1), element.text);
      } else if (element.label === 'Table' && element.table) {
        const t = element.table;
        console.log(`[table: ${t.numRows}x${t.numCols} on page ${page.pageNumber}]`);
      }
    }
  }
  ```

  ```bash curl theme={null}
  curl "https://api.meibel.ai/v2/documents/$JOB_ID/structured" \
    -H "Meibel-API-Key: $MEIBEL_API_KEY"
  ```
</CodeGroup>

The same document you submitted is now a set of pages, each a list of typed, positioned elements. You have the readable Markdown for people and models, and the structured result for your program.

## What you learned

You submitted a document, waited for the job, and read a single parse back as both Markdown and structured data. That same flow, submit then poll then fetch, handles any supported input: a digital PDF, a scan, or an office document all return the same structured content, whether you parse one file or run many through these steps. From here:

<CardGroup>
  <Card title="Choosing an output format" icon="list-check" href="/document-parsing/guides/choosing-an-output-format">
    When Markdown, the structured result, or annotated output fits your task.
  </Card>

  <Card title="Extracting tables" icon="table-cells" href="/document-parsing/guides/extracting-tables">
    Turn the table cells you saw here into rows your code can use.
  </Card>

  <Card title="How parsing works" icon="gears" href="/document-parsing/concepts/how-parsing-works">
    Understand the stages behind the result you just read.
  </Card>

  <Card title="Output schema" icon="code" href="/document-parsing/reference/output-schema">
    The full field list for the structured result you just printed.
  </Card>
</CardGroup>


# Managing Agents
Source: https://docs.meibel.ai/guides/agents

Create, configure, publish, version, and delete agents

Agents are the core building block of the Meibel platform. Each agent encapsulates a system prompt, model configuration, and optional datasources. This guide walks through the full lifecycle: creating, inspecting, updating, publishing, versioning, and deleting agents.

## Create an agent

Create a new agent definition by providing a name, description, and system prompt.

<CodeGroup>
  ```python Python theme={null}
  import os
  from meibel import MeibelClient
  from meibel.models import CreateAgentDefinitionRequest

  client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

  agent = client.agents.create(body=CreateAgentDefinitionRequest(
      display_name="Research Assistant",
      description="Answers questions using uploaded knowledge bases",
      instructions="You are a helpful research assistant. Answer questions accurately and cite your sources.",
  ))

  print(agent.id, agent.display_name)
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient } from 'meibel';

  const client = new MeibelClient({ apiKey: process.env.MEIBEL_API_KEY });

  const agent = await client.agents.create({
    displayName: 'Research Assistant',
    description: 'Answers questions using uploaded knowledge bases',
    instructions: 'You are a helpful research assistant. Answer questions accurately and cite your sources.',
  });

  console.log(agent.id, agent.displayName);
  ```

  ```go Go theme={null}
  import (
      "context"
      "fmt"
      "os"

      meibel "github.com/meibel-ai/meibel-go"
  )

  client := meibel.NewClient(meibel.WithAPIKey(os.Getenv("MEIBEL_API_KEY")))
  ctx := context.Background()

  agent, err := client.Agents.CreateAgent(ctx, meibel.CreateAgentDefinitionRequest{
      DisplayName:  "Research Assistant",
      Description:  "Answers questions using uploaded knowledge bases",
      Instructions: "You are a helpful research assistant. Answer questions accurately and cite your sources.",
  })
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(agent.ID, agent.DisplayName)
  ```

  ```bash CLI theme={null}
  meibel agents create --data '{
    "display_name": "Research Assistant",
    "description": "Answers questions using uploaded knowledge bases",
    "instructions": "You are a helpful research assistant. Answer questions accurately and cite your sources."
  }'
  ```
</CodeGroup>

The response includes the agent's `id`, `display_name`, `description`, `instructions`, and timestamps. Use the `id` for all subsequent operations.

## Get agent details

Retrieve the full configuration and metadata for an existing agent.

<CodeGroup>
  ```python Python theme={null}
  agent = client.agents.get(agent_id="agent_123")

  print(agent.display_name)
  print(agent.instructions)
  print(agent.created_at)
  ```

  ```typescript TypeScript theme={null}
  const agent = await client.agents.get('agent_123');

  console.log(agent.displayName);
  console.log(agent.instructions);
  console.log(agent.createdAt);
  ```

  ```go Go theme={null}
  agent, err := client.Agents.GetAgent(ctx, "agent_123")
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(agent.DisplayName)
  fmt.Println(agent.Instructions)
  fmt.Println(agent.CreatedAt)
  ```

  ```bash CLI theme={null}
  meibel agents get agent_123
  ```
</CodeGroup>

## Update an agent

Modify an agent's name, description, system prompt, or other configuration fields. Only the fields you include in the request body are changed.

<CodeGroup>
  ```python Python theme={null}
  from meibel.models import UpdateAgentDefinitionRequest

  updated = client.agents.update(
      agent_id="agent_123",
      body=UpdateAgentDefinitionRequest(
          display_name="Research Assistant v2",
          instructions="You are an expert research assistant. Always cite sources with page numbers.",
      ),
  )

  print(updated.version)
  ```

  ```typescript TypeScript theme={null}
  const updated = await client.agents.update('agent_123', {
    displayName: 'Research Assistant v2',
    instructions: 'You are an expert research assistant. Always cite sources with page numbers.',
  });

  console.log(updated.version);
  ```

  ```go Go theme={null}
  updated, err := client.Agents.UpdateAgent(ctx, "agent_123", meibel.UpdateAgentDefinitionRequest{
      DisplayName:  "Research Assistant v2",
      Instructions: "You are an expert research assistant. Always cite sources with page numbers.",
  })
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(updated.DisplayName)
  ```

  ```bash CLI theme={null}
  meibel agents update agent_123 --data '{
    "display_name": "Research Assistant v2",
    "instructions": "You are an expert research assistant. Always cite sources with page numbers."
  }'
  ```
</CodeGroup>

## Publish an agent

Publishing creates an immutable snapshot of the agent's current configuration. Include a commit message to document what changed.

<CodeGroup>
  ```python Python theme={null}
  from meibel.models import PublishAgentDefinitionRequest

  published = client.agents.publish(
      agent_id="agent_123",
      body=PublishAgentDefinitionRequest(
          commit_message="Improved citation formatting and added page-number references",
      ),
  )

  print(published.version)
  print(published.commit_message)
  ```

  ```typescript TypeScript theme={null}
  const published = await client.agents.publish('agent_123', {
    commitMessage: 'Improved citation formatting and added page-number references',
  });

  console.log(published.version);
  console.log(published.commitMessage);
  ```

  ```go Go theme={null}
  published, err := client.Agents.PublishAgent(ctx, "agent_123", meibel.PublishAgentDefinitionRequest{
      CommitMessage: "Improved citation formatting and added page-number references",
  })
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(published.Version)
  fmt.Println(published.CommitMessage)
  ```

  ```bash CLI theme={null}
  meibel agents publish agent_123 --data '{
    "commit_message": "Improved citation formatting and added page-number references"
  }'
  ```
</CodeGroup>

<Note>
  Publishing does not affect the draft agent. You can continue editing the draft and publish again when ready.
</Note>

## List agent versions

Retrieve all published versions for an agent. Versions are returned in reverse chronological order.

<CodeGroup>
  ```python Python theme={null}
  for version in client.agents.list_versions(agent_id="agent_123"):
      print(f"v{version.version}: {version.commit_message} ({version.published_at})")
  ```

  ```typescript TypeScript theme={null}
  for await (const version of client.agents.listVersions('agent_123')) {
    console.log(`v${version.version}: ${version.commitMessage} (${version.publishedAt})`);
  }
  ```

  ```go Go theme={null}
  iter := client.Agents.ListAgentVersions(ctx, "agent_123")
  for iter.Next(ctx) {
      version := iter.Item()
      fmt.Printf("v%d: %s (%s)\n", version.Version, version.CommitMessage, version.PublishedAt)
  }
  if err := iter.Err(); err != nil {
      log.Fatal(err)
  }
  ```

  ```bash CLI theme={null}
  meibel agents list-agent-versions agent_123
  ```
</CodeGroup>

## Delete an agent

Permanently remove an agent and all its published versions. This action cannot be undone.

<CodeGroup>
  ```python Python theme={null}
  client.agents.delete(agent_id="agent_123")
  ```

  ```typescript TypeScript theme={null}
  await client.agents.delete('agent_123');
  ```

  ```go Go theme={null}
  err := client.Agents.DeleteAgent(ctx, "agent_123")
  if err != nil {
      log.Fatal(err)
  }
  ```

  ```bash CLI theme={null}
  meibel agents delete agent_123
  ```
</CodeGroup>

<Warning>
  Deleting an agent removes all published versions and terminates any active sessions associated with it.
</Warning>


# Data Elements
Source: https://docs.meibel.ai/guides/data-elements

List, search, inspect, and update extracted data elements

Data elements are the individual items extracted from your datasource files during ingestion. Each element has an `id`, a `name`, a `media_type`, an optional `description`, and `metadata`. Agents can retrieve and cite them. This guide covers listing, searching, inspecting, and updating data elements.

## List data elements

Retrieve all data elements for a datasource. Results are paginated automatically by the SDK iterator.

<CodeGroup>
  ```python Python theme={null}
  import os
  from meibel import MeibelClient

  client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

  for element in client.datasources.data_elements.list(datasource_id="ds_123"):
      print(f"{element.id}: {element.name}")
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient } from 'meibel';

  const client = new MeibelClient({ apiKey: process.env.MEIBEL_API_KEY });

  for await (const element of client.datasources.dataElements.list('ds_123')) {
    console.log(`${element.id}: ${element.name}`);
  }
  ```

  ```go Go theme={null}
  import (
      "context"
      "fmt"
      "os"

      meibel "github.com/meibel-ai/meibel-go"
  )

  client := meibel.NewClient(meibel.WithAPIKey(os.Getenv("MEIBEL_API_KEY")))
  ctx := context.Background()

  iter := client.DataElements.ListDataElements(ctx, "ds_123", nil)
  for iter.Next(ctx) {
      element := iter.Item()
      content := element.Content
      if len(content) > 80 {
          content = content[:80] + "..."
      }
      fmt.Printf("%s: %s\n", element.ID, content)
  }
  if err := iter.Err(); err != nil {
      log.Fatal(err)
  }
  ```

  ```bash CLI theme={null}
  meibel data-elements list
  ```
</CodeGroup>

The iterator handles cursor-based pagination transparently. Each element includes `id`, `name`, `media_type`, `description`, and `metadata`.

## Get a data element

Retrieve the full details of a specific data element, including its name, media type, and metadata.

<CodeGroup>
  ```python Python theme={null}
  element = client.datasources.data_elements.get(
      data_element_id="de_456",
      datasource_id="ds_123",
  )

  print(element.id)
  print(element.name)
  print(element.media_type)
  print(element.metadata)
  ```

  ```typescript TypeScript theme={null}
  const element = await client.datasources.dataElements.get('de_456', 'ds_123');

  console.log(element.id);
  console.log(element.name);
  console.log(element.mediaType);
  console.log(element.metadata);
  ```

  ```go Go theme={null}
  element, err := client.DataElements.GetDataElement(ctx, "ds_123", "de_456")
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(element.ID)
  fmt.Println(element.Content)
  fmt.Println(element.SourceFile)
  fmt.Println(element.Metadata)
  ```

  ```bash CLI theme={null}
  meibel data-elements get de_456
  ```
</CodeGroup>

## Search data elements

Search across data elements within a datasource by filtering on the element name with a regular expression, optionally narrowing by media type. Matching elements are returned in `results.items`.

<CodeGroup>
  ```python Python theme={null}
  from meibel.models import DataElementSearchRequest

  results = client.datasources.data_elements.search(
      datasource_id="ds_123",
      body=DataElementSearchRequest(
          regex_filter="quarterly.*revenue",
      ),
  )

  for element in results.items:
      print(f"{element.id}: {element.name}")
  ```

  ```typescript TypeScript theme={null}
  const results = await client.datasources.dataElements.search('ds_123', {
    regexFilter: 'quarterly.*revenue',
  });

  for (const element of results.items) {
    console.log(`${element.id}: ${element.name}`);
  }
  ```

  ```go Go theme={null}
  results, err := client.DataElements.SearchDataElements(ctx, "ds_123", meibel.DataElementSearchRequest{
      RegexFilter: "quarterly.*revenue",
  }, nil)
  if err != nil {
      log.Fatal(err)
  }

  for _, result := range results {
      content := result.Content
      if len(content) > 100 {
          content = content[:100] + "..."
      }
      fmt.Printf("[%.2f] %s\n", result.Score, content)
  }
  ```

  ```bash CLI theme={null}
  meibel data-elements search --data '{
    "regex_filter": "quarterly.*revenue"
  }'
  ```
</CodeGroup>

Each entry in `results.items` is a data element with its `id`, `name`, `media_type`, and `metadata`. Page through larger result sets with `results.has_next` and `results.next_cursor`.

## Update a data element

Modify a data element's name, description, or metadata. This is useful for correcting extraction errors or enriching elements with additional context. See [Managing datasource metadata](/guides/datasource-metadata) for defining metadata fields and indexing them so you can filter and scope by them.

<CodeGroup>
  ```python Python theme={null}
  from meibel.models import UpdateDataElementRequest

  updated = client.datasources.data_elements.update(
      data_element_id="de_456",
      datasource_id="ds_123",
      body=UpdateDataElementRequest(
          description="Corrected figures for Q4 2025.",
          metadata={"reviewed": True, "reviewer": "finance-team"},
      ),
  )

  print(updated.id)
  print(updated.description)
  ```

  ```typescript TypeScript theme={null}
  const updated = await client.datasources.dataElements.update('de_456', 'ds_123', {
    description: 'Corrected figures for Q4 2025.',
    metadata: { reviewed: true, reviewer: 'finance-team' },
  });

  console.log(updated.id);
  console.log(updated.description);
  ```

  ```go Go theme={null}
  updated, err := client.DataElements.UpdateDataElement(ctx, "ds_123", "de_456", meibel.UpdateDataElementRequest{
      Content: "Corrected content with accurate figures for Q4 2025.",
      Metadata: map[string]interface{}{
          "reviewed": true,
          "reviewer": "finance-team",
      },
  })
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(updated.ID)
  fmt.Println(updated.Content)
  ```

  ```bash CLI theme={null}
  meibel data-elements update de_456 --data '{
    "content": "Corrected content with accurate figures for Q4 2025.",
    "metadata": {"reviewed": true, "reviewer": "finance-team"}
  }'
  ```
</CodeGroup>

<Note>
  Updates apply to a data element's `name`, `description`, and `metadata`. Metadata you set here can be indexed for filtering and scoping (see [Managing datasource metadata](/guides/datasource-metadata)).
</Note>


# Managing Datasource Metadata
Source: https://docs.meibel.ai/guides/datasource-metadata

Apply a metadata shape, define custom fields, annotate data elements, and index the fields you want to search and scope by

Metadata is the set of typed fields attached to a datasource's content. Good metadata sharpens retrieval, and once a field is indexed it becomes a dimension an [execution policy](/concepts/execution-policies) can scope a session by. This guide covers the ways to populate metadata and how to index a field so you can filter and govern by it. For the concepts behind metadata and its link to access control, see [Datasources](/concepts/datasources).

## Prerequisites

* A datasource. See [Managing datasources](/guides/datasources) to create one.
* Your API key in the `MEIBEL_API_KEY` environment variable.

## Apply a metadata shape

When your documents fit a familiar domain, a prebuilt shape saves you from defining fields by hand. The catalog ships shapes for common document types, including bibliography, legal, medical, and insurance. Browse it to find a shape and note its `model_id`.

<CodeGroup>
  ```python Python theme={null}
  import os
  from meibel import MeibelClient

  client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

  catalog = client.metadata_model_catalog.list()
  for entry in catalog.models:
      print(entry.model_id, entry.name)
  ```

  ```bash CLI theme={null}
  meibel metadata-model-catalog list
  ```
</CodeGroup>

Set the datasource's metadata configuration to that shape. On the next ingestion the platform extracts the shape's fields from every document.

<CodeGroup>
  ```python Python theme={null}
  from meibel.models import UpdateDatasourceRequest, MetadataConfigRequest

  client.datasources.update(
      datasource_id="ds_123",
      body=UpdateDatasourceRequest(
          metadata_config=MetadataConfigRequest(type="catalog", model_id="bibliography"),
      ),
  )
  ```

  ```bash CLI theme={null}
  meibel metadata-configuration update-config ds_123 --data '{
    "type": "catalog", "model_id": "bibliography"
  }'
  ```
</CodeGroup>

## Define custom fields

When no prebuilt shape fits, define the fields yourself. Each field carries a name, a type, and a description that tells the extractor what to capture. Set `index` to `true` on a field to make it filterable, which is what lets you search and scope by it later.

<CodeGroup>
  ```python Python theme={null}
  from meibel.models import UpdateDatasourceRequest, MetadataConfigRequest, MetadataField

  client.datasources.update(
      datasource_id="ds_123",
      body=UpdateDatasourceRequest(
          metadata_config=MetadataConfigRequest(
              type="custom",
              fields=[
                  MetadataField(
                      name="product_line",
                      type="string",
                      description="The product line a document covers, e.g. commercial or consumer",
                      index=True,
                  ),
                  MetadataField(
                      name="effective_date",
                      type="datetime",
                      description="The date the document takes effect",
                      index=True,
                  ),
              ],
          ),
      ),
  )
  ```

  ```bash CLI theme={null}
  meibel metadata-configuration update-config ds_123 --data '{
    "type": "custom",
    "fields": [
      { "name": "product_line", "type": "string", "description": "The product line a document covers, e.g. commercial or consumer", "index": true },
      { "name": "effective_date", "type": "datetime", "description": "The date the document takes effect", "index": true }
    ]
  }'
  ```
</CodeGroup>

A field's type is one of `string`, `integer`, `float`, `boolean`, `datetime`, `uuid`, `geo`, or `list[string]`.

<Note>
  A fixed number of fields can be indexed per data type, so index the fields you most need to filter and scope by. An unindexed field is still extracted and still describes its data, but cannot be filtered on.
</Note>

## Annotate a data element

Automatic extraction covers most values, but sometimes you know a value the extractor cannot infer, or you need to correct one it got wrong. Set metadata directly on a single data element.

<CodeGroup>
  ```python Python theme={null}
  from meibel.models import UpdateDataElementRequest

  client.datasources.data_elements.update(
      data_element_id="de_456",
      datasource_id="ds_123",
      body=UpdateDataElementRequest(
          metadata={"product_line": "commercial", "reviewed": True},
      ),
  )
  ```

  ```bash CLI theme={null}
  meibel data-element-metadata update ds_123 de_456 --data '{
    "metadata": { "product_line": "commercial", "reviewed": true }
  }'
  ```
</CodeGroup>

A value you set by hand is preserved when the datasource is re-ingested, so a later ingest keeps your annotation rather than overwriting it with a fresh extraction.

## Scope access with an indexed field

Indexing is what connects metadata to access control. Once a field is indexed, an [execution policy](/concepts/execution-policies) can filter on it, so you can hold each session to the slice of data a given user should see. With `product_line` indexed, a policy can restrict a session to a single line:

```json theme={null}
{
  "datasources": {
    "ds_123": {
      "documents": {
        "filter": { "product_line": "commercial" }
      }
    }
  }
}
```

A session created with this policy retrieves only documents whose `product_line` is `commercial`. A field that was never indexed cannot appear in a filter like this, which is why the fields you choose to index set the boundary of what you can govern. For the steps to create and apply policies, see [Managing execution policies](/guides/execution-policies).

## Related guides

<CardGroup>
  <Card title="Datasources" icon="database" href="/concepts/datasources">
    How metadata, retrieval, and access control fit together.
  </Card>

  <Card title="Managing Datasources" icon="database" href="/guides/datasources">
    Create datasources, add content, and trigger ingestion.
  </Card>

  <Card title="Data Elements" icon="cube" href="/guides/data-elements">
    List, search, and inspect the data elements metadata attaches to.
  </Card>

  <Card title="Managing Execution Policies" icon="shield-halved" href="/guides/execution-policies">
    Scope sessions by filtering on indexed metadata fields.
  </Card>
</CardGroup>


# Managing Datasources
Source: https://docs.meibel.ai/guides/datasources

Create datasources, upload files, trigger ingestion, and manage the data pipeline

A datasource is where Meibel keeps the data your agents draw on. It holds two shapes of content: unstructured documents and structured tables. Uploading a file only begins the process: when a datasource ingests your files, it parses each one, recovers the structure inside it, and extracts metadata as it goes. Your documents become data elements an agent can search by meaning, and your tables, along with other data suited to tabular representation, become queryable by their columns and values. By the time ingestion finishes, an agent bound to the datasource can retrieve from your files directly as it reasons through a task.

This guide walks through the full datasource lifecycle: creating a datasource, uploading files, triggering ingestion, retrieving its details and status, updating it, and deleting it. By the end you will know how to take a datasource from empty to queryable and how to manage it as your content changes. For detailed explanations of what a datasource is and how agents query it, see the [Datasources concept](/concepts/datasources). To learn how to configure the metadata you can search and scope by, see [Managing datasource metadata](/guides/datasource-metadata).

The examples work on one datasource throughout: **Q4 Financial Reports**, which holds quarterly earnings reports and analyst briefings. It starts with a single uploaded PDF, `earnings-q4.pdf`, and each step below acts on that same datasource, so the snippets follow in order as one walkthrough. Set your API key in the `MEIBEL_API_KEY` environment variable before you begin.

## Create a datasource

Every datasource starts empty. You create one with a name and a description to organize it by. You supply its content yourself by uploading files, which the next step covers.

<CodeGroup>
  ```python Python theme={null}
  import os
  from meibel import MeibelClient

  client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

  datasource = client.datasources.create(
      name="Q4 Financial Reports",
      description="Quarterly earnings reports and analyst briefings",
  )

  print(datasource.id, datasource.name)
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient } from 'meibel';

  const client = new MeibelClient({ apiKey: process.env.MEIBEL_API_KEY });

  const datasource = await client.datasources.create({
    name: 'Q4 Financial Reports',
    description: 'Quarterly earnings reports and analyst briefings',
  });

  console.log(datasource.id, datasource.name);
  ```
</CodeGroup>

The response includes the datasource `id`, `name`, `description`, and timestamps. Every operation that follows needs this `id`. The snippets reuse the `datasource` object returned here, so keep it in scope as you work through the steps.

## Upload files

With a datasource in place, add the files whose contents you want agents to reach. Each call uploads one file, so repeat it for every file you want to add. The SDK streams the file to the server in chunks rather than loading it into memory first, so a large document uploads without exhausting memory. Uploading stores a file but does not process it. Its contents become searchable only after ingestion, which the next step triggers.

<CodeGroup>
  ```python Python theme={null}
  with open("earnings-q4.pdf", "rb") as f:
      upload = client.datasources.file_uploads.upload_content(
          datasource_id=datasource.id,
          files=f,
          files_name="earnings-q4.pdf",
      )

  print(upload.upload_id)
  ```

  ```typescript TypeScript theme={null}
  import { readFile } from "node:fs/promises";

  const fileBlob = new Blob([await readFile("earnings-q4.pdf")]);
  const upload = await client.datasources.fileUploads.uploadContent(datasource.id, fileBlob, "earnings-q4.pdf");

  console.log(upload.uploadId);
  ```
</CodeGroup>

<Note>
  You can upload multiple files to the same datasource. Supported formats include PDF, DOCX, XLSX, CSV, TXT, and JSON.
</Note>

## Trigger ingestion

Ingestion is the step that turns uploaded files into queryable content. The pipeline parses each file, breaks documents into data elements, reads structured files into tables, and extracts metadata along the way. Trigger it once your files are in place. You can trigger it again later after adding more files or changing the datasource's metadata configuration, and the pipeline reprocesses the content accordingly.

<CodeGroup>
  ```python Python theme={null}
  result = client.datasources.ingest.trigger(datasource_id=datasource.id)

  print(result.message)
  ```

  ```typescript TypeScript theme={null}
  const result = await client.datasources.ingest.trigger(datasource.id);

  console.log(result.message);
  ```
</CodeGroup>

Ingestion runs asynchronously, so the call returns before processing finishes. Track its progress by polling the datasource's status, described next, or by subscribing to streaming events.

## Get datasource details

Fetching a datasource returns its current state, which is how you check where it stands. The `last_sync_status` field reflects the most recent ingest run, and `total_ingested_files` reports how many files have been ingested. Read the status here to confirm ingestion has finished before you rely on the datasource in an agent.

<CodeGroup>
  ```python Python theme={null}
  current = client.datasources.get(datasource_id=datasource.id)

  print(current.name)
  print(current.last_sync_status)
  print(current.total_ingested_files)
  ```

  ```typescript TypeScript theme={null}
  const current = await client.datasources.get(datasource.id);

  console.log(current.name);
  console.log(current.lastSyncStatus);
  console.log(current.totalIngestedFiles);
  ```
</CodeGroup>

## Update a datasource

Updating changes a datasource's name, description, or configuration after you create it. Here you rename **Q4 Financial Reports** to add the year and record that it now reflects the final audited numbers. The update is partial: only the fields you include in the request body change, and anything you leave out keeps its current value. Changing configuration that affects how content is processed, such as the metadata shape, takes effect on the next ingestion rather than immediately.

<CodeGroup>
  ```python Python theme={null}
  from meibel.models import UpdateDatasourceRequest

  updated = client.datasources.update(
      datasource_id=datasource.id,
      body=UpdateDatasourceRequest(
          name="Q4 2025 Financial Reports",
          description="Updated with final audited numbers",
      ),
  )

  print(updated.name)
  ```

  ```typescript TypeScript theme={null}
  const updated = await client.datasources.update(datasource.id, {
    name: 'Q4 2025 Financial Reports',
    description: 'Updated with final audited numbers',
  });

  console.log(updated.name);
  ```
</CodeGroup>

## Delete a datasource

Deleting removes a datasource along with every file and data element it holds. The removal is permanent, so reserve it for datasources you are sure you no longer need, and check what depends on the datasource before you delete it.

<CodeGroup>
  ```python Python theme={null}
  client.datasources.delete(datasource_id=datasource.id)
  ```

  ```typescript TypeScript theme={null}
  await client.datasources.delete(datasource.id);
  ```
</CodeGroup>

<Warning>
  Deleting a datasource removes all uploaded files and extracted data elements. Agents that reference this datasource will lose access to its content.
</Warning>


# Deep Transform
Source: https://docs.meibel.ai/guides/deep-transform

Extract schema-conformant JSON from a document, with per-value provenance

Deep transform extracts the entities you care about from a document and returns JSON that conforms to a schema you define. You describe those entities once as a JSON Schema, submit it with the document, and the extraction assembles them across every page into a single result. Each value carries the page region it came from, so you can trace any field in the output back to its source.

The work is entity-driven rather than page-driven, so the same schema runs against a three-page letter or a fifteen-hundred-page contract without change. Pages are processed in parallel, and references to the same entity on different pages land on the same place in the output.

<Note>
  Deep transform is a preview feature. The request and response shapes may change before general availability. The SDK helper methods shown here are available in the 2.0.4 Python and TypeScript SDKs.
</Note>

## Before you begin

You need three things to run an extraction:

* **An API key**, sent in the `Meibel-API-Key` header. The SDKs read it from the `MEIBEL_API_KEY` environment variable in the examples below.
* **A JSON Schema** describing the entities to extract. The shape of this schema is the shape of your output: objects become nested objects, arrays become collections, and scalar fields become single values. A field declared as a string comes back as a string; a field declared as an object comes back fully populated.
* **A document** to extract from, such as a PDF.
* **Domain guidance (optional)**, sent as `guidance`. This is free-text instruction about how your domain names and structures things: the terminology it uses, how to disambiguate values that look alike, and which fields to prioritize. Specialized documents extract more accurately when the model is told the conventions the document assumes its reader already knows.

## Submit an extraction

Submitting sends the document and schema together and returns a job ID right away, so the extraction runs server-side while your application stays responsive. The `root_name` names the top-level entity in your schema, and `max_pages` caps how many pages are processed, which is useful for a quick trial before a full run. Pass your domain instructions as `guidance` so the extraction reads the document the way a specialist in your field would.

<CodeGroup>
  ```python Python theme={null}
  import json
  import os
  from meibel import MeibelClient

  client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

  with open("schema.json") as f:
      schema = json.load(f)

  with open("guidance.md") as f:
      guidance = f.read()

  job = client.documents.submit_deep_transform(
      file="contract.pdf",
      schema=schema,
      root_name="contract",
      guidance=guidance,
  )

  print("Job submitted:", job.job_id)
  ```

  ```typescript TypeScript theme={null}
  import { readFile } from "node:fs/promises";
  import { MeibelClient } from "meibel";

  const client = new MeibelClient({ apiKey: process.env.MEIBEL_API_KEY });

  const schema = JSON.parse(await readFile("schema.json", "utf8"));
  const guidance = await readFile("guidance.md", "utf8");
  const file = new File([await readFile("contract.pdf")], "contract.pdf", {
    type: "application/pdf",
  });

  const job = await client.documents.submitDeepTransform({
    file,
    schema,
    rootName: "contract",
    guidance,
  });

  console.log("Job submitted:", job.jobId);
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.meibel.ai/v2/documents/deep-transform \
    -H "Meibel-API-Key: $MEIBEL_API_KEY" \
    -F "file=@contract.pdf;type=application/pdf" \
    -F "schema=<schema.json" \
    -F "root_name=contract" \
    -F "guidance=<guidance.md"
  ```
</CodeGroup>

Store the returned job ID to check status and download results. Submission is idempotent on the document and schema pair, so repeating the same request returns the same job rather than starting a new one.

The examples read `guidance` from a `guidance.md` file, which keeps longer instructions out of your code. Guidance is optional, so you can omit it for documents that need no domain context, and it can grow from a sentence to a full page as a domain demands.

## Check status

A deep transform runs asynchronously across the document's pages, so you poll the job until it reaches a terminal state. The status moves through `queued` and `running` to either `succeeded` or `failed`.

<CodeGroup>
  ```python Python theme={null}
  import time

  while True:
      status = client.documents.get_deep_transform_status(job.job_id)
      print("Status:", status.status)

      if status.status in ("succeeded", "failed"):
          break

      time.sleep(5)
  ```

  ```typescript TypeScript theme={null}
  let status;

  do {
    status = await client.documents.getDeepTransformStatus(job.jobId);
    console.log("Status:", status.status);

    if (status.status === "succeeded" || status.status === "failed") {
      break;
    }

    await new Promise((resolve) => setTimeout(resolve, 5000));
  } while (true);
  ```

  ```bash cURL theme={null}
  curl https://api.meibel.ai/v2/documents/deep-transform/$JOB_ID \
    -H "Meibel-API-Key: $MEIBEL_API_KEY"
  ```
</CodeGroup>

<Note>
  A succeeded job also reports `metrics` (such as wall-clock time and identity resolution rate) and an `aeq` extraction-quality score, alongside the list of `artifacts` you can download.
</Note>

## Download results

Once the job succeeds, the extraction is available as named artifacts. The `output.json` artifact holds the schema-conformant result. The `provenance.json` artifact maps each value in that result to its source spans, so a reviewer can land on the exact page region a field came from.

<CodeGroup>
  ```python Python theme={null}
  import json

  result = client.documents.download_deep_transform_artifact(
      job.job_id, "output.json"
  )

  # The artifact is a parsed JSON object shaped like your schema.
  print(json.dumps(result, indent=2))
  ```

  ```typescript TypeScript theme={null}
  const result = await client.documents.downloadDeepTransformArtifact(
    job.jobId,
    "output.json",
  );

  // The artifact is a parsed JSON object shaped like your schema.
  console.log(JSON.stringify(result, null, 2));
  ```

  ```bash cURL theme={null}
  curl https://api.meibel.ai/v2/documents/deep-transform/$JOB_ID/artifact/output.json \
    -H "Meibel-API-Key: $MEIBEL_API_KEY"
  ```
</CodeGroup>

The result conforms to the schema you submitted, and values for the same entity are merged from every page they appear on. The nesting and field names mirror your schema exactly.

<Note>
  You can review provenance visually in the Meibel app instead of reading `provenance.json` by hand. Open the Transform menu in the app sidebar and enter your job ID, or go straight to `https://app.meibel.ai/projects/{project_id}/transform/job/{job_id}` with your project and job IDs substituted. Either way you see each extracted value linked to the page region it came from.
</Note>

## Reuse a parsed document

If you have already parsed a document through the Documents API, you can run a deep transform against that parse instead of uploading the file again. Submit to `POST /documents/deep-transform/from-document` with the parse job ID and your schema, and the document is not parsed a second time.

<CardGroup>
  <Card title="Prompts & Artifact Schemas" icon="pen-fancy" href="/guides/prompts-and-schemas">
    Define the JSON Schema that shapes your extraction output.
  </Card>

  <Card title="Document Processing" icon="file" href="/guides/documents">
    Parse a document first, then reuse that parse in a deep transform.
  </Card>
</CardGroup>


# Document Processing
Source: https://docs.meibel.ai/guides/documents

Parse documents asynchronously or synchronously, poll for results, and stream trace events

The document processing API extracts structured content from uploaded files. You can process documents asynchronously (submit a job, poll for results) or synchronously (block until done). This guide covers both workflows, plus streaming trace events for real-time progress.

## Parse a document (async)

Submit a document for asynchronous parsing. The API returns a job ID immediately so your application stays responsive while the server processes the file.

<CodeGroup>
  ```python Python theme={null}
  import os
  from meibel import MeibelClient

  client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

  with open("contract.pdf", "rb") as f:
      job = client.documents.parse(file=f, file_name="contract.pdf")

  print(f"Job submitted: {job.job_id}")
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient } from 'meibel';
  import { readFile } from "node:fs/promises";

  const client = new MeibelClient({ apiKey: process.env.MEIBEL_API_KEY });

  const file = new Blob([await readFile("contract.pdf")]);
  const job = await client.documents.parse(file, "contract.pdf");

  console.log('Job submitted:', job.jobId);
  ```

  ```go Go theme={null}
  import (
      "context"
      "fmt"
      "os"

      meibel "github.com/meibel-ai/meibel-go"
  )

  client := meibel.NewClient(meibel.WithAPIKey(os.Getenv("MEIBEL_API_KEY")))
  ctx := context.Background()

  f, err := os.Open("contract.pdf")
  if err != nil {
      log.Fatal(err)
  }
  defer f.Close()

  job, err := client.Documents.ParseDocument(ctx, f, "contract.pdf")
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println("Job submitted:", job.JobID)
  ```

  ```bash CLI theme={null}
  meibel documents parse --file contract.pdf
  ```
</CodeGroup>

Store the `job_id` from the response to check status, retrieve results, and stream trace events.

## Poll for status

Check the processing status of a submitted document job. Poll until the status reaches `"completed"` or `"failed"`.

<CodeGroup>
  ```python Python theme={null}
  import time

  while True:
      status = client.documents.get_status(job_id=job.job_id)
      print(f"Status: {status.status}")

      if status.status == "completed":
          print("Processing finished")
          break
      elif status.status == "failed":
          print(f"Processing failed: {status.error}")
          break

      time.sleep(2)
  ```

  ```typescript TypeScript theme={null}
  let status;
  do {
    status = await client.documents.getStatus(job.jobId);
    console.log('Status:', status.status);

    if (status.status === 'failed') {
      console.error('Processing failed:', status.error);
      break;
    }

    if (status.status !== 'completed') {
      await new Promise((r) => setTimeout(r, 2000));
    }
  } while (status.status !== 'completed');

  console.log('Processing finished');
  ```

  ```go Go theme={null}
  for {
      status, err := client.Documents.GetDocumentStatus(ctx, job.JobID)
      if err != nil {
          log.Fatal(err)
      }

      fmt.Println("Status:", status.Status)

      if status.Status == "completed" {
          fmt.Println("Processing finished")
          break
      }
      if status.Status == "failed" {
          fmt.Println("Processing failed:", status.Error)
          break
      }

      time.Sleep(2 * time.Second)
  }
  ```

  ```bash CLI theme={null}
  meibel documents get-status "$JOB_ID"
  ```
</CodeGroup>

<Note>
  A 2-second polling interval is recommended. For long-running jobs, consider using the streaming trace endpoint instead.
</Note>

## Get results

Once processing is complete, retrieve the extracted content in markdown or structured JSON format.

<CodeGroup>
  ```python Python theme={null}
  # Get results as markdown
  markdown_result = client.documents.get_result(
      job_id=job.job_id,
      format="markdown",
  )
  print(markdown_result)

  # Get results as structured JSON
  json_result = client.documents.get_result(
      job_id=job.job_id,
      format="json",
  )
  print(json_result)
  ```

  ```typescript TypeScript theme={null}
  // Get results as markdown
  const markdownResult = await client.documents.getResult(job.jobId, {
    format: 'markdown',
  });
  console.log(markdownResult);

  // Get results as structured JSON
  const jsonResult = await client.documents.getResult(job.jobId, {
    format: 'json',
  });
  console.log(jsonResult);
  ```

  ```go Go theme={null}
  // Get results as markdown
  markdownFmt := "markdown"
  markdownResult, err := client.Documents.GetDocumentResult(ctx, job.JobID, &meibel.GetDocumentResultOptions{
      Format: &markdownFmt,
  })
  if err != nil {
      log.Fatal(err)
  }
  fmt.Println(markdownResult.Content)

  // Get results as structured JSON
  jsonFmt := "json"
  jsonResult, err := client.Documents.GetDocumentResult(ctx, job.JobID, &meibel.GetDocumentResultOptions{
      Format: &jsonFmt,
  })
  if err != nil {
      log.Fatal(err)
  }
  fmt.Println(jsonResult.Content)
  ```

  ```bash CLI theme={null}
  # Markdown format
  meibel documents get-result "$JOB_ID" --format markdown

  # JSON format
  meibel documents get-result "$JOB_ID" --format json
  ```
</CodeGroup>

The `markdown` format returns a clean, readable representation of the document. The `json` format returns structured data including headings, tables, and extracted metadata.

## Process synchronously

For smaller documents where you want the result in a single call, use the synchronous endpoint. It blocks until processing completes and returns the result directly.

<CodeGroup>
  ```python Python theme={null}
  with open("invoice.pdf", "rb") as f:
      result = client.documents.process(file=f, file_name="invoice.pdf")

  print(result.result)
  ```

  ```typescript TypeScript theme={null}
  const invoiceBlob = new Blob([await readFile("invoice.pdf")]);
  const result = await client.documents.process(invoiceBlob, "invoice.pdf");

  console.log(result.result);
  ```

  ```go Go theme={null}
  f, err := os.Open("invoice.pdf")
  if err != nil {
      log.Fatal(err)
  }
  defer f.Close()

  result, err := client.Documents.ProcessDocument(ctx, f, "invoice.pdf", nil)
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(result.Content)
  ```

  ```bash CLI theme={null}
  meibel documents parse --file invoice.pdf --wait
  ```
</CodeGroup>

<Note>
  The synchronous endpoint is best for small files (under 10 MB). For larger documents, use the async workflow with polling or trace streaming.
</Note>

## List child documents

Some documents (e.g., archives, multi-part files) produce child documents during processing. List them by job ID.

<CodeGroup>
  ```python Python theme={null}
  children = client.documents.list_children(job_id=job.job_id)

  for child in children:
      print(f"{child.filename}: {child.status}")
  ```

  ```typescript TypeScript theme={null}
  const children = await client.documents.listChildren(job.jobId);

  for (const child of children) {
    console.log(`${child.filename}: ${child.status}`);
  }
  ```

  ```go Go theme={null}
  children, err := client.Documents.ListDocumentChildren(ctx, job.JobID)
  if err != nil {
      log.Fatal(err)
  }

  for _, child := range children {
      fmt.Printf("%s: %s\n", child.FileName, child.Status)
  }
  ```

  ```bash CLI theme={null}
  meibel documents list-children "$JOB_ID"
  ```
</CodeGroup>

## Stream trace events

Stream real-time processing events for a document job. Trace events provide fine-grained progress updates such as page extraction, OCR steps, and content classification.

<CodeGroup>
  ```python Python theme={null}
  for event in client.documents.stream_trace(job_id=job.job_id):
      print(f"[{event.event}] {event.data}")
  ```

  ```typescript TypeScript theme={null}
  for await (const event of client.documents.streamTrace(job.jobId)) {
    console.log(event);
  }
  ```

  ```go Go theme={null}
  stream, err := client.Documents.StreamDocumentTrace(ctx, job.JobID)
  if err != nil {
      log.Fatal(err)
  }

  for event := range stream.Events() {
      fmt.Printf("[%s] %s\n", event.Type, event.Message)
  }
  if err := stream.Err(); err != nil {
      log.Fatal(err)
  }
  ```

  ```bash CLI theme={null}
  meibel documents stream-trace "$JOB_ID"
  ```
</CodeGroup>

Trace events are delivered as Server-Sent Events (SSE). In Python, each event is an `SSEEvent` with an `.event` name and a `.data` payload; call `event.json()` to parse the payload into a dict. In TypeScript, each event is the already-parsed payload object. The stream ends with a terminal event once processing completes.


# Managing Execution Policies
Source: https://docs.meibel.ai/guides/execution-policies

Create, apply, update, and delete execution policies that control data and tool access for agent sessions

Execution policies are [hardrails](/concepts/execution-policies) that limit what data and tools an agent session can access, enforced by the platform rather than by the LLM. You can store policies as named, reusable objects and then apply them when creating sessions, defining agents, or running batch jobs. This guide covers the full lifecycle: creating policies, managing them, and applying them to agent executions.

## Create an execution policy

Create a stored execution policy by providing a name and the policy payload. The policy defines datasource constraints, tool constraints, or both.

<CodeGroup>
  ```python Python theme={null}
  import os
  from meibel import MeibelClient
  from meibel.models import CreateExecutionPolicyRequest

  client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

  policy = client.execution_policies.create(body=CreateExecutionPolicyRequest(
      name="EU Region Policy",
      description="Restricts data access to EU region and hides sensitive columns",
      execution_policy={
          "datasources": {
              "ds_abc123": {
                  "tables": {
                      "filter": {
                          "table.__name__": {"$in": ["orders", "customers"]},
                          "orders.region": "EU",
                      },
                      "hidden_columns": {
                          "customers": ["ssn", "credit_card"],
                      },
                  },
              },
          },
          "tools": {
              "web_search": {"disabled": True},
          },
      },
  ))

  print(policy.id, policy.name)
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient } from 'meibel';

  const client = new MeibelClient({ apiKey: process.env.MEIBEL_API_KEY });

  const policy = await client.executionPolicies.createExecutionPolicy({
    name: 'EU Region Policy',
    description: 'Restricts data access to EU region and hides sensitive columns',
    executionPolicy: {
      datasources: {
        ds_abc123: {
          tables: {
            filter: {
              'table.__name__': { $in: ['orders', 'customers'] },
              'orders.region': 'EU',
            },
            hidden_columns: {
              customers: ['ssn', 'credit_card'],
            },
          },
        },
      },
      tools: {
        web_search: { disabled: true },
      },
    },
  });

  console.log(policy.id, policy.name);
  ```
</CodeGroup>

The response includes the policy's `id`, `name`, `description`, `execution_policy`, and timestamps. Use the `id` for all subsequent operations.

<Note>
  Policy names must be unique within your project. Creating a policy with a name that already exists returns an error.
</Note>

## List execution policies

Retrieve all execution policies in your project.

<CodeGroup>
  ```python Python theme={null}
  policies = client.execution_policies.list()

  for p in policies.data:
      print(f"{p.id}: {p.name}")
  ```

  ```typescript TypeScript theme={null}
  const policies = await client.executionPolicies.listExecutionPolicies();

  for (const p of policies.data) {
    console.log(`${p.id}: ${p.name}`);
  }
  ```
</CodeGroup>

## Get execution policy details

Retrieve a specific execution policy by ID.

<CodeGroup>
  ```python Python theme={null}
  policy = client.execution_policies.get(policy_id="pol_abc123")

  print(policy.name)
  print(policy.execution_policy)
  ```

  ```typescript TypeScript theme={null}
  const policy = await client.executionPolicies.getExecutionPolicy('pol_abc123');

  console.log(policy.name);
  console.log(policy.executionPolicy);
  ```
</CodeGroup>

## Update an execution policy

Modify a policy's name, description, or constraints. Only the fields you include in the request body are changed.

<CodeGroup>
  ```python Python theme={null}
  from meibel.models import UpdateExecutionPolicyRequest

  updated = client.execution_policies.update(
      policy_id="pol_abc123",
      body=UpdateExecutionPolicyRequest(
          description="Updated: EU region with tighter tool constraints",
          execution_policy={
              "datasources": {
                  "ds_abc123": {
                      "tables": {
                          "filter": {
                              "table.__name__": {"$in": ["orders", "customers"]},
                              "orders.region": "EU",
                          },
                          "hidden_columns": {
                              "customers": ["ssn", "credit_card"],
                          },
                      },
                  },
              },
              "tools": {
                  "web_search": {"disabled": True},
                  "send_email": {
                      "variables": {
                          "to": "support@acme.com",
                      },
                  },
              },
          },
      ),
  )

  print(updated.name)
  ```

  ```typescript TypeScript theme={null}
  const updated = await client.executionPolicies.updateExecutionPolicy('pol_abc123', {
    description: 'Updated: EU region with tighter tool constraints',
    executionPolicy: {
      datasources: {
        ds_abc123: {
          tables: {
            filter: {
              'table.__name__': { $in: ['orders', 'customers'] },
              'orders.region': 'EU',
            },
            hidden_columns: {
              customers: ['ssn', 'credit_card'],
            },
          },
        },
      },
      tools: {
        web_search: { disabled: true },
        send_email: {
          variables: {
            to: 'support@acme.com',
          },
        },
      },
    },
  });

  console.log(updated.name);
  ```
</CodeGroup>

## Delete an execution policy

Remove an execution policy. The policy is soft-deleted and no longer appears in list results.

<CodeGroup>
  ```python Python theme={null}
  client.execution_policies.delete(policy_id="pol_abc123")
  ```

  ```typescript TypeScript theme={null}
  await client.executionPolicies.deleteExecutionPolicy('pol_abc123');
  ```
</CodeGroup>

<Warning>
  Deleting a policy does not retroactively affect sessions or executions that already used it. Those sessions retain the policy that was composed at creation time.
</Warning>

## Apply policies to a session

Pass stored policy IDs, an inline policy, or both when creating a session. When both are provided, they are composed together.

<CodeGroup>
  ```python Python theme={null}
  from meibel.models import CreateSessionRequest

  # Apply a stored policy by ID
  session = client.agents.sessions.create(
      agent_id="agent_abc123",
      body=CreateSessionRequest(
          execution_policy_ids=["pol_abc123"],
      ),
  )

  print(session.session_id)
  ```

  ```typescript TypeScript theme={null}
  // Apply a stored policy by ID
  const session = await client.agents.sessions.create('agent_abc123', {
    executionPolicyIds: ['pol_abc123'],
  });

  console.log(session.sessionId);
  ```
</CodeGroup>

You can also pass an inline policy directly, which is useful for per-user constraints that do not need to be stored:

<CodeGroup>
  ```python Python theme={null}
  session = client.agents.sessions.create(
      agent_id="agent_abc123",
      body=CreateSessionRequest(
          execution_policy={
              "datasources": {
                  "ds_abc123": {
                      "documents": {
                          "filter": {
                              "data_element.__id__": {"$in": ["de_001", "de_002"]},
                          },
                      },
                  },
              },
          },
      ),
  )
  ```

  ```typescript TypeScript theme={null}
  const session = await client.agents.sessions.create('agent_abc123', {
    executionPolicy: {
      datasources: {
        ds_abc123: {
          documents: {
            filter: {
              'data_element.__id__': { $in: ['de_001', 'de_002'] },
            },
          },
        },
      },
    },
  });
  ```
</CodeGroup>

When you pass both `execution_policy_ids` and `execution_policy`, the stored policies are resolved first, then the inline policy is composed on top. All filters are merged with `$and` semantics, so the effective policy is the intersection of all constraints.

## Apply policies to an agent definition

Set default execution policies on an agent definition so they apply to every session created against that agent. These can be stored policy references, an inline policy, or both.

<CodeGroup>
  ```python Python theme={null}
  from meibel.models import CreateAgentDefinitionRequest

  agent = client.agents.create(body=CreateAgentDefinitionRequest(
      display_name="Support Agent",
      instructions="You are a support assistant. Answer questions using the knowledge base.",
      execution_policy_ids=["pol_abc123", "pol_def456"],
      execution_policy={
          "tools": {
              "delete_record": {"disabled": True},
          },
      },
  ))

  print(agent.id)
  ```

  ```typescript TypeScript theme={null}
  const agent = await client.agents.create({
    displayName: 'Support Agent',
    instructions: 'You are a support assistant. Answer questions using the knowledge base.',
    executionPolicyIds: ['pol_abc123', 'pol_def456'],
    executionPolicy: {
      tools: {
        delete_record: { disabled: true },
      },
    },
  });

  console.log(agent.id);
  ```
</CodeGroup>

Definition-level policies compose with session-level policies when a session is created. The definition policies are applied first, then session-level policies are layered on top. Because composition only narrows access, session-level policies cannot grant access beyond what the agent definition allows.

## Apply policies to a batch execution

Pass execution policies when executing a batch definition to scope the data each item in the batch can access.

<CodeGroup>
  ```python Python theme={null}
  execution = client.batches.execute(definition_id="bdef_abc123")

  print(execution.execution_id)
  ```

  ```typescript TypeScript theme={null}
  const execution = await client.batches.executeBatchDefinition('bdef_abc123', {
    executionPolicyIds: ['pol_abc123'],
    executionPolicy: {
      datasources: {
        ds_abc123: {
          tables: {
            filter: {
              'orders.region': 'EU',
            },
          },
        },
      },
    },
  });

  console.log(execution.executionId);
  ```
</CodeGroup>

As with agent sessions, batch-level policies compose with any policies set on the underlying batch definition.


# Prompts & Artifact Schemas
Source: https://docs.meibel.ai/guides/prompts-and-schemas

Create reusable prompts and define structured output schemas for agents

Prompts and artifact schemas let you define reusable building blocks for agent behavior. Prompts provide templated instructions that can be shared across agents, while artifact schemas define the structured output format agents should produce.

## Create a prompt

Create a reusable prompt template. Prompts can include variables using double-brace syntax (e.g., `{{topic}}`) that are filled in at runtime.

<CodeGroup>
  ```python Python theme={null}
  import os
  from meibel import MeibelClient
  from meibel.models import CreateAgentPromptRequest

  client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

  prompt = client.prompts.create_prompt(body=CreateAgentPromptRequest(
      name="summarize-document",
      description="Summarizes a document with key takeaways",
      content="Summarize the following document in {{length}} bullet points. Focus on {{focus_area}}. Include citations for each point.",
  ))

  print(prompt.id, prompt.name)
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient } from 'meibel';

  const client = new MeibelClient({ apiKey: process.env.MEIBEL_API_KEY });

  const prompt = await client.prompts.createPrompt({
    name: 'summarize-document',
    description: 'Summarizes a document with key takeaways',
    content: 'Summarize the following document in {{length}} bullet points. Focus on {{focus_area}}. Include citations for each point.',
  });

  console.log(prompt.id, prompt.name);
  ```

  ```go Go theme={null}
  import (
      "context"
      "fmt"
      "os"

      meibel "github.com/meibel-ai/meibel-go"
  )

  client := meibel.NewClient(meibel.WithAPIKey(os.Getenv("MEIBEL_API_KEY")))
  ctx := context.Background()

  prompt, err := client.Prompts.CreatePrompt(ctx, meibel.CreateAgentPromptRequest{
      Name:        "summarize-document",
      Description: "Summarizes a document with key takeaways",
      Content:     "Summarize the following document in {{length}} bullet points. Focus on {{focus_area}}. Include citations for each point.",
  })
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(prompt.ID, prompt.Name)
  ```

  ```bash CLI theme={null}
  meibel prompts create --data '{
    "name": "summarize-document",
    "description": "Summarizes a document with key takeaways",
    "content": "Summarize the following document in {{length}} bullet points. Focus on {{focus_area}}. Include citations for each point."
  }'
  ```
</CodeGroup>

The response includes the prompt `id`, `name`, `description`, and `content`. Use the `id` to reference the prompt from agents or other API calls.

## Get a prompt

Retrieve the full details of an existing prompt.

<CodeGroup>
  ```python Python theme={null}
  prompt = client.prompts.get_prompt(prompt_id="prompt_123")

  print(prompt.name)
  print(prompt.content)
  ```

  ```typescript TypeScript theme={null}
  const prompt = await client.prompts.getPrompt('prompt_123');

  console.log(prompt.name);
  console.log(prompt.content);
  ```

  ```go Go theme={null}
  prompt, err := client.Prompts.GetPrompt(ctx, "prompt_123")
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(prompt.Name)
  fmt.Println(prompt.Content)
  ```

  ```bash CLI theme={null}
  meibel prompts get prompt_123
  ```
</CodeGroup>

## Update a prompt

Modify a prompt's name, description, or content. Only the fields you include are changed.

<CodeGroup>
  ```python Python theme={null}
  from meibel.models import UpdateAgentPromptRequest

  updated = client.prompts.update_prompt(
      prompt_id="prompt_123",
      body=UpdateAgentPromptRequest(
          content="Provide a {{length}}-point summary of the document. Focus on {{focus_area}}. Cite sources with page numbers.",
      ),
  )

  print(updated.content)
  ```

  ```typescript TypeScript theme={null}
  const updated = await client.prompts.updatePrompt('prompt_123', {
    content: 'Provide a {{length}}-point summary of the document. Focus on {{focus_area}}. Cite sources with page numbers.',
  });

  console.log(updated.content);
  ```

  ```go Go theme={null}
  updated, err := client.Prompts.UpdatePrompt(ctx, "prompt_123", meibel.UpdateAgentPromptRequest{
      Content: "Provide a {{length}}-point summary of the document. Focus on {{focus_area}}. Cite sources with page numbers.",
  })
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(updated.Content)
  ```

  ```bash CLI theme={null}
  meibel prompts update prompt_123 --data '{
    "content": "Provide a {{length}}-point summary of the document. Focus on {{focus_area}}. Cite sources with page numbers."
  }'
  ```
</CodeGroup>

<Note>
  Updating a prompt does not affect agents that have already been published with the old version. Only new sessions or republished agents pick up the changes.
</Note>

## Delete a prompt

Permanently remove a prompt. This action cannot be undone.

<CodeGroup>
  ```python Python theme={null}
  client.prompts.delete_prompt(prompt_id="prompt_123")
  ```

  ```typescript TypeScript theme={null}
  await client.prompts.deletePrompt('prompt_123');
  ```

  ```go Go theme={null}
  err := client.Prompts.DeletePrompt(ctx, "prompt_123")
  if err != nil {
      log.Fatal(err)
  }
  ```

  ```bash CLI theme={null}
  meibel prompts delete prompt_123
  ```
</CodeGroup>

## Create an artifact schema

Artifact schemas define the structured output format that an agent should produce. Use JSON Schema to specify the fields, types, and validation rules.

<CodeGroup>
  ```python Python theme={null}
  schema = client.artifact_schemas.create(
      display_name="Meeting Summary",
      type="json",
      description="Structured output for meeting summaries",
      schema={
          "type": "object",
          "properties": {
              "title": {"type": "string", "description": "Meeting title"},
              "date": {"type": "string", "format": "date"},
              "attendees": {
                  "type": "array",
                  "items": {"type": "string"},
              },
              "action_items": {
                  "type": "array",
                  "items": {
                      "type": "object",
                      "properties": {
                          "description": {"type": "string"},
                          "assignee": {"type": "string"},
                          "due_date": {"type": "string", "format": "date"},
                      },
                      "required": ["description"],
                  },
              },
              "key_decisions": {
                  "type": "array",
                  "items": {"type": "string"},
              },
          },
          "required": ["title", "date", "action_items"],
      },
  )

  print(schema.id, schema.name)
  ```

  ```typescript TypeScript theme={null}
  const schema = await client.artifactSchemas.create({
    displayName: 'Meeting Summary',
    type: 'json',
    description: 'Structured output for meeting summaries',
    schema: {
      type: 'object',
      properties: {
        title: { type: 'string', description: 'Meeting title' },
        date: { type: 'string', format: 'date' },
        attendees: {
          type: 'array',
          items: { type: 'string' },
        },
        actionItems: {
          type: 'array',
          items: {
            type: 'object',
            properties: {
              description: { type: 'string' },
              assignee: { type: 'string' },
              dueDate: { type: 'string', format: 'date' },
            },
            required: ['description'],
          },
        },
        keyDecisions: {
          type: 'array',
          items: { type: 'string' },
        },
      },
      required: ['title', 'date', 'actionItems'],
    },
  });

  console.log(schema.id, schema.name);
  ```

  ```go Go theme={null}
  schema, err := client.ArtifactSchemas.CreateArtifactSchema(ctx, meibel.CreateAgentArtifactRequest{
      Name:        "meeting-summary",
      Description: "Structured output for meeting summaries",
      Schema: map[string]interface{}{
          "type": "object",
          "properties": map[string]interface{}{
              "title":     map[string]interface{}{"type": "string", "description": "Meeting title"},
              "date":      map[string]interface{}{"type": "string", "format": "date"},
              "attendees": map[string]interface{}{"type": "array", "items": map[string]interface{}{"type": "string"}},
              "action_items": map[string]interface{}{
                  "type": "array",
                  "items": map[string]interface{}{
                      "type": "object",
                      "properties": map[string]interface{}{
                          "description": map[string]interface{}{"type": "string"},
                          "assignee":    map[string]interface{}{"type": "string"},
                          "due_date":    map[string]interface{}{"type": "string", "format": "date"},
                      },
                      "required": []string{"description"},
                  },
              },
              "key_decisions": map[string]interface{}{
                  "type":  "array",
                  "items": map[string]interface{}{"type": "string"},
              },
          },
          "required": []string{"title", "date", "action_items"},
      },
  })
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(schema.ID, schema.Name)
  ```

  ```bash CLI theme={null}
  meibel artifact-schemas create --data '{
    "name": "meeting-summary",
    "description": "Structured output for meeting summaries",
    "schema": {
      "type": "object",
      "properties": {
        "title": {"type": "string", "description": "Meeting title"},
        "date": {"type": "string", "format": "date"},
        "attendees": {"type": "array", "items": {"type": "string"}},
        "action_items": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "description": {"type": "string"},
              "assignee": {"type": "string"},
              "due_date": {"type": "string", "format": "date"}
            },
            "required": ["description"]
          }
        },
        "key_decisions": {"type": "array", "items": {"type": "string"}}
      },
      "required": ["title", "date", "action_items"]
    }
  }'
  ```
</CodeGroup>

## Use a schema with an agent

Reference an artifact schema when creating or updating an agent so that its responses conform to the defined structure.

<CodeGroup>
  ```python Python theme={null}
  from meibel.models import CreateAgentDefinitionRequest

  agent = client.agents.create(body=CreateAgentDefinitionRequest(
      display_name="Meeting Summarizer",
      description="Produces structured meeting summaries",
      instructions="You are a meeting summarizer. Extract key information and format it according to the provided schema.",
      artifacts=[schema.name],
  ))

  print(agent.id, agent.name)
  ```

  ```typescript TypeScript theme={null}
  const agent = await client.agents.create({
    displayName: 'Meeting Summarizer',
    description: 'Produces structured meeting summaries',
    instructions: 'You are a meeting summarizer. Extract key information and format it according to the provided schema.',
    artifacts: [schema.name],
  });

  console.log(agent.id, agent.name);
  ```

  ```go Go theme={null}
  agent, err := client.Agents.CreateAgent(ctx, meibel.CreateAgentDefinitionRequest{
      DisplayName:  "Meeting Summarizer",
      Description:  "Produces structured meeting summaries",
      Instructions: "You are a meeting summarizer. Extract key information and format it according to the provided schema.",
      Artifacts:    []string{schema.ID},
  })
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(agent.ID, agent.Artifacts)
  ```

  ```bash CLI theme={null}
  meibel agents create --data '{
    "display_name": "Meeting Summarizer",
    "description": "Produces structured meeting summaries",
    "instructions": "You are a meeting summarizer. Extract key information and format it according to the provided schema.",
    "artifacts": ["'"$SCHEMA_ID"'"]
  }'
  ```
</CodeGroup>

When an agent has an artifact schema attached, its responses will include a structured `artifact` field alongside the natural language message. The artifact conforms to the JSON Schema you defined, making it straightforward to parse and store programmatically.


# Sessions & Chat
Source: https://docs.meibel.ai/guides/sessions-and-chat

Create sessions, send messages, and stream responses

Sessions represent a conversation between a user and an agent. Each session maintains its own message history and context window. This guide covers creating sessions, sending messages (synchronously or via streaming), and retrieving conversation history.

## Create a session

Start a new conversation by creating a session tied to a specific agent.

<CodeGroup>
  ```python Python theme={null}
  import os
  from meibel import MeibelClient
  from meibel.models import CreateSessionRequest

  client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

  session = client.agents.sessions.create(
      agent_id="agent_123",
      body=CreateSessionRequest(),
  )

  print(session.session_id)
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient } from 'meibel';

  const client = new MeibelClient({ apiKey: process.env.MEIBEL_API_KEY });

  const session = await client.agents.sessions.create('agent_123');

  console.log(session.sessionId);
  ```

  ```go Go theme={null}
  import (
      "context"
      "fmt"
      "os"

      meibel "github.com/meibel-ai/meibel-go"
  )

  client := meibel.NewClient(meibel.WithAPIKey(os.Getenv("MEIBEL_API_KEY")))
  ctx := context.Background()

  session, err := client.Agents.CreateSession(ctx, "agent_123", nil)
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(session.ID)
  fmt.Println(session.AgentID)
  ```

  ```bash CLI theme={null}
  meibel agents create-session agent_123
  ```
</CodeGroup>

The response includes the session `session_id`. Store it to send messages and retrieve history.

## Send a chat message (synchronous)

Send a message and wait for the complete response. This is the simplest approach and works well when you do not need to display partial results.

<CodeGroup>
  ```python Python theme={null}
  from meibel.models import ChatMessageRequest

  response = client.agents.sessions.send_chat_message(
      session_id=session.session_id,
      body=ChatMessageRequest(
          user_message="What are the key findings in the Q4 report?",
      ),
  )

  print(response.assistant_response)
  print(response.response.sources)
  ```

  ```typescript TypeScript theme={null}
  const response = await client.agents.sessions.sendChatMessage(session.sessionId, {
    userMessage: 'What are the key findings in the Q4 report?',
  });

  console.log(response.assistantResponse);
  console.log(response.response.sources);
  ```

  ```go Go theme={null}
  response, err := client.Sessions.SendChatMessage(ctx, session.ID, meibel.ChatMessageRequest{
      UserMessage: "What are the key findings in the Q4 report?",
  })
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(response.Message)
  fmt.Println(response.Citations)
  ```

  ```bash CLI theme={null}
  meibel sessions send-chat-message "$SESSION_ID" --data '{
    "user_message": "What are the key findings in the Q4 report?"
  }'
  ```
</CodeGroup>

The response includes the assistant's reply in `assistant_response`, any `sources` (under `response.sources`) referencing datasource content, and metadata such as token usage.

## Stream a chat response (SSE)

For a more responsive user experience, stream the response as Server-Sent Events. Each event delivers a chunk of the assistant's reply as it is generated.

<CodeGroup>
  ```python Python theme={null}
  import io

  stream = client.agents.sessions.send_chat_message_stream(
      session_id=session.session_id,
      file=io.BytesIO(b""),
      file_name="empty.txt",
      user_message="Summarize the revenue trends over the last 3 quarters.",
  )

  shown = 0
  for event in stream:
      if event.event == "partial_response":
          message = event.json()["data"]["message"]
          print(message[shown:], end="", flush=True)
          shown = len(message)
      elif event.event == "completion":
          print("\n\n[Stream complete]")
  ```

  ```typescript TypeScript theme={null}
  const stream = client.agents.sessions.sendChatMessageStream(
    session.sessionId,
    undefined,
    undefined,
    { userMessage: 'Summarize the revenue trends over the last 3 quarters.' },
  );

  let shown = 0;
  for await (const event of stream) {
    const e = event as { type?: string; data?: { message?: string } };
    if (e.type === 'partial_response') {
      const message = e.data?.message ?? '';
      process.stdout.write(message.slice(shown));
      shown = message.length;
    } else if (e.type === 'completion') {
      console.log('\n\n[Stream complete]');
    }
  }
  ```

  ```go Go theme={null}
  stream, err := client.Sessions.SendChatMessageStream(ctx, session.ID, meibel.ChatMessageRequest{
      UserMessage: "Summarize the revenue trends over the last 3 quarters.",
  })
  if err != nil {
      log.Fatal(err)
  }

  for event := range stream.Events {
      switch e := event.(type) {
      case *meibel.ContentEvent:
          fmt.Print(e.Delta)
      case *meibel.CitationsEvent:
          fmt.Println("\n\nCitations:", e.Citations)
      case *meibel.DoneEvent:
          fmt.Println("\n\n[Stream complete]")
      }
  }
  if err := stream.Err(); err != nil {
      log.Fatal(err)
  }
  ```

  ```bash CLI theme={null}
  meibel sessions send-chat-message-stream "$SESSION_ID" --data '{
    "user_message": "Summarize the revenue trends over the last 3 quarters."
  }'
  ```
</CodeGroup>

<Note>
  The streaming endpoint uses Server-Sent Events (SSE). In Python each event is an `SSEEvent` (read `event.event` for the name and `event.json()` for the payload); in TypeScript each event is the already-parsed payload. Common event names are `partial_response` (the cumulative reply so far) and `completion` (stream finished), with `connected` and `status` events along the way.
</Note>

## Get session message history

Retrieve all messages exchanged in a session, in chronological order.

<CodeGroup>
  ```python Python theme={null}
  messages = client.sessions.get_messages(session_id=session.session_id)

  for msg in messages.messages:
      print(f"[{msg.type}] {msg.message}")
  ```

  ```typescript TypeScript theme={null}
  const messages = await client.sessions.getMessages(session.sessionId);

  for (const msg of messages.messages) {
    console.log(`[${msg.type}] ${msg.message}`);
  }
  ```

  ```go Go theme={null}
  messages, err := client.Sessions.GetSessionMessages(ctx, session.ID)
  if err != nil {
      log.Fatal(err)
  }

  for _, msg := range messages {
      fmt.Printf("[%s] %s\n", msg.Role, msg.Message)
  }
  ```

  ```bash CLI theme={null}
  meibel sessions get-session-messages "$SESSION_ID"
  ```
</CodeGroup>

Each message includes `type` (either `"user"` or `"assistant"`), the `message` content, and a timestamp.

## Get session details

Retrieve metadata about a session, including its agent and status.

<CodeGroup>
  ```python Python theme={null}
  session = client.sessions.get(session_id="session_456")

  print(session.agent_id)
  print(session.agent_name)
  print(session.status)
  ```

  ```typescript TypeScript theme={null}
  const session = await client.sessions.get('session_456');

  console.log(session.agentId);
  console.log(session.agentName);
  console.log(session.status);
  ```

  ```go Go theme={null}
  session, err := client.Sessions.GetSession(ctx, "session_456")
  if err != nil {
      log.Fatal(err)
  }

  fmt.Println(session.ID)
  fmt.Println(session.AgentID)
  fmt.Println(session.CreatedAt)
  ```

  ```bash CLI theme={null}
  meibel sessions get-session session_456
  ```
</CodeGroup>


# Meibel
Source: https://docs.meibel.ai/index

Parse, transform, ingest, and put agents on your data. Powerful primitives, one API.

<div>
  <div>
    <p>Meibel documentation</p>

    # Powerful primitives. <br />One API. <br />Deeply integrated.

    <p>
      Parse documents, transform their content into your schema, ingest it into a queryable datasource, and put agents on top. Each through a single API, with no vector store to run, no graph database to wire up, and no separate orchestration layer to maintain.
    </p>

    <div>
      <a href="/installation">Get started</a>
      <a href="/api-reference/introduction">API reference</a>
    </div>
  </div>

  <div>
    <div>Where to go next</div>
    <h2>Browse the docs</h2>
    <p>Jump into any section of the documentation.</p>
  </div>

  <CardGroup>
    <Card title="Get started" icon="rocket" href="/installation">
      Install the SDK, set your API key, and make your first call.
    </Card>

    <Card title="Core concepts" icon="cube" href="/concepts/agents">
      Agents, datasources, batches, execution policies, and confidence scoring.
    </Card>

    <Card title="Guides" icon="book-open" href="/guides/agents">
      Task-first walkthroughs for agents, sessions, data, and documents.
    </Card>

    <Card title="API reference" icon="code" href="/api-reference/introduction">
      Every endpoint, parameter, and response, with a live playground.
    </Card>

    <Card title="SDKs" icon="terminal" href="/sdk/python">
      Python, TypeScript, Go, and CLI. One client covers every primitive.
    </Card>

    <Card title="Tutorials" icon="graduation-cap" href="/tutorials/datasource-csv-tutorial">
      End-to-end examples like asking a million-row CSV precise questions.
    </Card>
  </CardGroup>

  <div>
    <div>The pipeline in code</div>
    <h2>From data to agent</h2>
    <p>One pipeline, four stages: parse a document, transform to your schema, ingest into a datasource, put an agent on top.</p>
  </div>

  <Tabs>
    <Tab title="1. Parse">
      ```python parse.py theme={null}
      import os
      from meibel import MeibelClient

      client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

      # Parse any document into structured content, with
      # hierarchy, tables, and visual elements preserved.
      with open("rfq.pdf", "rb") as f:
          parsed = client.documents.process(file=f, file_name="rfq.pdf")

      print(parsed.result)
      ```
    </Tab>

    <Tab title="2. Transform">
      ```python transform.py theme={null}
      # Reshape document content into your schema,
      # with per-field confidence scores. Reference a
      # schema by name, or pass a dict or Pydantic model.
      extracted = client.documents.transform(
          file="rfq.pdf",
          schema="quote_v3",
      )

      print(extracted.data)
      ```
    </Tab>

    <Tab title="3. Ingest">
      ```python ingest.py theme={null}
      from meibel.models import MoveDocumentsRequest

      # Move the parsed document straight into a new datasource,
      # by job ID. No re-upload, no vector store to manage.
      moved = client.documents.move(
          body=MoveDocumentsRequest(
              documents=[parsed.job_id],
              new_datasource_name="rfqs",
          )
      )

      client.datasources.ingest.trigger(datasource_id=moved.datasource_id)
      ```
    </Tab>

    <Tab title="4. Agent">
      ```python agent.py theme={null}
      from meibel.models import (
          CreateAgentDefinitionRequest,
          ChatMessageRequest,
      )

      # An agent binds to one or more datasources, holding
      # unstructured documents, structured tables, or both.
      agent = client.agents.create(
          body=CreateAgentDefinitionRequest(
              display_name="Quote reviewer",
              instructions="Answer using only the ingested RFQs.",
              datasources=[moved.datasource_id],
          )
      )

      # Open a session, then chat.
      session = client.agents.sessions.create(agent_id=agent.id)

      reply = client.agents.sessions.send_chat_message(
          session_id=session.session_id,
          body=ChatMessageRequest(user_message="Draft a response to this RFQ."),
      )
      print(reply.assistant_response)
      ```
    </Tab>
  </Tabs>

  <div>
    <div>Building blocks</div>
    <h2>Powerful primitives</h2>
    <p>Each through a single API. Use one, or compose them into a full pipeline.</p>
  </div>

  <CardGroup>
    <Card title="01 · Parsing" icon="file-lines" href="/guides/documents">
      Turn PDFs, spreadsheets, and images into typed content, with hierarchy, tables, and visual elements preserved.
    </Card>

    <Card title="02 · Transform" icon="wand-magic-sparkles" href="/guides/deep-transform">
      Reshape any source into your schema with per-field confidence scores.
    </Card>

    <Card title="03 · Datasources" icon="database" href="/guides/datasources">
      A living, queryable store for structured and unstructured data, with no vector database to manage.
    </Card>

    <Card title="04 · Agents" icon="robot" href="/guides/agents">
      Reasoning that knows your data and your rules, scoped by execution policies and run one-off or in batches.
    </Card>
  </CardGroup>

  <div>
    <div>How the stack composes</div>
    <h2>Deeply integrated.</h2>
    <p>The primitives aren't just co-located. Each one folds its output into the next, so quality compounds down the pipeline instead of degrading. What parsing captures, transform reshapes; what transform yields, ingestion indexes; what ingestion structures, agents reason over; and policy and confidence thread through all of it.</p>
  </div>

  <div>
    <div />

    <div />

    <div />

    <div>
      <div>
        <span>04</span>

        <div>
          <div>
            <span>Agents</span>
            <span>REASONING</span>
          </div>

          <p>Reason over structured and unstructured data, on retrieval sharpened by everything below. The cleaner the layers below, the sharper the answers.</p>
        </div>
      </div>
    </div>

    <div>
      <div>
        <span>03</span>

        <div>
          <div>
            <span>Ingest</span>
            <span>INDEX</span>
          </div>

          <p>Runs two paths at once: structured data lands in typed, queryable tables, while unstructured content is chunked along semantic boundaries into dense and sparse embeddings for hybrid retrieval. Both live in one datasource, with no vector store to manage.</p>
        </div>
      </div>
    </div>

    <div>
      <div>
        <span>02</span>

        <div>
          <div>
            <span>Transform</span>
            <span>SCHEMA</span>
          </div>

          <p>Reshapes parsed content into your business schema, with per-field confidence. The structured fields it yields fold into ingestion and sharpen what agents retrieve.</p>
        </div>
      </div>
    </div>

    <div>
      <div>
        <span>01</span>

        <div>
          <div>
            <span>Parse</span>
            <span>FOUNDATION</span>
          </div>

          <p>Captures deep structure (sections, headings, tables, lists) plus context for charts, graphs and formulas. Every extracted token carries a confidence score. Everything above inherits this structure.</p>
        </div>
      </div>
    </div>
  </div>

  <div>
    <span>
      <span />

      structure & quality compound upward
    </span>
  </div>

  <div>
    <div>Enforced by the platform</div>
    <h2>Hardrails, not guardrails</h2>
    <p>Lock agents to the data and tools you trust, enforced in code, not vibes. Execution policies strictly define what an agent can see and do. They're evaluated by the platform, outside the LLM, so the model can't bypass them through prompting.</p>
    <p>The result: agents that stay in-bounds by construction, not persuasion.</p>
  </div>

  <Tabs>
    <Tab title="Unstructured data">
      ```json policy.json theme={null}
      {
        "datasources": {
          "ds_abc123": {
            "documents": {
              "filter": {
                "data_element.__id__": { "$in": ["de_abc", "de_def"] }
              }
            }
          }
        }
      }
      ```
    </Tab>

    <Tab title="Structured data">
      ```json policy.json theme={null}
      {
        "datasources": {
          "ds_abc123": {
            "tables": {
              "filter": {
                "table.__name__": { "$in": ["orders", "customers"] },
                "orders.region": "EU"
              },
              "hidden_columns": {
                "customers": ["ssn", "credit_card"]
              }
            }
          }
        }
      }
      ```
    </Tab>

    <Tab title="Tools">
      ```json policy.json theme={null}
      {
        "tools": {
          "send_email": {
            "variables": {
              "to": "support@acme.com",
              "max_attachments": { "$lte": 3 }
            }
          },
          "web_search": { "disabled": true }
        }
      }
      ```
    </Tab>
  </Tabs>

  <div>
    <CardGroup>
      <Card title="Before the LLM call" icon="filter">Disabled tools and off-limits tables are stripped out of the schemas the agent sees. Constraints show up directly in parameter descriptions.</Card>
      <Card title="During data access" icon="database">Table filters are pushed down into SQL and document filters into vector search. The agent never even sees rows or documents that don't match.</Card>
      <Card title="At tool execution" icon="stamp">Every tool call is validated against the policy before it runs. Non-compliant calls are blocked: no side effects, no exceptions.</Card>
    </CardGroup>
  </div>

  <div>
    <div>Get started</div>
    <h2>Connected in under a minute</h2>
    <p>Install the SDK, verify your key, and make your first call.</p>
  </div>

  <Steps>
    <Step title="Install the SDK">
      ```bash theme={null}
      pip install meibel
      ```
    </Step>

    <Step title="Verify your key">
      ```python verify.py theme={null}
      import os
      from meibel import MeibelClient

      client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

      result = client.datasources.list()
      print(f"Connected: {len(result.datasources)} datasource(s) found")
      ```
    </Step>

    <Step title="Make your first call">
      Head to the [quickstart](/quickstart) for a full walkthrough.
    </Step>
  </Steps>
</div>


# Installation
Source: https://docs.meibel.ai/installation

Install Meibel SDKs in your development environment

## SDK Installation

Choose the SDK for your language and install it with your package manager.

<CodeGroup>
  ```bash Python theme={null}
  pip install meibel
  ```

  ```bash TypeScript theme={null}
  npm install meibel
  ```

  ```bash Go theme={null}
  go get github.com/meibel-ai/meibel-go
  ```
</CodeGroup>

## Environment Setup

All SDKs authenticate with an API key. Get yours from the [Meibel Console](https://app.meibel.ai), then set it as an environment variable.

<CodeGroup>
  ```bash macOS / Linux theme={null}
  export MEIBEL_API_KEY="mbl_your_api_key_here"
  ```

  ```powershell Windows theme={null}
  $env:MEIBEL_API_KEY = "mbl_your_api_key_here"
  ```

  ```bash .env file theme={null}
  MEIBEL_API_KEY=mbl_your_api_key_here
  ```
</CodeGroup>

<Tip>
  Using a `.env` file? The Python SDK loads it automatically with [python-dotenv](https://pypi.org/project/python-dotenv/). For TypeScript, use the [dotenv](https://www.npmjs.com/package/dotenv) package.
</Tip>

## CLI Installation

The Meibel CLI lets you interact with the API from your terminal. Install it with your preferred method.

<CodeGroup>
  ```bash Homebrew (macOS) theme={null}
  brew install meibel-ai/tap/meibel
  ```

  ```powershell Scoop (Windows) theme={null}
  scoop bucket add meibel https://github.com/meibel-ai/scoop-bucket
  scoop install meibel
  ```

  ```bash Binary Download (Linux / manual) theme={null}
  # Download the latest release for your platform from:
  # https://github.com/meibel-ai/meibel-cli/releases
  #
  # Extract and move to your PATH:
  tar -xzf meibel_linux_amd64.tar.gz
  sudo mv meibel /usr/local/bin/
  ```
</CodeGroup>

After installing, configure the CLI with your API key:

```bash theme={null}
meibel config init
```

## Verify Installation

Run a quick API call to confirm everything is working. This example lists your datasources -- an empty list means the connection is healthy.

<CodeGroup>
  ```python Python theme={null}
  import os
  from meibel import MeibelClient

  client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

  datasources = client.datasources.list()
  print(f"Connected — {len(datasources.datasources)} datasource(s) found")
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient } from "meibel";

  const client = new MeibelClient({
    apiKey: process.env.MEIBEL_API_KEY,
  });

  const datasources = await client.datasources.list();
  console.log(`Connected — ${datasources.datasources.length} datasource(s) found`);
  ```

  ```go Go theme={null}
  package main

  import (
      "context"
      "fmt"
      "os"

      meibel "github.com/meibel-ai/meibel-go"
  )

  func main() {
      client := meibel.NewClient(
          meibel.WithAPIKey(os.Getenv("MEIBEL_API_KEY")),
      )

      datasources, err := client.Datasources.ListDatasources(context.Background(), nil)
      if err != nil {
          panic(err)
      }
      fmt.Printf("Connected — %d datasource(s) found\n", len(datasources.Items))
  }
  ```

  ```bash CLI theme={null}
  meibel datasources list
  ```
</CodeGroup>

If you see a count (even zero), your installation and API key are working correctly.

## Next Steps

<CardGroup>
  <Card title="Quick Start" icon="rocket" href="/quickstart">
    Build your first AI-powered workflow end-to-end
  </Card>

  <Card title="API Reference" icon="book" href="/api-reference/introduction">
    Explore the full API reference
  </Card>
</CardGroup>


# Getting Started
Source: https://docs.meibel.ai/quickstart

Parse a document, build a knowledge base, chat with an agent, and extract structured data one document at a time and across a whole datasource

This guide takes you from an empty project to a working example that introduces the core functionality of the Meibel platform. You will parse a document, build a searchable knowledge base, create an agent and chat with it, extract structured data from a document, then run that same extraction across an entire datasource as a batch. By the end you will have a small but complete pipeline: raw PDFs going in, searchable knowledge and structured data coming out.

The example uses material safety data sheets (MSDS), the documents that ship with chemical products to describe their hazards and safe handling. They work well for this because they are real documents built around a standard set of fields, yet they still vary from one manufacturer to the next. That mix of structure and variation is what you meet in most real-world data, and it gives each step something meaningful to work on.

<Note>
  Before you begin, [install an SDK and set your API key](/installation). Each example below assumes `MEIBEL_API_KEY` is set in your environment.
</Note>

## 1. Parse a document

Start by parsing a single document. It is the fastest way to see the platform do real work, and it confirms your SDK and API key are set up correctly before you build anything larger. Parsing turns a PDF, including scanned or image-based pages, into clean structured content you can read or hand to another step. The call runs synchronously, so the result comes straight back.

<CodeGroup>
  ```python Python theme={null}
  import os
  from meibel import MeibelClient

  client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

  with open("document.pdf", "rb") as f:
      parsed = client.documents.process(file=f, file_name="document.pdf")

  print(parsed.result)
  ```

  ```typescript TypeScript theme={null}
  import { MeibelClient } from "meibel";
  import { readFile } from "node:fs/promises";

  const client = new MeibelClient({ apiKey: process.env.MEIBEL_API_KEY });

  const parsed = await client.documents.process(
    new Blob([await readFile("document.pdf")]),
    "document.pdf",
  );

  console.log(parsed.result);
  ```
</CodeGroup>

This single call handles parsing, OCR, and structuring, then returns the extracted content in `result`. Running it synchronously like this suits smaller files or testing a lighter workflow. For larger documents or higher volumes, submit the job asynchronously and poll for the result instead. See the [document processing guide](/guides/documents) for that workflow.

## 2. Create a datasource and upload content

Parsing one document is useful on its own, but most work spans many documents that you want to search across and keep up to date. A **datasource** is a managed knowledge base for exactly that: you add files to it, and the platform parses, analyzes, indexes, and keeps them searchable. The agent you build in the next step will draw on this datasource to ground its answers.

In this step you create a datasource, upload a few MSDS PDFs, and trigger ingestion. Download the sample sheets first, then upload them from your working directory:

* [Acetone safety data sheet (Avantor)](https://media.vwr.com/stibo/search/sds000001650_ca_en.pdf)
* [Ethanol safety data sheet (Decon Laboratories)](https://deconlabs.com/sds/Ethanol_Decon_200%20Proof%20SDS.pdf)
* [Acetic acid safety data sheet (SeaStar Chemicals)](https://seastarchemicals.com/wp-content/uploads/2023/03/06AceticGlacialSDS_Rev202208_SSN_EN.pdf)
* [Citric acid safety data sheet (Chemfax)](https://chemfax.com/wp-content/uploads/2020/12/Citric-Acid-SDS-Version-6-2021.pdf)
* [Potassium chlorate safety data sheet (Fisher Scientific)](https://www.fishersci.com/store/msds?partNumber=P212100\&productDescription=POTASSIUM+CHLORATE+CERT+100GM\&vendorId=VN00033897\&countryCode=US\&language=en)

<CodeGroup>
  ```python Python theme={null}
  # Create a datasource. Omit the connector for a file-upload knowledge base.
  datasource = client.datasources.create(
      name="Safety Data Sheets",
      description="MSDS PDFs for the getting started example",
  )
  ds_id = datasource.id
  print(f"Created datasource: {ds_id}")

  # Upload the sheets into the datasource, one file per call.
  filenames = [
      "sds000001650_ca_en.pdf",
      "Ethanol_Decon_200 Proof SDS.pdf",
      "06AceticGlacialSDS_Rev202208_SSN_EN.pdf",
      "Citric-Acid-SDS-Version-6-2021.pdf",
      "POTASSIUM-CHLORATE-CERT-100GM.pdf",
  ]
  for name in filenames:
      with open(name, "rb") as fh:
          client.datasources.file_uploads.upload_content(
              datasource_id=ds_id, files=fh, files_name=name
          )

  # Trigger ingestion so the content becomes searchable.
  client.datasources.ingest.trigger(datasource_id=ds_id)

  # Wait for ingestion to finish. The datasource must be fully ingested before an
  # agent can query it (step 3) or a batch can run against it (step 5).
  import time

  while True:
      ingest = client.datasources.ingest.get_status(datasource_id=ds_id)
      if ingest.status in ("completed", "failed"):
          break
      time.sleep(5)
  print(f"Ingestion {ingest.status}")
  ```

  ```typescript TypeScript theme={null}
  import { readFile } from "node:fs/promises";

  // Create a datasource. Omit the connector for a file-upload knowledge base.
  const datasource = await client.datasources.create({
    name: "Safety Data Sheets",
    description: "MSDS PDFs for the getting started example",
  });
  const dsId = datasource.id;
  console.log(`Created datasource: ${dsId}`);

  // Upload the sheets into the datasource, one file per call.
  const filenames = [
    "sds000001650_ca_en.pdf",
    "Ethanol_Decon_200 Proof SDS.pdf",
    "06AceticGlacialSDS_Rev202208_SSN_EN.pdf",
    "Citric-Acid-SDS-Version-6-2021.pdf",
    "POTASSIUM-CHLORATE-CERT-100GM.pdf",
  ];
  for (const name of filenames) {
    await client.datasources.fileUploads.uploadContent(
      dsId,
      new Blob([await readFile(name)]),
      name,
    );
  }

  // Trigger ingestion so the content becomes searchable.
  await client.datasources.ingest.trigger(dsId);

  // Wait for ingestion to finish. The datasource must be fully ingested before an
  // agent can query it (step 3) or a batch can run against it (step 5).
  let ingest;
  while (true) {
    ingest = await client.datasources.ingest.getStatus(dsId);
    if (ingest.status === "completed" || ingest.status === "failed") break;
    await new Promise((r) => setTimeout(r, 5000));
  }
  console.log(`Ingestion ${ingest.status}`);
  ```
</CodeGroup>

Ingestion runs asynchronously. Once your files are uploaded, each one is parsed, its content is extracted, and the results are indexed into searchable data elements. The loop above polls until ingestion reaches a terminal state, because the agent you build next can only draw on content that is fully ingested. The [datasources guide](/guides/datasources) covers tracking ingestion status in more detail.

## 3. Create an agent and chat with it

An agent is where your context turns into something you can use. It brings together what it knows, how it reasons, and what it produces: the datasources it can draw on, a system prompt that shapes its responses, and an optional schema for structured output. It can also call tools to improve an answer or take action, such as running a targeted search or querying a database. A sensitive tool can require human approval before it runs. Bound to your MSDS datasource, an agent retrieves the relevant content on its own to ground each answer, point to the source it drew from, and decline to guess when the answer is not there.

Create an agent over the datasource from the previous step, then publish it. Publishing freezes the configuration as a versioned, reproducible release that can hold chat sessions.

<CodeGroup>
  ```python Python theme={null}
  from meibel.models import (
      CreateAgentDefinitionRequest,
      PublishAgentDefinitionRequest,
  )

  agent = client.agents.create(
      body=CreateAgentDefinitionRequest(
          display_name="Safety Assistant",
          description="Answers questions about the uploaded safety data sheets",
          instructions=(
              "You are a safety data sheet assistant. Answer using only the uploaded "
              "sheets. If the answer is not in them, say so."
          ),
          datasources=[ds_id],
      )
  )
  agent_id = agent.id

  client.agents.publish(
      agent_id=agent_id,
      body=PublishAgentDefinitionRequest(commit_message="Initial release"),
  )
  print(f"Published agent: {agent_id}")
  ```

  ```typescript TypeScript theme={null}
  const agent = await client.agents.create({
    displayName: "Safety Assistant",
    description: "Answers questions about the uploaded safety data sheets",
    instructions:
      "You are a safety data sheet assistant. Answer using only the uploaded " +
      "sheets. If the answer is not in them, say so.",
    datasources: [dsId],
  });
  const agentId = agent.id;

  await client.agents.publish(agentId, { commitMessage: "Initial release" });
  console.log(`Published agent: ${agentId}`);
  ```
</CodeGroup>

Now open a session and ask a question. A session keeps its own conversation history, so the agent can follow up on earlier messages. Each response carries both the answer and the sources the agent drew on, so you can check where it came from.

<CodeGroup>
  ```python Python theme={null}
  from meibel.models import ChatMessageRequest

  session = client.agents.sessions.create(agent_id=agent_id)

  reply = client.agents.sessions.send_chat_message(
      session_id=session.session_id,
      body=ChatMessageRequest(
          user_message="What protective equipment does the acetone sheet recommend?",
      ),
  )

  print(reply.assistant_response)
  for source in reply.response.sources or []:
      print(f"  source: {source.title}")
  ```

  ```typescript TypeScript theme={null}
  const session = await client.agents.sessions.create(agentId);

  const reply = await client.agents.sessions.sendChatMessage(session.sessionId, {
    userMessage: "What protective equipment does the acetone sheet recommend?",
  });

  console.log(reply.assistantResponse);
  for (const source of reply.response.sources ?? []) {
    console.log(`  source: ${source.title}`);
  }
  ```
</CodeGroup>

### Stream a response

Streaming sends the response back in pieces as it is generated, instead of making you wait for the whole thing. It helps anywhere a wait would otherwise feel slow: a chat interface that shows the answer as it forms, a long-running task you want to report progress on, or a downstream process that can start on early output before the rest arrives. Send the message to the streaming endpoint and read events as they come in.

<CodeGroup>
  ```python Python theme={null}
  import json

  stream = client.agents.sessions.send_chat_message_stream(
      session_id=session.session_id,
      user_message="Summarize the handling precautions across the sheets.",
  )
  answer = ""
  shown = 0
  for event in stream:
      if not event.data:
          continue
      payload = json.loads(event.data)
      if event.event == "partial_response":
          # partial_response carries the full response so far; print only the new part
          full = payload["data"].get("message") or ""
          if len(full) > shown:
              print(full[shown:], end="", flush=True)
              shown = len(full)
      elif event.event == "completion":
          # the authoritative, complete answer
          answer = payload["data"]["message"]
  print("\n\n" + answer)
  ```

  ```typescript TypeScript theme={null}
  const stream = client.agents.sessions.sendChatMessageStream(
    session.sessionId,
    undefined, // no file attachment
    undefined, // no file name
    { userMessage: "Summarize the handling precautions across the sheets." },
  );

  let answer = "";
  let shown = 0;
  for await (const event of stream) {
    const e = event as { type?: string; data?: { message?: string } };
    if (e.type === "partial_response") {
      // partial_response carries the full response so far; print only the new part
      const full = e.data?.message ?? "";
      if (full.length > shown) {
        process.stdout.write(full.slice(shown));
        shown = full.length;
      }
    } else if (e.type === "completion") {
      // the authoritative, complete answer
      answer = e.data?.message ?? "";
    }
  }
  console.log("\n\n" + answer);
  ```
</CodeGroup>

The stream delivers Server-Sent Events. Each event carries a type and a JSON payload. Types include `connected`, `status`, `tool_call` / `tool_result` (emitted when the agent retrieves from a datasource), `partial_response` (empty while a tool runs), and `completion`. The **complete answer is always in the `completion` event's `data.message`**. Read that for the final text. Each `partial_response` carries the full text generated up to that point, so print the newly-added suffix for a live typing effect. See the [streaming guide](/api-ref-guides/streaming) for the full event reference and semantics. To attach a file to a streaming turn, pass a file and file name as the second and third arguments (both optional).

## 4. Extract structured data from a document

Chat gives you answers in prose. Often you want structured data instead: the same fields, in the same shape, ready to store or compare. For that you define an **artifact schema**, a JSON Schema that names the fields you want, and have the platform extract a document into that shape.

Start with the schema. It lists the chemical-property and safety fields to pull from each sheet. Every field is optional, so the platform returns `null` for anything a given sheet does not contain.

<CodeGroup>
  ```python Python theme={null}
  schema = client.artifact_schemas.create(
      display_name="Chemical Properties",
      type="json",
      description="Chemical properties and safety information from a safety data sheet",
      schema={
          "type": "object",
          "properties": {
              "product_name": {"type": "string"},
              "cas_number": {"type": "string"},
              "molecular_formula": {"type": "string"},
              "physical_state": {"type": "string"},
              "appearance": {"type": "string"},
              "odor": {"type": "string"},
              "melting_point": {"type": "string"},
              "boiling_point": {"type": "string"},
              "flash_point": {"type": "string"},
              "ph": {"type": "string"},
              "specific_gravity": {"type": "string"},
              "solubility": {"type": "string"},
              "hazard_classification": {"type": "string"},
              "signal_word": {"type": "string"},
              "hazard_statements": {"type": "array", "items": {"type": "string"}},
              "first_aid_inhalation": {"type": "string"},
              "first_aid_skin": {"type": "string"},
              "first_aid_eyes": {"type": "string"},
              "storage_conditions": {"type": "string"},
              "manufacturer": {"type": "string"},
          },
      },
  )
  print(f"Created schema: {schema.id}")
  ```

  ```typescript TypeScript theme={null}
  const schema = await client.artifactSchemas.create({
    displayName: "Chemical Properties",
    type: "json",
    description: "Chemical properties and safety information from a safety data sheet",
    schema: {
      type: "object",
      properties: {
        product_name: { type: "string" },
        cas_number: { type: "string" },
        molecular_formula: { type: "string" },
        physical_state: { type: "string" },
        appearance: { type: "string" },
        odor: { type: "string" },
        melting_point: { type: "string" },
        boiling_point: { type: "string" },
        flash_point: { type: "string" },
        ph: { type: "string" },
        specific_gravity: { type: "string" },
        solubility: { type: "string" },
        hazard_classification: { type: "string" },
        signal_word: { type: "string" },
        hazard_statements: { type: "array", items: { type: "string" } },
        first_aid_inhalation: { type: "string" },
        first_aid_skin: { type: "string" },
        first_aid_eyes: { type: "string" },
        storage_conditions: { type: "string" },
        manufacturer: { type: "string" },
      },
    },
  });
  console.log(`Created schema: ${schema.id}`);
  ```
</CodeGroup>

Now extract a single sheet against that schema. This runs synchronously and returns the structured data directly, so you can confirm the fields come back the way you expect before running it at scale.

<CodeGroup>
  ```python Python theme={null}
  extracted = client.documents.transform(
      file="sds000001650_ca_en.pdf",
      # transform() resolves a string schema reference by NAME (or a urn: catalog URN),
      # never by the UUID id. You can also pass the schema dict or a Pydantic model directly.
      schema=schema.name,
  )

  print(extracted.data)
  ```

  ```typescript TypeScript theme={null}
  const extracted = await client.documents.transform({
    file: "sds000001650_ca_en.pdf",
    // transform() resolves a string schema reference by NAME (or a urn: catalog URN),
    // never by the UUID id. You can also pass the schema object or a Zod schema directly.
    schema: schema.name,
  });

  console.log(extracted.data);
  ```
</CodeGroup>

The result comes back as structured data keyed by the fields you defined, ready to store, compare, or hand to another system. The same schema drives the batch run in the next step.

## 5. Run extraction across a datasource in batch

Extracting a document at a time works well for interactive, on-demand extraction, and for checking that your schema behaves. When you need the same structured extraction from every document in a datasource, run it as a **batch**: point an agent at the datasource, and it processes each file and returns one structured result per input document.

The agent's instructions are what drive the extraction, so give it a focused extraction prompt. This one works well for safety data sheets:

<CodeGroup>
  ```python Python theme={null}
  extraction_prompt = """You are an MSDS (Material Safety Data Sheet) data extractor. For the attached document, extract the following chemical properties and safety information.

  Return your extraction as a structured JSON artifact named "chemical_properties".

  Extract these fields (use null for any field not found in the document):

  - product_name: The chemical or product name
  - cas_number: CAS registry number
  - molecular_formula: Chemical formula if listed
  - physical_state: solid, liquid, gas, powder, etc.
  - appearance: Color and physical description
  - odor: Described odor
  - melting_point: Melting point with units
  - boiling_point: Boiling point with units
  - flash_point: Flash point with units
  - ph: pH value or range
  - specific_gravity: Specific gravity / relative density
  - solubility: Water solubility description
  - hazard_classification: GHS or other hazard classification
  - signal_word: Danger or Warning
  - hazard_statements: List of H-statements or hazard descriptions
  - first_aid_inhalation: First aid for inhalation
  - first_aid_skin: First aid for skin contact
  - first_aid_eyes: First aid for eye contact
  - storage_conditions: Recommended storage conditions
  - manufacturer: Manufacturer or supplier name"""
  ```

  ```typescript TypeScript theme={null}
  const extractionPrompt = `You are an MSDS (Material Safety Data Sheet) data extractor. For the attached document, extract the following chemical properties and safety information.

  Return your extraction as a structured JSON artifact named "chemical_properties".

  Extract these fields (use null for any field not found in the document):

  - product_name: The chemical or product name
  - cas_number: CAS registry number
  - molecular_formula: Chemical formula if listed
  - physical_state: solid, liquid, gas, powder, etc.
  - appearance: Color and physical description
  - odor: Described odor
  - melting_point: Melting point with units
  - boiling_point: Boiling point with units
  - flash_point: Flash point with units
  - ph: pH value or range
  - specific_gravity: Specific gravity / relative density
  - solubility: Water solubility description
  - hazard_classification: GHS or other hazard classification
  - signal_word: Danger or Warning
  - hazard_statements: List of H-statements or hazard descriptions
  - first_aid_inhalation: First aid for inhalation
  - first_aid_skin: First aid for skin contact
  - first_aid_eyes: First aid for eye contact
  - storage_conditions: Recommended storage conditions
  - manufacturer: Manufacturer or supplier name`;
  ```
</CodeGroup>

A few things make this a strong extraction prompt:

* **A clear role and task.** "You are an MSDS data extractor" and "extract the following ... from the attached document" keep the agent focused on extraction rather than conversation.
* **An explicit output contract.** It asks for a structured JSON artifact by name, matching the schema you registered.
* **Every field named and described.** A short description per field tells the agent exactly what to pull and resolves ambiguity between similar fields.
* **A rule for missing data.** "Use null for any field not found" keeps the agent faithful to the document instead of guessing.
* **Grounded to the source.** "From the attached document" anchors the extraction to the file rather than the model's prior knowledge.

The chat agent from step 3 is tuned for conversation. Rather than repurpose it, create a dedicated extraction agent: give it the extraction prompt as its instructions and attach the schema it should produce, referenced by name as in step 4. Keeping the two separate leaves your chat assistant untouched and makes each agent's job explicit.

<CodeGroup>
  ```python Python theme={null}
  extractor = client.agents.create(
      body=CreateAgentDefinitionRequest(
          display_name="SDS Extractor",
          description="Extracts chemical properties from safety data sheets",
          instructions=extraction_prompt,
          datasources=[ds_id],
          artifacts=[schema.name],  # attach the schema by name, not its id
      )
  )
  extractor_id = extractor.id

  client.agents.publish(
      agent_id=extractor_id,
      body=PublishAgentDefinitionRequest(commit_message="Initial release"),
  )
  print(f"Published extraction agent: {extractor_id}")
  ```

  ```typescript TypeScript theme={null}
  const extractor = await client.agents.create({
    displayName: "SDS Extractor",
    description: "Extracts chemical properties from safety data sheets",
    instructions: extractionPrompt,
    datasources: [dsId],
    artifacts: [schema.name], // attach the schema by name, not its id
  });
  const extractorId = extractor.id;

  await client.agents.publish(extractorId, { commitMessage: "Initial release" });
  console.log(`Published extraction agent: ${extractorId}`);
  ```
</CodeGroup>

Now define a batch over the datasource, execute it, and poll for results.

<CodeGroup>
  ```python Python theme={null}
  import time
  from meibel.models import CreateBatchDefinitionRequest

  batch = client.batches.create(
      body=CreateBatchDefinitionRequest(
          name="chemical-properties-extraction",
          agent_id=extractor_id,
          input_datasource_id=ds_id,
          user_message="Extract the chemical properties from each sheet.",
      )
  )

  execution = client.batches.execute(definition_id=batch.id)
  print(f"Execution: {execution.execution_id}")

  while True:
      status = client.batches.executions.get_by_id(
          execution_id=execution.execution_id,
      )
      print(f"{status.status}: {status.succeeded or 0} succeeded, {status.failed or 0} failed")
      if status.status in ("COMPLETED", "FAILED"):
          break
      time.sleep(2)

  for item in status.items or []:
      print(item.filename, item.output_artifacts)
  ```

  ```typescript TypeScript theme={null}
  const batch = await client.batches.create({
    name: "chemical-properties-extraction",
    agentId: extractorId,
    inputDatasourceId: dsId,
    userMessage: "Extract the chemical properties from each sheet.",
  });

  const execution = await client.batches.execute(batch.id);
  console.log(`Execution: ${execution.executionId}`);

  let status;
  while (true) {
    status = await client.batches.executions.getById(execution.executionId);
    console.log(`${status.status}: ${status.succeeded ?? 0} succeeded, ${status.failed ?? 0} failed`);
    if (status.status === "COMPLETED" || status.status === "FAILED") break;
    await new Promise((r) => setTimeout(r, 2000));
  }

  for (const item of status.items ?? []) {
    console.log(item.filename, item.outputArtifacts);
  }
  ```
</CodeGroup>

When the batch run completes, the execution reports how many items succeeded and failed, and each item carries the structured data the agent produced for its document. For a long-running batch, you can stream live progress instead of polling. See the [error handling guide](/api-ref-guides/error-handling) for retrying failed items.

A batch definition is reusable. Execute it again whenever the datasource changes, and each run works against its latest ingested state. By default, each run writes its results to a new output datasource that the platform creates for you, so the results live on as data you can query later. To collect results in a specific place, pin an output datasource when you define the batch.

<CodeGroup>
  ```python Python theme={null}
  output_ds = client.datasources.create(name="Chemical Properties Results")

  batch = client.batches.create(
      body=CreateBatchDefinitionRequest(
          name="chemical-properties-extraction",
          agent_id=extractor_id,
          input_datasource_id=ds_id,
          output_datasource_id=output_ds.id,
          user_message="Extract the chemical properties from each sheet.",
      )
  )
  ```

  ```typescript TypeScript theme={null}
  const outputDs = await client.datasources.create({ name: "Chemical Properties Results" });

  const batch = await client.batches.create({
    name: "chemical-properties-extraction",
    agentId: extractorId,
    inputDatasourceId: dsId,
    outputDatasourceId: outputDs.id,
    userMessage: "Extract the chemical properties from each sheet.",
  });
  ```
</CodeGroup>

## What's next

You now have a complete Meibel pipeline: documents parsed, a searchable knowledge base, an agent you can chat with, and structured data pulled from a single document and from a whole datasource at once. Each step here is the simplest version of something you can take much further. Explore the ideas behind them next:

<CardGroup>
  <Card title="Agents" icon="robot" href="/concepts/agents">
    Agent definitions, tools, publishing, and versioning
  </Card>

  <Card title="Confidence Scoring" icon="chart-bar" href="/concepts/confidence-scoring">
    How Meibel evaluates the quality of an agent's work
  </Card>

  <Card title="Streaming" icon="bolt" href="/api-ref-guides/streaming">
    Streaming patterns for chat and processing
  </Card>
</CardGroup>


# CLI
Source: https://docs.meibel.ai/sdk/cli

Install and use the Meibel CLI to parse documents, manage datasources, run agents, and script pipelines from the command line.

The Meibel CLI provides a convenient way to interact with the API from the command line.

## Installation

<Tabs>
  <Tab title="Go Install">
    ```bash theme={null}
    go install github.com/meibel-ai/meibel-go/v2/cmd/meibel@latest
    ```
  </Tab>
</Tabs>

## Configuration

Set your API key:

```bash theme={null}
export MEIBEL_API_KEY="your-api-key"
```

Or use a config file at `~/.meibel/config.yaml`:

```yaml theme={null}
api_key: your-api-key
base_url: https://api.example.com
```

## Commands

### Agents

```bash theme={null}
meibel agents list
meibel agents create
meibel agents get --agent-id "agent_id_value"
meibel agents update --agent-id "agent_id_value"
meibel agents delete --agent-id "agent_id_value"
meibel agents publish --agent-id "agent_id_value"
meibel agents list-versions --agent-id "agent_id_value"
```

### AgentsSessions

```bash theme={null}
meibel agents agents-sessions create-by-name --name "name_value"
meibel agents agents-sessions list --agent-id "agent_id_value"
meibel agents agents-sessions create --agent-id "agent_id_value"
meibel agents agents-sessions send-chat-message --session-id "session_id_value"
meibel agents agents-sessions send-chat-message-stream --session-id "session_id_value"
```

### ArtifactSchemas

```bash theme={null}
meibel artifact-schemas list
meibel artifact-schemas create
meibel artifact-schemas get --artifact-id "artifact_id_value"
meibel artifact-schemas update --artifact-id "artifact_id_value"
meibel artifact-schemas delete --artifact-id "artifact_id_value"
```

### Batches

```bash theme={null}
meibel batches list
meibel batches create
meibel batches get-by-catalog-urn
meibel batches get-by-id --definition-id "definition_id_value"
meibel batches update-by-id --definition-id "definition_id_value"
meibel batches delete-by-id --definition-id "definition_id_value"
meibel batches list-versions --definition-id "definition_id_value"
meibel batches execute --definition-id "definition_id_value"
```

### Executions

```bash theme={null}
meibel batches executions list
meibel batches executions create
meibel batches executions get-by-id --execution-id "execution_id_value"
meibel batches executions update-by-id --execution-id "execution_id_value"
meibel batches executions get-realtime-progress --execution-id "execution_id_value"
meibel batches executions retry-failed-items --execution-id "execution_id_value"
meibel batches executions cancel --execution-id "execution_id_value"
```

### ConfidenceScoring

```bash theme={null}
meibel confidence-scoring get-job --job-id "job_id_value"
meibel confidence-scoring list-jobs
meibel confidence-scoring get-agent-summary --agent-name "agent_name_value"
meibel confidence-scoring get-agent-session-summary --agent-name "agent_name_value" --session-id "session_id_value"
```

### Datasources

```bash theme={null}
meibel datasources list
meibel datasources create
meibel datasources get --datasource-id "datasource_id_value"
meibel datasources update --datasource-id "datasource_id_value"
meibel datasources delete --datasource-id "datasource_id_value"
```

### DataElements

```bash theme={null}
meibel datasources data-elements get --data-element-id "data_element_id_value" --datasource-id "datasource_id_value"
meibel datasources data-elements update --data-element-id "data_element_id_value" --datasource-id "datasource_id_value"
meibel datasources data-elements list --datasource-id "datasource_id_value"
meibel datasources data-elements search --datasource-id "datasource_id_value"
```

### Downloads

```bash theme={null}
meibel datasources downloads create-job --datasource-id "datasource_id_value"
meibel datasources downloads stream-progress --job-id "job_id_value" --datasource-id "datasource_id_value"
meibel datasources downloads file --job-id "job_id_value" --datasource-id "datasource_id_value"
meibel datasources downloads process --datasource-id "datasource_id_value"
```

### FileUploads

```bash theme={null}
meibel datasources file-uploads list-content --datasource-id "datasource_id_value"
meibel datasources file-uploads content --datasource-id "datasource_id_value"
meibel datasources file-uploads stream-progress --upload-id "upload_id_value"
```

### Ingest

```bash theme={null}
meibel datasources ingest trigger --datasource-id "datasource_id_value"
meibel datasources ingest get-status --datasource-id "datasource_id_value"
```

### Tables

```bash theme={null}
meibel datasources tables list --datasource-id "datasource_id_value"
meibel datasources tables update-descriptions --datasource-id "datasource_id_value"
meibel datasources tables list-columns --table-name "table_name_value" --datasource-id "datasource_id_value"
meibel datasources tables update-column-descriptions --table-name "table_name_value" --datasource-id "datasource_id_value"
```

### Documents

```bash theme={null}
meibel documents list-deep-transforms
meibel documents submit-deep-transform-from
meibel documents get-deep-transform-status --job-id "job_id_value"
meibel documents download-deep-transform-artifact --job-id "job_id_value" --name "name_value"
meibel documents parse
meibel documents get-status --job-id "job_id_value"
meibel documents get-result --job-id "job_id_value"
meibel documents get-structured-result --job-id "job_id_value"
meibel documents list-children --job-id "job_id_value"
meibel documents stream-trace --job-id "job_id_value"
meibel documents move
```

### ExecutionPolicies

```bash theme={null}
meibel execution-policies list
meibel execution-policies create
meibel execution-policies get --policy-id "policy_id_value"
meibel execution-policies update --policy-id "policy_id_value"
meibel execution-policies delete --policy-id "policy_id_value"
```

### MetadataModelCatalog

```bash theme={null}
meibel metadata-model-catalog list
meibel metadata-model-catalog get-entry --model-id "model_id_value"
```

### Sessions

```bash theme={null}
meibel sessions get --session-id "session_id_value"
meibel sessions get-messages --session-id "session_id_value"
```

## Output Formats

The CLI supports multiple output formats:

```bash theme={null}
# JSON output (default)
meibel resource list --output json

# Table output
meibel resource list --output table

# YAML output
meibel resource list --output yaml
```


# Go SDK
Source: https://docs.meibel.ai/sdk/go

Install and use the official Meibel Go SDK: idiomatic Go client with typed request and response structs and examples for every endpoint.

The official Go SDK provides an idiomatic Go interface to the Meibel API.

## Installation

```bash theme={null}
go get github.com/meibel-ai/meibel-go/v2
```

## Quick Start

```go theme={null}
package main

import (
    "context"
    "github.com/meibel-ai/meibel-go/v2"
)

func main() {
    // Initialize the client
    client := v2.NewClient(v2.WithAPIKey("your-api-key"))
    ctx := context.Background()

    // Make API calls
    // ...
}
```

## Configuration

```go theme={null}
client := v2.NewClient(
    v2.WithAPIKey("your-api-key"),
    v2.WithBaseURL("https://api.example.com"),
    v2.WithTimeout(30 * time.Second),
)
```

## Resources

* `client.Agents` - Create, manage, and publish AI agents
* `client.Agents.Sessions` - Create, manage, and chat with agent sessions
* `client.ArtifactSchemas` - Define structured output schemas for agent artifacts
* `client.Batches` - Create and manage batch processing definitions
* `client.Batches.Executions` - Run, monitor, and manage batch executions
* `client.ConfidenceScoring` - Track and analyze confidence scores
* `client.Datasources` - Manage data source connections
* `client.Datasources.DataElements` - Manage data elements within datasources
* `client.Datasources.Downloads` - Export datasources as downloadable files
* `client.Datasources.FileUploads` - Upload and manage content files
* `client.Datasources.Ingest` - Trigger and monitor datasource ingestion
* `client.Datasources.Tables` - Manage table and column descriptions for AI context
* `client.Documents` - Parse and transform documents into structured data
* `client.ExecutionPolicies` - Manage execution policies
* `client.MetadataModelCatalog` - Browse available metadata extraction models
* `client.Sessions` - View session details across agents

## Context Support

All operations accept a `context.Context` as the first parameter for cancellation and timeout handling:

```go theme={null}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

result, err := client.Resource.Operation(ctx, params)
```


# TypeScript SDK
Source: https://docs.meibel.ai/sdk/nodejs

Install and use the official Meibel TypeScript SDK: a fully typed client with promises, streaming, and generated types for every endpoint.

The official TypeScript SDK provides a fully typed interface to the Meibel API.

## Installation

```bash theme={null}
npm install meibel
# or
yarn add meibel
# or
pnpm add meibel
```

## Quick Start

```typescript theme={null}
import { MeibelClient } from 'meibel';

// Initialize the client
const client = new MeibelClient({ apiKey: 'your-api-key' });

// Make API calls
// ...
```

## Configuration

```typescript theme={null}
const client = new MeibelClient({
  apiKey: 'your-api-key',
  baseUrl: 'https://api.example.com',  // Custom base URL
  timeout: 30000,  // Request timeout in milliseconds
});
```

## Resources

* `client.agents` - Create, manage, and publish AI agents
* `client.agents.sessions` - Create, manage, and chat with agent sessions
* `client.artifactSchemas` - Define structured output schemas for agent artifacts
* `client.batches` - Create and manage batch processing definitions
* `client.batches.executions` - Run, monitor, and manage batch executions
* `client.confidenceScoring` - Track and analyze confidence scores
* `client.datasources` - Manage data source connections
* `client.datasources.dataElements` - Manage data elements within datasources
* `client.datasources.downloads` - Export datasources as downloadable files
* `client.datasources.fileUploads` - Upload and manage content files
* `client.datasources.ingest` - Trigger and monitor datasource ingestion
* `client.datasources.tables` - Manage table and column descriptions for AI context
* `client.documents` - Parse and transform documents into structured data
* `client.executionPolicies` - Manage execution policies
* `client.metadataModelCatalog` - Browse available metadata extraction models
* `client.sessions` - View session details across agents

## TypeScript Support

This SDK is written in TypeScript and provides:

* Full type definitions for all API types
* Zod schemas for runtime validation
* IntelliSense support in VS Code and other IDEs


# Python SDK
Source: https://docs.meibel.ai/sdk/python

Install and use the official Meibel Python SDK: async client, typed models, streaming responses, and idiomatic examples for every endpoint.

The official Python SDK provides a convenient way to interact with the Meibel API.

## Installation

```bash theme={null}
pip install meibel
```

## Quick Start

```python theme={null}
from meibel import MeibelClient

# Initialize the client
client = MeibelClient(api_key="your-api-key")

# Make API calls
# ...
```

## Async Support

The SDK provides both synchronous and asynchronous clients:

```python theme={null}
from meibel import AsyncMeibelClient
import asyncio

async def main():
    client = AsyncMeibelClient(api_key="your-api-key")
    # Make async API calls
    # ...
    await client.close()

asyncio.run(main())
```

## Configuration

```python theme={null}
client = MeibelClient(
    api_key="your-api-key",
    base_url="https://api.example.com",  # Custom base URL
    timeout=30.0,  # Request timeout in seconds
)
```

## Resources

* `client.agents` - Create, manage, and publish AI agents
* `client.agents.sessions` - Create, manage, and chat with agent sessions
* `client.artifact_schemas` - Define structured output schemas for agent artifacts
* `client.batches` - Create and manage batch processing definitions
* `client.batches.executions` - Run, monitor, and manage batch executions
* `client.confidence_scoring` - Track and analyze confidence scores
* `client.datasources` - Manage data source connections
* `client.datasources.data_elements` - Manage data elements within datasources
* `client.datasources.downloads` - Export datasources as downloadable files
* `client.datasources.file_uploads` - Upload and manage content files
* `client.datasources.ingest` - Trigger and monitor datasource ingestion
* `client.datasources.tables` - Manage table and column descriptions for AI context
* `client.documents` - Parse and transform documents into structured data
* `client.execution_policies` - Manage execution policies
* `client.metadata_model_catalog` - Browse available metadata extraction models
* `client.sessions` - View session details across agents


# Troubleshooting
Source: https://docs.meibel.ai/troubleshooting

Diagnose and fix common Meibel issues: authentication errors, ingestion failures, rate limits, agent errors, and SDK setup problems.

# Troubleshooting

Find solutions to common issues when working with Meibel.

## Authentication Issues

### Invalid API Key Error

**Problem**: Getting "401 Unauthorized" or "Invalid API key" errors.

**Solutions**:

1. Verify your API key is set:
   ```bash theme={null}
   echo $MEIBEL_API_KEY
   ```
2. Check the key hasn't expired in the [Dashboard](https://app.meibel.ai)
3. Ensure you're using the correct header format:
   ```bash theme={null}
   curl -H "Meibel-API-Key: your-key" https://api.meibel.ai/v2/datasources
   ```

### Permission Denied

**Problem**: "403 Forbidden" errors when accessing resources.

**Solutions**:

* Verify your API key has the necessary permissions
* Check if you're accessing resources in the correct project
* Ensure you're not exceeding plan limits

## Connection Issues

### Timeout Errors

**Problem**: Requests timing out before completion.

**Solutions**:

<CodeGroup>
  ```python Python theme={null}
  # Increase timeout
  client = MeibelClient(
      api_key=os.getenv("MEIBEL_API_KEY"),
      timeout=60.0,  # 60 seconds
  )
  ```

  ```typescript TypeScript theme={null}
  const client = new MeibelClient({
    apiKey: process.env.MEIBEL_API_KEY,
    timeout: 60000,  // 60 seconds
  });
  ```
</CodeGroup>

### SSL Certificate Errors

**Problem**: SSL verification failures.

**Solutions**:

1. Update your certificates:
   ```bash theme={null}
   pip install --upgrade certifi
   ```

## Rate Limiting

### 429 Too Many Requests

**Problem**: Hitting rate limits.

**Solutions**:

1. Implement exponential backoff (see [Rate Limits](/api-reference/rate-limits))
2. Batch operations when possible
3. Consider upgrading your plan for higher limits

## SDK-Specific Issues

### Python: Import Errors

```bash theme={null}
pip install meibel
python -c "from meibel import MeibelClient; print('OK')"
```

### TypeScript: Build Errors

```bash theme={null}
npm install meibel
npx tsc --noEmit  # Check for type errors
```

### CLI: Command Not Found

```bash theme={null}
# Reinstall
brew reinstall meibel-ai/tap/meibel

# Or check PATH
which meibel
```

## Getting Help

If you're still experiencing issues:

1. **Documentation**: Review the [API Reference](/api-reference/introduction)
2. **Support**: Contact [support@meibel.ai](mailto:support@meibel.ai) with:
   * Error messages and stack traces
   * Steps to reproduce the issue
   * SDK version and language version


# Ask a Million-Row CSV Precise Questions
Source: https://docs.meibel.ai/tutorials/datasource-csv-tutorial

Load a public dataset of almost two million records as a datasource and ask an agent precise questions about it in plain language

Suppose you have a large table of records, up to millions of rows, and you want to ask questions about it in plain language whose answers depend on counting and comparing across every row.

A language model cannot answer those questions by reading the rows directly. A table that size does not fit in its context window, and semantic search over the data only surfaces a handful of rows that look relevant, without computing anything across all of them.

A Meibel datasource solves this. It ingests the data and gives an agent a way to query it directly, so the agent pulls exactly what a question needs instead of scanning everything. The agent answers by writing a query that the database runs over the whole table: the model turns each question into a query, and the engine does the counting. The answer is exact for any number of rows.

This tutorial builds that setup around the Social Security Administration's (SSA) national baby-name records, a public dataset of about 1.8 million rows. By the end you will have an agent you can ask questions like "how many babies have been named Mary since 1880?" and get back an exact count.

## Prerequisites

* The Meibel Python SDK: `pip install meibel`. See [Installation](/installation).
* Your API key in the `MEIBEL_API_KEY` environment variable.

## 1. Get the dataset

The SSA dataset records, for every year since 1880, how many babies of each sex were given each name in the United States. Each row is one name in one year, with the columns `year`, `name`, `sex`, and `count`. That comes to about 1.8 million rows in roughly 30 MB, well within the 250 MB free-tier upload limit.

The data is public domain, but the SSA blocks scripted downloads from its own site, so Meibel hosts a copy for this tutorial. Download and unpack it:

```bash theme={null}
curl -fsSL https://storage.googleapis.com/meibel-examples/tutorials/ssa-baby-names.csv.gz | gunzip > names.csv
```

## 2. Create a datasource for the CSV

A datasource is the container Meibel stores your data in and queries against. A single one can hold structured tables, documents, or both. Create an empty one now, then load the CSV into it over the next steps.

```python theme={null}
import os
from meibel import MeibelClient

client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

datasource = client.datasources.create(
    name="Baby Names",
    description="SSA national baby-name counts by year",
)

print(datasource.id)
```

## 3. Upload the CSV and wait for it to land

Uploading a file to a datasource is asynchronous: the call returns before the file has finished arriving. Upload the CSV, then wait for the transfer to complete before moving on; the later steps have nothing to work with until the file has fully landed.

```python theme={null}
upload = client.datasources.file_uploads.upload_content(
    datasource_id=datasource.id,
    files=open("names.csv", "rb"),
    files_name="names.csv",
)

# The progress stream ends once the file has fully landed.
for _event in client.datasources.file_uploads.stream_upload_progress(upload.upload_id):
    pass
```

## 4. Ingest into a queryable table

Ingestion registers the CSV as a table, one column per field, and infers each column's type. It also generates a description for every column from the column's own values, which the agent later uses to write accurate queries. Trigger it and poll until it completes (about 20 seconds for this file).

```python theme={null}
import time

client.datasources.ingest.trigger(datasource_id=datasource.id)

while True:
    status = client.datasources.ingest.get_status(datasource_id=datasource.id)
    if "completed" in str(status.status).lower():
        break
    time.sleep(3)
```

Confirm the table and its columns were registered.

```python theme={null}
for table in client.datasources.tables.list(datasource_id=datasource.id, include_columns=True):
    print(table.table_name, [c.column_name for c in table.columns])
# names ['count', 'name', 'sex', 'year']
```

<Note>
  Column descriptions are generated automatically, and you can refine them for clarity. Clear descriptions help the agent map a plain-language question to the right columns. See [Managing datasource metadata](/guides/datasource-metadata) for more information.
</Note>

## 5. Give an agent access to the table

So far the data sits in a datasource, but nothing can yet answer questions about it. That job belongs to an agent, which responds to a question by choosing among the tools available to it and calling them to assemble an answer. An agent can reach a datasource only once that datasource is bound to it in the agent's configuration. Binding the baby-name datasource is what gives this agent a tool for querying the table, and the agent's `instructions` field steers it toward that tool whenever a question calls for a count or an aggregate.

Create the agent and bind the datasource in a single call:

```python theme={null}
from meibel.models import CreateAgentDefinitionRequest

agent = client.agents.create(body=CreateAgentDefinitionRequest(
    display_name="Names Analyst",
    description="Answers quantitative questions about the baby-name data",
    instructions="You answer questions about the baby-name data. For counts, totals, rankings, and other aggregates, query the table so the numbers are exact. State the figure plainly.",
    datasources=[datasource.id],
))

print(agent.id)
```

## 6. Ask for exact numbers

A session is a single instance of a conversation with the agent. Open one and send it questions that only an exact computation can answer. The agent turns each question into a query that the database runs over every row, so the totals it reports are always true totals rather than estimates.

```python theme={null}
from meibel.models import ChatMessageRequest

session = client.agents.sessions.create(agent_id=agent.id)

def ask(question):
    answer = client.agents.sessions.send_chat_message(
        session_id=session.session_id,
        body=ChatMessageRequest(user_message=question),
    )
    print(question)
    print(answer.response.message)
    print()

ask("How many name records are in the dataset?")
ask("What was the most popular girls' name in 1990, and how many babies received it?")
ask("How many babies have been named Mary in total since 1880?")
ask("Which five names were the most common across the 1980s?")
```

Each answer is computed across the full table. The count reflects every row, the total for "Mary" sums every matching year, and the ranking considers every name. Because the database performs the computation, the numbers stay exact whether the table holds a few thousand rows or a few million.

## Why the answers are precise

Two things separate this from asking a model to read the data. First, the model behind the agent only has to translate your question into a query; it never tallies the rows itself, so it cannot approximate or lose count across a long file. Second, the query runs against the complete table the datasource ingested, with no estimation and no retrieval of a subset, so an aggregate reflects the whole dataset. A question about a total or a top-N returns the same answer the database would give a SQL analyst.

This is the structured-data half of a datasource. For questions whose answers live in prose rather than tables, an agent searches documents through retrieval instead of querying a table. A single datasource can hold both structured tables and documents, and it indexes each kind so an agent bound to it can move between them. See [Datasources](/concepts/datasources) for how the two shapes work together.

## What you learned

The steps you just followed work for any tabular data: any CSV you can load as a table becomes something an agent answers exactly. Create a datasource, upload and ingest the file, bind it to an agent, then ask in plain language. Because the database does the counting, the approach holds whether the table has a few thousand rows or many millions. From here, you can refine the column descriptions so the agent maps questions to columns more reliably, or add documents to the same datasource so one agent handles both figures and prose.

<CardGroup>
  <Card title="Datasources" icon="database" href="/concepts/datasources">
    How structured and unstructured data are queried and combined.
  </Card>

  <Card title="Managing Datasource Metadata" icon="tags" href="/guides/datasource-metadata">
    Refine column descriptions so queries stay accurate.
  </Card>

  <Card title="Sessions & Chat" icon="comments" href="/guides/sessions-and-chat">
    Send messages, stream responses, and read session history.
  </Card>
</CardGroup>