Getting Started

This guide takes you from an empty project to a working example that introduces the core functionality of the Meibel platform. You will parse a document, build a searchable knowledge base, create an agent and chat with it, extract structured data from a document, then run that same extraction across an entire datasource as a batch. By the end you will have a small but complete pipeline: raw PDFs going in, searchable knowledge and structured data coming out. The example uses material safety data sheets (MSDS), the documents that ship with chemical products to describe their hazards and safe handling. They work well for this because they are real documents built around a standard set of fields, yet they still vary from one manufacturer to the next. That mix of structure and variation is what you meet in most real-world data, and it gives each step something meaningful to work on.

Before you begin, install an SDK and set your API key. Each example below assumes MEIBEL_API_KEY is set in your environment.

1. Parse a document

Start by parsing a single document. It is the fastest way to see the platform do real work, and it confirms your SDK and API key are set up correctly before you build anything larger. Parsing turns a PDF, including scanned or image-based pages, into clean structured content you can read or hand to another step. The call runs synchronously, so the result comes straight back.

import os
from meibel import MeibelClient

client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

with open("document.pdf", "rb") as f:
    parsed = client.documents.process(file=f, file_name="document.pdf")

print(parsed.result)

import { MeibelClient } from "meibel";
import { readFile } from "node:fs/promises";

const client = new MeibelClient({ apiKey: process.env.MEIBEL_API_KEY });

const parsed = await client.documents.process(
  new Blob([await readFile("document.pdf")]),
  "document.pdf",
);

console.log(parsed.result);

This single call handles parsing, OCR, and structuring, then returns the extracted content in result. Running it synchronously like this suits smaller files or testing a lighter workflow. For larger documents or higher volumes, submit the job asynchronously and poll for the result instead. See the document processing guide for that workflow.

2. Create a datasource and upload content

Parsing one document is useful on its own, but most work spans many documents that you want to search across and keep up to date. A datasource is a managed knowledge base for exactly that: you add files to it, and the platform parses, analyzes, indexes, and keeps them searchable. The agent you build in the next step will draw on this datasource to ground its answers. In this step you create a datasource, upload a few MSDS PDFs, and trigger ingestion. Download the sample sheets first, then upload them from your working directory:

# Create a datasource. Omit the connector for a file-upload knowledge base.
datasource = client.datasources.create(
    name="Safety Data Sheets",
    description="MSDS PDFs for the getting started example",
)
ds_id = datasource.id
print(f"Created datasource: {ds_id}")

# Upload the sheets into the datasource, one file per call.
filenames = [
    "sds000001650_ca_en.pdf",
    "Ethanol_Decon_200 Proof SDS.pdf",
    "06AceticGlacialSDS_Rev202208_SSN_EN.pdf",
    "Citric-Acid-SDS-Version-6-2021.pdf",
    "POTASSIUM-CHLORATE-CERT-100GM.pdf",
]
for name in filenames:
    with open(name, "rb") as fh:
        client.datasources.file_uploads.upload_content(
            datasource_id=ds_id, files=fh, files_name=name
        )

# Trigger ingestion so the content becomes searchable.
client.datasources.ingest.trigger(datasource_id=ds_id)

# Wait for ingestion to finish. The datasource must be fully ingested before an
# agent can query it (step 3) or a batch can run against it (step 5).
import time

while True:
    ingest = client.datasources.ingest.get_status(datasource_id=ds_id)
    if ingest.status in ("completed", "failed"):
        break
    time.sleep(5)
print(f"Ingestion {ingest.status}")

import { readFile } from "node:fs/promises";

// Create a datasource. Omit the connector for a file-upload knowledge base.
const datasource = await client.datasources.create({
  name: "Safety Data Sheets",
  description: "MSDS PDFs for the getting started example",
});
const dsId = datasource.id;
console.log(`Created datasource: ${dsId}`);

// Upload the sheets into the datasource, one file per call.
const filenames = [
  "sds000001650_ca_en.pdf",
  "Ethanol_Decon_200 Proof SDS.pdf",
  "06AceticGlacialSDS_Rev202208_SSN_EN.pdf",
  "Citric-Acid-SDS-Version-6-2021.pdf",
  "POTASSIUM-CHLORATE-CERT-100GM.pdf",
];
for (const name of filenames) {
  await client.datasources.fileUploads.uploadContent(
    dsId,
    new Blob([await readFile(name)]),
    name,
  );
}

// Trigger ingestion so the content becomes searchable.
await client.datasources.ingest.trigger(dsId);

// Wait for ingestion to finish. The datasource must be fully ingested before an
// agent can query it (step 3) or a batch can run against it (step 5).
let ingest;
while (true) {
  ingest = await client.datasources.ingest.getStatus(dsId);
  if (ingest.status === "completed" || ingest.status === "failed") break;
  await new Promise((r) => setTimeout(r, 5000));
}
console.log(`Ingestion ${ingest.status}`);

Ingestion runs asynchronously. Once your files are uploaded, each one is parsed, its content is extracted, and the results are indexed into searchable data elements. The loop above polls until ingestion reaches a terminal state, because the agent you build next can only draw on content that is fully ingested. The datasources guide covers tracking ingestion status in more detail.

3. Create an agent and chat with it

An agent is where your context turns into something you can use. It brings together what it knows, how it reasons, and what it produces: the datasources it can draw on, a system prompt that shapes its responses, and an optional schema for structured output. It can also call tools to improve an answer or take action, such as running a targeted search or querying a database. A sensitive tool can require human approval before it runs. Bound to your MSDS datasource, an agent retrieves the relevant content on its own to ground each answer, point to the source it drew from, and decline to guess when the answer is not there. Create an agent over the datasource from the previous step, then publish it. Publishing freezes the configuration as a versioned, reproducible release that can hold chat sessions.

from meibel.models import (
    CreateAgentDefinitionRequest,
    PublishAgentDefinitionRequest,
)

agent = client.agents.create(
    body=CreateAgentDefinitionRequest(
        display_name="Safety Assistant",
        description="Answers questions about the uploaded safety data sheets",
        instructions=(
            "You are a safety data sheet assistant. Answer using only the uploaded "
            "sheets. If the answer is not in them, say so."
        ),
        datasources=[ds_id],
    )
)
agent_id = agent.id

client.agents.publish(
    agent_id=agent_id,
    body=PublishAgentDefinitionRequest(commit_message="Initial release"),
)
print(f"Published agent: {agent_id}")

const agent = await client.agents.create({
  displayName: "Safety Assistant",
  description: "Answers questions about the uploaded safety data sheets",
  instructions:
    "You are a safety data sheet assistant. Answer using only the uploaded " +
    "sheets. If the answer is not in them, say so.",
  datasources: [dsId],
});
const agentId = agent.id;

await client.agents.publish(agentId, { commitMessage: "Initial release" });
console.log(`Published agent: ${agentId}`);

Now open a session and ask a question. A session keeps its own conversation history, so the agent can follow up on earlier messages. Each response carries both the answer and the sources the agent drew on, so you can check where it came from.

from meibel.models import ChatMessageRequest

session = client.agents.sessions.create(agent_id=agent_id)

reply = client.agents.sessions.send_chat_message(
    session_id=session.session_id,
    body=ChatMessageRequest(
        user_message="What protective equipment does the acetone sheet recommend?",
    ),
)

print(reply.assistant_response)
for source in reply.response.sources or []:
    print(f"  source: {source.title}")

const session = await client.agents.sessions.create(agentId);

const reply = await client.agents.sessions.sendChatMessage(session.sessionId, {
  userMessage: "What protective equipment does the acetone sheet recommend?",
});

console.log(reply.assistantResponse);
for (const source of reply.response.sources ?? []) {
  console.log(`  source: ${source.title}`);
}

Stream a response

Streaming sends the response back in pieces as it is generated, instead of making you wait for the whole thing. It helps anywhere a wait would otherwise feel slow: a chat interface that shows the answer as it forms, a long-running task you want to report progress on, or a downstream process that can start on early output before the rest arrives. Send the message to the streaming endpoint and read events as they come in.

import json

stream = client.agents.sessions.send_chat_message_stream(
    session_id=session.session_id,
    user_message="Summarize the handling precautions across the sheets.",
)
answer = ""
shown = 0
for event in stream:
    if not event.data:
        continue
    payload = json.loads(event.data)
    if event.event == "partial_response":
        # partial_response carries the full response so far; print only the new part
        full = payload["data"].get("message") or ""
        if len(full) > shown:
            print(full[shown:], end="", flush=True)
            shown = len(full)
    elif event.event == "completion":
        # the authoritative, complete answer
        answer = payload["data"]["message"]
print("\n\n" + answer)

const stream = client.agents.sessions.sendChatMessageStream(
  session.sessionId,
  undefined, // no file attachment
  undefined, // no file name
  { userMessage: "Summarize the handling precautions across the sheets." },
);

let answer = "";
let shown = 0;
for await (const event of stream) {
  const e = event as { type?: string; data?: { message?: string } };
  if (e.type === "partial_response") {
    // partial_response carries the full response so far; print only the new part
    const full = e.data?.message ?? "";
    if (full.length > shown) {
      process.stdout.write(full.slice(shown));
      shown = full.length;
    }
  } else if (e.type === "completion") {
    // the authoritative, complete answer
    answer = e.data?.message ?? "";
  }
}
console.log("\n\n" + answer);

The stream delivers Server-Sent Events. Each event carries a type and a JSON payload. Types include connected, status, tool_call / tool_result (emitted when the agent retrieves from a datasource), partial_response (empty while a tool runs), and completion. The complete answer is always in the completion event’s data.message. Read that for the final text. Each partial_response carries the full text generated up to that point, so print the newly-added suffix for a live typing effect. See the streaming guide for the full event reference and semantics. To attach a file to a streaming turn, pass a file and file name as the second and third arguments (both optional).

4. Extract structured data from a document

Chat gives you answers in prose. Often you want structured data instead: the same fields, in the same shape, ready to store or compare. For that you define an artifact schema, a JSON Schema that names the fields you want, and have the platform extract a document into that shape. Start with the schema. It lists the chemical-property and safety fields to pull from each sheet. Every field is optional, so the platform returns null for anything a given sheet does not contain.

schema = client.artifact_schemas.create(
    display_name="Chemical Properties",
    type="json",
    description="Chemical properties and safety information from a safety data sheet",
    schema={
        "type": "object",
        "properties": {
            "product_name": {"type": "string"},
            "cas_number": {"type": "string"},
            "molecular_formula": {"type": "string"},
            "physical_state": {"type": "string"},
            "appearance": {"type": "string"},
            "odor": {"type": "string"},
            "melting_point": {"type": "string"},
            "boiling_point": {"type": "string"},
            "flash_point": {"type": "string"},
            "ph": {"type": "string"},
            "specific_gravity": {"type": "string"},
            "solubility": {"type": "string"},
            "hazard_classification": {"type": "string"},
            "signal_word": {"type": "string"},
            "hazard_statements": {"type": "array", "items": {"type": "string"}},
            "first_aid_inhalation": {"type": "string"},
            "first_aid_skin": {"type": "string"},
            "first_aid_eyes": {"type": "string"},
            "storage_conditions": {"type": "string"},
            "manufacturer": {"type": "string"},
        },
    },
)
print(f"Created schema: {schema.id}")

const schema = await client.artifactSchemas.create({
  displayName: "Chemical Properties",
  type: "json",
  description: "Chemical properties and safety information from a safety data sheet",
  schema: {
    type: "object",
    properties: {
      product_name: { type: "string" },
      cas_number: { type: "string" },
      molecular_formula: { type: "string" },
      physical_state: { type: "string" },
      appearance: { type: "string" },
      odor: { type: "string" },
      melting_point: { type: "string" },
      boiling_point: { type: "string" },
      flash_point: { type: "string" },
      ph: { type: "string" },
      specific_gravity: { type: "string" },
      solubility: { type: "string" },
      hazard_classification: { type: "string" },
      signal_word: { type: "string" },
      hazard_statements: { type: "array", items: { type: "string" } },
      first_aid_inhalation: { type: "string" },
      first_aid_skin: { type: "string" },
      first_aid_eyes: { type: "string" },
      storage_conditions: { type: "string" },
      manufacturer: { type: "string" },
    },
  },
});
console.log(`Created schema: ${schema.id}`);

Now extract a single sheet against that schema. This runs synchronously and returns the structured data directly, so you can confirm the fields come back the way you expect before running it at scale.

extracted = client.documents.transform(
    file="sds000001650_ca_en.pdf",
    # transform() resolves a string schema reference by NAME (or a urn: catalog URN),
    # never by the UUID id. You can also pass the schema dict or a Pydantic model directly.
    schema=schema.name,
)

print(extracted.data)

const extracted = await client.documents.transform({
  file: "sds000001650_ca_en.pdf",
  // transform() resolves a string schema reference by NAME (or a urn: catalog URN),
  // never by the UUID id. You can also pass the schema object or a Zod schema directly.
  schema: schema.name,
});

console.log(extracted.data);

The result comes back as structured data keyed by the fields you defined, ready to store, compare, or hand to another system. The same schema drives the batch run in the next step.

5. Run extraction across a datasource in batch

Extracting a document at a time works well for interactive, on-demand extraction, and for checking that your schema behaves. When you need the same structured extraction from every document in a datasource, run it as a batch: point an agent at the datasource, and it processes each file and returns one structured result per input document. The agent’s instructions are what drive the extraction, so give it a focused extraction prompt. This one works well for safety data sheets:

extraction_prompt = """You are an MSDS (Material Safety Data Sheet) data extractor. For the attached document, extract the following chemical properties and safety information.

Return your extraction as a structured JSON artifact named "chemical_properties".

Extract these fields (use null for any field not found in the document):

- product_name: The chemical or product name
- cas_number: CAS registry number
- molecular_formula: Chemical formula if listed
- physical_state: solid, liquid, gas, powder, etc.
- appearance: Color and physical description
- odor: Described odor
- melting_point: Melting point with units
- boiling_point: Boiling point with units
- flash_point: Flash point with units
- ph: pH value or range
- specific_gravity: Specific gravity / relative density
- solubility: Water solubility description
- hazard_classification: GHS or other hazard classification
- signal_word: Danger or Warning
- hazard_statements: List of H-statements or hazard descriptions
- first_aid_inhalation: First aid for inhalation
- first_aid_skin: First aid for skin contact
- first_aid_eyes: First aid for eye contact
- storage_conditions: Recommended storage conditions
- manufacturer: Manufacturer or supplier name"""

const extractionPrompt = `You are an MSDS (Material Safety Data Sheet) data extractor. For the attached document, extract the following chemical properties and safety information.

Return your extraction as a structured JSON artifact named "chemical_properties".

Extract these fields (use null for any field not found in the document):

- product_name: The chemical or product name
- cas_number: CAS registry number
- molecular_formula: Chemical formula if listed
- physical_state: solid, liquid, gas, powder, etc.
- appearance: Color and physical description
- odor: Described odor
- melting_point: Melting point with units
- boiling_point: Boiling point with units
- flash_point: Flash point with units
- ph: pH value or range
- specific_gravity: Specific gravity / relative density
- solubility: Water solubility description
- hazard_classification: GHS or other hazard classification
- signal_word: Danger or Warning
- hazard_statements: List of H-statements or hazard descriptions
- first_aid_inhalation: First aid for inhalation
- first_aid_skin: First aid for skin contact
- first_aid_eyes: First aid for eye contact
- storage_conditions: Recommended storage conditions
- manufacturer: Manufacturer or supplier name`;

A few things make this a strong extraction prompt:

A clear role and task. “You are an MSDS data extractor” and “extract the following … from the attached document” keep the agent focused on extraction rather than conversation.
An explicit output contract. It asks for a structured JSON artifact by name, matching the schema you registered.
Every field named and described. A short description per field tells the agent exactly what to pull and resolves ambiguity between similar fields.
A rule for missing data. “Use null for any field not found” keeps the agent faithful to the document instead of guessing.
Grounded to the source. “From the attached document” anchors the extraction to the file rather than the model’s prior knowledge.

The chat agent from step 3 is tuned for conversation. Rather than repurpose it, create a dedicated extraction agent: give it the extraction prompt as its instructions and attach the schema it should produce, referenced by name as in step 4. Keeping the two separate leaves your chat assistant untouched and makes each agent’s job explicit.

extractor = client.agents.create(
    body=CreateAgentDefinitionRequest(
        display_name="SDS Extractor",
        description="Extracts chemical properties from safety data sheets",
        instructions=extraction_prompt,
        datasources=[ds_id],
        artifacts=[schema.name],  # attach the schema by name, not its id
    )
)
extractor_id = extractor.id

client.agents.publish(
    agent_id=extractor_id,
    body=PublishAgentDefinitionRequest(commit_message="Initial release"),
)
print(f"Published extraction agent: {extractor_id}")

const extractor = await client.agents.create({
  displayName: "SDS Extractor",
  description: "Extracts chemical properties from safety data sheets",
  instructions: extractionPrompt,
  datasources: [dsId],
  artifacts: [schema.name], // attach the schema by name, not its id
});
const extractorId = extractor.id;

await client.agents.publish(extractorId, { commitMessage: "Initial release" });
console.log(`Published extraction agent: ${extractorId}`);

Now define a batch over the datasource, execute it, and poll for results.

import time
from meibel.models import CreateBatchDefinitionRequest

batch = client.batches.create(
    body=CreateBatchDefinitionRequest(
        name="chemical-properties-extraction",
        agent_id=extractor_id,
        input_datasource_id=ds_id,
        user_message="Extract the chemical properties from each sheet.",
    )
)

execution = client.batches.execute(definition_id=batch.id)
print(f"Execution: {execution.execution_id}")

while True:
    status = client.batches.executions.get_by_id(
        execution_id=execution.execution_id,
    )
    print(f"{status.status}: {status.succeeded or 0} succeeded, {status.failed or 0} failed")
    if status.status in ("COMPLETED", "FAILED"):
        break
    time.sleep(2)

for item in status.items or []:
    print(item.filename, item.output_artifacts)

const batch = await client.batches.create({
  name: "chemical-properties-extraction",
  agentId: extractorId,
  inputDatasourceId: dsId,
  userMessage: "Extract the chemical properties from each sheet.",
});

const execution = await client.batches.execute(batch.id);
console.log(`Execution: ${execution.executionId}`);

let status;
while (true) {
  status = await client.batches.executions.getById(execution.executionId);
  console.log(`${status.status}: ${status.succeeded ?? 0} succeeded, ${status.failed ?? 0} failed`);
  if (status.status === "COMPLETED" || status.status === "FAILED") break;
  await new Promise((r) => setTimeout(r, 2000));
}

for (const item of status.items ?? []) {
  console.log(item.filename, item.outputArtifacts);
}

When the batch run completes, the execution reports how many items succeeded and failed, and each item carries the structured data the agent produced for its document. For a long-running batch, you can stream live progress instead of polling. See the error handling guide for retrying failed items. A batch definition is reusable. Execute it again whenever the datasource changes, and each run works against its latest ingested state. By default, each run writes its results to a new output datasource that the platform creates for you, so the results live on as data you can query later. To collect results in a specific place, pin an output datasource when you define the batch.

output_ds = client.datasources.create(name="Chemical Properties Results")

batch = client.batches.create(
    body=CreateBatchDefinitionRequest(
        name="chemical-properties-extraction",
        agent_id=extractor_id,
        input_datasource_id=ds_id,
        output_datasource_id=output_ds.id,
        user_message="Extract the chemical properties from each sheet.",
    )
)

const outputDs = await client.datasources.create({ name: "Chemical Properties Results" });

const batch = await client.batches.create({
  name: "chemical-properties-extraction",
  agentId: extractorId,
  inputDatasourceId: dsId,
  outputDatasourceId: outputDs.id,
  userMessage: "Extract the chemical properties from each sheet.",
});

What’s next

You now have a complete Meibel pipeline: documents parsed, a searchable knowledge base, an agent you can chat with, and structured data pulled from a single document and from a whole datasource at once. Each step here is the simplest version of something you can take much further. Explore the ideas behind them next:

Agents

Agent definitions, tools, publishing, and versioning

Confidence Scoring

How Meibel evaluates the quality of an agent’s work

Streaming

Streaming patterns for chat and processing

Get started

Tutorials

Core concepts

Agents and sessions

Data and documents

Operations

Support

Getting Started

1. Parse a document

2. Create a datasource and upload content

3. Create an agent and chat with it

Stream a response

4. Extract structured data from a document

5. Run extraction across a datasource in batch

What’s next

Agents

Confidence Scoring

Streaming

​1. Parse a document

​2. Create a datasource and upload content

​3. Create an agent and chat with it

​Stream a response

​4. Extract structured data from a document

​5. Run extraction across a datasource in batch

​What’s next

Agents

Confidence Scoring

Streaming

1. Parse a document

2. Create a datasource and upload content

3. Create an agent and chat with it

Stream a response

4. Extract structured data from a document

5. Run extraction across a datasource in batch

What’s next