Ask a Million-Row CSV Precise Questions

Suppose you have a large table of records, up to millions of rows, and you want to ask questions about it in plain language whose answers depend on counting and comparing across every row. A language model cannot answer those questions by reading the rows directly. A table that size does not fit in its context window, and semantic search over the data only surfaces a handful of rows that look relevant, without computing anything across all of them. A Meibel datasource solves this. It ingests the data and gives an agent a way to query it directly, so the agent pulls exactly what a question needs instead of scanning everything. The agent answers by writing a query that the database runs over the whole table: the model turns each question into a query, and the engine does the counting. The answer is exact for any number of rows. This tutorial builds that setup around the Social Security Administration’s (SSA) national baby-name records, a public dataset of about 1.8 million rows. By the end you will have an agent you can ask questions like “how many babies have been named Mary since 1880?” and get back an exact count.

Prerequisites

The Meibel Python SDK: pip install meibel. See Installation.
Your API key in the MEIBEL_API_KEY environment variable.

1. Get the dataset

The SSA dataset records, for every year since 1880, how many babies of each sex were given each name in the United States. Each row is one name in one year, with the columns year, name, sex, and count. That comes to about 1.8 million rows in roughly 30 MB, well within the 250 MB free-tier upload limit. The data is public domain, but the SSA blocks scripted downloads from its own site, so Meibel hosts a copy for this tutorial. Download and unpack it:

curl -fsSL https://storage.googleapis.com/meibel-examples/tutorials/ssa-baby-names.csv.gz | gunzip > names.csv

2. Create a datasource for the CSV

A datasource is the container Meibel stores your data in and queries against. A single one can hold structured tables, documents, or both. Create an empty one now, then load the CSV into it over the next steps.

import os
from meibel import MeibelClient

client = MeibelClient(api_key=os.environ["MEIBEL_API_KEY"])

datasource = client.datasources.create(
    name="Baby Names",
    description="SSA national baby-name counts by year",
)

print(datasource.id)

3. Upload the CSV and wait for it to land

Uploading a file to a datasource is asynchronous: the call returns before the file has finished arriving. Upload the CSV, then wait for the transfer to complete before moving on; the later steps have nothing to work with until the file has fully landed.

upload = client.datasources.file_uploads.upload_content(
    datasource_id=datasource.id,
    files=open("names.csv", "rb"),
    files_name="names.csv",
)

# The progress stream ends once the file has fully landed.
for _event in client.datasources.file_uploads.stream_upload_progress(upload.upload_id):
    pass

4. Ingest into a queryable table

Ingestion registers the CSV as a table, one column per field, and infers each column’s type. It also generates a description for every column from the column’s own values, which the agent later uses to write accurate queries. Trigger it and poll until it completes (about 20 seconds for this file).

import time

client.datasources.ingest.trigger(datasource_id=datasource.id)

while True:
    status = client.datasources.ingest.get_status(datasource_id=datasource.id)
    if "completed" in str(status.status).lower():
        break
    time.sleep(3)

Confirm the table and its columns were registered.

for table in client.datasources.tables.list(datasource_id=datasource.id, include_columns=True):
    print(table.table_name, [c.column_name for c in table.columns])
# names ['count', 'name', 'sex', 'year']

Column descriptions are generated automatically, and you can refine them for clarity. Clear descriptions help the agent map a plain-language question to the right columns. See Managing datasource metadata for more information.

5. Give an agent access to the table

So far the data sits in a datasource, but nothing can yet answer questions about it. That job belongs to an agent, which responds to a question by choosing among the tools available to it and calling them to assemble an answer. An agent can reach a datasource only once that datasource is bound to it in the agent’s configuration. Binding the baby-name datasource is what gives this agent a tool for querying the table, and the agent’s instructions field steers it toward that tool whenever a question calls for a count or an aggregate. Create the agent and bind the datasource in a single call:

from meibel.models import CreateAgentDefinitionRequest

agent = client.agents.create(body=CreateAgentDefinitionRequest(
    display_name="Names Analyst",
    description="Answers quantitative questions about the baby-name data",
    instructions="You answer questions about the baby-name data. For counts, totals, rankings, and other aggregates, query the table so the numbers are exact. State the figure plainly.",
    datasources=[datasource.id],
))

print(agent.id)

6. Ask for exact numbers

A session is a single instance of a conversation with the agent. Open one and send it questions that only an exact computation can answer. The agent turns each question into a query that the database runs over every row, so the totals it reports are always true totals rather than estimates.

from meibel.models import ChatMessageRequest

session = client.agents.sessions.create(agent_id=agent.id)

def ask(question):
    answer = client.agents.sessions.send_chat_message(
        session_id=session.session_id,
        body=ChatMessageRequest(user_message=question),
    )
    print(question)
    print(answer.response.message)
    print()

ask("How many name records are in the dataset?")
ask("What was the most popular girls' name in 1990, and how many babies received it?")
ask("How many babies have been named Mary in total since 1880?")
ask("Which five names were the most common across the 1980s?")

Each answer is computed across the full table. The count reflects every row, the total for “Mary” sums every matching year, and the ranking considers every name. Because the database performs the computation, the numbers stay exact whether the table holds a few thousand rows or a few million.

Why the answers are precise

Two things separate this from asking a model to read the data. First, the model behind the agent only has to translate your question into a query; it never tallies the rows itself, so it cannot approximate or lose count across a long file. Second, the query runs against the complete table the datasource ingested, with no estimation and no retrieval of a subset, so an aggregate reflects the whole dataset. A question about a total or a top-N returns the same answer the database would give a SQL analyst. This is the structured-data half of a datasource. For questions whose answers live in prose rather than tables, an agent searches documents through retrieval instead of querying a table. A single datasource can hold both structured tables and documents, and it indexes each kind so an agent bound to it can move between them. See Datasources for how the two shapes work together.

What you learned

The steps you just followed work for any tabular data: any CSV you can load as a table becomes something an agent answers exactly. Create a datasource, upload and ingest the file, bind it to an agent, then ask in plain language. Because the database does the counting, the approach holds whether the table has a few thousand rows or many millions. From here, you can refine the column descriptions so the agent maps questions to columns more reliably, or add documents to the same datasource so one agent handles both figures and prose.

Datasources

How structured and unstructured data are queried and combined.

Managing Datasource Metadata

Refine column descriptions so queries stay accurate.

Sessions & Chat

Send messages, stream responses, and read session history.

​Prerequisites

​1. Get the dataset

​2. Create a datasource for the CSV

​3. Upload the CSV and wait for it to land

​4. Ingest into a queryable table

​5. Give an agent access to the table

​6. Ask for exact numbers

​Why the answers are precise

​What you learned

Datasources

Managing Datasource Metadata

Sessions & Chat

Prerequisites

1. Get the dataset

2. Create a datasource for the CSV

3. Upload the CSV and wait for it to land

4. Ingest into a queryable table

5. Give an agent access to the table

6. Ask for exact numbers

Why the answers are precise

What you learned