Datasets

client.datasets reads and edits datasets the way the dataset page does. A dataset is an upload (one recording session), a merged dataset (several uploads reviewed and trained as one) or a mixture (kept up to date from other datasets). Every method takes a dataset id or its full name, accounts/{a}/datasets/{d}.

These calls go through the qualia.v1 Datasets API, so any API key that can read the dataset’s workspace can make them; a key restricted to workspaces is refused. Writes need the data:write scope and the same role the dashboard asks for.

Read datasets

from qualia import Qualia

client = Qualia()

page = client.datasets.list(filter="kind = MIXTURE", limit=50)
dataset = client.datasets.get(page.items[0].name)
print(dataset.kind, dataset.counts.episodes, dataset.etag)

list filters on kind (UPLOAD, MERGED, MIXTURE), state, labels.*, target_robot, source_robot, slug, display_name (: for a substring), reads, readers and merged_dataset. It leaves out uploads merged into another dataset, unless the filter names one slug, one readers or one merged_dataset (which lists the uploads that merged dataset holds). get also says which Tasks reference the dataset’s episodes (task_references).

Edit a dataset

Only the arguments you pass change. Pass the etag you read to refuse a stale write (409).

client.datasets.update(dataset.name, display_name="Bench, day 2", etag=dataset.etag)
client.datasets.update(dataset.name, target_robot="accounts/<a>/robots/<r>")
client.datasets.update(
    dataset.name,
    reach_settings={"solver": "DLS", "base_m": [0.0, 0.1, 0.0], "home_heading_rad": 0.0},
)

Argument	What it sets
display_name	The name
description	Free text; "" clears it
target_robot	The robot it plays on, accounts/{a}/robots/{r}; "" clears it
reach_settings	How the reach check walks it, replaced as a whole: solver (PYROKI, DLS), base_m (empty or x, y, z metres), home_heading_rad. {} resets

A merged dataset writes its robots and reach settings to every upload it holds. Saving reach settings starts nothing; the next reach check run uses them.

Merge uploads

A merge is final. Preview it first: validate_only=True writes nothing and returns what differs between the uploads.

uploads = [first.name, second.name]
conflicts = client.datasets.merge(uploads, display_name="Bench", validate_only=True)
for c in conflicts:
    print(c.field, c.channel, c.blocking, [side.value for side in c.sides])

merged = client.datasets.merge(
    uploads,
    display_name="Bench",
    resolutions=[{"field": "target_robot", "channel": "", "value": first.target_robot}],
)

A blocking conflict (what training freezes: features, cameras, channel names, fps, where the poses are) settles only by leaving uploads out. The others take one value each from resolutions; labels and the Task default to the uploads’ commonest. into= adds uploads to an existing merged dataset.

Curate from a version

A branch is a mixture that reads one upload as a version froze it. It is named after the version unless you name it.

branch = client.datasets.create(curated_from=f"{dataset.name}/versions/1.0")

Delete a dataset

client.datasets.delete(dataset.name, force=True)

This deletes a mixture, or retires an upload or a merged dataset (with the uploads it holds). Nothing is erased, and versions cut from it keep resolving. It is refused while a mixture reads it, and while Tasks reference its episodes unless force=True, which removes those references.

Map channels onto canonical keys

result = client.datasets.remap_channels(
    dataset.name, {"/arm/joint_0": "left.joint_0.pos", "/arm/joint_1": ""}
)
print(result.interface_hash)

Renames add to earlier ones; "" resets a channel to the key read from its name. A key another channel holds is refused and nothing is written.

Re-ingest an MCAP upload

toml = open("ingest.toml").read()
reingest = client.datasets.reingest(dataset.name, toml)
done = client.datasets.wait_reingest(reingest.name, timeout=1800)
print(done.episode_count, done.frame_count)

The new meta/ingest.toml replaces the published one and the upload’s next generation is published. The published config again runs nothing (unchanged). A config with [episodes] spans re-cuts the upload and is refused while any episode holds a person’s ruling, issue, note or edit. wait_reingest raises QualiaError with the reason when it fails, and TimeoutError when it has not ended; the work keeps running either way. get_reingest reads one at any time.

Resolve a handheld upload’s poses

started = client.datasets.resolve_poses(dataset.name, gripper="<gripper id>")
client.datasets.resolve_poses(dataset.name, "root")  # undo: back at the gripper root

The gripper’s tracker-to-TCP calibration is read and sent as it stands; a gripper with none is refused before anything is sent. Without gripper= it is the upload’s capture gripper. The work re-derives the whole upload, and the call waits for it; wait=False returns once it is accepted, and started.operation names the re-ingest it started (wait_resolve(started.operation), or wait_reingest, which waits for the same thing). no_op means the poses were already there.

Read the ingest report

report = client.datasets.ingest_report(dataset.name)
if report.omission != "OMISSION_UNSPECIFIED":
    raise RuntimeError(f"report unreadable: {report.omission}")
for source in report.report["sources"]:
    print(source["path"], [c["camera"] for c in source.get("cameras", [])])

report.report is the document ingest wrote. A merged dataset answers its uploads’ reports folded into one: the sources of its base upload, findings rolled up across uploads, sync_agreement and the uploads read so far.