Checks

client.checks runs the checks the dataset page runs and reads what they found. A check never keeps, rejects or deletes an episode: its findings are issues (confident failures) and notes (uncertain readings).

The default check is the VLM check (vlm): a model reads each episode against the Task it is linked to. It costs credits; estimate shows the price and starts nothing. The catalog’s checks (task.graph, task.criteria, …) run as one selection with check_ids, categories or all.

Checks speak qualia.v1, in the API key’s own account. Every method takes a dataset id or its full name (accounts/{account}/datasets/{dataset}).

The VLM check

from qualia import Qualia

client = Qualia()

estimate = client.checks.estimate(dataset_id, max_episodes=50)
print(estimate.credits, estimate.episode_count, estimate.models)

started = client.checks.run(dataset_id, max_episodes=50, max_cost=estimate.credits)
run = client.checks.wait(started.check_runs[-1])   # SUCCEEDED or FAILED
for issue in client.checks.findings(dataset_id).items:
    print(issue.episode, issue.finding.verdict, issue.text)

check="reach" runs the reach solver instead; check="-" runs the dataset’s default set (episode-rules on an upload, mixture on a mixture).

Is the VLM check set up on this deployment?

client.checks.get("vlm").runnable

Checks from the catalog

for check in client.checks.list(dataset_id).items:
    print(check.check_id, check.kind, check.runnable, check.unavailable_reason)

bound = client.checks.estimate(dataset_id, categories=["task"])
started = client.checks.run(dataset_id, categories=["task"], max_cost=200)
run = client.checks.wait(started.check_run)

page = client.checks.list_findings(dataset_id, verdict="FAIL")
for finding in page.items:
    print(finding.episode, finding.title, finding.severity)

A catalog run needs max_cost covering the bound estimate gives for the same selection. run sends a request_id, so a retry with the same id returns the first run.

list_findings(dataset, check="-", *, check_run=, episode=, verdict=) reads one check’s findings ("task.graph") or every check’s.

Reference ranges

The episode rules judge each episode against its robot’s reference ranges: one per robot interface (or robot) and metric, with the fences an episode is judged against.

page = client.checks.reference_ranges(interface_hash="b3:…", metric="duration_seconds")
r = page.items[0]
print(r.metric, r.fence_low, r.fence_high, r.sample_count)

reference_range(name) reads one.

Reach

status = client.checks.reach_status(dataset_id)
print(status.target_robot, status.current_key, status.stale)
if status.last_run:
    print(status.last_run.state, status.last_run.done, status.last_run.total)

client.checks.run(dataset_id, "reach")        # the solver; queued, not waited for
client.checks.cancel_reach(dataset_id)        # stop it; solved takes keep results

for result in client.checks.results(dataset_id, state="SUCCEEDED").items:
    print(result.episode, result.reach.longest_seconds, len(result.reach.stretches))

results(dataset, check="reach", *, episode=, state=, robot=, filter=) lists each take on the dataset’s target robot at its current reach key, NOT_RUN where nothing is solved under it yet. Pass None for every stored result in your workspace.

Methods

Method	Call	Returns
get(check="vlm")	GetCheck	Check
estimate(dataset, *, check_ids=, categories=, all=, episodes=, filter=, model=, max_episodes=, max_cost=)	RunCheck or RunDatasetChecks, validate_only	CheckEstimate, or RunDatasetChecksResponse for a catalog selection
run(dataset, check="vlm", *, …same…, request_id=)	RunCheck or RunDatasetChecks	RunCheckResponse or RunDatasetChecksResponse
get_run(run), wait(run, *, timeout=, interval=)	GetCheckRun	CheckRun
list_runs(check="-", *, dataset=, state=, filter=)	ListCheckRuns	page of CheckRun
list(dataset)	ListDatasetChecks	page of DatasetCheck
list_findings(dataset, check="-", *, check_run=, episode=, verdict=)	ListCheckFindings	page of CheckFinding
findings(dataset, check="vlm", *, filter=)	Issues.ListIssues	page of Issue
reference_ranges(check="episode-rules", *, interface_hash=, robot=, metric=, filter=)	ListReferenceRanges	page of ReferenceRange
reference_range(name)	GetReferenceRange	ReferenceRange
reach_status(dataset)	GetReachStatus	ReachStatus
cancel_reach(dataset)	CancelReachStatus	ReachStatus
results(dataset, check="reach", *, episode=, state=, robot=, filter=)	ListCheckResults	page of CheckResult

Pages take limit= and cursor= and return items with page_info.end_cursor. A key restricted to workspaces reaches none of these; the CLI is qualia check.