Checks_RunDatasetChecks
const url = 'https://dev-api.qualiastudios.dev/v1/accounts/example/datasets/example/checks:run';const options = { method: 'POST', headers: {'X-API-Key': '<X-API-Key>', 'Content-Type': 'application/json'}, body: '{"dataset":"example","checkIds":["example"],"categories":["example"],"all":true,"selection":{"names":{"names":["example"]},"filter":"example"},"maxEpisodes":1,"maxCost":"example","validateOnly":true,"model":"example","requestId":"example","kind":"KIND_UNSPECIFIED"}'};
try { const response = await fetch(url, options); const data = await response.json(); console.log(data);} catch (error) { console.error(error);}curl --request POST \ --url https://dev-api.qualiastudios.dev/v1/accounts/example/datasets/example/checks:run \ --header 'Content-Type: application/json' \ --header 'X-API-Key: <X-API-Key>' \ --data '{ "dataset": "example", "checkIds": [ "example" ], "categories": [ "example" ], "all": true, "selection": { "names": { "names": [ "example" ] }, "filter": "example" }, "maxEpisodes": 1, "maxCost": "example", "validateOnly": true, "model": "example", "requestId": "example", "kind": "KIND_UNSPECIFIED" }'Runs catalog checks over a dataset as one run (QUA-3524): the union of
check_ids, categories and all, deduplicated, on the episodes of
selection, sampled to max_episodes. The selection is resolved and
pinned before the estimate (check versions, Task revisions, episodes),
and the run reads exactly what was pinned. One model call per episode
reads every selected VLM check. An episode a check cannot read (no
published Task for a Task check) is skipped for that check, never
passed. all and a category take the checks that can run and list the
rest in skipped_checks. INVALID_ARGUMENT, before anything starts, for
an empty selection, an unknown check id or category, a named check that
cannot run on this dataset, a selection none of whose checks can run, or
when no episode is eligible for any selected check;
FAILED_PRECONDITION on a deployment with no checks runner, and, for a
person’s run (estimate included), while a selected check is still running
on this dataset: the message names that run, which
DatasetCheck.running_check_run also lists. A retry with the first call’s
request_id is answered with its run, not refused. With
validate_only nothing starts and nothing is reserved: the response
carries the resolved selection and pipeline-api’s bound for it, the
same bound a run reserves. A run needs max_cost at or above that
bound. Each run is a CheckRun under checks/vlm.
Authorizations
Section titled “Authorizations ”Parameters
Section titled “ Parameters ”Path Parameters
Section titled “Path Parameters ”The account id.
The dataset id.
Request Body required
Section titled “Request Body required ”Request for Checks.RunDatasetChecks.
object
Accounts/{account}/datasets/{dataset}
Check ids to run, e.g. task.graph.
Categories to run every runnable check of the selected kind.
Run every runnable check of the selected kind.
Only these episodes (names under dataset, or a ListEpisodes filter);
unset means all.
object
Episode names under the request’s parent.
object
Accounts/{account}/datasets/{dataset}/episodes/{episode}, at most 10000.
A ListEpisodes filter over the parent (e.g. what the filter bar leaves).
Sample at most this many eligible episodes, evenly spread and the same for the same request. Must be positive; unset reads all of them.
The most credits the run may reserve and spend. Required to start a
VLM run, and must cover the bound; optional with validate_only.
Heuristic runs have a bound of zero and do not require max_cost.
Resolve the selection and answer the bound, starting and reserving nothing.
The model that reads the VLM checks, one of CheckEstimate.models. Empty: the runner’s default.
Idempotency key (AIP-155): a retry with the same id within 24 hours returns the first call’s run and its pinned selection instead of starting a second one. Required to start a run.
Kind selected by all and categories. Unset: the kind check_ids names;
else VLM when a VLM check they reach can run here, else heuristic. A run
reads one kind; the checks it leaves out are listed in skipped_checks.
Explicit check_ids may name either kind, but cannot mix kinds in one run.
Responses
Section titled “ Responses ”OK
Response for Checks.RunDatasetChecks.
object
The resolved checks, by category then check id.
One selected check, as the run pins it.
object
Accounts/{account}/datasets/{dataset}/checks/{check}
The check version pinned.
How it reads an episode.
Selected episodes it will read.
Selected episodes it skips, with skip_reason.
Why it skips them (e.g. no published Task); empty when it skips none.
Episodes selected: after the selection and sample cap.
Episodes of the selection that no selected check can read.
What the run reserves: pipeline-api’s bound for the pinned selection.
object
Credits the run would reserve: on a merged dataset, the sum over its uploads’ runs.
Episodes the model would read: linked to a published task, in a published version, with a camera.
Episodes of the selection it would leave out: not linked to a task, linked to one with no published revision, not in a published version, or without a camera.
The model it would read with: the request’s, else the template’s default.
Every model this deployment’s VLM check offers, the default first.
Accounts/{account}/checks/vlm/runs/{run}: the run; empty with
validate_only.
Checks all or categories reached that the run leaves out, by category
then check id, with why. An id in check_ids that cannot run is refused
instead. The VLM check’s own steps (graph.observe, graph.evaluate) run
only inside it and are left out unlisted.
A check the selection reached that the run leaves out.
object
Accounts/{account}/datasets/{dataset}/checks/{check}
Its active version.
How it reads an episode.
Why the run leaves it out: it cannot run here, or it is of the other kind and runs as its own run.
Example
{ "checks": [ { "kind": "KIND_UNSPECIFIED" } ], "skippedChecks": [ { "kind": "KIND_UNSPECIFIED" } ]}default
Section titled “default ”Default error response
The Status type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by gRPC. Each Status message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the API Design Guide.
object
The status code, which should be an enum value of [google.rpc.Code][google.rpc.Code].
A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the [google.rpc.Status.details][google.rpc.Status.details] field, or localized by the client.
A list of messages that carry the error details. There is a common set of message types for APIs to use.
Contains an arbitrary serialized message along with a @type that describes the type of the serialized message.
object
The type of the serialized message.
Example generated
{ "code": 1, "message": "example", "details": [ { "@type": "example" } ]}