How to use this site

Use a small document sample to compare an extraction setup with one change, check both outputs against the source, and save corrections you can reuse.

Sign in, then open Experiments. Runs shows individual execution records; Corpora shows the document collections available for evaluation.

1. Set up a small comparison

1. In Experiments, choose a baseline for the document type you want to test. A baseline is a saved starting configuration from the extractor's code; it may differ from the current production version. The variant is your edited copy.
2. Choose a corpus (a versioned collection of documents), then select Load calibration documents before editing. Start with a few documents you can check carefully. The selector offers matching calibration documents from locked corpus versions.
3. Give the experiment a descriptive name, such as “Bank statements · clearer transaction instructions”. Change one variable at a time: the prompt, model, response schema, or generation settings. The model list contains the configured Vertex models supported by this app.
4. Set Maximum calls per configuration and Maximum USD per configuration. Each side gets those limits: a pair with 3 calls and USD 2 per configuration can use up to 6 calls and USD 4 in total, subject to the other configured limits.
5. Select Save baseline and variant experiment. Saving creates the comparison and makes no provider call.

2. Run and watch progress

Select Run both configurations to queue the baseline and variant for sequential execution. Select Refresh progress to see updated statuses, failures, estimated cost and recorded response latency. The page does not update itself.

Cancel pending work stops work that has not started. An in-flight provider call may still finish and consume allowance. If execution is unavailable, see the help below.

3. Review the disagreements

1. Select Continue review or Review document, then Open / refresh private preview. The default Needs review view shows exceptions. Valid model agreements are used automatically and require no clicks.
2. For a disputed field, choose Use baseline, Use variant, or Enter value after checking the source. Use Not in source for a confirmed absence or Cannot read for an unreadable field. Leave anything unresolved if needed. Empty text, zero, false, missing and null stay distinct.
3. For unmatched records, select corresponding baseline and variant records, then Save review to expose their remaining field differences. Associate a saved source record when one exists. For one-sided records, confirm whether they are present, absent or unreadable. Matching confirms identity; it does not approve every contained value. Dates, amounts and row positions never establish identity automatically.
4. Select Save review or Save & next document. An evidence page and note are optional. Result shows saved human corrections plus model-agreed values, with unresolved sections listed separately. Viewing or saving never starts a model call.
5. Use All values whenever you want to inspect agreements, saved decisions or source inventory. Open Full source review there to check every field, confirm complete records or add a record both models missed. Its typed controls and Save human reference revision action remain available. Save your draft before switching to that editor.

Ordinary review is partial. Only in Full source review, select I checked every value and every row in the source after checking the complete source inventory. Confirm context records explicitly before declaring completeness. An empty disagreement queue does not mark the source fully human-verified. Both models can share a mistake or omission; optional source spot-checks and full review help find those.

In Full source review, confirm a whole record or choose Edit fields for individual checks. Use the blank forms to add records and Add more blank record forms when needed. If the whole source is unreadable, select Source cannot be read; this saves no guessed values.

Opening a review or preview never saves truth. Model-agreed values are labeled separately from source-checked human decisions. Compatible future experiments reuse saved references; partial record corrections need explicit identity to score the relevant model record. If another reference revision or model output changed, reload before saving. Technical details include scores and the original responses; model confidence is not measured accuracy.

4. Decide what to try next

Compare human-reviewed quality alongside estimated cost, latency and failures before expanding the sample. Scores count the reviewed values and records within the reference's scope; they are not whole-corpus accuracy. Unresolved items still need attention. A small calibration sample can guide your next change, but does not establish performance on unseen documents.

If something is missing or unavailable