A is the baseline pz-59-agent-capture-baseline · 3214a1c5d383; B is the candidate pz-61-agent-capture-row-changed · 33ba22f318c0.
What changed? chose this comparison: compared with pz-59-agent-capture-baseline, the most recent earlier run whose final output differs.
The short answer
The runs first differ at agent_step, where 2 inputs changed.
Same: 0 · differ: 1.iThe comparison's rows, by the SDK's verdict: 1 FIRST DIFFERENCE = 1.
agent_stepFirst difference
Checked from the records. The same answer every time.
Model-call records, compared by onetrace
Each step where both runs hold a model-call record of the same name, in the words of the installed onetrace's comparator: its result, the mode it applied, what it rested on, and the rest of its report. It reads the two records; it runs no model.
Compared with the comparator's defaults: mode 1, fingerprint form tolerant.
result: could not compare: input not comparable (input.token_ids_digest: not observed)
mode 1
rests on: nothing beyond the preconditions
fingerprint: not compared (input not comparable (input.token_ids_digest: not observed))
output: not compared (input not comparable (input.token_ids_digest: not observed))
not compared; conditions not met: input not comparable (input.token_ids_digest: not observed); fingerprint 'tolerant' not recorded on A and B
records: A sha256:f0a2a8a78b96…, B sha256:9a3712103a2b…Each fingerprint in fullsha256:f0a2a8a78b9604ed99e1136147feb07c5f092bf0ca49d3909e0d27f138309d8bsha256:9a3712103a2bffde2d3438f3a69f1cee3d85d33720f131f3c30c5e8e5d2760fc
receipts: A verified, B verified
not claimed: hidden reasoning; semantic correctness; causality from a differing value alone; reproducibility across GPUs, versions or engines; hosted models' internals; anything not observed. Record equivalence is not correctness.
result: could not compare: batch context not observed: batch.size, batch.padded_width, batch.position, batch.members, batch.fillers, batch.batch_invariant
mode 1
rests on: nothing beyond the preconditions
fingerprint: not compared (batch context not observed: batch.size, batch.padded_width, batch.position, batch.members, batch.fillers, batch.batch_invariant)
output: not compared (batch context not observed: batch.size, batch.padded_width, batch.position, batch.members, batch.fillers, batch.batch_invariant)
not compared; conditions not met: batch context not observed: batch.size, batch.padded_width, batch.position, batch.members, batch.fillers, batch.batch_invariant; input not comparable (input.request_digest: not observed); fingerprint 'tolerant' not observed
records: A sha256:97c5212188b9…, B sha256:97c5212188b9…Each fingerprint in fullsha256:97c5212188b9b0ae0cc275225cef3f3ce635cc9f892006471c049454fb6c79fa
receipts: A verified, B verified
not claimed: hidden reasoning; semantic correctness; causality from a differing value alone; reproducibility across GPUs, versions or engines; hosted models' internals; anything not observed. Record equivalence is not correctness.
baseline: nested run pz-60-agent-capture-baseline-subagent, its head recorded as sha256:66521c85c6b6…sha256:66521c85c6b63ac150f8879decccf464d910046ecfd912e684fdfd5b66664dd3 · open its page
candidate: nested run pz-62-agent-capture-row-changed-subagent, its head recorded as sha256:72e26c8224b3…sha256:72e26c8224b3594d45af79eba68673abcef4583702017eb2dba4522cdb106345 · open its page
At step agent_step: spans from your app's traces (the files under traces/)
From the app's own traces, as reported. Not checked by onetrace.
onetrace.stage agent_step
attribute
baseline
candidate
duration
28.0 ms
17.0 ms
differs
onetrace.app.span_id
5d3d846c2804fb50
20fd583e7d0ffc03
differs
onetrace.app.trace_id
a3219e9d5de594d74ee68b731d07b464
b728f85bab6e787aaa4373f6e15231c9
differs
onetrace.inputs.digests
["sha256:d77ccb69e3f9…", "sha256:66521c85c6b6…"]Each fingerprint in fullsha256:d77ccb69e3f915ed75b4a2db3ff5893fdd0f433e02fdff100441fb843aae585bsha256:66521c85c6b63ac150f8879decccf464d910046ecfd912e684fdfd5b66664dd3
["sha256:d77ccb69e3f9…", "sha256:72e26c8224b3…"]Each fingerprint in fullsha256:d77ccb69e3f915ed75b4a2db3ff5893fdd0f433e02fdff100441fb843aae585bsha256:72e26c8224b3594d45af79eba68673abcef4583702017eb2dba4522cdb106345
differs
onetrace.outputs.digests
["sha256:6b2e802ce2ef…", "sha256:f0a2a8a78b96…", "sha256:97c5212188b9…", "sha256:3036e8bf5149…"]Each fingerprint in fullsha256:6b2e802ce2ef5f0c5bbe648eaf721379fed6b1da575407368d3af6c2a606952esha256:f0a2a8a78b9604ed99e1136147feb07c5f092bf0ca49d3909e0d27f138309d8bsha256:97c5212188b9b0ae0cc275225cef3f3ce635cc9f892006471c049454fb6c79fasha256:3036e8bf514997c0c20570671c14c5d05fe6a431e1e1ad9b2d931a62df71aabc
["sha256:b891d1cf41d0…", "sha256:9a3712103a2b…", "sha256:97c5212188b9…", "sha256:3036e8bf5149…"]Each fingerprint in fullsha256:b891d1cf41d086267ab96ca798215aeb800c8785fe8a6b68872568d8a026c37asha256:9a3712103a2bffde2d3438f3a69f1cee3d85d33720f131f3c30c5e8e5d2760fcsha256:97c5212188b9b0ae0cc275225cef3f3ce635cc9f892006471c049454fb6c79fasha256:3036e8bf514997c0c20570671c14c5d05fe6a431e1e1ad9b2d931a62df71aabc
The app's spans at the first difference, side by side
Facts from the app's traces, not checked by onetrace, ranked: a status error first, then retries, a model, a span on one side only, tokens, then time.
No span view for the candidate run here: its trace file holds no span of the trace this step recorded.
What these spans do not say
The spans are the app's own report of what it did.
onetrace checks the join, the trace context it recorded for this step, not what the spans say.
A span difference is not a verdict and changes no verdict.
At the first difference: the model's rows and choices
Where both runs' model calls at the first differing step kept a per-step capture, what the two captures say there, from the steps files as the records name them. It reads the files; it runs no model.
model_call.json
Logits rows (L3)
The logits row changed at token 2; the chosen token first changed at token 4.
2 positions before token 4 had a changed logits row and the same chosen token.
the capture, as recorded
baseline
candidate
capture level
capture level L3
capture level L3
digest method
digest method sha256-tree-512/v0
digest method sha256-tree-512/v0
model runner
model runner V1
model runner V1
batch-invariant mode
batch-invariant mode on
batch-invariant mode on
digests computed beyond the delivered tokens, counted and never compared: baseline 0, candidate 0
L3 shows whether the row the sampler chose from was the same bytes, not why it changed.
The cost and repeat figures in onetrace's docs are from one GPU, two models and greedy decoding.
A digest difference is not a verdict and changes no verdict.
Candidates (L2)
At token 4, the first token whose choice differs:
the candidate's token 40 is not among the baseline's top 4
the baseline's token 15 is not among the candidate's top 4
example data, constructed: the demo's own, not a real run's
k as recorded: 4 and 4
the capture, as recorded
baseline
candidate
capture level
capture level L3
capture level L3
k requested
k requested 4
k requested 4
k returned, fewest
k returned, fewest 4
k returned, fewest 4
k returned, most
k returned, most 4
k returned, most 4
sampling: allowed_token_ids
sampling: allowed_token_ids none
sampling: allowed_token_ids none
sampling: bad_words
sampling: bad_words none
sampling: bad_words none
sampling: frequency_penalty
sampling: frequency_penalty 0.0
sampling: frequency_penalty 0.0
sampling: logit_bias
sampling: logit_bias none
sampling: logit_bias none
sampling: logprobs
sampling: logprobs 4
sampling: logprobs 4
sampling: min_tokens
sampling: min_tokens 0
sampling: min_tokens 0
sampling: presence_penalty
sampling: presence_penalty 0.0
sampling: presence_penalty 0.0
sampling: repetition_penalty
sampling: repetition_penalty 1.0
sampling: repetition_penalty 1.0
sampling: structured_outputs
sampling: structured_outputs none
sampling: structured_outputs none
sampling: temperature
sampling: temperature 0.0
sampling: temperature 0.0
sampling: thinking_token_budget
sampling: thinking_token_budget none
sampling: thinking_token_budget none
A rank shows how the candidates were ordered, not why a different one was chosen.
L2 shows no probabilities unless the record holds them.
A candidate difference is not a verdict and changes no verdict.
Settings that changed
No recorded setting differs between the two runs.
The text diff
Shown with at most 200 lines per output (the SDK's default).
210210 "steps": {211211 "source": "model_call.steps.json",212212 "status": "observed",213- "value": "sha256:6b2e802ce2ef5f0c5bbe648eaf721379fed6b1da575407368d3af6c2a606952e"213+ "value": "sha256:b891d1cf41d086267ab96ca798215aeb800c8785fe8a6b68872568d8a026c37a"214214 },215215 "steps.count": {216216 "source": "example data, constructed for the demo; no model was called",
By member, as JSON
Numbers are compared as written, so 1.0 and 1.00 differ.
Not shown for graph-shaped runs: where the change started, and other runs that recorded the same value.
This console draws where the change started only for two linear runs; the SDK's localize report for this pair is below, as it wrote it.
The localize report as the SDK wrote it, with names from the records escaped
baseline a pz-59-agent-capture-baseline (run)
candidate b pz-61-agent-capture-row-changed (run)
localize: diverged
node A B verdict
--------------------------------------------------------------------
agent_step 1 1 FIRST DIFFERENCE
input run:pz-60-agent-capture-baseline-subagent differs
input run:pz-62-agent-capture-row-changed-subagent differs
output model_call.steps.json differs
output model_call.json differs
first difference agent_step
comparisons performed 1 of 1 nodes; same 0, FIRST DIFFERENCE 1, downstream 0, reconverged 0, COULD NOT CHECK 0
FIRST DIFFERENCE at node 'agent_step'
instrument agent-planner 1.0.0 -> 1.0.0
config 0e5774cac5af -> 0e5774cac5af
input run:pz-60-agent-capture-baseline-subagent differs
input run:pz-62-agent-capture-row-changed-subagent differs
blast radius not applicable — run-scoped
carried:
[baseline] [NOT-RUN] originality on 2 receipts -- record is declared unanchored; this verifier can establish consistency, not originality
[candidate] [NOT-RUN] originality on 2 receipts -- record is declared unanchored; this verifier can establish consistency, not originality
signing [baseline]: [SIGNED-UNTRUSTED] signatures/1.json: signature -- signed by an untrusted key sha256:6e20550ae0dc6b4471d54336d09d74d14f3f623b735045eddd053c444ec3b14c -- signer not on the reader's trust list
signing [candidate]: [SIGNED-UNTRUSTED] signatures/1.json: signature -- signed by an untrusted key sha256:6e20550ae0dc6b4471d54336d09d74d14f3f623b735045eddd053c444ec3b14c -- signer not on the reader's trust list