Skip to content
Hands-on Coding Assistants

Part 5 — Prove It on a Change Chapter 24

Impact Measurement & Continuous Improvement

In this chapter

Count the whole change

A suggestion arrives in seconds. Now count the time it takes to inspect the diff, supply a missing constraint, rerun the checks, and accept the patch. A randomized study of experienced open-source developers using early-2025 models found that developers took longer in the AI-enabled condition in that study. Measure your own task, model, and review effort.

Compare manual and delegated setup

Create two clean worktrees at lab-start and confirm they have the same commit.

git worktree add --detach ../task-api-manual lab-start
git worktree add --detach ../task-api-delegated lab-start
git -C ../task-api-manual rev-parse HEAD
git -C ../task-api-delegated rev-parse HEAD

In the first, time yourself running the two commands from the registration exercise.

npm test
npm run test:registration

The base tests should pass and the contract should fail. In the second worktree, select your model in a fresh Plan session and record its ID. Take the same measurements.

Run npm test and npm run test:registration as separate shell calls.
Use the exact commands. Report each observed result and leave the
repository unchanged.

Include the time you spend checking what ran. If a provider quota, missing tool support, or denied command interrupts the task, record the point where it stopped. Preserve any outcomes you can confirm from individual events. A run that ends before verification has a different outcome from a verified run.

RecordWhere to check it
Elapsed timeTime from starting the task through the last verification or recorded interruption
Actions and retry countVisible tool activity and human intervention in the session
Tokens or chargesReported token categories and the provider’s usage or billing record, when available
VerificationCommand, exit status, and working-tree status
ReworkSpec clarifications, changes to the patch, reruns, and approvals you performed

An export can supply token counts when the client receives them. Use the session ID from opencode session list. Keep cache, input, output, and reasoning counts separate. Store observed commands and results in your own reviewed record.

Change one thing

Try the registration implementation on two clean copies of the same base. Give one session a prompt with the spec path. Give the other the same prompt and the path to the acceptance test. Use the same spec, model, runtime, and verification procedure. A completed result passes both suites and has a diff a person can accept against the spec. Time the work through that review.

Record whether identifying the contract reduced the number of missed criteria or corrections. Repeat before treating a difference as stable. If you repeat across models, record the exact IDs and alternate run order. Quotas, hardware and context settings can change the result.

Account for energy, water, and local impacts

A free quota sets the price you see. For a local run, a wall meter can give you energy for the whole machine. Measure an idle interval of the same length if you want to estimate the extra use above baseline. For a hosted run, use provider measurements with comparable scope where available. Tokens and call counts alone leave energy use unknown.

The Berkeley Lab report distinguishes water used directly at a data center from water associated with its electricity supply. What those impacts mean for a specific run depends on its infrastructure and location. A Virginia legislative review also describes construction-driven economic benefits, local tax revenue, and reported effects of persistent noise at some sites. Keep these observations tied to the places and scopes the reports studied.

Keep the measurement with the task and the control you changed. In the final project you can use it to explain where a second session helped and where a direct check was quicker.

References