Part 5 — Prove It on a Change Chapter 24
Impact Measurement & Continuous Improvement
In this chapter
Count the whole change
A suggestion arrives in seconds. Now count the time it takes to inspect the diff, supply a missing constraint, rerun the checks, and accept the patch. A randomized study of experienced open-source developers using early-2025 models found that developers took longer in the AI-enabled condition in that study. Measure your own task, model, and review effort.
Compare manual and delegated setup
Create two clean worktrees at lab-start and confirm they have the same commit.
git worktree add --detach ../task-api-manual lab-start
git worktree add --detach ../task-api-delegated lab-start
git -C ../task-api-manual rev-parse HEAD
git -C ../task-api-delegated rev-parse HEAD
In the first, time yourself running the two commands from the registration exercise.
npm test
npm run test:registration
The base tests should pass and the contract should fail. In the second worktree, select your model in a fresh Plan session and record its ID. Take the same measurements.
Run npm test and npm run test:registration as separate shell calls.
Use the exact commands. Report each observed result and leave the
repository unchanged.
Include the time you spend checking what ran. If a provider quota, missing tool support, or denied command interrupts the task, record the point where it stopped. Preserve any outcomes you can confirm from individual events. A run that ends before verification has a different outcome from a verified run.
| Record | Where to check it |
|---|---|
| Elapsed time | Time from starting the task through the last verification or recorded interruption |
| Actions and retry count | Visible tool activity and human intervention in the session |
| Tokens or charges | Reported token categories and the provider’s usage or billing record, when available |
| Verification | Command, exit status, and working-tree status |
| Rework | Spec clarifications, changes to the patch, reruns, and approvals you performed |
An export can supply token counts when the client receives them. Use the session ID from opencode session list. Keep cache, input, output, and reasoning counts separate. Store observed commands and results in your own reviewed record.
Change one thing
Try the registration implementation on two clean copies of the same base. Give one session a prompt with the spec path. Give the other the same prompt and the path to the acceptance test. Use the same spec, model, runtime, and verification procedure. A completed result passes both suites and has a diff a person can accept against the spec. Time the work through that review.
Record whether identifying the contract reduced the number of missed criteria or corrections. Repeat before treating a difference as stable. If you repeat across models, record the exact IDs and alternate run order. Quotas, hardware and context settings can change the result.
Account for energy, water, and local impacts
A free quota sets the price you see. For a local run, a wall meter can give you energy for the whole machine. Measure an idle interval of the same length if you want to estimate the extra use above baseline. For a hosted run, use provider measurements with comparable scope where available. Tokens and call counts alone leave energy use unknown.
The Berkeley Lab report distinguishes water used directly at a data center from water associated with its electricity supply. What those impacts mean for a specific run depends on its infrastructure and location. A Virginia legislative review also describes construction-driven economic benefits, local tax revenue, and reported effects of persistent noise at some sites. Keep these observations tied to the places and scopes the reports studied.
Keep the measurement with the task and the control you changed. In the final project you can use it to explain where a second session helped and where a direct check was quicker.
References
- Joel Becker et al. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. arXiv preprint 2507.09089v2, submitted 12 July 2025 and revised 25 July 2025. DOI 10.48550/arXiv.2507.09089. The arXiv record lists no publication venue.
- OpenCode maintainers. CLI commands and Session commands. Official OpenCode 2 documentation.
- International Energy Agency. Key Questions on Energy and AI. IEA report, 16 April 2026.
- Arman Shehabi et al. 2024 United States Data Center Energy Usage Report. Lawrence Berkeley National Laboratory technical report LBNL-2001637, 19 December 2024. DOI 10.71468/P1WC7Q.
- Joint Legislative Audit and Review Commission. Data Centers in Virginia. Report 598 to the Governor and General Assembly of Virginia, 9 December 2024. Study summary.