Analytics, ROI & rollout

You can't govern what you don't measure, and different audiences care about different numbers. Group the metrics by the question they answer:

• Adoption (is the team using it?): DAU/WAU/MAU and sessions per user.
• Output (is work shipping?): PRs created and merged, sessions by category.
• Efficiency (at what cost?): ACU per merged PR, session size distribution.
• Quality (is the work good?): PR acceptance rate. This is the same 70% baseline climbing to 80–90% you've seen in earlier courses, now measured at org level, and it's the single best summary of how well an org's context and habits are working.

Where to get them: the in-app analytics and consumption pages cover day-to-day steering; the metrics and consumption APIs export everything to your own dashboards for exec reporting; audit logs cover compliance reporting. Session Insights, which you met in Lesson 6 as a cost tool, doubles as a quality lens on individual sessions.

One principle keeps reports honest: measure outcomes, not activity. Raw session count is the easiest number to grow and the least meaningful; a team can run hundreds of sessions that merge nothing. Merged PRs, acceptance rate, and ACU per merged PR tell you whether Devin is shipping work at a reasonable cost. Report those.

The last piece is the rollout playbook. Lesson 4 covered the rollout machinery (Testing → Available → Enabled by default, canary orgs, rollback); this is the organizational playbook, and it ties the whole course together. Each phase has an exit criterion you should be able to name:

1. Pilot with champions. One org, staffed with people who want this to work. Exit: champions merging Devin PRs weekly and an acceptance rate trending toward the baseline.
2. Verify the environment. Builds green, sessions starting clean (Lesson 4). Exit: a stable snapshot with no recurring build failures.
3. Seed context. Knowledge onboarding, proven playbooks, repo skills (Lesson 7). Exit: pilot sessions stop needing the same corrective follow-ups.
4. Expand org by org. Repeat the day-one onboarding for each team, using the pilot org as the template. Exit: each new org reaches pilot-level acceptance rate.
5. Govern. ACU limits, guardrails, and analytics review as an operating cadence (Lessons 5 and 6). Exit: this isn't a rollout anymore; it's how you run Devin.

Everything in this course lands in that sequence, and the sandbox that follows runs you through a full day-one onboarding: org, identity, repos, environment, secrets, governance, verify.