Intern Final Review · Office of the CAIO
Estimating what a chatbot
is quietly saving us.
Brolin O’Connell
· A Bayesian causal estimate
· 2026
The question
How much time did
the chatbot actually save
a technician per work order?
The data
Two tables. One before, one after.
PRE & POST · WORK ORDERS
hours_worked
project_cost
employee_id
wo_start_date
wo_end_date
location, wo_number
POST ONLY · CHATBOT SESSIONS
session_duration
user_id
wo_referenced
joined on employee_id = user_id,
wo_number = wo_referenced.
03The data
The problem
hours_worked bundles two things.
hours_worked
=
physical labor
+
retrieval time
The chatbot only touches retrieval. To isolate its effect we must split retrieval from labor, and do it for a pre-period where retrieval was never recorded.
04The problem
The idea
Learn labor from the period where we can see it. Predict it for the period where we can't.
STEP 1 — POST
L = hours_workedpost − session_duration
STEP 2 — PRE
R̂manual = hours_workedpre − L̂pre
To do step 2 we need a labor model.
log(L) = μ + β · log(project_cost) + αj + ε
THE MECHANICS — A SINGLE WORK ORDER
Total recorded time, minus the chatbot session, leaves physical labor.
05The idea
Why Bayesian — the mechanics
The model doesn't return a number.
It returns a chain of guesses that teach us how sure to be.
PROPOSE
Pick a nearby value for (μ, β, α). Maybe better, maybe worse.
ACCEPT OR REJECT
Keep it with probability proportional to how well it fits the data and the prior.
REPEAT — THOUSANDS OF TIMES
The chain wanders through plausible worlds. Each step is one draw from the posterior.
06Why Bayesian
From chain to posterior
Each draw is one plausible world. Stack them — the shape of our uncertainty appears.
The width is the honest part. A regression gives a single line and a standard error; the posterior shows the full geography of what the data will let us believe.
07From chain to posterior
What this shows us
Not just "the chatbot saves X minutes."
A RANGE
A credible interval for the savings, not just a point.
A PROBABILITY
A real probability the tool helped at all.
A CAVEAT, BUILT IN
Uncertainty in the labor estimate flows through to the savings. Nothing is hidden.
The one assumption that keeps the whole argument honest: physical labor, for a job of given cost and technician, hasn't drifted between periods.
08What this shows us
PART TWO
Beyond the
chatbot.
The specific problem was work-order lookup time. The shape of the problem is everywhere.
The scale of the problem
Every node is a place
this thinking could live.
Work orders · procurement cycles · mission ops handoffs · FOIA & records · onboarding · data-sharing agreements · incident resolution · travel · contracts · training · vendor rights · center-to-center transfers, all the same shape, none yet instrumented the same way.
10The web of NASA's data
What you'd actually look at
The inference makes a dashboard possible.
TIME SAVED · POSTERIOR MEDIAN
~12 min
90% CI
excludes zero
P(Δ R > 0)
> 0.99
signal strength
HOURS SAVED · PERIOD
~80 hrs
post-deployment
POSTERIOR · Δ R (MIN/LOOKUP)
012 ← median24
ESTIMATED RETRIEVAL TIME · BY WEEK
PRE  ·  POST
Concept. Illustrative values. Every tile reads from the same posterior, so every number comes with a range, because every number already has one.
11Dashboard concept
What I actually learned
The modeling was the straightforward part.
Knowing what to model was the work.
Thank you.
QUESTIONS WELCOME.
12Thank you