Is it Worth the Phone Call?
On pricing a missing data point
These days, most talk about the value of data is about big data, the terabytes of training data that AI companies care about. However, as algorithms are increasingly used to make decisions about individuals, data acquires a value at the individual level, that differs from person to person.
Algorithms don't perform equally well for every person, and whether that's good or bad for the person the algorithm makes a prediction about depends entirely on what the algorithm is for. You'd probably like a medical algorithm to be very good at predicting the right treatment for you. At the same time you'd probably prefer a marketing algorithm to be quite bad at predicting what you can be tempted to spend money on. Depending on the application, being hard to predict is sometimes a liability and sometimes a protection.
How well an algorithm predicts for a specific instance depends on data in several forms: how much training data there was and what kind, how rare cases were handled during fitting, the quality of the record at hand and whether anything in that record is missing. This post is about a small version of the last of those: In a situation where a deployed algorithm has to make a prediction about someone whose record is missing one variable, and that variable could be obtained at a cost, how much should we be willing to pay for it?
The rest of the post works through a toy example to introduce the expected value of information — an old idea from decision theory that, applied one individual and one variable at a time, does roughly what I've just described. The maths below is right as far as it goes, nonetheless this should be viewed more as a thought experiment rather than a serious proposal of how this problem should be solved (hopefully this will come at some later point in time). Comments, objections and ideas are very welcome. — Write me an email.
A patient with a gap in his file
Let's imagine the following situation: You work the front desk at a hospital. Last year they installed an algorithm that predicts how long each incoming patient will stay, so the management can plan bed capacity ahead of time. The algorithm takes five inputs: age, sex, how long the patient's last hospital stay was, why they were admitted that time, and the reason for their current admission.
Now a 68-year-old man is admitted with pneumonia. The last time he had been to the hospital was due to a bad fall. However, that was at a different hospital, so you don't have the length of the stay recorded in the hospital's databank and the patient doesn't remember. Was it four days? Two weeks? He just shrugs.
You could call the other hospital. Realistically that's twenty-five minutes of hold music and being transferred to three departments, and maybe a 70% chance anyone finds the record. And it might turn out that knowing the answer wouldn't have changed the prediction anyway.
How do you decide whether chasing the missing information is worth it?
This looks like a small operational question (and it admittedly is) but I do think it is secretly interesting and the answer is not just relevant for this specific situation: There are more and more deployed models making decisions about individuals whose records have gaps, and someone has to decide when those gaps are worth filling.
Why the answer depends on the patient
The first instinct might be to try to figure out whether the previous length of stay is a variable that actually matters for the model's prediction. This could be answered based on standard variable importance measures (e.g. SHAP, permutation importance).
But actually that's not the question you are faced with: Nobody is deciding whether to ever collect previous length of stay. The decision is whether to spend twenty-five minutes on this particular man, standing here now, with this particular set of things already known about him.
Those two questions can have different answers. This becomes clearer when separating the two factors influencing what you could gain from a phone call:
(1) How much you'd learn. If men like him almost always spent three days in hospital after a fall, you already know the answer to within a rounding error, and there's nothing to chase.
(2) How much it would matter. Even if the previous stay could be anything from three days to three weeks, that only helps if the model actually does something with it. If this man's predicted stay is going to come out at four days regardless, you've spent twenty-five minutes buying a number that changes nothing. This depends on the rest of the patient's record. The same missing field, can be worth a lot for one patient and nothing for the next. That's what makes this a question about the individual rather than about the variable.
Actually pricing it
Now, let's try to find out the Expected Value of the (missing) Information.
Step 1What could the missing value have been?
First, we want to estimate the range of possible values for the missing data point. To do so, search the records for patients who match this man on everything you do know: roughly his age, male, admitted for pneumonia, previous admission was a fall and previous stay duration was recorded. Say you find five. Their previous stays were:
That is a fairly wide spread since "Fall" could mean a lot of things and result in very different stay durations, ranging from a bruised hip with an overnight stay to three weeks in an orthopaedic ward.
Step 2Run each one through the model
Take each of those five durations, plug it into our patient's record, and see what the model says.
| imputed previous stay v (days) | ||||||
|---|---|---|---|---|---|---|
| 3 | 5 | 8 | 14 | 20 | mean | |
| ŷᵢ(v) (days) | 3.99 | 4.29 | 4.74 | 5.64 | 6.54 | 5.04 |
The average is 5.04 days. That's your best guess without the call.1
Step 3Look at the spread
Those five predictions run from 3.99 to 6.54 days. Making the phone call would collapse that spread down to one number, i.e. the prediction that corresponds to the true previous stay. Thus what the call resolves is exactly the observed spread. Measured as variance:
which on a scale of days is $\sqrt{1.091} \approx 1.04$ or, equivalently, 25 hours.
It's worth noting that the whole calculation only needs (1) the models' predictions (but no knowledge about its training parameters, structure or training data) and (2) no information about the realized stay duration of any of the five similar patients from the records (i.e. no outcome data).
Step 4Money
Next the question is whether twenty-five hours of variance in prediction is worth a 25 minute phone call. Suppose being off by a day in a bed reservation costs the hospital about €400, once you count idle capacity and the cost of scrambling when you're short. Your loaded wage is €36/hour, the call takes twenty-five minutes, so it costs €15. And it only works 70% of the time. So essentially, your manager probably wants you to call if the following holds:
Call whenever the missing value could move the prediction by more than about five and a half hours.
Our patient moves it by twenty-five. Expected gain €306, cost €15. →
What about this other guy?
Now assume a second patient walks in (let's call him patient B). Also sixty-eight, also male, also had a fall a couple of years ago at some other hospital, also can't remember how long he was in. Identical file, the only difference is that he has been admitted for a scheduled hip replacement, not pneumonia.
Should you call this time around?
Your reference group is now different people, the search matches on everything you know, including what he's admitted for now. Thus you'd look for men his age having hip replacements whose previous admission was a fall. Different five patients, different five durations.
But there's no particular reason those durations should be more or less spread out. What's being recorded is how long a fall put someone in hospital. That depends on the fall and on the patient, not necessarily on what they happen to turn up with two years later.2
Thus, we assume we observe a fairly similar sample in terms of stay duration:
| previous stays on record (days) | mean | variance | |||||
|---|---|---|---|---|---|---|---|
| Patient A's group | 3 | 5 | 8 | 14 | 20 | 10.0 | 48.5 |
| Patient B's group | 3 | 6 | 9 | 13 | 21 | 10.4 | 48.8 |
While those represent two different groups of people, the variance is almost identical and differs only by 0.6%.
Now run the same procedure:
| imputed v, and the resulting prediction (days) | mean | |||||
|---|---|---|---|---|---|---|
| Patient A's v | 3 | 5 | 8 | 14 | 20 | |
| prediction (pneumonia) | 3.99 | 4.29 | 4.74 | 5.64 | 6.54 | 5.04 |
| Patient B's v | 3 | 6 | 9 | 13 | 21 | |
| prediction (hip) | 3.78 | 3.87 | 3.96 | 4.08 | 4.32 | 4.00 |
Five hours. Below the five-and-a-half-hour threshold. Expected gain of €12 against a €15 call. →
So, contrary to the first case, here the decision should be not to call.
Same blank, different prices
The difference is that a scheduled hip replacement is a highly standardized procedure. There's a protocol. It's designed to get people out on a predictable day, and it mostly works. The model learned this from four years of records, and it means that essentially nothing in this patient's history is going to move the prediction much. He could have spent three days or three weeks recovering from that fall; either way he's out in about four days.
Whereas pneumonia has no such protocol. How long you're in depends on how well you tolerate being ill, and how long you took to bounce back last time is a decent proxy for that. So the previous stay actually does something.
If you'd asked a variable-importance method how much "duration of previous stay" matters, you'd have gotten one number, computed by averaging over the dataset. Permutation importance, SHAP summary plots, etc. all answer "how much does this column matter?"
However, in our example, both our patients are missing the same column. So they'd get exactly the same score. The method cannot see the difference, not because it's a bad method but because the difference isn't a property of the column. It's a property of how that column interacts with the rest of this particular record.
What could go wrong (a lot...)
In the previous sections I have been computing a number and calling it "the value of the call." It's worth being precise about what has to be true for that to be at least a somewhat credible number. Specifically, there are four assumptions that need to hold to make sure the quantity we are computing matches the value we are after at least in expectation3:
Your reference patients resemble your patient. They were selected precisely because their previous stay was on file, which is not a neutral property. If people whose records are incomplete differ systematically from people whose records aren't, beyond what your observed variables capture, your reference group is the wrong comparison (i.e. we assume data is missing at random conditional on the observed variables).
The hospital actually acts on the prediction. The €400-per-day-squared calculation assumes reservations follow the model. If your bed manager adds a fixed safety margin, or overrides the recommendation, the way we price the decision is wrong. The EVI is the value of the information to the decision, not to the model.
The prediction contains what the decision needs. The hospital's cost here depends on the average length of stay, and the model reports an average, so they match. But suppose what actually mattered was whether the patient overruns the six free bed-days on the ward. Then two patients could both be predicted to stay five days while having completely different chances of overrunning — one whose stay is near-certain to land between four and six, another who might plausibly leave in two days or twelve. Same prediction, different costs. In this case you'd need a model that predicts an overrun probability not a point prediction.
The model is right about this variable. If the model is systematically two days optimistic for this kind of patient, that's a real problem for bed planning, but it doesn't affect the value of the phone call — it'd be equally two days optimistic with or without the information. Constant bias cancels. However, if the model's error changes as the missing value changes, the EVI will be wrong.
For example, suppose the model over-responds to the previous stay. This is a plausible failure since it was fitted on data where that field was always filled in, and models trained that way tend to lean on a variable harder than they should. In this case the EVI will be wrongly recommending to call in cases where it shouldn't.
Beyond hospitals
The generic setup of this example - a deployed model, a fixed decision, an incomplete record, and a way to fill the gap for a price - is not rare. With more and more algorithms regularly making decisions about individuals, the general question of when to fill in a missing value, is one of increasing relevance.
Moreover, everything above asks when a decision-maker would choose to fill in a missing value. However, when an algorithm is deciding a benefit claim or a risk score rather than a bed, the costs and benefits land on different people. The individual has to deal with the consequences of a badly-targeted decision, the institution considers only the cost of the phone call. This opens up an additional ethical / fairness question: at what point should an institution be required to go get the missing value before letting the model decide?
- Another obvious idea could be to impute the average previous stay (10 days) and predict once. In our stylized toy example this gives the same answer, because the model happens to respond linearly to the missing field (see Appendix for further details). In general however these two calculations don't lead to the same result and the two aren't equally good. What you want is your best guess about this patient, which means averaging over the values his missing field might take — and that's an average of predictions, not a prediction at an average. Imagine a model that had learned a U-shape, where both very short and very long previous stays predict a longer stay now. Predicting at the midpoint would return a number that none of the plausible values supports. (It also isn't always available: if the missing field were categorical, there'd be no average category to impute.)↩
- Strictly this needs the previous stay to be independent of the current diagnosis given age, sex, and the previous admission reason. This might not be exactly true since both are partly downstream of how robust the patient is. However it's close enough that there is no reason to expect the two reference groups to look systematically different.↩
- One of the empirical challenges is that both ingredients of the EVI have to be estimated, and the number inherits their error. The spread of plausible values is the noisier one: read off $k$ donors, a sample variance has a relative standard error of about $\sqrt{2/(k-1)}$, which for the five donors above is roughly 70%. The slope is the more dangerous one, because its error does not average out favourably. If $\hat\beta_3$ is unbiased with sampling variance $v$, the reported EVI has expectation $(\beta_3^2 + v)\,\tau^2$ while the true value of acting on it is $(\beta_3^2 - v)\,\tau^2$: the plug-in estimate overstates the value of the call by $2v\tau^2$, and the call stops being worth making at all once $v$ exceeds $\beta_3^2$. This is a separate problem from the outcome being noisy — the residual variance is the same whether or not you call, so it cancels out of the comparison (A.1) and a large $\sigma^2$ does not by itself make the individual EVI wrong.↩
Where the numbers came from
Both the "truth" and the "model predictions" for this toy example have been generated from a simple linear regression:
with $\gamma_{\mathrm{fall}} = 0.60$, $\delta_{\mathrm{pneumonia}} = 0$ (the reference category) and $\delta_{\mathrm{hip}} = 0.15$. The slope on the previous stay depends on the current diagnosis:
Each additional day of a pneumonia patient's previous stay lengthens the predicted current one by about four hours; for the surgical pathway it buys forty-five minutes. This interaction is the difference between the two patients.
A.1Calculating EVI from the model
For a 68-year-old man whose previous admission followed a fall, everything but the missing feature is fixed. The intercept comes to $0.50 + 0.03(68) + 0.40 + 0.60 = 3.54$ for pneumonia, and $0.15$ more for the hip replacement, so the model collapses to a single line in $v$ for each patient:
Evaluating these at the respective donor values reproduces the two rows of predictions used in the text.
Because the equation above serves as both the data-generating process and the prediction function, the true conditional distribution of the outcome is $y \mid X_i^{\mathrm{obs}}, v \sim \mathcal{N}\big(\hat y_i(v),\, 1.5^2\big)$: the model's prediction is the centre of the distribution the outcome is actually drawn from. The true risk is therefore
while the model risk is $R(a, \hat y_i(v)) = c(a - \hat y_i(v))^2$. Differencing across two actions removes the second term, and the two differences coincide exactly: $\varepsilon = 0$.
With the model in hand, EVI can be computed without imputing anything.
Under the model above, $\mathrm{EVI}_i = \beta_3(d_i)^2 \cdot \operatorname{Var}\big(v \mid X_i^{\mathrm{obs}}\big)$.
Proof. By the collapse argument above, $\mathrm{EVI}_i$ is the variance of $\hat y_i(v)$ over $v$. By the line equations this is $\beta_3(d_i) v$ plus a constant, and $d_i$ is observed, so the constant drops out of the variance and the slope comes out squared. ∎
Checking against the text: the reference durations have sample variances $48.5$ and $48.8$ days$^2$ respectively, so
matching the values obtained by imputation.
A.2Estimation error due to miscalibration
Suppose the model's slope were fitted rather than known, and came out at $\hat\beta_3 = 0.35$ against a true $0.15$. The prediction and the truth now come apart: writing $m_i(v) := \mathbb{E}[y \mid X_i^{\mathrm{obs}}, v]$ for the true conditional mean, the model's error $\hat y_i(v) - m_i(v)$ grows with the missing feature at rate $0.35 - 0.15 = 0.20$. Working through the algebra gives
where $\tau^2 = \operatorname{Var}(v \mid X_i^{\mathrm{obs}})$. The call has positive value if and only if $0 < \hat\beta_3 < 2\beta_3$: at exactly twice the true slope the sharpened prediction is as wrong as the averaged one, and beyond that, acquiring the missing value makes the reservation worse.
With the figures above, $\mathrm{EVI}_A$ would be reported as $5.94$ days$^2$. This is a claimed expected saving of some €1,660 — while the true value of the call is $-$€340. The accuracy of the $EVI$ thus heavily depends on the calibration of the underlying model.