{predictNMB}

Rex Parsons

predictNMB Team:

  • Rex Parsons
  • Adrian Barnett
  • Robin Blythe
  • Steven McPhail(Supervisor)
  • Susanna Cramb(Supervisor)





  • Adam Sparks
  • Tinula Kariyawasam
  • Emi Tanaka

What is a clinical Decision Support System?

What is a clinical Decision Support System?

What is a clinical prediction model?

Diagnostic

What is a clinical prediction model?

Prognostic

What is a clinical prediction model?

Prognostic

Why we should we evaluate in terms of costs

Why we should we evaluate in terms of costs

Why we should we evaluate in terms of costs

Why we should we evaluate in terms of costs

Why we should we evaluate in terms of costs

Why we should we evaluate in terms of costs

Why we should we evaluate in terms of costs

Why we should use {predictNMB}

{predictNMB}

  • User provides inputs for setting & intervention (w/ uncertainty)
  • Simulates & evaluates clinical prediction models within CDSS
  • Estimates patient benefit and healthcare costs

DEMO

# install.packages("predictNMB")
library(predictNMB)
library(ggplot2)
library(parallel)

Example problem and inputs required - inpatient falls

  • Falls leads to about 0.04 lost Quality-Adjusted Life Years (QALYs) (Latimer et al. 2013) and has an approximate beta distribution of: \[\mathrm{B}(\alpha = 2.95, \beta = 32.25)\]

  • There are also additional healthcare costs of about $6669 (Morello et al. 2015) and follows an approximate gamma distribution of: \[\Gamma (\alpha = 22.05, \beta = 0.0033) \]

  • Fall prevention education…

    • has a fixed, known cost of $77.3 per patient (Hill et al. 2015)
    • reduces probability of fall by 45% (Haines et al. 2011) - the log hazard ratio follows an approximate normal distribution of: \[\mathcal{N}(\mu = -0.844, \sigma = 0.304) \]

Example problem and inputs required - inpatient falls

  • The willingness-to-pay (WTP) for us is $28033 AUD

  • Current practice: Everyone gets the fall prevention intervention (treat-all approach).

Input Distribution R code
QALYs lost \[\mathrm{B}(\alpha = 2.95, \beta = 32.25)\] rbeta(n = 1, shape1 = 2.95, shape2 = 32.25)
Healthcare costs \[\Gamma (\alpha = 22.05, \beta = 0.0033) \] rgamma(n = 1, shape = 22.05, rate = 0.0033)
Treatment effect (hazard) \[\exp(\mathcal{N}(\mu = -0.844, \sigma = 0.304)) \] exp(rnorm(n = 1, mean = -0.844, sd = 0.304))
Treatment cost $77.30 -
WTP $28033 -

Calculations and code for using details in paper cited papers above is described in (Parsons et al. 2023). We used {fitdistrplus} but you can also use a shiny app by Nicole White and Robin Blythe: ShinyPrior (White and Blythe 2023).

Objectives/Questions

  • We have a prediction model which has an AUC of about 0.8 and we want to know whether it’ll be worthwhile implementing it within a CDSS to reduce healthcare costs (giving people that are unlikely to fall the intervention at $77.3 a pop!)

  • We are currently in a geriatric ward where the fall rate is about 0.1 (1 in 10 admitted patients have a fall) but are also interested in implementing the same model in the acute care ward (fall rate = 0.03). Would we expect to make the same conclusion?

  • We think we can improve the performance of the model up to 0.95 with some extra effort by the models - would this change our conclusion?

Making our samplers

validation_sampler <- get_nmb_sampler(
  outcome_cost = function()  rgamma(1, shape = 22.05, rate = 0.0033),
  wtp = 28033,
  qalys_lost = function() rbeta(1, shape1 = 2.95, shape2 = 32.25),
  high_risk_group_treatment_effect = function() exp(rnorm(1, mean = -0.844, sd = 0.304)),
  high_risk_group_treatment_cost = 77.3,
  low_risk_group_treatment_effect = 0,
  low_risk_group_treatment_cost = 0,
  use_expected_values = FALSE
)

Making our samplers

validation_sampler <- get_nmb_sampler(
  outcome_cost = function()  rgamma(1, shape = 22.05, rate = 0.0033),
  wtp = 28033,
  qalys_lost = function() rbeta(1, shape1 = 2.95, shape2 = 32.25),
  high_risk_group_treatment_effect = function() exp(rnorm(1, mean = -0.844, sd = 0.304)),
  high_risk_group_treatment_cost = 77.3,
  low_risk_group_treatment_effect = 0,
  low_risk_group_treatment_cost = 0,
  use_expected_values = FALSE
)


training_sampler <- get_nmb_sampler(
  outcome_cost = function()  rgamma(1, shape = 22.05, rate = 0.0033),
  wtp = 28033,
  qalys_lost = function() rbeta(1, shape1 = 2.95, shape2 = 32.25),
  high_risk_group_treatment_effect = function() exp(rnorm(1, mean = -0.844, sd = 0.304)),
  high_risk_group_treatment_cost = 77.3,
  low_risk_group_treatment_effect = 0,
  low_risk_group_treatment_cost = 0,
  use_expected_values = TRUE
)

Primary analyses

Running our simulation (primary use-case)

cl <- makeCluster(detectCores() - 1)

primary_sim <- do_nmb_sim(
  n_sims = 500,
  n_valid = 10000,
  sim_auc = 0.8,
  event_rate = 0.1,
  cutpoint_methods = c("all", "none", "youden", "value_optimising"),
  fx_nmb_training = training_sampler,
  fx_nmb_evaluation = validation_sampler,
  show_progress = TRUE,
  cl = cl
)

Interpreting the results

summary(primary_sim)
# A tibble: 4 × 3
  method           median `95% CI`         
  <chr>             <dbl> <chr>            
1 all               -580. -937.8 to -282.3 
2 none              -906. -1338.4 to -563.2
3 value optimising  -591. -934.8 to -288.8 
4 youden            -649. -1063.6 to -340.3

Interpreting the results

autoplot(primary_sim) + theme_sim()

Interpreting the results

autoplot(primary_sim, what = "cutpoints") + theme_sim()

Interpreting the results

autoplot(primary_sim, what = "inb", inb_ref_col = "all") + theme_sim()

Interpreting the results

autoplot(primary_sim, what = "qalys") + theme_sim()

Interpreting the results

ce_plot(primary_sim, ref_col = "all", add_prop_ce = TRUE)

Acute care setting

cl <- makeCluster(detectCores() - 1)

acute_care_sim <- do_nmb_sim(
  n_sims = 500,
  n_valid = 10000,
  sim_auc = 0.8,
  event_rate = 0.03,
  cutpoint_methods = c("all", "none", "youden", "value_optimising"),
  fx_nmb_training = training_sampler,
  fx_nmb_evaluation = validation_sampler,
  show_progress = TRUE,
  cl = cl
)

Interpreting the results - acute care setting

autoplot(acute_care_sim) + theme_sim()

Interpreting the results - acute care setting

ce_plot(acute_care_sim, ref_col = "all", add_prop_ce = TRUE)

Bonus question - what happens when our inputs change?

screen_simulation_inputs(
  n_sims = 500,
  n_valid = 10000,
  sim_auc = 0.8,
  event_rate = 0.03,
  fx_nmb_training = training_sampler,
  fx_nmb_evaluation = validation_sampler,
  cutpoint_methods = c("all", "none", "youden", "value_optimising"),
  show_progress = TRUE,
  cl = cl
)

What happens when the intervention cost changes?

see RWParsons/rmed2023-predictNMB/demo-code/cost-of-treatment-screen.R for code to make cost_screen using screen_simulation_inputs()!

autoplot(cost_screen) +
  scale_x_discrete(labels = function(x) gsub("[A-Z]\\-", "$", x))

{predictNMB}

docs.ropensci.org/predictNMB/


@RexParsons8

RWParsons/venables-award-2023-preso


Value-optimising cutpoint

predictNMB