THE 4 SCENARIO PREDICTION MODEL CHALLENGE (Courtesy of @VickersBiostats) I’ll present 4 scenarios w/ model evaluation results and ask you what you would do. Try to answer each one, keeping in mind that the goal is to do the most good w/ what’s available! Ready to begin? 👇
#1: 2 models have been proposed to select patients at high-risk for aggressive prostate cancer amongst those with ⬆️ PSA. These 2 models are evaluated on an independent data set. AUC: Model A: 0.81 Model B: 0.79
#1: Should either of these models be used to advise men on prostate biopsy? If so, is one preferable?
#2: Cancer patients routinely have lymph nodes removed, but this often causes adverse effects. There’s a model developed on a different dataset that predicts lymph node invasion w/ an AUC 0.75.
#2: Should this model, developed on a different data set, be used to determine which patients should have lymph node dissection and which can be spared this technique?
#3: Two models have been proposed to select patients at high-risk for aggressive prostate cancer amongst those with elevated PSA. These two models are evaluated on an independent data set. AUC: Model A: 0.78 Model B: 0.83
#3: Model A is the standard model used in many clinics. Model B is a new model that adds in the results of new imaging test. The new test is somewhat unpleasant for patients. Should we use the new model, subjecting patients to the unpleasant new test?
#4: Pts with adv. cancer sometimes get palliative surgery for pain relief. This surgery is contraindicated if pts have a high risk of death w/in 6 mo. There are 2 models, Models A and B — B also includes some add’l markers from routine bloodwork. AUC: Model A: 0.68 Model B: 0.78
#4: Should either model be used? If so, which one?
If you feel like you couldn’t answer one or more of these, what information do you want to know? Name the scenario and I’ll try to answer to the best of my knowledge.
After folks have had a chance to work through this, I’ll post a separate thread (and link here) with my thought process on each one and how I answered. Then I’ll share how decision curves can be used in each of these scenarios (courtesy of slides from @VickersBiostats).
@kdpsinghlab @ADAlthousePhD @VickersBiostats Have any of these models followed TRIPOD guidelines on prediction models and validation? If not, don't use them.
@kdpsinghlab @VickersBiostats Platt scale the models
@nagpalchirag @VickersBiostats Let’s say you do Platt scale the models on your data. Are any of them good enough to use?