Researchers have demonstrated that the publicly trained large language models (LLM) can decipher not only the pain location of musculoskeletal disorders (like lower back, knee, and shoulder pain), but also the acuity of pain in all three of these conditions – with exceptional efficiency and accuracy. This information is now available at the point of care level, assisting clinicians with the management of complex musculoskeletal conditions.
The novel use of the publicly trained LLM can effectively reduce the barriers to using sophisticated AI models at the point of care level in medicine. It will also allow frontline clinicians to improve the quality and delivery of care, more quickly.
To examine the suitability of LLMs, researchers gathered 26,551 patient notes from five Mount Sinai facilities, using a simple text match for initial selection. Data was drawn across a five year period.
For the exercise, clinical personnel manually labelled 1,714 notes from 1,155 patients for pain location and acuity. 19 percent of the final dataset arose from within the primary care setting, 51 percent from Internal Medicine, and 30 percent from Orthopaedics. Labels were created for the location (such as for shoulders, lower back, knee, or “other” pain), and the acuity (that is, acute, chronic and acute-on-chronic).
These data were used to fine-tune a publicly available foundational language model named LLaMA-7B.
The researchers also trained another LLaMA-7B using a note dataset combined with the Alpaca dataset which contains over 50,000 general-purpose instructions paired to expected responses.
Researchers then used a method called Group Shuffle Splitting to partition data into 75 percent training, 5 percent validation, and 20 percent testing groups. For parsing clinical notes, both LLaMA-7B models outperformed baseline models except in a few cases.
The data showed that the LLaMA-7B model trained on patient notes alone achieved high classification accuracies for knee pain and for other pain locations. The LLaMA-7B model trained with the extended Alpaca dataset exhibited slightly better accuracy for shoulder pain but performed similarly or slightly worse in other categories.
The researchers’ models also categorized pain acuity as acute, chronic, or acute-on-chronic to a high level of accuracy. This means that pre-trained Large Language Models can serve as a robust foundation for creating fine-tuned models capable of effectively parsing unstructured clinical notes in a directed manner.
In terms of practical application, this suggests that such models can be deployed as specialized conversational agents or chatbots, helping clinicians swiftly access pertinent patients, maintain data privacy, and potentially streamline clinical workflow.
The research appears in the medical journal The Lancet and it is titled “Using fine-tuned large language models to parse clinical notes in musculoskeletal pain disorders.”
