Reinforcement learning in large, structured action spaces: A simulation study of decision support for spinal cord injury rehabilitation
Bibliographic record
Abstract
Reinforcement learning (RL) has helped improve decision-making in several domains but can be challenging to apply; this is the case for rehabilitation of people with a spinal cord injury (SCI). Among other factors, applying RL in this domain is difficult because there are many possible treatments (i.e., large action space) and few detailed records of longitudinal treatments and outcomes (i.e., limited training data). Applying Fitted Q Iteration in this domain with linear models and the most natural state and action representation results in problems with convergence and overfitting. However, isolating treatments from one another can mitigate the convergence issue, and treatments for SCIs have meaningful groupings that can be used to combat overfitting. We propose two approaches to grouping treatments so that an RL agent can learn effectively from limited data. One relies on domain knowledge of SCI rehabilitation and the other learns similarities among treatments using an embedding technique. After re-interpreting the data using these treatment grouping approaches in conjunction with our process that isolates the treatment groups, we use Fitted Q Iteration to train an agent that learns to select better treatments. Through a simulation study designed to reflect the properties of SCI rehabilitation, we find that agents trained after using either grouping method can help improve the treatment decisions of individual physiotherapists, but the approach based on domain knowledge offers better performance. Our findings provide a proof of concept that applying RL has the potential to help improve the treatment of those with an SCI and indicates that continued efforts to gather data and apply RL to this domain are worthwhile. • Reinforcement learning is challenging to apply to spinal cord injury rehabilitation. • We created a simulator to imitate spinal cord injury rehabilitation. • Grouping treatments facilitated effective reinforcement learning in the simulator. • Simulated patient outcomes were improved by using reinforcement learning.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.002 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".