Learning state representations in reinforcement learning using mixed policy successor features
Bibliographic record
Abstract
Reinforcement learning (RL) algorithms often suffer from sample inefficiency compared to humans on the same task.This is because humans have strong priors (especially in terms of good representations) that they have obtained through their evolutionary history and life experiences.Unsupervised pre-training in the environment can also be used to incorporate such priors, though this requires some experience with the environment, if not with rewards.Successor features (SFs) are one strategy to encode priors into state representations.They provide a representation scheme in which each state is represented by expected discounted future occupancy of other states.However, in their conventional formulation, SFs suffer from dependence on policy, dependence on context, and sample inefficiency (requiring extensive training on the task).We propose to address these three problems with a novel approach to pretraining SFs that can serve as good state representations for new agents, akin to evolutionary and life experience priors in the brain.We learn them by minimizing successor feature loss in an unsupervised training phase where data is collected from exploring the environment using a mixture of several (often good) policies with additional exploration, an approach we refer to as "mixed policy successor features" (MPSFs).We show how this approach can help us to learn policy-general and context-general representations that enable sample efficient RL.We also explore how MPSF can enable sample efficient unsupervised pre-training by incorporating a unsupervised initialization phase.We empirically demonstrate the effectiveness of these approaches in grid world environments.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.001 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.001 | 0.000 |
| Science and technology studies | 0.004 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.002 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".