Learning state representations in reinforcement learning using mixed policy successor features
Bibliographic record
Abstract
Reinforcement learning (RL) algorithms often suffer from sample inefficiency compared to humans on the same task.This is because humans have strong priors (especially in terms of good representations) that they have obtained through their evolutionary history and life experiences.Unsupervised pre-training in the environment can also be used to incorporate such priors, though this requires some experience with the environment, if not with rewards.Successor features (SFs) are one strategy to encode priors into state representations.They provide a representation scheme in which each state is represented by expected discounted future occupancy of other states.However, in their conventional formulation, SFs suffer from dependence on policy, dependence on context, and sample inefficiency (requiring extensive training on the task).We propose to address these three problems with a novel approach to pretraining SFs that can serve as good state representations for new agents, akin to evolutionary and life experience priors in the brain.We learn them by minimizing successor feature loss in an unsupervised training phase where data is collected from exploring the environment using a mixture of several (often good) policies with additional exploration, an approach we refer to as "mixed policy successor features" (MPSFs).We show how this approach can help us to learn policy-general and context-general representations that enable sample efficient RL.We also explore how MPSF can enable sample efficient unsupervised pre-training by incorporating a unsupervised initialization phase.We empirically demonstrate the effectiveness of these approaches in grid world environments.1
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.006 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.002 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.002 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".