A Novel Competency Tagging Method Through Semantic Search Using Fine-Tuned LLM
Bibliographic record
Abstract
Competency tagging plays an essential role in both academic and industrial settings, enabling the alignment of learning content, job postings, and resumes with specific skill sets. However, traditional manual tagging is costly, time-intensive, and prone to inconsistencies. In this work, we propose an automated competency tagging method leveraging semantic search with fine-tuned Large Language Model (LLM). Our approach encodes textual data from learning materials and competency descriptions into a shared embedding space, enabling efficient retrieval of relevant competency tags via similarity search. We systematically evaluate semantic matching at different levels of granularity-document, paragraph, and sentence-to optimize retrieval performance. Furthermore, we fine-tune the LLM using Low-Rank Adaptation (LoRA) to improve competency tagging while maintaining efficiency. Experiments on a dataset of 164 pages of learning content and 96 competencies demonstrate the effectiveness of our method, achieving a recall@10 of $80.29 \%$. Notably, fine-tuning with LoRA led to a $6 \%$ improvement in recall@10, highlighting its impact on enhancing retrieval performance. Our findings underscore the potential of fine-tuned LLMs for high-precision competency tagging.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.005 |
| Meta-epidemiology (narrow) | 0.002 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.004 | 0.002 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.001 | 0.003 |
| Open science | 0.002 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.003 | 0.005 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".