An Evaluation of Methods to Assess Team Research Consultations
Bibliographic record
Abstract
Abstract Objective – Due to the individualized nature of consultations and institutional constraints, research consultations can be challenging to assess. At Texas A&M University Libraries, subject librarians use research consultations to teach information literacy to upper-division engineering student teams working on a technical paper project. This paper describes an action research project designed to evaluate which assessment method for consultations with student teams would provide the most actionable data about the instruction and the consultation logistics as well as optimize librarian time. Methods – For three semesters, we simultaneously used up to four consultation assessment methods: one-minute papers, team process interviews, retrospective interviews, and questionnaires. We followed the action research cycle to plan the assessments, implement the assessments, reflect on the data collected and our experiences implementing the assessments, and revise the assessments for the next semester. Each assessment method was distributed to students enrolled in an engineering course at a different point in the technical paper project. The one-minute paper was given immediately after the consultation. The team process interviews occurred after project deliverables. The questionnaire was distributed in-person on the last day of class. Focus groups were planned for after the assignment was completed, but low participation meant that instead of focus groups we conducted retrospective interviews. We used three criteria to compare the assessments: information provided related to the effectiveness of the instruction, information provided about the logistics of the consultation, and suitability as an assessment method in our context. After comparing the results of the assessment methods and reflecting on our experiences implementing the assessments, we modified the consultation and the assessment methods for the next semester. Results – Each assessment method had strengths and weaknesses. The one-minute papers provided the best responses about the effectiveness of the instruction when questions were framed positively, but required the most staff buy-in to distribute. The team process interviews were time intensive, but provided an essential understanding of how students think about and prepare for each progress report. Recruiting for and scheduling the focus groups required more time and effort than the data collected about the instruction and logistics warranted. The questionnaire provided student perspectives about their learning after the assignment had been completed, collected feedback about the logistics of the consultations, was easy to modify each semester, and required minimal librarian time. Conclusion – Utilizing multiple assessment methods at the same time allowed us to determine what would work best in our context. The questionnaire, which allowed us to collect data on the instruction and consultation logistics, was the most suitable assessment method for us. The description of our assessment methods and our findings can assist other libraries with planning and implementing consultation assessment.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.193 | 0.381 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.007 | 0.004 |
| Science and technology studies | 0.004 | 0.003 |
| Scholarly communication | 0.007 | 0.006 |
| Open science | 0.005 | 0.010 |
| Research integrity | 0.003 | 0.002 |
| Insufficient payload (model declined to judge) | 0.005 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".