An Evaluation of Methods to Assess Team Research Consultations
Notice bibliographique
Résumé
Abstract Objective – Due to the individualized nature of consultations and institutional constraints, research consultations can be challenging to assess. At Texas A&M University Libraries, subject librarians use research consultations to teach information literacy to upper-division engineering student teams working on a technical paper project. This paper describes an action research project designed to evaluate which assessment method for consultations with student teams would provide the most actionable data about the instruction and the consultation logistics as well as optimize librarian time. Methods – For three semesters, we simultaneously used up to four consultation assessment methods: one-minute papers, team process interviews, retrospective interviews, and questionnaires. We followed the action research cycle to plan the assessments, implement the assessments, reflect on the data collected and our experiences implementing the assessments, and revise the assessments for the next semester. Each assessment method was distributed to students enrolled in an engineering course at a different point in the technical paper project. The one-minute paper was given immediately after the consultation. The team process interviews occurred after project deliverables. The questionnaire was distributed in-person on the last day of class. Focus groups were planned for after the assignment was completed, but low participation meant that instead of focus groups we conducted retrospective interviews. We used three criteria to compare the assessments: information provided related to the effectiveness of the instruction, information provided about the logistics of the consultation, and suitability as an assessment method in our context. After comparing the results of the assessment methods and reflecting on our experiences implementing the assessments, we modified the consultation and the assessment methods for the next semester. Results – Each assessment method had strengths and weaknesses. The one-minute papers provided the best responses about the effectiveness of the instruction when questions were framed positively, but required the most staff buy-in to distribute. The team process interviews were time intensive, but provided an essential understanding of how students think about and prepare for each progress report. Recruiting for and scheduling the focus groups required more time and effort than the data collected about the instruction and logistics warranted. The questionnaire provided student perspectives about their learning after the assignment had been completed, collected feedback about the logistics of the consultations, was easy to modify each semester, and required minimal librarian time. Conclusion – Utilizing multiple assessment methods at the same time allowed us to determine what would work best in our context. The questionnaire, which allowed us to collect data on the instruction and consultation logistics, was the most suitable assessment method for us. The description of our assessment methods and our findings can assist other libraries with planning and implementing consultation assessment.
Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.
Comment cette classification a été obtenuedéplier
Prédiction machine sur la base complète
Imitation des enseignantsNi prévalence calibrée, ni vérité terrain. Validation humaine à venir. Le volet Gemma est une étiquette directe du modèle pour chaque travail de la base, lue sur la notice réduite au titre. Le volet Codex est un classifieur appris des 10 348 étiquettes directes de Codex et calibré sur les taux pondérés de l'échantillon; les champs sans appui suffisant ne portent aucun appel Codex. Le mode candidate est l'union des deux volets; le consensus est leur intersection. Ces sorties portent le statut machine_predicted_unvalidated et ne sont pas des étiquettes humaines.
Scores du classifieur distillé par catégorie (deux têtes)
| Catégorie | Codex | Gemma |
|---|---|---|
| Métarecherche | 0,193 | 0,381 |
| Méta-épidémiologie (sens strict) | 0,002 | 0,001 |
| Méta-épidémiologie (sens large) | 0,001 | 0,002 |
| Bibliométrie | 0,007 | 0,004 |
| Études des sciences et des technologies | 0,004 | 0,003 |
| Communication savante | 0,007 | 0,006 |
| Science ouverte | 0,005 | 0,010 |
| Intégrité de la recherche | 0,003 | 0,002 |
| Charge utile insuffisante (le modèle a refusé de juger) | 0,005 | 0,002 |
Scores machine (provisoires)
Les deux têtes enseignantes du modèle étudiant, lues sur ce travail. Un score ordonne la base pour la relecture; il n'affirme jamais une catégorie, et le statut de validation accompagne chaque rangée tel quel.
Scores de référence d'un modèle non mature (critères de maturité non atteints, 7 itérations). Un score ordonne; il n'affirme jamais une catégorie.
score_only:v0-immature-baseline · tel quel depuis la passe de notation : score_only signifie que le nombre peut ordonner les travaux, et qu'aucune étiquette de catégorie n'en découleClassification
machine, non validéePrédiction automatique; l’étiquette directe de Gemma et le classifieur distillé Codex s’accordent sur ce qui est montré ici.
Le détail, modèle par modèle et score par score, se trouve en fin de page sous « Comment cette classification a été obtenue ».