Academic Libraries’ Citation Guides to ChatGPT Show Mixed Levels of Accuracy and Currency
Bibliographic record
Abstract
A Review of: Moulaison-Sandy, H. (2023). What is a person? Emerging interpretations of AI authorship and attribution. Proceedings of the Association for Information Science & Technology, 60(1), 279–290. https://doi.org/10.1002/pra2.788 Objective – To examine how and which academic libraries are responding to emerging guidelines on citing ChatGPT in the American Psychological Association (APA) style through guidance published on the libraries’ websites. Design – Analysis of search results and webpage content. Setting – Websites of academic libraries in the United States. Subjects – Library webpages addressing how ChatGPT should be cited in APA format. Methods – Google search results for academic library webpages providing guidance on citing ChatGPT in APA format were retrieved on a weekly basis using the query “chatgpt apa citation site:.edu” over a six-week period that covered the weeks before and immediately after the APA issued official guidance for citing ChatGPT. The first three pages of relevant search results were coded in MAXQDA and analyzed to determine the type of institution, using the Carnegie Classification and membership in the Association of American Universities (AAU). As this was a period during which APA style recommendations for citing ChatGPT were shifting, the accuracy of the library webpage content was also assessed and tracked across the studied time period. Main Results – During the six-week period, the number of library webpages with guidance for citing ChatGPT in APA format increased. Although doctoral universities accounted for the largest number of webpages each week, baccalaureate colleges, baccalaureate/associate’s colleges, and associates’ colleges were also well-represented in the search results. Institutions belonging to the AAU were represented by a relatively small number throughout the study. Over half of the pages made some mention of APA’s recommendations being interim or evolving, though the exact number fluctuated throughout the period. Prior to the collection period, APA had revised its initial recommendations to cite ChatGPT as a webpage or as personal communication, but 40% to 60% of library webpages continued to offer this outdated guidance. Of the library webpages, 13% to 40% provided verbatim guidance from ChatGPT responses on how it should be cited. The final two weeks of the collection period occurred after April 7, 2023, when APA had published official recommendations for citing ChatGPT. In the week following this change, none of the webpages in the first three pages of results had been updated to fully capture the new recommendations. The study analyzed the nine webpages appearing in the first page of results for the second week after APA’s official recommendations were published, showing that three linked to the APA’s blog, zero provided further explanation on how to apply the recommendations, five included outdated guidance, and three gave guidance from ChatGPT’s responses to questions on how it should be cited. Conclusion – The author sees the results of the study as reflecting three interrelated components: a new technology, gaps in librarians’ knowledge related to large language models (LLMs) and how they are currently being discussed in terms of authorship, and Google’s inability to rank the results in a way that prioritizes correct information. The substantial presence of institutions serving undergraduates leads the author to conclude that this is the population most in need of guidance for citing ChatGPT and the responsiveness on the part of the librarians shows an understanding of this need, even if the guidance itself is inaccurate.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.054 | 0.348 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.001 |
| Bibliometrics | 0.057 | 0.063 |
| Science and technology studies | 0.004 | 0.003 |
| Scholarly communication | 0.014 | 0.013 |
| Open science | 0.003 | 0.006 |
| Research integrity | 0.002 | 0.002 |
| Insufficient payload (model declined to judge) | 0.013 | 0.010 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".