A Content Analysis of Google Scholar: Coverage Varies by Discipline and by Database
Bibliographic record
Abstract
Objective – To ascertain the coverage by discipline, publication date, publication language, and upload frequency of the scholarly articles found in Google Scholar. Design – Comparative content analyses. Setting – Electronic information resources accessible via the internet (both freely accessible and for-fee databases). Subjects – Forty-seven online databases and Google Scholar. Methods – The study compared the content of 47 databases (21 Internet resources freely available to the general public; 26 restricted-access databases) covering a variety of subjects with the content of Google Scholar. Each database was assigned to one of the following discipline categories: business, education, humanities, science and medicine, social science, and multidisciplinary. From April through July 2005, researchers generated random samples of 50 article titles from each of the 47 databases and searched the titles on Google Scholar to determine inclusion. Related studies were conducted for publication date and publication language analysis, and for the Google Scholar upload frequency study. For the publication date study, random samples from one database (PsycINFO) with a high degree of variability in Google Scholar coverage were searched for 1990, 2000, and 2004. For the publication language study, Google Scholar coverage of PsycINFO articles in English was compared to coverage of PsycINFO articles published in non-English languages. For the upload frequency study, two databases chosen for their high degree of coverage (BioMed Central and PubMed) were monitored to determine how often the new content was uploaded to Google Scholar. Main Results – This study revealed that content covered by Google Scholar varies greatly from database to database and from discipline to discipline. Of the 47 databases studied, coverage ranged from 6% to 100%. Mean and median values of coverage for all databases were both 60%. The mean discipline category scores varied from the humanities databases at 10% coverage, to the social sciences and education at 39% and 41% respectively, to science and medicine databases at 76% coverage. Mean coverage was 77% for the multidisciplinary databases. Mean coverage of open access journal databases was 95%, freely accessible databases had 84% mean coverage, and single publisher databases had 83% mean coverage. The publication language study found a bias towards English language publications. As well, a publication date bias was found – coverage of earlier dates was not as thorough as coverage of more recent publications. In the upload frequency study, for BioMed Central and PubMed there appears to be an approximately 15-week delay in the uploading of new material to Google Scholar. Conclusions – The results of this study serve to alert researchers and information professionals that Google Scholar (in beta test mode at the time of the study) has poor coverage in certain areas. To those with access to commercial databases, this serves as a cautionary tale. To those with a dearth of commercial databases, Google Scholar is a welcome site and can provide at least some information. The researchers state that the search engine itself could make future content studies unnecessary if it decides to make its content collection methodology transparent to users. Upload frequency, Google Scholar’s linking services, the advanced search option, and the “cited by” feature could all be subjects of future studies. For its first year in operation, Google Scholar offers a broad range of discipline coverage with substantial depth in some areas. At the time of the study, Google Scholar was working with libraries and vendors to connect search results to library-licensed full text.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.047 | 0.294 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.002 | 0.002 |
| Bibliometrics | 0.066 | 0.089 |
| Science and technology studies | 0.001 | 0.002 |
| Scholarly communication | 0.006 | 0.006 |
| Open science | 0.002 | 0.005 |
| Research integrity | 0.001 | 0.000 |
| Insufficient payload (model declined to judge) | 0.003 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".