Assessing the Effectiveness of AI Tools (Elicit, SciSpace, and Consensus) in Literature Review and Research
Bibliographic record
Abstract
The authors share the sentiments of other researchers that conducting a literature review and creating a matrix is a cumbersome task. The development of research tools that benefit researchers in their study conduct is always interesting and welcome. Thus, the objectives of this study are: (1) To evaluate the features and functionalities of Elicit, SciSpace, and Consensus in facilitating literature review and research and recommend the applicability of each tool for a type of research; (2) Assess and identify the strengths of these tools and align the impact with research workflows and efficiency; and (3) Utilize documented user feedback and opinions to ensure the appropriate AI tool is available for the researchers. This study adopts a mixed-methods approach. Firstly, a systematic literature review is conducted to gather relevant studies, user feedback, and expert opinions on the AI tools. These same studies will be used as the sample variables for this study. Secondly, a comparative evaluation of each AI tool against the original document and each other is conducted to assess each tool against the established evaluation criteria, which include search capabilities, document retrieval, summarization accuracy, citation analysis, and integration with existing research workflows. Lastly, the strengths and weaknesses of each tool are identified in relation to the criteria, and the effectiveness of the AI tool is determined based on the original content of the sample material. The findings of this study include (a) identified AI tools, such as Elicit, SciSpace, and Consensus, which offer valuable contributions to the research community through productivity-boosting features. (b) Each tool has strengths and weaknesses in search capabilities, document retrieval, summarization accuracy, citation analysis, and integration with existing research workflows; and (c) Documented user feedback indicates positive experiences with the usability and effectiveness of the tools, highlighting their potential to enhance research workflows. This study acknowledges potential limitations, including the reliance on user feedback and the subjective nature of user experiences. The evaluation is based on a specific set of criteria, and the results may vary depending on individual research needs and preferences. This study offers practical implications for researchers, students, and professionals seeking efficient and effective tools for conducting literature reviews. Elicit, SciSpace, and Consensus offer insights into their strengths, weaknesses, and potential applications. The findings contribute to informed decision-making regarding the adoption and utilization of these AI tools in research. This study examines AI tools designed explicitly for conducting literature reviews, distinguishing itself from existing research that predominantly emphasizes the application of AI in academic research settings. It offers an analysis of how various AI features can enhance the literature review process, thereby contributing a unique perspective to the ongoing discourse on the integration of AI in research methodologies.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.611 | 0.789 |
| Meta-epidemiology (narrow) | 0.002 | 0.003 |
| Meta-epidemiology (broad) | 0.003 | 0.005 |
| Bibliometrics | 0.074 | 0.048 |
| Science and technology studies | 0.006 | 0.007 |
| Scholarly communication | 0.021 | 0.028 |
| Open science | 0.005 | 0.021 |
| Research integrity | 0.004 | 0.004 |
| Insufficient payload (model declined to judge) | 0.004 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".