Research report
Does Turnitin Flag Human-Written Research Papers from the PLOS Corpus as AI-generated?
Benchmark Experiment: Evaluating Turnitin AI Detection on Human-Written Research Abstracts
Overview
This report presents a benchmark experiment assessing the accuracy of Turnitin's AI detection tool on a dataset of 504 human-written research article abstracts from the PLOS Corpus, published in or before 2021. The dataset comprises 135,712 words across diverse domains and majors. The primary metric is word-level accuracy, defined as the percentage of words correctly identified as human-written (i.e., not flagged as AI-generated).
- Overall word-level accuracy is 100.0% across all 504 articles and 135,712 words.
- No words were incorrectly flagged as AI-generated; every word was correctly identified as human-written.
- Domain breakdown:
- Business: 1,291 words, 100.0% accuracy
- Humanities: 2,942 words, 100.0% accuracy
- Psychology: 10,870 words, 100.0% accuracy
- STEM: 120,609 words, 100.0% accuracy
- Major breakdown: All 18 majors (e.g., Biology, Computer Science, Economics, Physics) achieved 100.0% accuracy.
- Word count buckets: Abstracts in the 400-word bucket (503 articles, 134,954 words) and the 800-word bucket (1 article, 758 words) both showed 100.0% accuracy.
- Dataset: 504 human-written research article abstracts from the PLOS Corpus, published in 2021 or earlier.
- Total word count: 135,712 words (reference/bibliography excluded).
- Domains covered: Business, Humanities, Psychology, STEM.
- Majors covered: 18 majors including Biology, Biomedical Engineering, Chemistry, Computer Science, Economics, Physics, and others.
- Word count per abstract: 400–800 words.
- Procedure: Each abstract was submitted to Turnitin's AI detection tool. The tool's output was compared to the ground truth (all human-written). Word-level accuracy was calculated as the percentage of words correctly identified as human-written.
- Scoring unit: accuracy is measured in words. First identify correct paragraphs under the expectation below, then sum the words in those paragraphs for the correct-word count; total words are summed over all counted paragraphs.
- Correct paragraph: a counted paragraph that was not flagged as AI. Correct words are the words in those paragraphs.
- References excluded: from a References/Bibliography-style heading onward, those paragraphs are left out of word counts.
- Dataset scope: Only abstracts from the PLOS Corpus were tested; results may not generalize to full papers or other corpora.
- Temporal limitation: All articles were published in or before 2021; newer writing styles or AI-generated text patterns may differ.
Overall Results
- Total articles: 504
- Total words: 135,712
- Correct words: 135,712
- Word accuracy: 100.0%
- All words across all articles were correctly identified as human-written, with no false AI flags.
Results by Domain
- Business: 6 articles, 1,291 words, 100.0% accuracy
- Humanities: 11 articles, 2,942 words, 100.0% accuracy
- Psychology: 40 articles, 10,870 words, 100.0% accuracy
- STEM: 447 articles, 120,609 words, 100.0% accuracy
- Every domain achieved perfect word-level accuracy.
Results by Major
- Biology: 280 articles, 72,983 words, 100.0% accuracy
- Biomedical Engineering: 36 articles, 9,784 words, 100.0% accuracy
- Chemistry: 1 article, 309 words, 100.0% accuracy
- Civil Engineering: 1 article, 271 words, 100.0% accuracy
- Communications: 4 articles, 1,077 words, 100.0% accuracy
- Computer Science: 12 articles, 2,914 words, 100.0% accuracy
- Criminal Justice: 1 article, 162 words, 100.0% accuracy
- Data Science: 17 articles, 4,026 words, 100.0% accuracy
- Economics: 6 articles, 1,291 words, 100.0% accuracy
- Environmental Science: 15 articles, 3,915 words, 100.0% accuracy
- Health Sciences: 80 articles, 25,137 words, 100.0% accuracy
- History: 1 article, 285 words, 100.0% accuracy
- Mechanical Engineering: 1 article, 223 words, 100.0% accuracy
- Nursing: 2 articles, 543 words, 100.0% accuracy
- Physics: 2 articles, 504 words, 100.0% accuracy
- Political Science: 2 articles, 588 words, 100.0% accuracy
- Psychology: 40 articles, 10,870 words, 100.0% accuracy
- Sociology: 3 articles, 830 words, 100.0% accuracy
- All 18 majors achieved 100.0% word accuracy.
Results by Word Count Bucket
- 400-word bucket: 503 articles, 134,954 words, 100.0% accuracy
- 800-word bucket: 1 article, 758 words, 100.0% accuracy
- Both word count ranges showed perfect accuracy.