AI Detection Accuracy in Academia
Navigating the treacherous waters of academic honesty has become an increasingly daunting task. But it isn’t a new one. And tools to help you detect academic dishonesty have been around long enough that there are some proverbial 800-pound-gorillas in the room.
You’re likely familiar with Turnitin, a well established tool for detecting traditional plagiarism. Recently, it stepped into the arena of AI detection as well with the promise of being a one-stop solution now also detecting AI-written content.
But how accurate is it, really? We decided to put it to the test:
We collected 600 documents known to be written by AI and 600 documents known to be written by humans. With this sample dataset of 1200 articles, we scanned each using both Turnitin’s AI detector and our detector, powered by Originality.AI. Curious to see what we’ve discovered? Keep reading.
When a Positive is a Negative
False positives are a significant concern in Academia, and rightly so. They lead to unnecessary confusion, and false negatives threaten the integrity of academic enforcement. Because of this, trusting your detection tool of choice should be a top priority when incorporating one into your grading workflow. So, let’s start by looking at when the detectors were positive a document was written by AI.
True Positives: Passed.AI vs. Turnitin
In our comprehensive test of 1200 documents, we assessed how well both tools performed.
While Turnitin did correctly identify 231 AI-written documents, it missed 369 AI-written documents, incorrectly attributing them as human-written documents. When tasked with the same documents, our Passed.AI detector correctly identified 578 of the 600 documents as AI-written, catching almost 2.5 times as many instances of academic dishonesty as Turnitin.
When it came to False Positives, Turnitin incorrectly identified 21 documents as AI-written when they were in fact human-written for a False Positive Rate of 3.5%. Our rate was slightly lower, at 2.5% (15/600). Both of those False Positive Rates are well below the “industry average” in our experience. Yet, 3 students in a class of 100 being falsely accused of submitting work that is not their own is still trouble and a good reminder of why secondary verification, as we offer with our Document Audits and Replay remain a crucial part of the desired solution.
False Negatives: The Hidden Threat
A false negative is an instance where potential academic dishonesty is missed. In this regard, Turnitin recorded 369 false negatives. That’s a significant number of papers where potential academic dishonesty was completely overlooked.
In contrast, Passed.AI’s false negative rate stands at a mere 22 instances. This low rate is a testament to the robustness of training our AI has undergone. Simply, Passed.AI is far less likely to let a dishonest paper slip through an initial AI scan unnoticed and when you add our secondary verification tools, you will have a much more accurate and certain position.
Conclusion: The Superiority of Passed.AI
Our findings clearly show that Passed.AI significantly outperforms Turnitin in detecting AI-written documents, catching nearly 2.5 times as many instances of academic dishonesty. Furthermore, Passed.AI’s lower rates of false positives and false negatives ensure a more accurate and reliable detection process.
While no system is perfect, the combination of our advanced AI detection and secondary verification tools offers a comprehensive and trustworthy solution for maintaining academic integrity in the digital age. With Passed.AI, you can be confident in your ability to uphold the highest standards of academic honesty.
Note: Are you from Academia and would like to run this test yourselves? Reach out to me and I’d be happy to provide you with the dataset.
