Are the AI detectors still effective?

Since the launch of AI tools like ChatGPT, the term “AI” has become a buzzword in industries ranging from education to content creation. Writers and students now face a new challenge defending their work, whether it is AI or human content. 

According to statistics by Stastia last year, the term “AI” was searched more than 30.4 million times in July 2023, which is triple the number in the month before. As AI became more prevalent, teachers and editors had to find an alternative way to determine whether the text presented to them was AI or not. This led to the rise of AI detectors.

However, many writers and educators have questioned the authenticity of AI. One of them is  Christoper Pen, the co-founder of TrustInsights.ai, who stated in a LinkedIn post that “AI detectors” are a joke. Pen is not the only one who thinks that AI detectors are not reliable. Dianna Mason, SEO content specialist, stated in a post, “AI content detectors don’t work.”

According to another post on Reddit, one user stated, “Yes, AI detectors are a scam. There has never been an academic paper proving that an AI detector can work reliably. There also isn’t a promising theoretical framework for how a reliable AI detector would work. “

Several well-publicized examples show the inefficiency of AI detectors. For instance, one detector misidentified 97.75% of the U.S. Declaration of Independence as AI-composed. 

In another instance, texts from the Bible were found to be AI-written. The Bhagavad Gita, a Hindu scripture, was also misidentified by the AI. Such incidents raise very serious questions about the credibility of AI detectors.

To further investigate the effectiveness of AI detectors, we tested an excerpt from the Bible with four different detectors. The results were quite different

• Quilbott- According to this site, the excerpt used was likely 0%, not AI-generated.
• Grammaly- Grammaly also identified the text as 100% human-written.
• Gpt Zero- This site gave the text a 96% probability of having been written by Humans.
• Zero Gpt- This was the last site i tested and the results were quite mind blowing, according to the site, the text was 96% AI written.

How AI Detectors Work

AI detectors, or GPT detectors, are systems that are designed to identify content generated by AI. These detectors compare text in terms of patterns and word arrangement, just like LLMs do with text.  LLMs work with probabilities that are inferred from data and base their predictions purely on sequence detection of words. The AI detectors operate on these probabilities to detect AI content. 

AI detectors mainly use two measures, namely perplexity and burstiness. Perplexity measures the probability that the next word will be correct, and burstiness is the variability of the number of words and phrases. While human writing tends to exhibit more variety, AI-generated text is often more uniform. Despite this, as the likes of GPT-4 AI models advance, they can replicate human writing with higher proficiency, thus making detection even harder. 

According to Jodie Cook , the founder of Coachvox, “Right now we laugh at AI-generated LinkedIn comments…. But as the technology improves, we won’t have a clue what’s real and not real. No one will. “

For writers, students, and businesses, the stakes are high. This begs the question can AI detectors be trusted?