The article presents the findings of a study conducted by researchers from the Department of Computer Science, Lyle School of Engineering at Southern Methodist University. The study aimed to test the hypothesis that AI models would have an advantage in detecting their own content due to the unique artifacts generated by each model. The researchers tested three AI models – ChatGPT-3.5 by OpenAI, Bard by Google, and Claude by Anthropic.
The study involved creating a dataset of fifty different topics and prompting each AI model to generate essays on each topic, as well as a rewritten version of the original essays. The researchers used a zero-shot prompting method to self-detect the AI-generated content.
The results of the study revealed that Bard and ChatGPT were able to self-detect their own content at a similar rate, while ZeroGPT performed slightly worse. However, Claude, the AI model by Anthropic, was unable to detect its own content, indicating that it produced fewer detectable artifacts. This was unexpected, as it suggested that Claude’s output was of higher quality in terms of outputting fewer AI artifacts.
The study also tested the AI models’ ability to detect each other’s content, and found that Bard-generated content was the easiest to detect, while ChatGPT and Claude had difficulty detecting each other’s content.
One of the most interesting findings was that Claude was able to self-detect the paraphrased content, even though it was unable to detect the original essays. The researchers suggested that this may be due to the inner workings of the transformer models.
The researchers acknowledged that their sample size was small and did not claim that their results were definitive. They emphasized the need for further research on larger datasets with more diversity of AI-generated text, as well as testing of additional AI models and prompt engineering.
Overall, the study confirmed that detecting AI-generated content is not an easy task, and self-detection remains an interesting area for continued research. The full research paper and abstract are available for reference.
Read Full Article