SlopTotal is a project that describes itself as the "VirusTotal of AI text," offering a service that incorporates 23 AI text detectors to analyze texts locally on a device. However, the docker run command in its README fails on Mac devices with Apple Silicon, requiring the use of docker compose instead. The project's effectiveness has been tested by running 36 diverse texts through the service, including 24 texts generated by AI models such as Claude, ChatGPT, and Gemini, and 12 human-written texts.
Out of the 12 human-written texts, 4 were incorrectly identified as AI-generated, representing a 33% error rate. Conversely, 9 out of the 24 AI-generated texts were incorrectly identified as human work, a 38% error rate. The README claims better results, but the user notes that the sample tested was small and may not be representative of all text types, particularly news or books in English.
The user also references previous instances where AI detection systems have been found unreliable. For example, OpenAI removed its own classifier in July 2023 due to low accuracy, and Vanderbilt disabled Turnitin's AI detector the following month. Stanford researchers tested 7 detectors in 2023 on 91 human TOEFL writings by non-English speakers, with an average of 61% being incorrectly identified as AI-generated.
Recent studies on commercial detectors, such as those conducted by the NBER and Jonathan Karr and his colleagues, found varying results. The NBER study found near-zero false positive rates with Pangram, 0.1 to 0.3% with OriginalityAI, and about 0.7% with GPTZero on 1,992 human texts from before 2020. Karr's study found no AI positives when testing with Pangram and GPTZero on summaries from 2013 to 2015. However, the studies differ on humanization, with the NBER finding that Pangram holds up against the humanizer StealthGPT, while less than 4% of AI texts processed by Undetectable AI remain flagged by Karr. Karr also notes that a human-written text edited by an AI can be flagged as AI in 38 to 80% of cases.
The user concludes that while SlopTotal can be used to form an opinion, it should not be the sole evidence for accusations. They note that human texts modified or reorganized by an LLM can be flagged as AI-generated, even if they were originally written or dictated by humans.
Evaluation of SlopTotal AI Text Detection Service Reveals Inconsistencies
AI-rewritten from original reportingHow it works
aitextdetectionslop-totalai-errorreliabilityai-tools
Original sources:
- 🇫🇷Korben



