Bots have surpassed humans at solving CAPTCHAs.
CAPTCHA Type: reCAPTCHA (click)
Human Time: 3.1-4.9 seconds
Human Accuracy: 71-85%
Bot Time: 1.4
Bot Accuracy: 100%
CAPTCHA Type: Geetest
Human Time: 28-30
Human Accuracy: N/A
Bot Time: 5.3
Bot Accuracy: 96%
CAPTCHA Type: Arkose
Human Time: 18-42
Human Accuracy: N/A
Bot Time: N/A
Bot Accuracy: N/A
CAPTCHA Type: Distorted Text
Human Time: 9-15.3
Human Accuracy: 50-84%
Bot Time: <1
Bot Accuracy: 99.8%
CAPTCHA Type: reCAPTCHA (image)
Human Time: 15-26
Human Accuracy: 81%
Bot Time: 17.5
Bot Accuracy: 85%
CAPTCHA Type: hCAPTCHA
Human Time: 18-32
Human Accuracy: 71-81%
Bot Time: 14.9
Bot Accuracy: 98%
Quoting from the paper about how the human scores were calculated.
"To understand the landscape of modern CAPTCHAS and guide the design of the subsequent user study, we manually inspected the 200 most popular websites from the Alexa Top Website list."
"Our goal was to imitate a normal user's web experience and trigger CAPTCHAS in a natural setting. Although CAPTCHAS can be used to protect any section or action on a website,
they are often encountered during user account creation to prevent bots creating accounts."
"The most prevalent types were: reCAPTCHA was the most prevalent, appearing on 68 websites (34% of the inspected websites)."
"Slider-based CAPTCHAS appeared on 14 websites (7%)."
"Distorted Text CAPTCHAS appeared on 14 websites (7%)."
"Game-based CAPTCHAS appeared on 9 websites (4.5%)."
"hCAPTCHA appeared on 1 website."
"Other CAPTCHAs found during our inspection included: a CAPTCHA resembling a scratch-off lottery ticket; a CAPTCHA asking users to locate Chinese characters within an image;
and a proprietary CAPTCHA service called 'NuCaptcha'."
"Having identified the relevant CAPTCHA types, we conducted a 1,000 participant online user study to evaluate real users' solving times and preferences for these types of CAPTCHAS. Our study was run using using Amazon MTurk."
They have a table where they list out the ages, countries, education, gender, device type, input method, and internet use. They seem to have a sufficiently broad spectrum of humans.
Ages: 30-39: 531, 20-29: 403, 40-49: 271, 50-59: 106, 60+: 58, 18-19: 31. Countries: USA 985, India: 240, Brazil: 50, Italy: 27, UK: 24, Other: 74. Education: Bachelors: 822, Masters: 243, high school: 210, Associates: 98, PhD: 24, no degree: 3. Gender: male: 832, female: 557, nonbinary: 11. Device type: computer: 1301, phone: 74, tablet: 25. Input method: keyboard 1261, touch: 125, other: 14. Internet use: work: 860, web surf: 397, education: 87, gaming: 30, other: 26.
"Direct setting: This setting was designed to match previous CAPTCHA user studies, in which participants are directly asked to solve CAPTCHAS."
"Contextualized setting: This setting was designed to measure CAPTCHA solving behavior in the context of a typical web activity."
"For reCAPTCHA, the selection between image- or click based tasks is made dynamically by Google. Whilst we know that 85% and 71% of participants (easy and hard setting) were shown a click-based CAPTCHA, the exact task-to-participant mapping is not revealed to website operators. We therefore assume that the slowest solving times correspond to imagebased tasks. After disambiguation, click-based reCAPTCHA had the lowest median solving time at 3.7 seconds. Curiously, there was little difference between easy and difficult settings."
"The next lowest median solving times were for distorted text CAPTCHAS. As expected, simple distorted text CAPTCHAS were solved the fastest. Masked and moving versions had very similar solving times. For hCAPTCHA, there is a clear distinction between easy and difficult settings."
"The latter consistently served either a harder image-based task or increased the number of rounds. However, for both hCAPTCHA settings, the fastest solving times are similar to those of reCAPTCHA and distorted text. Finally, the gamebased and slider-based CAPTCHAS generally yielded higher median solving times, though some participants still solved
these relatively quickly (e.g., < 10 seconds)."
"With the exception of reCAPTCHA (click) and distorted text, we observed that solving times for other types have a relatively high variance."
"reCAPTCHA: The accuracy of image classification was 81% and 81.7% on the easy and hard settings respectively. Surprisingly, the difficulty appeared not to impact accuracy."
"hCAPTCHA: The accuracy was 81.4% and 70.6% on the easy and hard settings respectively. This shows that, unlike reCAPTCHA, the difficulty has a direct impact on accuracy."
"Distorted Text: We evaluated agreement among participants as a proxy for accuracy."
If you're wondering what AI systems they used to crack the CAPTCHAs... well... They didn't actually run any AI systems on the CAPTCHAs. They scoured "the literature" for AI performance scores. And they didn't provide a convenient table listing the sources for all their numbers on the AI performance. They have a section of references, of course, but there are 77 references. The paper focuses totally on the human testing and demographic breakdowns of it.
