JarCastingCloudflare-IUAM-Solver
A simple "Pure Java" library and cli tool to breaking through the Cloudflare's anti-bot mechanism (a.k.a "I'm Under Attack Mode", or IUAM), implemented with HTMLUnit.
Prerequisites
- JDK 11
CLI Tool
Install
$ curl -LO https://github.com/ninja-beans/cloudflare-iuam-solver/releases/download/0.1.0/cfis
$ chmod +x cfis
Usage
Print a cookie string.
$ ./cfis -c https://www.example.com
cf_clearance=XXXXXXXXXXXXXXXXXXXX-XXXXXXXXXX-X-XXX;__cfduid=XXXXXXXXXXXXXXXXXXXX;
Download a html content with curl.
$ ./cfis -c > cookie.txt
$ ./cfis -u > ua.txt
$ curl -s --cookie "$(cat cookie.txt)" -A "$(cat ua.txt)" https://www.example.com/
Extract all images with curl and xmllint.
$ eval $(./cfis --curl https://www.example.com/) | xmllint --xpath "//img" --html - 2> /dev/null
Java Library
Install
<dependency>
<groupId>com.ninja-beans.crawler</groupId>
<artifactId>cloudflare-iuam-solver-parent</artifactId>
<version>0.1.0</version>
<type>pom</type>
</dependency>
Usage
Scraping with Java 11 HttpClient and Jsoup.
public class App {
public static void main(final String[] args) throws IOException, InterruptedException {
var url = args[0];
var result = IuamSolver.solve(url);
// 1. Create HttpClient
var client = HttpClient
.newBuilder()
.version(Version.HTTP_1_1)
.followRedirects(Redirect.NORMAL)
.cookieHandler(result.getCookieManager()).build();
// 2. Send the request and get the response
var request = HttpRequest.newBuilder().header("Accept", "*/*")
.header("User-Agent", result.getResponse().getUserAgent())
.GET()
.uri(URI.create(url))
.build();
var response = client.send(request, BodyHandlers.ofString(StandardCharsets.UTF_8));
// 3. Parse the response
var doc = Jsoup.parse(response.body(), url);
var elm = doc.getElementById("title");
System.out.println(doc.title());
System.out.println(elm.html());
}
}