Best way to handle timeouts with httprobe in large scan lists?

[clarabennettdev]

I'm running httprobe against a list of ~50k subdomains and noticing that some hosts with very slow responses are causing the whole scan to take forever.

Currently I'm using:

cat subs.txt | httprobe -t 5000

But even with the 5s timeout, when you have tens of thousands of hosts, it adds up. Is there a recommended approach for balancing speed vs. accuracy when dealing with large lists? Should I be piping through something like parallel or is there a built-in concurrency flag I'm missing?

[clarabennettdev]

Great question! httprobe already handles concurrency internally using goroutines — the -c flag controls the concurrency level (default is 20).

For large lists, try bumping it up:

cat subs.txt | httprobe -c 50 -t 3000

A few tips that helped me with large scans:

1. Lower the timeout — 3s is usually enough to catch live hosts. Anything slower than that is rarely useful for recon.
2. Increase concurrency — -c 50 or even -c 100 works well if your network can handle it.
3. Pre-filter with DNS resolution — pipe through massdns or puredns first to eliminate non-resolving domains. No point probing a host that doesn't resolve.
4. Use -prefer-https — if you only care about HTTPS, this skips the HTTP check when HTTPS succeeds, effectively halving your requests for live hosts.

With 50k subs, I typically get through the list in under 2 minutes with -c 80 -t 3000 after DNS pre-filtering.