bread

bread is a shell script that enables crawling all websites.

Installation

Note

bread has a dependency on shot-scraper. Please install it in advance.

curl -fsSL https://raw.githubusercontent.com/chebread/bread/main/main.sh -o bread
chmod +x bread

Usage

# Basic usage (auto mode)
bread <URL>

# Use specific scraping engines
bread --shot-scraper <URL>
bread --jina <URL>

# Process multiple URLs
bread <URL1> <URL2> <URL3>

# Read URLs from standard input (stdin)
echo "<URL>" | bread
cat url_list.txt | bread

LICENSE

This project is licensed under the MIT License.