First Major Release v1.0.0

[indrajithi]

This is a place holder Issue for the first major release v1.0.0

Please feel free to create issue from this list

Scope and Features: First major version v1.0.0

Functional Requirements

• Basic Crawling Functionality Feature: Basic features and package the project #1
• Configurable options for maximum links to crawl Feature: Basic features and package the project #1
• Handle both relative and absolute URLs Feature: Basic features and package the project #1
• Save crawl results to a specified file Feature: Basic features and package the project #1
• Configurable verbosity levels for logging add support for concurrent workers, custom delay and optional verbose #7
• Concurrency and custom delay add support for concurrent workers, custom delay and optional verbose #7
• Support Regular expression Url regex matching #16
• Crawl internal / external links only Feature: Support flag to crawl only the root website. Do not hop to external links #11
• Return optional html in response Added option to include page body in crawl results #19
• Crawl depth per website/domain Crawl depth per domain #37
• Logging Implement logging #38
• Retry mechanism for transient errors Implement a retry mechanism for transient errors #39
• Support Javascript heavy dynamic websites Feature: Support for crawling dynamic javascript heavy site #10
• (Optional) Respect Robots.txt Respect robots.txt when crawling when set as True #42
• (Optional) User-Agent Customization
• (Optional) Proxy support
• (Optional) Use Asynchronous I/O
• (Optional) Crawl output to database (Mongo mabye)

Non Functional Requirements

• Git workflow for CI/CD python publish workflow #4
• Documentation (API and Developer) Docs: Auto generate documentation #18
• Test coverage above 80% Test coverage above 80% #28
• Git hooks Add mypy and pytest to pre-commit #22
• Modular and Extensible Architecture Housekeeping: Refactor the code base to a more Modular and Extensible Architecture #17
• (Optional) Memory Benchmark: Monitor Monitor memory usage during the crawling process
• (optional) Security considerations (e.g., handling of malicious content)

[Mews]

You forgot to check "Return optional html in response #19" ;)

[Mews]

@indrajithi On Git hooks maybe you should link my second pr on that feature (#25 ) so people also see the pre-commit install --hook-type pre-push command :)

[Mews]

@indrajithi you can check "Test coverage above 80%" now ;)