Does AdaptivePlaywrightCrawler always invokes Playwright browser?

[danielvanmilgx]

Hello,

I'm trying to scrape some websites and determine what AdaptivePlaywrightCrawler strategy is used to scrape the website.
Therefor I have implemented a request handler like:

    @crawler.router.default_handler
    async def request_handler(context: AdaptivePlaywrightCrawlingContext) -> None:

And check the context.page in the handler to determine what strategy is used.

It looks like the handler is always invoked twice: one time for the Playwright client strategy and one time for the static strategy.

So, is it a fact Crawlee AdaptivePlaywrightCrawler always tries the "client" first to determine what strategy to use next for sub levels and/or similar pages?

How can I finally determine what strategy AdaptivePlaywrightCrawler has or will use for the crawl as I have to generate statistics about the amount of websites that need the client vs the static strategy.

Thanks!

[Pijukatel]

Hello,

you can inspect the Crawler statistics state. In case of AdaptivePlaywrightCrawler it will be AdaptivePlaywrightCrawlerStatisticState and it has fields like browser_request_handler_runs or http_only_request_handler_runs

...
crawler.log.info(crawler.statistics.state)

So, is it a fact Crawlee AdaptivePlaywrightCrawler always tries the "client" first to determine what strategy to use next for sub levels and/or similar pages?

Initially, it will run both to compare results. It will gradually lean towards one of the approaches for a group of similar urls and it will, from time to time, recheck the assumption with both.

To further understand how the prediction works, you can check https://crawlee.dev/python/docs/guides/adaptive-playwright-crawler#prediction-related-arguments

[danielvanmilgx]

Thanks for the quick reply @janbuchar Pijukatel!

• But question remains: the stats mention http_only_request_handler_runs=0 browser_request_handler_runs=1 for a simple non SPA website. So for the first request(s) always Playwright is used and so the statistics are not very useful?

[Pijukatel]

Yes, the freshly started AdaptivePlaywrightCrawler will try both on the first request. It is learning by comparing the results of both; if they are the same, no need to use the browser. The statistics become useful if you let it run for a while.