mc_89_add_scraping_logic_for_non_pending_urls

[maxachis]

#89

This PR adds scraping logic for non-pending URLs. The URL HTML task has been updated to scrape non-pending URLs as well, storing their compressed HTML data, as well as other information from the scrape, in the database.

To ensure better information on scrapes, and to avoid disrupting the status of existing URLs, several other changes were performed:

• Addition of a url_web_metadata table, which provides information on the HTTP Status of the link when accessed, as well as the content type returned (which is in turn used in scraping).
• Addition of a url_scrape_info table, which describes the outcome of a scrape