Empowering users to scrape the products data from John's Test Store website.
Overview • Prerequisites • Architecture • Demo • Support • License
The primary goal of this project revolves around the retrieval of comprehensive products data from the John's Test Store website and analyze it.
The project repository exhibits the following structure:
Test-Store-Data-Analysis/
├── ⚙️.env
├── 📜.gitignore
├── ⚙️.pre-commit-config.yaml
├── 🔑LICENSE
├── 🐍main.py
├── 🔒poetry.lock
├── 📇pyproject.toml
├── 📝README.md
├── 🗒️requirements.txt
├── 🐍setup.py
├── 🐍template.py
├── 📁.github
│ └── 📂workflows
│ └── 📃actions.yaml
├── 📁conf
│ └── 📃configs.yaml
├── 📁data
│ ├── 📂external
│ │ ├── 📑products_link.csv
│ │ └── 📑scraped_products.csv
│ └── 📂processed
│ └── 📑products.csv
├── 📁images
│ └── 🖼️topmate_featured.png
├── 📁logs
│ └── 🧾2024_02_04_02_44_21_PM.log
├── 📁notebooks
│ ├── 📙01_web_scraping_tests.ipynb
│ └── 📙02_data_preprocessing.ipynb
├── 📁reports
│ └── .gitkeep
└── 📁src
├── 🐍constants.py
├── 🐍exception.py
├── 🐍logger.py
├── 🐍__init__.py
├── 📂components
│ ├── 🐍data_preprocessor.py
│ ├── 🐍link_extraction.py
│ └── 🐍product_scraper.py
├── 📂pipelines
│ ├── 🐍stage_01_data_extraction.py
│ └── 🐍stage_02_data_preprocessor.py
└── 📂utils
└── 🐍basic_utils.py
To fully grasp the concepts and processes involved in this project, it is recommended to have a solid understanding of the following skills:
- Fundamental knowledge of Python & Modular coding
- Familiarity with the Python libraries listed in the 🗒️requirements.txt file
- Basic familiarity with data analytics and Power BI
Having these skills as a foundation will help to ensure a smooth and effective experience while working on this project.
The selection of applications and their installation process may differ depending on personal preferences and computer configurations.
[CONTENT TO BE ADDED]
[CONTENT TO BE ADDED]
If you have any questions, concerns, or suggestions, feel free to reach out to me through any of the following channels:
If you find my work valuable, you can show your appreciation by buying me a coffee
This license allows reusers to distribute, remix, adapt, and build upon the material in any medium or format for noncommercial purposes only, and only so long as attribution is given to the creator. If you remix, adapt, or build upon the material, you must license the modified material under identical terms.
