feat(etl): Script to export github data into bigquery

[dklawren]

No description provided.

[Copilot]

Pull request overview

This PR adds a comprehensive GitHub ETL pipeline that extracts pull request data from GitHub repositories and loads it into Google BigQuery. The implementation uses a streaming/chunked architecture to process data in batches of 100 PRs, includes rate limit handling, and provides local testing capabilities through mock services.

• Implements chunked extraction, transformation, and loading of GitHub PR data to BigQuery
• Adds mock GitHub API server and BigQuery emulator for local testing without rate limits
• Includes comprehensive documentation with setup instructions and schema definitions

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 19 comments.

Show a summary per file

File	Description
requirements.txt	Adds BigQuery client library and testing dependencies (pytest, pytest-mock, pytest-cov)
mock_github_api.py	Creates Flask-based mock GitHub API server that generates 250 sample PRs with commits, comments, and reviewers for testing
main.py	Implements the core ETL pipeline with chunked processing: extraction from GitHub API with pagination/rate limiting, data transformation to BigQuery format, and insertion using BigQuery client
docker-compose.yml	Orchestrates three services: mock GitHub API, BigQuery emulator, and the ETL service for local development
data.yml	Defines BigQuery table schemas for pull_requests, commits, reviewers, and comments tables used by the emulator
README.md	Provides comprehensive documentation including setup, architecture, authentication methods, and usage examples
Dockerfile.mock	Creates container image for the mock GitHub API service using Python 3.11 and Flask

Comments suppressed due to low confidence (1)

main.py:18

• Import of 'pprint' is not used.

import pprint

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 7 changed files in this pull request and generated 15 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

@dklawren I've opened a new pull request, #3, to work on those changes. Once the pull request is ready, I'll request review from you.

[Copilot]

@dklawren I've opened a new pull request, #4, to work on those changes. Once the pull request is ready, I'll request review from you.