[WIP] Add distributed job queue and worker processing system

[Copilot]

Distributed Job Queue & Worker Processing System Implementation Plan

Building a scalable, distributed background job processing system with Node.js, Redis, and BullMQ.

Implementation Checklist:

• Initialize Node.js project with package.json
• Install dependencies (Express, BullMQ, Redis, etc.)
• Create project structure (src/, config/, etc.)
• Implement Redis connection configuration
• Create job queue manager with BullMQ
• Implement REST API server for job submission
• Create worker process for job processing
• Add support for different job types (image processing, PDF generation, email, scraping, batch computation)
• Add environment configuration (.env.example)
• Create Docker Compose setup for Redis
• Add documentation (README with usage instructions)
• Add error handling and logging
• Test the complete system

Original prompt

Distributed Job Queue & Worker Processing System
This project is a scalable, distributed background job processing system built with Node.js, Redis, and BullMQ. It allows clients to submit tasks (jobs) through an API, and those jobs are processed asynchronously by independent worker processes.
The system is designed to handle CPU-intensive operations, such as image processing, PDF generation, sending emails, data scraping, or batch computation, without blocking the main application.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.