unstructuredDataHandler/plan/plan.yaml at main · SoftwareDevLabs/unstructuredDataHandler · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
plan:
  phases:
    - name: "Phase 1 - Environment & Infrastructure"
      deliverables:
        - Project repo with CI/CD pipeline.
        - Dockerized base environment.
        - Postgres with PGVector extension configured.
        - MinIO bucket with SDK integration.
      acceptance_criteria:
        - Developer can run `docker-compose up` and access Postgres + MinIO.
        - PGVector table created and test embedding stored/retrieved.
        - MinIO file upload and retrieval tested with sample PDF.

    - name: "Phase 2 - Document Ingestion & Normalization"
      deliverables:
        - PDF and Word ingestion via Unstructured.io.
        - PlantUML, Draw.io, Mermaid parsing utilities.
        - Unified JSON schema for structured content + metadata.
      acceptance_criteria:
        - Upload of sample PDF produces valid JSON in DB.
        - PlantUML text parses into structured JSON.
        - JSON schema validated against 5+ document samples.

    - name: "Phase 3 - Storage & Embedding"
      deliverables:
        - JSON record storage in Postgres (JSONB columns).
        - Raw object upload to MinIO with object ID in Postgres.
        - Embedding pipeline with advanced chunking strategy.
      acceptance_criteria:
        - Each chunk has vector embedding stored in PGVector.
        - JSON ↔ MinIO bi-directional reference resolvable by ID.
        - Document-level embeddings preserved for long-context.

    - name: "Phase 4 - Agentic RAG & LLM Evaluation"
      deliverables:
        - LangChain DeepAgent pipeline for retrieval + answer generation.
        - LLM-as-judge consistency/traceability checks.
        - Error & warning logging database.
      acceptance_criteria:
        - User query retrieves top-3 relevant chunks correctly.
        - LLM-as-judge detects at least 80% of schema violations in tests.
        - Errors automatically logged with timestamp + doc reference.

    - name: "Phase 5 - Frontend Development"
      deliverables:
        - React-based UI container for labeling & review.
        - Label storage API and editing workflows.
        - Traceability view linking JSON records to raw files.
      acceptance_criteria:
        - User can log in, review parsed document, add/edit labels.
        - UI displays linked raw document (from MinIO).
        - Edits propagate back to Postgres.

    - name: "Phase 6 - Deployment & Scaling"
      deliverables:
        - Kubernetes manifests and Helm chart.
        - Multi-LLM provider integration.
        - Monitoring + observability (LangSmith, Prometheus/Grafana).
      acceptance_criteria:
        - Cluster runs frontend + backend containers in separate pods.
        - Queries work with OpenAI + local LLaMA2 backend.
        - Metrics & logs visible in monitoring dashboard.