New folder project structure

[tmikula-dev]

Overview

The project is getting more complex, so there is need of having a new folder clear project structure.

Release Notes:

• New src folder structure
• New tests folder structure

Closes #77

Summary by CodeRabbit

Release Notes

• Refactor

  • Reorganized internal code structure into clearer logical subpackages for improved maintainability and consistency.
  • Updated all module imports to use absolute paths for better clarity and reduced fragility.

• Tests

  • Updated test suite to align with the reorganized code structure.

[coderabbitai]

Walkthrough

Restructures the project by organizing writer modules into src/writers/ and utility modules into src/utils/, converting all relative imports to absolute imports throughout source and test files. Adjusts conf path root calculation and updates all corresponding test expectations.

Changes

Cohort / File(s)	Summary
Package Initializers src/utils/__init__.py, src/writers/__init__.py, tests/__init__.py, tests/utils/__init__.py, tests/writers/__init__.py	Added Apache 2.0 license headers to establish new package directories. No functional code introduced.
Configuration Path Utilities src/utils/conf_path.py	Recomputed project root as two levels up (instead of one) with intermediate variables and clarifying comments; resolution order remains unchanged.
Main Event Handler src/event_gate_lambda.py	Replaced fragile relative ImportError fallbacks with direct absolute imports from src.writers and src.utils.conf_path; introduced internal aliases _CONF_DIR and _INVALID_CONF_ENV.
Trace Logging Utilities src/utils/trace_logging.py	Updated relative imports to absolute: .logging_levels → src.utils.logging_levels, .safe_serialization → src.utils.safe_serialization.
Writer Modules src/writers/writer_eventbridge.py, src/writers/writer_kafka.py, src/writers/writer_postgres.py	Changed trace logging imports from relative to absolute paths (src.utils.trace_logging); preserved all control flow, error handling, and public signatures.
Conf Path Tests tests/utils/test_conf_path.py	Updated module import from src.conf_path to src.utils.conf_path; adjusted test expectations to compute repository root conf directory from module file location (three levels up).
Conf Validation Tests tests/utils/test_conf_validation.py	Adjusted CONF_DIR path from tests/conf to ../conf to reference repository root conf directory.
Safe Serialization Tests tests/utils/test_safe_serialization.py	Updated imports and patch targets from src.safe_serialization to src.utils.safe_serialization.
Trace Logging Tests tests/utils/test_trace_logging.py	Updated imports: TRACE_LEVEL from src.utils.logging_levels and writer modules from src.writers.* (previously src.writer_*).
Writer Module Tests tests/writers/test_writer_eventbridge.py, tests/writers/test_writer_kafka.py, tests/writers/test_writer_postgres.py	Updated module import paths: src.writer_* → src.writers.writer_*; removed unused mock imports in test file.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

• conf_path.py: Logic change in root calculation (two levels up) requires verification of correctness across all resolution paths and existing deployments.
• Test expectations: Multiple test files updated with new path computations; verify expected behavior aligns with actual module locations and repository structure.
• Import consistency: Review that all absolute import paths correctly reference reorganized packages and no circular dependencies are introduced.
• writer_postgres.py: Most substantial writer update with additional typing adjustments; verify all control flow and error handling remain equivalent.

Possibly related PRs

• Enhance error handling and import robustness in event_gate_lambda #62: Updates writer imports in event_gate_lambda.py with guarded try/except/fallback logic versus this PR's direct absolute imports.
• Add configuration directory resolution and update event_gate_lambda to use it #57: Introduces centralized conf_path resolver with CONF_DIR and INVALID_CONF_ENV utilities alongside similar writer import reorganization.
• Add TRACE logging level for verbose payload logging across event writers #61: Modifies trace logging imports and reorganizes package paths for trace utilities and writer modules.

Suggested reviewers

• ABMC831
• Zejnilovic
• oto-macenauer-absa
• petr-pokorny-absa

Poem

🐰 The burrow grows tidy, the warren runs deep,
Writers and loggers in folders now keep,
No relative paths that lead us astray,
Absolute imports light up the way!
Hopping through tests, all expectations aligned,
A structured warren, so cleanly designed. ✨

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 72.73% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'New folder project structure' directly and clearly summarizes the main change of restructuring the codebase into new organizational folders.
Linked Issues check	✅ Passed	The PR successfully implements all coding objectives from issue #77: introduces writers, utils (with logging, serialization, configuration submodules), creates corresponding test structure, and maintains event_gate_lambda.py as the main entry point.
Out of Scope Changes check	✅ Passed	All changes are in-scope restructuring work: converting relative imports to absolute paths, reorganizing files into src/writers, src/utils, tests/writers, and tests/utils folders as required by issue #77.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feature/77-new-folder-project-structure

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

Trivy has completed a full security repository scan ✅ You can find the analysis results for this PR branch on this overview.
Below is the summary of the findings:

TRIVY	CRITICAL	HIGH	MEDIUM	LOW	TOTAL
vulnerability	0	0	0	0	0
secret	0	6	0	0	6
misconfiguration	0	0	1	10	11
license	0	0	0	0	0
➡️ Total	0	6	1	10	17

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (1)

src/writers/writer_postgres.py (1)

30-30: Remove unused noqa directive.

The # noqa: F401 directive is unnecessary since psycopg2 is used later in the code (line 37 onwards).

Apply this diff:

-    import psycopg2  # noqa: F401
+    import psycopg2

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 7c2045b and 1acf137.

📒 Files selected for processing (18)

• src/event_gate_lambda.py (1 hunks)
• src/utils/__init__.py (1 hunks)
• src/utils/conf_path.py (3 hunks)
• src/utils/trace_logging.py (1 hunks)
• src/writers/__init__.py (1 hunks)
• src/writers/writer_eventbridge.py (1 hunks)
• src/writers/writer_kafka.py (1 hunks)
• src/writers/writer_postgres.py (1 hunks)
• tests/__init__.py (1 hunks)
• tests/utils/__init__.py (1 hunks)
• tests/utils/test_conf_path.py (5 hunks)
• tests/utils/test_conf_validation.py (1 hunks)
• tests/utils/test_safe_serialization.py (2 hunks)
• tests/utils/test_trace_logging.py (1 hunks)
• tests/writers/__init__.py (1 hunks)
• tests/writers/test_writer_eventbridge.py (1 hunks)
• tests/writers/test_writer_kafka.py (1 hunks)
• tests/writers/test_writer_postgres.py (1 hunks)

🧰 Additional context used

🧬 Code graph analysis (5)

src/utils/trace_logging.py (1)

src/utils/safe_serialization.py (1)

• safe_serialize_for_log (47-84)

src/writers/writer_kafka.py (3)

src/utils/trace_logging.py (1)

• log_payload_at_trace (29-48)

tests/utils/test_trace_logging.py (2)

• produce (26-29)
• flush (31-32)

tests/writers/test_writer_kafka.py (7)

• produce (10-13)
• produce (20-22)
• produce (92-93)
• flush (15-16)
• flush (31-38)
• flush (47-49)
• flush (58-60)

src/writers/writer_eventbridge.py (1)

src/utils/trace_logging.py (1)

• log_payload_at_trace (29-48)

src/writers/writer_postgres.py (3)

src/utils/trace_logging.py (1)

• log_payload_at_trace (29-48)

tests/writers/test_writer_postgres.py (7)

• client (243-244)
• cursor (174-175)
• execute (34-35)
• execute (159-160)
• connect (191-192)
• connect (227-228)
• commit (177-178)

tests/utils/test_trace_logging.py (4)

• cursor (59-60)
• execute (49-50)
• connect (72-73)
• commit (62-63)

tests/utils/test_safe_serialization.py (1)

src/utils/safe_serialization.py (1)

• safe_serialize_for_log (47-84)

🪛 GitHub Check: Trivy

src/writers/writer_kafka.py

[failure] 73-73: Plaintext password literal
Artifact: src/writers/writer_kafka.py
Type:
Secret Plaintext password literal
Severity: HIGH
Match: "ssl.key.********************************************

src/writers/writer_postgres.py

[failure] 67-67: Plaintext secret literal
Artifact: src/writers/writer_postgres.py
Type:
Secret Plaintext secret literal
Severity: HIGH
Match: ********************************************************

---

[failure] 68-68: Plaintext secret literal
Artifact: src/writers/writer_postgres.py
Type:
Secret Plaintext secret literal
Severity: HIGH
Match: ************************************************************

---

[failure] 70-71: Plaintext secret literal
Artifact: src/writers/writer_postgres.py
Type:
Secret Plaintext secret literal
Severity: HIGH
Match: if secret_name and **************

---

[failure] 72-72: Plaintext secret literal
Artifact: src/writers/writer_postgres.py
Type:
Secret Plaintext secret literal
Severity: HIGH
Match: ************************************************************************************

---

[failure] 282-282: Plaintext password literal
Artifact: src/writers/writer_postgres.py
Type:
Secret Plaintext password literal
Severity: HIGH
Match: ******************************

🪛 Ruff (0.14.4)

src/writers/writer_kafka.py

58-58: Avoid specifying long messages outside the exception class

(TRY003)

---

109-109: Unused lambda argument: msg

(ARG005)

src/writers/writer_postgres.py

30-30: Unused noqa directive (non-enabled: F401)

Remove unused noqa directive

(RUF100)

---

90-120: Possible SQL injection vector through string-based query construction

(S608)

---

149-171: Possible SQL injection vector through string-based query construction

(S608)

---

186-206: Possible SQL injection vector through string-based query construction

(S608)

---

229-247: Possible SQL injection vector through string-based query construction

(S608)

---

299-299: Use explicit conversion flag

Replace with conversion flag

(RUF010)

🔇 Additional comments (10)

tests/writers/test_writer_eventbridge.py (1)

4-4: Imports aligned with new package layout.

Switching to src.writers.writer_eventbridge keeps the tests in step with the reorganized module structure. Looks good.

tests/utils/test_trace_logging.py (1)

4-7: LGTM!

The import path updates correctly reflect the new package structure with writers under src.writers/ and utilities under src.utils/.

tests/writers/test_writer_postgres.py (1)

22-22: LGTM!

The import path correctly reflects the new package structure.

src/writers/writer_kafka.py (1)

29-29: LGTM!

The import path correctly reflects the new utility module location under src.utils/.

src/utils/conf_path.py (1)

37-39: LGTM!

The project root calculation correctly reflects the new file location at src/utils/conf_path.py (two levels up from root instead of one). The intermediate variable parent_utils_dir improves clarity.

tests/utils/test_conf_validation.py (1)

22-22: LGTM!

The CONF_DIR path calculation correctly navigates from the new test location (tests/utils/) to the repository root conf/ directory.

src/writers/writer_postgres.py (1)

34-34: LGTM!

The import path correctly reflects the new utility module location under src.utils/.

src/event_gate_lambda.py (1)

34-39: LGTM!

The updated imports correctly reflect the new package structure, and the internal aliases maintain compatibility within the module. The removal of conditional import fallbacks simplifies the code.

tests/utils/test_conf_path.py (2)

7-7: LGTM!

The import path correctly reflects the new module location.

---

25-26: LGTM!

The test assertions correctly compute the repository root conf path by going three levels up from the module file location (src/utils/conf_path.py), which aligns with the updated project_root calculation in the source module.

Also applies to: 83-84, 125-126, 156-157

[oto-macenauer-absa]

lgtm, the aquasec warning are nonsense/false positives