TASKS.md

Implementation Task List: IDA, HAPPE, and IPC

This task list breaks down the implementation steps outlined in PLANNING.md.

Phase 1: Define Shared IPC Communication (@ipc Crate)

• Module Setup:
  • Create directory ipc/src/internal_messages/.
  • Create file ipc/src/internal_messages/mod.rs.
  • Create file ipc/src/internal_messages/types.rs.
  • Declare internal_messages module in ipc/src/lib.rs (pub mod internal_messages;).
• Define Structs & Enums (HAPPE <-> IDA):
  • Define MemoryItem, ConversationTurn in internal_messages/types.rs.
  • Define InternalMessage enum (GetMemoriesRequest, GetMemoriesResponse, StoreTurnRequest) in internal_messages/mod.rs.
  • Add #[derive(Serialize, Deserialize, Debug, Clone, PartialEq)] to all.
  • Re-export types from types.rs in mod.rs.
• Define Structs & Enums (Client <-> HAPPE):
  • Create ipc/src/happe_request/mod.rs and types.rs.
  • Define HappeQueryRequest { query: String } struct.
  • Define HappeQueryResponse { response: String, error: Option<String> } struct.
  • Add #[derive(Serialize, Deserialize, Debug, Clone, PartialEq)] to all.
  • Add happe_request module to ipc/src/lib.rs.
• Dependencies:
  • Ensure serde (with derive feature) is listed under [dependencies] in ipc/Cargo.toml.
  • Ensure gemini_core is added if shared types are used.
• Verification:
  • Run cargo check -p gemini-ipc to ensure the crate compiles.
• Milestone: All IPC message definitions complete and build successfully.

Phase 2: Implement IDA Daemon Skeleton (@ida Crate)

• Binary Skeleton & Config: (Basic setup)
• IPC Server (ida/src/ipc_server.rs): (Listens for HAPPE)
• Connection Handler (ida/src/ipc_server.rs): (Handles InternalMessage)
• Background Storage Task (ida/src/storage.rs): (Placeholder logic)
• MCP Client Placeholder (ida/src/memory_mcp_client.rs): (Placeholder functions)
• Dependencies (ida/Cargo.toml): (Initial dependencies)
• Verification: (Basic build)
• Milestone: ida-daemon builds and runs, successfully accepting IPC connections from HAPPE and processing known message types (using placeholders for external calls).

Phase 3: Implement HAPPE Daemon Skeleton (@happe Crate)

• Binary Skeleton & Config: (Basic setup)
• IDA IPC Client (happe/src/ida_client.rs): (Connects to IDA)
• Core Interaction Loop (happe/src/coordinator.rs): (Placeholder loop calling IDA client and placeholder LLM/MCP)
• LLM Client Placeholder (happe/src/llm_client.rs):
• MCP Client Placeholder (happe/src/mcp_client.rs):
• Dependencies (happe/Cargo.toml): (Initial dependencies)
• Verification: (Basic build)
• Milestone: happe-daemon builds and runs, successfully connecting to IDA, sending requests, receiving responses, and sending async notifications (using placeholders for external calls).

Phase 4: Initial Integration Test (HAPPE <-> IDA Placeholders)

• Build Workspace:
• Direct IDA Test (Optional):
• Concurrent Daemon Test:
• Interaction Flow Test (via stdin): (Verify logs for HAPPE <-> IDA placeholder flow)
• Repeat:
• Milestone: Successful execution of the basic HAPPE <-> IDA communication flow verified via logs using placeholder logic.

Phase 5: Implement HAPPE Core Logic & API (@happe Crate)

• Core Interaction Implementation (happe/src/coordinator.rs):
  • Define core process_query(config: &AppConfig, mcp_provider: &mcp::host::McpHost, gemini_client: &core::client::GeminiClient, query: String) -> Result<String, Error> function.
    • Adapt structure from cli/src/app.rs::process_prompt.
  • Implement real prompt construction (using gemini_core::prompt, memories from IDA, MCP info).
    • Reference cli/src/app.rs lines ~107-121, ~115 for structure.
    • Use mcp::gemini::build_mcp_system_prompt.
  • Integrate real llm_client::generate_response call (using implementation from below).
  • Implement logic to handle LLM function calls.
    • Use mcp_provider to get capabilities (core::rpc_types::ServerCapabilities).
    • Use mcp::gemini::generate_gemini_function_declarations to create core::types::Tool list.
    • Use llm_client to parse function calls from response (mcp::gemini::FunctionCall).
    • Call happe/src/mcp_client.rs::execute_tool.
    • Reference cli/src/app.rs lines ~81-112, ~211, ~232-247.
  • Call IdaClient::store_turn_async with ipc::internal_messages::ConversationTurn.
    • Reference cli/src/app.rs lines ~218-227, ~250-270.
• LLM Client Implementation (happe/src/llm_client.rs):
  • Implement API call logic (using reqwest, core::client::GeminiClient, core::types::GenerateContentRequest).
    • Adapt from gemini-core::client & cli/src/app.rs lines ~189-211.
  • Implement response parsing (extract_text_from_response, extract_function_calls_from_response using mcp::gemini::parse_function_calls).
  • Define proper error types (consider using/extending core::errors).
• MCP Client Implementation (happe/src/mcp_client.rs - If needed):
  • Implement get_capabilities function (likely wrapper around mcp::host::McpHost::get_all_capabilities).
    • Reference cli/src/app.rs::get_capabilities (~line 1235).\
  • Implement execute_tool function (using mcp::host::McpHost::execute_tool).
    • Reference cli/src/app.rs::execute_tool (~line 1209).\
  • Ensure HAPPE has an McpProvider instance (mcp::host::McpHost).
• HAPPE Input API & Servers:
  • Implement IPC server (happe/src/ipc_server.rs) using ipc::happe_request types.
  • Implement HTTP server (happe/src/http_server.rs) using axum/warp.
  • Update happe-daemon.rs to initialize core::client::GeminiClient and mcp::host::McpHost and pass them to handlers.
• Binary Implementation (happe/src/bin/happe-daemon.rs):
  • Implement real configuration loading (using core::config::GeminiConfig as base, loading MCP config with mcp::config::load_mcp_servers).
    • Migrate from cli/src/config.rs & cli/src/main.rs.\
  • Define comprehensive CLI arguments (clap).
• Dependencies Update (happe/Cargo.toml):
  • Add @core, @mcp, @ipc.
  • Add reqwest, axum/warp, serde_json.
  • Ensure features match usage.
• Verification (Full):
  • Run cargo check --all-targets / build --all-targets.
• Milestone: happe-daemon fully functional: processes queries via IPC/HTTP, interacts with real LLM and MCP servers, and uses placeholder IDA for memory.

Phase 6: Implement IDA Core Logic (@ida Crate)

• Storage Logic (ida/src/storage.rs):
  • Replace placeholder logic in handle_storage.
  • Implement analysis of ipc::internal_messages::ConversationTurn.
  • Implement memory summarization/formatting.
    • Consider adapting cli/src/history.rs::summarize_conversation.
  • Implement interaction with Memory Backend:
    • Option A (Direct Access): Use gemini_memory::store::MemoryStore.
      • Initialize MemoryStore in ida-daemon.rs.
      • Call MemoryStore methods (add, check duplicates, etc.).
      • Handle conversion between gemini_memory::memory::Memory and ipc::internal_messages::MemoryItem.
      • Replaces cli/src/memory/mod.rs, cli/src/memory_broker.rs.
      • Include async queue/workers if needed, migrating from cli/src/memory/mod.rs.
    • Option B (MCP Access): Use @mcp::host::McpHost as a client.
      • Initialize McpHost in ida-daemon.rs, configured for the Memory MCP server.
      • Define and call MCP methods for store_memory, check_duplicates, etc.
  • Implement duplicate checking logic (either via MemoryStore or MCP call).
• Retrieval Logic (ida/src/memory_mcp_client.rs or direct):
  • Replace placeholder retrieve_memories.
  • Implement query logic against Memory Backend:
    • Option A (Direct Access): Use gemini_memory::store::MemoryStore::query_memories.
      • Handle conversion from gemini_memory::memory::Memory to ipc::internal_messages::MemoryItem.
    • Option B (MCP Access): Call MCP method for query_memories.
      • Replaces calls like enhance_prompt in cli/src/app.rs.
• Configuration (ida/src/bin/ida-daemon.rs):
  • Add config based on chosen backend access (MemoryStore path/config OR Memory MCP server address).
  • Add config for async workers, etc.
    • Migrate from cli/src/config.rs::AsyncMemoryConfigExt.
• Dependencies (ida/Cargo.toml):
  • Add @ipc.
  • Option A: Add @memory.
  • Option B: Add @mcp.
  • Maybe add @core (for errors/config base).
• Verification:
  • Run cargo check/build.
  • Unit tests for storage/retrieval logic.
• Milestone: ida-daemon fully functional: stores, summarizes, and retrieves memories based on real logic, interacting with the memory backend.

Phase 7: Refactor CLI (@cli Crate)

• Modify Main Logic (cli/src/main.rs, cli/src/app.rs):
  • Remove LLM/MCP/Memory/History logic.
  • Keep input loop (run_interactive_chat, etc.).
  • Add IPC client (cli/src/happe_client.rs?) to connect to happe-daemon query socket.
  • Send query via IPC, receive response, display using output.rs.
• Update CLI Arguments (cli/src/cli.rs::Args):
  • Remove LLM/memory/etc. args. Add HAPPE connection arg.
• Update Configuration (cli/src/config.rs):
  • Remove old config logic. Add HAPPE connection config.
• Remove Unused Code:
  • Gut/delete history.rs, memory/, memory_broker.rs.
  • Remove unused dependencies (gemini-core, gemini-memory, @mcp) from cli/Cargo.toml.
  • Clean up app.rs.
• Dependencies (cli/Cargo.toml):
  • Add @ipc (for HappeQueryRequest/Response).
  • Add necessary IPC client library (tokio?).
• Verification:
  • Run cargo check/build.
• Milestone: cli executable successfully acts as a front-end to happe-daemon via IPC, core logic removed.

Phase 8: End-to-End Integration Testing

• Setup: Run ida-daemon, happe-daemon, and any required MCP servers.
• CLI Test: Use the refactored @cli to interact with happe-daemon.
  • Verify queries are processed correctly.
  • Verify responses are displayed.
  • Verify memory retrieval influences responses over time (check IDA logs).
  • Verify turns are stored (check IDA logs).
• HTTP Test: Use curl or similar to send queries to happe-daemon's HTTP endpoint.
  • Verify responses are correct.
  • Verify interaction affects memory via IDA logs.
• Tool Call Test: If MCP tools are implemented, trigger them via queries.
  • Verify HAPPE logs show function call parsing and execution via MCP.
  • Verify tool results are sent back to LLM and influence final response.
• Milestone: Full system (CLI/HTTP -> HAPPE -> LLM/MCP & IDA -> Memory Backend) functions correctly for multiple interaction turns.

General Tasks (Ongoing)

• Code Formatting: Run cargo fmt regularly across the workspace.
• Linting: Run cargo clippy --all-targets --all-features regularly and address warnings.
• Unit Tests: Add unit tests for specific functions with complex logic.
• Documentation: Add doc comments (///) for public functions, structs, and enums.
• Error Handling: Ensure errors are propagated or handled gracefully.
• Configuration: Refine configuration loading (e.g., use environment variables, config files).
• Logging: Improve logging messages for clarity and debugging.
• Version Control: Commit changes frequently with clear messages.

Phase 9: Enhance IDA with Broker LLM (Direct Integration)

This phase implements the plan to integrate a "broker" LLM directly within IDA to refine memory retrieval results before sending them to HAPPE.

Sub-Phase 1: Prerequisites and Setup

• Broker LLM Configuration Loading:
  • Verify MemoryBrokerConfig fields (provider, api_key, model_name, base_url) are loaded correctly into IdaConfig.
  • Add loaded IdaConfig to shared DaemonState struct.
  • Update DaemonState instantiation in ida/src/bin/ida-daemon.rs.
  • Ensure handle_message in ida/src/ipc_server.rs can access the IdaConfig.
• Ensure MCP Access Reuse in IDA:
  • Identify where MemoryStore is initialized in ida/src/bin/ida-daemon.rs.
  • Store the Arc<dyn McpHostInterface + Send + Sync> used for MemoryStore in the DaemonState.
  • Update DaemonState definition and instantiation.
• Integrate Broker LLM Client into IDA:
  • Configuration (core/src/config.rs):
    • Modify MemoryBrokerConfig struct (add provider, base_url, etc.).
    • Update corresponding IdaConfig in ida/src/config.rs and its From impl.
    • Update default config generation (install or tools) for [ida.memory_broker].
  • Implementation (IDA Crate):
    • Add dependencies (reqwest, serde, serde_json, async-trait) to ida/Cargo.toml.
    • Create ida/src/llm_clients.rs.
    • Define trait LLMClient { async fn generate(&self, prompt: &str) -> Result<String>; }.
    • Implement the trait for supported providers (e.g., GeminiClient, OllamaClient).
    • Create factory function create_llm_client(config: &CoreMemoryBrokerConfig) -> Result<Option<Arc<dyn LLMClient>>>.
    • Add Option<Arc<dyn LLMClient>> field to DaemonState.
    • Initialize the client in ida/src/bin/ida-daemon.rs using the factory.

Sub-Phase 2: Core Logic Implementation in IDA

• Modify ida::memory_mcp_client::retrieve_memories:
  • Change signature to accept broker_llm_client: &Option<Arc<dyn LLMClient>> and conversation_context: Option<String>.
  • After semantic search, if results exist and client exists:
    • Construct broker prompt (query, context, candidates, instructions).
    • Call client.generate(&broker_prompt).await.
    • Parse response (e.g., comma-separated keys).
    • Filter semantic search results based on broker response.
    • Implement fallback logic on broker error.
  • Update call site in ida/src/ipc_server.rs (handle_message) to pass the client and context.

Sub-Phase 3: Integrating Conversation Context (IPC Changes)

• Modify IPC Message (ipc/src/internal_messages.rs):
  • Add conversation_context: Option<String> to GetMemoriesRequest.
• Update HAPPE Client (happe/src/ida_client.rs):
  • Update get_memories signature and request construction.
• Update HAPPE Orchestrator (happe/src/coordinator.rs or similar):
  • Gather context and pass it to ida_client::get_memories.
• Update IDA Server (ida/src/ipc_server.rs):
  • Extract context and pass it to memory_mcp_client::retrieve_memories.

Sub-Phase 4: Testing and Refinement

• Testing: Unit, Integration, End-to-End tests.
• Refinement: Prompt engineering, latency analysis, fallback logic, configuration.

Phase 10: Implement Configurable Session Management in HAPPE

This phase adds a stateful session management system to HAPPE, allowing it to maintain conversation history across multiple requests within a session. This is crucial for providing proper context to IDA for memory retrieval.

Sub-Phase 1: Define Core Session Store Trait & In-Memory Adapter

• Module Setup:
  • Create directory happe/src/session/.
  • Create directory happe/src/session/adapters/.
  • Create file happe/src/session/mod.rs.
  • Create file happe/src/session/store.rs.
  • Create file happe/src/session/adapters/in_memory.rs.
  • Declare session module in happe/src/lib.rs (pub mod session;).
• Define SessionStore Trait (happe/src/session/store.rs):
  • Define #[async_trait] pub trait SessionStore: Send + Sync.
  • Add methods for session management:
    • create_session
    • get_session
    • save_session
    • delete_session
    • cleanup_expired_sessions
• Implement InMemorySessionStore (happe/src/session/adapters/in_memory.rs):
  • Define struct InMemorySessionStore.
  • Add thread-safe storage using Arc<RwLock<HashMap<String, Session>>>.
  • Implement SessionStore trait.
  • Add expiration and cleanup logic.
• Dependencies (happe/Cargo.toml):
  • Add async-trait = "0.1".
  • Add uuid = { version = "1", features = ["v4"] } (for session ID generation).
  • Ensure tokio features include sync.

Sub-Phase 2: Integrate Session Store into HAPPE State

• State (happe/src/http_server.rs, happe/src/ipc_server.rs):
  • Add session_store: SessionStoreRef to AppState struct (http_server.rs).
  • Add session_store: SessionStoreRef to IpcServerState struct (ipc_server.rs).
  • Initialize session store in HTTP and IPC servers.
  • Add session cleanup task to the IPC server.

Sub-Phase 3: Modify Coordinator & IPC Handler

• Coordinator (happe/src/coordinator.rs):
  • Modify process_query signature to accept session: &Session.
  • Add helper function get_conversation_history to extract history from the session.
  • Add helper function update_session_history to store turns in the session.
• IPC Handler (happe/src/ipc_server.rs):
  • Modify handle_connection:
    • Get or create session for the request.
    • Pass session to coordinator::process_query.
    • Update session with new conversation turn.
    • Save session back to the store.
• HTTP Handler (happe/src/http_server.rs):
  • Modify handle_query:
    • Extract session ID from request or create a new one.
    • Get or create session.
    • Pass session to coordinator::process_query.
    • Update and save session.

Sub-Phase 4: Update IPC Request & Client (@cli)

• IPC Request (ipc/src/happe_request/types.rs or mod.rs):
  • Add session_id: Option<String> field to HappeQueryRequest struct.
  • Add session_id: Option<String> field to HappeQueryResponse struct.
• CLI (@cli Crate):
  • Identify where the main interaction loop/HAPPE client logic resides (e.g., cli/src/app.rs or cli/src/happe_client.rs).
  • On CLI startup, generate a persistent session_id for the duration of the run (e.g., let session_id = uuid::Uuid::new_v4().to_string();).
  • Modify the code that creates and sends HappeQueryRequest to include this session_id.

Sub-Phase 5: Testing & Refinement

• Unit Tests:
  • Add tests for the Session struct in store.rs.
  • Add tests for InMemorySessionStore.
• Integration Tests: Test IPC handler with session history.
• End-to-End Tests (@cli -> happe-daemon): Verify context is maintained across multiple turns within a single CLI run.
• Refinement: Assess performance, history pruning, error handling, security of session IDs.

Phase 11: Rewrite MCP Servers (Hybrid Approach - Python Facades, Rust Core Memory)

This phase replaces filesystem-mcp and command-mcp with Python implementations. memory-store-mcp becomes a Python facade process, while its core logic (including vector store via gemini-memory) is embedded and executed directly within mcp-hostd.rs.

Sub-Phase 1: Investigation & Design Refinement

• Analyze mcp-hostd.rs & gemini_mcp Crate: (Config loading/launching confirmed).
• Analyze install Crate: (Binary/config handling understood).
• Analyze gemini-memory Dependency: (Will be embedded in mcp-hostd).
• Refine Design: Confirm hybrid strategy:
  • mcp-hostd embeds gemini_memory::MemoryStore.
  • mcp-hostd intercepts & handles memory-store-mcp/* tool calls internally.
  • Minimal memory-store-mcp.py facade launched via stdio for handshake.
  • Standard filesystem-mcp.py and command-mcp.py servers.

Sub-Phase 2: Implementation

• Modify mcp-hostd.rs (Initialization):
  • Ensure gemini-memory dependency available to mcp-hostd.
  • Initialize MemoryStore instance during startup.
  • Load necessary MemoryStore config (DB path, etc.) from main config.toml.
  • Store Arc<MemoryStore> where process_request can access it.
• Modify mcp-hostd.rs (process_request):
  • Add logic to intercept ExecuteTool for server == "memory-store-mcp".
  • Implement internal calls to embedded MemoryStore for intercepted requests.
  • Bypass standard host.execute_tool for these intercepted requests.
• Modify mcp-hostd.rs (Capabilities):
  • Manually add tool definitions for internal memory operations under the memory-store-mcp server name during capability aggregation.
• Implement python_mcp/servers/base_server.py (as before).
• Implement python_mcp/servers/filesystem_mcp.py (as before).
• Implement python_mcp/servers/command_mcp.py (as before).
• Simplify python_mcp/servers/memory_store_mcp.py:
  • Remove MemoryStore class and tool handlers.
  • main should only create McpBaseServer, register no tools, and call run().
• Project Setup (python_mcp, requirements.txt - as before).

Sub-Phase 3: Integration & Installation

• Modify install/src/main.rs:
  • Remove Rust MCP binary installation steps.
  • Add step to copy python_mcp directory to installation target.
  • Modify install_unified_config:
    • Generate mcp_servers.json entries pointing all 3 servers (memory-store-mcp, filesystem-mcp, command-mcp) to their Python scripts.
    • Ensure main config.toml generation includes necessary [memory] / [ida] sections for embedded MemoryStore config.

Sub-Phase 4: Testing & Deployment

• Build & Install modified install crate.
• Verify Python scripts and config files placement.
• Install Python dependencies (pip install -r requirements.txt).
• Start mcp-hostd. Check logs for embedded MemoryStore init & Python process handshakes.
• IPC Client Testing:
  • Test GetCapabilities (should show memory tools listed under memory-store-mcp).
  • Test filesystem-mcp tools (check Python logs).
  • Test command-mcp tools (check Python logs).
  • Test memory-store-mcp tools (check mcp-hostd logs & DB state).
• Shutdown Testing.
• Deployment.