[Node] smart CSV import/export with templates

.github/workflows/main.yml

@@ -110,6 +110,12 @@ jobs:
         python-version: ${{ matrix.version }}
         architecture: ${{ matrix.arch }}
 
+    - name: Set up Node.js
+      if: matrix.package == 'octobot'
+      uses: actions/setup-node@v6
+      with:
+        node-version: 22
+
     - name: Download build wheel artifact
       uses: actions/download-artifact@v8
       with:
@@ -181,6 +187,16 @@ jobs:
         cd ${{ matrix.package }}
         pytest tests --backend=rust -v
 
+    - name: Run frontend tests
+      if: matrix.package == 'octobot'
+      run: |
+        find packages/tentacles -name "package.json" -not -path "*/node_modules/*" | while read pkg; do
+          cd "$(dirname "$pkg")"
+          npm ci
+          npm test
+          cd - > /dev/null
+        done
+
   docker:
     name: Build & Push Docker images
     needs: [ build, tests ]

.gitignore

@@ -4,6 +4,9 @@ __pycache__/
 *$py.class
 *.orig
 
+# Node
+node_modules/
+
 # C extensions
 *.so
 

docs/content/developers/packages/node.md

@@ -28,19 +28,19 @@ Log messages emitted inside any workflow or step are routed to a per-workflow fi
 
 Task payloads are optionally encrypted using a hybrid RSA/AES-GCM/ECDSA scheme. Each encryption call generates a fresh AES-256-GCM key and IV; the AES key is wrapped with RSA-OAEP so the bulk payload never travels under the asymmetric key directly. An ECDSA signature over the ciphertext — computed as `ciphertext + encrypted_aes_key + iv` concatenated — is verified before any decryption attempt, preventing chosen-ciphertext attacks.
 
-The central design constraint is directional key separation. There are two independent key pairs — one for task inputs, one for task outputs — and each pair is split across the two sides. The node holds the private key to decrypt incoming inputs and the public key to encrypt outgoing results. The producer holds the complementary keys. Neither side has all eight keys. A compromised node can read the task inputs it needs to execute, but cannot decrypt the results it just produced — those are only readable by the producer.
+**Key split between server and browser.** Keys are divided strictly by who needs them at runtime. The server holds the four INPUTS keys — used to decrypt incoming task content and verify its signature before execution, and to re-sign and re-encrypt results before storage. These are the only keys the node process ever loads, set via four environment variables (`TASKS_INPUTS_RSA_PRIVATE_KEY`, `TASKS_INPUTS_ECDSA_PUBLIC_KEY`, `TASKS_INPUTS_RSA_PUBLIC_KEY`, `TASKS_INPUTS_ECDSA_PRIVATE_KEY`). The four OUTPUTS keys — used to decrypt results produced by the node — are never loaded by the server. They exist only in the browser, entered once in the Settings page and stored locally. A node process running with only the four INPUTS keys in its environment is fully functional; the OUTPUTS keys have no server-side role.
 
 **Metadata format asymmetry.** The accompanying metadata envelope — which carries `ENCRYPTED_AES_KEY_B64`, `IV_B64`, and `SIGNATURE_B64` — is serialised differently depending on direction. For task inputs, `encrypt_task_content` returns the metadata as `base64(JSON)`: the JSON object is first serialised to a string, then base64-encoded again. The corresponding `decrypt_task_content` therefore base64-decodes then JSON-parses. For task outputs, `encrypt_task_result` returns the metadata as a plain JSON string with no outer base64 layer, and `decrypt_task_result` JSON-parses directly. The asymmetry exists because input metadata is transmitted as a CSV or API field where a single opaque string is easier to embed without escaping, while output metadata is stored in the database and consumed programmatically by code that already handles JSON. Being aware of this distinction matters when building a producer or any tooling that reads raw database records.
 
 **`encrypted_task` context manager.** This wraps each task execution transparently. On entry it decrypts `task.content` if the input keys are present and `task.content_metadata` is non-null. On exit it encrypts `task.result` if the output keys are present. The two directions are independent — a node can be configured to decrypt inputs only, encrypt outputs only, or both. If decryption fails, the context manager does not propagate the exception into the workflow; instead it writes a structured error dict to `task.result`, so the failure is observable via the API without crashing the scheduler.
 
 **Security boundary with `octobot_flow`.** The `encrypted_task` context manager wraps the call to `octobot_flow`'s `AutomationJob.run()` inside the node's workflow step. This means the scheduler — the master node that stores and routes tasks — only ever sees encrypted payloads. Task content is decrypted just before execution on the consumer node that holds the private keys, and the result is re-encrypted immediately after. The scheduler database, its API, and any intermediary never handle plaintext. A compromised scheduler leaks only ciphertext. From flow's perspective, nothing changes — it receives a plaintext `AutomationState` dict and returns an updated one. The flow package has no awareness of encryption, which means the same engine works identically in encrypted node deployments, unencrypted nodes, and standalone bots.
 
-**Key loading and validation.** All eight keys are accepted as PEM-encoded strings via environment variables, decoded to `bytes` at process startup by a `BeforeValidator` in the pydantic `Settings` model. There is no lazy loading — `settings` is a module-level singleton instantiated at import time, so a misconfigured key value fails fast before any requests are served. Two computed properties — `is_node_side_encryption_enabled` and `is_producer_side_encryption_enabled` — check whether all four keys for each role are present simultaneously, enabling clean conditional logic elsewhere in the codebase without repeating null checks. The broader `tasks_encryption_enabled` flag requires all eight keys, which is only meaningful when a single process is acting as both producer and node (uncommon outside of testing).
+**Key loading and validation.** The four INPUTS keys are accepted as PEM-encoded strings via environment variables, decoded to `bytes` at process startup by a `BeforeValidator` in the pydantic `Settings` model. There is no lazy loading — `settings` is a module-level singleton instantiated at import time, so a misconfigured key value fails fast before any requests are served. The `is_node_side_encryption_enabled` computed property checks whether all four INPUTS keys are present simultaneously, and `tasks_encryption_enabled` is an alias for it — enabling clean conditional logic throughout the codebase without repeating null checks.
 
-**CSV bulk tools.** The `tools/` directory contains three standalone CLI scripts — `encrypt_csv_tasks.py`, `decrypt_csv_tasks.py`, and `decrypt_csv_results.py` — intended for batch operations outside the running service. All three load keys either from PEM files passed on the command line or from the environment variables, falling back gracefully between the two with an explicit priority order. A companion `csv_utils.py` implements the column-merging logic shared with the TypeScript `node_web_interface` frontend: it reads a flat CSV where arbitrary columns are collapsed into a JSON `content` object on encryption, and expands an encrypted result dict back into separate columns on decryption. This symmetry means a producer can author tasks in a spreadsheet-friendly multi-column format, encrypt in bulk, and upload the resulting file directly to the API without any intermediate reformatting. The `generate_and_save_keys` helper in `csv_utils.py` also handles first-time key provisioning — generating all four key pairs (4096-bit RSA and ECDSA) and writing them to a `task_encryption_keys.json` file — which is the recommended path for setting up a fresh deployment.
+**Browser key storage.** The OUTPUTS keys entered in the browser Settings page, and the login passphrase, are both stored in `IndexedDB` encrypted with a device-bound, non-extractable AES-256-GCM key. That device key is generated on first login using `crypto.subtle.generateKey` with `extractable: false`, stored in the same `IndexedDB` database, and can be used by the browser but never exported or read as raw bytes — not even from a filesystem dump of the database file. The device key is origin-bound, so it cannot be used from another domain or browser profile. Neither the passphrase nor the OUTPUTS keys are ever stored in `localStorage` or sent to the server.
 
-**Key generation.** For a fresh deployment, `generate_and_save_keys` creates all four key pairs at once — two 4096-bit RSA pairs (one for inputs, one for outputs) and two ECDSA pairs on the SECP256R1 curve — and writes them to a single `task_encryption_keys.json` file. The node operator and producer each extract their half of the keys from that file. This single-generation approach guarantees that the key pairs are mathematically matched and avoids the error-prone step of generating keys on each side independently. Keys can also be generated manually with openssl:
+**Key generation.** For a fresh deployment, `generate_and_save_keys` creates all four key pairs at once — two 4096-bit RSA pairs (one for inputs, one for outputs) and two ECDSA pairs on the SECP256R1 curve — and writes them to a single `task_encryption_keys.json` file. From that file, the server operator copies the four INPUTS keys into the node's environment variables, and enters the four OUTPUTS keys into the browser Settings page. Keys can also be generated manually with openssl:
 
 ```bash
 # RSA 4096-bit key pair (repeat for inputs and outputs)
@@ -52,6 +52,6 @@ openssl ecparam -genkey -name prime256v1 -noout -out ecdsa_private.pem
 openssl ec -in ecdsa_private.pem -pubout -out ecdsa_public.pem
 ```
 
-The resulting PEM strings are set as environment variables on the appropriate side — four keys per role (node and producer).
+The four INPUTS PEM strings go into the node's `.env` file. The four OUTPUTS PEM strings are entered in the browser Settings page and stored locally — they are never sent to the server.
 
-Encryption is opt-in. If the relevant keys are absent, the corresponding path is skipped and the field stays plaintext, which is the backward-compatible default.
+Encryption is opt-in. If the INPUTS keys are absent from the environment, the corresponding path is skipped and the field stays plaintext, which is the backward-compatible default.

packages/node/.env.sample

@@ -9,4 +9,10 @@ BACKEND_CORS_ORIGINS="http://localhost,http://localhost:5173,http://localhost:80
 SCHEDULER_POSTGRES_URL=
 SCHEDULER_SQLITE_FILE=tasks.db
 
-BLOCKCHAIN_WALLETS_EXTRA_CONFIG={}
+BLOCKCHAIN_WALLETS_EXTRA_CONFIG={}
+
+# Task encryption keys (server-side only — OUTPUTS keys are configured in browser settings)
+# TASKS_INPUTS_RSA_PRIVATE_KEY=
+# TASKS_INPUTS_ECDSA_PUBLIC_KEY=
+# TASKS_INPUTS_RSA_PUBLIC_KEY=
+# TASKS_INPUTS_ECDSA_PRIVATE_KEY=

packages/node/octobot_node/config.py

@@ -95,17 +95,11 @@ def all_cors_origins(self) -> list[str]:
     ADMIN_USERNAME: EmailStr = DEFAULT_ADMIN_USERNAME
     ADMIN_PASSWORD: str = DEFAULT_ADMIN_PASSWORD
 
-    # Used to decrypt inputs and encrypt outputs
+    # Task encryption keys (server-side)
     TASKS_INPUTS_RSA_PRIVATE_KEY: Annotated[bytes | None, BeforeValidator(parse_key_to_bytes)] = None
     TASKS_INPUTS_ECDSA_PUBLIC_KEY: Annotated[bytes | None, BeforeValidator(parse_key_to_bytes)] = None
-    TASKS_OUTPUTS_RSA_PUBLIC_KEY: Annotated[bytes | None, BeforeValidator(parse_key_to_bytes)] = None
-    TASKS_OUTPUTS_ECDSA_PRIVATE_KEY: Annotated[bytes | None, BeforeValidator(parse_key_to_bytes)] = None
-
-    # Used to encrypt inputs and decrypt outputs
     TASKS_INPUTS_RSA_PUBLIC_KEY: Annotated[bytes | None, BeforeValidator(parse_key_to_bytes)] = None
     TASKS_INPUTS_ECDSA_PRIVATE_KEY: Annotated[bytes | None, BeforeValidator(parse_key_to_bytes)] = None
-    TASKS_OUTPUTS_RSA_PRIVATE_KEY: Annotated[bytes | None, BeforeValidator(parse_key_to_bytes)] = None
-    TASKS_OUTPUTS_ECDSA_PUBLIC_KEY: Annotated[bytes | None, BeforeValidator(parse_key_to_bytes)] = None
 
     USE_DEDICATED_LOG_FILE_PER_AUTOMATION: bool = True
 
@@ -115,24 +109,14 @@ def is_node_side_encryption_enabled(self) -> bool:
         return all([
             self.TASKS_INPUTS_RSA_PRIVATE_KEY,
             self.TASKS_INPUTS_ECDSA_PUBLIC_KEY,
-            self.TASKS_OUTPUTS_RSA_PUBLIC_KEY,
-            self.TASKS_OUTPUTS_ECDSA_PRIVATE_KEY,
-        ])
-
-    @computed_field
-    @property
-    def is_producer_side_encryption_enabled(self) -> bool:
-        return all([
             self.TASKS_INPUTS_RSA_PUBLIC_KEY,
             self.TASKS_INPUTS_ECDSA_PRIVATE_KEY,
-            self.TASKS_OUTPUTS_RSA_PRIVATE_KEY,
-            self.TASKS_OUTPUTS_ECDSA_PUBLIC_KEY,
         ])
 
     @computed_field
     @property
     def tasks_encryption_enabled(self) -> bool:
-        return self.is_node_side_encryption_enabled and self.is_producer_side_encryption_enabled
+        return self.is_node_side_encryption_enabled
 
     def _check_default_secret(self, var_name: str, value: str | None, default_value: EmailStr | None) -> None:
         if value == default_value:

packages/node/octobot_node/constants.py

@@ -21,3 +21,10 @@
 
 AUTOMATION_LOGS_FOLDER = f"{BASE_LOGS_FOLDER}/automations"
 PARENT_WORKFLOW_ID_LENGTH = 36 # length of a UUID4
+
+TASKS_ENCRYPTION_ENV_VARS = [
+    "TASKS_INPUTS_RSA_PRIVATE_KEY",
+    "TASKS_INPUTS_ECDSA_PUBLIC_KEY",
+    "TASKS_INPUTS_RSA_PUBLIC_KEY",
+    "TASKS_INPUTS_ECDSA_PRIVATE_KEY",
+]

packages/node/octobot_node/models.py

@@ -50,6 +50,7 @@ class Execution(BaseModel):
     name: typing.Optional[str] = None
     description: typing.Optional[str] = None
     actions: typing.Optional[str] = None
+    content_metadata: typing.Optional[str] = None
     type: typing.Optional[str] = None
     status: typing.Optional[TaskStatus] = None
     result: typing.Optional[str] = None
@@ -66,6 +67,7 @@ class Task(BaseModel):
     content_metadata: typing.Optional[str] = None
     type: typing.Optional[str] = None
     executions: list[Execution] = []
+    error: typing.Optional[str] = None
 
 class Node(BaseModel):
     node_type: str

packages/node/octobot_node/scheduler/api.py

@@ -109,11 +109,14 @@ def _build_tasks_from_executions(
     tasks = []
     for parent_id, group in grouped.items():
         active = _get_active_execution(group)
+        error = active.error if active else None
         tasks.append(octobot_node.models.Task(
             id=parent_id,
             name=active.name if active else None,
             content=active.actions if active else None,
+            content_metadata=active.content_metadata if active else None,
             executions=group,
+            error=error,
         ))
     return tasks
 

packages/node/octobot_node/scheduler/encryption/task_outputs.py

@@ -18,7 +18,6 @@
 import base64
 
 from typing import Tuple, Optional
-from octobot_node.config import settings
 from octobot_node.scheduler.encryption import (
     ENCRYPTED_AES_KEY_B64_METADATA_KEY, 
     IV_B64_METADATA_KEY, 
@@ -31,20 +30,20 @@
 import octobot_commons.cryptography as cryptography
 
 
-def encrypt_task_result(result: str) -> Tuple[str, str]:
+def encrypt_task_result(result: str, rsa_public_key: bytes, ecdsa_private_key: bytes) -> Tuple[str, str]:
     aes_encryption_key = cryptography.generate_aes_key()
     iv = cryptography.generate_iv()
 
     encrypted_result = cryptography.aes_gcm_encrypt(result.encode('utf-8'), aes_encryption_key, iv)
     if not encrypted_result:
         raise EncryptionTaskError("Failed to encrypt result")
 
-    encrypted_aes_key = cryptography.rsa_encrypt_aes_key(aes_encryption_key, settings.TASKS_OUTPUTS_RSA_PUBLIC_KEY)
+    encrypted_aes_key = cryptography.rsa_encrypt_aes_key(aes_encryption_key, rsa_public_key)
     if not encrypted_aes_key:
         raise EncryptionTaskError("Failed to encrypt AES key")
 
     data_to_sign = encrypted_result + encrypted_aes_key + iv
-    signature = cryptography.sign_data(data_to_sign, settings.TASKS_OUTPUTS_ECDSA_PRIVATE_KEY)
+    signature = cryptography.sign_data(data_to_sign, ecdsa_private_key)
     if not signature:
         raise EncryptionTaskError("Failed to sign data")
 
@@ -57,7 +56,12 @@ def encrypt_task_result(result: str) -> Tuple[str, str]:
     return encrypted_result_b64, json.dumps(metadata)
 
 
-def decrypt_task_result(encrypted_result: str, metadata: Optional[str] = None) -> str:
+def decrypt_task_result(
+    encrypted_result: str,
+    rsa_private_key: bytes,
+    ecdsa_public_key: bytes,
+    metadata: Optional[str] = None,
+) -> str:
     if metadata is None:
         raise MissingMetadataError("No metadata provided for result decryption")
 
@@ -81,10 +85,10 @@ def decrypt_task_result(encrypted_result: str, metadata: Optional[str] = None) -
         raise MetadataParsingError(f"Failed to decode base64-encoded data: {e}")
 
     data_to_verify = encrypted_result_bytes + encrypted_aes_key + iv
-    if not cryptography.verify_signature(data_to_verify, settings.TASKS_OUTPUTS_ECDSA_PUBLIC_KEY, signature):
+    if not cryptography.verify_signature(data_to_verify, ecdsa_public_key, signature):
         raise SignatureVerificationError("Signature verification failed")
 
-    decrypted_aes_key = cryptography.rsa_decrypt_aes_key(encrypted_aes_key, settings.TASKS_OUTPUTS_RSA_PRIVATE_KEY)
+    decrypted_aes_key = cryptography.rsa_decrypt_aes_key(encrypted_aes_key, rsa_private_key)
     if not decrypted_aes_key:
         raise EncryptionTaskError("Failed to decrypt AES key")
 

packages/node/octobot_node/scheduler/scheduler.py

@@ -29,6 +29,7 @@
 import octobot_node.constants
 import octobot_node.scheduler.workflows_util as workflows_util
 import octobot_node.scheduler.workflows.params as workflow_params
+import octobot_node.scheduler.encryption as encryption
 try:
     from octobot import VERSION
 except ImportError:
@@ -208,10 +209,19 @@ async def get_results(self) -> list[octobot_node.models.Execution]:
                         except Exception as e:
                             self.logger.warning(f"Failed to parse output for workflow {completed_workflow_status.workflow_id}: {e}")
                             output = workflow_params.AutomationWorkflowOutput()
-                        result = output.state or task.content
+                        if output.state:
+                            result = output.state
+                            metadata = output.state_metadata
+                        else:
+                            result = task.content
+                            metadata = task.content_metadata
+                        if octobot_node.config.settings.is_node_side_encryption_enabled and result and metadata:
+                            try:
+                                result = encryption.decrypt_task_content(result, metadata)
+                            except Exception as decrypt_err:
+                                self.logger.warning(f"Failed to decrypt result for workflow {completed_workflow_status.workflow_id}: {decrypt_err}")
                         description = "Completed"
                         status = octobot_node.models.TaskStatus.COMPLETED
-                        metadata = task.content_metadata
                         task_name = task.name
                         error = output.error
                     else:
@@ -227,6 +237,7 @@ async def get_results(self) -> list[octobot_node.models.Execution]:
                         name=task_name,
                         description=description,
                         status=status,
+                        content_metadata=task.content_metadata if task else None,
                         result=result or "",
                         result_metadata=metadata,
                         scheduled_at=completed_workflow_status.created_at,
@@ -255,11 +266,13 @@ def _parse_workflow_status(
                 task_type = task.type
                 task_actions = task.content #todo confi
 
+        task_content_metadata = task.content_metadata if task else None
         return octobot_node.models.Execution(
             id=task_id,
             name=task_name,
             description=description,
             actions=task_actions,
+            content_metadata=task_content_metadata,
             type=task_type,
             status=status,
         )