0.8.8 - Bug fixes: IN clause, RETURNING clause, not Santa Claus

Fixed

• IN Clause Datatype Mismatch - Fixed issue #63 where IN clauses with parameterised lists caused datatype mismatch errors due to automatic JSON encoding of lists
• SQL Comment Query Detection - Fixed Protocol.UndefinedError when queries start with SQL comments (both -- and /* */ styles) by properly skipping comments before detecting query type
• RETURNING Clause for update_all/delete_all - Added RETURNING clause generation when using update_all/delete_all with select clauses, fixing Protocol.UndefinedError with Oban job fetching

Changed

• Removed Unsupported Replication Tests - Removed replication integration tests that were testing unsupported features

Full Changelog: 0.8.7...0.8.8

0.8.7 - R*Tree spatial indexing, CHECK constraints, better DEFAULT and type handling

Added

• CHECK Constraint Support - Column-level CHECK constraints in migrations
• R*Tree Spatial Indexing - Full support for SQLite R*Tree virtual tables with 1D-5D indexing, validation, and comprehensive test coverage
• ecto_sqlite3 Compatibility Test Suite - Comprehensive tests ensuring feature parity with ecto_sqlite3
• Type Encoding Improvements - Automatic JSON encoding for plain maps, DateTime/Decimal parameter encoding, improved type coercion
• Comprehensive Type Loader/Dumper Support - Full support for encoding/decoding temporal types (DateTime, NaiveDateTime, Date, Time), Decimal, and special nil values with proper ISO 8601 formatting
• Default Value Type Handling - Support for Decimal, DateTime, NaiveDateTime, Date, Time, and :null as default values in migrations with warning logging for unsupported types
• Connection Recovery Testing - Test suite for connection failure scenarios and recovery patterns
• Query Encoding Improvements - Explicit test coverage for query parameter encoding with various data types and edge cases

Fixed

• DateTime Microsecond Type Loading - Fixed :utc_datetime_usec, :naive_datetime_usec, and :time_usec loading from ISO 8601 strings with microsecond precision
• Parameter Encoding - Automatic map-to-JSON conversion, DateTime/Decimal encoding for compatibility with Oban and other libraries
• Migration Robustness - Handle :serial/:bigserial types, improved default value handling with warnings for unsupported types
• JSON and RETURNING Clauses - Fixed JSON encoding in RETURNING queries and datetime function calls
• Test Isolation - Comprehensive database cleanup across all test suites, per-test table clearing, improved resource management
• DateTime Type Handling - Fixed datetime_decode to handle timezone-aware ISO 8601 strings and nil value encoding for date/time/bool types
• Decimal Type Handling - Updated assertions to accept both numeric and string representations of decimal values in database queries
• Datetime Roundtrip Preservation - Strengthened microsecond precision preservation in datetime round-trip tests

Changed

• Test Suite Consolidation - Streamlined and improved test organisation with better coverage of edge cases, error handling, and concurrent operations
• Documentation - Updated documentation with SQLite-specific query limitations, compatibility testing results, and guidance for type encoding edge cases

Full Changelog: 0.8.6...0.8.7

0.8.6 - JSON, UPSERT, EXPLAIN, and Rust fixes

Release v0.8.6

Major Features

Named Parameters Execution Support

Full support for SQLite named parameter syntax (:name, @name, $name) in prepared statements and direct execution. Parameters are transparently converted from maps to positional arguments internally.

# Use named parameters in queries
{:ok, _, result, state} = EctoLibSql.handle_execute(
  "SELECT * FROM users WHERE email = :email AND status = :status",
  %{"email" => "alice@example.com", "status" => "active"},
  [],
  state
)

Works seamlessly with prepared statements, transactions, batch operations, and cursor streaming.

EXPLAIN QUERY PLAN Support

Full support for SQLite's EXPLAIN QUERY PLAN via Ecto's Repo.explain/2 and Repo.explain/3. Returns structured query plans for optimisation and debugging.

{:ok, plan} = Repo.explain(:all, from(u in User, where: u.active == true))
# Returns: [%{"id" => 2, "parent" => 0, "notused" => 0, "detail" => "SCAN users"}]

CTE (Common Table Expression) Support

Full support for SQL WITH clauses including recursive CTEs. Enables complex hierarchical queries and improved query organisation.

query = "hierarchy"
        |> with_cte("hierarchy", as: ^base_query)
        |> recursive_ctes(true)
        |> select([h], h.name)
Repo.all(query)

Query-Based UPSERT Support

Extended on_conflict support to handle query-based updates with keyword list syntax for dynamic operations.

Repo.insert(changeset,
  on_conflict: [set: [name: "updated", updated_at: DateTime.utc_now()]],
  conflict_target: [:email]
)

STRICT Table Option

Added support for SQLite's STRICT table option for stronger type enforcement at INSERT/UPDATE time.

create table(:users, options: [strict: true]) do
  add :name, :string
  add :age, :integer
end

Security Enhancements

CVE-2025-47736 Protection

Defence-in-depth measures against SQL injection via named parameters:

• Comprehensive parameter validation to prevent atom table exhaustion
• Improved parameter extraction to avoid malicious input exploitation
• Validates all named parameters against statement introspection
• Proper error handling for invalid or malicious parameter names

See SECURITY.md for full details.

Bug Fixes & Improvements

Statement Caching

• Replaced unbounded persistent_term cache with bounded ETS LRU cache
• Prevents memory leaks from unlimited prepared statement caching
• Configurable cache size with automatic eviction

Error Handling

• Propagate parameter introspection errors instead of silently falling back
• Descriptive errors for invalid argument types
• Improved error messages throughout

Code Quality

• Fixed all Credo warnings
• Improved test reliability and coverage
• Better state threading and error handling
• Removed redundant UTF-8 validation code

Documentation

• Added generated/computed columns documentation
• Enhanced JSON/JSONB function documentation
• Comprehensive test coverage for all new features
• Cross-connection security test suite

🔗 Resources

• CHANGELOG - Full changelog
• AGENTS.md - API reference
• SECURITY.md - Security policy

---

Full Changelog: 0.8.3...0.8.6

0.8.3 - SQLite extensions & fuzz testing

v0.8.3 Release Notes

New Features

RANDOM ROWID Support (libSQL Extension)

• Generate pseudorandom row IDs instead of sequential integers for security/privacy
• Prevents ID enumeration attacks and leaking business metrics
• Usage: create table(:sessions, options: [random_rowid: true])

SQLite Extension Loading

• Load SQLite extensions dynamically via enable_extensions/2 and load_ext/3
• Supports FTS5, JSON1, R-Tree, PCRE, and custom extensions
• Security-first: disabled by default, must be explicitly enabled

Enhanced Statement Introspection

• stmt_parameter_name/3 - Get named parameter names (:name, @name, $name)
• reset_stmt/2 - Explicitly reset statements for efficient reuse
• get_stmt_columns/2 - Get full column metadata (name, origin, declared type)

Remote Encryption Support

• New remote_encryption_key option for Turso encrypted databases
• Works alongside existing local encryption_key for end-to-end encryption

Quality & Testing

• Added Credo, Dialyxir, and Sobelow for comprehensive Elixir code analysis
• Property-based fuzz testing with StreamData (SQL injection, transactions, edge cases)
• Rust fuzz testing infrastructure with cargo-fuzz
• Ported key tests from Ecto.Adapters.SQL for compatibility verification
• Modernised Rust code: std::sync::LazyLock, stricter Clippy lints

Fixes

• SQL injection prevention in Pragma module table name validation
• Dialyzer type error in disconnect/2 spec
• Improved fuzz test stability for savepoints and binary data

Changelog: https://github.com/ocean/ecto_libsql/blob/main/CHANGELOG.md
Full Changelog: 0.8.1...0.8.3

0.8.1 - Constraints bug fix

Fixed

• Constraint Error Handling: Index Name Reconstruction (Issue #34)
• Improved constraint name extraction to reconstruct full index names from SQLite error messages
• Now follows Ecto's naming convention: table_column1_column2_index
• Single-column constraints: "UNIQUE constraint failed: users.email" → "users_email_index" (previously just "email")
• Multi-column constraints: "UNIQUE constraint failed: users.slug, users.parent_slug" → "users_slug_parent_slug_index"
• Backtick handling: Properly strips trailing backticks appended by libSQL to error messages
• Enhanced error messages: Preserves custom index names from enhanced format (index: custom_index_name)
• NOT NULL constraints: Reconstructs index names following same convention
• Enables accurate unique_constraint/3 and check_constraint/3 matching with custom index names in Ecto changesets

Full Changelog: 0.8.0...0.8.1

0.8.0 - Rust refactor

Overview

Major code refactoring and critical thread safety fixes with zero breaking changes.

Key Changes

Code Refactoring

• Modularised Rust codebase: Split 2,302-line monolithic lib.rs into 13 focused modules (connection, query, batch, statement, transaction, savepoint, cursor, replication, metadata, utils, constants, models, decode)
• Reorganised test suite: Refactored 1,194-line tests.rs into structured modules (integration, constants, utils)
• Improved maintainability: Better code navigation and contributor onboarding with clearer separation of concerns
• Zero behaviour changes: All APIs and functionality preserved

Thread Safety & Performance Fixes

• Registry lock management: Fixed all functions to drop registry locks before async operations (prevents deadlocks)
• Scheduler annotations: Added #[rustler::nif(schedule = "DirtyIo")] to blocking NIFs
• Atom naming consistency: Fixed remote_primary → remote atom mismatch
• Runtime optimisation: Use shared global TOKIO_RUNTIME instead of creating per-connection (prevents resource exhaustion)
• Replication performance: Eliminated unnecessary async overhead for synchronous operations

Bug Fixes

• Fixed prepared statement column introspection tests (enabled previously skipped tests)
• Enhanced constraint error message handling with index name support
• Improved remote Turso test stability
• Better error handling for allocation failures
• Proper SQL identifier quoting in PRAGMA queries

---

Note: This release is fully backward compatible. The refactoring is purely organisational with performance and stability improvements under the hood.

• Rust refactoring, fix constraint error message handling by @ocean in #33

Full Changelog: 0.7.5...0.8.0

0.7.5 - query routing bug fix & performance improvements

Release 0.7.5 - Query Routing & Performance Improvements

This release focuses on critical bug fixes for batch operations and significant performance optimisations.

Fixed

Query/Execute Routing for Batch Operations

• Implemented proper query() vs execute() routing in batch operations based on statement type
• execute_batch() now automatically detects SELECT and RETURNING clauses to use the correct LibSQL method
• execute_transactional_batch() applies the same intelligent routing logic for atomic operations
• execute_batch_native() and execute_transactional_batch_native() now properly route SQL batch execution
• Eliminates "Statement does not return data" errors for operations that should return rows
• All operations with RETURNING clauses now correctly use the query() method

Performance: Batch Operation Optimisations

• Eliminated per-statement argument clones in batch operations for better memory efficiency
• Changed batch_stmts.iter() to batch_stmts.into_iter() to consume vectors by value
• Removed unnecessary args.clone() calls in both transactional and non-transactional batches
• Reduces memory allocations during batch execution for improved throughput

Lock Coupling Reduction

• Dropped outer LibSQLConn mutex guard earlier in batch operations to reduce contention
• Extract inner Arc<Mutexlibsql::Connection> before entering async blocks
• Only hold inner connection lock during actual I/O operations
• Applied to all four batch operation variants:
  • execute_batch()
  • execute_transactional_batch()
  • execute_batch_native()
  • execute_transactional_batch_native()
• Reduces contention and deadlock surface area by following the established pattern from query_args()

Test Coverage & Documentation

• Enhanced should_use_query() test coverage for block comment handling
• Added explicit assertion documenting known limitation: RETURNING in block comments detected as false positive (safe behaviour)
• Documented CTE and EXPLAIN detection limitations with clear scope notes
• Added comprehensive future improvement recommendations with priority levels and implementation sketches
• Added performance budget notes for optimisation efforts

Impact

• Correctness: Batch operations with RETURNING clauses now work correctly
• Performance: Reduced memory allocations and lock contention in batch operations
• Reliability: Lower deadlock risk through improved lock coupling patterns
• Maintainability: Better test coverage and documentation for edge cases

Full Changelog: 0.7.0...0.7.5

0.7.0

🚀 EctoLibSql v0.7.0 - Performance & Feature Release

Major performance improvements, new connection management features, and comprehensive SQLite configuration support.

🐛 Bug Fixes

• ON CONFLICT Support (Issue #25) - Full upsert support with composite unique indexes

  • :on_conflict: :nothing with conflict targets
  • :on_conflict: :replace_all for upserts
  • Custom field replacement with {fields, _, targets}

• Binary ID Type System (Issue #23) - Complete resolution

  • Fixed autogenerate(:binary_id) to generate string UUIDs
  • Fixed INSERT without RETURNING clause (was causing CaseClauseError)
  • Fixed BLOB encoding in Rust NIF (now returns Elixir binaries)

• Remote Test Stability - Fixed vector operations test failures from stale data

⚡ Performance Improvements

Prepared Statement Caching - Statements are now cached and reused with automatic binding cleanup, delivering ~10-15x faster query execution for repeated queries. Benchmark: 100 cached executions in ~33ms (~330µs per query).

✨ New Features

Connection Management

• busy_timeout/2 - Configure database lock timeout (default: 5000ms)
• reset/1 - Reset connection state without closing
• interrupt/1 - Cancel long-running queries

PRAGMA Configuration

New EctoLibSql.Pragma module with comprehensive SQLite configuration:

• Foreign keys control (enable_foreign_keys/1, disable_foreign_keys/1)
• Journal mode (set_journal_mode/2 - WAL, DELETE, MEMORY, etc.)
• Cache size (set_cache_size/2)
• Synchronous level (set_synchronous/2 - OFF, NORMAL, FULL, EXTRA)
• Table introspection (table_info/2, table_list/1)
• User version tracking (user_version/1, set_user_version/2)

Transaction Features

• Savepoints - Nested transaction support with partial rollback
  • create_savepoint/2 - Create named savepoint
  • release_savepoint_by_name/2 - Commit savepoint changes
  • rollback_to_savepoint_by_name/2 - Rollback to savepoint
• Transaction Ownership Validation - Prevents cross-connection transaction manipulation

Batch Operations

• execute_batch_sql/2 - Execute multiple SQL statements (non-transactional)
• execute_transactional_batch_sql/2 - Atomic batch execution

Prepared Statement Introspection

• stmt_parameter_count/2 - Get number of parameters
• stmt_column_count/2 - Get number of result columns
• stmt_column_name/3 - Get column name by index

Advanced Replica Sync

• get_frame_number_for_replica/1 - Monitor replication frame
• sync_until_frame/2 - Wait for specific frame (30s timeout)
• flush_and_get_frame/1 - Push pending writes
• max_write_replication_index/1 - Read-your-writes consistency tracking

🔒 Security Fixes

• SQL Injection Prevention - Added strict alphanumeric validation for savepoint names
• Prepared Statement Validation - Enhanced parameter binding checks

📚 Documentation

• Updated AGENTS.md with all new features
• Updated README.md with concise examples

🔄 Backward Compatibility

All changes are backward compatible. Prepared statement API unchanged - caching happens transparently with improved performance.

Full Changelog: https://github.com/ocean/ecto_libsql/blob/main/CHANGELOG.md
Full list of changes: 0.6.0...0.7.0

0.6.0 - Migrations, remote replicas, better libSQL support

What's Changed

• fix: Fix panic in prepared statement exec, add streaming support by @ocean in #19
• fix: Fix Ecto migrations bug for issue #20 and add some more tests by @ocean in #21
• feature: Add some extended libSQL support for ALTER TABLE etc by @ocean in #22

• Remote Sync Performance & Reliability
  • Removed redundant manual .sync() calls after write operations for embedded replicas
  • LibSQL automatically handles sync to remote primary database - manual syncs were causing double-sync overhead
  • Added 30-second timeout to connection establishment to prevent indefinite hangs
  • All Turso remote tests now pass reliably (previously 4 tests timed out)
  • Test suite execution time improved significantly (~107s vs timing out at 60s+)

According to Turso documentation: "Writes are sent to the remote primary database by default, then the local database updates automatically once the remote write succeeds." Manual sync is only needed when explicitly pulling down changes from remote (e.g., after reconnecting to an existing replica).

• Ecto Migrations Compatibility (Issue #20)

  • Fixed DDL function grouping that was causing compilation errors
  • Added comprehensive migration test suite (759 lines) covering all SQLite ALTER TABLE operations
  • Improved handling of SQLite's limited ALTER TABLE support
  • Added tests for column operations, constraint management, and index creation

• Prepared Statement Execution

  • Fixed panic in prepared statement execution that could crash the BEAM VM
  • Added proper error handling for prepared statement operations
  • Improved error messages for prepared statement failures

• Extended LibSQL DDL Support

  • Added support for additional ALTER TABLE operations compatible with LibSQL
  • Improved DDL operation grouping and execution order
  • Better handling of SQLite dialect quirks

Added

• Cursor Streaming Support

  • Implemented cursor-based streaming for large result sets
  • Added handle_declare/4, handle_fetch/4, and handle_deallocate/4 DBConnection callbacks
  • Memory-efficient processing of large queries
  • Rust NIF functions: declare_cursor/3, fetch_cursor/2, cursor registry management

• Comprehensive Test Coverage

  • Added 138 new DDL generation tests in test/ecto_connection_test.exs
  • Added 759 lines of migration tests in test/ecto_migration_test.exs
  • Improved error handling test coverage
  • All 162 tests passing (0 failures)

Changed

• Sync Behaviour for Embedded Replicas

  • Automatic sync after writes has been removed (LibSQL handles this natively)
  • Manual sync() via EctoLibSql.Native.sync/1 still available for explicit control
  • Improved sync timeout handling with configurable DEFAULT_SYNC_TIMEOUT_SECS (30s)
  • Added connection timeout to prevent hangs during initial replica sync

• Documentation Updates

  • Updated all documentation to reflect sync behaviour changes
  • Added clarification about when manual sync is needed vs automatic
  • Improved Turso/LibSQL compatibility documentation references

Technical Details

Sync Performance Before:

• Manual .sync() called after every write operation
• Double sync overhead (LibSQL auto-sync + manual sync)
• 120-second timeout causing long test hangs
• 4 tests timing out after 60+ seconds each

Sync Performance After:

• LibSQL's native auto-sync used correctly
• No redundant manual sync calls
• 30-second connection timeout for fast failure
• All tests passing in ~107 seconds

Migration Notes

This is a non-breaking change for normal usage. However, if you were relying on automatic sync behaviour after writes in embedded replica mode, you may now need to explicitly call EctoLibSql.Native.sync/1 when you need to ensure remote data is pulled down (e.g., after reconnecting to an existing local database).

Recommended Actions:

1. Review code that uses embedded replicas with sync: true
2. Add explicit sync() calls after reconnecting to existing local databases if you need to pull down remote changes
3. Remove any redundant manual sync() calls after write operations

Full Changelog: 0.5.0...0.6.0

0.5.0 - Massively improve Rust error handling

Changed

• Rust NIF Error Handling
  • Eliminated all 146 unwrap() calls from production Rust code
  • Added safe_lock() and safe_lock_arc() helper functions for safe mutex locking
  • All NIF errors now return {:error, message} tuples to Elixir instead of panicking
  • Mutex poisoning errors are handled gracefully with descriptive context
  • Invalid connection/transaction/statement/cursor IDs return proper errors

Fixed

• VM Stability - NIF errors no longer crash the entire BEAM VM
  • Invalid operations (bad connection IDs, missing resources) now return error tuples
  • Processes survive NIF errors, allowing supervision trees to work properly
  • Error messages include descriptive context for easier debugging

Added

• Comprehensive Error Handling Tests
  • Added test/error_demo_test.exs with 7 tests demonstrating graceful error handling
  • Added test/error_handling_test.exs with 14 comprehensive error coverage tests
  • All tests verify that NIF errors return proper error tuples instead of crashing the BEAM VM

Technical Details

Before 0.5.0:

• 146 unwrap() calls in Rust production code
• Mutex/registry errors → panic → entire BEAM VM crash
• Invalid IDs → panic → VM crash
• Supervision trees ineffective for NIF errors

After 0.5.0:

• 0 unwrap() calls in Rust production code (100% eliminated)
• All errors return {:error, "descriptive message"} tuples
• Processes can handle errors and recover
• Supervision trees work as expected

Migration Guide

This is a non-breaking change for normal Ecto usage. Your existing code will continue to work exactly as before, but is now significantly more stable.

What Changed:

• NIF functions that previously panicked now return {:error, reason} tuples
• Your existing error handling code will now catch errors that previously crashed the VM

Recommended Actions:

1. Review error handling in code that uses EctoLibSql.Native functions directly
2. Ensure supervision strategies are in place for database operations
3. Consider adding retry logic for transient errors (connection timeouts, etc.)

Notes

This release represents a major stability improvement for production deployments. The refactoring ensures that ecto_libsql handles errors the "Elixir way" - returning error tuples that can be supervised, rather than panicking at the Rust level and crashing the VM.