Refined memory/cpu cost models for ValueData and UnValueData

[Unisay]

Context

• What: Replace constant memory costs with linear models for ValueData and UnValueData builtins
• Why: Existing constant memory costs (1 unit) don't reflect actual memory behavior, leading to inaccurate costing
• Approach: Empirical measurement using new memory-analysis tooling to derive accurate linear coefficients

Problem Statement

The current cost models for ValueData and UnValueData use constant memory costs of 1 unit, regardless of input size. This is inaccurate because:

• ValueData: Serializes a Value to Data, memory should scale with serialized size
• UnValueData: Deserializes Data to Value, memory should scale with node count in the Data structure

Inaccurate memory models can lead to budget misestimation in smart contracts.

Solution Approach

This PR implements a comprehensive solution in 6 logical commits:

1. Infrastructure: Add memory-analysis executable with plotting and regression utilities
2. Core Implementation: Introduce DataNodeCount newtype for node-based memory tracking
3. Type System Integration: Wire DataNodeCount into the DefaultUni type system
4. Builtin Application: Apply specialized wrappers (ValueTotalSize, DataNodeCount) to builtins
5. Benchmark Alignment: Update benchmarks to use the new memory wrappers
6. Cost Model Update: Replace constant costs with empirically-derived linear models

Memory Measurement Strategy

• ValueData: Use ValueTotalSize wrapper (already exists) - measures total serialized size
• UnValueData: Use new DataNodeCount wrapper - performs lazy node traversal of Data structure

This approach separates concerns:

• Memory measurement logic lives in ExMemoryUsage.hs (slope applied per node/byte)
• Cost coefficients live in JSON cost models (intercept + slope multiplier)

Design Decisions

Why node count for UnValueData?

• UnValueData converts Data → Value by traversing the Data tree structure
• Memory cost scales with tree complexity measured as node count
• Node count reflects the structural traversal work performed
• Lazy traversal (via CostRose) ensures accurate accounting

Why separate wrappers?

• Type safety: each wrapper represents a specific memory measurement strategy
• Flexibility: can tune coefficients independently via JSON
• Clarity: explicit in builtin signatures what memory model is used

Changes

Memory Analysis Tooling

New executable: plutus-benchmark:memory-analysis

• PlutusBenchmark.MemoryAnalysis: Main analysis framework
• PlutusBenchmark.MemoryAnalysis.Experiments: Memory behavior experiments
• PlutusBenchmark.MemoryAnalysis.Generators: Test data generators
• PlutusBenchmark.Plotting: Chart generation utilities
• PlutusBenchmark.RegressionInteger: Regression with asymmetric loss

This tooling enabled empirical measurement of memory behavior to derive the coefficients used in the cost models.

Core Memory Tracking

plutus-core/src/PlutusCore/Evaluation/Machine/ExMemoryUsage.hs

• Added DataNodeCount newtype wrapping Data
• Implemented ExMemoryUsage instance using countNodesRoseScaled
• Helper function performs lazy node traversal with slope-per-node accounting
• Language extensions: AllowAmbiguousTypes, BlockArguments, InstanceSigs, KindSignatures, ScopedTypeVariables

plutus-core/src/PlutusCore/Default/Universe.hs

• Added KnownTypeAst instance for DataNodeCount
• Added MakeKnownIn and ReadKnownIn instances for marshalling
• Minor refactoring: use void instead of (() <$) for clarity

Builtin Updates

plutus-core/src/PlutusCore/Default/Builtins.hs

• ValueData: Changed signature from Value -> Data to ValueTotalSize -> Data
• UnValueData: Changed signature from Data -> BuiltinResult Value to DataNodeCount -> BuiltinResult Value
• Both changes enable accurate memory accounting via specialized wrappers

Benchmark Alignment

plutus-core/cost-model/budgeting-bench/Benchmarks/Values.hs

• Updated valueDataBenchmark to use createOneTermBuiltinBenchWithWrapper with ValueTotalSize
• Updated unValueDataBenchmark to use createOneTermBuiltinBenchWithWrapper with DataNodeCount
• Ensures benchmarks measure the same memory behavior as production builtins

Cost Model Data

plutus-core/cost-model/data/builtinCostModel{A,B,C}.json

Updated memory models (all three variants updated identically):

"valueData": {
  "memory": {
    "arguments": {
      "intercept": 6,
      "slope": 38
    },
    "type": "linear_in_x"
  }
}

Memory = 38 × size + 6 (was constant 1)

"unValueData": {
  "cpu": {
    "arguments": {
      "intercept": 1000,
      "slope": 290658
    },
    "type": "linear_in_x"
  },
  "memory": {
    "arguments": {
      "intercept": 0,
      "slope": 8
    },
    "type": "linear_in_x"
  }
}

Memory = 8 × nodes + 0 (was constant 1)
CPU = 290658 × nodes + 1000 (updated from 43200 × arg + 1000)

plutus-core/cost-model/data/benching-conway.csv

Regenerated benchmark data (404 lines changed) with new memory measurement approach.

Impact

Budget Changes

Scripts using ValueData or UnValueData will see different memory budget consumption:

• Small values: Similar or slightly higher memory cost
• Large values: Significantly more accurate memory cost (linear scaling)
• CPU costs: UnValueData CPU costs updated to reflect node-based measurement

Conformance Tests

Expect budget differences in conformance tests that use these builtins. The new costs are more accurate than the previous constant models.

4. Memory Analysis

The memory-analysis executable can reproduce the experiments:

cabal run plutus-benchmark:memory-analysis

This generates plots and regression analysis in plutus-benchmark/memory-analysis/data/.

5. Conformance Tests

cabal test plutus-conformance

Expect budget differences but correct behavior.

Notes for Reviewers

Commit Structure

The PR is organized as 6 atomic commits following dependency order:

1. Infrastructure (analysis tools)
2. Core implementation (DataNodeCount)
3. Type system integration
4. Builtin application
5. Benchmark alignment
6. Cost model data

Each commit is buildable and represents a logical unit of change.

The updated CPU models for ValueData and UnValueData could be previewed here.

[ana-pantilie]

I don't think we should implement our own linear regression algorithm when R is the domain standard for this.