Huge amount of memory for spdz2k client communication scenario

[gionut]

Running spdz2k over the following program (https://github.com/gionut/PRIDA-MP-SPDZ/blob/main/PRIDA/Programs/Source/prida%2B.mpc). I'm expecting that the memory footprint would be similar to the estimated memory, however, it's much larger. For 500 clients the memory spikes up to 3GB, while the memory-usage script would predict only several MB. For 1500 clients, the kernel kills the processes due to OOM.

I was thinking the reason for the memory increase could be keeping the entire socket array in memory, so I was thinking to communicate with the clients in batches. This would probably allow me to run the 1500 clients use case. However, even for 500 clients, the memory usage seems way too large for what the program is doing.

To reproduce

# Clone repo and submodules
clone https://github.com/gionut/PRIDA-MP-SPDZ
cd PRIDA-MP-SPDZ
git submodule init 
git submodule update

# Copy clients to MP-SPDZ/ExternalIo directory
cp PRIDA/Programs/Source/data_owner.py ../MP-SPDZ/ExternalIo/data_owner.py
cp PRIDA/Programs/data_customer.py ../MP-SPDZ/ExternalIo/data_customer.py

# Start container from PRIDA-MP-SPDZ/PRIDA
docker run -d -it --name prida_plus --mount type=bind,source="$(pwd)"/logs,target=/usr/src/MP-SPDZ/logs --mount type=bind,source="$(pwd)"/Programs/Source,target=/usr/src/MP-SPDZ/Programs/Source --mount type=bind,source="$(pwd)"/Player-Data,target=/usr/src/MP-SPDZ/Player-Data --mount type=bind,source="$(pwd)"/../MP-SPDZ/ExternalIO,target=/usr/src/MP-SPDZ/ExternalIO --mount type=bind,source="$(pwd)"/Programs/Schedules,target=/usr/src/MP-SPDZ/Programs/Schedules --mount type=bind,source="$(pwd)"/Programs/Bytecode,target=/usr/src/MP-SPDZ/Programs/Bytecode mpspdz:spdz2k-party

# Setup SSL certificates
../MP-SPDZ/Scripts/setup-ssl.sh 2
../MP-SPDZ/Scripts/setup-clients.sh $(N)+$(M) > /dev/null 2>&1

# Running the computation
docker exec prida_plus sh -c "python Programs/Source/one-run.py -N 1000 -M 1 --batch-size 100 --timeout 60"
# one-run script starts the clients and the two main parties.

Outputs
Virtual machine outputs using --verbose

Using SPDZ2k security parameter 64
Using statistical security parameter 40
Trying to run 64-bit computation
Setup took 0.0771036 seconds.
Party 0 is listening on port 14000 for external client connections.
Listening for client connections on base port 14000
Party 0 received external client connection from client id: 37
..
Party 0 received external client connection from client id: 499
Party 0 received external client connection from client id: 500
Compiler: ./compile.py prida+ -M -E spdz2k 500 1
Main thread communication:
Broadcasting 0.00082 MB in 31 rounds, taking 0.00136305 seconds
Exchanging one-to-one 1.46389 MB in 12 rounds, taking 0.00199452 seconds
Receiving directly 0.004896 MB in 7 rounds, taking 0.000244258 seconds
Receiving one-to-one 1.37796 MB in 11 rounds, taking 0.0079064 seconds
Sending directly 0.004896 MB in 7 rounds, taking 0.000112279 seconds
Sending one-to-one 1.37796 MB in 11 rounds, taking 0.000588243 seconds
Sending/receiving 3.2e-05 MB in 2 rounds, taking 0.000912159 seconds
Client communication: 4.8e-05 MB in 2.57386 seconds and 1 rounds 
	99 triples of SPDZ2^(64+64) left
Significant amount of unused triples of SPDZ2^(64+64) distorting the benchmark. For more accurate benchmarks, consider reducing the batch size with --batch-size.
Client communication: 0.032 MB in 0.801504 seconds and 500 rounds 
Client communication: 0.032 MB in 0.763966 seconds and 500 rounds 
Detailed costs:
             1 integer multiplications
             4 integer openings
3 threads spent a total of 4.16787 seconds (0.000472 MB, 21 rounds) on the online phase, 0.398068 seconds (20.7251 MB, 362 rounds) on the preprocessing/offline phase, and 6.336 seconds idling.
Communication details (rounds and time in parallel threads counted double):
Broadcasting 0.002524 MB in 93 rounds, taking 0.00320992 seconds (at most 0.00146991)
Exchanging one-to-one 16.1025 MB in 116 rounds, taking 0.0709732 seconds (at most 0.0346893)
Receiving directly 0.052896 MB in 27 rounds, taking 0.00307216 seconds (at most 0.00211925)
Receiving one-to-one 4.56764 MB in 59 rounds, taking 0.0079064 seconds
Sending directly 0.052896 MB in 27 rounds, taking 0.000667386 seconds (at most 0.000382438)
Sending one-to-one 4.56764 MB in 59 rounds, taking 0.000588243 seconds
Sending/receiving 3.2e-05 MB in 2 rounds, taking 0.000912159 seconds
CPU time = 1.52198 (overall core time)
The following benchmarks are including preprocessing (offline phase).
Time = 3.63289 seconds 
Data sent = 20.7256 MB in ~383 rounds (party 0 only; rounds counted double due to multi-threading)
Global data sent = 41.4512 MB (all parties)
Actual preprocessing cost of program:
  Type int
          1001        Triples
             2        Randoms
             2          Opens
Coordination took 0.00209848 seconds
Command line: spdz2k-party.x prida+-500-1 -p 0 -h 0.0.0.0 -N 2 -v --batch-size 100

Memory Usage Scripts + Docker stats

# docker stats show the container uses 3GB RAM
# ./Scripts/memory-usage.py /prida+-500-1    
Memory:
     9708 regint
     10743 sint
      8192 cint
      8192 cgf2n
      8192 sgf2n
Registers in main thread:
       501 regint
         3 cint
         2 sint
Registers in other threads:
        11 regint
         3 cint
         5 sint
The program requires at least an estimated 0.000364-0.001094 GB of RAM per party.

MP-SPDZ version
0.4.2 (Dec 24, 2025)

[mkskeller]

This might be due to the memory used for SSL connections, which isn't accounted for in the memory usage estimate. You might be able to reduce this using closeclientconnection: https://mp-spdz.readthedocs.io/en/latest/instructions.html#Compiler.instructions.closeclientconnection

If you run all the clients on the same machine, you might also want to stagger them.

[gionut]

Hi! Thanks for the suggestion.

I've batched the client connections and managed to keep memory usage within reasonable bounds.

Do you have any insight into why the connections consume so much memory (~1MB per connection?) ? Is this related to how Linux handles sockets, or is it specific to the SSL implementation?

[mkskeller]

I don't know. You can further investigate this by putting MY_CFLAGS += -DNO_CLIENT_TLS into CONFIG.mine and the compiling the C++ code again, but this only works with C++ client.

[gionut]

Thanks!
I'll try this and come back when I have new results.