feat: delegation program should handle large accounts and large data size increases during commits

[thlorenz]

I got the committor tests including proccess commit with the delegation program to basically
work for small accounts, but ran into the following issue with larger ones:

First Attempt

1. Create small FlexiCounter
2. Delegate it
3. Increase the data size by 1MB (or whichever requires multiple reallocs)
4. Successfully create the commit buffer account (using reallocs to make it work) and write
   all data chunks to it
5. Process the commit buffer to write the chunks to the original account on chain
   => InvalidRealloc

Here the delegation program is unable to increase the size of the on chain account enough to
hold the new amounts of data.

Second Attempt

So I figured for now to get the test to work I'll try to increase the size of the account before
delegation and did the following:

1. Create small FlexiCounter
2. Airdrop to it so it can hold more data
3. Run multiple reallocs to increase the size of the account
4. Delegate it
   => InvalidRealloc

Due to creating the temporary buffer for the owner change:

// Create the Buffer PDA
create_pda(
    accounts.buffer,
    accounts.owner_program.key,
    data_len,
    buffer_signer_seeds,
    accounts.system_program,
    accounts.payer,
)?;

which errors below:

https://github.com/magicblock-labs/ephemeral-rollups-sdk/blob/main/rust/sdk/src/utils.rs#L56

Summary

There are multiple issues here:

1. It is currently impossible to delegate large accounts
2. When committing account changes, the data size cannot change by more than 10_240 bytes

Solutions

1. can be solved as follows

When trying to delegate a very large account we need to prepare the temporary buffer account in
separate instructions (possibly requiring multiple transactions).
However that would require to temporarily pay the rent until we can close it again after
delegation completes

2. can be solved as follows

We can prealloc the extra space needed before committing to the account. However that would
make the commit less transactional since the realloc is not part of the actual commit
instruction.
Thus we could end up running the reallocs but never commit.
That might be acceptable

Solving both via a limitation

We could also limit the size of the delegated accounts to a certain size and require the user to split
data accordingly. We also could enforce a max account size of writables in our validator.

At that point we wouldn't need the whole strategy of reallocs I added to the committor and
could just commit the data directly to the account.

[Vinayapr23]

Is this implemented
If no one is working on it can i take a look

[GabrielePicco]

Closing as duplicate of #80.

Thanks @Vinayapr23, this is a potentially major change, if you are interested, feel free to propose a potential implementation opening a discussion