[WIP] [core] fix group offloading when using torchao#13276

sayakpaul wants to merge 1 commit intomainhuggingface/diffusers:mainfrom

fix-torchao-groupoffloadinghuggingface/diffusers:fix-torchao-groupoffloadingCopy head branch name to clipboard