Tensor paralleled circuit tracing should be naturally supported with sparse dictionary in TP. However, due to lack of ops supported in DTensor (e.g. index_put_, diagonal_backward, etc.), some manual intervention is required. The result consistency in TP and non-TP settings should also be checked.
Tensor paralleled circuit tracing should be naturally supported with sparse dictionary in TP. However, due to lack of ops supported in DTensor (e.g.
index_put_,diagonal_backward, etc.), some manual intervention is required. The result consistency in TP and non-TP settings should also be checked.