chore: fix test caching #6469

Job	Run time
E2E tests (ubuntu-latest)	7m 42s
E2E tests (windows-latest)	15m 7s
E2E tests (macos-latest)	9m 20s
Build	1m 25s
Unit tests (windows-latest)	2m 24s
Unit tests (ubuntu-latest)	1m 12s
Integration tests (macos-latest)	1m 59s
Unit tests (macos-latest)	1m 32s
Format check	39s
Integration tests (windows-latest)	3m 14s
Linter	2m 22s
Integration tests (ubuntu-latest)	1m 36s
	48m 32s

Provide feedback