Question about calvin inference

[Hitjss]

Thank you for sharing your excellent work!
When I evaluated the model provided for calvin [univla-7b-224-sft-calvin]using a single RTX 4090, I did not achieve the results reported in the paper (95.5 vs 55.6). Here is my config.

Could you please point out any issues or offer some suggestions? Thank you very much.

[PuzhenYuan]

@Hitjss
I am also working on evaluating univla on calvin. May I ask you how you configured your Calvin environment?
Do you use original univla environment (python 3.10) and run calvin's sh install.sh?
Or do you use default calvin environment (python 3.8) and install univla dependencies?
Any kind help is appreciated!

[retsuh-bqw]

@Hitjss You may refer to this issue to check whether it is a rendering issue caused by inappropriate environment installation.