-
Notifications
You must be signed in to change notification settings - Fork 27
ISSUE-690: Create AI model whitelist configuration #693
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
94bd5bf to
96d131b
Compare
donny-wong
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @Naragod , I think you misunderstood the requirements for this Issue. We specifically would like to whitelist what is allowed to put in the remote_url argument for the RemoteModel class.
1c5c352 to
a908525
Compare
Hi @donny-wong I have made some changes. Please take a look. |
Will mention here
Will document here what we spoke about: We will not use a whitelist file but instead put the list of allowed endpoint urls in the settings.yml file. |
34b2621 to
831c2b4
Compare
0cbcffe to
362ccd9
Compare
05f7be1 to
362ccd9
Compare
for more information, see https://pre-commit.ci
|
|
||
| from ..tester import Test, Tester | ||
| from ..specs import TestSpecs | ||
| from ...config import config as server_config |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm getting an import error from here when running the AI test. Did you encounter this?
Extra Info
[AI Feedback]
Messages on stderr:
Traceback (most recent call last):
File "<string>", line 1, in <module>
import sys, json; sys.path.append("/app/autotest_server"); from testers.ai.ai_tester import AiTester as Tester; from testers.specs import TestSpecs; Tester(resource_settings=[], specs=TestSpecs.from_json(sys.stdin.read())).run()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/autotest_server/testers/ai/ai_tester.py", line 7, in <module>
from ...config import config as server_config
ImportError: attempted relative import beyond top-level package
Description:
We wish to dynamically whitelist AI models available for use. To prevent the potential exposure of sensitive data when autograding/autotesting assignments, we wish to restrict usage to remote models.
Implementation
Allow users when creating an assignment the ability to submit a whitelist of available models. When the autotest is ran, any models used missing from this list will fail to execute.