Skip to content

Conversation

@Naragod
Copy link
Contributor

@Naragod Naragod commented Jan 13, 2026

Description:

We wish to dynamically whitelist AI models available for use. To prevent the potential exposure of sensitive data when autograding/autotesting assignments, we wish to restrict usage to remote models.

Implementation

Allow users when creating an assignment the ability to submit a whitelist of available models. When the autotest is ran, any models used missing from this list will fail to execute.

@Naragod Naragod added this to the v2.9.1 milestone Jan 13, 2026
@Naragod Naragod requested a review from donny-wong January 13, 2026 19:04
@Naragod Naragod force-pushed the ISSUE-690_enable_ai_model_whitelist_configuration branch 2 times, most recently from 94bd5bf to 96d131b Compare January 13, 2026 19:13
Copy link
Contributor

@donny-wong donny-wong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Naragod , I think you misunderstood the requirements for this Issue. We specifically would like to whitelist what is allowed to put in the remote_url argument for the RemoteModel class.

@Naragod Naragod force-pushed the ISSUE-690_enable_ai_model_whitelist_configuration branch from 1c5c352 to a908525 Compare February 1, 2026 00:26
@Naragod
Copy link
Contributor Author

Naragod commented Feb 1, 2026

Hi @Naragod , I think you misunderstood the requirements for this Issue. We specifically would like to whitelist what is allowed to put in the remote_url argument for the RemoteModel class.

Hi @donny-wong I have made some changes. Please take a look.

@donny-wong
Copy link
Contributor

Hi @Naragod , I think you misunderstood the requirements for this Issue. We specifically would like to whitelist what is allowed to put in the remote_url argument for the RemoteModel class.

Hi @donny-wong I have made some changes. Please take a look.

Will mention here

Hi @Naragod , I think you misunderstood the requirements for this Issue. We specifically would like to whitelist what is allowed to put in the remote_url argument for the RemoteModel class.

Hi @donny-wong I have made some changes. Please take a look.

Will document here what we spoke about: We will not use a whitelist file but instead put the list of allowed endpoint urls in the settings.yml file.

@Naragod Naragod force-pushed the ISSUE-690_enable_ai_model_whitelist_configuration branch from 34b2621 to 831c2b4 Compare February 2, 2026 16:56
@Naragod Naragod force-pushed the ISSUE-690_enable_ai_model_whitelist_configuration branch from 0cbcffe to 362ccd9 Compare February 2, 2026 18:03
@Naragod Naragod requested a review from donny-wong February 3, 2026 15:25
@Naragod Naragod force-pushed the ISSUE-690_enable_ai_model_whitelist_configuration branch from 05f7be1 to 362ccd9 Compare February 6, 2026 16:32
@Naragod Naragod requested a review from david-yz-liu February 6, 2026 16:33

from ..tester import Test, Tester
from ..specs import TestSpecs
from ...config import config as server_config
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm getting an import error from here when running the AI test. Did you encounter this?

Extra Info
[AI Feedback]
Messages on stderr: 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import sys, json; sys.path.append("/app/autotest_server"); from testers.ai.ai_tester import AiTester as Tester; from testers.specs import TestSpecs; Tester(resource_settings=[], specs=TestSpecs.from_json(sys.stdin.read())).run()
                                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/autotest_server/testers/ai/ai_tester.py", line 7, in <module>
    from ...config import config as server_config
ImportError: attempted relative import beyond top-level package

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create a configuration setting for a list of allowed remote AI models

2 participants