Refactoring of status texts + add help tooltips

[prandla]

this is an implementation of #1347. it took me this long to get around to it because i wanted to implement it Properly, so i refactored how the messages are stored in the DB, in order to store the messages as IDs instead of english strings. the IDs look like "evaluation:timelimit", a namespace and a message id in the namespace. (i defined four namespaces, compilation/evaluation/execution/outputonly. these correspond to the existing two MessageCollections, and I moved some stray messages into the new namespaces).

this has the added benefit that we can change the english strings without breaking old statuses.

I might want to move the help-button-rendering to some other function, and in any case i need to write DB migrations and fix the test suite, but i thought i'd ask for comments about this general design before doing that.

on second thoughts maybe the namespacing stuff is a bit overcomplicated and we can just merge all the messages into one big MessageCollection.

[codecov]

❌ 16 Tests Failed:

Tests completed	Failed	Passed	Skipped
695	16	679	8

View the top 3 failed test(s) by shortest run time

cmstestsuite/unit_tests/grading/init_test.py::TestFormatStatusText::test_success_with_placeholders

Stack Traces | 0.001s run time

self = <cmstestsuite.unit_tests.grading.init_test.TestFormatStatusText testMethod=test_success_with_placeholders>

    def test_success_with_placeholders(self):
>       self.assertEqual(format_status_text(["%s", "ASD"]), "ASD")
E       AssertionError: 'N/A' != 'ASD'
E       - N/A
E       + ASD

.../unit_tests/grading/init_test.py:48: AssertionError

cmstestsuite/unit_tests/grading/init_test.py::TestFormatStatusText::test_success_with_translator

Stack Traces | 0.001s run time

self = <cmstestsuite.unit_tests.grading.init_test.TestFormatStatusText testMethod=test_success_with_translator>

    def test_success_with_translator(self):
        self.assertEqual(format_status_text([""], self._tr), "")
>       self.assertEqual(format_status_text(["ASD"], self._tr), "ESD")
E       AssertionError: 'N/E' != 'ESD'
E       - N/E
E       + ESD

.../unit_tests/grading/init_test.py:54: AssertionError

cmstestsuite/unit_tests/grading/steps/trusted_test.py::TestExtractOutcomeAndText::test_following_lines_ignored

Stack Traces | 0.001s run time

self = <cmstestsuite.unit_tests.grading.steps.trusted_test.TestExtractOutcomeAndText testMethod=test_following_lines_ignored>

    def test_following_lines_ignored(self):
        self.sandbox.fake_file("o", b"0.45\nNothing\n")
        self.sandbox.fake_file("e", b"Text to return.\nto see here")
        outcome, text = extract_outcome_and_text(self.sandbox)
        self.assertEqual(outcome, 0.45)
>       self.assertEqual(text, ["Text to return."])
E       AssertionError: Lists differ: ['custom:Text to return.'] != ['Text to return.']
E       
E       First differing element 0:
E       'custom:Text to return.'
E       'Text to return.'
E       
E       - ['custom:Text to return.']
E       ?   -------
E       
E       + ['Text to return.']

.../grading/steps/trusted_test.py:59: AssertionError

cmstestsuite/unit_tests/grading/steps/trusted_test.py::TestExtractOutcomeAndText::test_success

Stack Traces | 0.001s run time

self = <cmstestsuite.unit_tests.grading.steps.trusted_test.TestExtractOutcomeAndText testMethod=test_success>

    def test_success(self):
        self.sandbox.fake_file("o", b"0.45\n")
        self.sandbox.fake_file("e", "你好.\n".encode("utf-8"))
        outcome, text = extract_outcome_and_text(self.sandbox)
        self.assertEqual(outcome, 0.45)
>       self.assertEqual(text, ["你好."])
E       AssertionError: Lists differ: ['custom:你好.'] != ['你好.']
E       
E       First differing element 0:
E       'custom:你好.'
E       '你好.'
E       
E       - ['custom:你好.']
E       + ['你好.']

.../grading/steps/trusted_test.py:52: AssertionError

cmstestsuite/unit_tests/grading/steps/trusted_test.py::TestExtractOutcomeAndText::test_text_is_stripped

Stack Traces | 0.001s run time

self = <cmstestsuite.unit_tests.grading.steps.trusted_test.TestExtractOutcomeAndText testMethod=test_text_is_stripped>

    def test_text_is_stripped(self):
        self.sandbox.fake_file("o", b"   0.45\t \nignored")
        self.sandbox.fake_file("e", b"\t  Text to return.\r\n")
        outcome, text = extract_outcome_and_text(self.sandbox)
        self.assertEqual(outcome, 0.45)
>       self.assertEqual(text, ["Text to return."])
E       AssertionError: Lists differ: ['custom:Text to return.'] != ['Text to return.']
E       
E       First differing element 0:
E       'custom:Text to return.'
E       'Text to return.'
E       
E       - ['custom:Text to return.']
E       ?   -------
E       
E       + ['Text to return.']

.../grading/steps/trusted_test.py:73: AssertionError

cmstestsuite/unit_tests/grading/steps/trusted_test.py::TestExtractOutcomeAndText::test_text_is_translated

Stack Traces | 0.001s run time

self = <cmstestsuite.unit_tests.grading.steps.trusted_test.TestExtractOutcomeAndText testMethod=test_text_is_translated>

    def test_text_is_translated(self):
        self.sandbox.fake_file("o", b"0.45\n")
        self.sandbox.fake_file("e", b"translate:success\n")
        outcome, text = extract_outcome_and_text(self.sandbox)
        self.assertEqual(outcome, 0.45)
>       self.assertEqual(text, ["Output is correct"])
E       AssertionError: Lists differ: ['evaluation:success'] != ['Output is correct']
E       
E       First differing element 0:
E       'evaluation:success'
E       'Output is correct'
E       
E       - ['evaluation:success']
E       + ['Output is correct']

.../grading/steps/trusted_test.py:80: AssertionError

cmstestsuite/unit_tests/grading/steps/trusted_test.py::TestExtractOutcomeAndText::test_works_without_newlines

Stack Traces | 0.001s run time

self = <cmstestsuite.unit_tests.grading.steps.trusted_test.TestExtractOutcomeAndText testMethod=test_works_without_newlines>

    def test_works_without_newlines(self):
        self.sandbox.fake_file("o", b"0.45")
        self.sandbox.fake_file("e", b"Text to return.")
        outcome, text = extract_outcome_and_text(self.sandbox)
        self.assertEqual(outcome, 0.45)
>       self.assertEqual(text, ["Text to return."])
E       AssertionError: Lists differ: ['custom:Text to return.'] != ['Text to return.']
E       
E       First differing element 0:
E       'custom:Text to return.'
E       'Text to return.'
E       
E       - ['custom:Text to return.']
E       ?   -------
E       
E       + ['Text to return.']

.../grading/steps/trusted_test.py:66: AssertionError

cmstestsuite/unit_tests/grading/init_test.py::TestFormatStatusText::test_success_no_placeholders

Stack Traces | 0.002s run time

self = <cmstestsuite.unit_tests.grading.init_test.TestFormatStatusText testMethod=test_success_no_placeholders>

    def test_success_no_placeholders(self):
        self.assertEqual(format_status_text([]), "N/A")
        self.assertEqual(format_status_text([""]), "")
>       self.assertEqual(format_status_text(["ASD"]), "ASD")
E       AssertionError: 'N/A' != 'ASD'
E       - N/A
E       + ASD

.../unit_tests/grading/init_test.py:44: AssertionError

cmstestsuite/unit_tests/grading/steps/compilation_test.py::TestCompilationStep::test_single_command_compilation_failed_nonzero_return

Stack Traces | 0.002s run time

self = <cmstestsuite.unit_tests.grading.steps.compilation_test.TestCompilationStep testMethod=test_single_command_compilation_failed_nonzero_return>

    def test_single_command_compilation_failed_nonzero_return(self):
        # This case is a "success" for the sandbox (it's the user's fault),
        # but compilation is unsuccessful (no executable).
        expected_stats = get_stats(
            0.1, 0.5, 1000 * 1024, Sandbox.EXIT_NONZERO_RETURN,
            stdout="o", stderr="e")
        with patch("cms.grading.steps.compilation.generic_step",
                   return_value=expected_stats):
            success, compilation_success, text, stats = compilation_step(
                self.sandbox, ONE_COMMAND)
    
        # User's fault, no error needs to be logged.
        self.assertLoggedError(False)
        self.assertTrue(success)
        self.assertFalse(compilation_success)
>       self.assertEqual(text, [COMPILATION_MESSAGES.get("fail").message])
E       AssertionError: Lists differ: ['compilation:fail'] != ['Compilation failed']
E       
E       First differing element 0:
E       'compilation:fail'
E       'Compilation failed'
E       
E       - ['compilation:fail']
E       ?   ^          ^
E       
E       + ['Compilation failed']
E       ?   ^          ^    ++

.../grading/steps/compilation_test.py:82: AssertionError

cmstestsuite/unit_tests/grading/steps/compilation_test.py::TestCompilationStep::test_single_command_compilation_failed_signal

Stack Traces | 0.002s run time

self = <cmstestsuite.unit_tests.grading.steps.compilation_test.TestCompilationStep testMethod=test_single_command_compilation_failed_signal>

    def test_single_command_compilation_failed_signal(self):
        # This case is a "success" for the sandbox (it's the user's fault),
        # but compilation is unsuccessful (no executable).
        expected_stats = get_stats(
            0.1, 0.5, 1000 * 1024, Sandbox.EXIT_SIGNAL, signal=11,
            stdout="o", stderr="e")
        with patch("cms.grading.steps.compilation.generic_step",
                   return_value=expected_stats):
            success, compilation_success, text, stats = compilation_step(
                self.sandbox, ONE_COMMAND)
    
        # User's fault, no error needs to be logged.
        self.assertLoggedError(False)
        self.assertTrue(success)
        self.assertFalse(compilation_success)
>       self.assertEqual(text,
                         [COMPILATION_MESSAGES.get("signal").message, "11"])
E       AssertionError: Lists differ: ['compilation:signal', '11'] != ['Compilation killed with signal %s', '11']
E       
E       First differing element 0:
E       'compilation:signal'
E       'Compilation killed with signal %s'
E       
E       - ['compilation:signal', '11']
E       + ['Compilation killed with signal %s', '11']

.../grading/steps/compilation_test.py:136: AssertionError

cmstestsuite/unit_tests/grading/steps/compilation_test.py::TestCompilationStep::test_single_command_compilation_failed_timeout

Stack Traces | 0.002s run time

self = <cmstestsuite.unit_tests.grading.steps.compilation_test.TestCompilationStep testMethod=test_single_command_compilation_failed_timeout>

    def test_single_command_compilation_failed_timeout(self):
        # This case is a "success" for the sandbox (it's the user's fault),
        # but compilation is unsuccessful (no executable).
        expected_stats = get_stats(
            0.1, 0.5, 1000 * 1024, Sandbox.EXIT_TIMEOUT,
            stdout="o", stderr="e")
        with patch("cms.grading.steps.compilation.generic_step",
                   return_value=expected_stats):
            success, compilation_success, text, stats = compilation_step(
                self.sandbox, ONE_COMMAND)
    
        # User's fault, no error needs to be logged.
        self.assertLoggedError(False)
        self.assertTrue(success)
        self.assertFalse(compilation_success)
>       self.assertEqual(text, [COMPILATION_MESSAGES.get("timeout").message])
E       AssertionError: Lists differ: ['compilation:timeout'] != ['Compilation timed out']
E       
E       First differing element 0:
E       'compilation:timeout'
E       'Compilation timed out'
E       
E       - ['compilation:timeout']
E       ?   ^          ^
E       
E       + ['Compilation timed out']
E       ?   ^          ^    ++

.../grading/steps/compilation_test.py:100: AssertionError

cmstestsuite/unit_tests/grading/steps/compilation_test.py::TestCompilationStep::test_single_command_compilation_failed_timeout_wall

Stack Traces | 0.002s run time

self = <cmstestsuite.unit_tests.grading.steps.compilation_test.TestCompilationStep testMethod=test_single_command_compilation_failed_timeout_wall>

    def test_single_command_compilation_failed_timeout_wall(self):
        # This case is a "success" for the sandbox (it's the user's fault),
        # but compilation is unsuccessful (no executable).
        expected_stats = get_stats(
            0.1, 0.5, 1000 * 1024, Sandbox.EXIT_TIMEOUT_WALL,
            stdout="o", stderr="e")
        with patch("cms.grading.steps.compilation.generic_step",
                   return_value=expected_stats):
            success, compilation_success, text, stats = compilation_step(
                self.sandbox, ONE_COMMAND)
    
        # User's fault, no error needs to be logged.
        self.assertLoggedError(False)
        self.assertTrue(success)
        self.assertFalse(compilation_success)
>       self.assertEqual(text, [COMPILATION_MESSAGES.get("timeout").message])
E       AssertionError: Lists differ: ['compilation:timeout'] != ['Compilation timed out']
E       
E       First differing element 0:
E       'compilation:timeout'
E       'Compilation timed out'
E       
E       - ['compilation:timeout']
E       ?   ^          ^
E       
E       + ['Compilation timed out']
E       ?   ^          ^    ++

.../grading/steps/compilation_test.py:118: AssertionError

cmstestsuite/unit_tests/grading/steps/compilation_test.py::TestCompilationStep::test_single_command_success

Stack Traces | 0.002s run time

self = <cmstestsuite.unit_tests.grading.steps.compilation_test.TestCompilationStep testMethod=test_single_command_success>

    def test_single_command_success(self):
        expected_stats = get_stats(
            0.1, 0.5, 1000 * 1024, Sandbox.EXIT_OK, stdout="o", stderr="你好")
        with patch("cms.grading.steps.compilation.generic_step",
                   return_value=expected_stats) as mock_generic_step:
            success, compilation_success, text, stats = compilation_step(
                self.sandbox, ONE_COMMAND)
    
        mock_generic_step.assert_called_once_with(
            self.sandbox, ONE_COMMAND, "compilation", collect_output=True)
        self.assertLoggedError(False)
        self.assertTrue(success)
        self.assertTrue(compilation_success)
>       self.assertEqual(text, [COMPILATION_MESSAGES.get("success").message])
E       AssertionError: Lists differ: ['compilation:success'] != ['Compilation succeeded']
E       
E       First differing element 0:
E       'compilation:success'
E       'Compilation succeeded'
E       
E       - ['compilation:success']
E       ?   ^          ^     ^^
E       
E       + ['Compilation succeeded']
E       ?   ^          ^     ^^^^

.../grading/steps/compilation_test.py:64: AssertionError

cmstestsuite/unit_tests/grading/tasktypes/OutputOnlyTest.py::TestEvaluate::test_diff_missing_file

Stack Traces | 0.002s run time

self = <cmstestsuite.unit_tests.grading.tasktypes.OutputOnlyTest.TestEvaluate testMethod=test_diff_missing_file>

    def test_diff_missing_file(self):
        tt, job = self.prepare(["diff"], {
            "output_001.txt": FILE_001,
        })
    
        tt.evaluate(job, self.file_cacher)
    
        self.eval_output.assert_not_called()
>       self.assertResultsInJob(job,
                                True, str(0.0), ["File not submitted"], {})

.../grading/tasktypes/OutputOnlyTest.py:90: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../grading/tasktypes/OutputOnlyTest.py:67: in assertResultsInJob
    self.assertEqual(job.text, text)
E   AssertionError: Lists differ: ['outputonly:nofile'] != ['File not submitted']
E   
E   First differing element 0:
E   'outputonly:nofile'
E   'File not submitted'
E   
E   - ['outputonly:nofile']
E   + ['File not submitted']

cmstestsuite/unit_tests/grading/steps/compilation_test.py::TestCompilationStep::test_multiple_commands_success

Stack Traces | 0.003s run time

self = <cmstestsuite.unit_tests.grading.steps.compilation_test.TestCompilationStep testMethod=test_multiple_commands_success>

    def test_multiple_commands_success(self):
        expected_stats = get_stats(
            0.1, 0.5, 1000 * 1024, Sandbox.EXIT_OK, stdout="o", stderr="你好")
        with patch("cms.grading.steps.compilation.generic_step",
                   return_value=expected_stats) as mock_generic_step:
            success, compilation_success, text, stats = compilation_step(
                self.sandbox, TWO_COMMANDS)
    
        mock_generic_step.assert_called_once_with(
            self.sandbox, TWO_COMMANDS, "compilation", collect_output=True)
        self.assertLoggedError(False)
        self.assertTrue(success)
        self.assertTrue(compilation_success)
>       self.assertEqual(text, [COMPILATION_MESSAGES.get("success").message])
E       AssertionError: Lists differ: ['compilation:success'] != ['Compilation succeeded']
E       
E       First differing element 0:
E       'compilation:success'
E       'Compilation succeeded'
E       
E       - ['compilation:success']
E       ?   ^          ^     ^^
E       
E       + ['Compilation succeeded']
E       ?   ^          ^     ^^^^

.../grading/steps/compilation_test.py:166: AssertionError

cmstestsuite/unit_tests/grading/steps/trusted_test.py::TestCheckerStep::test_success

Stack Traces | 0.005s run time

self = <cmstestsuite.unit_tests.grading.steps.trusted_test.TestCheckerStep testMethod=test_success>

    def test_success(self):
        self.mock_trusted_step.return_value = (True, True, {})
        self.set_checker_output(b"0.123\n", b"Text.\n")
    
        ret = checker_step(self.sandbox, "c_dig", "i_dig", "co_dig", "o")
    
>       self.assertEqual(ret, (True, 0.123, ["Text."]))
E       AssertionError: Tuples differ: (True, 0.123, ['custom:Text.']) != (True, 0.123, ['Text.'])
E       
E       First differing element 2:
E       ['custom:Text.']
E       ['Text.']
E       
E       - (True, 0.123, ['custom:Text.'])
E       ?                 -------
E       
E       + (True, 0.123, ['Text.'])

.../grading/steps/trusted_test.py:263: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}