Rewrite MathTex to make it more robust regarding splitting

[henrikmidtiby]

Overview: What does this pull request change?

This PR changes a large part of the implementation of the MathTex class.
The reason is to deal with the issues related to splitting the tex string and then try to match the generated objects with parts of the original tex string.
The idea for the PR is based on this discussion on discord.
https://discord.com/channels/581738731934056449/1406977902788218892

The MathTex implementation has been updated to make it more robust and fix a number of issues.
A beneficial side effect is that named groups in svg files can now be accessed through SVGMobject.

Motivation and Explanation: Why and how do your changes improve the library?

This PR changes how MathTex objects are created / rendered.
To illustrate the changes in the process, please look at the following example
of calling MathTex.

exp1 = MathTex("{a}^2+{b}^2", 
               "=", 
               "{c}^2", 
               substrings_to_isolate=["a", "b", "c"])

The first step in rendering the MathTex equation, is to join the three separate strings to a single string and add some markup to the string that will allow manim to extract certain parts of the equation afterwards. Each of the three input strings are put in the following markup.

\special{dvisvgm:raw <g id='unique000'>} < input string > \special{dvisvgm:raw </g>}

This ends up looking like the following:

\special{dvisvgm:raw <g id='unique000'>}{a}^2+{b}^2 \special{dvisvgm:raw </g>}
\special{dvisvgm:raw <g id='unique001'>}= \special{dvisvgm:raw </g>}
\special{dvisvgm:raw <g id='unique002'>}{c}^2\special{dvisvgm:raw </g>}

This allows us to access a named group in the formed svg file corresponding to each of the three tex strings.
For each string, the elements of substrings_to_isolate is also located, and marked up with a similar markup.
E.g. the a in the equation ends up like this, where the element is placed in the named group 'unique000ss'.

\special{dvisvgm:raw <g id='unique000ss'>}a\special{dvisvgm:raw </g>}

All there marked up strings are joined to the following string (newlines were added to increase readability).

\special{dvisvgm:raw <g id='unique000'>}{
\special{dvisvgm:raw <g id='unique000ss'>}a
\special{dvisvgm:raw </g>}}^2+{
\special{dvisvgm:raw <g id='unique001ss'>}b
\special{dvisvgm:raw </g>}}^2 
\special{dvisvgm:raw </g>}
\special{dvisvgm:raw <g id='unique001'>}= 
\special{dvisvgm:raw </g>}
\special{dvisvgm:raw <g id='unique002'>}{
\special{dvisvgm:raw <g id='unique002ss'>}c
\special{dvisvgm:raw </g>}}^2
\special{dvisvgm:raw </g>}

This string is then saved to a tex file and converted to first a .dvi and then a .svg file using the latex and dvisvgm command line tools.
Finally the svg file is loaded by a SVGMobject from which the named groups can be accessed as demonstrated here.

exp1.scale(2)
exp1.set_color_by_tex("a", BLUE)
exp1[2].set_color(ORANGE)
self.add(exp1)

The full code for this example is provided here

class Scene27(Scene):
    def construct(self):
        exp1 = MathTex("{a}^2+{b}^2", 
                       "=", 
                       "{c}^2", 
                       substrings_to_isolate=["a", "b", "c"])
        exp1.scale(2)
        exp1.set_color_by_tex("a", BLUE)
        exp1[2].set_color(ORANGE)
        self.add(exp1)

and it renders as follows

Links to added or changed documentation pages

Further Information and Comments

This PR is an extension of #4473 "Maintain groups when importing svg files"

The PR removes these methods from MathTex: get_parts_by_tex and index_of_part_by_tex.

Reviewer Checklist

• The PR title is descriptive enough for the changelog, and the PR is labeled correctly
• If applicable: newly added non-private functions and classes have a docstring including a short summary and a PARAMETERS section
• If applicable: newly added functions and classes are tested

[henrikmidtiby]

And here are some example code and the generated output.

class ExampleScene8(Scene):
    def construct(self):
        formula = MathTex(
            r"P(X=k) =",
            r"\binom{12}{k}",
            r"0.5^{k}",
            r"(1-0.5)^{12-k}",
            substrings_to_isolate=["k", "1", "12", "0.5"],
        ).scale(1.3)

        formula.set_color_by_tex("k", GREEN)
        formula.set_color_by_tex("12", RED)
        formula.set_color_by_tex("1", YELLOW)
        formula.set_color_by_tex("0.5", BLUE_D)
        self.add(formula)

class ExampleScene11(Scene):
    def construct(self):
        t2cm = {"n": RED, "1": GREEN, "x": YELLOW}
        eq = MathTex(r"\sum_{1}^{n} x", tex_to_color_map=t2cm).scale(1.3)

        self.add(eq)

[henrikmidtiby]

I have located a minor issue related to the "Using text" guide.
https://docs.manim.community/en/stable/guides/using_text.html#substrings-and-parts

In the example IncorrectLaTeXSubstringColoring, the equation is no longer colored as in the current documentation.

[behackl]

This is great and I love the much more robust implementation. Kudos and thank you!

Here are some more suggestions:

• It might be nice to have some sort of validation in SVGMobject.__init__ that checks whether all ids from the SVG are contained in the id map at the end of initialising (and log warnings if some are missing?)
• The PR description should mention removal/renaming of get_parts_by_tex, index_of_part_by_tex

Additionally, my review clanker suggests adding two properties to MathTex:

@property
def _substring_matches(self) -> list[tuple[str, str]]:
    """Return only the 'ss' (substring_to_isolate) matches."""
    return [(tex, id_) for tex, id_ in self.matched_strings_and_ids if id_.endswith("ss")]

@property
def _main_matches(self) -> list[tuple[str, str]]:
    """Return only the main tex_string matches."""
    return [(tex, id_) for tex, id_ in self.matched_strings_and_ids if not id_.endswith("ss")]

in order to make break_up_by_substring more readable,

for tex_string, tex_string_id in self._main_matches:
    mtp = MathTexPart()
    mtp.tex_string = tex_string
    mtp.add(*self.id_to_vgroup_dict[tex_string_id].submobjects)
    new_submobjects.append(mtp)

Not sure whether that is worth it, and/or whether 'ss' should perhaps be made more explicit (I'd like that) -- but other than these I don't really have any substantial feedback. I like it.

Left a few more inline comments, please do take a look!

[henrikmidtiby]

It might be nice to have some sort of validation in SVGMobject.__init__ that checks whether all ids from the SVG are contained in the id map at the end of initialising (and log warnings if some are missing?)

I like this idea, but is uncertain about how to implement it right now.
If I locate the id's manually by iterating through all elements in the svg, I should also get all of them in in get_mobjects_from.
Right now I cannot see how to get a list of id's from the SVG file directly from svgelements [1]

[1] https://pypi.org/project/svgelements/.

• The PR description should mention removal/renaming of get_parts_by_tex, index_of_part_by_tex

Done

Additionally, my review clanker suggests adding two properties to MathTex:
...
in order to make break_up_by_substring more readable,

Implemented.

Not sure whether that is worth it, and/or whether 'ss' should perhaps be made more explicit (I'd like that) -- but other than these I don't really have any substantial feedback. I like it.

The string can now be customized, as it is defined as follows

MATHTEX_SUBSTRING = "substring"

Left a few more inline comments, please do take a look!
Thanks for the suggestions.

The error messages are now given to the logger.

Regarding the additional parsing of sub and superscripts I would like to avoid that for now.

[behackl]

lets get this merged -- thank you very much for your effort with this! if you have a second, could you add a short description for the changelog to either a comment or in an edit of the pr description?

[henrikmidtiby]

@behackl
I would suggest the following text for the changelog.

The MathTex implementation has been updated to make it more robust and fix a number of issues.
A beneficial side effect is that named groups in svg files can now be accessed through SVGMobject.