PEP 679: Modify proposal and improve sections

peps/pep-0679.rst

@@ -1,137 +1,215 @@
 PEP: 679
-Title: Allow parentheses in assert statements
-Author: Pablo Galindo Salgado <pablogsal@python.org>
-Discussions-To: https://discuss.python.org/t/pep-679-allow-parentheses-in-assert-statements/13003
+Title: New assert statement syntax with parentheses
+Author: Pablo Galindo Salgado <pablogsal@python.org>,
+        Stan Ulbrych <stanulbrych@gmail.com>,
+Discussions-To: Pending
 Status: Draft
 Type: Standards Track
 Created: 07-Jan-2022
-Python-Version: 3.12
+Python-Version: 3.15
+Post-History: `10-Jan-2022 <https://discuss.python.org/t/pep-679-allow-parentheses-in-assert-statements/13003>`__
 
 
 Abstract
 ========
 
-This PEP proposes to allow parentheses surrounding the two-argument form of
-assert statements. This will cause the interpreter to reinterpret what before
-would have been an assert with a two-element tuple that will always be True
-(``assert (expression, message)``) to an assert statement with a subject and a
-failure message, equivalent to the statement with the parentheses removed
-(``assert expression, message``).
+This PEP proposes allowing parentheses in the two-argument form of :keyword:`assert`.
+The interpreter will reinterpret ``assert (expr, msg)`` as ``assert expr, msg``,
+eliminating the common pitfall where such code was previously treated as
+asserting a two-element :class:`tuple`, which is always truthy.
 
 
 Motivation
 ==========
 
-It is a common user mistake when using the form of the assert statement that includes
-the error message to surround it with parentheses. Unfortunately, this mistake
-passes undetected as the assert will always pass, because it is
-interpreted as an assert statement where the expression is a two-tuple, which
-always has truth-y value.
+It is a common user mistake when using the form of the :keyword:`assert`
+statement that includes the error message to surround it with parentheses [#SO1]_ [#RD]_.
+This is because many beginners assume :keyword:`!assert` is a function.
+The prominent :mod:`unittest` methods, particularly :meth:`~unittest.TestCase.assertTrue`,
+also require parentheses around the assertion and message.
 
-The mistake most often happens when extending the test or description beyond a
-single line, as parentheses are the natural way to do that.
+Unfortunately, this mistake passes undetected as the ``assert`` will always pass
+[#exception]_, because it is interpreted as an ``assert`` statement where the
+expression is a two-tuple, which always has truth-y value.
+The mistake also often occurs when extending the test or description beyond a
+single line, as parentheses are a natural way to do that.
 
-This is so common that a ``SyntaxWarning`` is `now emitted by the compiler
-<https://bugs.python.org/issue35029>`_.
+This is so common that a :exc:`SyntaxWarning` is `emitted by the compiler
+<https://github.com/python/cpython/issues/79210>`_ since 3.10 and several
+code linters [#fl8]_ [#pylint]_.
 
 Additionally, some other statements in the language allow parenthesized forms
-in one way or another like ``import`` statements (``from x import (a,b,c)``) and
-``del`` statements (``del (a,b,c)``).
+in one way or another, for example, ``import`` statements
+(``from x import (a,b,c)``) or ``del`` statements (``del (a,b,c)``).
 
-Allowing parentheses not only will remove the common mistake but also will allow
+Allowing parentheses not only will remove the pitfall but also will allow
 users and auto-formatters to format long assert statements over multiple lines
 in what the authors of this document believe will be a more natural way.
-Although is possible to currently format long ``assert`` statements over
-multiple lines as::
+Although it is possible to currently format long :keyword:`assert` statements
+over multiple lines with backslashes (as is recommended by
+:pep:`8#maximum-line-length`) or parentheses and a comma::
 
-  assert (
+    assert (
       very very long
-      expression
-  ), (
+      test
+    ), (
       "very very long "
-      "message"
-  )
+      "error message"
+    )
 
-the authors of this document believe the parenthesized form is more clear and more consistent with
-the formatting of other grammar constructs::
+the authors of this document believe the proposed parenthesized form is more
+clear and intuitive, along with being more consistent with the formatting of
+other grammar constructs::
 
-  assert (
+    assert (
       very very long
-      expression,
+      test,
 
       "very very long "
-      "message",
-  )
+      "message"
+    )
 
-This change has been originally discussed and proposed in [bpo-46167]_.
 
 Rationale
 =========
 
-This change can be implemented in the parser or in the compiler. We have
-selected implementing this change in the parser because doing it in the compiler
-will require re-interpreting the AST of an assert statement with a two-tuple::
-
-  Module(
-      body=[
-          Assert(
-              test=Tuple(
-                  elts=[
-                      Name(id='x', ctx=Load()),
-                      Name(id='y', ctx=Load())],
-                  ctx=Load()))],
-      type_ignores=[])
-
-as the AST of an assert statement with an expression and a message::
-
-  Module(
-      body=[
-          Assert(
-              test=Name(id='x', ctx=Load()),
-              msg=Name(id='y', ctx=Load()))],
-      type_ignores=[])
-
-The problem with this approach is that the AST of the first form will
-technically be "incorrect" as we already have a specialized form for the AST of
-an assert statement with a test and a message (the second one). This
-means that many tools that deal with ASTs will need to be aware of this change
-in semantics, which will be confusing as there is already a correct form that
-better expresses the new meaning.
+Due to backwards compatibility concerns (see section below), to inform users
+of the new change of how what was previously a two element tuple is parsed,
+a :exc:`SyntaxWarning` with a message like
+``"new assertion syntax, will assert first element of tuple"``
+will be raised till Python 3.17. For example, when using the new syntax:
+
+.. code-block:: pycon
+
+    >>> assert ('Petr' == 'Pablo', "That doesn't look right!")
+    <python-input-0>:0: SyntaxWarning: new assertion syntax, will assert first element of tuple
+    Traceback (most recent call last):
+      File "<python-input-0>", line 1, in <module>
+        assert ('Petr' == 'Pablo', "That doesn't look right!")
+                ^^^^^^^^^^^^^^^^^
+    AssertionError: That doesn't look right!
+
+Note that improving syntax warnings in general
+is out of the scope of this PEP.
+
 
 Specification
 =============
 
-This PEP proposes changing the grammar of the ``assert`` statement to: ::
+The formal grammar of the :keyword:`assert` statement will change to [#edgecase]_:
+
+.. code-block::
 
     | 'assert' '(' expression ',' expression [','] ')' &(NEWLINE | ';')
     | 'assert' a=expression [',' expression ]
 
 Where the first line is the new form of the assert statement that allows
-parentheses. The lookahead is needed so statements like ``assert (a, b) <= c,
-"something"`` are still parsed correctly and to prevent the parser to eagerly
-capture the tuple as the full statement.
+parentheses and will raise a :exc:`SyntaxWarning` till 3.17.
+The lookahead is needed to prevent the parser from eagerly capturing the
+tuple as the full statement, so statements like ``assert (a, b) <= c, "something"``
+are still parsed correctly.
+
+
+Implementation Notes
+====================
+
+This change can be implemented in the parser or in the compiler.
+The specification that a :exc:`SyntaxWarning` be raised informing users
+of the new syntax complicates the implementation, as warnings
+should be raised during compilation.
+
+The authors believe that an ideal implementation would be in the parser [#edgecase]_,
+resulting in ``assert (x,y)`` having the same AST as ``assert x,y``.
+This necessitates a two-step implementation plan, with a necessary temporary
+compromise.
+
+
+Implementing in the parser
+--------------------------
 
-Optionally, new "invalid" rule can be added to produce custom syntax errors to
-cover tuples with 0, 1, 3 or more elements.
+It is not possible to have a pure parser implementation with the warning
+specification.
+(Note that, without the warning specification the pure parser implementation is
+a small grammar change [#previmp]_).
+To raise the warning, the compiler must
+be aware of the new syntax, which means that a flag would be necessary as
+otherwise the information is lost during parsing.
+As such, the AST of an :keyword:`assert` would look like so,
+with a ``paren_syntax`` flag::
+
+    >>> print(ast.dump(ast.parse('assert(True, "Error message")'), indent=4))
+    Module(
+        body=[
+            Assert(
+                test=Constant(value=True),
+                msg=Constant(value='Error message'),
+                paren_syntax=1)])
+
+The flag would be removed in 3.18 along with the :exc:`SyntaxWarning`.
+
+
+Implementing in the compiler
+----------------------------
+
+The new syntax can be implemented in the compiler by special casing tuples
+of length two. This however, will have the side-effect of not modifying the
+AST whatsoever during the transition period while the :exc:`SyntaxWarning`
+is being emitted.
+
+Once the :exc:`SyntaxWarning` is removed, the implementation
+can be moved to the parser level, where the parenthesized form would be
+parsed directly into the same AST structure as ``assert expression, message``.
+This approach is more backwards-compatible, as the many tools that deal with
+ASTs will have more time to adapt.
 
 
 Backwards Compatibility
 =======================
 
-The change is not technically backwards compatible, as parsing ``assert (x,y)``
-is currently interpreted as an assert statement with a 2-tuple as the subject,
-while after this change it will be interpreted as ``assert x,y``.
+The change is not technically backwards compatible. Whether implemented initially
+in the parser or the compiler, ``assert (x,y)``,
+which is currently interpreted as an assert statement with a 2-tuple as the
+subject and is always truth-y, will be interpreted as ``assert x,y``.
 
 On the other hand, assert statements of this kind always pass, so they are
 effectively not doing anything in user code. The authors of this document think
 that this backwards incompatibility nature is beneficial, as it will highlight
-these cases in user code while before they will have passed unnoticed (assuming that
-these cases still exist because users are ignoring syntax warnings).
-
-Security Implications
-=====================
-
-There are no security implications for this change.
+these cases in user code while before they will have passed unnoticed. This case
+has already raised a :exc:`SyntaxWarning` since Python 3.10, so there has been
+a deprecation period of over 5 years.
+The continued raising of a :exc:`!SyntaxWarning` should mitigate surprises.
+
+The change will also result in changes to the AST of ``assert (x,y)``,
+which currently is:
+
+.. code-block:: text
+
+    Module(
+        body=[
+            Assert(
+                test=Tuple(
+                    elts=[
+                        Name(id='x', ctx=Load()),
+                        Name(id='y', ctx=Load())],
+                    ctx=Load()))],
+        type_ignores=[])
+
+the final implementation, in Python 3.18, will result in the following AST:
+
+.. code-block:: text
+
+    Module(
+        body=[
+            Assert(
+                test=Name(id='x', ctx=Load()),
+                msg=Name(id='y', ctx=Load()))],
+        type_ignores=[])
+
+The problem with this is that the AST of the first form will
+technically be "incorrect" as we already have a specialized form for the AST of
+an assert statement with a test and a message (the second one).
+Implementing initially in the compiler will delay this change, alleviating
+backwards compatibility concerns, as tools will have more time to adjust.
 
 
 How to Teach This
@@ -141,21 +219,94 @@ The new form of the ``assert`` statement will be documented as part of the langu
 standard.
 
 When teaching the form with error message of the ``assert`` statement to users,
-now it can be noted that adding parentheses also work as expected, which allows to break
-the statement over multiple lines.
+now it can be noted that adding parentheses also work as expected, which allows
+to break the statement over multiple lines.
 
 
 Reference Implementation
 ========================
 
-A proposed draft PR with the change exist in [GH-30247]_.
+A reference implementation in the parser can be found in this
+`branch <https://github.com/python/cpython/compare/main...StanFromIreland:assert-prototype?expand=1>`__
+and reference implementation in the compiler can be found in this
+`branch <https://github.com/python/cpython/compare/main...StanFromIreland:assert-codegen?expand=1>`__.
 
 
-References
-==========
+Rejected Ideas
+==============
+
+Adding a syntax with a keyword
+------------------------------
+
+Everywhere else in Python syntax, the comma separates variable-length “lists”
+of homogeneous elements, like the the items of a :class:`tuple` or :class:`list`,
+parameters/arguments of functions, or import targets.
+After Python 3.0 introduced :keyword:`except...as <except>`,
+the :keyword:`assert` statement remains as the only exception to this convention.
+
+It's possible that user confusion stems, at least partly, from an expectation
+that comma-separated items are equivalent.
+Enclosing an :keyword:`!assert` statement's expression and message in
+parentheses would visually bind them together even further.
+Making ``assert`` look more similar to a function call encourages a wrong
+mentality.
+
+As a possible solution, it was proposed [#assertwith]_ to replace the comma with
+a keyword, and the form would allow parentheses, for example::
+
+    assert condition else "message"
+    assert (condition else "message")
+
+The comma could then be slowly and carefully deprecated, starting with
+the case where they appear in parentheses, which already raises a
+:exc:`SyntaxWarning`.
+
+The authors of this PEP believe that adding a completely new syntax will,
+first and foremost, not solve the common beginner pitfall that this PEP aims to
+patch, and will not improve the formatting of assert statements across multiple
+lines, which the authors believe the proposed syntax improves.
+
+
+Security Implications
+=====================
+
+There are no security implications for this change.
+
+
+Acknowledgements
+================
+
+This change was originally discussed and proposed in :cpython-issue:`90325`.
+
+Many thanks to Petr Viktorin for his help during the drafting process of this PEP.
+
+
+Footnotes
+=========
+
+.. [#SO1] `StackOverflow: "'assert' statement with or without parentheses" <https://stackoverflow.com/questions/3112171/assert-statement-with-or-without-parentheses>`_
+
+.. [#RD] `/r/python: "Rant: use that second expression in assert! " <https://www.reddit.com/r/Python/comments/1n87g91/rant_use_that_second_expression_in_assert/>`_
+
+.. [#fl8] `flake8: Rule F631 <https://flake8.pycqa.org/en/latest/user/error-codes.html>`_
+
+.. [#pylint] `pylint: assert-on-tuple (W0199) <https://pylint.pycqa.org/en/latest/user_guide/checkers/features.html>`_
+
+.. [#previmp] For the previous parser implementation, see :cpython-pr:`30247`
+
+.. [#exception] During the updating of this PEP, an exception
+                (``assert (*(t := ()),)``) was found, contradicting the warning.
+
+.. [#assertwith] `[DPO] Pre-PEP: Assert-with: Dedicated syntax for assertion messages <https://discuss.python.org/t/pre-pep-assert-with-dedicated-syntax-for-assertion-messages/13247>`_
+
+.. [#edgecase] An edge case arises with constructs like:
+
+                 >>> x = (0,)
+                 >>> assert (*x, "edge cases aren't fun:-(")
 
-.. [bpo-46167] https://bugs.python.org/issue46167
-.. [GH-30247] https://github.com/python/cpython/pull/30247
+               This form is currently parsed as a single tuple expression, not
+               as a condition/message pair, and will need explicit handling in
+               the compiler.
 
 
 Copyright