Skip to content

gh-139922: Tail calling for MSVC (VS 2026) #139962

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Fidget-Spinner wants to merge 33 commits into from
Closed

gh-139922: Tail calling for MSVC (VS 2026) #139962

Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
82d1259
Brandt's changes
Fidget-Spinner Aug 21, 2025
3248658
Hulon's and mine changes
Fidget-Spinner Aug 21, 2025
9ac430d
push stackrefs into tstate
Fidget-Spinner Aug 21, 2025
085c1d7
Fix last few remaining problems
Fidget-Spinner Aug 21, 2025
0b12f2e
fix handling of recursion
Fidget-Spinner Aug 24, 2025
acf48f5
Changes by Chris to support clang-cl
Fidget-Spinner Oct 11, 2025
35e96c1
Make the restrict MSVC only for now
Fidget-Spinner Oct 11, 2025
d7737e9
Merge remote-tracking branch 'upstream/main' into msvc-tailcall-take-two
Fidget-Spinner Oct 11, 2025
86f19cf
Change restricts to MSVC only
Fidget-Spinner Oct 11, 2025
40013cc
📜🤖 Added by blurb_it.
blurb-it[bot] Oct 11, 2025
48db59e
Reduce diff
Fidget-Spinner Oct 11, 2025
bc9d23c
Merge branch 'msvc-tailcall-take-two' of github.com:Fidget-Spinner/cp…
Fidget-Spinner Oct 11, 2025
19e02c2
Work around specialization
Fidget-Spinner Oct 11, 2025
66ec774
Fix VS 2026
Fidget-Spinner Oct 11, 2025
50f8ff7
fix a typo
Fidget-Spinner Oct 11, 2025
5d908b4
Move to macros to internal header
Fidget-Spinner Oct 11, 2025
0786133
Merge branch 'msvc-tailcall-take-two' of github.com:Fidget-Spinner/cp…
Fidget-Spinner Oct 11, 2025
e699d40
Reduce number of restricts
Fidget-Spinner Oct 13, 2025
66d6c39
Merge branch 'msvc-tailcall-take-two' of https://github.com/Fidget-Sp…
Fidget-Spinner Oct 13, 2025
5584fec
Reduce restrict use even more, reduce usage
Fidget-Spinner Oct 13, 2025
7eeeaa8
Apply Chris' changes
Fidget-Spinner Oct 23, 2025
6f3d525
Merge remote-tracking branch 'upstream/main' into msvc-tailcall-take-two
Fidget-Spinner Nov 4, 2025
81618e2
Merge remote-tracking branch 'upstream/main' into msvc-tailcall-take-two
Fidget-Spinner Nov 11, 2025
2008d1d
Use choco for now to get VS 2026
Fidget-Spinner Nov 11, 2025
7c84388
Update tail-call.yml
Fidget-Spinner Nov 11, 2025
68b41cf
Update tail-call.yml
Fidget-Spinner Nov 11, 2025
c7316fc
Update tail-call.yml
Fidget-Spinner Nov 11, 2025
9214d5b
Update tail-call.yml
Fidget-Spinner Nov 11, 2025
52c6f9c
Update tail-call.yml
Fidget-Spinner Nov 11, 2025
34d98d3
Update tail-call.yml
Fidget-Spinner Nov 11, 2025
7ec626e
Update tail-call.yml
Fidget-Spinner Nov 11, 2025
ad1c5a2
Update tail-call.yml
Fidget-Spinner Nov 11, 2025
4155337
Update tail-call.yml
Fidget-Spinner Nov 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 24 additions & 2 deletions Include/internal/pycore_ceval.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -303,9 +303,9 @@ PyAPI_FUNC(PyObject *)_PyEval_MatchKeys(PyThreadState *tstate, PyObject *map, Py
PyAPI_FUNC(void) _PyEval_MonitorRaise(PyThreadState *tstate, _PyInterpreterFrame *frame, _Py_CODEUNIT *instr);
PyAPI_FUNC(int) _PyEval_UnpackIterableStackRef(PyThreadState *tstate, PyObject *v, int argcnt, int argcntafter, _PyStackRef *sp);
PyAPI_FUNC(void) _PyEval_FrameClearAndPop(PyThreadState *tstate, _PyInterpreterFrame *frame);
PyAPI_FUNC(PyObject **) _PyObjectArray_FromStackRefArray(_PyStackRef *input, Py_ssize_t nargs, PyObject **scratch);
PyAPI_FUNC(PyObject **) _PyObjectArray_FromStackRefArray(_PyThreadStateImpl *_tstate, _PyStackRef *input, Py_ssize_t nargs);

PyAPI_FUNC(void) _PyObjectArray_Free(PyObject **array, PyObject **scratch);
PyAPI_FUNC(void) _PyObjectArray_Free(_PyThreadStateImpl *_tstate, PyObject **array, Py_ssize_t nargs, PyObject **temp_arr);

PyAPI_FUNC(PyObject *) _PyEval_GetANext(PyObject *aiter);
PyAPI_FUNC(void) _PyEval_LoadGlobalStackRef(PyObject *globals, PyObject *builtins, PyObject *name, _PyStackRef *writeto);
Expand Down Expand Up @@ -391,6 +391,28 @@ _PyForIter_VirtualIteratorNext(PyThreadState* tstate, struct _PyInterpreterFrame
#define SPECIAL___AEXIT__ 3
#define SPECIAL_MAX 3

// Special counterparts of ceval functions for performance reasons
PyAPI_FUNC(int) _PyEval_Mapping_GetOptionalItem(PyObject *obj, PyObject *key, PyObject **result);
Copy link
Member

@markshannon markshannon Oct 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the only difference the restrict annotation?
We could just add the annotation to PyMapping_GetOptionalItem the result pointer can never alias the other parameters.

Copy link
Member Author

@Fidget-Spinner Fidget-Spinner Oct 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. We can't because PyMapping_GetOptionalItem is part of the public API, and we can't change the signature of it.

Copy link
Member

@markshannon markshannon Oct 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We aren't changing the signature, just telling the compiler than one of its parameters can't alias the others.
I'm surprised we need to do this anyway. I would have thought that strict aliasing already tells MSVC that they can't alias.

Copy link
Member

@chris-eibl chris-eibl Oct 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is, that strict aliasing can't help here, since in

PyMapping_GetOptionalItem(PyObject *obj, PyObject *key, PyObject **result)

all parameters are of type PyObject. And by using restrict we tell the compiler that "we assure you, that none of these PyObjects point to the same memory", and if we brake that contract, UB kicks in.


#if defined(_MSC_VER) && !defined(__clang__) && _Py_TAIL_CALL_INTERP
# define Py_NO_INLINE_MSVC_TAILCALL Py_NO_INLINE
#else
# define Py_NO_INLINE_MSVC_TAILCALL
#endif

// Tells the compiler that this variable cannot be alised.
Copy link
Member

@zooba zooba Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grammar nit, but "cannot be aliased" is ambiguous (because it's not clear whether we are choosing not to alias it, or telling the compiler that it cannot choose to alias it).

I think we're promising that it won't be aliased? So "Tells the compiler to trust that we haven't aliased this variable" better explains what it's doing. Or just "... this variable is not going to be aliased".

Alternatively, "Tells the compiler not to alias the variable" (but I don't think that even makes sense, let alone being correct here).

#if defined(_MSC_VER) && !defined(__clang__)
# define Py_UNALIASED(var) restrict var
#else
# define Py_UNALIASED(var) var
#endif

// Just a scope. Hints to the programmer and compiler
// That any local variable defined within this block MUST
// not escape from the current definition.
# define Py_BEGIN_LOCALS_MUST_NOT_ESCAPE {
# define Py_END_LOCALS_MUST_NOT_ESCAPE }

#ifdef __cplusplus
}
#endif
Expand Down
5 changes: 5 additions & 0 deletions Include/internal/pycore_tstate.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ struct _gc_thread_state {
};
#endif

/* How much scratch space to give stackref to PyObject* conversion. */
#define MAX_STACKREF_SCRATCH 1024
Copy link
Member

@zooba zooba Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How arbitrary is this amount? What is the usual amount of space required for conversions?

Copy link
Member Author

@Fidget-Spinner Fidget-Spinner Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We used to arbitrarily always consume 10 PyObject/PyStackRef per call. Meaning even if you had a single item, the whole 10 slot space would be used.


// Every PyThreadState is actually allocated as a _PyThreadStateImpl. The
// PyThreadState fields are exposed as part of the C API, although most fields
// are intended to be private. The _PyThreadStateImpl fields not exposed.
Expand All @@ -47,6 +50,8 @@ typedef struct _PyThreadStateImpl {
struct _qsbr_thread_state *qsbr; // only used by free-threaded build
struct llist_node mem_free_queue; // delayed free queue

PyObject *stackref_scratch[MAX_STACKREF_SCRATCH];
int n_stackref_scratch_used;
#ifdef Py_GIL_DISABLED
// Stack references for the current thread that exist on the C stack
struct _PyCStackRef *c_stack_refs;
Expand Down
1 change: 1 addition & 0 deletions Misc/NEWS.d/next/Core_and_Builtins/2025-10-11-17-01-21.gh-issue-139922.RUkXyd.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Allow building CPython with the tail calling interpreter on Visual Studio 2026 MSVC. This provides a performance gain over the prior interpreter for MSVC. Patch by Ken Jin, Brandt Bucher, and Chris Eibl. With help from the MSVC team including Hulon Jenkins.
6 changes: 6 additions & 0 deletions Objects/abstract.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,12 @@ PyMapping_GetOptionalItem(PyObject *obj, PyObject *key, PyObject **result)
return 0;
}

Py_NO_INLINE_MSVC_TAILCALL int
_PyEval_Mapping_GetOptionalItem(PyObject *obj, PyObject *key, PyObject **Py_UNALIASED(result))
{
return PyMapping_GetOptionalItem(obj, key, result);
}

int
PyObject_SetItem(PyObject *o, PyObject *key, PyObject *value)
{
Expand Down
2 changes: 1 addition & 1 deletion Objects/dictobject.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2225,7 +2225,7 @@ _PyDict_NewPresized(Py_ssize_t minused)
return dict_new_presized(minused, false);
}

PyObject *
Py_NO_INLINE_MSVC_TAILCALL PyObject *
_PyDict_FromItems(PyObject *const *keys, Py_ssize_t keys_offset,
PyObject *const *values, Py_ssize_t values_offset,
Py_ssize_t length)
Expand Down
4 changes: 3 additions & 1 deletion PCbuild/pythoncore.vcxproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -595,7 +595,9 @@
<ClCompile Include="..\Python\bltinmodule.c" />
<ClCompile Include="..\Python\bootstrap_hash.c" />
<ClCompile Include="..\Python\brc.c" />
<ClCompile Include="..\Python\ceval.c" />
<ClCompile Include="..\Python\ceval.c">
<AdditionalOptions Condition="'$(UseTailCallInterp)' == 'true' and $(PlatformToolset) != 'ClangCL'">/std:clatest %(AdditionalOptions)</AdditionalOptions>
Copy link
Member

@zooba zooba Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if we pass this option to clang-cl? Does it break?

Any possibility of passing a specific /std:c rather than "latest"?

Copy link
Member Author

@Fidget-Spinner Fidget-Spinner Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I've worked with Chris on this and apparently it builds with clang-cl still the last time I checked with him? @chris-eibl

Copy link
Member

@chris-eibl chris-eibl Oct 13, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if we pass this option to clang-cl? Does it break?

Yupp, clang-cl doesn't like /std:clatest, hence I omitted it, because it doesn't need it.
Seems the clang-cl versions we are using (>=19)

  • support by default at least C99 (which is would be needed for restrict)
  • and support must tail, etc, in the "clang specific notation"

But I assume MSVC needs /std:clatest for [[msvc::musttail]] - @Fidget-Spinner correct?
And at least MSVC definitely needs at least /std:c11 for restrict (https://learn.microsoft.com/en-us/cpp/build/reference/std-specify-language-standard-version?view=msvc-170).

Any possibility of passing a specific /std:c rather than "latest"?

Yeah, we could explicitely enforce /std:c11for clang-cl, but we didn't need it in the past and AFAICT this PR didn't change anything that clang-cl sees, due to carefully crafted combinations of _MSC_VER and __clang__ like

#if defined(_MSC_VER) && !defined(__clang__)

and apparently it builds with clang-cl still the last time I checked with him?

Yes, it still does. Having done a lot of builds recently and still playing around.
And tail calling CI is green as well: https://github.com/python/cpython/actions/runs/18435075900/job/52527447202

Copy link
Member Author

@Fidget-Spinner Fidget-Spinner Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm under the impression we need clatest for msvc musttail as well.

Copy link
Member

@chris-eibl chris-eibl Oct 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've just verified again:

  • MSVC needs /std:clatest for [[msvc::musttail]]
  • using /std:c17 fails with error C2143: syntax error: missing ';' before '['
  • clang-cl doesn't accept /std:clatest

So atm we're stuck with this construct until MSVC introduces a new "fixed" option (or maybe accepts [[msvc::musttail]] in /std:c17).

I agree with Steve #135927 (comment)

For the record, I don't like building with Clatest and if we're considering making that change upstream, we should pick a version.

and we should revisit this once we know a fixed version option for it.

</ClCompile>
<ClCompile Include="..\Python\codecs.c" />
<ClCompile Include="..\Python\codegen.c" />
<ClCompile Include="..\Python\compile.c" />
Expand Down
52 changes: 31 additions & 21 deletions Python/bytecodes.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1535,7 +1535,7 @@ dummy_func(

inst(LOAD_BUILD_CLASS, ( -- bc)) {
PyObject *bc_o;
int err = PyMapping_GetOptionalItem(BUILTINS(), &_Py_ID(__build_class__), &bc_o);
int err = _PyEval_Mapping_GetOptionalItem(BUILTINS(), &_Py_ID(__build_class__), &bc_o);
ERROR_IF(err < 0);
if (bc_o == NULL) {
_PyErr_SetString(tstate, PyExc_NameError,
Expand Down Expand Up @@ -1739,7 +1739,7 @@ dummy_func(
inst(LOAD_FROM_DICT_OR_GLOBALS, (mod_or_class_dict -- v)) {
PyObject *name = GETITEM(FRAME_CO_NAMES, oparg);
PyObject *v_o;
int err = PyMapping_GetOptionalItem(PyStackRef_AsPyObjectBorrow(mod_or_class_dict), name, &v_o);
int err = _PyEval_Mapping_GetOptionalItem(PyStackRef_AsPyObjectBorrow(mod_or_class_dict), name, &v_o);
PyStackRef_CLOSE(mod_or_class_dict);
ERROR_IF(err < 0);
if (v_o == NULL) {
Expand All @@ -1762,11 +1762,11 @@ dummy_func(
else {
/* Slow-path if globals or builtins is not a dict */
/* namespace 1: globals */
int err = PyMapping_GetOptionalItem(GLOBALS(), name, &v_o);
int err = _PyEval_Mapping_GetOptionalItem(GLOBALS(), name, &v_o);
ERROR_IF(err < 0);
if (v_o == NULL) {
/* namespace 2: builtins */
int err = PyMapping_GetOptionalItem(BUILTINS(), name, &v_o);
int err = _PyEval_Mapping_GetOptionalItem(BUILTINS(), name, &v_o);
ERROR_IF(err < 0);
if (v_o == NULL) {
_PyEval_FormatExcCheckArg(
Expand Down Expand Up @@ -1932,7 +1932,7 @@ dummy_func(
assert(class_dict);
assert(oparg >= 0 && oparg < _PyFrame_GetCode(frame)->co_nlocalsplus);
name = PyTuple_GET_ITEM(_PyFrame_GetCode(frame)->co_localsplusnames, oparg);
int err = PyMapping_GetOptionalItem(class_dict, name, &value_o);
int err = _PyEval_Mapping_GetOptionalItem(class_dict, name, &value_o);
if (err < 0) {
ERROR_NO_POP();
}
Expand Down Expand Up @@ -1983,7 +1983,7 @@ dummy_func(
ERROR_IF(true);
}
PyObject *str_o = _PyUnicode_JoinArray(&_Py_STR(empty), pieces_o, oparg);
STACKREFS_TO_PYOBJECTS_CLEANUP(pieces_o);
STACKREFS_TO_PYOBJECTS_CLEANUP(pieces_o, oparg);
DECREF_INPUTS();
ERROR_IF(str_o == NULL);
str = PyStackRef_FromPyObjectSteal(str_o);
Expand Down Expand Up @@ -2108,7 +2108,7 @@ dummy_func(
values_o, 2,
values_o+1, 2,
oparg);
STACKREFS_TO_PYOBJECTS_CLEANUP(values_o);
STACKREFS_TO_PYOBJECTS_CLEANUP(values_o, oparg*2);
DECREF_INPUTS();
ERROR_IF(map_o == NULL);
map = PyStackRef_FromPyObjectStealMortal(map_o);
Expand All @@ -2122,7 +2122,7 @@ dummy_func(
ERROR_IF(true);
}
/* check if __annotations__ in locals()... */
int err = PyMapping_GetOptionalItem(LOCALS(), &_Py_ID(__annotations__), &ann_dict);
int err = _PyEval_Mapping_GetOptionalItem(LOCALS(), &_Py_ID(__annotations__), &ann_dict);
ERROR_IF(err < 0);
if (ann_dict == NULL) {
ann_dict = PyDict_New();
Expand Down Expand Up @@ -2226,8 +2226,11 @@ dummy_func(
}
// we make no attempt to optimize here; specializations should
// handle any case whose performance we care about
PyObject *super;
Py_BEGIN_LOCALS_MUST_NOT_ESCAPE;
PyObject *stack[] = {class, self};
PyObject *super = PyObject_Vectorcall(global_super, stack, oparg & 2, NULL);
super = PyObject_Vectorcall(global_super, stack, oparg & 2, NULL);
Py_END_LOCALS_MUST_NOT_ESCAPE;
if (opcode == INSTRUMENTED_LOAD_SUPER_ATTR) {
PyObject *arg = oparg & 2 ? class : &_PyInstrumentation_MISSING;
if (super == NULL) {
Expand Down Expand Up @@ -2286,8 +2289,12 @@ dummy_func(
PyObject *name = GETITEM(FRAME_CO_NAMES, oparg >> 2);
PyTypeObject *cls = (PyTypeObject *)class;
int method_found = 0;
PyObject *attr_o = _PySuper_Lookup(cls, self, name,
Py_TYPE(self)->tp_getattro == PyObject_GenericGetAttr ? &method_found : NULL);
PyObject *attr_o;
Py_BEGIN_LOCALS_MUST_NOT_ESCAPE;
int *method_found_ptr = &method_found;
attr_o = _PySuper_Lookup(cls, self, name,
Py_TYPE(self)->tp_getattro == PyObject_GenericGetAttr ? method_found_ptr : NULL);
Py_END_LOCALS_MUST_NOT_ESCAPE;
if (attr_o == NULL) {
ERROR_NO_POP();
}
Expand Down Expand Up @@ -3510,10 +3517,13 @@ dummy_func(
}
assert(PyStackRef_IsTaggedInt(lasti));
(void)lasti; // Shut up compiler warning if asserts are off
PyObject* res_o;
Py_BEGIN_LOCALS_MUST_NOT_ESCAPE;
PyObject *stack[5] = {NULL, PyStackRef_AsPyObjectBorrow(exit_self), exc, val_o, tb};
int has_self = !PyStackRef_IsNull(exit_self);
PyObject *res_o = PyObject_Vectorcall(exit_func_o, stack + 2 - has_self,
res_o = PyObject_Vectorcall(exit_func_o, stack + 2 - has_self,
(3 + has_self) | PY_VECTORCALL_ARGUMENTS_OFFSET, NULL);
Py_END_LOCALS_MUST_NOT_ESCAPE;
Py_XDECREF(original_tb);
ERROR_IF(res_o == NULL);
res = PyStackRef_FromPyObjectSteal(res_o);
Expand Down Expand Up @@ -3755,7 +3765,7 @@ dummy_func(
callable_o, args_o,
total_args | PY_VECTORCALL_ARGUMENTS_OFFSET,
NULL);
STACKREFS_TO_PYOBJECTS_CLEANUP(args_o);
STACKREFS_TO_PYOBJECTS_CLEANUP(args_o, total_args);
if (opcode == INSTRUMENTED_CALL) {
PyObject *arg = total_args == 0 ?
&_PyInstrumentation_MISSING : PyStackRef_AsPyObjectBorrow(arguments[0]);
Expand Down Expand Up @@ -3910,7 +3920,7 @@ dummy_func(
callable_o, args_o,
total_args | PY_VECTORCALL_ARGUMENTS_OFFSET,
NULL);
STACKREFS_TO_PYOBJECTS_CLEANUP(args_o);
STACKREFS_TO_PYOBJECTS_CLEANUP(args_o, total_args);
assert((res_o != NULL) ^ (_PyErr_Occurred(tstate) != NULL));
DECREF_INPUTS();
ERROR_IF(res_o == NULL);
Expand Down Expand Up @@ -4193,7 +4203,7 @@ dummy_func(
ERROR_IF(true);
}
PyObject *res_o = tp->tp_vectorcall((PyObject *)tp, args_o, total_args, NULL);
STACKREFS_TO_PYOBJECTS_CLEANUP(args_o);
STACKREFS_TO_PYOBJECTS_CLEANUP(args_o, total_args);
DECREF_INPUTS();
ERROR_IF(res_o == NULL);
res = PyStackRef_FromPyObjectSteal(res_o);
Expand Down Expand Up @@ -4264,7 +4274,7 @@ dummy_func(
PyCFunction_GET_SELF(callable_o),
args_o,
total_args);
STACKREFS_TO_PYOBJECTS_CLEANUP(args_o);
STACKREFS_TO_PYOBJECTS_CLEANUP(args_o, total_args);
assert((res_o != NULL) ^ (_PyErr_Occurred(tstate) != NULL));
DECREF_INPUTS();
ERROR_IF(res_o == NULL);
Expand Down Expand Up @@ -4300,7 +4310,7 @@ dummy_func(
ERROR_IF(true);
}
PyObject *res_o = cfunc(PyCFunction_GET_SELF(callable_o), args_o, total_args, NULL);
STACKREFS_TO_PYOBJECTS_CLEANUP(args_o);
STACKREFS_TO_PYOBJECTS_CLEANUP(args_o, total_args);
assert((res_o != NULL) ^ (_PyErr_Occurred(tstate) != NULL));
DECREF_INPUTS();
ERROR_IF(res_o == NULL);
Expand Down Expand Up @@ -4480,7 +4490,7 @@ dummy_func(
PyCFunctionFastWithKeywords cfunc =
_PyCFunctionFastWithKeywords_CAST(meth->ml_meth);
PyObject *res_o = cfunc(self, (args_o + 1), nargs, NULL);
STACKREFS_TO_PYOBJECTS_CLEANUP(args_o);
STACKREFS_TO_PYOBJECTS_CLEANUP(args_o, total_args);
assert((res_o != NULL) ^ (_PyErr_Occurred(tstate) != NULL));
DECREF_INPUTS();
ERROR_IF(res_o == NULL);
Expand Down Expand Up @@ -4559,7 +4569,7 @@ dummy_func(
}
PyCFunctionFast cfunc = _PyCFunctionFast_CAST(meth->ml_meth);
PyObject *res_o = cfunc(self, (args_o + 1), nargs);
STACKREFS_TO_PYOBJECTS_CLEANUP(args_o);
STACKREFS_TO_PYOBJECTS_CLEANUP(args_o, total_args);
assert((res_o != NULL) ^ (_PyErr_Occurred(tstate) != NULL));
DECREF_INPUTS();
ERROR_IF(res_o == NULL);
Expand Down Expand Up @@ -4658,7 +4668,7 @@ dummy_func(
callable_o, args_o,
positional_args | PY_VECTORCALL_ARGUMENTS_OFFSET,
kwnames_o);
STACKREFS_TO_PYOBJECTS_CLEANUP(args_o);
STACKREFS_TO_PYOBJECTS_CLEANUP(args_o, total_args);
if (opcode == INSTRUMENTED_CALL_KW) {
PyObject *arg = total_args == 0 ?
&_PyInstrumentation_MISSING : PyStackRef_AsPyObjectBorrow(arguments[0]);
Expand Down Expand Up @@ -4815,7 +4825,7 @@ dummy_func(
positional_args | PY_VECTORCALL_ARGUMENTS_OFFSET,
kwnames_o);
PyStackRef_CLOSE(kwnames);
STACKREFS_TO_PYOBJECTS_CLEANUP(args_o);
STACKREFS_TO_PYOBJECTS_CLEANUP(args_o, total_args);
assert((res_o != NULL) ^ (_PyErr_Occurred(tstate) != NULL));
DECREF_INPUTS();
ERROR_IF(res_o == NULL);
Expand Down
21 changes: 15 additions & 6 deletions Python/ceval.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -956,10 +956,12 @@ extern void _PyUOpPrint(const _PyUOpInstruction *uop);


PyObject **
_PyObjectArray_FromStackRefArray(_PyStackRef *input, Py_ssize_t nargs, PyObject **scratch)
_PyObjectArray_FromStackRefArray(_PyThreadStateImpl *_tstate, _PyStackRef *restrict input, Py_ssize_t nargs)
{
PyObject **result;
if (nargs > MAX_STACKREF_SCRATCH) {
/* +1 because vectorcall might use -1 to write self */
PyObject **target = _tstate->stackref_scratch + _tstate->n_stackref_scratch_used + nargs + 1;
if (target > _tstate->stackref_scratch + MAX_STACKREF_SCRATCH) {
// +1 in case PY_VECTORCALL_ARGUMENTS_OFFSET is set.
result = PyMem_Malloc((nargs + 1) * sizeof(PyObject *));
if (result == NULL) {
Expand All @@ -968,7 +970,9 @@ _PyObjectArray_FromStackRefArray(_PyStackRef *input, Py_ssize_t nargs, PyObject
result++;
}
else {
result = scratch;
result = _tstate->stackref_scratch + _tstate->n_stackref_scratch_used;
_tstate->n_stackref_scratch_used += (int)nargs + 1;
assert(_tstate->n_stackref_scratch_used < MAX_STACKREF_SCRATCH);
}
for (int i = 0; i < nargs; i++) {
result[i] = PyStackRef_AsPyObjectBorrow(input[i]);
Expand All @@ -977,10 +981,15 @@ _PyObjectArray_FromStackRefArray(_PyStackRef *input, Py_ssize_t nargs, PyObject
}

void
_PyObjectArray_Free(PyObject **array, PyObject **scratch)
_PyObjectArray_Free(_PyThreadStateImpl *_tstate, PyObject **array, Py_ssize_t nargs, PyObject **temp_arr)
{
if (array != scratch) {
PyMem_Free(array);
/* -1 because we +1 previously */
if (array == temp_arr) {
_tstate->n_stackref_scratch_used -= ((int)nargs + 1);
assert(_tstate->n_stackref_scratch_used >= 0);
}
else {
PyMem_Free(array-1);
}
}

Expand Down
28 changes: 14 additions & 14 deletions Python/ceval_macros.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,12 +79,15 @@
#endif

#if _Py_TAIL_CALL_INTERP
// Note: [[clang::musttail]] works for GCC 15, but not __attribute__((musttail)) at the moment.
# define Py_MUSTTAIL [[clang::musttail]]
# define Py_PRESERVE_NONE_CC __attribute__((preserve_none))
Py_PRESERVE_NONE_CC typedef PyObject* (*py_tail_call_funcptr)(TAIL_CALL_PARAMS);

# define TARGET(op) Py_PRESERVE_NONE_CC PyObject *_TAIL_CALL_##op(TAIL_CALL_PARAMS)
# if defined(_MSC_VER) && !defined(__clang__)
# define Py_MUSTTAIL [[msvc::musttail]]
# define Py_PRESERVE_NONE_CC __preserve_none
# else
# define Py_MUSTTAIL __attribute__((musttail))
# define Py_PRESERVE_NONE_CC __attribute__((preserve_none))
# endif
typedef PyObject *(Py_PRESERVE_NONE_CC *py_tail_call_funcptr)(TAIL_CALL_PARAMS);
# define TARGET(op) Py_NO_INLINE PyObject *Py_PRESERVE_NONE_CC _TAIL_CALL_##op(TAIL_CALL_PARAMS)
# define DISPATCH_GOTO() \
do { \
Py_MUSTTAIL return (((py_tail_call_funcptr *)instruction_funcptr_table)[opcode])(TAIL_CALL_ARGS); \
Expand Down Expand Up @@ -390,17 +393,14 @@ do { \

/* Stackref macros */

/* How much scratch space to give stackref to PyObject* conversion. */
#define MAX_STACKREF_SCRATCH 10

#define STACKREFS_TO_PYOBJECTS(ARGS, ARG_COUNT, NAME) \
/* +1 because vectorcall might use -1 to write self */ \
PyObject *NAME##_temp[MAX_STACKREF_SCRATCH+1]; \
PyObject **NAME = _PyObjectArray_FromStackRefArray(ARGS, ARG_COUNT, NAME##_temp + 1);
_PyThreadStateImpl *_tstate = (_PyThreadStateImpl *)tstate; \
PyObject **NAME##_temp = _tstate->stackref_scratch + _tstate->n_stackref_scratch_used; \
PyObject **NAME = _PyObjectArray_FromStackRefArray(_tstate, ARGS, ARG_COUNT);

#define STACKREFS_TO_PYOBJECTS_CLEANUP(NAME) \
/* +1 because we +1 previously */ \
_PyObjectArray_Free(NAME - 1, NAME##_temp);
#define STACKREFS_TO_PYOBJECTS_CLEANUP(NAME, ARG_COUNT) \
_PyObjectArray_Free(_tstate, NAME, ARG_COUNT, NAME##_temp);

#define CONVERSION_FAILED(NAME) ((NAME) == NULL)

Expand Down
Loading
Loading