From f1565109603123c39bd2a0c6002f5ea15a62b6e7 Mon Sep 17 00:00:00 2001 From: Trent Nelson Date: Fri, 25 Sep 2026 11:24:14 -0700 Subject: [PATCH 1/4] gh-151518: Avoid STW starvation of attaching threads (#152826) * gh-151518: Avoid STW starvation of attaching threads Free-threaded stop-the-world pauses can otherwise starve a thread trying to reattach after it was suspended while detached. A tight manual gc.collect() loop can release and immediately request the next stop-the-world pause, repeatedly parking the detached thread before it can attach and make progress. Add a distinct _Py_THREAD_SUSPENDED_DETACHED state for tstates parked from DETACHED. tstate_wait_attach() marks an attach waiter only after observing that detached-origin suspended state, and park_detached_threads() skips only those active waiters on later stop-the-world passes. The ordinary successful tstate_try_attach() path remains the baseline CAS-only path. Teach the related stop-the-world paths about both suspended states, including start_the_world() and tstate_delete_common(). Keep the new wait flag after the existing hot free-threaded _PyThreadStateImpl fields so their offsets do not move. Add a free-threaded GC regression test that runs a subprocess with a tight gc.collect() worker and verifies the main thread can reattach after sleeping and stop the worker. * gh-151518: Track attach waiters in the thread state Represent active attach waiters with suspended-waiting and detached-waiting states. Preserve waiter registration when the world resumes, and keep passive detached threads immediately parkable. Restore the thread-state padding and retain the single-CAS uncontended attach path. * gh-151518: Simplify the GC fairness regression Use explicit warmup imports and a joined non-daemon collector. Run the child through script_helper with a faulthandler watchdog so a stalled attachment still fails with a traceback. Describe the bug as a fairness issue in the NEWS entry. * gh-151518: Exercise STW fairness directly * gh-151518: Preserve attach waiters when resuming BRC suspensions Share the waiter-aware resume transition between stop-the-world pauses and biased reference count merging so a concurrently registering waiter keeps its opportunity to attach. Restore the support import needed by the new upstream GC regression and align the thread-state constants. * gh-151518: Strengthen the STW attach fairness regression Hold each test pause for 10 ms and repeat pauses in C to reduce the opportunities for a waiting thread to run between pause requests. Clarify that parking rechecks the thread state before sleeping. * gh-151518: Restrict STW test pauses to free-threaded builds STW is a no-op in GIL builds. Avoid waiting in the helper there, since WASI cannot perform the blocking futex operation. * gh-151518: Wake only registered attach waiters Avoid a parking-lot lookup when resuming a suspended thread that has not tried to reattach. Use the old state from the successful CAS so concurrent waiter registration still receives a wakeup. --- Include/cpython/pystate.h | 3 +- Include/internal/pycore_pystate.h | 47 ++++++------ .../test_free_threading/test_threading.py | 39 +++++++++- ...-06-16-19-20-00.gh-issue-151518.e6v0Js.rst | 2 + Modules/_testinternalcapi.c | 19 +++++ Python/pystate.c | 71 +++++++++++++++---- 6 files changed, 142 insertions(+), 39 deletions(-) create mode 100644 Misc/NEWS.d/next/Core_and_Builtins/2026-06-16-19-20-00.gh-issue-151518.e6v0Js.rst diff --git a/Include/cpython/pystate.h b/Include/cpython/pystate.h index f367146e262bfe8..e551f624b6289b4 100644 --- a/Include/cpython/pystate.h +++ b/Include/cpython/pystate.h @@ -118,8 +118,7 @@ struct _ts { int _whence; - /* Thread state (_Py_THREAD_ATTACHED, _Py_THREAD_DETACHED, _Py_THREAD_SUSPENDED). - See Include/internal/pycore_pystate.h for more details. */ + /* Thread state. See Include/internal/pycore_pystate.h for details. */ int state; int py_recursion_remaining; diff --git a/Include/internal/pycore_pystate.h b/Include/internal/pycore_pystate.h index 6caa7a5d30116e9..8b8f0a98bedf76b 100644 --- a/Include/internal/pycore_pystate.h +++ b/Include/internal/pycore_pystate.h @@ -21,32 +21,32 @@ extern "C" { // interpreter at the same time. Only the "bound" thread may perform the // transitions between "attached" and "detached" on its own PyThreadState. // -// The "suspended" state is used to implement stop-the-world pauses, such as -// for cyclic garbage collection. It is only used in `--disable-gil` builds. -// The "suspended" state is similar to the "detached" state in that in both -// states the thread is not allowed to call most Python APIs. However, unlike -// the "detached" state, a thread may not transition itself out from the -// "suspended" state. Only the thread performing a stop-the-world pause may -// transition a thread from the "suspended" state back to the "detached" state. +// The "suspended" states are used to implement stop-the-world pauses and to +// merge biased reference counts on behalf of detached threads. They are only +// used in `--disable-gil` builds. +// They are similar to the "detached" state in that the thread is not allowed +// to call most Python APIs. A suspended thread trying to attach marks itself +// as "suspended-waiting". Only the thread responsible for suspending it may +// resume it, moving it to "detached" or "detached-waiting". +// A "detached-waiting" thread must attach before it can be suspended again. // // The "shutting down" state is used when the interpreter is being finalized. // Threads in this state can't do anything other than block the OS thread. // (See _PyThreadState_HangThread). // -// State transition diagram: -// -// (bound thread) (stop-the-world thread) -// [attached] <-> [detached] <-> [suspended] -// | ^ -// +---------------------------->---------------------------+ -// (bound thread) -// -// The (bound thread) and (stop-the-world thread) labels indicate which thread -// is allowed to perform the transition. -#define _Py_THREAD_DETACHED 0 -#define _Py_THREAD_ATTACHED 1 -#define _Py_THREAD_SUSPENDED 2 -#define _Py_THREAD_SHUTTING_DOWN 3 +// State transitions: +// Bound thread: attached <-> detached +// attached -> suspended +// suspended -> suspended-waiting +// detached-waiting -> attached +// Suspending thread: detached <-> suspended +// suspended-waiting -> detached-waiting +#define _Py_THREAD_DETACHED 0 +#define _Py_THREAD_ATTACHED 1 +#define _Py_THREAD_SUSPENDED 2 +#define _Py_THREAD_SHUTTING_DOWN 3 +#define _Py_THREAD_SUSPENDED_WAITING 4 +#define _Py_THREAD_DETACHED_WAITING 5 /* Check if the current thread is the main thread. @@ -162,8 +162,9 @@ extern void _PyThreadState_Suspend(PyThreadState *tstate); // Returns 1 on success, 0 if the thread was not in the "detached" state. extern int _PyThreadState_TrySuspendDetached(PyThreadState *tstate); -// Undo a successful _PyThreadState_TrySuspendDetached(): switch the thread -// back to "detached" and wake it if it is waiting to attach. +// Resume a thread suspended by _PyThreadState_TrySuspendDetached() or a +// stop-the-world pause: switch it back to "detached" or "detached-waiting" +// and wake it if it is waiting to attach. extern void _PyThreadState_ResumeDetached(PyThreadState *tstate); #endif diff --git a/Lib/test/test_free_threading/test_threading.py b/Lib/test/test_free_threading/test_threading.py index b5a5ca272b9405a..ecc42db7b44f3af 100644 --- a/Lib/test/test_free_threading/test_threading.py +++ b/Lib/test/test_free_threading/test_threading.py @@ -1,5 +1,8 @@ import unittest -from test.support import threading_helper +import textwrap + +from test import support +from test.support import script_helper, threading_helper threading_helper.requires_working_threading(module=True) @@ -22,5 +25,39 @@ def mutate_thread(): threading_helper.run_concurrently([repr_thread, mutate_thread]) +class TestThreadState(unittest.TestCase): + @support.requires_subprocess() + def test_tight_stw_loop_does_not_starve_attach(self): + script = textwrap.dedent(f""" + import faulthandler + + faulthandler.dump_traceback_later({support.SHORT_TIMEOUT}, exit=True) + + import _testinternalcapi + import threading + import time + + started = threading.Event() + stop = threading.Event() + + def stop_the_world(): + _testinternalcapi.test_stop_the_world() + started.set() + while not stop.is_set(): + _testinternalcapi.test_stop_the_world() + + thread = threading.Thread(target=stop_the_world) + thread.start() + started.wait() + # Each reattachment must make progress between consecutive pauses. + for _ in range(50): + time.sleep(0.02) + stop.set() + thread.join() + faulthandler.cancel_dump_traceback_later() + """) + script_helper.assert_python_ok("-X", "gil=0", "-c", script) + + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2026-06-16-19-20-00.gh-issue-151518.e6v0Js.rst b/Misc/NEWS.d/next/Core_and_Builtins/2026-06-16-19-20-00.gh-issue-151518.e6v0Js.rst new file mode 100644 index 000000000000000..5da739e8b732235 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2026-06-16-19-20-00.gh-issue-151518.e6v0Js.rst @@ -0,0 +1,2 @@ +Fix a free-threaded stop-the-world fairness issue that could starve a thread +reattaching after being suspended while detached. diff --git a/Modules/_testinternalcapi.c b/Modules/_testinternalcapi.c index c01ac65dd4cc042..049b990d65be5fa 100644 --- a/Modules/_testinternalcapi.c +++ b/Modules/_testinternalcapi.c @@ -30,6 +30,7 @@ #include "pycore_instruction_sequence.h" // _PyInstructionSequence_New() #include "pycore_interpframe.h" // _PyFrame_GetFunction() #include "pycore_jit.h" // _PyJIT_AddressInJitCode() +#include "pycore_lock.h" // PyEvent_WaitTimed() #include "pycore_object.h" // _PyObject_IsFreed() #include "pycore_optimizer.h" // _Py_Executor_DependsOn #include "pycore_pathconfig.h" // _PyPathConfig_ClearGlobal() @@ -208,6 +209,23 @@ get_stack_margin(PyObject *self, PyObject *Py_UNUSED(args)) return PyLong_FromSize_t(_PyOS_STACK_MARGIN_BYTES); } +static PyObject * +test_stop_the_world(PyObject *self, PyObject *Py_UNUSED(args)) +{ +#ifdef Py_GIL_DISABLED + PyInterpreterState *interp = _PyInterpreterState_GET(); + // Request consecutive pauses without running Python code between them. + for (int i = 0; i < 100; i++) { + _PyEval_StopTheWorld(interp); + // Give detached threads time to try to reattach during the pause. + PyEvent event = {0}; + PyEvent_WaitTimed(&event, 10 * 1000 * 1000, /*detach=*/0); + _PyEval_StartTheWorld(interp); + } +#endif + Py_RETURN_NONE; +} + #ifdef MS_WINDOWS static const char * classify_address(uintptr_t addr, int jit_enabled, PyInterpreterState *interp) @@ -3298,6 +3316,7 @@ static PyMethodDef module_functions[] = { {"get_c_recursion_remaining", get_c_recursion_remaining, METH_NOARGS}, {"get_stack_pointer", get_stack_pointer, METH_NOARGS}, {"get_stack_margin", get_stack_margin, METH_NOARGS}, + {"test_stop_the_world", test_stop_the_world, METH_NOARGS}, {"classify_stack_addresses", classify_stack_addresses, METH_VARARGS}, {"get_jit_code_ranges", get_jit_code_ranges, METH_NOARGS}, {"get_jit_backend", get_jit_backend, METH_NOARGS}, diff --git a/Python/pystate.c b/Python/pystate.c index 75bb7520c9ec8cc..737bf0f216bd34e 100644 --- a/Python/pystate.c +++ b/Python/pystate.c @@ -1939,7 +1939,10 @@ tstate_delete_common(PyThreadState *tstate, int release_gil) if (tstate->next) { tstate->next->prev = tstate->prev; } - if (tstate->state != _Py_THREAD_SUSPENDED) { + int state = _Py_atomic_load_int_relaxed(&tstate->state); + if (state != _Py_THREAD_SUSPENDED && + state != _Py_THREAD_SUSPENDED_WAITING) + { // Any ongoing stop-the-world request should not wait for us because // our thread is getting deleted. if (interp->stoptheworld.requested) { @@ -2223,6 +2226,22 @@ tstate_try_attach(PyThreadState *tstate) #endif } +static int +tstate_try_attach_detached(PyThreadState *tstate, int *state) +{ +#ifdef Py_GIL_DISABLED + assert(*state == _Py_THREAD_DETACHED || + *state == _Py_THREAD_DETACHED_WAITING); + return _Py_atomic_compare_exchange_int(&tstate->state, + state, + _Py_THREAD_ATTACHED); +#else + assert(tstate->state == _Py_THREAD_DETACHED); + tstate->state = _Py_THREAD_ATTACHED; + return 1; +#endif +} + static void tstate_set_detached(PyThreadState *tstate, int detached_state) { @@ -2237,10 +2256,20 @@ tstate_set_detached(PyThreadState *tstate, int detached_state) static void tstate_wait_attach(PyThreadState *tstate) { - do { + for (;;) { int state = _Py_atomic_load_int_relaxed(&tstate->state); if (state == _Py_THREAD_SUSPENDED) { - // Wait until we're switched out of SUSPENDED to DETACHED. + // Register an active attach waiter. The next stop-the-world + // request must let this thread attach before suspending it again. + if (!_Py_atomic_compare_exchange_int( + &tstate->state, &state, _Py_THREAD_SUSPENDED_WAITING)) + { + continue; + } + state = _Py_THREAD_SUSPENDED_WAITING; + } + if (state == _Py_THREAD_SUSPENDED_WAITING) { + // Park rechecks the state before sleeping, in case we were resumed. _PyParkingLot_Park(&tstate->state, &state, sizeof(tstate->state), /*timeout=*/-1, NULL, /*detach=*/0); } @@ -2249,10 +2278,13 @@ tstate_wait_attach(PyThreadState *tstate) _PyThreadState_HangThread(tstate); } else { - assert(state == _Py_THREAD_DETACHED); + assert(state == _Py_THREAD_DETACHED || + state == _Py_THREAD_DETACHED_WAITING); + if (tstate_try_attach_detached(tstate, &state)) { + return; + } } - // Once we're back in DETACHED we can re-attach - } while (!tstate_try_attach(tstate)); + } } void @@ -2394,10 +2426,24 @@ void _PyThreadState_ResumeDetached(PyThreadState *tstate) { assert(tstate != _PyThreadState_GET()); - assert(_Py_atomic_load_int_relaxed(&tstate->state) == _Py_THREAD_SUSPENDED); - _Py_atomic_store_int(&tstate->state, _Py_THREAD_DETACHED); + int state = _Py_atomic_load_int_relaxed(&tstate->state); + int next_state; + do { + assert(state == _Py_THREAD_SUSPENDED || + state == _Py_THREAD_SUSPENDED_WAITING); + if (state == _Py_THREAD_SUSPENDED_WAITING) { + next_state = _Py_THREAD_DETACHED_WAITING; + } + else { + next_state = _Py_THREAD_DETACHED; + } + // Retry if an attach waiter registered concurrently. + } while (!_Py_atomic_compare_exchange_int( + &tstate->state, &state, next_state)); // Wake the thread if it is parked in tstate_wait_attach(). - _PyParkingLot_UnparkAll(&tstate->state); + if (state == _Py_THREAD_SUSPENDED_WAITING) { + _PyParkingLot_UnparkAll(&tstate->state); + } } #endif @@ -2442,6 +2488,8 @@ park_detached_threads(struct _stoptheworld_state *stw) _Py_FOR_EACH_STW_INTERP(stw, i) { _Py_FOR_EACH_TSTATE_UNLOCKED(i, t) { int state = _Py_atomic_load_int_relaxed(&t->state); + // DETACHED_WAITING threads remain counted until they attach and + // stop, so repeated pauses cannot prevent them from attaching. if (state == _Py_THREAD_DETACHED) { // Atomically transition to "suspended" if in "detached" state. if (_Py_atomic_compare_exchange_int( @@ -2530,10 +2578,7 @@ start_the_world(struct _stoptheworld_state *stw) _Py_FOR_EACH_STW_INTERP(stw, i) { _Py_FOR_EACH_TSTATE_UNLOCKED(i, t) { if (t != stw->requester) { - assert(_Py_atomic_load_int_relaxed(&t->state) == - _Py_THREAD_SUSPENDED); - _Py_atomic_store_int(&t->state, _Py_THREAD_DETACHED); - _PyParkingLot_UnparkAll(&t->state); + _PyThreadState_ResumeDetached(t); } } } From 198bc7683641f6b44d23fe5dc228f732a7cf3df8 Mon Sep 17 00:00:00 2001 From: Sergey B Kirpichev Date: Fri, 25 Sep 2026 21:51:59 +0300 Subject: [PATCH 2/4] gh-151464: exclude '<>' token from tokenize output (#154854) * gh-151464: exclude '<>' token from tokenize output Was: ``` $ echo '1 <> 2' | python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,4: OP '<>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` Now (regardless on ``__future__.barry_as_FLUFL`` import): ``` $ echo '1 <> 2' | ./python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,3: OP '<' 1,3-1,4: OP '>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` in accordance with the Grammar: https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters Also adds a custom error message for ``<>`` ("not equal" in Pascal and Python 2). * +1 * address review: lowercase and move invalid rule * address review: news * address review: move test_guido_as_bdfl_ineq_tokens() * address review: revert _PyTokenizer_From* changes * + revert unrelated change * address review: remove whatsnew entry --- Grammar/python.gram | 8 + Include/internal/pycore_token.h | 2 +- Lib/test/test_syntax.py | 25 ++ Lib/test/test_tokenize.py | 14 + ...-07-29-08-43-14.gh-issue-151464.o_mtmz.rst | 2 + Parser/action_helpers.c | 2 + Parser/lexer/lexer.c | 2 +- Parser/lexer/state.h | 1 + Parser/parser.c | 424 ++++++++++-------- Parser/pegen.c | 8 +- Parser/token.c | 4 +- Tools/build/generate_token.py | 9 +- 12 files changed, 314 insertions(+), 187 deletions(-) create mode 100644 Misc/NEWS.d/next/Core_and_Builtins/2026-07-29-08-43-14.gh-issue-151464.o_mtmz.rst diff --git a/Grammar/python.gram b/Grammar/python.gram index 28f5ed3a6e61d4a..f2c9993aa4f80a7 100644 --- a/Grammar/python.gram +++ b/Grammar/python.gram @@ -789,6 +789,7 @@ compare_op_bitwise_or_pair[CmpopExprPair*]: | (tok='!=' { _PyPegen_check_barry_as_flufl(p, tok) ? NULL : tok }) a=bitwise_or { _PyPegen_cmpop_expr_pair(p, NotEq, a) } | '<=' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, LtE, a) } + | invalid_noteq | '<' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, Lt, a) } | '>=' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, GtE, a) } | '>' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, Gt, a) } @@ -1615,3 +1616,10 @@ invalid_bitwise_or: ? RAISE_SYNTAX_ERROR_KNOWN_RANGE(b, c, "invalid syntax. Maybe you meant 'or' or '|' instead of '||'?") : NULL } + +invalid_noteq: + | a='<' b='>' { + _PyPegen_tokens_are_adjacent(a, b) + ? RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "invalid syntax. Maybe you meant '!=' instead of '<>'?") + : NULL + } diff --git a/Include/internal/pycore_token.h b/Include/internal/pycore_token.h index 5de1f719a2f1a28..8dcd4226745dd0a 100644 --- a/Include/internal/pycore_token.h +++ b/Include/internal/pycore_token.h @@ -101,7 +101,7 @@ extern "C" { // Export these 4 symbols for 'test_peg_generator' PyAPI_DATA(const char * const) _PyParser_TokenNames[]; /* Token names */ PyAPI_FUNC(int) _PyToken_OneChar(int); -PyAPI_FUNC(int) _PyToken_TwoChars(int, int); +PyAPI_FUNC(int) _PyToken_TwoChars(int, int, int); PyAPI_FUNC(int) _PyToken_ThreeChars(int, int, int); #ifdef __cplusplus diff --git a/Lib/test/test_syntax.py b/Lib/test/test_syntax.py index 2001b70739fd26e..39ab53bb1f8b5fe 100644 --- a/Lib/test/test_syntax.py +++ b/Lib/test/test_syntax.py @@ -3644,6 +3644,31 @@ def test_ifexp_body_stmt_else_stmt(self): ]: self._check_error(f"x = {lhs_stmt} if 1 else {rhs_stmt}", msg) + def test_diamond_operator(self): + self._check_error( + "1<>2", + r"Maybe you meant '!=' instead of '<>'\?", + lineno=1, + end_lineno=1, + offset=2, + end_offset=4, + ) + + def test_diamond_operator_barry_as_flufl(self): + # Under barry_as_FLUFL, '<>' is the valid "not equal" operator + compile( + "from __future__ import barry_as_FLUFL\n1<>2", + "", "exec", + ) + self._check_error( + "from __future__ import barry_as_FLUFL\na != b", + "with Barry as BDFL, use '<>' instead of '!='", + lineno=2, + end_lineno=2, + offset=3, + end_offset=5, + ) + def test_double_ampersand(self): self._check_error( "a && b", diff --git a/Lib/test/test_tokenize.py b/Lib/test/test_tokenize.py index 53215eceeb8aed3..cb8f58511aab877 100644 --- a/Lib/test/test_tokenize.py +++ b/Lib/test/test_tokenize.py @@ -1234,6 +1234,20 @@ def test_multiline_non_ascii_fstring_with_expr(self): FSTRING_END \'"\' (2, 2) (2, 3) """) + def test_ineq_tokens(self): + self.check_tokenize("1 != 2", """\ + NUMBER '1' (1, 0) (1, 1) + OP '!=' (1, 2) (1, 4) + NUMBER '2' (1, 5) (1, 6) +""") + self.check_tokenize("1 <> 2", """\ + NUMBER '1' (1, 0) (1, 1) + OP '<' (1, 2) (1, 3) + OP '>' (1, 3) (1, 4) + NUMBER '2' (1, 5) (1, 6) +""") + + class GenerateTokensTest(TokenizeTest): def check_tokenize(self, s, expected): # Format the tokens in s in a table format. diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2026-07-29-08-43-14.gh-issue-151464.o_mtmz.rst b/Misc/NEWS.d/next/Core_and_Builtins/2026-07-29-08-43-14.gh-issue-151464.o_mtmz.rst new file mode 100644 index 000000000000000..ba7dbe132540373 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2026-07-29-08-43-14.gh-issue-151464.o_mtmz.rst @@ -0,0 +1,2 @@ +Exclude invalid token ``<>`` from :mod:`tokenize` output. :exc:`SyntaxError` +for ``<>`` now suggests ``!=``. diff --git a/Parser/action_helpers.c b/Parser/action_helpers.c index 88a9df98aa5b62a..2d51795d887993d 100644 --- a/Parser/action_helpers.c +++ b/Parser/action_helpers.c @@ -6,6 +6,7 @@ #include "pegen.h" #include "string_parser.h" // _PyPegen_decode_string() +#include "lexer/state.h" // tok_state void * @@ -2133,6 +2134,7 @@ _PyPegen_checked_from_import(Parser *p, asdl_seq *dots, expr_ty module_name, alias_ty alias = asdl_seq_GET(names, i); if (PyUnicode_CompareWithASCIIString(alias->name, "barry_as_FLUFL") == 0) { p->flags |= PyPARSE_BARRY_AS_BDFL; + p->tok->barry_as_bdfl = 1; } } } diff --git a/Parser/lexer/lexer.c b/Parser/lexer/lexer.c index f27fee8d61b9085..4652103107367ef 100644 --- a/Parser/lexer/lexer.c +++ b/Parser/lexer/lexer.c @@ -409,7 +409,7 @@ _PyLexer_get_normal(struct tok_state *tok, ftstring_state *current, struct token /* Check for two-character token */ { int c2 = tok_nextc(tok); - int current_token = _PyToken_TwoChars(c, c2); + int current_token = _PyToken_TwoChars(c, c2, tok->barry_as_bdfl); if (current_token != OP) { int c3 = tok_nextc(tok); int current_token3 = _PyToken_ThreeChars(c, c2, c3); diff --git a/Parser/lexer/state.h b/Parser/lexer/state.h index dff67c2ba83dae4..3b5d47e12a6c38f 100644 --- a/Parser/lexer/state.h +++ b/Parser/lexer/state.h @@ -108,6 +108,7 @@ struct tok_state { #ifdef Py_DEBUG int debug; #endif + int barry_as_bdfl; }; static inline ftstring_state * diff --git a/Parser/parser.c b/Parser/parser.c index b3d73dd4fafb762..896c089e3360035 100644 --- a/Parser/parser.c +++ b/Parser/parser.c @@ -350,184 +350,185 @@ static char *soft_keywords[] = { #define invalid_type_params_type 1261 #define invalid_bitwise_and_type 1262 // Left-recursive #define invalid_bitwise_or_type 1263 // Left-recursive -#define _loop0_1_type 1264 -#define _loop1_2_type 1265 -#define _loop0_3_type 1266 -#define _gather_4_type 1267 -#define _tmp_5_type 1268 -#define _tmp_6_type 1269 -#define _tmp_7_type 1270 -#define _tmp_8_type 1271 -#define _tmp_9_type 1272 -#define _tmp_10_type 1273 -#define _tmp_11_type 1274 -#define _loop1_12_type 1275 -#define _loop0_13_type 1276 -#define _gather_14_type 1277 -#define _tmp_15_type 1278 -#define _tmp_16_type 1279 -#define _loop0_17_type 1280 -#define _loop1_18_type 1281 -#define _loop0_19_type 1282 -#define _gather_20_type 1283 -#define _tmp_21_type 1284 -#define _loop0_22_type 1285 -#define _gather_23_type 1286 -#define _loop1_24_type 1287 -#define _tmp_25_type 1288 -#define _tmp_26_type 1289 -#define _loop0_27_type 1290 -#define _loop0_28_type 1291 -#define _loop1_29_type 1292 -#define _loop1_30_type 1293 -#define _loop0_31_type 1294 -#define _loop1_32_type 1295 -#define _loop0_33_type 1296 -#define _gather_34_type 1297 -#define _tmp_35_type 1298 -#define _loop1_36_type 1299 -#define _loop1_37_type 1300 -#define _loop1_38_type 1301 -#define _loop0_39_type 1302 -#define _gather_40_type 1303 -#define _tmp_41_type 1304 -#define _tmp_42_type 1305 -#define _tmp_43_type 1306 -#define _loop0_44_type 1307 -#define _gather_45_type 1308 -#define _loop0_46_type 1309 -#define _gather_47_type 1310 -#define _tmp_48_type 1311 -#define _loop0_49_type 1312 -#define _gather_50_type 1313 -#define _loop0_51_type 1314 -#define _gather_52_type 1315 -#define _loop0_53_type 1316 -#define _gather_54_type 1317 -#define _loop1_55_type 1318 -#define _loop1_56_type 1319 -#define _loop0_57_type 1320 -#define _gather_58_type 1321 -#define _loop1_59_type 1322 -#define _loop1_60_type 1323 -#define _loop1_61_type 1324 -#define _tmp_62_type 1325 -#define _loop0_63_type 1326 -#define _gather_64_type 1327 -#define _tmp_65_type 1328 -#define _tmp_66_type 1329 -#define _tmp_67_type 1330 -#define _tmp_68_type 1331 -#define _tmp_69_type 1332 -#define _loop0_70_type 1333 -#define _loop0_71_type 1334 -#define _loop1_72_type 1335 -#define _loop1_73_type 1336 -#define _loop0_74_type 1337 -#define _loop1_75_type 1338 -#define _loop0_76_type 1339 -#define _loop0_77_type 1340 -#define _loop0_78_type 1341 -#define _loop0_79_type 1342 -#define _loop1_80_type 1343 -#define _loop1_81_type 1344 -#define _loop0_82_type 1345 -#define _gather_83_type 1346 -#define _tmp_84_type 1347 -#define _loop0_85_type 1348 -#define _gather_86_type 1349 -#define _loop1_87_type 1350 -#define _loop0_88_type 1351 -#define _tmp_89_type 1352 -#define _loop0_90_type 1353 -#define _gather_91_type 1354 -#define _tmp_92_type 1355 -#define _loop0_93_type 1356 -#define _gather_94_type 1357 -#define _tmp_95_type 1358 -#define _loop0_96_type 1359 -#define _gather_97_type 1360 -#define _loop0_98_type 1361 -#define _tmp_99_type 1362 -#define _tmp_100_type 1363 -#define _loop0_101_type 1364 -#define _gather_102_type 1365 -#define _loop0_103_type 1366 -#define _gather_104_type 1367 -#define _tmp_105_type 1368 -#define _tmp_106_type 1369 -#define _loop0_107_type 1370 -#define _gather_108_type 1371 -#define _tmp_109_type 1372 -#define _tmp_110_type 1373 -#define _tmp_111_type 1374 -#define _tmp_112_type 1375 -#define _loop1_113_type 1376 -#define _tmp_114_type 1377 -#define _tmp_115_type 1378 -#define _tmp_116_type 1379 -#define _tmp_117_type 1380 -#define _tmp_118_type 1381 -#define _loop0_119_type 1382 -#define _loop0_120_type 1383 -#define _tmp_121_type 1384 -#define _tmp_122_type 1385 -#define _tmp_123_type 1386 -#define _tmp_124_type 1387 -#define _tmp_125_type 1388 -#define _tmp_126_type 1389 -#define _tmp_127_type 1390 -#define _tmp_128_type 1391 -#define _loop0_129_type 1392 -#define _gather_130_type 1393 -#define _tmp_131_type 1394 -#define _tmp_132_type 1395 -#define _tmp_133_type 1396 -#define _tmp_134_type 1397 -#define _loop0_135_type 1398 -#define _gather_136_type 1399 -#define _tmp_137_type 1400 -#define _loop0_138_type 1401 -#define _gather_139_type 1402 -#define _loop0_140_type 1403 -#define _gather_141_type 1404 -#define _tmp_142_type 1405 -#define _loop0_143_type 1406 -#define _tmp_144_type 1407 -#define _tmp_145_type 1408 -#define _tmp_146_type 1409 -#define _tmp_147_type 1410 -#define _tmp_148_type 1411 -#define _tmp_149_type 1412 -#define _tmp_150_type 1413 -#define _tmp_151_type 1414 -#define _tmp_152_type 1415 -#define _tmp_153_type 1416 -#define _tmp_154_type 1417 -#define _tmp_155_type 1418 -#define _tmp_156_type 1419 -#define _tmp_157_type 1420 -#define _tmp_158_type 1421 -#define _tmp_159_type 1422 -#define _tmp_160_type 1423 -#define _tmp_161_type 1424 -#define _tmp_162_type 1425 -#define _tmp_163_type 1426 -#define _tmp_164_type 1427 -#define _tmp_165_type 1428 -#define _tmp_166_type 1429 -#define _tmp_167_type 1430 -#define _tmp_168_type 1431 -#define _tmp_169_type 1432 -#define _tmp_170_type 1433 -#define _tmp_171_type 1434 -#define _loop0_172_type 1435 -#define _tmp_173_type 1436 -#define _tmp_174_type 1437 -#define _tmp_175_type 1438 -#define _tmp_176_type 1439 -#define _tmp_177_type 1440 -#define _tmp_178_type 1441 +#define invalid_noteq_type 1264 +#define _loop0_1_type 1265 +#define _loop1_2_type 1266 +#define _loop0_3_type 1267 +#define _gather_4_type 1268 +#define _tmp_5_type 1269 +#define _tmp_6_type 1270 +#define _tmp_7_type 1271 +#define _tmp_8_type 1272 +#define _tmp_9_type 1273 +#define _tmp_10_type 1274 +#define _tmp_11_type 1275 +#define _loop1_12_type 1276 +#define _loop0_13_type 1277 +#define _gather_14_type 1278 +#define _tmp_15_type 1279 +#define _tmp_16_type 1280 +#define _loop0_17_type 1281 +#define _loop1_18_type 1282 +#define _loop0_19_type 1283 +#define _gather_20_type 1284 +#define _tmp_21_type 1285 +#define _loop0_22_type 1286 +#define _gather_23_type 1287 +#define _loop1_24_type 1288 +#define _tmp_25_type 1289 +#define _tmp_26_type 1290 +#define _loop0_27_type 1291 +#define _loop0_28_type 1292 +#define _loop1_29_type 1293 +#define _loop1_30_type 1294 +#define _loop0_31_type 1295 +#define _loop1_32_type 1296 +#define _loop0_33_type 1297 +#define _gather_34_type 1298 +#define _tmp_35_type 1299 +#define _loop1_36_type 1300 +#define _loop1_37_type 1301 +#define _loop1_38_type 1302 +#define _loop0_39_type 1303 +#define _gather_40_type 1304 +#define _tmp_41_type 1305 +#define _tmp_42_type 1306 +#define _tmp_43_type 1307 +#define _loop0_44_type 1308 +#define _gather_45_type 1309 +#define _loop0_46_type 1310 +#define _gather_47_type 1311 +#define _tmp_48_type 1312 +#define _loop0_49_type 1313 +#define _gather_50_type 1314 +#define _loop0_51_type 1315 +#define _gather_52_type 1316 +#define _loop0_53_type 1317 +#define _gather_54_type 1318 +#define _loop1_55_type 1319 +#define _loop1_56_type 1320 +#define _loop0_57_type 1321 +#define _gather_58_type 1322 +#define _loop1_59_type 1323 +#define _loop1_60_type 1324 +#define _loop1_61_type 1325 +#define _tmp_62_type 1326 +#define _loop0_63_type 1327 +#define _gather_64_type 1328 +#define _tmp_65_type 1329 +#define _tmp_66_type 1330 +#define _tmp_67_type 1331 +#define _tmp_68_type 1332 +#define _tmp_69_type 1333 +#define _loop0_70_type 1334 +#define _loop0_71_type 1335 +#define _loop1_72_type 1336 +#define _loop1_73_type 1337 +#define _loop0_74_type 1338 +#define _loop1_75_type 1339 +#define _loop0_76_type 1340 +#define _loop0_77_type 1341 +#define _loop0_78_type 1342 +#define _loop0_79_type 1343 +#define _loop1_80_type 1344 +#define _loop1_81_type 1345 +#define _loop0_82_type 1346 +#define _gather_83_type 1347 +#define _tmp_84_type 1348 +#define _loop0_85_type 1349 +#define _gather_86_type 1350 +#define _loop1_87_type 1351 +#define _loop0_88_type 1352 +#define _tmp_89_type 1353 +#define _loop0_90_type 1354 +#define _gather_91_type 1355 +#define _tmp_92_type 1356 +#define _loop0_93_type 1357 +#define _gather_94_type 1358 +#define _tmp_95_type 1359 +#define _loop0_96_type 1360 +#define _gather_97_type 1361 +#define _loop0_98_type 1362 +#define _tmp_99_type 1363 +#define _tmp_100_type 1364 +#define _loop0_101_type 1365 +#define _gather_102_type 1366 +#define _loop0_103_type 1367 +#define _gather_104_type 1368 +#define _tmp_105_type 1369 +#define _tmp_106_type 1370 +#define _loop0_107_type 1371 +#define _gather_108_type 1372 +#define _tmp_109_type 1373 +#define _tmp_110_type 1374 +#define _tmp_111_type 1375 +#define _tmp_112_type 1376 +#define _loop1_113_type 1377 +#define _tmp_114_type 1378 +#define _tmp_115_type 1379 +#define _tmp_116_type 1380 +#define _tmp_117_type 1381 +#define _tmp_118_type 1382 +#define _loop0_119_type 1383 +#define _loop0_120_type 1384 +#define _tmp_121_type 1385 +#define _tmp_122_type 1386 +#define _tmp_123_type 1387 +#define _tmp_124_type 1388 +#define _tmp_125_type 1389 +#define _tmp_126_type 1390 +#define _tmp_127_type 1391 +#define _tmp_128_type 1392 +#define _loop0_129_type 1393 +#define _gather_130_type 1394 +#define _tmp_131_type 1395 +#define _tmp_132_type 1396 +#define _tmp_133_type 1397 +#define _tmp_134_type 1398 +#define _loop0_135_type 1399 +#define _gather_136_type 1400 +#define _tmp_137_type 1401 +#define _loop0_138_type 1402 +#define _gather_139_type 1403 +#define _loop0_140_type 1404 +#define _gather_141_type 1405 +#define _tmp_142_type 1406 +#define _loop0_143_type 1407 +#define _tmp_144_type 1408 +#define _tmp_145_type 1409 +#define _tmp_146_type 1410 +#define _tmp_147_type 1411 +#define _tmp_148_type 1412 +#define _tmp_149_type 1413 +#define _tmp_150_type 1414 +#define _tmp_151_type 1415 +#define _tmp_152_type 1416 +#define _tmp_153_type 1417 +#define _tmp_154_type 1418 +#define _tmp_155_type 1419 +#define _tmp_156_type 1420 +#define _tmp_157_type 1421 +#define _tmp_158_type 1422 +#define _tmp_159_type 1423 +#define _tmp_160_type 1424 +#define _tmp_161_type 1425 +#define _tmp_162_type 1426 +#define _tmp_163_type 1427 +#define _tmp_164_type 1428 +#define _tmp_165_type 1429 +#define _tmp_166_type 1430 +#define _tmp_167_type 1431 +#define _tmp_168_type 1432 +#define _tmp_169_type 1433 +#define _tmp_170_type 1434 +#define _tmp_171_type 1435 +#define _loop0_172_type 1436 +#define _tmp_173_type 1437 +#define _tmp_174_type 1438 +#define _tmp_175_type 1439 +#define _tmp_176_type 1440 +#define _tmp_177_type 1441 +#define _tmp_178_type 1442 static mod_ty file_rule(Parser *p); static mod_ty interactive_rule(Parser *p); @@ -793,6 +794,7 @@ static void *invalid_factor_rule(Parser *p); static void *invalid_type_params_rule(Parser *p); static void *invalid_bitwise_and_rule(Parser *p); static void *invalid_bitwise_or_rule(Parser *p); +static void *invalid_noteq_rule(Parser *p); static asdl_seq *_loop0_1_rule(Parser *p); static asdl_seq *_loop1_2_rule(Parser *p); static asdl_seq *_loop0_3_rule(Parser *p); @@ -12712,6 +12714,7 @@ comparison_rule(Parser *p) // | '==' bitwise_or // | ('!=') bitwise_or // | '<=' bitwise_or +// | invalid_noteq // | '<' bitwise_or // | '>=' bitwise_or // | '>' bitwise_or @@ -12812,6 +12815,25 @@ compare_op_bitwise_or_pair_rule(Parser *p) D(fprintf(stderr, "%*c%s compare_op_bitwise_or_pair[%d-%d]: %s failed!\n", p->level, ' ', p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "'<=' bitwise_or")); } + if (p->call_invalid_rules) { // invalid_noteq + if (p->error_indicator) { + p->level--; + return NULL; + } + D(fprintf(stderr, "%*c> compare_op_bitwise_or_pair[%d-%d]: %s\n", p->level, ' ', _mark, p->mark, "invalid_noteq")); + void *invalid_noteq_var; + if ( + (invalid_noteq_var = invalid_noteq_rule(p)) // invalid_noteq + ) + { + D(fprintf(stderr, "%*c+ compare_op_bitwise_or_pair[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "invalid_noteq")); + _res = invalid_noteq_var; + goto done; + } + p->mark = _mark; + D(fprintf(stderr, "%*c%s compare_op_bitwise_or_pair[%d-%d]: %s failed!\n", p->level, ' ', + p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "invalid_noteq")); + } { // '<' bitwise_or if (p->error_indicator) { p->level--; @@ -27269,6 +27291,52 @@ invalid_bitwise_or_rule(Parser *p) return _res; } +// invalid_noteq: '<' '>' +static void * +invalid_noteq_rule(Parser *p) +{ + if (p->level++ == MAXSTACK || _PyPegen_stack_exhausted(p)) { + _Pypegen_stack_overflow(p); + } + if (p->error_indicator) { + p->level--; + return NULL; + } + void * _res = NULL; + int _mark = p->mark; + { // '<' '>' + if (p->error_indicator) { + p->level--; + return NULL; + } + D(fprintf(stderr, "%*c> invalid_noteq[%d-%d]: %s\n", p->level, ' ', _mark, p->mark, "'<' '>'")); + Token * a; + Token * b; + if ( + (a = _PyPegen_expect_token(p, 20)) // token='<' + && + (b = _PyPegen_expect_token(p, 21)) // token='>' + ) + { + D(fprintf(stderr, "%*c+ invalid_noteq[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'<' '>'")); + _res = _PyPegen_tokens_are_adjacent ( a , b ) ? RAISE_SYNTAX_ERROR_KNOWN_RANGE ( a , b , "invalid syntax. Maybe you meant '!=' instead of '<>'?" ) : NULL; + if ((_res == NULL || p->error_indicator) && PyErr_Occurred()) { + p->error_indicator = 1; + p->level--; + return NULL; + } + goto done; + } + p->mark = _mark; + D(fprintf(stderr, "%*c%s invalid_noteq[%d-%d]: %s failed!\n", p->level, ' ', + p->error_indicator ? "ERROR!" : "-", _mark, p->mark, "'<' '>'")); + } + _res = NULL; + done: + p->level--; + return _res; +} + // _loop0_1: NEWLINE static asdl_seq * _loop0_1_rule(Parser *p) diff --git a/Parser/pegen.c b/Parser/pegen.c index 5edf5542c7a8f99..7b228ca874bc0ba 100644 --- a/Parser/pegen.c +++ b/Parser/pegen.c @@ -7,6 +7,7 @@ #include "pycore_unicodeobject.h" // _PyUnicode_InternImmortal #include +#include "lexer/state.h" #include "tokenizer/tokenizer.h" #include "pegen.h" @@ -1124,6 +1125,7 @@ _PyPegen_run_parser_from_file_pointer(FILE *fp, int start_rule, PyObject *filena PyCompilerFlags *flags, int *errcode, PyObject **interactive_src, PyArena *arena) { + int parser_flags = compute_parser_flags(flags); struct tok_state *tok = _PyTokenizer_FromFile(fp, enc, ps1, ps2); if (tok == NULL) { if (PyErr_Occurred()) { @@ -1135,6 +1137,7 @@ _PyPegen_run_parser_from_file_pointer(FILE *fp, int start_rule, PyObject *filena } return NULL; } + tok->barry_as_bdfl = parser_flags & PyPARSE_BARRY_AS_BDFL; // From here on we need to clean up even if there's an error mod_ty result = NULL; @@ -1146,7 +1149,6 @@ _PyPegen_run_parser_from_file_pointer(FILE *fp, int start_rule, PyObject *filena _PyTokenizer_SetContext(tok, filename_ob, module); Py_DECREF(module); - int parser_flags = compute_parser_flags(flags); Parser *p = _PyPegen_Parser_New(tok, start_rule, parser_flags, PY_MINOR_VERSION, errcode, NULL, arena); if (p == NULL) { @@ -1176,6 +1178,7 @@ _PyPegen_run_parser_from_string(const char *str, int start_rule, PyObject *filen PyCompilerFlags *flags, PyArena *arena, PyObject *module) { int exec_input = start_rule == Py_file_input; + int parser_flags = compute_parser_flags(flags); struct tok_state *tok; if (flags != NULL && flags->cf_flags & PyCF_IGNORE_COOKIE) { @@ -1193,12 +1196,13 @@ _PyPegen_run_parser_from_string(const char *str, int start_rule, PyObject *filen } return NULL; } + tok->barry_as_bdfl = parser_flags & PyPARSE_BARRY_AS_BDFL; + _PyTokenizer_SetContext(tok, filename_ob, module); // We need to clear up from here on mod_ty result = NULL; - int parser_flags = compute_parser_flags(flags); int feature_version = flags && (flags->cf_flags & PyCF_ONLY_AST) ? flags->cf_feature_version : PY_MINOR_VERSION; Parser *p = _PyPegen_Parser_New(tok, start_rule, parser_flags, feature_version, diff --git a/Parser/token.c b/Parser/token.c index a6e7ce35dfad855..0a729e5d8a368b3 100644 --- a/Parser/token.c +++ b/Parser/token.c @@ -113,7 +113,7 @@ _PyToken_OneChar(int c1) } int -_PyToken_TwoChars(int c1, int c2) +_PyToken_TwoChars(int c1, int c2, int BARRY_AS_BDFL) { switch (c1) { case '!': @@ -163,7 +163,7 @@ _PyToken_TwoChars(int c1, int c2) switch (c2) { case '<': return LEFTSHIFT; case '=': return LESSEQUAL; - case '>': return NOTEQUAL; + case '>': return BARRY_AS_BDFL ? NOTEQUAL : OP; } break; case '=': diff --git a/Tools/build/generate_token.py b/Tools/build/generate_token.py index 9bc96d338fdafa2..ae9e69ab9cef425 100755 --- a/Tools/build/generate_token.py +++ b/Tools/build/generate_token.py @@ -94,7 +94,7 @@ def update_file(file, content): // Export these 4 symbols for 'test_peg_generator' PyAPI_DATA(const char * const) _PyParser_TokenNames[]; /* Token names */ PyAPI_FUNC(int) _PyToken_OneChar(int); -PyAPI_FUNC(int) _PyToken_TwoChars(int, int); +PyAPI_FUNC(int) _PyToken_TwoChars(int, int, int); PyAPI_FUNC(int) _PyToken_ThreeChars(int, int, int); #ifdef __cplusplus @@ -142,7 +142,7 @@ def make_h(infile, outfile='Include/internal/pycore_token.h'): } int -_PyToken_TwoChars(int c1, int c2) +_PyToken_TwoChars(int c1, int c2, int BARRY_AS_BDFL) { %s\ return OP; @@ -171,7 +171,10 @@ def generate_chars_to_token(mapping, n=1): write(indent) write(' break;\n') else: - write("case '%s': return %s;\n" % (c, value)) + if c == '>' and value == 'NOTEQUAL': + write("case '%s': return BARRY_AS_BDFL ? %s : OP;\n" % (c, value)) + else: + write("case '%s': return %s;\n" % (c, value)) write(indent) write('}\n') return ''.join(result) From b71989d65780683dbbcc3882afc0ef0d794a6ddb Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sat, 26 Sep 2026 00:31:44 +0200 Subject: [PATCH 3/4] gh-157649: Merge test_cppext into test_cext (#158176) * test_cext now uses a single virtual environment for all tests. It makes the test way faster since creating a virtual environment is quite slow. * Add Py_TARGET_ABI3T test on C++. * Move macro tests from module exec function to test_macros(). * Call Py_BEGIN_CRITICAL_SECTION() on a fresh dictionary object, rather than on the module. * Rename some environment variables to always use "CPYTHON_TEST_" prefix. * Replace support.MS_WINDOWS with MSVC to be more explicit. * Move macro tests to test_macros(). * On macOS, use C++11 by default. --- Lib/test/pythoninfo.py | 27 +- Lib/test/test_cext/__init__.py | 155 ++++++++--- Lib/test/test_cext/extension.c | 423 +++++++++++++++++++++++++---- Lib/test/test_cext/extension.cpp | 3 + Lib/test/test_cext/setup.py | 105 +++++-- Lib/test/test_cppext/__init__.py | 148 ---------- Lib/test/test_cppext/extension.cpp | 375 ------------------------- Lib/test/test_cppext/setup.py | 126 --------- Makefile.pre.in | 1 - 9 files changed, 582 insertions(+), 781 deletions(-) create mode 100644 Lib/test/test_cext/extension.cpp delete mode 100644 Lib/test/test_cppext/__init__.py delete mode 100644 Lib/test/test_cppext/extension.cpp delete mode 100644 Lib/test/test_cppext/setup.py diff --git a/Lib/test/pythoninfo.py b/Lib/test/pythoninfo.py index a6099cda28f07c5..1f0753b9839cdd2 100644 --- a/Lib/test/pythoninfo.py +++ b/Lib/test/pythoninfo.py @@ -917,18 +917,17 @@ def collect_support_threading_helper(info_add): copy_attributes(info_add, threading_helper, 'support_threading_helper.%s', attributes) -def collect_cc(info_add): +def get_compiler_version(sysconfig_var): import sysconfig - - CC = sysconfig.get_config_var('CC') - if not CC: + program = sysconfig.get_config_var(sysconfig_var) + if not program: return try: import shlex - args = shlex.split(CC) + args = shlex.split(program) except ImportError: - args = CC.split() + args = program.split() args.append('--version') stdout = run_command(args) @@ -942,7 +941,21 @@ def collect_cc(info_add): text = first_line(stdout) text = normalize_text(text) - info_add('CC.version', text) + if text: + text = f'[{program}] {text}' + return text + + +def collect_cc(info_add): + # C compiler + version = get_compiler_version('CC') + if version: + info_add('CC.version', version) + + # C++ compiler + version = get_compiler_version('CXX') + if version: + info_add('CXX.version', version) def collect_gdbm(info_add): diff --git a/Lib/test/test_cext/__init__.py b/Lib/test/test_cext/__init__.py index dfc7d230426cf8c..457925413c94ec8 100644 --- a/Lib/test/test_cext/__init__.py +++ b/Lib/test/test_cext/__init__.py @@ -1,28 +1,37 @@ -# gh-116869: Build a basic C test extension to check that the Python C API -# does not emit C compiler warnings. +# gh-116869: Build a C/C++ test extension to check that the Python C API does +# not emit compiler warnings. # # The Python C API must be compatible with building # with the -Werror=declaration-after-statement compiler flag. import os.path +import platform import shlex import shutil import subprocess -import sysconfig import sys +import sysconfig import unittest from test import support +from test.support import os_helper + + +if not support.has_subprocess_support: + raise unittest.SkipTest("requires subprocess support") +SOURCE_DIR = os.path.dirname(__file__) SOURCES = [ - os.path.join(os.path.dirname(__file__), 'extension.c'), + os.path.join(SOURCE_DIR, 'extension.c'), + os.path.join(SOURCE_DIR, 'extension.cpp'), + os.path.join(SOURCE_DIR, 'setup.py'), ] -SETUP = os.path.join(os.path.dirname(__file__), 'setup.py') +MSVC = support.MS_WINDOWS # With MSVC on a debug build, the linker fails with: cannot open file # 'python311.lib', it should look 'python311_d.lib'. -@unittest.skipIf(support.MS_WINDOWS and support.Py_DEBUG, +@unittest.skipIf(MSVC and support.Py_DEBUG, 'test fails on Windows debug build') # Building and running an extension in clang sanitizing mode is not # straightforward @@ -34,41 +43,38 @@ @support.requires_resource('cpu') class BaseTests: TEST_INTERNAL_C_API = False - - # Default build with no options - def test_build(self): - self.check_build('_test_cext') + LANGUAGE = None def check_build(self, extension_name, std=None, limited=False, - abi3t=False): - venv_dir = 'env' - with support.setup_venv_with_pip_setuptools(venv_dir) as python_exe: - self._check_build(extension_name, python_exe, - std=std, limited=limited, - abi3t=abi3t) - - def _check_build(self, extension_name, python_exe, std, limited, - abi3t): + abi3t=False, extra_cflags=None): + if self.LANGUAGE == 'C++' and not std and sys.platform == 'darwin': + # Old Apple clang++ default C++ std is gnu++98, use C++11 instead + std = 'c++11' + pkg_dir = 'pkg' os.mkdir(pkg_dir) - shutil.copy(SETUP, os.path.join(pkg_dir, os.path.basename(SETUP))) + self.addCleanup(os_helper.rmtree, pkg_dir) + for source in SOURCES: dest = os.path.join(pkg_dir, os.path.basename(source)) shutil.copy(source, dest) def run_cmd(operation, cmd): env = os.environ.copy() + env['CPYTHON_TEST_EXT_NAME'] = extension_name + env['CPYTHON_TEST_LANG'] = self.LANGUAGE if std: env['CPYTHON_TEST_STD'] = std if limited: env['CPYTHON_TEST_LIMITED'] = '1' if abi3t: env['CPYTHON_TEST_ABI3T'] = '1' - if support.MS_WINDOWS and sysconfig.is_python_build(): - env['CPYTHON_EXTRA_INCDIRS'] = os.path.split(sysconfig.get_config_h_filename())[0] - env['CPYTHON_EXTRA_LIBDIRS'] = os.path.split(sys.executable)[0] - env['CPYTHON_TEST_EXT_NAME'] = extension_name - env['TEST_INTERNAL_C_API'] = str(int(self.TEST_INTERNAL_C_API)) + if MSVC and sysconfig.is_python_build(): + env['CPYTHON_TEST_EXTRA_INCDIRS'] = os.path.split(sysconfig.get_config_h_filename())[0] + env['CPYTHON_TEST_EXTRA_LIBDIRS'] = os.path.split(sys.executable)[0] + env['CPYTHON_TEST_INTERNAL_C_API'] = str(int(self.TEST_INTERNAL_C_API)) + if extra_cflags: + env['CPYTHON_TEST_EXTRA_CFLAGS'] = extra_cflags if support.verbose: print('Run:', ' '.join(map(shlex.quote, cmd))) subprocess.run(cmd, check=True, env=env) @@ -85,6 +91,7 @@ def run_cmd(operation, cmd): f"{operation} failed with exit code {proc.returncode}") # Build and install the C extension + python_exe = PYTHON_EXE cmd = [python_exe, '-X', 'dev', '-m', 'pip', 'install', '--no-build-isolation', os.path.abspath(pkg_dir)] @@ -101,7 +108,7 @@ def run_cmd(operation, cmd): '-c', 'pass'] run_cmd('Reference run', cmd) - # Import the C extension + # Import the C/C++ extension cmd = [python_exe, '-X', 'dev', '-X', 'showrefcount', @@ -109,31 +116,101 @@ def run_cmd(operation, cmd): run_cmd('Import', cmd) -class TestPublicCAPI(BaseTests, unittest.TestCase): +class TestPublicC(BaseTests, unittest.TestCase): + LANGUAGE = 'C' + + # Default build with no options + def test_build(self): + self.check_build('_test_cext') + + @unittest.skipIf(MSVC, "MSVC doesn't support /std:c99") + def test_build_c99(self): + # In public docs, we say C API is compatible with C11. However, + # in practice we do maintain C99 compatibility in public headers. + # Please ask the C API WG before adding a new C11-only feature. + self.check_build('_test_cext_c99', std='c99') + + def test_build_c11(self): + self.check_build('_test_cext_c11', std='c11') + def test_build_limited(self): - self.check_build('_test_limited_cext', limited=True) + self.check_build('_test_cext_limited', limited=True) def test_build_limited_c11(self): - self.check_build('_test_limited_c11_cext', limited=True, std='c11') + self.check_build('_test_cext_limited_c11', limited=True, std='c11') - def test_build_c11(self): - self.check_build('_test_c11_cext', std='c11') + def test_build_abi3t(self): + # Test with Py_TARGET_ABI3T + self.check_build('_test_cext_abi3t', abi3t=True) + + +class TestPublicCpp(BaseTests, unittest.TestCase): + LANGUAGE = 'C++' + + def test_build(self): + self.check_build('_test_cppext') + + def test_build_cpp03(self): + # In public docs, we say C API is compatible with C++11. However, + # in practice we do maintain C++03 compatibility in public headers. + # Please ask the C API WG before adding a new C++11-only feature. + self.check_build('_test_cppext_cpp03', std='c++03') + + @unittest.skipIf(MSVC, "MSVC doesn't support /std:c++11") + def test_build_cpp11(self): + self.check_build('_test_cppext_cpp11', std='c++11') + + # Only test C++14 on MSVC. + # On s390x RHEL7, GCC 4.8.5 doesn't support C++14. + @unittest.skipIf(not MSVC, "need MSVC") + def test_build_cpp14(self): + self.check_build('_test_cppext_cpp14', std='c++14') + + # Test that headers compile with Intel asm syntax, which may conflict + # with inline assembly in free-threading headers that use AT&T syntax. + @unittest.skipIf(MSVC, "MSVC doesn't support -masm=intel") + @unittest.skipUnless(platform.machine() in ('x86_64', 'i686', 'AMD64'), + "x86-specific flag") + def test_build_intel_asm(self): + self.check_build('_test_cppext_intel_asm', extra_cflags='-masm=intel') + + def test_build_limited(self): + self.check_build('_test_cppext_limited', limited=True) + + def test_build_limited_cpp03(self): + self.check_build('_test_cppext_limited_cpp03', std='c++03', limited=True) def test_build_abi3t(self): # Test with Py_TARGET_ABI3T - self.check_build('_test_abi3t', abi3t=True) + self.check_build('_test_cppext_abi3t', abi3t=True) - @unittest.skipIf(support.MS_WINDOWS, "MSVC doesn't support /std:c99") - def test_build_c99(self): - # In public docs, we say C API is compatible with C11. However, - # in practice we do maintain C99 compatibility in public headers. - # Please ask the C API WG before adding a new C11-only feature. - self.check_build('_test_c99_cext', std='c99') + +class TestInteralC(BaseTests, unittest.TestCase): + LANGUAGE = 'C' + TEST_INTERNAL_C_API = True + + # Default build with no options + def test_build(self): + self.check_build('_test_cext_internal') -class TestInteralCAPI(BaseTests, unittest.TestCase): +class TestInteralCpp(BaseTests, unittest.TestCase): + LANGUAGE = 'C++' TEST_INTERNAL_C_API = True + def test_build(self): + self.check_build('_test_cppext_internal') + + +def setUpModule(): + global VENV_CONTEXT, PYTHON_EXE + VENV_CONTEXT = support.setup_venv_with_pip_setuptools('env') + PYTHON_EXE = VENV_CONTEXT.__enter__() + + +def tearDownModule(): + VENV_CONTEXT.__exit__(None, None, None) + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_cext/extension.c b/Lib/test/test_cext/extension.c index 0a26a0d8753711f..038f1a2af46c671 100644 --- a/Lib/test/test_cext/extension.c +++ b/Lib/test/test_cext/extension.c @@ -1,7 +1,10 @@ -// gh-116869: Basic C test extension to check that the Python C API -// does not emit C compiler warnings. +// gh-116869: C/C++ extension module used to check that the Python C API does +// not emit compiler warnings. // // Test also the internal C API if the TEST_INTERNAL_C_API macro is defined. +// +// Declare variables at the top of each function to support building with +// -Werror=declaration-after-statement. // Always enable assertions #undef NDEBUG @@ -13,21 +16,42 @@ #include "Python.h" #include "datetime.h" -#ifdef TEST_INTERNAL_C_API - // gh-135906: Check for compiler warnings in the internal C API. - // - Cython uses pycore_critical_section.h, pycore_frame.h and - // pycore_template.h. - // - greenlet uses pycore_frame.h, pycore_interpframe_structs.h and - // pycore_interpframe.h. -# include "internal/pycore_critical_section.h" -# include "internal/pycore_frame.h" -# include "internal/pycore_gc.h" -# include "internal/pycore_interp.h" -# include "internal/pycore_interpframe.h" -# include "internal/pycore_interpframe_structs.h" -# include "internal/pycore_object.h" -# include "internal/pycore_pystate.h" -# include "internal/pycore_template.h" +#ifdef __cplusplus +# ifdef TEST_INTERNAL_C_API + // gh-135906: Check for compiler warnings in the internal C API + // - Cython uses pycore_critical_section.h, pycore_frame.h and + // pycore_template.h. + // - greenlet uses pycore_frame.h, pycore_interpframe_structs.h and + // pycore_interpframe.h. +# include "internal/pycore_frame.h" +# include "internal/pycore_interpframe_structs.h" +# include "internal/pycore_template.h" + + // mimalloc emits compiler warnings on Windows. +# if !defined(MS_WINDOWS) +# include "internal/pycore_backoff.h" +# include "internal/pycore_cell.h" +# include "internal/pycore_critical_section.h" +# include "internal/pycore_interpframe.h" +# endif +# endif +#else +# ifdef TEST_INTERNAL_C_API + // gh-135906: Check for compiler warnings in the internal C API. + // - Cython uses pycore_critical_section.h, pycore_frame.h and + // pycore_template.h. + // - greenlet uses pycore_frame.h, pycore_interpframe_structs.h and + // pycore_interpframe.h. +# include "internal/pycore_critical_section.h" +# include "internal/pycore_frame.h" +# include "internal/pycore_gc.h" +# include "internal/pycore_interp.h" +# include "internal/pycore_interpframe.h" +# include "internal/pycore_interpframe_structs.h" +# include "internal/pycore_object.h" +# include "internal/pycore_pystate.h" +# include "internal/pycore_template.h" +# endif #endif #ifndef MODULE_NAME @@ -37,13 +61,13 @@ #define _STR(NAME) #NAME #define STR(NAME) _STR(NAME) -PyDoc_STRVAR(_testcext_add_doc, +PyDoc_STRVAR(test_add_doc, "add(x, y)\n" "\n" "Return the sum of two integers: x + y."); static PyObject * -_testcext_add(PyObject *Py_UNUSED(module), PyObject *args) +test_add(PyObject *Py_UNUSED(module), PyObject *args) { long i, j, res; if (!PyArg_ParseTuple(args, "ll:foo", &i, &j)) { @@ -54,6 +78,50 @@ _testcext_add(PyObject *Py_UNUSED(module), PyObject *args) } +static PyObject * +test_macros(PyObject *Py_UNUSED(module), PyObject *Py_UNUSED(args)) +{ + PyObject *obj, *dict; + + // test Py_BUILD_ASSERT() and Py_BUILD_ASSERT_EXPR() + Py_BUILD_ASSERT(sizeof(int) == sizeof(unsigned int)); + assert(Py_BUILD_ASSERT_EXPR(sizeof(int) == sizeof(unsigned int)) == 0); + + // Test Py_MIN(), Py_MAX(), Py_ABS() + assert(Py_MIN(5, 11) == 5); + assert(Py_MAX(5, 11) == 11); + assert(Py_ABS(-5) == 5); + + // Test Py_CLEAR(): use typeof()/__typeof__() if available, or memcpy() + obj = Py_None; + Py_CLEAR(obj); + assert(obj == _Py_NULL); + +#ifndef Py_LIMITED_API + // Test Py_SETREF(): use typeof()/__typeof__() if available, or memcpy() + obj = Py_None; + Py_SETREF(obj, _Py_NULL); + assert(obj == _Py_NULL); + + // Test Py_XSETREF(): use typeof()/__typeof__() if available, or memcpy() + obj = Py_None; + Py_XSETREF(obj, _Py_NULL); + assert(obj == _Py_NULL); +#endif + + // Test that Py_BEGIN_CRITICAL_SECTION is available + dict = PyDict_New(); + if (dict == NULL) { + return NULL; + } + Py_BEGIN_CRITICAL_SECTION(dict); + Py_END_CRITICAL_SECTION(); + Py_DECREF(dict); + + Py_RETURN_NONE; +} + + static PyObject * test_datetime(PyObject *Py_UNUSED(module), PyObject *Py_UNUSED(args)) { @@ -69,52 +137,261 @@ test_datetime(PyObject *Py_UNUSED(module), PyObject *Py_UNUSED(args)) } -static PyMethodDef _testcext_methods[] = { - {"add", _testcext_add, METH_VARARGS, _testcext_add_doc}, - {"test_datetime", test_datetime, METH_NOARGS, NULL}, - {NULL, NULL, 0, NULL} // sentinel +static PyObject * +test_unicode(PyObject *Py_UNUSED(module), PyObject *Py_UNUSED(args)) +{ + PyObject *str; +#ifndef Py_LIMITED_API + const void* data; + int kind; + const void* const_data; + unsigned int ukind; +#endif + + str = PyUnicode_FromString("abc"); + if (str == _Py_NULL) { + return _Py_NULL; + } + + assert(PyUnicode_Check(str)); + + assert(PyUnicode_GetLength(str) == 3); + assert(PyUnicode_ReadChar(str, 0) == 'a'); + assert(PyUnicode_ReadChar(str, 1) == 'b'); + +#ifndef Py_LIMITED_API + assert(PyUnicode_GET_LENGTH(str) == 3); + + // gh-92800: test PyUnicode_READ() + data = PyUnicode_DATA(str); + assert(data != _Py_NULL); + kind = PyUnicode_KIND(str); + assert(kind == PyUnicode_1BYTE_KIND); + assert(PyUnicode_READ(kind, data, 0) == 'a'); + + // gh-92800: test PyUnicode_READ() casts + const_data = PyUnicode_DATA(str); +#ifdef __cplusplus + ukind = static_cast(kind); +#else + ukind = (unsigned int)kind; +#endif + assert(PyUnicode_READ(ukind, const_data, 2) == 'c'); + + assert(PyUnicode_READ_CHAR(str, 1) == 'b'); +#endif + + Py_DECREF(str); + Py_RETURN_NONE; +} + + +#ifdef __cplusplus +// Class to test operator casting an object to PyObject* +class StrongRef +{ +public: + StrongRef(PyObject *obj) : m_obj(obj) { + Py_INCREF(this->m_obj); + } + + ~StrongRef() { + Py_DECREF(this->m_obj); + } + + // Cast to PyObject*: get a borrowed reference + inline operator PyObject*() const { return this->m_obj; } + +private: + PyObject *m_obj; // Strong reference +}; + + +static PyObject * +test_api_casts(PyObject *Py_UNUSED(module), PyObject *Py_UNUSED(args)) +{ + PyObject *obj = Py_BuildValue("(ii)", 1, 2); + if (obj == _Py_NULL) { + return _Py_NULL; + } + Py_ssize_t refcnt = Py_REFCNT(obj); + assert(refcnt >= 1); + +#ifndef Py_LIMITED_API + // gh-92138: For backward compatibility, functions of Python C API accepts + // "const PyObject*". Check that using it does not emit C++ compiler + // warnings. + const PyObject *const_obj = obj; + Py_INCREF(const_obj); + Py_DECREF(const_obj); + PyTypeObject *type = Py_TYPE(const_obj); + assert(Py_REFCNT(const_obj) == refcnt); + assert(type == &PyTuple_Type); + assert(PyTuple_GET_SIZE(const_obj) == 2); + PyObject *one = PyTuple_GET_ITEM(const_obj, 0); + assert(PyLong_AsLong(one) == 1); +#endif + + // gh-92898: StrongRef doesn't inherit from PyObject but has an operator to + // cast to PyObject*. + StrongRef strong_ref(obj); + assert(Py_TYPE(strong_ref) == &PyTuple_Type); + assert(Py_REFCNT(strong_ref) == (refcnt + 1)); + Py_INCREF(strong_ref); + Py_DECREF(strong_ref); + + // gh-93442: Pass 0 as NULL for PyObject* + Py_XINCREF(0); + Py_XDECREF(0); +#if __cplusplus >= 201103 + // Test nullptr passed as PyObject* + Py_XINCREF(nullptr); + Py_XDECREF(nullptr); +#endif + + Py_DECREF(obj); + Py_RETURN_NONE; +} +#endif // __cplusplus + + +// VirtualPyObject is incompatible with opaque PyObject +#if defined(__cplusplus) && !defined(Py_TARGET_ABI3T) +/* Test a `new`-allocated object with a virtual method. + * (https://github.com/python/cpython/issues/94731) */ + +class VirtualPyObject : public PyObject { +public: + VirtualPyObject(); + virtual ~VirtualPyObject() { + delete [] internal_data; + --instance_count; + } + virtual void set_internal_data() { + internal_data[0] = 1; + } + static void dealloc(PyObject* o) { + delete static_cast(o); + } + + // Number of "living" instances + static int instance_count; +private: + // buffer that can get corrupted + int* internal_data; +}; + +int VirtualPyObject::instance_count = 0; + +// Converting from function pointer to void* has undefined behavior, but +// works on all known platforms, and CPython's module and type slots currently +// need it. +// (GCC doesn't have a narrower category for this than -Wpedantic.) +_Py_COMP_DIAG_PUSH +#if defined(__GNUC__) +#pragma GCC diagnostic ignored "-Wpedantic" +#elif defined(__clang__) +#pragma clang diagnostic ignored "-Wpedantic" +#endif + +PyType_Slot VirtualPyObject_Slots[] = { + {Py_tp_free, (void*)VirtualPyObject::dealloc}, + {0, _Py_NULL}, +}; + +_Py_COMP_DIAG_POP + +PyType_Spec VirtualPyObject_Spec = { + /* .name */ STR(MODULE_NAME) ".VirtualPyObject", + /* .basicsize */ sizeof(VirtualPyObject), + /* .itemsize */ 0, + /* .flags */ Py_TPFLAGS_DEFAULT, + /* .slots */ VirtualPyObject_Slots, +}; + +VirtualPyObject::VirtualPyObject() { + // Create a temporary type (just so we don't need to store it) + PyObject *type = PyType_FromSpec(&VirtualPyObject_Spec); + // no good way to signal failure from a C++ constructor, so use assert + // for error handling + assert(type); + assert(PyObject_Init(this, (PyTypeObject *)type)); + Py_DECREF(type); + internal_data = new int[50]; + ++instance_count; +} + +static PyObject * +test_virtual_object(PyObject *Py_UNUSED(module), PyObject *Py_UNUSED(args)) +{ + VirtualPyObject* obj = new VirtualPyObject(); + obj->set_internal_data(); + Py_DECREF(obj); + if (VirtualPyObject::instance_count != 0) { + return PyErr_Format( + PyExc_AssertionError, + "instance_count should be 0, got %d", + VirtualPyObject::instance_count); + } + Py_RETURN_NONE; +} +#endif // __cplusplus && !Py_TARGET_ABI3T + + +static PyMethodDef module_methods[] = { + {"add", test_add, METH_VARARGS, test_add_doc}, + {"test_macros", test_macros, METH_NOARGS, _Py_NULL}, + {"test_datetime", test_datetime, METH_NOARGS, _Py_NULL}, + {"test_unicode", test_unicode, METH_NOARGS, _Py_NULL}, +#ifdef __cplusplus + {"test_api_casts", test_api_casts, METH_NOARGS, _Py_NULL}, +#endif +#if defined(__cplusplus) && !defined(Py_TARGET_ABI3T) + {"test_virtual_object", test_virtual_object, METH_NOARGS, _Py_NULL}, +#endif + {_Py_NULL, _Py_NULL, 0, _Py_NULL}, // sentinel }; static int -_testcext_exec(PyObject *module) +module_exec(PyObject *module) { - PyObject *result, *obj; + PyObject *result; #ifdef __STDC_VERSION__ if (PyModule_AddIntMacro(module, __STDC_VERSION__) < 0) { return -1; } #endif +#ifdef __cplusplus + if (PyModule_AddIntMacro(module, __cplusplus) < 0) { + return -1; + } +#endif - result = PyObject_CallMethod(module, "test_datetime", ""); + result = PyObject_CallMethod(module, "test_macros", ""); if (!result) return -1; Py_DECREF(result); - // Test Py_BUILD_ASSERT() and Py_BUILD_ASSERT_EXPR() - Py_BUILD_ASSERT(sizeof(int) == sizeof(unsigned int)); - assert(Py_BUILD_ASSERT_EXPR(sizeof(int) == sizeof(unsigned int)) == 0); - - // Test Py_MIN(), Py_MAX(), Py_ABS() - assert(Py_MIN(5, 11) == 5); - assert(Py_MAX(5, 11) == 11); - assert(Py_ABS(-5) == 5); + result = PyObject_CallMethod(module, "test_datetime", ""); + if (!result) return -1; + Py_DECREF(result); - // Test Py_CLEAR(): use typeof()/__typeof__() if available, or memcpy() - obj = Py_None; - Py_CLEAR(obj); - assert(obj == NULL); + result = PyObject_CallMethod(module, "test_unicode", ""); + if (!result) return -1; + Py_DECREF(result); -#ifndef Py_LIMITED_API - // Test Py_SETREF(): use typeof()/__typeof__() if available, or memcpy() - obj = Py_None; - Py_SETREF(obj, NULL); - assert(obj == NULL); +#ifdef __cplusplus + result = PyObject_CallMethod(module, "test_api_casts", ""); + if (!result) return -1; + Py_DECREF(result); #endif - // Test that Py_BEGIN_CRITICAL_SECTION is available - Py_BEGIN_CRITICAL_SECTION(module); - Py_END_CRITICAL_SECTION(); +#if defined(__cplusplus) && !defined(Py_TARGET_ABI3T) + result = PyObject_CallMethod(module, "test_virtual_object", ""); + if (!result) return -1; + Py_DECREF(result); +#endif return 0; } @@ -122,38 +399,68 @@ _testcext_exec(PyObject *module) #define _FUNC_NAME(NAME) PyModExport_ ## NAME #define FUNC_NAME(NAME) _FUNC_NAME(NAME) +#ifdef __cplusplus +PyDoc_STRVAR(module_doc, "C++ test extension."); +#else +PyDoc_STRVAR(module_doc, "C test extension."); +#endif +PyABIInfo_VAR(abi_info); + +#ifdef __cplusplus + +// Need to ignore "-Wpedantic" warnings; see VirtualPyObject_Slots above +_Py_COMP_DIAG_PUSH +#if defined(__GNUC__) +# pragma GCC diagnostic ignored "-Wpedantic" +#elif defined(__clang__) +# pragma clang diagnostic ignored "-Wpedantic" +#endif + +static PySlot module_slots[] = { + PySlot_PTR_STATIC(Py_mod_abi, &abi_info), + PySlot_PTR_STATIC(Py_mod_name, (void*)STR(MODULE_NAME)), + PySlot_PTR_STATIC(Py_mod_doc, (void*)(char*)module_doc), + PySlot_PTR_STATIC(Py_mod_exec, (void*)module_exec), + PySlot_PTR_STATIC(Py_mod_methods, module_methods), + PySlot_PTR_STATIC(Py_mod_gil, Py_MOD_GIL_NOT_USED), + PySlot_END, +}; + +_Py_COMP_DIAG_POP + +#else + // Converting from function pointer to void* has undefined behavior, but // works on all known platforms, and CPython's module and type slots currently // need it. // (GCC doesn't have a narrower category for this than -Wpedantic.) _Py_COMP_DIAG_PUSH #if defined(__GNUC__) -#pragma GCC diagnostic ignored "-Wpedantic" -#pragma GCC diagnostic ignored "-Wcast-qual" +# pragma GCC diagnostic ignored "-Wpedantic" +# pragma GCC diagnostic ignored "-Wcast-qual" #elif defined(__clang__) -#pragma clang diagnostic ignored "-Wpedantic" -#pragma clang diagnostic ignored "-Wcast-qual" +# pragma clang diagnostic ignored "-Wpedantic" +# pragma clang diagnostic ignored "-Wcast-qual" #endif -PyDoc_STRVAR(_testcext_doc, "C test extension."); -PyABIInfo_VAR(abi_info); - -static PySlot _testcext_slots[] = { +static PySlot module_slots[] = { PySlot_STATIC_DATA(Py_mod_abi, &abi_info), PySlot_STATIC_DATA(Py_mod_name, STR(MODULE_NAME)), - PySlot_STATIC_DATA(Py_mod_doc, (void*)(char*)_testcext_doc), - PySlot_FUNC(Py_mod_exec, (void*)_testcext_exec), - PySlot_STATIC_DATA(Py_mod_methods, _testcext_methods), + PySlot_STATIC_DATA(Py_mod_doc, (void*)(char*)module_doc), + PySlot_FUNC(Py_mod_exec, (void*)module_exec), + PySlot_STATIC_DATA(Py_mod_methods, module_methods), PySlot_DATA(Py_mod_gil, Py_MOD_GIL_NOT_USED), PySlot_END, }; _Py_COMP_DIAG_POP +#endif // !__cplusplus + PyMODEXPORT_FUNC FUNC_NAME(MODULE_NAME)(void) { - return _testcext_slots; + return module_slots; } // Also define the soft-deprecated entrypoint to ensure it isn't called diff --git a/Lib/test/test_cext/extension.cpp b/Lib/test/test_cext/extension.cpp new file mode 100644 index 000000000000000..8889e1ca7b34893 --- /dev/null +++ b/Lib/test/test_cext/extension.cpp @@ -0,0 +1,3 @@ +// extension.cpp just copy/paste extension.c. This file is used to pass +// a filename with ".cpp" extension to select a C++ compiler. +#include "extension.c" diff --git a/Lib/test/test_cext/setup.py b/Lib/test/test_cext/setup.py index 1eca44bdf823dc2..58bee3e255a4b38 100644 --- a/Lib/test/test_cext/setup.py +++ b/Lib/test/test_cext/setup.py @@ -1,5 +1,5 @@ -# gh-91321: Build a basic C test extension to check that the Python C API is -# compatible with C and does not emit C compiler warnings. +# gh-91321: Build a basic C or C++ test extension module to check that the +# Python C API does not emit compiler warnings. import os import shlex import sys @@ -8,10 +8,17 @@ from setuptools import setup, Extension +SOURCE = { + 'C': 'extension.c', + 'C++': 'extension.cpp', +} -SOURCE = 'extension.c' +MSVC = support.MS_WINDOWS -if not support.MS_WINDOWS: + +### C flags ################################################################# + +if not MSVC: # C compiler flags for GCC and clang BASE_CFLAGS = [ # The purpose of test_cext extension is to check that building a C @@ -59,32 +66,74 @@ ] +### C++ flags ############################################################### + +if not MSVC: + # C++ compiler flags for GCC and clang + CPPFLAGS = [ + # gh-91321: The purpose of _testcppext extension is to check that building + # a C++ extension using the Python C API does not emit C++ compiler + # warnings + '-Werror', + ] + + CPPFLAGS_PEDANTIC = [ + # Ask for strict(er) compliance with the standard. + # We cannot do this for c++03 unlimited API, since several headers in + # Include/cpython/ use commas at end of `enum` declarations, a C++11 + # feature for which GCC has no narrower option than -Wpedantic itself. + '-pedantic-errors', + + # We also use `long long`, a C++11 feature we can enable individually. + '-Wno-long-long', + ] +else: + # MSVC compiler flags + CPPFLAGS = [ + # Display warnings level 1 to 4 + '/W4', + # Treat all compiler warnings as compiler errors + '/WX', + ] + CPPFLAGS_PEDANTIC = [] + + def main(): - std = os.environ.get("CPYTHON_TEST_STD", "") module_name = os.environ["CPYTHON_TEST_EXT_NAME"] + language = os.environ.get("CPYTHON_TEST_LANG", "C") + std = os.environ.get("CPYTHON_TEST_STD", "") limited = bool(os.environ.get("CPYTHON_TEST_LIMITED", "")) abi3t = bool(os.environ.get("CPYTHON_TEST_ABI3T", "")) - internal = bool(int(os.environ.get("TEST_INTERNAL_C_API", "0"))) - incdirs = os.environ.get("CPYTHON_EXTRA_INCDIRS", "") - libdirs = os.environ.get("CPYTHON_EXTRA_LIBDIRS", "") + internal = bool(int(os.environ.get("CPYTHON_TEST_INTERNAL_C_API", "0"))) + incdirs = os.environ.get("CPYTHON_TEST_EXTRA_INCDIRS", "") + libdirs = os.environ.get("CPYTHON_TEST_EXTRA_LIBDIRS", "") + extra_cflags = os.environ.get("CPYTHON_TEST_EXTRA_CFLAGS", "") - sources = [SOURCE] + source = SOURCE[language] - if not internal: - cflags = list(PUBLIC_CFLAGS) + if language == 'C++': + flags = list(CPPFLAGS) else: - cflags = list(INTERNAL_CFLAGS) - cflags.append(f'-DMODULE_NAME={module_name}') + if not internal: + flags = list(PUBLIC_CFLAGS) + else: + flags = list(INTERNAL_CFLAGS) + flags.append(f'-DMODULE_NAME={module_name}') # Add -std=STD or /std:STD (MSVC) compiler flag if std: - if support.MS_WINDOWS: - cflags.append(f'/std:{std}') + if MSVC: + flags.append(f'/std:{std}') else: - cflags.append(f'-std={std}') + flags.append(f'-std={std}') - # Remove existing -std or /std options from CC command line. - # Python adds -std=c11 option. + if language == 'C++' and (limited or (std != 'c++03') and not internal): + # See CPPFLAGS_PEDANTIC docstring + flags.extend(CPPFLAGS_PEDANTIC) + + # gh-105776: When "gcc -std=11" is used as the C++ compiler, -std=c11 + # option emits a C++ compiler warning. Remove "-std11" option from the + # CC command. cmd = (sysconfig.get_config_var('CC') or '') if cmd is not None: if support.MS_WINDOWS: @@ -99,13 +148,13 @@ def main(): # Define opt-in macros if limited: - cflags.append(f'-DPy_LIMITED_API={sys.hexversion:#x}') - + flags.append(f'-DPy_LIMITED_API={sys.hexversion:#x}') if abi3t: - cflags.append(f'-DPy_TARGET_ABI3T={sys.hexversion:#x}') - + flags.append(f'-DPy_TARGET_ABI3T={sys.hexversion:#x}') if internal: - cflags.append('-DTEST_INTERNAL_C_API=1') + flags.append('-DTEST_INTERNAL_C_API=1') + if extra_cflags: + flags.extend(shlex.split(extra_cflags)) # Add additional include and library directories, typically for in-tree # testing where not all directories are inferred @@ -119,17 +168,19 @@ def main(): library_dirs.extend(libdirs.split(os.pathsep)) # Display information to help debugging - for env_name in ('CC', 'CFLAGS', 'CPPFLAGS'): + print(f"Language: {language}") + print(f"Source: {source}") + for env_name in ('CC', 'CXX', 'CFLAGS', 'CPPFLAGS', 'CXXFLAGS'): if env_name in os.environ: print(f"{env_name} env var: {os.environ[env_name]!r}") else: print(f"{env_name} env var: ") - print(f"extra_compile_args: {cflags!r}") + print(f"extra_compile_args: {flags!r}") ext = Extension( module_name, - sources=sources, - extra_compile_args=cflags, + sources=[source], + extra_compile_args=flags, include_dirs=include_dirs, library_dirs=library_dirs) setup(name=f'internal_{module_name}', diff --git a/Lib/test/test_cppext/__init__.py b/Lib/test/test_cppext/__init__.py deleted file mode 100644 index db7f41d9ef7a11f..000000000000000 --- a/Lib/test/test_cppext/__init__.py +++ /dev/null @@ -1,148 +0,0 @@ -# gh-91321: Build a basic C++ test extension to check that the Python C API is -# compatible with C++ and does not emit C++ compiler warnings. -import os.path -import platform -import shlex -import shutil -import subprocess -import sys -import sysconfig -import unittest -from test import support - - -SOURCE = os.path.join(os.path.dirname(__file__), 'extension.cpp') -SETUP = os.path.join(os.path.dirname(__file__), 'setup.py') - - -# With MSVC on a debug build, the linker fails with: cannot open file -# 'python311.lib', it should look 'python311_d.lib'. -@unittest.skipIf(support.MS_WINDOWS and support.Py_DEBUG, - 'test fails on Windows debug build') -# Building and running an extension in clang sanitizing mode is not -# straightforward -@support.skip_if_sanitizer('test does not work with analyzing builds', - address=True, memory=True, ub=True, thread=True) -# the test uses venv+pip: skip if it's not available -@support.requires_venv_with_pip() -@support.requires_subprocess() -@support.requires_resource('cpu') -class BaseTests: - TEST_INTERNAL_C_API = False - - def check_build(self, extension_name, std=None, limited=False, - extra_cflags=None): - venv_dir = 'env' - with support.setup_venv_with_pip_setuptools(venv_dir) as python_exe: - self._check_build(extension_name, python_exe, - std=std, limited=limited, - extra_cflags=extra_cflags) - - def _check_build(self, extension_name, python_exe, std, limited, - extra_cflags=None): - pkg_dir = 'pkg' - os.mkdir(pkg_dir) - shutil.copy(SETUP, os.path.join(pkg_dir, os.path.basename(SETUP))) - shutil.copy(SOURCE, os.path.join(pkg_dir, os.path.basename(SOURCE))) - - def run_cmd(operation, cmd): - env = os.environ.copy() - if std: - env['CPYTHON_TEST_CPP_STD'] = std - if limited: - env['CPYTHON_TEST_LIMITED'] = '1' - if support.MS_WINDOWS and sysconfig.is_python_build(): - env['CPYTHON_EXTRA_INCDIRS'] = os.path.split(sysconfig.get_config_h_filename())[0] - env['CPYTHON_EXTRA_LIBDIRS'] = os.path.split(sys.executable)[0] - env['CPYTHON_TEST_EXT_NAME'] = extension_name - env['TEST_INTERNAL_C_API'] = str(int(self.TEST_INTERNAL_C_API)) - if extra_cflags: - env['CPYTHON_TEST_EXTRA_CFLAGS'] = extra_cflags - if support.verbose: - print('Run:', ' '.join(map(shlex.quote, cmd))) - subprocess.run(cmd, check=True, env=env) - else: - proc = subprocess.run(cmd, - env=env, - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT, - text=True) - if proc.returncode: - print('Run:', ' '.join(map(shlex.quote, cmd))) - print(proc.stdout, end='') - self.fail( - f"{operation} failed with exit code {proc.returncode}") - - # Build and install the C++ extension - cmd = [python_exe, '-X', 'dev', - '-m', 'pip', 'install', '--no-build-isolation', - os.path.abspath(pkg_dir)] - if support.verbose: - cmd.append('-v') - run_cmd('Install', cmd) - - # Do a reference run. Until we test that running python - # doesn't leak references (gh-94755), run it so one can manually check - # -X showrefcount results against this baseline. - cmd = [python_exe, - '-X', 'dev', - '-X', 'showrefcount', - '-c', 'pass'] - run_cmd('Reference run', cmd) - - # Import the C++ extension - cmd = [python_exe, - '-X', 'dev', - '-X', 'showrefcount', - '-c', f"import {extension_name}"] - run_cmd('Import', cmd) - - -class TestPublicCAPI(BaseTests, unittest.TestCase): - def test_build(self): - self.check_build('_testcppext') - - def test_build_limited_cpp03(self): - self.check_build('_test_limited_cpp03ext', std='c++03', limited=True) - - def test_build_limited(self): - self.check_build('_testcppext_limited', limited=True) - - def test_build_cpp03(self): - # In public docs, we say C API is compatible with C++11. However, - # in practice we do maintain C++03 compatibility in public headers. - # Please ask the C API WG before adding a new C++11-only feature. - self.check_build('_testcpp03ext', std='c++03') - - @unittest.skipIf(support.MS_WINDOWS, "MSVC doesn't support /std:c++11") - def test_build_cpp11(self): - self.check_build('_testcpp11ext', std='c++11') - - # Only test C++14 on MSVC. - # On s390x RHEL7, GCC 4.8.5 doesn't support C++14. - @unittest.skipIf(not support.MS_WINDOWS, "need Windows") - def test_build_cpp14(self): - self.check_build('_testcpp14ext', std='c++14') - - # Test that headers compile with Intel asm syntax, which may conflict - # with inline assembly in free-threading headers that use AT&T syntax. - @unittest.skipIf(support.MS_WINDOWS, "MSVC doesn't support -masm=intel") - @unittest.skipUnless(platform.machine() in ('x86_64', 'i686', 'AMD64'), - "x86-specific flag") - def test_build_intel_asm(self): - self.check_build('_testcppext_asm', extra_cflags='-masm=intel') - - -class TestInteralCAPI(BaseTests, unittest.TestCase): - TEST_INTERNAL_C_API = True - - def test_build(self): - kwargs = {} - if sys.platform == 'darwin': - # Old Apple clang++ default C++ std is gnu++98 - kwargs['std'] = 'c++11' - self.check_build('_testcppext_internal', **kwargs) - - -if __name__ == "__main__": - unittest.main() diff --git a/Lib/test/test_cppext/extension.cpp b/Lib/test/test_cppext/extension.cpp deleted file mode 100644 index 1ff56d0e7fd25a0..000000000000000 --- a/Lib/test/test_cppext/extension.cpp +++ /dev/null @@ -1,375 +0,0 @@ -// gh-91321: Basic C++ test extension to check that the Python C API is -// compatible with C++ and does not emit C++ compiler warnings. -// -// The code is only built, not executed. - -// Always enable assertions -#undef NDEBUG - -#ifdef TEST_INTERNAL_C_API -# define Py_BUILD_CORE_MODULE 1 -#endif - -#include "Python.h" -#include "datetime.h" - -#ifdef TEST_INTERNAL_C_API - // gh-135906: Check for compiler warnings in the internal C API - // - Cython uses pycore_critical_section.h, pycore_frame.h and - // pycore_template.h. - // - greenlet uses pycore_frame.h, pycore_interpframe_structs.h and - // pycore_interpframe.h. -# include "internal/pycore_frame.h" -# include "internal/pycore_interpframe_structs.h" -# include "internal/pycore_template.h" - - // mimalloc emits compiler warnings on Windows. -# if !defined(MS_WINDOWS) -# include "internal/pycore_backoff.h" -# include "internal/pycore_cell.h" -# include "internal/pycore_critical_section.h" -# include "internal/pycore_interpframe.h" -# endif -#endif - -#ifndef MODULE_NAME -# error "MODULE_NAME macro must be defined" -#endif - -#define _STR(NAME) #NAME -#define STR(NAME) _STR(NAME) - -PyDoc_STRVAR(_testcppext_add_doc, -"add(x, y)\n" -"\n" -"Return the sum of two integers: x + y."); - -static PyObject * -_testcppext_add(PyObject *Py_UNUSED(module), PyObject *args) -{ - long i, j; - if (!PyArg_ParseTuple(args, "ll:foo", &i, &j)) { - return _Py_NULL; - } - long res = i + j; - return PyLong_FromLong(res); -} - - -// Class to test operator casting an object to PyObject* -class StrongRef -{ -public: - StrongRef(PyObject *obj) : m_obj(obj) { - Py_INCREF(this->m_obj); - } - - ~StrongRef() { - Py_DECREF(this->m_obj); - } - - // Cast to PyObject*: get a borrowed reference - inline operator PyObject*() const { return this->m_obj; } - -private: - PyObject *m_obj; // Strong reference -}; - - -static PyObject * -test_api_casts(PyObject *Py_UNUSED(module), PyObject *Py_UNUSED(args)) -{ - PyObject *obj = Py_BuildValue("(ii)", 1, 2); - if (obj == _Py_NULL) { - return _Py_NULL; - } - Py_ssize_t refcnt = Py_REFCNT(obj); - assert(refcnt >= 1); - -#ifndef Py_LIMITED_API - // gh-92138: For backward compatibility, functions of Python C API accepts - // "const PyObject*". Check that using it does not emit C++ compiler - // warnings. - const PyObject *const_obj = obj; - Py_INCREF(const_obj); - Py_DECREF(const_obj); - PyTypeObject *type = Py_TYPE(const_obj); - assert(Py_REFCNT(const_obj) == refcnt); - assert(type == &PyTuple_Type); - assert(PyTuple_GET_SIZE(const_obj) == 2); - PyObject *one = PyTuple_GET_ITEM(const_obj, 0); - assert(PyLong_AsLong(one) == 1); -#endif - - // gh-92898: StrongRef doesn't inherit from PyObject but has an operator to - // cast to PyObject*. - StrongRef strong_ref(obj); - assert(Py_TYPE(strong_ref) == &PyTuple_Type); - assert(Py_REFCNT(strong_ref) == (refcnt + 1)); - Py_INCREF(strong_ref); - Py_DECREF(strong_ref); - - // gh-93442: Pass 0 as NULL for PyObject* - Py_XINCREF(0); - Py_XDECREF(0); -#if __cplusplus >= 201103 - // Test nullptr passed as PyObject* - Py_XINCREF(nullptr); - Py_XDECREF(nullptr); -#endif - - Py_DECREF(obj); - Py_RETURN_NONE; -} - - -static PyObject * -test_unicode(PyObject *Py_UNUSED(module), PyObject *Py_UNUSED(args)) -{ - PyObject *str = PyUnicode_FromString("abc"); - if (str == _Py_NULL) { - return _Py_NULL; - } - - assert(PyUnicode_Check(str)); - - assert(PyUnicode_GetLength(str) == 3); - assert(PyUnicode_ReadChar(str, 0) == 'a'); - assert(PyUnicode_ReadChar(str, 1) == 'b'); - -#ifndef Py_LIMITED_API - assert(PyUnicode_GET_LENGTH(str) == 3); - - // gh-92800: test PyUnicode_READ() - const void* data = PyUnicode_DATA(str); - assert(data != _Py_NULL); - int kind = PyUnicode_KIND(str); - assert(kind == PyUnicode_1BYTE_KIND); - assert(PyUnicode_READ(kind, data, 0) == 'a'); - - // gh-92800: test PyUnicode_READ() casts - const void* const_data = PyUnicode_DATA(str); - unsigned int ukind = static_cast(kind); - assert(PyUnicode_READ(ukind, const_data, 2) == 'c'); - - assert(PyUnicode_READ_CHAR(str, 1) == 'b'); -#endif - - Py_DECREF(str); - Py_RETURN_NONE; -} - -// VirtualPyObject is incompatible with opaque PyObject -#ifndef Py_TARGET_ABI3T -/* Test a `new`-allocated object with a virtual method. - * (https://github.com/python/cpython/issues/94731) */ - -class VirtualPyObject : public PyObject { -public: - VirtualPyObject(); - virtual ~VirtualPyObject() { - delete [] internal_data; - --instance_count; - } - virtual void set_internal_data() { - internal_data[0] = 1; - } - static void dealloc(PyObject* o) { - delete static_cast(o); - } - - // Number of "living" instances - static int instance_count; -private: - // buffer that can get corrupted - int* internal_data; -}; - -int VirtualPyObject::instance_count = 0; - -// Converting from function pointer to void* has undefined behavior, but -// works on all known platforms, and CPython's module and type slots currently -// need it. -// (GCC doesn't have a narrower category for this than -Wpedantic.) -_Py_COMP_DIAG_PUSH -#if defined(__GNUC__) -#pragma GCC diagnostic ignored "-Wpedantic" -#elif defined(__clang__) -#pragma clang diagnostic ignored "-Wpedantic" -#endif - -PyType_Slot VirtualPyObject_Slots[] = { - {Py_tp_free, (void*)VirtualPyObject::dealloc}, - {0, _Py_NULL}, -}; - -_Py_COMP_DIAG_POP - -PyType_Spec VirtualPyObject_Spec = { - /* .name */ STR(MODULE_NAME) ".VirtualPyObject", - /* .basicsize */ sizeof(VirtualPyObject), - /* .itemsize */ 0, - /* .flags */ Py_TPFLAGS_DEFAULT, - /* .slots */ VirtualPyObject_Slots, -}; - -VirtualPyObject::VirtualPyObject() { - // Create a temporary type (just so we don't need to store it) - PyObject *type = PyType_FromSpec(&VirtualPyObject_Spec); - // no good way to signal failure from a C++ constructor, so use assert - // for error handling - assert(type); - assert(PyObject_Init(this, (PyTypeObject *)type)); - Py_DECREF(type); - internal_data = new int[50]; - ++instance_count; -} - -static PyObject * -test_virtual_object(PyObject *Py_UNUSED(module), PyObject *Py_UNUSED(args)) -{ - VirtualPyObject* obj = new VirtualPyObject(); - obj->set_internal_data(); - Py_DECREF(obj); - if (VirtualPyObject::instance_count != 0) { - return PyErr_Format( - PyExc_AssertionError, - "instance_count should be 0, got %d", - VirtualPyObject::instance_count); - } - Py_RETURN_NONE; -} -#endif // Py_TARGET_ABI3T - - -static PyObject * -test_datetime(PyObject *Py_UNUSED(module), PyObject *Py_UNUSED(args)) -{ - // datetime.h is excluded from the limited C API -#ifndef Py_LIMITED_API - PyDateTime_IMPORT; - if (PyErr_Occurred()) { - return NULL; - } -#endif - - Py_RETURN_NONE; -} - -static PyMethodDef _testcppext_methods[] = { - {"add", _testcppext_add, METH_VARARGS, _testcppext_add_doc}, - {"test_api_casts", test_api_casts, METH_NOARGS, _Py_NULL}, - {"test_unicode", test_unicode, METH_NOARGS, _Py_NULL}, -#ifndef Py_TARGET_ABI3T - {"test_virtual_object", test_virtual_object, METH_NOARGS, _Py_NULL}, -#endif - {"test_datetime", test_datetime, METH_NOARGS, _Py_NULL}, - // Note: _testcppext_exec currently runs all test functions directly. - // When adding a new one, add a call there. - - {_Py_NULL, _Py_NULL, 0, _Py_NULL} /* sentinel */ -}; - - -static int -_testcppext_exec(PyObject *module) -{ - if (PyModule_AddIntMacro(module, __cplusplus) < 0) { - return -1; - } - - PyObject *result; - - result = PyObject_CallMethod(module, "test_api_casts", ""); - if (!result) return -1; - Py_DECREF(result); - - result = PyObject_CallMethod(module, "test_unicode", ""); - if (!result) return -1; - Py_DECREF(result); - -#ifndef Py_TARGET_ABI3T - result = PyObject_CallMethod(module, "test_virtual_object", ""); - if (!result) return -1; - Py_DECREF(result); -#endif - - result = PyObject_CallMethod(module, "test_datetime", ""); - if (!result) return -1; - Py_DECREF(result); - - // test Py_BUILD_ASSERT() and Py_BUILD_ASSERT_EXPR() - Py_BUILD_ASSERT(sizeof(int) == sizeof(unsigned int)); - assert(Py_BUILD_ASSERT_EXPR(sizeof(int) == sizeof(unsigned int)) == 0); - - // Test Py_MIN(), Py_MAX(), Py_ABS() - assert(Py_MIN(5, 11) == 5); - assert(Py_MAX(5, 11) == 11); - assert(Py_ABS(-5) == 5); - - // Test Py_CLEAR(): use typeof()/__typeof__() if available, or memcpy() - PyObject *obj = Py_None; - Py_CLEAR(obj); - assert(obj == _Py_NULL); - -#ifndef Py_LIMITED_API - // Test Py_SETREF(): use typeof()/__typeof__() if available, or memcpy() - obj = Py_None; - Py_SETREF(obj, _Py_NULL); - assert(obj == _Py_NULL); -#endif - - // Test that Py_BEGIN_CRITICAL_SECTION is available - Py_BEGIN_CRITICAL_SECTION(module); - Py_END_CRITICAL_SECTION(); - - return 0; -} - - -PyDoc_STRVAR(_testcppext_doc, "C++ test extension."); -PyABIInfo_VAR(abi_info); - -// Need to ignore "-Wpedantic" warnings; see VirtualPyObject_Slots above -_Py_COMP_DIAG_PUSH -#if defined(__GNUC__) -#pragma GCC diagnostic ignored "-Wpedantic" -#elif defined(__clang__) -#pragma clang diagnostic ignored "-Wpedantic" -#endif - -static PySlot _testcppext_slots[] = { - PySlot_PTR_STATIC(Py_mod_abi, &abi_info), - PySlot_PTR_STATIC(Py_mod_name, (void*)STR(MODULE_NAME)), - PySlot_PTR_STATIC(Py_mod_doc, (void*)(char*)_testcppext_doc), - PySlot_PTR_STATIC(Py_mod_exec, (void*)_testcppext_exec), - PySlot_PTR_STATIC(Py_mod_methods, _testcppext_methods), - PySlot_PTR_STATIC(Py_mod_gil, Py_MOD_GIL_NOT_USED), - PySlot_END, -}; - -_Py_COMP_DIAG_POP - - -#define _FUNC_NAME(NAME) PyModExport_ ## NAME -#define FUNC_NAME(NAME) _FUNC_NAME(NAME) - -PyMODEXPORT_FUNC -FUNC_NAME(MODULE_NAME)(void) -{ - return _testcppext_slots; -} - -// Also define the soft-deprecated entrypoint to ensure it isn't called - -#define _INITFUNC_NAME(NAME) PyInit_ ## NAME -#define INITFUNC_NAME(NAME) _INITFUNC_NAME(NAME) - -PyMODINIT_FUNC -INITFUNC_NAME(MODULE_NAME)(void) -{ - PyErr_SetString( - PyExc_AssertionError, - "PyInit_* function called while a PyModExport_* one is available"); - return NULL; -} diff --git a/Lib/test/test_cppext/setup.py b/Lib/test/test_cppext/setup.py deleted file mode 100644 index 5d004ca6e3ad788..000000000000000 --- a/Lib/test/test_cppext/setup.py +++ /dev/null @@ -1,126 +0,0 @@ -# gh-91321: Build a basic C++ test extension to check that the Python C API is -# compatible with C++ and does not emit C++ compiler warnings. -import os -import shlex -import sys -import sysconfig -from test import support - -from setuptools import setup, Extension - - -SOURCE = 'extension.cpp' - -if not support.MS_WINDOWS: - # C++ compiler flags for GCC and clang - CPPFLAGS = [ - # gh-91321: The purpose of _testcppext extension is to check that building - # a C++ extension using the Python C API does not emit C++ compiler - # warnings - '-Werror', - ] - - CPPFLAGS_PEDANTIC = [ - # Ask for strict(er) compliance with the standard. - # We cannot do this for c++03 unlimited API, since several headers in - # Include/cpython/ use commas at end of `enum` declarations, a C++11 - # feature for which GCC has no narrower option than -Wpedantic itself. - '-pedantic-errors', - - # We also use `long long`, a C++11 feature we can enable individually. - '-Wno-long-long', - ] -else: - # MSVC compiler flags - CPPFLAGS = [ - # Display warnings level 1 to 4 - '/W4', - # Treat all compiler warnings as compiler errors - '/WX', - ] - CPPFLAGS_PEDANTIC = [] - - -def main(): - cppflags = list(CPPFLAGS) - std = os.environ.get("CPYTHON_TEST_CPP_STD", "") - module_name = os.environ["CPYTHON_TEST_EXT_NAME"] - limited = bool(os.environ.get("CPYTHON_TEST_LIMITED", "")) - internal = bool(int(os.environ.get("TEST_INTERNAL_C_API", "0"))) - incdirs = os.environ.get("CPYTHON_EXTRA_INCDIRS", "") - libdirs = os.environ.get("CPYTHON_EXTRA_LIBDIRS", "") - - cppflags = list(CPPFLAGS) - cppflags.append(f'-DMODULE_NAME={module_name}') - - # Add -std=STD or /std:STD (MSVC) compiler flag - if std: - if support.MS_WINDOWS: - cppflags.append(f'/std:{std}') - else: - cppflags.append(f'-std={std}') - - if limited or (std != 'c++03') and not internal: - # See CPPFLAGS_PEDANTIC docstring - cppflags.extend(CPPFLAGS_PEDANTIC) - - # gh-105776: When "gcc -std=11" is used as the C++ compiler, -std=c11 - # option emits a C++ compiler warning. Remove "-std11" option from the - # CC command. - cmd = (sysconfig.get_config_var('CC') or '') - if cmd is not None: - if support.MS_WINDOWS: - std_prefix = '/std' - else: - std_prefix = '-std' - cmd = shlex.split(cmd) - cmd = [arg for arg in cmd if not arg.startswith(std_prefix)] - cmd = shlex.join(cmd) - # CC env var overrides sysconfig CC variable in setuptools - os.environ['CC'] = cmd - - # Define Py_LIMITED_API macro - if limited: - version = sys.hexversion - cppflags.append(f'-DPy_LIMITED_API={version:#x}') - - if internal: - cppflags.append('-DTEST_INTERNAL_C_API=1') - - extra_cflags = os.environ.get("CPYTHON_TEST_EXTRA_CFLAGS", "") - if extra_cflags: - cppflags.extend(shlex.split(extra_cflags)) - - # Add additional include and library directories, typically for in-tree - # testing where not all directories are inferred - include_dirs = [] - library_dirs = [] - if incdirs: - print("Add incdirs:", incdirs) - include_dirs.extend(incdirs.split(os.pathsep)) - if libdirs: - print("Add libdirs:", libdirs) - library_dirs.extend(libdirs.split(os.pathsep)) - - # Display information to help debugging - for env_name in ('CC', 'CXX', 'CFLAGS', 'CPPFLAGS', 'CXXFLAGS'): - if env_name in os.environ: - print(f"{env_name} env var: {os.environ[env_name]!r}") - else: - print(f"{env_name} env var: ") - print(f"extra_compile_args: {cppflags!r}") - - ext = Extension( - module_name, - sources=[SOURCE], - language='c++', - extra_compile_args=cppflags, - include_dirs=include_dirs, - library_dirs=library_dirs) - setup(name=f'internal_{module_name}', - version='0.0', - ext_modules=[ext]) - - -if __name__ == "__main__": - main() diff --git a/Makefile.pre.in b/Makefile.pre.in index b29976ee041099c..42480f28ad7f315 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -2611,7 +2611,6 @@ TESTSUBDIRS= idlelib/idle_test \ test/test_capi \ test/test_cext \ test/test_concurrent_futures \ - test/test_cppext \ test/test_ctypes \ test/test_dataclasses \ test/test_doctest \ From a000129df30819c3b762592d7586de4d2536f92b Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sat, 26 Sep 2026 01:02:29 +0200 Subject: [PATCH 4/4] gh-158001: Fix test_ctypes.test_version_var() on s390x (#158193) Py_Version type is "unsigned long", not "int". The bug is visible on big endian machines such as s390x. --- Lib/test/test_ctypes/test_values.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_ctypes/test_values.py b/Lib/test/test_ctypes/test_values.py index 0d62702c0c0fd35..5ce2fdbe77c2260 100644 --- a/Lib/test/test_ctypes/test_values.py +++ b/Lib/test/test_ctypes/test_values.py @@ -8,7 +8,7 @@ import sys import unittest from ctypes import (Structure, CDLL, POINTER, pythonapi, - c_ubyte, c_char_p, c_int) + c_ubyte, c_char_p, c_int, c_ulong) from test import support from test.support import import_helper, thread_unsafe @@ -77,7 +77,7 @@ def test_version_var(self): # This test accesses the Py_Version integer, which is # exported by the Python dll and should match the sys.hexversion value - version = c_int.in_dll(pythonapi, "Py_Version").value + version = c_ulong.in_dll(pythonapi, "Py_Version").value self.assertEqual(version, sys.hexversion) @thread_unsafe('overrides frozen modules')