Skip to content

Conversation

@ThanhNguyxn
Copy link

Summary

  • Reset lexer state in shlex.push_source() so pushing a new source after EOF can continue tokenization.
  • Add regression test test_push_source_after_eof in Lib/test/test_shlex.py covering both POSIX and non-POSIX modes.
  • Add NEWS entry for gh-140950.

Repro

from shlex import shlex
lexer = shlex('a')
lexer.get_token()  # 'a'
lexer.get_token()  # EOF
lexer.push_source('b')
lexer.get_token()  # now returns 'b'

Verification

  • python -m py_compile Lib/shlex.py Lib/test/test_shlex.py
  • python -c "import importlib.util, sys, unittest; shlex_spec=importlib.util.spec_from_file_location('shlex', r'D:\Code\cpython\Lib\shlex.py'); shlex_mod=importlib.util.module_from_spec(shlex_spec); shlex_spec.loader.exec_module(shlex_mod); sys.modules['shlex']=shlex_mod; test_spec=importlib.util.spec_from_file_location('test_shlex_local', r'D:\Code\cpython\Lib\test\test_shlex.py'); test_mod=importlib.util.module_from_spec(test_spec); test_spec.loader.exec_module(test_mod); suite=unittest.TestSuite([test_mod.ShlexTest('test_push_source_after_eof')]); result=unittest.TextTestRunner(verbosity=2).run(suite); raise SystemExit(0 if result.wasSuccessful() else 1)"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant