Context
Since 13.0.2 (markdown-it/markdown-it#847), markdown-it JS stops with an error when a rule reports a match without consuming input, instead of looping:
ParserBlock.tokenize throws block rule didn't increment state.line (parser_block.ts);
ParserInline.skipToken and ParserInline.tokenize throw inline rule didn't increment state.pos (parser_inline.ts).
markdown-it-py has no equivalent check in ParserBlock.tokenize or in ParserInline.skipToken / tokenize. A plugin rule that returns True without advancing makes parse() loop forever. It usually pushes a token on every pass, so memory grows until the process is killed:
from markdown_it import MarkdownIt
md = MarkdownIt("commonmark")
md.block.ruler.before("paragraph", "stuck", lambda state, start, end, silent: True)
md.parse("text\n") # never returns; same with an inline rule that returns True
This is not hypothetical. executablebooks/mdit-py-plugins#156 (texmath, input "> $$ a=1\n\n$$\n") and, earlier, executablebooks/mdit-py-plugins#117 (amsmath) were both this bug in a plugin. In each case the only symptom was a hang.
Proposal
Port the JS guards: after a rule returns True, raise if state.line (block) or state.pos (inline) has not advanced. A buggy plugin would then fail fast with a clear message that names the problem, instead of hanging and exhausting memory. Built-in rules always advance, so this only affects plugins that are already broken.
Reproduced on markdown-it-py 4.2.0 and master (6f58654), Python 3.12. I'm happy to open a PR if you'd like one.
Context
Since 13.0.2 (markdown-it/markdown-it#847), markdown-it JS stops with an error when a rule reports a match without consuming input, instead of looping:
ParserBlock.tokenizethrowsblock rule didn't increment state.line(parser_block.ts);ParserInline.skipTokenandParserInline.tokenizethrowinline rule didn't increment state.pos(parser_inline.ts).markdown-it-py has no equivalent check in
ParserBlock.tokenizeor inParserInline.skipToken/tokenize. A plugin rule that returnsTruewithout advancing makesparse()loop forever. It usually pushes a token on every pass, so memory grows until the process is killed:This is not hypothetical. executablebooks/mdit-py-plugins#156 (
texmath, input"> $$ a=1\n\n$$\n") and, earlier, executablebooks/mdit-py-plugins#117 (amsmath) were both this bug in a plugin. In each case the only symptom was a hang.Proposal
Port the JS guards: after a rule returns
True, raise ifstate.line(block) orstate.pos(inline) has not advanced. A buggy plugin would then fail fast with a clear message that names the problem, instead of hanging and exhausting memory. Built-in rules always advance, so this only affects plugins that are already broken.Reproduced on markdown-it-py 4.2.0 and
master(6f58654), Python 3.12. I'm happy to open a PR if you'd like one.