gh-102555: Fix comment parsing in HTMLParser #135664

serhiy-storchaka · 2025-06-18T10:32:23Z

"--!>" now ends the comment.
"-- >" no longer ends the comment.
Support abnormally ended empty comments "<-->" and "<--->".

Issue: HTMLParser differences from the HTML5 specification #135661

Issue: XSS in html.parser library #102555

* "--!>" now ends the comment. * "-- >" no longer ends the comment. * Support abnormally ended empty comments "<-->" and "<--->".

Co-author: Kerim Kabirov <the.privat33r+gh@pm.me>

Privat33r-dev

Solid implementation overall :)

Privat33r-dev · 2025-06-25T13:49:06Z

Lib/html/parser.py

@@ -309,6 +310,21 @@ def parse_html_declaration(self, i):
        else:
            return self.parse_bogus_comment(i)

+    # Internal -- parse comment, return length or -1 if not terminated
+    # see https://html.spec.whatwg.org/multipage/parsing.html#comment-start-state
+    def parse_comment(self, i, report=True):


I wonder if the change should be made in the _markupbase.

cpython/Lib/_markupbase.py

Lines 165 to 175 in c2f2fd4

def parse_comment(self, i, report=1):

rawdata = self.rawdata

if rawdata[i:i+4] != '<!--':

raise AssertionError('unexpected call to parse_comment()')

match = _commentclose.search(rawdata, i+4)

if not match:

return -1

if report:

j = match.start(0)

self.handle_comment(rawdata[i+4: j])

return match.end(0)

If the method is overloaded here, then there are no other use cases, and the original method becomes dead code.
https://github.com/search?q=repo%3Apython%2Fcpython%20parse_comment&type=code

I think it's fine to override the method here. Since _markupbase has been made internal/private in Python 3 and it's only used by html.parser, it makes sense to me to add new code directly to html.parser (and possibly even merging _markupbase into html.parser eventually).

Regarding the (now) dead code, we could either let it be, adding a comment noting that the method is unused/overridden, or delete it. The first two options are less destructive, but since the module is private there shouldn't be much concern about breaking backward compatibility (and if anyone is relying on the original implementation, they are probably using it through HTMLParser anyway).

_markupbase can be used in third-party code (if it is also the base for the SGML parser or other parsers), so it is better to not touch it in maintained versions. This change can break it if SGML has other rules for comments. But in the developing branch we can remove it, after finishing all other bug fixes.

ezio-melotti · 2025-07-04T04:44:55Z

Lib/html/parser.py

@@ -309,6 +310,21 @@ def parse_html_declaration(self, i):
        else:
            return self.parse_bogus_comment(i)

+    # Internal -- parse comment, return length or -1 if not terminated
+    # see https://html.spec.whatwg.org/multipage/parsing.html#comment-start-state
+    def parse_comment(self, i, report=True):


I think it's fine to override the method here. Since _markupbase has been made internal/private in Python 3 and it's only used by html.parser, it makes sense to me to add new code directly to html.parser (and possibly even merging _markupbase into html.parser eventually).

Regarding the (now) dead code, we could either let it be, adding a comment noting that the method is unused/overridden, or delete it. The first two options are less destructive, but since the module is private there shouldn't be much concern about breaking backward compatibility (and if anyone is relying on the original implementation, they are probably using it through HTMLParser anyway).

Misc/NEWS.d/next/Security/2025-06-18-13-28-08.gh-issue-102555.nADrzJ.rst

…nADrzJ.rst Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>

miss-islington-app · 2025-07-04T07:00:26Z

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.9, 3.10, 3.11, 3.12, 3.13, 3.14.
🐍🍒⛏🤖

…TML5 standard (pythonGH-135664) * "--!>" now ends the comment. * "-- >" no longer ends the comment. * Support abnormally ended empty comments "<-->" and "<--->". --------- (cherry picked from commit 8ac7613) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-author: Kerim Kabirov <the.privat33r+gh@pm.me> Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>

bedevere-app · 2025-07-04T07:00:37Z

GH-136271 is a backport of this pull request to the 3.14 branch.

…TML5 standard (pythonGH-135664) * "--!>" now ends the comment. * "-- >" no longer ends the comment. * Support abnormally ended empty comments "<-->" and "<--->". --------- (cherry picked from commit 8ac7613) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-author: Kerim Kabirov <the.privat33r+gh@pm.me> Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>

bedevere-app · 2025-07-04T07:00:43Z

GH-136272 is a backport of this pull request to the 3.13 branch.

bedevere-app · 2025-07-04T07:00:48Z

GH-136273 is a backport of this pull request to the 3.12 branch.

…TML5 standard (pythonGH-135664) * "--!>" now ends the comment. * "-- >" no longer ends the comment. * Support abnormally ended empty comments "<-->" and "<--->". --------- (cherry picked from commit 8ac7613) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-author: Kerim Kabirov <the.privat33r+gh@pm.me> Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>

bedevere-app · 2025-07-04T07:00:55Z

GH-136274 is a backport of this pull request to the 3.11 branch.

…TML5 standard (pythonGH-135664) * "--!>" now ends the comment. * "-- >" no longer ends the comment. * Support abnormally ended empty comments "<-->" and "<--->". --------- (cherry picked from commit 8ac7613) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-author: Kerim Kabirov <the.privat33r+gh@pm.me> Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>

bedevere-app · 2025-07-04T07:01:02Z

GH-136275 is a backport of this pull request to the 3.10 branch.

…TML5 standard (pythonGH-135664) * "--!>" now ends the comment. * "-- >" no longer ends the comment. * Support abnormally ended empty comments "<-->" and "<--->". --------- (cherry picked from commit 8ac7613) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-author: Kerim Kabirov <the.privat33r+gh@pm.me> Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>

bedevere-app · 2025-07-04T07:01:08Z

GH-136276 is a backport of this pull request to the 3.9 branch.

…HTML5 standard (GH-135664) (GH-136272) * "--!>" now ends the comment. * "-- >" no longer ends the comment. * Support abnormally ended empty comments "<-->" and "<--->". --------- (cherry picked from commit 8ac7613) Co-author: Kerim Kabirov <the.privat33r+gh@pm.me> Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>

…HTML5 standard (GH-135664) (GH-136271) * "--!>" now ends the comment. * "-- >" no longer ends the comment. * Support abnormally ended empty comments "<-->" and "<--->". --------- (cherry picked from commit 8ac7613) Co-author: Kerim Kabirov <the.privat33r+gh@pm.me> Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>

pythongh-135661: Fix comment parsing in HTMLParser

2706182

* "--!>" now ends the comment. * "-- >" no longer ends the comment. * Support abnormally ended empty comments "<-->" and "<--->".

serhiy-storchaka requested a review from ezio-melotti as a code owner June 18, 2025 10:32

serhiy-storchaka added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Jun 18, 2025

bedevere-app bot added the awaiting core review label Jun 18, 2025

bedevere-app bot mentioned this pull request Jun 18, 2025

HTMLParser differences from the HTML5 specification #135661

Open

serhiy-storchaka added 3 commits June 25, 2025 14:56

Merge branch 'main' into htmlparser-comment

0e6d20a

Add more tests: "<!---!>" and "<!--!>".

f1c9efa

Move to pythongh-102555.

0f05afc

Co-author: Kerim Kabirov <the.privat33r+gh@pm.me>

serhiy-storchaka mentioned this pull request Jun 25, 2025

gh-102555 Increase HTML standard compliance for closing comment tags #117406

Closed

serhiy-storchaka changed the title ~~gh-135661: Fix comment parsing in HTMLParser~~ gh-102555: Fix comment parsing in HTMLParser Jun 25, 2025

bedevere-app bot mentioned this pull request Jun 25, 2025

XSS in html.parser library #102555

Open

Privat33r-dev reviewed Jun 25, 2025

View reviewed changes

serhiy-storchaka added 2 commits July 3, 2025 16:38

Merge branch 'main' into htmlparser-comment

3ec0926

Move to Security.

b146249

ezio-melotti approved these changes Jul 4, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting core review labels Jul 4, 2025

serhiy-storchaka and others added 2 commits July 4, 2025 09:02

Update Misc/NEWS.d/next/Security/2025-06-18-13-28-08.gh-issue-102555.…

c8d9740

…nADrzJ.rst Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>

Merge branch 'main' into htmlparser-comment

1503ae7

serhiy-storchaka added type-security A security issue needs backport to 3.9 only security fixes needs backport to 3.10 only security fixes needs backport to 3.11 only security fixes needs backport to 3.12 only security fixes labels Jul 4, 2025

serhiy-storchaka enabled auto-merge (squash) July 4, 2025 06:08

serhiy-storchaka merged commit 8ac7613 into python:main Jul 4, 2025
81 of 83 checks passed

bedevere-app bot removed the awaiting merge label Jul 4, 2025

bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Jul 4, 2025

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Jul 4, 2025

bedevere-app bot removed the needs backport to 3.12 only security fixes label Jul 4, 2025

bedevere-app bot removed the needs backport to 3.11 only security fixes label Jul 4, 2025

bedevere-app bot removed the needs backport to 3.10 only security fixes label Jul 4, 2025

bedevere-app bot removed the needs backport to 3.9 only security fixes label Jul 4, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

gh-102555: Fix comment parsing in HTMLParser #135664

gh-102555: Fix comment parsing in HTMLParser #135664

serhiy-storchaka commented Jun 18, 2025 •

edited by bedevere-app bot

Loading

Uh oh!

Privat33r-dev left a comment

Uh oh!

Privat33r-dev Jun 25, 2025 •

edited

Loading

Uh oh!

ezio-melotti Jul 4, 2025

Uh oh!

serhiy-storchaka Jul 4, 2025

Uh oh!

ezio-melotti Jul 4, 2025

Uh oh!

Uh oh!

Uh oh!

miss-islington-app bot commented Jul 4, 2025

Uh oh!

bedevere-app bot commented Jul 4, 2025

Uh oh!

bedevere-app bot commented Jul 4, 2025

Uh oh!

bedevere-app bot commented Jul 4, 2025

Uh oh!

bedevere-app bot commented Jul 4, 2025

Uh oh!

bedevere-app bot commented Jul 4, 2025

Uh oh!

bedevere-app bot commented Jul 4, 2025

Uh oh!

Uh oh!

TMZ Celebrity News – Breaking Stories, Videos & Gossip

🎥 Watch TMZ Live

	def parse_comment(self, i, report=1):
	rawdata = self.rawdata
	if rawdata[i:i+4] != '<!--':
	raise AssertionError('unexpected call to parse_comment()')
	match = _commentclose.search(rawdata, i+4)
	if not match:
	return -1
	if report:
	j = match.start(0)
	self.handle_comment(rawdata[i+4: j])
	return match.end(0)

Uh oh!

gh-102555: Fix comment parsing in HTMLParser #135664

gh-102555: Fix comment parsing in HTMLParser #135664

Conversation

serhiy-storchaka commented Jun 18, 2025 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Privat33r-dev left a comment

Choose a reason for hiding this comment

Uh oh!

Privat33r-dev Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ezio-melotti Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

ezio-melotti Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

miss-islington-app bot commented Jul 4, 2025

Uh oh!

bedevere-app bot commented Jul 4, 2025

Uh oh!

bedevere-app bot commented Jul 4, 2025

Uh oh!

bedevere-app bot commented Jul 4, 2025

Uh oh!

bedevere-app bot commented Jul 4, 2025

Uh oh!

bedevere-app bot commented Jul 4, 2025

Uh oh!

bedevere-app bot commented Jul 4, 2025

Uh oh!

Uh oh!

TMZ Celebrity News – Breaking Stories, Videos & Gossip

🎥 Watch TMZ Live

serhiy-storchaka commented Jun 18, 2025 •

edited by bedevere-app bot

Loading

Privat33r-dev Jun 25, 2025 •

edited

Loading