Update docs

[twm]

When trying to figure out how to sanitize some HTML I noticed that the docs have lagged behind the implementation. So I removed the mention of HTMLSanitizer and then... things got out of hand. Summary of changes:

• Add tox -e doc environment so that it's easy to build the docs.
• Remove documentation of HTMLTokenizer, as it is now private.
• Remove documentation of HTMLSanitizer, as it no longer exists.
• Add basic documentation of the treeadapters package.
• Linkify many references to classes, functions, and modules.
• Fix various Sphinx warnings and formatting issues.
• Add __version__ to html5lib.__all__
• Remove sub-modules from html5lib.treeadapters.__all__ (as this caused Sphinx warnings and didn't really make sense).

I didn't squash as suggested in CONTRIBUTING.md because recent PRs don't seem to be following that procedure. I am happy to do so if you like, or to try to break this into smaller PRs.

[willkg]

The sanitizer still exists, but it got rewritten as a filter and is now in html5lib.filters.sanitizer. You can see it here:

https://github.com/html5lib/html5lib-python/blob/17499b9763a090f7715af49555d21fe4b558958b/html5lib/filters/sanitizer.py

Just a drive-by comment on the off chance it helps.

[twm]

@willkg Yup, and the new form is documented. The old docs just weren't removed.

[willkg]

I'm really sorry this has been sitting around so long. I appreciate you working on it and fixing so many issues!

I have some comments. If you have time, can you look through them? If not, let me know and I can work through them.

Thank you! Looking forward to landing this!

[willkg]

I don't think we want __version__ to get imported when importing *.

[twm]

As __version__ is part of the public interface of this module, which __all__ defines, I think it should be here.

[willkg]

I wouldn't think about __all__ that way. It's really for specifying what gets exported if the user does from html5lib import *. I don't want __version__ to get exported in that scenario. Please undo this change.

[willkg]

I'd remove this. Amongst other things, it's not going to be true for much longer.

[twm]

Thank goodness! Will do.

[willkg]

Is it really an AssertionError? If so, we should write up an issue to change that.

[twm]

Yeah, the implementation is basically all assert statements:

html5lib-python/html5lib/filters/lint.py

Lines 24 to 79 in 7bbde54

	assert namespace is None or isinstance(namespace, text_type)
	assert namespace != ""
	assert isinstance(name, text_type)
	assert name != ""
	assert isinstance(token["data"], dict)
	if (not namespace or namespace == namespaces["html"]) and name in voidElements:
	assert type == "EmptyTag"
	else:
	assert type == "StartTag"
	if type == "StartTag" and self.require_matching_tags:
	open_elements.append((namespace, name))
	for (namespace, name), value in token["data"].items():
	assert namespace is None or isinstance(namespace, text_type)
	assert namespace != ""
	assert isinstance(name, text_type)
	assert name != ""
	assert isinstance(value, text_type)
	elif type == "EndTag":
	namespace = token["namespace"]
	name = token["name"]
	assert namespace is None or isinstance(namespace, text_type)
	assert namespace != ""
	assert isinstance(name, text_type)
	assert name != ""
	if (not namespace or namespace == namespaces["html"]) and name in voidElements:
	assert False, "Void element reported as EndTag token: %(tag)s" % {"tag": name}
	elif self.require_matching_tags:
	start = open_elements.pop()
	assert start == (namespace, name)
	elif type == "Comment":
	data = token["data"]
	assert isinstance(data, text_type)
	elif type in ("Characters", "SpaceCharacters"):
	data = token["data"]
	assert isinstance(data, text_type)
	assert data != ""
	if type == "SpaceCharacters":
	assert data.strip(spaceCharacters) == ""
	elif type == "Doctype":
	name = token["name"]
	assert name is None or isinstance(name, text_type)
	assert token["publicId"] is None or isinstance(name, text_type)
	assert token["systemId"] is None or isinstance(name, text_type)
	elif type == "Entity":
	assert isinstance(token["name"], text_type)
	elif type == "SerializerError":
	assert isinstance(token["data"], text_type)
	else:
	assert False, "Unknown token type: %(type)s" % {"type": type}

[willkg]

Given that what follows is a bulleted list, can you add a : to the end of this line?

[willkg]

The leader has "a few tools", but this list has two items. Further, the items seem like a sentence. I'd either change the bullet list into a sentence or unsentencify the bullet items. Maybe something like this?:

html5lib provides two tools for consuming token streams:

* :class:`~html5lib.serializer.HTMLSerializer` for generating a stream of bytes
* filters for manipulating the token stream

[twm]

I ended up going with a sentence, rather than the bulleted list, as there aren't exactly two (really there's the serializer and a bunch of filters), so it's really two categories of token consumer, but that distinction isn't really useful to point out.

[willkg]

Why take the header out here?

[twm]

Otherwise there are two headers in a row with exactly the same text.

[twm]

I think that I have addressed all the issues you noted as appropriate. Please let me know if anything else is required! Thanks!

[willkg]

I have one outstanding issue with __version__ being listed in __all__. Otherwise this is good to go. Thank you!

[willkg]

@twm Thank you so much for this!