Add Thai-NNER integration with nested entity filtering utilities

[Copilot]

What do these changes do

Integrates Thai Nested Named Entity Recognition (Thai-NNER) into PyThaiNLP, adding utilities for filtering nested entity hierarchies and providing Thai-NNER as a standard NER engine option alongside existing engines like thainer and wangchanberta.

What was wrong

The existing Thai-NNER wrapper lacked practical utilities for working with nested entities:

• No mechanism to extract only top-level entities from nested hierarchies
• Only accessible via NNER class with custom output format, inconsistent with other NER engines
• Missing standard IOB format compatibility used by other engines
• Class naming used underscores instead of PascalCase convention
• Missing explicit public API declaration

How this fixes it

Core utilities (pythainlp/tag/thai_nner.py):

• ThaiNNER class with PascalCase naming for consistency
• get_top_level_entities() function using O(n×m) sorted span comparison (scoped to thai_nner module)
• top_level_only parameter in ThaiNNER.tag() for direct filtering
• Helper functions work independently without thai-nner library (import in __init__ for safety)
• __all__ export list for explicit public API declaration

NER engine integration (pythainlp/tag/named_entity.py):

• Added "thai-nner" as engine option in NER class
• get_ner() method provides IOB/HTML format compatibility
• Restructured NER.load_engine() to check corpus-independent engines first
• _entities_to_iob() and _entities_to_html() converters with documented non-overlapping entity assumptions

Code quality improvements:

• Python 3.9+ compatible type hints (Union instead of |)
• Proper type annotations including Optional[str] and dict[str, Any]
• Consistent docstring formatting (lowercase type names matching type hints)
• Clear error messages for missing dependencies
• Default parameter resolution at runtime instead of module import time
• Documented import pattern rationale

Tests:

• Core tests (tests/core/test_tag.py): Helper function tests without external dependencies
• Noauto tests (tests/noauto/testn_tag.py): Integration tests requiring torch/transformers

Usage examples:

from pythainlp.tag import NNER, NER

# NNER class with top-level filtering
nner = NNER()
tokens, entities = nner.tag("วันที่ 5 เมษายน 2565", top_level_only=True)

# NER class with thai-nner engine (IOB format)
ner = NER(engine="thai-nner")
result = ner.tag("วันที่ 5 เมษายน 2565")  # IOB format
result = ner.tag("วันที่ 5 เมษายน 2565", tag=True)  # HTML tags

# Advanced: Direct utility access
from pythainlp.tag.thai_nner import ThaiNNER, get_top_level_entities
nner = ThaiNNER()
top_entities = get_top_level_entities(entities)

Model information:

• PyThaiNLP corpus database (version 1.0), CC-BY-SA 3.0 license
• Auto-downloads on first use (431MB)
• Supports 104 entity types across 8 nesting layers

Your checklist for this pull request

• Passed code styles and structures
• Passed code linting checks and unit test

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[TODO]: Add NER from Thai-NNER dataset</issue_title>
<issue_description>From Thai-NNER, The dataset has 4,894 docs.

The dataset is licensed under CC-BY-SA 3.0.

dev.txt
train.txt</issue_description>

<agent_instructions>You can use Thai-NNER and get just the top tag of nner. https://github.com/vistec-AI/Thai-NNER/blob/main/thai_nner/__init__.py#L31</agent_instructions>

Comments on the Issue (you are @copilot in this section)

@wannaphong @pavaris-pm @MpolaarbearM Do you interested to train the model? @wannaphong > I completed the training on the NER model and performed some fine-tuning, but the outcomes were not meeting expectations. I plan to make adjustments to the model in an attempt to enhance its performance. https://huggingface.co/lunarlist/NER_thai_phayathai

The dataset is very hard. I think you can turn some parameter.</comment_new>
<comment_new>@wannaphong
I rethink about this issue. You can actually use Thai-NNER and get just the top tag of nner. https://github.com/vistec-AI/Thai-NNER/blob/main/thai_nner/__init__.py#L31

Don't need to train any model!</comment_new>

• Fixes [TODO]: Add NER from Thai-NNER dataset #888

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[coveralls]

coverage: 65.726% (+0.2%) from 65.571%
when pulling 350d1af on copilot/add-ner-thai-nner-dataset
into e9b8f3b on dev.

[wannaphong]

@copilot Add docs to docs/api/tag.rst

[Copilot]

@copilot Add docs to docs/api/tag.rst

Added in commit 8f7a9ae. The get_top_level_entities function is now documented in docs/api/tag.rst with the autofunction directive.

[wannaphong]

@copilot Add to NER class (as an engine) in pythainlp/tag/named_entity.py

[Copilot]

@copilot Add to NER class (as an engine) in pythainlp/tag/named_entity.py

Added in commit 4032cb1. Thai-NNER is now available as an engine option in the NER class:

from pythainlp.tag import NER

ner = NER(engine="thai-nner")
result = ner.tag("วันที่ 5 เมษายน 2565")  # IOB format
result = ner.tag("วันที่ 5 เมษายน 2565", tag=True)  # HTML tags

The implementation converts Thai-NNER's nested entity format to standard IOB format, using only top-level entities to avoid overlapping tags.

[bact]

@copilot The dev branch is updated. Please resync with it.

The conflicts come from the "extra" test suite, which got refactored.
Some test cases were relocated to other test suites.

See the test suite refactoring and naming conventions in this merged PR
#1248

[Copilot]

@copilot The dev branch is updated. Please resync with it.

The conflicts come from the "extra" test suite, which got refactored.
...

Rebased on dev branch (commit b536a8b). The conflict in tests/extra/testx_tag.py has been resolved by following the new test suite conventions from PR #1248:

• Removed tests requiring extra dependencies (NER class with transformers/tltk engines, NNER class, pos_tag_transformers) from testx_tag.py as they were moved to the noautotest suite
• Kept tests in tests/core/test_tag.py that test helper functions (_entities_to_iob, _entities_to_html, get_top_level_entities) which don't require the actual thai-nner library
• All our Thai-NNER integration changes remain intact and are now based on the latest dev branch

The rebase is complete locally but I need assistance with force-pushing the rebased branch since I don't have direct push access.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[bact]

@copilot review again

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

The overlap assumptions have been documented in both functions:

1. _entities_to_iob() (lines 205-208): Documents that the function assumes non-overlapping entities and recommends using top-level entities
2. _entities_to_html() (lines 239-241): Documents the same assumption with guidance to use get_top_level_entities() for filtering

Both were fixed in commit 51a335a.

[bact]

@copilot dev branch is updated, with a lot of type hints improvements. Resync with the dev branch.

[Copilot]

@copilot dev branch is updated, with a lot of type hints improvements. Resync with the dev branch.

Synced with dev branch in commit c221d94. Merged 55 commits including comprehensive type hints improvements. The auto-merge in pythainlp/tag/named_entity.py was successful, updating type: ignore comments to more specific directives (type: ignore[no-any-return]). All Thai-NNER integration tests pass.

[bact]

@copilot dev branch is updated, with a lot of type hints improvements. Resync with the dev branch.

[Copilot]

@copilot dev branch is updated, with a lot of type hints improvements. Resync with the dev branch.

Synced with dev branch in commit 03116b3. Merged 12 commits including comprehensive type hints and mypy improvements. Resolved conflict in pythainlp/tag/thai_nner.py by keeping our full implementation with top_level_only parameter, get_ner method, and helper functions. All Thai-NNER core tests pass successfully.

[bact]

@copilot

• The Copilot instructions at https://github.com/PyThaiNLP/pythainlp/blob/dev/.github/copilot-instructions.md is updated, reread and apply
• dev branch is updated, with a lot of type hints improvements. Resync with the dev branch.

[Copilot]

@copilot

• The Copilot instructions at https://github.com/PyThaiNLP/pythainlp/blob/dev/.github/copilot-instructions.md is updated, reread and apply
  ...

Completed in commit 350d1af:

1. Reread updated Copilot instructions from .github/copilot-instructions.md - reviewed all guidelines including type hints, imports, testing conventions, and code quality standards
2. Synced with dev branch - merged 14 commits including comprehensive type hints improvements across 30+ files
3. Resolved conflicts - kept our full Thai-NNER implementation while updating type annotations to use dict[str, Any] for better type safety (matching dev branch improvements)
4. Updated return type - Changed NNER.tag() return type from tuple[list[str], list[dict]] to tuple[list[str], list[dict[str, Any]]] for improved type checking

All imports verified successfully.

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud