[python] feature request - add api.md to generation

[swathipil]

Add a step to the Python TypeSpec emitter to automatically generate an api.md file as part of the emitter pipeline.

Prerequirements:

• Export-APIViewMarkdown.ps1 should be available under eng/emitters/scripts or eng/common/scripts. (cc/ @timotheeguerin) addressed here: Copy Export-APIViewMarkdown.ps1 into eng/emitters/scripts/ #10096

Scope

• Generate api.md.
• By default, it should be generated, but allow for an opt‑out.
• Any apistubgen errors should warn and continue, and NOT block.
• The api.md file should NOT be included in the whl (exclude from pyproject.toml/setup.py).

Proposed implementation

Extend the Python emitter pipeline from:

preprocess → generate → format (black)

to:

preprocess → generate → format (black) → apistubgen → api.md 

1. Run the apistubgen command on the package to generate the token JSON file. This file will not be added to the package artifacts.
2. Pass the token file to Export-APIViewMarkdown.ps1 on the JSON token file to generate the api markdown file.

[msyyc]

Hope to get more context here:

• we already configure https://github.com/microsoft/typespec/blob/d3b7f61f9e1f39a1d522fc5aed80edc39bd71bac/packages/http-c… to run apiview command, so we need to run this scrip after that to generate a file named api.md?
• According to issue description, i understand the new script will block CI if failed?
• why do we need such a step in pipeline? The script is actually not in scope of python emitter. For apiview command, we could say "oh, we need to make sure our generated SDK could be used to generate a ApiView json". For this script which consumes the ApiView json, it is more far away from the scope of python emitter.
  CC @swathipil / @iscai-msft

[swathipil]

@msyyc -

• It looks like the apiview-stub-generator package is only added to the tox test environment. In our case, we want the python emitter to actually run this during the SDK generation, so it seems like it will need to be a dependency in the package. But yes like you said, the steps will be:

1. Run apiview-stub-generator to produce the token file.
2. Pass the token file to this script to generate the api.md

• No, this step should not be blocking. If it fails, the emitter should just warn and continue. Updated the issue to note this.
• More context:
  • This api.md feature was requested by Python architects and I've discussed it with Laurent as well.
  • The script was originally part of the parser, but we decided to move it out since the JSON token file -> md conversion logic is the same across languages. Otherwise, this same script would be duplicated across parsers.
  • So in order to get the api.md file, we need to run both the parser and this script.

[swathipil]

Offline discussion: This feature is blocked by:

• [Python APIView] investigate perf bottlenecks when generating stubs, especially for larger packages Azure/azure-sdk-tools#14659.
• [Python APIView] Update supported versions Azure/azure-sdk-tools#9557

I'm working to unblock these. Once these are resolved, I'll investigate whether it's possible to use the CodeModel to directly generate the token file/api.md, rather than CodeModel -> generated SDK -> Python API parser -> token file -> api.md. That should be much faster.

For now, we'll be running a nightly pipeline to regen api.md in SDK folders when needed.

[swathipil]

Resolved both blockers, and the fixes will be in the next release. Adding an option to the emitter to run the api parser to generate from code-model.yaml for faster api.md generation. This can be used when the package is fully generated. Otherwise users can opt out and generate manually.