Sync Arize and Phoenix skills from arize-skills#1749
Open
jimbobbennett wants to merge 1 commit into
Open
Conversation
…d10a776b2317d9 and phoenix@3ab13bc62ca1f30966531f59223dfb168f6effdc
github-actions
Bot
added
new-submission
Contributor
🔍 Skill Validator Results
Summary
Full validator output```text Found 3 skill(s) [arize-compliance-audit] 📊 arize-compliance-audit: 4,071 BPE tokens [chars/4: 4,430] (standard ~), 27 sections, 2 code blocks [arize-compliance-audit] ⚠ Skill is 4,071 BPE tokens (chars/4 estimate: 4,430) — approaching "comprehensive" range where gains diminish. [arize-instrumentation] 📊 arize-instrumentation: 6,183 BPE tokens [chars/4: 6,271] (comprehensive ✗), 19 sections, 5 code blocks [arize-instrumentation] ⚠ Skill is 6,183 BPE tokens (chars/4 estimate: 6,271) — "comprehensive" skills hurt performance by 2.9pp on average. Consider splitting into 2–3 focused skills. [phoenix-tracing] 📊 phoenix-tracing: 1,488 BPE tokens [chars/4: 1,690] (detailed ✓), 17 sections, 1 code blocks ✅ All checks passed (3 skill(s)) ``` |
github-actions
Bot
added
the
skill-check-warning
Contributor
There was a problem hiding this comment.
Pull request overview
This PR syncs Arize/Phoenix skill content by updating tracing reference docs and adding a new Arize compliance audit skill with supporting regulatory reference material.
Changes:
- Adds a new
arize-compliance-auditskill and reference checklist/framework files. - Updates Phoenix tracing docs for project routing via OTLP HTTP headers and LLM tools masking.
- Updates Arize instrumentation guidance for Go setup and credential handling.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
skills/phoenix-tracing/references/projects-typescript.md |
Adds OTLP HTTP x-project-name project routing guidance. |
skills/phoenix-tracing/references/projects-python.md |
Adds OTLP HTTP header routing examples for Python and collectors. |
skills/phoenix-tracing/references/production-typescript.md |
Adds OPENINFERENCE_HIDE_LLM_TOOLS masking guidance. |
skills/phoenix-tracing/references/production-python.md |
Adds Python hide_llm_tools masking guidance. |
skills/arize-instrumentation/SKILL.md |
Refines Go installation/exporter instructions and manual span error status handling. |
skills/arize-compliance-audit/SKILL.md |
Adds a new compliance audit skill workflow. |
skills/arize-compliance-audit/references/us-ai-compliance.md |
Adds US AI compliance reference material. |
skills/arize-compliance-audit/references/iso-42001.md |
Adds ISO 42001 developer reference material. |
skills/arize-compliance-audit/references/eu-ai-act-gpai.md |
Adds EU AI Act/GPAI reference material. |
skills/arize-compliance-audit/references/compliance-checklist-template.md |
Adds reusable compliance checklist template. |
docs/README.skills.md |
Adds the new compliance audit skill to the skill index. |
Comments suppressed due to low confidence (5)
skills/arize-compliance-audit/SKILL.md:73
- This repeats the non-existent
AskUserQuestiontool reference. Use the normal skill phrasing to ask the user directly, otherwise the agent may fail before it can collect the required use-case information.
Use the `AskUserQuestion` tool to ask: **What does your AI application do?**
skills/arize-compliance-audit/SKILL.md:236
- Requiring the non-existent
AskUserQuestiontool here can break the remediation flow in clients that only support normal conversational confirmation. Existing skills use plain instructions such as asking the user or proceeding only after the user confirms.
After presenting the checklist, offer to implement specific fixes. **Always use the `AskUserQuestion` tool to confirm before making any changes.**
skills/arize-compliance-audit/SKILL.md:263
- This instruction again depends on
AskUserQuestion, which is not a repository-supported tool name. It should tell the agent to get explicit user confirmation in chat instead of invoking a tool that may not exist.
- Show exactly what will change (file, code diff concept) then use the `AskUserQuestion` tool to get confirmation before applying.
skills/arize-compliance-audit/SKILL.md:270
- This orchestration step also names the unsupported
AskUserQuestiontool. Leaving one occurrence behind would still cause agents following this skill to attempt an unavailable tool call before invoking related skills.
When gaps identified in Phase 1 or 2 require capabilities from other Arize skills, offer to invoke them. **Always use the `AskUserQuestion` tool to ask before invoking another skill** and explain why it is relevant to the compliance gap.
skills/arize-compliance-audit/SKILL.md:225
- This repeated “quoted code snippets” requirement has the same secret-leak risk for hardcoded credential findings. The final report instructions should explicitly redact secret values instead of requiring verbatim snippets in all cases.
- One subsection per Non-compliant or Partial domain, each containing: exact file paths and line numbers, quoted code snippets, app-specific risk explanation, and a precise description of what is missing. This section is mandatory — never omit it.
|
|
||
| ### Step 1 — Framework selection | ||
|
|
||
| Use the `AskUserQuestion` tool to ask the user which frameworks apply. **Do not infer or auto-select** — always ask explicitly. |
| @@ -0,0 +1,314 @@ | |||
| --- | |||
| name: arize-compliance-audit | |||
| description: "INVOKE THIS SKILL when auditing an AI agent or LLM app for regulatory compliance. Covers EU AI Act, GPAI Code of Practice, GDPR, NIST AI RMF, Colorado AI Act, HIPAA, and ISO 42001. Scans the codebase for compliance gaps, cross-references Arize instrumentation for audit trail coverage, and produces an actionable remediation checklist tailored to the selected frameworks." | |||
|
|
||
| For each domain rated Non-compliant or Partial, write a dedicated subsection that includes: | ||
|
|
||
| 1. **The exact code path** — file path(s), line number(s), and the relevant code snippet showing where the gap exists. Do not paraphrase; quote the actual code. |
| 2. **Mark items from Phase 1.** Items where evidence was found: mark as `Compliant`. Items with gaps: mark as `Non-compliant` with a concrete remediation suggestion. | ||
| 3. **Prioritise correctly.** Critical = enforcement risk or system prohibition. High = required by regulation. Medium = recommended by framework. Low = best practice. | ||
| 4. **Be specific in remediation.** Instead of "implement input validation", say "add a guardrail library like `guardrails-ai` to validate LLM inputs and outputs against your content policy". | ||
| 5. **Include the instrumentation cross-reference table** from the template. If Arize tracing is not set up, flag this as a Critical gap — audit trails are required by EU Art. 12 and NIST MAN-2.1. |
| | Retention requirements | Trace data retained for required period | EU: appropriate period (min 6 months for high-risk); HIPAA: 6 years | | ||
| | Bias monitoring | Demographic or group attributes | Check for metadata attributes that enable fairness analysis | | ||
|
|
||
| If Arize tracing is **not** set up, this is a significant compliance gap. Offer: "Shall I run the `arize-instrumentation` skill to set up audit-trail tracing? Regulatory frameworks (EU AI Act Art. 12, NIST AI RMF MAN-2.1) require event logging for AI systems." |
Comment on lines
+2
to
+3
| name: arize-compliance-audit | ||
| description: "INVOKE THIS SKILL when auditing an AI agent or LLM app for regulatory compliance. Covers EU AI Act, GPAI Code of Practice, GDPR, NIST AI RMF, Colorado AI Act, HIPAA, and ISO 42001. Scans the codebase for compliance gaps, cross-references Arize instrumentation for audit trail coverage, and produces an actionable remediation checklist tailored to the selected frameworks." |
Contributor
|
@jimbobbennett - we've expanded the policy for external plugins, would it make sense to migrate this across? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Checklist
npm startand verified thatREADME.mdis up to date.stagedbranch for this pull request.Description
Updates the Arize AX and Phoenix skills to the latest version
Type of Contribution
Additional Notes
By submitting this pull request, I confirm that my contribution abides by the Code of Conduct and will be licensed under the MIT License.