Skip to content

feat: grammar fixes and add new studio-highlighting.scm file #34

Merged
davem-intersys merged 11 commits intointersystems:mainfrom
hkimura-intersys:studio
Apr 10, 2026
Merged

feat: grammar fixes and add new studio-highlighting.scm file #34
davem-intersys merged 11 commits intointersystems:mainfrom
hkimura-intersys:studio

Conversation

@hkimura-intersys
Copy link
Copy Markdown
Collaborator

Overview

This PR adds a new studio-highlighting.scm file to group nodes the same way studio does for highlighting purposes. Additionally, fixes to the core grammar were added (see below for details)

Key Changes

1. Parser and scanner behavior were tightened

The core scanner gained explicit handling for inactive preprocessor blocks and a
broader set of command-like tokens.

  • common/scanner.h adds POUND_IF_SPECIAL_CASE*, MNEMONIC, and
    TAG_END_IF scanner tokens.
  • #if 0 ... #endif blocks are now recognized as a special case so the parser
    can treat unreachable code differently and queries can highlight it as
    inactive. When #IF 0 happens, iris actually registers everything from that line up to #ELSE ,#ENDIF, and #ELSEIF 1 as a comment (so it is technically valid to write whatever you want within those lines). Additionally, here the #ELSE and #ELSEIF 1 happens 100% of the time, so IRIS treats this the same way as an #ENDIF, and makes it so #ENDIF is optional. There were some cases where someone started a block between the #ELSE and #ENDIF, which caused the parser to break. To fix this, in this special case, the #ENDIF is treated as a special tag if #ELSE has already been seen.
  • Statement detection (for the objectscript playground grammar) at column 1 was broadened for more command forms and abbreviations, including MV*, ZALLOCATE, ZLOAD, ZUSE, and related
    Z* commands.
  • Scanner logic was improved around dotted-line continuations, slash-prefixed
    mnemonics, comments between a condition and a following block, and
    argumentless loop/command termination.

2. Routine Grammar Changes

Before, it was valid to have anything in the first three lines (would return compiled_header token). Now, the second line must be %RO (as all compiled versions have this) to get the compiled_header token. Valid options are compiled_header, routine header (ROUTINE name [type = ]), or an objectscript statement (added this as it is allowed in studio).

  • objectscript_routine/grammar.js introduces routine_definition and allows a
    source file to begin with a compiled header, a ROUTINE definition, or a
    top-level statement.
  • objectscript_routine/src/scanner.c simplifies the compiled-header and
    ROUTINE detection path and keeps routine-specific lexing active without the
    older staged gating flags.
  • rtn_dot handling is now part of the routine grammar extras, which better
    matches real .mac/.rtn content.
  • README.md and CONTRIBUTING.md were updated to describe
    objectscript_routine as a routine-file grammar instead of a
    routine-header grammar.

3. Expression, core, and UDL grammars were broadened and cleaned up

Grammar Changes include:

  • expr/grammar.js
    • replaces the older pattern-expression regex with a more structured nested
      form (allows ( and ) also)
    • adds $BITLOGIC as a separate built in function, as this one function allows operators that normal expressions don't
    • allows extrinsic indirection with optional method args
    • broadens $SYSTEM.* chaining (as the last item is allowed to be an oref chain expr, not just a regular method)
    • folds parameter-like member access into oref_parameter
  • core/grammar.js
    • aligns ZBREAK parsing with the new mnemonic and option tokens
    • allows macros in NEW (since a macro can evaluate to a local var)
    • generalizes MERGE arguments to set_target
    • adds ZLOAD
    • introduces a macro-command form (a macro can be an if statement -> $$$MACRO { w hi} is valid)
    • relaxes some GOTO/argument parsing cases
  • udl/grammar.js
    • reuses return_type for property, relationship, and projection typing
    • improves typename parsing for Type(...) Of ...
    • treats more keyword values as type-like names for cleaner trees and
      highlighting
  • common/keywords.js and common/identifiers.js were adjusted to support the
    new aliases and identifier behavior.

4. Studio-specific highlighting

  • New studio-highlights.scm files were added for:
    • objectscript/queries/
    • objectscript_routine/queries/
    • udl/queries/
  • Matching packaged copies were added where needed in Python query directories
    for routine and UDL bindings.
  • Rust bindings now export STUDIO_HIGHLIGHTS_QUERY constants in:
    • bindings/rust/lib.rs
    • bindings/rust-routine/lib.rs
    • bindings/rust-playground/lib.rs
  • The staged Rust crates now include those query files in their package
    manifests and staging scripts.

5. Standard highlight queries were updated to match the new tree shape

The existing highlights.scm files across expr, core, udl,
objectscript, and objectscript_routine were refreshed.

These changes include:

  • parameter_name highlighting moved to oref_parameter
  • builtin/system captures were split more cleanly from ordinary identifiers
  • new captures were added for zbreak_command_option, tag_end_if,
    command_keyword, keyword_zload, and dotted_statement
  • #if 0 regions can now be tagged as inactive code
  • UDL/objectscript queries were updated to reflect the return_type-based type
    cleanup

.tsqueryrc.json was updated at the same time to document new capture names
such as:

  • comment.inactive
  • punctuation.special.dots
  • variable.member.oref
  • variable.member.sql
  • punctuation.bracket.json

6. Query sync and packaging workflows were updated for the new assets

The repo now treats Studio highlight files as related, but separate, artifacts.

  • scripts/sync_queries.py still manages the canonical trio:
    • highlights.scm
    • indents.scm
    • injections.scm
  • studio-highlights.scm is now explicitly excluded from the normal composed
    query sync and Python query-copy verification path.
  • scripts/rust_routine_crate.sh and scripts/rust_playground_crate.sh now
    copy studio-highlights.scm into staged crate directories.
  • cargo_readme.md, cargo_readme_playground.md, and
    cargo_readme_routine.md were updated to document the new exported query
    constants.

Testing

  • all parsers and queries have been tested and are passing:
image

This was tested on many real rtn, mac files as well.

@hkimura-intersys hkimura-intersys marked this pull request as draft April 9, 2026 14:18
@hkimura-intersys hkimura-intersys marked this pull request as ready for review April 9, 2026 14:24
@hkimura-intersys hkimura-intersys marked this pull request as draft April 9, 2026 17:20
@hkimura-intersys hkimura-intersys marked this pull request as ready for review April 9, 2026 19:55
@davem-intersys davem-intersys merged commit d1f119c into intersystems:main Apr 10, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants