Add LSP Client and long running tests in tools by chrisqm-dev · Pull Request #511 · aws-cloudformation/cloudformation-languageserver

chrisqm-dev · 2026-03-31T21:53:38Z

Description of changes:

Add lsp client methods to start up and interact with the cfn language server
Can be used for telemetry generator as well
Add long-running tests
- This tests the standalone.js and ensures it is usable for extended periods of time
- Tests the following:
  - Completion on document updates
  - Hover on document updates
  - Region switching, and therefore public schema retrieval
  - Ensures performance doesn't degrade over time

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

chrisqm-dev · 2026-03-31T21:56:24Z

tools/lspClient/LspClient.ts

+        const output = data.toString().trim();
+
+        // Readiness detection
+        if (output.includes('cfn-lint version')) {


Currently, the only way to determine if lint, guard, and the schemas are ready is through the logs. #507 Introduces a new LSP request which will return the guard, lint, and schema statuses. This addition will make testing more robust and speed up test time, but this PR is functional without this change

satyakigh · 2026-04-01T16:41:29Z

Our folder and file names are usually camelCased unless its a bin/cli command file

satyakigh · 2026-04-01T16:40:35Z

tools/lsp-client/index.ts

@@ -0,0 +1,3 @@
+export { LspClient } from './LspClient';


We don't use index.ts

Removed index.ts file

satyakigh · 2026-04-01T16:43:18Z

tools/lsp-client/types.ts

@@ -0,0 +1,49 @@
+import { InitializeParams } from 'vscode-languageserver-protocol';


Can all of this go in LspConnectionInterface? Should LspConnectionInterface be LspConnection ? our file names don't usually have interface naming

Renamed LspConnectionInterface and file to LspConnection. Moved types.ts content into LspConnection.ts.

satyakigh · 2026-04-01T16:43:39Z

tools/lsp-client/types.ts

@@ -0,0 +1,49 @@
+import { InitializeParams } from 'vscode-languageserver-protocol';
+
+export interface LspClientConfig {


Think a lot of these should be required not optional

Made every property required except for:

env

storageDir

suppressLogLevels

Also, added clientInfo and extensionInfo properties

satyakigh · 2026-04-01T16:44:43Z

tools/lsp-client/types.ts

+    suppressLogLevels?: string[];
+}
+
+export interface ReadinessFlags {


Can't all the types from src be used?

Modified to use all the src types

satyakigh · 2026-04-01T16:45:10Z

.gitignore

 .DS_Store
 tools/*
 !tools/*.ts
+!tools/lsp-client/


The previous line can be updated to !tools/**/*.ts instead

modified to use the lines: !tools//
!tools/**/.ts

!tools/*/ is necessary because without it the subdirectories will be ignored

satyakigh · 2026-04-01T16:51:56Z

tools/stability/Monitoring.ts

+export type StandaloneTestMetrics = {
+    operationsAttempted: number;
+    operationsFailed: number;
+    averageDuration: number | null;


Can't these just be an array of numbers and the user decides what to do with them

Because this test could run for a very long time and there are thousands of operations per minute, the duration array could get very large. We should keep the min, max, and average attributes to avoid having to calculate from the array. Added the durations array in case there's any other operations we use in the future, such as median or p99.

satyakigh · 2026-04-01T16:52:34Z

tools/stability/Monitoring.ts

+    }
+}
+
+export function generateFinalReport(testStartTime: number): void {


What will this report be used for?

This prints out to the github actions logs upon completing successfully. I wanted to replicate a test coverage output report similar to what you see in vitest. We can't use vitest since it will not work with node 18

satyakigh · 2026-04-01T16:53:45Z

tools/stability/runStabilityTest.ts

+    }
+}
+
+void main();


This will swallow the exception and succeed the run

Added logic so that the exception is not swallowed in the event an exception (such as hover validation failure) occurs, and it fails fast. In the catch, we will explicitly throw the error

satyakigh · 2026-04-01T16:54:11Z

tools/long-running/Templates.ts

+
+const cache = new Map<string, string>();
+
+const loadTemplate = (name: string): string => {


The template utils from src can be used directly

Switched to using Templates const from the tst/utils

satyakigh · 2026-04-01T16:54:52Z

tools/long-running/TestOrchestrator.ts

+        console.log('Initialization complete');
+    }
+
+    private async waitForSystemReadiness(): Promise<void> {


The system is "ready" but the orchestrator is "initializing/initialized"

Modified verbiage, not using "ready", only initializing/initialized

satyakigh · 2026-04-01T16:55:44Z

Lots of references to "long-running" the caller of the main file defines the duration

chrisqm-dev · 2026-04-02T15:25:02Z

Our folder and file names are usually camelCased unless its a bin/cli command file

Fixed folder and file names so that folders are camelCased and files are PascalCased, which matches the styling in the rest of the package

chrisqm-dev · 2026-04-02T16:40:07Z

Lots of references to "long-running" the caller of the main file defines the duration

Changed to "Stability" tests instead of calling it LongRunning

chrisqm-dev requested a review from a team as a code owner March 31, 2026 21:53

chrisqm-dev commented Mar 31, 2026

View reviewed changes

satyakigh reviewed Apr 1, 2026

View reviewed changes

chrisqm-dev added 2 commits April 2, 2026 14:30

Add LSP Client and long running tests in tools

b68fc4f

Address comments

2ddb466

chrisqm-dev force-pushed the long-running branch from eed185f to 2ddb466 Compare April 2, 2026 18:31

chrisqm-dev added 2 commits April 3, 2026 12:39

Modify LspConnection inteface

9852f10

Merge branch 'main' into long-running

562aac8

		@@ -0,0 +1,49 @@
		import { InitializeParams } from 'vscode-languageserver-protocol';

		@@ -0,0 +1,49 @@
		import { InitializeParams } from 'vscode-languageserver-protocol';

		export interface LspClientConfig {


		const cache = new Map<string, string>();

		const loadTemplate = (name: string): string => {

Conversation

chrisqm-dev commented Mar 31, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

satyakigh commented Apr 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chrisqm-dev Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

satyakigh commented Apr 1, 2026

Uh oh!

chrisqm-dev commented Apr 2, 2026

Uh oh!

chrisqm-dev commented Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chrisqm-dev Apr 2, 2026 •

edited

Loading