diff --git a/README.md b/README.md index e5ac77871..278bc6518 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ and [OpenAPI 3.1.x](https://github.com/OAI/OpenAPI-Specification/blob/master/ver - [outputFormat](#outputformat) - [ruleset](#ruleset) - [summaryOnly](#summaryonly) - - [produceImpactScore](#produceimpactscore) + - [produceQualityScore](#producequalityscore) - [markdownReport](#markdownreport) * [Programmatic Usage](#programmatic-usage) - [Validator Output](#validator-output) @@ -176,7 +176,7 @@ Options: -n, --no-colors disable colorizing of the output (default is false) -r, --ruleset use Spectral ruleset contained in `` ("default" forces use of default IBM Cloud Validation Ruleset) -s, --summary-only include only the summary information and skip individual errors and warnings (default is false) - -q, --impact-score compute scores representing the API impact of rule violations and include with the results (default is false) + -q, --quality-score compute scores representing the API quality of rule violations and include with the results (default is false) -m, --markdown-report generate a Markdown file with a report on all validator results (default is false) -w, --warnings-limit set warnings limit to (default is -1) --version output the version number @@ -701,7 +701,7 @@ module.exports = { -##### produceImpactScore +##### produceQualityScore @@ -709,13 +709,13 @@ module.exports = { - @@ -734,20 +734,20 @@ for more information about the purpose of these scores and how they are computed @@ -765,7 +765,7 @@ module.exports = {
Default
The produceImpactScore configuration property corresponds to the --q/--impact-score command-line option. If set to true, +The produceQualityScore configuration property corresponds to the +-q/--quality-score command-line option. If set to true, the validator will, in addition to reporting individual rule violations, use the -rule violation data to produce API impact scores based on the categories of usability, +rule violation data to produce API quality scores based on the categories of usability, security, robustness, and cost of evolution. By default, the data demonstrating how the scores are calculated from each rule is displayed. If this option is combined with -the "summary only" configuration option, only the categorized impact scores are displayed. +the "summary only" configuration option, only the categorized quality scores are displayed. These scores are useful for "Automated Quality Screening". See [this documentation](docs/automated-quality-screening.md) for more information about the purpose of these scores and how they are computed.
-produceImpactScore: true
+produceQualityScore: true
 
 {
-  "produceImpactScore": true
+  "produceQualityScore": true
 }
 
 module.exports = {
-  produceImpactScore: true
+  produceQualityScore: true
 };
 
The markdownReport configuration property corresponds to the -m/--markdown-report command-line option. If set to true, the validator will generate a Markdown file containing a report on all of the validator results, -including the individual rule violations, the impact scores, and the data used to compute the impact scores. +including the individual rule violations, the quality scores, and the data used to compute the quality scores. It provides a single location to see all of the information produced by the validator. A default filename is always used and is based on the name of the API definition file provided to the validator. If a file of the same name already exists, it will be overwritten by default. This is because the primary use-case @@ -881,9 +881,9 @@ warnings, and finally a summary section. - The `-s`/`--summary-only` command-line option or the `summaryOnly` configuration property causes only the summary to be displayed. - The `-e`/`--errors-only` option or `errorsOnly` configuration property causes only error-level violations to be displayed. -- The `-q`/`--impact-score` option or `produceImpactScore` configuration property causes the validator to show aggregated impact scores. See the example below: +- The `-q`/`--quality-score` option or `produceQualityScore` configuration property causes the validator to show aggregated quality scores. See the example below: -Example of impact score tables that are appended to the standard output: +Example of quality score tables that are appended to the standard output: ``` ┌────────────────┬───────────┐ @@ -896,7 +896,7 @@ Example of impact score tables that are appended to the standard output: │ overall (mean) │ 91 /100 │ └────────────────┴───────────┘ ┌──────────────────────────────┬───────┬─────────────────┬──────────────────┬─────────────────┬───────────────────┬──────────────────┬────────────┐ -│ rule │ count │ func │ usability impact │ security impact │ robustness impact │ evolution impact │ rule total │ +│ rule │ count │ func │ usability quality │ security quality │ robustness quality │ evolution quality │ rule total │ ├──────────────────────────────┼───────┼─────────────────┼──────────────────┼─────────────────┼───────────────────┼──────────────────┼────────────┤ │ operation-operationId-unique │ 1 │ 1×3÷operations │ 1 │ │ 2 │ 3 │ 6 │ │ ibm-no-array-responses │ 2 │ 2×10÷operations │ │ │ │ 20 │ 20 │ @@ -908,7 +908,7 @@ Example of impact score tables that are appended to the standard output: When displaying JSON output, the validator will produce a JSON object which complies with [this JSON schema](packages/validator/src/schemas/results-object.yaml). The JSON data will include information about all rule violations, -as well as all impact score information computed from the rule violations. +as well as all quality score information computed from the rule violations. Here is an example of JSON output: ```json @@ -987,7 +987,7 @@ Here is an example of JSON output: } }, "hasResults": true - "impactScore": { + "qualityScore": { "categorizedSummary": { "usability": 94, "security": 100, @@ -1020,7 +1020,7 @@ Here is an example of JSON output: ``` The JSON output is also affected by the `-s`/`--summary-only` and `-e`/`--errors-only` options as well as the `summaryOnly` and `errorsOnly` -configuration properties. It is _not_ affected by the `-q`/`--impact-score` option or `produceImpactScore` property. +configuration properties. It is _not_ affected by the `-q`/`--quality-score` option or `produceQualityScore` property. ## Logging diff --git a/docs/automated-quality-screening.md b/docs/automated-quality-screening.md index 841cd9942..034ac6529 100644 --- a/docs/automated-quality-screening.md +++ b/docs/automated-quality-screening.md @@ -1,11 +1,11 @@ # Automated Quality Screening The validator provides Automated Quality Screening (AQS) scores via the `-q` (or -`--impact-score`) flag. These scores help you evaluate risk and make decisions about investing in +`--quality-score`) flag. These scores help you evaluate risk and make decisions about investing in the quality of your service's API. AQS scores are not a substitute for expert review, but minimum AQS scores may be a prerequisite to a review. -AQS scores give you insight about four impact dimensions for API quality: usability, security, +AQS scores give you insight about four quality dimensions for API quality: usability, security, robustness, and cost of evolution. Each dimension will be scored from 0 to 100, and the overall score will be the average (mean) of the four individual scores. @@ -23,26 +23,26 @@ of a recalibration. Recalibrations will occur as a part of minor (but not patch) validator. Findings (errors and warnings) produced by each rule of the validator will reduce the AQS score for -one or more impact dimensions. The size of the reduction is determined by the number of findings for +one or more quality dimensions. The size of the reduction is determined by the number of findings for the rule and a qualitative weight assigned to the rule, scaled to the overall size of the API. -For each impact dimension, there is distinct criteria for assigning impact: +For each quality dimension, there is distinct criteria for assigning impact: - **Usability** is the broadest category, as most problems with an API will manifest as (at least) usability problems. Anything from stylistic deviation from standards (for example `camelCase` vs. `snake_case`) to underspecified behavior has a usability impact. -- **Security** impact can come from clear deviations from best practices within an API design, but +- **Security** quality can come from clear deviations from best practices within an API design, but very often comes from undefined or excessively permissive validation constraints on values handled in an API. While some constraints may be enforced by a service implementation without being defined in its API definition, it must be assumed that they are not. Overly permissive constraints can indicate a susceptibility to code-injection or denial-of-service attacks. -- **Robustness** impact comes from undefined behavior that a client may rely on, API design likely +- **Robustness** quality comes from undefined behavior that a client may rely on, API design likely to be misunderstood or misapplied, and missing robustness features such as optimistic locking that may be needed to mitigate race conditions for certain kinds of requests. -- **Evolution** impact comes from designing or specifying an API in such a way that it could be +- **Evolution** quality comes from designing or specifying an API in such a way that it could be difficult, expensive, or impossible to provide backward compatibility when certain kinds of new features are added to a service. -Note that rules that identify undefined behavior often have an impact across all four impact +Note that rules that identify undefined behavior often have an impact across all four quality dimensions. diff --git a/packages/validator/README.md b/packages/validator/README.md index 4bc050282..7862c654e 100644 --- a/packages/validator/README.md +++ b/packages/validator/README.md @@ -25,7 +25,7 @@ Options: -n, --no-colors disable colorizing of the output (default is false) -r, --ruleset use Spectral ruleset contained in `` ("default" forces use of default IBM Cloud Validation Ruleset) -s, --summary-only include only the summary information and skip individual errors and warnings (default is false) - -q, --impact-score compute scores representing the API impact of rule violations and include with the results (default is false) + -q, --quality-score compute scores representing the API quality of rule violations and include with the results (default is false) -m, --markdown-report generate a Markdown file with a report on all validator results (default is false) -w, --warnings-limit set warnings limit to (default is -1) --version output the version number @@ -38,4 +38,4 @@ Again, this page displays abbreviated information. The following links may be he - [Detailed information about the configuration options](../../README.md#configuration) - [Detailed information about the default ruleset](../../docs/ibm-cloud-rules.md) -- [Detailed information about the `--impact-score` feature](../../docs/automated-quality-screening.md) +- [Detailed information about the `--quality-score` feature](../../docs/automated-quality-screening.md) diff --git a/packages/validator/src/cli-validator/run-validator.js b/packages/validator/src/cli-validator/run-validator.js index d4c923e96..52dd657a0 100644 --- a/packages/validator/src/cli-validator/run-validator.js +++ b/packages/validator/src/cli-validator/run-validator.js @@ -25,7 +25,7 @@ const { } = require('./utils'); const { runSpectral } = require('../spectral'); -const { produceImpactScore, printScoreTables } = require('../scoring-tool'); +const { produceQualityScore, printScoreTables } = require('../scoring-tool'); const { printMarkdownReport } = require('../markdown-report'); let logger; @@ -157,11 +157,11 @@ async function runValidator(cliArgs, parseOptions = {}) { return Promise.reject(2); } - // If multiple files were specified and the impact score is requested, exit with an error. + // If multiple files were specified and the quality score is requested, exit with an error. // We could change this behavior in the future. - if (filesToValidate.length > 1 && context.config.produceImpactScore) { + if (filesToValidate.length > 1 && context.config.produceQualityScore) { logger.error( - 'At most one file can be specified when the impact score is requested.' + 'At most one file can be specified when the quality score is requested.' ); return Promise.reject(2); } @@ -238,30 +238,30 @@ async function runValidator(cliArgs, parseOptions = {}) { continue; } - // Compute scoring information if 1) the user requested the "impact score" + // Compute scoring information if 1) the user requested the "quality score" // option, 2) if JSON output is requested, or 3) if the markdown report is // requested. The JSON output and markdown report always include all results, // including the standard rule violations and the scoring information. - let impactScoreInformation = {}; + let qualityScoreInformation = {}; if ( - context.config.produceImpactScore || + context.config.produceQualityScore || outputIsJSON(context) || context.config.markdownReport ) { - logger.info('Impact scores are being calculated...'); - impactScoreInformation = await produceImpactScore(results, context); + logger.info('Quality scores are being calculated...'); + qualityScoreInformation = await produceQualityScore(results, context); } else { logger.info( - 'Impact scores are not being calculated. Scores are calculated when' + + 'Quality scores are not being calculated. Scores are calculated when' + 'requested, or when JSON output or a Markdown report is requested.' ); } - // Combine validator and impact score results into one object. + // Combine validator and quality score results into one object. results = { ...results, - impactScore: { - ...impactScoreInformation, + qualityScore: { + ...qualityScoreInformation, }, }; @@ -293,7 +293,7 @@ async function runValidator(cliArgs, parseOptions = {}) { // If summary output is requested, filter out extraneous information here. if (context.config.summaryOnly) { // Remove verbose scoring data. - results.impactScore.scoringData = []; + results.qualityScore.scoringData = []; // Remove individual rule violation results. ['error', 'warning', 'info', 'hint'].forEach(sev => { @@ -307,9 +307,9 @@ async function runValidator(cliArgs, parseOptions = {}) { } else if (results.hasResults) { printResults(context, results); - // If the user requested the "impact score" option, print + // If the user requested the "quality score" option, print // the scoring tables in addition to the standard output. - if (context.config.produceImpactScore) { + if (context.config.produceQualityScore) { printScoreTables(context, results); } } else { diff --git a/packages/validator/src/cli-validator/utils/cli-options.js b/packages/validator/src/cli-validator/utils/cli-options.js index e8f48abfb..fc17f805b 100644 --- a/packages/validator/src/cli-validator/utils/cli-options.js +++ b/packages/validator/src/cli-validator/utils/cli-options.js @@ -76,12 +76,12 @@ function createCLIOptions() { 'include only the summary information and skip individual errors and warnings (default is false)' ) .option( - '-q, --impact-score', - 'compute scores representing the API impact of rule violations and include with the results' + '-q, --quality-score', + 'compute scores representing the API quality of rule violations and include with the results' ) .option( '-m, --markdown-report', - 'write a markdown report to a file, consisting of rule violations and impact scores' + 'write a markdown report to a file, consisting of rule violations and quality scores' ) .option( '-w, --warnings-limit ', diff --git a/packages/validator/src/cli-validator/utils/configuration-manager.js b/packages/validator/src/cli-validator/utils/configuration-manager.js index d40e82453..c7474e499 100644 --- a/packages/validator/src/cli-validator/utils/configuration-manager.js +++ b/packages/validator/src/cli-validator/utils/configuration-manager.js @@ -42,7 +42,7 @@ const defaultConfig = { outputFormat: 'text', ruleset: null, summaryOnly: false, - produceImpactScore: false, + produceQualityScore: false, markdownReport: false, }; @@ -238,8 +238,8 @@ async function processArgs(args, cliParseOptions) { configObj.limits.warnings = opts.warningsLimit; } - if ('impactScore' in opts) { - configObj.produceImpactScore = true; + if ('qualityScore' in opts) { + configObj.produceQualityScore = true; } if ('markdownReport' in opts) { diff --git a/packages/validator/src/markdown-report/report.js b/packages/validator/src/markdown-report/report.js index 71071ab2a..4a02cf7d8 100644 --- a/packages/validator/src/markdown-report/report.js +++ b/packages/validator/src/markdown-report/report.js @@ -20,7 +20,7 @@ Version: ${apiDefinition.info.version} ## Quick view ${primary(results)} -The API impact score, also known as the "Automated Quality Screening" score, is calculated +The API quality score, also known as the "Automated Quality Screening" score, is calculated by the IBM OpenAPI Validator to help users understand the impact of the rule violations reported by the validator. The scores are designed to help users evaluate risk and make decisions about investing in the quality of their service's API. @@ -29,7 +29,7 @@ For more information, see [the AQS documentation](https://github.com/IBM/openapi ## Breakdown by category ${categorizedScores(results)} -The "overall" impact score is the average (mean) of the categorized scores. The categorized scores are +The "overall" quality score is the average (mean) of the categorized scores. The categorized scores are inherently weighted by the scoring algorithm, so that security violations are 5 times as severe as usability violations, evolution 3 times, and robustness 2 times. diff --git a/packages/validator/src/markdown-report/tables/categorized-scores.js b/packages/validator/src/markdown-report/tables/categorized-scores.js index c0658553c..5f67d3ee1 100644 --- a/packages/validator/src/markdown-report/tables/categorized-scores.js +++ b/packages/validator/src/markdown-report/tables/categorized-scores.js @@ -5,9 +5,9 @@ const MarkdownTable = require('../markdown-table'); -function getTable({ impactScore }) { - const { categorizedSummary } = impactScore; - const table = new MarkdownTable('Category', 'Impact Score'); +function getTable({ qualityScore }) { + const { categorizedSummary } = qualityScore; + const table = new MarkdownTable('Category', 'Quality Score'); for (const [category, score] of Object.entries(categorizedSummary)) { // Bold the "overall" score. diff --git a/packages/validator/src/markdown-report/tables/primary.js b/packages/validator/src/markdown-report/tables/primary.js index 61eab29fd..3fb8c89ae 100644 --- a/packages/validator/src/markdown-report/tables/primary.js +++ b/packages/validator/src/markdown-report/tables/primary.js @@ -5,15 +5,15 @@ const MarkdownTable = require('../markdown-table'); -function getTable({ impactScore, error, warning }) { +function getTable({ qualityScore, error, warning }) { const table = new MarkdownTable( - 'Impact Score', + 'Quality Score', 'Error Count', 'Warning Count' ); table.addRow( - `${impactScore.categorizedSummary.overall} / 100`, + `${qualityScore.categorizedSummary.overall} / 100`, error.summary.total, warning.summary.total ); diff --git a/packages/validator/src/markdown-report/tables/scoring-data.js b/packages/validator/src/markdown-report/tables/scoring-data.js index 35574c989..bf1daf311 100644 --- a/packages/validator/src/markdown-report/tables/scoring-data.js +++ b/packages/validator/src/markdown-report/tables/scoring-data.js @@ -5,16 +5,16 @@ const MarkdownTable = require('../markdown-table'); -function getTable({ impactScore }) { - const { scoringData } = impactScore; +function getTable({ qualityScore }) { + const { scoringData } = qualityScore; const table = new MarkdownTable( 'Rule', 'Count', 'Func', - 'Usability Impact', - 'Security Impact', - 'Robustness Impact', - 'Evolution Impact', + 'Usability Quality', + 'Security Quality', + 'Robustness Quality', + 'Evolution Quality', 'Rule Total' ); diff --git a/packages/validator/src/schemas/config-file.yaml b/packages/validator/src/schemas/config-file.yaml index 1c9c35daa..852213116 100644 --- a/packages/validator/src/schemas/config-file.yaml +++ b/packages/validator/src/schemas/config-file.yaml @@ -72,13 +72,13 @@ properties: A flag that causes the validator to display only the summary in the output type: boolean default: false - produceImpactScore: + produceQualityScore: description: > - A flag that causes the validator to display an aggregated score based on the API impact of rule violations + A flag that causes the validator to display an aggregated score based on the API quality of rule violations type: boolean default: false markdownReport: description: > - A flag that causes the validator to generate a report with rule violations and impact scores as a Markdown file + A flag that causes the validator to generate a report with rule violations and quality scores as a Markdown file type: boolean default: false diff --git a/packages/validator/src/schemas/results-object.yaml b/packages/validator/src/schemas/results-object.yaml index 168e1a41b..d398b0094 100644 --- a/packages/validator/src/schemas/results-object.yaml +++ b/packages/validator/src/schemas/results-object.yaml @@ -21,7 +21,7 @@ properties: hasResults: type: boolean description: A convenience flag indicating there is at least one result at any severity level - impactScore: + qualityScore: type: object description: Data translating the validation results into numeric scores required: @@ -30,7 +30,7 @@ properties: properties: categorizedSummary: type: object - description: Holds the max score for each API impact category + description: Holds the max score for each API quality category required: # Must match the category enum values in the "rubric-entry" schema. - usability @@ -75,7 +75,7 @@ properties: description: A string representation of the formula used to compute the demerits demerits: type: object - description: The categorized, numeric impact a rule violation has on the score + description: The categorized, numeric quality a rule violation has on the score # Must match the category enum values in the "rubric-entry" schema. properties: usability: @@ -162,10 +162,10 @@ $defs: description: The percentage of total violations for a given severity that this rule comprises MaxScore: type: integer - description: A number describing the maximum score for an API after accounting for the impact of rule violations + description: A number describing the maximum score for an API after accounting for the quality of rule violations minimum: 0 maximum: 100 Demerit: type: number - description: A number describing the demerit impact a rule has on an API + description: A number describing the demerit quality a rule has on an API minimum: 0.01 diff --git a/packages/validator/src/scoring-tool/index.js b/packages/validator/src/scoring-tool/index.js index 14de1ac97..3c2898a1d 100644 --- a/packages/validator/src/scoring-tool/index.js +++ b/packages/validator/src/scoring-tool/index.js @@ -10,7 +10,10 @@ const { const { scoreResults } = require('./score'); const { computeMetrics } = require('./compute-metrics'); -async function produceImpactScore(validatorResults, { apiDefinition, logger }) { +async function produceQualityScore( + validatorResults, + { apiDefinition, logger } +) { const metrics = await computeMetrics(apiDefinition); logger.debug(`API scaling metrics: ${metrics.toString()}`); @@ -18,11 +21,11 @@ async function produceImpactScore(validatorResults, { apiDefinition, logger }) { return scoreResults(validatorResults, metrics, logger); } -function printScoreTables({ config }, { impactScore }) { - printCategorizedScoresTable(impactScore); +function printScoreTables({ config }, { qualityScore }) { + printCategorizedScoresTable(qualityScore); if (!config.summaryOnly) { - printScoringDataTable(impactScore); + printScoringDataTable(qualityScore); } // Print a new line at the end to be consistent with other validator output. @@ -30,6 +33,6 @@ function printScoreTables({ config }, { impactScore }) { } module.exports = { - produceImpactScore, + produceQualityScore, printScoreTables, }; diff --git a/packages/validator/src/scoring-tool/output.js b/packages/validator/src/scoring-tool/output.js index 3212e3eb5..4e22b610e 100644 --- a/packages/validator/src/scoring-tool/output.js +++ b/packages/validator/src/scoring-tool/output.js @@ -30,7 +30,7 @@ function printCategorizedScoresTable({ categorizedSummary }) { } // Print out a table with all of the rule-based data used to compute the score. -// rule │ count │ func │ usability impact │ security impact │ robustness impact │ evolution impact │ total +// rule │ count │ func │ usability quality │ security quality │ robustness quality │ evolution quality │ total function printScoringDataTable({ scoringData }) { const table = new Table(); @@ -44,7 +44,7 @@ function printScoringDataTable({ scoringData }) { }; getCategories().forEach(c => { - row[`${c} impact`] = sd.demerits[c]; + row[`${c} quality`] = sd.demerits[c]; }); row['rule total'] = sd.demerits.total; diff --git a/packages/validator/src/scoring-tool/score.js b/packages/validator/src/scoring-tool/score.js index 822c0e3e5..a0143e412 100644 --- a/packages/validator/src/scoring-tool/score.js +++ b/packages/validator/src/scoring-tool/score.js @@ -8,7 +8,7 @@ const rubric = require('./rubric'); /** * Uses the validator results to calculate categorized demerits for each rule - * violation, which are then used to compute categorized impact scores for the API. + * violation, which are then used to compute categorized quality scores for the API. * * @param object result - the validator results * @param Metrics metrics - the metric tracking class instance for scaling rule violations @@ -46,7 +46,7 @@ function scoreResults(result, metrics, logger) { demeritSumsByCategory[category] += demerit; } - // After adding category totals, we can calculate the total impact for the given rule. + // After adding category totals, we can calculate the total quality score for the given rule. data.demerits.total = includeDecimals( Object.values(data.demerits).reduce((sum, num) => sum + num, 0.0), 2 diff --git a/packages/validator/test/cli-validator/mock-files/config/config1.yaml b/packages/validator/test/cli-validator/mock-files/config/config1.yaml index 76b74354c..29d54110b 100644 --- a/packages/validator/test/cli-validator/mock-files/config/config1.yaml +++ b/packages/validator/test/cli-validator/mock-files/config/config1.yaml @@ -12,4 +12,4 @@ logLevels: 'ibm-schema-description-exists': 'debug' outputFormat: 'text' summaryOnly: false -produceImpactScore: false +produceQualityScore: false diff --git a/packages/validator/test/cli-validator/tests/configuration-manager.test.js b/packages/validator/test/cli-validator/tests/configuration-manager.test.js index 2e0ae4a39..37f332ab2 100644 --- a/packages/validator/test/cli-validator/tests/configuration-manager.test.js +++ b/packages/validator/test/cli-validator/tests/configuration-manager.test.js @@ -47,7 +47,7 @@ describe('Configuration Manager tests', function () { expect(defaultConfig.outputFormat).toBe('text'); expect(defaultConfig.ruleset).toBe(null); expect(defaultConfig.summaryOnly).toBe(false); - expect(defaultConfig.produceImpactScore).toBe(false); + expect(defaultConfig.produceQualityScore).toBe(false); expect(defaultConfig.markdownReport).toBe(false); }); }); @@ -189,7 +189,7 @@ describe('Configuration Manager tests', function () { }, outputFormat: 'text', summaryOnly: false, - produceImpactScore: false, + produceQualityScore: false, markdownReport: false, ruleset: null, }); @@ -213,7 +213,7 @@ describe('Configuration Manager tests', function () { '--summary-only', '--warnings-limit', '-1', - '--impact-score', + '--quality-score', '--markdown-report', ], cliParseOptions @@ -235,7 +235,7 @@ describe('Configuration Manager tests', function () { }, ruleset: 'my-rules.yml', summaryOnly: true, - produceImpactScore: true, + produceQualityScore: true, markdownReport: true, }); }); @@ -274,7 +274,7 @@ describe('Configuration Manager tests', function () { }, outputFormat: 'text', summaryOnly: false, - produceImpactScore: false, + produceQualityScore: false, markdownReport: false, }; @@ -410,11 +410,11 @@ describe('Configuration Manager tests', function () { } ); - it.each(['-q', '--impact-score'])( - `should produce correct config with -q/--impact-score option`, + it.each(['-q', '--quality-score'])( + `should produce correct config with -q/--quality-score option`, async function (option) { const expectedConfig = { - produceImpactScore: true, + produceQualityScore: true, }; const { context } = await processArgs([option], cliParseOptions); diff --git a/packages/validator/test/cli-validator/tests/option-handling.test.js b/packages/validator/test/cli-validator/tests/option-handling.test.js index d936ec29d..bb90f007f 100644 --- a/packages/validator/test/cli-validator/tests/option-handling.test.js +++ b/packages/validator/test/cli-validator/tests/option-handling.test.js @@ -197,19 +197,21 @@ describe('cli tool - test option handling', function () { 'Every operation must have unique "operationId".' ); - // Representative sample of the impact score information. + // Representative sample of the quality score information. expect( - outputObject.impactScore.categorizedSummary.usability + outputObject.qualityScore.categorizedSummary.usability ).toBeTruthy(); - expect(outputObject.impactScore.categorizedSummary.security).toBeTruthy(); expect( - outputObject.impactScore.categorizedSummary.robustness + outputObject.qualityScore.categorizedSummary.security ).toBeTruthy(); expect( - outputObject.impactScore.categorizedSummary.evolution + outputObject.qualityScore.categorizedSummary.robustness ).toBeTruthy(); - expect(outputObject.impactScore.categorizedSummary.overall).toBeTruthy(); - expect(outputObject.impactScore.scoringData.length).toBeGreaterThan(0); + expect( + outputObject.qualityScore.categorizedSummary.evolution + ).toBeTruthy(); + expect(outputObject.qualityScore.categorizedSummary.overall).toBeTruthy(); + expect(outputObject.qualityScore.scoringData.length).toBeGreaterThan(0); // json output should comply with written schema const outputSchemaPath = @@ -256,8 +258,8 @@ describe('cli tool - test option handling', function () { expect(outputObject.warning.summary.total).toBeGreaterThan(0); }); - it.each(['-q', '--impact-score'])( - 'should print two scoring tables when the -q/--impact-score option is specified', + it.each(['-q', '--quality-score'])( + 'should print two scoring tables when the -q/--quality-score option is specified', async function (option) { await testValidator([ option, @@ -282,10 +284,10 @@ describe('cli tool - test option handling', function () { 'rule', 'count', 'func', - 'usability impact', - 'security impact', - 'robustness impact', - 'evolution impact', + 'usability quality', + 'security quality', + 'robustness quality', + 'evolution quality', 'rule total', ]); } diff --git a/packages/validator/test/markdown-report/tables/categorized-scores.test.js b/packages/validator/test/markdown-report/tables/categorized-scores.test.js index 46b0e5408..9020ab39b 100644 --- a/packages/validator/test/markdown-report/tables/categorized-scores.test.js +++ b/packages/validator/test/markdown-report/tables/categorized-scores.test.js @@ -11,7 +11,7 @@ describe('categorizedScores table tests', function () { const tableRows = categorizedScores(validatorResults).split('\n'); expect(tableRows).toHaveLength(7); - expect(tableRows[0]).toBe('| Category | Impact Score |'); + expect(tableRows[0]).toBe('| Category | Quality Score |'); expect(tableRows[1]).toBe('| --- | --- |'); expect(tableRows[2]).toBe('| usability | 94 / 100 |'); expect(tableRows[3]).toBe('| security | 96 / 100 |'); diff --git a/packages/validator/test/markdown-report/tables/primary.test.js b/packages/validator/test/markdown-report/tables/primary.test.js index 3d3238639..124f88955 100644 --- a/packages/validator/test/markdown-report/tables/primary.test.js +++ b/packages/validator/test/markdown-report/tables/primary.test.js @@ -11,7 +11,9 @@ describe('primary table tests', function () { const tableRows = primary(validatorResults).split('\n'); expect(tableRows).toHaveLength(3); - expect(tableRows[0]).toBe('| Impact Score | Error Count | Warning Count |'); + expect(tableRows[0]).toBe( + '| Quality Score | Error Count | Warning Count |' + ); expect(tableRows[1]).toBe('| --- | --- | --- |'); expect(tableRows[2]).toBe('| 96 / 100 | 2 | 1 |'); }); diff --git a/packages/validator/test/markdown-report/tables/scoring-data.test.js b/packages/validator/test/markdown-report/tables/scoring-data.test.js index 91d943498..123282f3a 100644 --- a/packages/validator/test/markdown-report/tables/scoring-data.test.js +++ b/packages/validator/test/markdown-report/tables/scoring-data.test.js @@ -12,7 +12,7 @@ describe('scoringData table tests', function () { expect(tableRows).toHaveLength(5); expect(tableRows[0]).toBe( - '| Rule | Count | Func | Usability Impact | Security Impact | Robustness Impact | Evolution Impact | Rule Total |' + '| Rule | Count | Func | Usability Quality | Security Quality | Robustness Quality | Evolution Quality | Rule Total |' ); expect(tableRows[1]).toBe( '| --- | --- | --- | --- | --- | --- | --- | --- |' diff --git a/packages/validator/test/scoring-tool/output.test.js b/packages/validator/test/scoring-tool/output.test.js index 4ef55058b..c2a4697fb 100644 --- a/packages/validator/test/scoring-tool/output.test.js +++ b/packages/validator/test/scoring-tool/output.test.js @@ -53,10 +53,10 @@ describe('scoring-tool output tests', function () { 'rule', 'count', 'func', - 'usability impact', - 'security impact', - 'robustness impact', - 'evolution impact', + 'usability quality', + 'security quality', + 'robustness quality', + 'evolution quality', 'rule total', ]); diff --git a/packages/validator/test/test-utils/mock-json-output.json b/packages/validator/test/test-utils/mock-json-output.json index e99aa8fcb..6e264c798 100644 --- a/packages/validator/test/test-utils/mock-json-output.json +++ b/packages/validator/test/test-utils/mock-json-output.json @@ -81,7 +81,7 @@ } }, "hasResults": true, - "impactScore": { + "qualityScore": { "categorizedSummary": { "usability": 94, "security": 96,