From 79c33baa3274fc28197c6afac5c1174f0c615be4 Mon Sep 17 00:00:00 2001 From: Ezequiel <73295972+iw-ezequiel@users.noreply.github.com> Date: Mon, 9 Aug 2021 12:55:07 +0100 Subject: [PATCH 1/5] Clarification/context Some added context and clarification around the definitions of green/amber --- quality-checks.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/quality-checks.md b/quality-checks.md index 5089a93d..a05b64d4 100644 --- a/quality-checks.md +++ b/quality-checks.md @@ -26,10 +26,12 @@ All of these checks are important, even where their purpose overlaps with other We rate our applications against each of these checks as follows: * Green = the quality check is applied frequently and consistently (in practice this typically means [automated](./patterns/automate-everything.md) via [continuous integration / continuous deployment](./practices/continuous-integration.md)), the output of the check is a quality gate (as opposed to just a warning / for information), and the tolerances for that quality gate (e.g. code coverage %) are agreed and understood -* Amber = the quality check is applied, but not all conditions for green are met - for example the check generates warnings that may or may not be acted on +* Amber = the quality check is applied, but not all conditions for green are met - for example the check generates warnings that may or may not be acted on, or is executed on an ad-hoc basis and cannot offer a consistent quality gate guarantee * Red = the quality check is not applied * N/A = the quality check is not applicable +There may be further nuances to these definitions, and the sections detailing each check can expand considerably on the criteria as required. Everyone is encouraged to contribute to those, and discuss how they can be best be tailored. + ## Tracking progress We recommend tracking progress on an Engineering Quality dashboard, for example: From 6a02d8b8f4f7580b92963e72016c8364348a4d17 Mon Sep 17 00:00:00 2001 From: iw-ezequiel Date: Wed, 11 Aug 2021 09:47:10 +0100 Subject: [PATCH 2/5] Further definitions of green/amber --- quality-checks.md | 44 ++++++++++++++++++++++---------------------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/quality-checks.md b/quality-checks.md index a05b64d4..c448cf53 100644 --- a/quality-checks.md +++ b/quality-checks.md @@ -40,25 +40,25 @@ We recommend tracking progress on an Engineering Quality dashboard, for example: ## Details -| Quality check | Classification | Applicability | What it means | Why we care | Tolerances for green | Endorsed tools / configuration | Further details | -| :--------------------- | :-------------- | :------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------ | -| Unit tests | Functionality | Universal | Logic tests for individual blocks of code, e.g. individual methods | This is the quickest (to execute) type of functional test, so these are essential to achieve both rapid and thorough functional testing | Builds fail if any tests fail | - | [Test practices](./practices/testing.md) | -| Integration tests | Functionality | Universal | Check interactions with other components and dependant systems. e.g. across microservices, authentication layers, database, third-party systems. Ideally includes full end-to-end testing across all components | When components are developed in isolation, it's vital that we regularly test them working together. Changes in one component can break the whole system | Builds fail if any tests fail | | | -| API / contract tests | Functionality | Contextual | Check whether the API interface adheres to the agreed contract | Any API interface is an integration point with another component or a software systems. An extra care has to be taken to ensure compatibility and stability of that integration are maintained so that we don't break applications that depend on our APIs | Builds fail if any tests fail | Postman | [Automate Your API Tests with Postman](https://www.postman.com/use-cases/api-testing-automation/) | -| UI tests | Functionality | Contextual | Check that the user interface components behave as expected, particularly checking the visual elements to verify that they are functioning according to requirements | As the only aspects of software that end users come into contact with, it is essential that these elements behave as expected and allow users to get only what they need from our software applications | Builds fail if any tests fail | | | -| Secret scanning | Security | Universal | Check for secrets (e.g. passwords, IP addresses, etc) accidentally included in software code | This protects us against accidentally leaking secrets (in source code) which could compromise the security of the application | Review the list of patterns and update it as necessary

... then:

Full repository (including history) scan, and all secrets removed

And:

Local (on developer laptop) scanning to block commits containing the patterns

And:

Server-side scanning within the code repository for new commits containing the patterns| [Security practices](/practices/security.md#secret-scanning) | | -| Security code analysis | Security | Universal | Check for indications of possible security issues (for example injection weaknesses) | This gives fast feedback about security issues: it's not as thorough as security testing, but it's much quicker to execute, so both are important to achieve both rapid and thorough security testing | If using SonarQube, must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met | One option is [SonarQube](tools/sonarqube.md). For the purpose of security code analysis, Developer Edition or higher is required as it includes advanced OWASP scanning. | | -| Security testing | Security | Contextual | Check for security issues (for example injection weaknesses) | More thorough than security code scanning, but much slower to execute, so both are important to achieve both rapid and thorough security testing | | | | -| Dependency scanning | Security | Universal | Check for security issues and vulnerabilities in dependent areas of code that are outside of our direct control | Without this we have no way of knowing of any issues or security vulnerabilities of third party components that we are not responsible for | | | | -| Performance tests | Resilience | Contextual | Check whether application performance is acceptable at different levels of load | Without this test, we don't know how load will affect the performance of the application | | | | -| Capacity tests | Resilience | Contextual | Identify the application's breaking point in terms of heavy load | Without this test, we don't know how much load the application can handle before the application breaks | | | | -| Stress tests | Resilience | Contextual | Check whether sudden spikes in load cause a problem | Without this test, we don't know if the application will fail under a sharp increase in load | | | | -| Soak tests | Resilience | Contextual | Check whether sustained heavy load causes a problem | Without this test, we don't know if application performance will suffer under prolonged heavy load | | | | -| Chaos tests | Resilience | Contextual | Cause failures in a system to test the resiliency of that system and its environment, and our ability to respond to failures | Give the team confidence that failures in a given environment will not lead to unplanned downtime or a negative user experience

Ensures that the team has visibility (e.g. dashboards and alerts) to be able to identify issues

Help the team to understand their mean time to recovery (MTTR) and to build muscle memory & confidence for recovery activities | Regular (at least every couple of months) game days, and:

Builds fail if any test fails - note, these tests are slow, and are likely to be part of an infrequently-triggered (e.g. overnight) build

The tests cover whether the system self-heals, auto-scales, and alerts as expected | | | -| Code coverage | Maintainability | Universal | The proportion of the application code which is executed (in this context: during testing) | The higher the code coverage, the more thorough the testing, and therefore the higher the likelihood of detecting functional issues early | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway test coverage is not met

For [new code](tools/sonarqube.md#definition-of-new-code), must meet coverage specified in default Sonarqube gateway.
For legacy code, coverage amount can be initially lowered to deal with historic tech debt. But must have a plan for increasing coverage over time. | [SonarQube](tools/sonarqube.md) (in conjunction with testing frameworks) | | -| Duplicate code scan | Maintainability | Universal | Check whether the same code is used in multiple places | Duplicate code increases the cost and risk of making software changes and will increase exponentially over time | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met | [SonarQube](tools/sonarqube.md) | | -| Code smells scan | Maintainability | Universal | Check whether the software code adheres to the principles, patterns and practices of writing clean code | The code is written once but read and executed many times. If the code is not clean, the cost and risk of making software changes both increase exponentially over time | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met | [SonarQube](tools/sonarqube.md) | [Clean Code: Smells and Heuristics](https://moderatemisbehaviour.github.io/clean-code-smells-and-heuristics/) | -| Dead code scan | Maintainability | Universal | Detecting unused code and files that are not needed | Code is written once but read and executed many times. The more code you have, the greater the risk of somethign going wrong | | | | -| Code review | Other | Universal | A second person manually checking a code change | Quality check by a human, as opposed to via a tool | Enforced & audited step within workflow | TBC | [Code review guidance](./patterns/everything-as-code.md#code-review) | -| Accessibility tests | Other | Universal | The practice of making applications usable by as many people as possible. | It is a regulatory requirement that our applications are accessible by as many people as possible. Catching accessibility failures up front is essential to maximise the accessibility | | | | -| Tech radar check | Other | Universal | Checking that the tools to be used are in line with organisational / team standards | To prevent the unnecessary proliferation of a wide variety of tools and technologies, which would have a negative impact on overall effectiveness | | | | +| Quality check | Classification | Applicability | What it means | Why we care | Tolerances for green | Minimum for Amber | Endorsed tools / configuration | Further details | +|:---------------------------------------------- |:--------------- |:------------- |:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |:------------------------------------------------------------------------------------------------------------- | +| Unit tests | Functionality | Universal | Logic tests for individual blocks of code, e.g. individual methods | This is the quickest (to execute) type of functional test, so these are essential to achieve both rapid and thorough functional testing | Builds fail if any tests fail. | All new code has relevant unit tests and CI is in the process of implementation. | - | [Test practices](./practices/testing.md) | +| Integration tests | Functionality | Universal | Check interactions with other components and dependant systems. e.g. across microservices, authentication layers, database, third-party systems. Ideally includes full end-to-end testing across all components | When components are developed in isolation, it's vital that we regularly test them working together. Changes in one component can break the whole system | Builds fail if any tests fail. | Automated scripted tests exist for key interfaces and are executed at least before every release. | | | +| API / contract tests | Functionality | Contextual | Check whether the API interface adheres to the agreed contract | Any API interface is an integration point with another component or a software systems. An extra care has to be taken to ensure compatibility and stability of that integration are maintained so that we don't break applications that depend on our APIs | Builds fail if any tests fail. | Automated scripted tests exist for key journeys and are executed at least before every release. | Postman | [Automate Your API Tests with Postman](https://www.postman.com/use-cases/api-testing-automation/) | +| UI tests | Functionality | Contextual | Check that the user interface components behave as expected, particularly checking the visual elements to verify that they are functioning according to requirements | As the only aspects of software that end users come into contact with, it is essential that these elements behave as expected and allow users to get only what they need from our software applications | Builds fail if any tests fail. | Automated scripted tests exist for key journeys and are executed at least before every release. | | | +| Secret scanning | Security | Universal | Check for secrets (e.g. passwords, IP addresses, etc) accidentally included in software code | This protects us against accidentally leaking secrets (in source code) which could compromise the security of the application | Review the list of patterns and update it as necessary

... then:

Full repository (including history) scan, and all secrets removed

And:

Local (on developer laptop) scanning to block commits containing the patterns

And:

Server-side scanning within the code repository for new commits containing the patterns. | Manual secret scanning can be executed on all repos. Team has ascertained there are no secrets to be removed. | [Security practices](/practices/security.md#secret-scanning) | | +| Security code analysis | Security | Universal | Check for indications of possible security issues (for example injection weaknesses) | This gives fast feedback about security issues: it's not as thorough as security testing, but it's much quicker to execute, so both are important to achieve both rapid and thorough security testing | If using SonarQube, must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met. | Static analysis tool being used and run regularly by engineers locally. | One option is [SonarQube](tools/sonarqube.md). For the purpose of security code analysis, Developer Edition or higher is required as it includes advanced OWASP scanning. | | +| Security testing | Security | Contextual | Check for security issues (for example injection weaknesses) | More thorough than security code scanning, but much slower to execute, so both are important to achieve both rapid and thorough security testing | | Penetration tests are conducted at least annually. | | | +| Dependency scanning | Security | Universal | Check for security issues and vulnerabilities in dependent areas of code that are outside of our direct control | Without this we have no way of knowing of any issues or security vulnerabilities of third party components that we are not responsible for | Automated before every release to a production-like environment - release fails if vulnerable dependencies are found | Manual dependency scans run frequently on an ad-hoc basis. | | | +| Performance tests | Resilience | Contextual | Check whether application performance is acceptable at different levels of load | Without this test, we don't know how load will affect the performance of the application | | | | | +| Capacity tests | Resilience | Contextual | Identify the application's breaking point in terms of heavy load | Without this test, we don't know how much load the application can handle before the application breaks | Routinely as part of part of checks prior to production releases. | At least every 6 months, and whenever usage profile is predicted to change (i.e. the system is not expected to break under load) | | | +| Stress tests | Resilience | Contextual | Check whether sudden spikes in load cause a problem | Without this test, we don't know if the application will fail under a sharp increase in load | Routinely as part of part of checks prior to production releases. | At least every 6 months, and whenever usage profile is predicted to change (i.e. peak load is expected to be satisfied at all times) | | | +| Soak tests | Resilience | Contextual | Check whether sustained heavy load causes a problem | Without this test, we don't know if application performance will suffer under prolonged heavy load | Routinely as part of part of checks prior to production releases. | At least every 6 months, and whenever usage profile is predicted to change (i.e. sustained load is expected to be satisfied at all times) | | | +| Chaos tests | Resilience | Contextual | Cause failures in a system to test the resiliency of that system and its environment, and our ability to respond to failures | Give the team confidence that failures in a given environment will not lead to unplanned downtime or a negative user experience

Ensures that the team has visibility (e.g. dashboards and alerts) to be able to identify issues

Help the team to understand their mean time to recovery (MTTR) and to build muscle memory & confidence for recovery activities | Regular (at least every couple of months) game days, and:

Builds fail if any test fails - note, these tests are slow, and are likely to be part of an infrequently-triggered (e.g. overnight) build

The tests cover whether the system self-heals, auto-scales, and alerts as expected | Manual chaos days on a quarterly (or more frequent) basis. | | | +| Code coverage | Maintainability | Universal | The proportion of the application code which is executed (in this context: during testing) | The higher the code coverage, the more thorough the testing, and therefore the higher the likelihood of detecting functional issues early | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway test coverage is not met

For [new code](tools/sonarqube.md#definition-of-new-code), must meet coverage specified in default Sonarqube gateway.
For legacy code, coverage amount can be initially lowered to deal with historic tech debt. But must have a plan for increasing coverage over time. | | [SonarQube](tools/sonarqube.md) (in conjunction with testing frameworks) | | +| Duplicate code scan | Maintainability | Universal | Check whether the same code is used in multiple places | Duplicate code increases the cost and risk of making software changes and will increase exponentially over time | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met | | [SonarQube](tools/sonarqube.md) | | +| Code smells scan | Maintainability | Universal | Check whether the software code adheres to the principles, patterns and practices of writing clean code | The code is written once but read and executed many times. If the code is not clean, the cost and risk of making software changes both increase exponentially over time | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met | | [SonarQube](tools/sonarqube.md) | [Clean Code: Smells and Heuristics](https://moderatemisbehaviour.github.io/clean-code-smells-and-heuristics/) | +| Dead code scan | Maintainability | Universal | Detecting unused code and files that are not needed | Code is written once but read and executed many times. The more code you have, the greater the risk of somethign going wrong | | | | | +| Code review | Other | Universal | A second person manually checking a code change | Quality check by a human, as opposed to via a tool | Enforced & audited step within workflow. | No amber tier, given this is a fundamental practice. | TBC | [Code review guidance](./patterns/everything-as-code.md#code-review) | +| Accessibility tests | Other | Universal | The practice of making applications usable by as many people as possible. | It is a regulatory requirement that our applications are accessible by as many people as possible. Catching accessibility failures up front is essential to maximise the accessibility | | | | | +| Tech radar check | Other | Universal | Checking that the tools to be used are in line with organisational / team standards | To prevent the unnecessary proliferation of a wide variety of tools and technologies, which would have a negative impact on overall effectiveness | | | | | From d4c7799603289a1ee71b39d4847b2f6ac089e52a Mon Sep 17 00:00:00 2001 From: iw-ezequiel Date: Wed, 11 Aug 2021 10:18:52 +0100 Subject: [PATCH 3/5] Clarification on usage and definitions --- quality-checks.md | 47 ++++++++++++++++++++++++----------------------- 1 file changed, 24 insertions(+), 23 deletions(-) diff --git a/quality-checks.md b/quality-checks.md index c448cf53..b8a06ce2 100644 --- a/quality-checks.md +++ b/quality-checks.md @@ -15,12 +15,13 @@ The checks are classified here based on the concerns they help to address: ## Usage -All applicable quality checks should be applied. Not all checks are applicable in all contexts, for example accessibility testing is only applicable to applications with a user interface. +All applicable quality checks should be applied. Not all checks are applicable in all contexts, for example accessibility testing is only applicable to applications with a user interface. The [Details](#details) section lists the applicability of each of the checks. The majority of these checks should be [automated](./patterns/automate-everything.md) via [continuous integration / continuous deployment](./practices/continuous-integration.md): the optimal sequencing of these checks within CI/CD pipelines will depend on the project's branching strategy, deployment strategy, etc. All of these checks are important, even where their purpose overlaps with other checks in the list. For example, comprehensive functional testing could be achieved without unit testing, instead only using the other functional test types on this list - this would result in a very long-running and inefficient test suite, precluding fast feedback and impeding rapid and safe delivery of software changes. For further details please see [test practices](./practices/testing.md), and especially the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html). +Although we separate resilience checks into their notional components, when it comes to implementation these may be joined up where useful. For example, an automated test suite may test the application for peak load and then drop down to sustained load in order to conduct a soak test. ## RAG scale We rate our applications against each of these checks as follows: @@ -40,25 +41,25 @@ We recommend tracking progress on an Engineering Quality dashboard, for example: ## Details -| Quality check | Classification | Applicability | What it means | Why we care | Tolerances for green | Minimum for Amber | Endorsed tools / configuration | Further details | -|:---------------------------------------------- |:--------------- |:------------- |:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |:------------------------------------------------------------------------------------------------------------- | -| Unit tests | Functionality | Universal | Logic tests for individual blocks of code, e.g. individual methods | This is the quickest (to execute) type of functional test, so these are essential to achieve both rapid and thorough functional testing | Builds fail if any tests fail. | All new code has relevant unit tests and CI is in the process of implementation. | - | [Test practices](./practices/testing.md) | -| Integration tests | Functionality | Universal | Check interactions with other components and dependant systems. e.g. across microservices, authentication layers, database, third-party systems. Ideally includes full end-to-end testing across all components | When components are developed in isolation, it's vital that we regularly test them working together. Changes in one component can break the whole system | Builds fail if any tests fail. | Automated scripted tests exist for key interfaces and are executed at least before every release. | | | -| API / contract tests | Functionality | Contextual | Check whether the API interface adheres to the agreed contract | Any API interface is an integration point with another component or a software systems. An extra care has to be taken to ensure compatibility and stability of that integration are maintained so that we don't break applications that depend on our APIs | Builds fail if any tests fail. | Automated scripted tests exist for key journeys and are executed at least before every release. | Postman | [Automate Your API Tests with Postman](https://www.postman.com/use-cases/api-testing-automation/) | -| UI tests | Functionality | Contextual | Check that the user interface components behave as expected, particularly checking the visual elements to verify that they are functioning according to requirements | As the only aspects of software that end users come into contact with, it is essential that these elements behave as expected and allow users to get only what they need from our software applications | Builds fail if any tests fail. | Automated scripted tests exist for key journeys and are executed at least before every release. | | | -| Secret scanning | Security | Universal | Check for secrets (e.g. passwords, IP addresses, etc) accidentally included in software code | This protects us against accidentally leaking secrets (in source code) which could compromise the security of the application | Review the list of patterns and update it as necessary

... then:

Full repository (including history) scan, and all secrets removed

And:

Local (on developer laptop) scanning to block commits containing the patterns

And:

Server-side scanning within the code repository for new commits containing the patterns. | Manual secret scanning can be executed on all repos. Team has ascertained there are no secrets to be removed. | [Security practices](/practices/security.md#secret-scanning) | | -| Security code analysis | Security | Universal | Check for indications of possible security issues (for example injection weaknesses) | This gives fast feedback about security issues: it's not as thorough as security testing, but it's much quicker to execute, so both are important to achieve both rapid and thorough security testing | If using SonarQube, must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met. | Static analysis tool being used and run regularly by engineers locally. | One option is [SonarQube](tools/sonarqube.md). For the purpose of security code analysis, Developer Edition or higher is required as it includes advanced OWASP scanning. | | -| Security testing | Security | Contextual | Check for security issues (for example injection weaknesses) | More thorough than security code scanning, but much slower to execute, so both are important to achieve both rapid and thorough security testing | | Penetration tests are conducted at least annually. | | | -| Dependency scanning | Security | Universal | Check for security issues and vulnerabilities in dependent areas of code that are outside of our direct control | Without this we have no way of knowing of any issues or security vulnerabilities of third party components that we are not responsible for | Automated before every release to a production-like environment - release fails if vulnerable dependencies are found | Manual dependency scans run frequently on an ad-hoc basis. | | | -| Performance tests | Resilience | Contextual | Check whether application performance is acceptable at different levels of load | Without this test, we don't know how load will affect the performance of the application | | | | | -| Capacity tests | Resilience | Contextual | Identify the application's breaking point in terms of heavy load | Without this test, we don't know how much load the application can handle before the application breaks | Routinely as part of part of checks prior to production releases. | At least every 6 months, and whenever usage profile is predicted to change (i.e. the system is not expected to break under load) | | | -| Stress tests | Resilience | Contextual | Check whether sudden spikes in load cause a problem | Without this test, we don't know if the application will fail under a sharp increase in load | Routinely as part of part of checks prior to production releases. | At least every 6 months, and whenever usage profile is predicted to change (i.e. peak load is expected to be satisfied at all times) | | | -| Soak tests | Resilience | Contextual | Check whether sustained heavy load causes a problem | Without this test, we don't know if application performance will suffer under prolonged heavy load | Routinely as part of part of checks prior to production releases. | At least every 6 months, and whenever usage profile is predicted to change (i.e. sustained load is expected to be satisfied at all times) | | | -| Chaos tests | Resilience | Contextual | Cause failures in a system to test the resiliency of that system and its environment, and our ability to respond to failures | Give the team confidence that failures in a given environment will not lead to unplanned downtime or a negative user experience

Ensures that the team has visibility (e.g. dashboards and alerts) to be able to identify issues

Help the team to understand their mean time to recovery (MTTR) and to build muscle memory & confidence for recovery activities | Regular (at least every couple of months) game days, and:

Builds fail if any test fails - note, these tests are slow, and are likely to be part of an infrequently-triggered (e.g. overnight) build

The tests cover whether the system self-heals, auto-scales, and alerts as expected | Manual chaos days on a quarterly (or more frequent) basis. | | | -| Code coverage | Maintainability | Universal | The proportion of the application code which is executed (in this context: during testing) | The higher the code coverage, the more thorough the testing, and therefore the higher the likelihood of detecting functional issues early | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway test coverage is not met

For [new code](tools/sonarqube.md#definition-of-new-code), must meet coverage specified in default Sonarqube gateway.
For legacy code, coverage amount can be initially lowered to deal with historic tech debt. But must have a plan for increasing coverage over time. | | [SonarQube](tools/sonarqube.md) (in conjunction with testing frameworks) | | -| Duplicate code scan | Maintainability | Universal | Check whether the same code is used in multiple places | Duplicate code increases the cost and risk of making software changes and will increase exponentially over time | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met | | [SonarQube](tools/sonarqube.md) | | -| Code smells scan | Maintainability | Universal | Check whether the software code adheres to the principles, patterns and practices of writing clean code | The code is written once but read and executed many times. If the code is not clean, the cost and risk of making software changes both increase exponentially over time | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met | | [SonarQube](tools/sonarqube.md) | [Clean Code: Smells and Heuristics](https://moderatemisbehaviour.github.io/clean-code-smells-and-heuristics/) | -| Dead code scan | Maintainability | Universal | Detecting unused code and files that are not needed | Code is written once but read and executed many times. The more code you have, the greater the risk of somethign going wrong | | | | | -| Code review | Other | Universal | A second person manually checking a code change | Quality check by a human, as opposed to via a tool | Enforced & audited step within workflow. | No amber tier, given this is a fundamental practice. | TBC | [Code review guidance](./patterns/everything-as-code.md#code-review) | -| Accessibility tests | Other | Universal | The practice of making applications usable by as many people as possible. | It is a regulatory requirement that our applications are accessible by as many people as possible. Catching accessibility failures up front is essential to maximise the accessibility | | | | | -| Tech radar check | Other | Universal | Checking that the tools to be used are in line with organisational / team standards | To prevent the unnecessary proliferation of a wide variety of tools and technologies, which would have a negative impact on overall effectiveness | | | | | +| Quality check | Classification | Applicability | What it means | Why we care | Tolerances for green | Minimum for Amber | Endorsed tools / configuration | Further details | +|:---------------------------------------------- |:--------------- |:------------- |:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |:------------------------------------------------------------------------------------------------------------- | +| Unit tests | Functionality | Universal | Logic tests for individual blocks of code, e.g. individual methods | This is the quickest (to execute) type of functional test, so these are essential to achieve both rapid and thorough functional testing | Builds fail if any tests fail. | All new code has relevant unit tests and CI is in the process of implementation. | - | [Test practices](./practices/testing.md) | +| Integration tests | Functionality | Universal | Check interactions with other components and dependant systems. e.g. across microservices, authentication layers, database, third-party systems. Ideally includes full end-to-end testing across all components | When components are developed in isolation, it's vital that we regularly test them working together. Changes in one component can break the whole system | Builds fail if any tests fail. | Automated scripted tests exist for key interfaces and are executed at least before every release. | | | +| API / contract tests | Functionality | Contextual | Check whether the API interface adheres to the agreed contract | Any API interface is an integration point with another component or a software systems. An extra care has to be taken to ensure compatibility and stability of that integration are maintained so that we don't break applications that depend on our APIs | Builds fail if any tests fail. | Automated scripted tests exist for key journeys and are executed at least before every release. | Postman | [Automate Your API Tests with Postman](https://www.postman.com/use-cases/api-testing-automation/) | +| UI tests | Functionality | Contextual | Check that the user interface components behave as expected, particularly checking the visual elements to verify that they are functioning according to requirements | As the only aspects of software that end users come into contact with, it is essential that these elements behave as expected and allow users to get only what they need from our software applications | Builds fail if any tests fail. | Automated scripted tests exist for key journeys and are executed at least before every release. | | | +| Secret scanning | Security | Universal | Check for secrets (e.g. passwords, IP addresses, etc) accidentally included in software code | This protects us against accidentally leaking secrets (in source code) which could compromise the security of the application | Review the list of patterns and update it as necessary

... then:

Full repository (including history) scan, and all secrets removed

And:

Local (on developer laptop) scanning to block commits containing the patterns

And:

Server-side scanning within the code repository for new commits containing the patterns. | Manual secret scanning can be executed on all repos. Team has ascertained there are no secrets to be removed. | [Security practices](/practices/security.md#secret-scanning) | | +| Security code analysis | Security | Universal | Check for indications of possible security issues (for example injection weaknesses) | This gives fast feedback about security issues: it's not as thorough as security testing, but it's much quicker to execute, so both are important to achieve both rapid and thorough security testing | If using SonarQube, must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met. | Static analysis tool being used and run regularly by engineers locally. | One option is [SonarQube](tools/sonarqube.md). For the purpose of security code analysis, Developer Edition or higher is required as it includes advanced OWASP scanning. | | +| Security testing | Security | Contextual | Check for security issues (for example injection weaknesses) | More thorough than security code scanning, but much slower to execute, so both are important to achieve both rapid and thorough security testing | | Penetration tests are conducted at least annually. | | | +| Dependency scanning | Security | Universal | Check for security issues and vulnerabilities in dependent areas of code that are outside of our direct control | Without this we have no way of knowing of any issues or security vulnerabilities of third party components that we are not responsible for | Automated before every release to a production-like environment - release fails if vulnerable dependencies are found | Manual dependency scans run frequently on an ad-hoc basis. | | | +| Performance tests | Resilience | Contextual | Check whether application performance is acceptable at different levels of load | Without this test, we don't know how load will affect the performance of the application | | | | | +| Capacity tests | Resilience | Contextual | Identify the application's breaking point in terms of heavy load | Without this test, we don't know how much load the application can handle before the application breaks | Routinely as part of part of checks prior to production releases. | At least every 6 months, and whenever usage profile is predicted to change (i.e. the system is not expected to break under load) | | | +| Stress tests | Resilience | Contextual | Check whether sudden spikes in load cause a problem | Without this test, we don't know if the application will fail under a sharp increase in load | Routinely as part of part of checks prior to production releases. | At least every 6 months, and whenever usage profile is predicted to change (i.e. peak load is expected to be satisfied at all times) | | | +| Soak tests | Resilience | Contextual | Check whether sustained heavy load causes a problem | Without this test, we don't know if application performance will suffer under prolonged heavy load | Routinely as part of part of checks prior to production releases. | At least every 6 months, and whenever usage profile is predicted to change (i.e. sustained load is expected to be satisfied at all times) | | | +| Chaos tests | Resilience | Contextual | Cause failures in a system to test the resiliency of that system and its environment, and our ability to respond to failures | Give the team confidence that failures in a given environment will not lead to unplanned downtime or a negative user experience

Ensures that the team has visibility (e.g. dashboards and alerts) to be able to identify issues

Help the team to understand their mean time to recovery (MTTR) and to build muscle memory & confidence for recovery activities | Regular (at least every couple of months) game days, and:

Builds fail if any test fails - note, these tests are slow, and are likely to be part of an infrequently-triggered (e.g. overnight) build

The tests cover whether the system self-heals, auto-scales, and alerts as expected | Manual chaos days on a quarterly (or more frequent) basis. | | | +| Code coverage | Maintainability | Universal | The proportion of the application code which is executed (in this context: during testing) | The higher the code coverage, the more thorough the testing, and therefore the higher the likelihood of detecting functional issues early | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway test coverage is not met

For [new code](tools/sonarqube.md#definition-of-new-code), must meet coverage specified in default Sonarqube gateway.
For legacy code, coverage amount can be initially lowered to deal with historic tech debt. But must have a plan for increasing coverage over time. | | [SonarQube](tools/sonarqube.md) (in conjunction with testing frameworks) | | +| Duplicate code scan | Maintainability | Universal | Check whether the same code is used in multiple places | Duplicate code increases the cost and risk of making software changes and will increase exponentially over time | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met | | [SonarQube](tools/sonarqube.md) | | +| Code smells scan | Maintainability | Universal | Check whether the software code adheres to the principles, patterns and practices of writing clean code | The code is written once but read and executed many times. If the code is not clean, the cost and risk of making software changes both increase exponentially over time | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met | | [SonarQube](tools/sonarqube.md) | [Clean Code: Smells and Heuristics](https://moderatemisbehaviour.github.io/clean-code-smells-and-heuristics/) | +| Dead code scan | Maintainability | Universal | Detecting unused code and files that are not needed | Code is written once but read and executed many times. The more code you have, the greater the risk of somethign going wrong | | | | | +| Code review | Other | Universal | A second person manually checking a code change | Quality check by a human, as opposed to via a tool | Enforced & audited step within workflow. | No amber tier, given this is a fundamental practice. | TBC | [Code review guidance](./patterns/everything-as-code.md#code-review) | +| Accessibility tests | Other | Universal | The practice of making applications usable by as many people as possible. | It is a regulatory requirement that our applications are accessible by as many people as possible. Catching accessibility failures up front is essential to maximise the accessibility | | | | | +| Tech radar check | Other | Universal | Checking that the tools to be used are in line with organisational / team standards | To prevent the unnecessary proliferation of a wide variety of tools and technologies, which would have a negative impact on overall effectiveness | | | | | From 71486a67b789a34f27f7318be4741a8397b4f9ef Mon Sep 17 00:00:00 2001 From: iw-ezequiel Date: Fri, 13 Aug 2021 09:53:39 +0100 Subject: [PATCH 4/5] Further definitions of amber/green --- quality-checks.md | 46 +++++++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/quality-checks.md b/quality-checks.md index b8a06ce2..82d63267 100644 --- a/quality-checks.md +++ b/quality-checks.md @@ -15,7 +15,7 @@ The checks are classified here based on the concerns they help to address: ## Usage -All applicable quality checks should be applied. Not all checks are applicable in all contexts, for example accessibility testing is only applicable to applications with a user interface. The [Details](#details) section lists the applicability of each of the checks. +All applicable quality checks should be applied. Not all checks are universal, i.e. applicable in all contexts. For example, accessibility testing is only applicable to applications with a user interface. The [Details](#details) section lists universal checks in black, and contextual checks in grey. The majority of these checks should be [automated](./patterns/automate-everything.md) via [continuous integration / continuous deployment](./practices/continuous-integration.md): the optimal sequencing of these checks within CI/CD pipelines will depend on the project's branching strategy, deployment strategy, etc. @@ -41,25 +41,25 @@ We recommend tracking progress on an Engineering Quality dashboard, for example: ## Details -| Quality check | Classification | Applicability | What it means | Why we care | Tolerances for green | Minimum for Amber | Endorsed tools / configuration | Further details | -|:---------------------------------------------- |:--------------- |:------------- |:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |:------------------------------------------------------------------------------------------------------------- | -| Unit tests | Functionality | Universal | Logic tests for individual blocks of code, e.g. individual methods | This is the quickest (to execute) type of functional test, so these are essential to achieve both rapid and thorough functional testing | Builds fail if any tests fail. | All new code has relevant unit tests and CI is in the process of implementation. | - | [Test practices](./practices/testing.md) | -| Integration tests | Functionality | Universal | Check interactions with other components and dependant systems. e.g. across microservices, authentication layers, database, third-party systems. Ideally includes full end-to-end testing across all components | When components are developed in isolation, it's vital that we regularly test them working together. Changes in one component can break the whole system | Builds fail if any tests fail. | Automated scripted tests exist for key interfaces and are executed at least before every release. | | | -| API / contract tests | Functionality | Contextual | Check whether the API interface adheres to the agreed contract | Any API interface is an integration point with another component or a software systems. An extra care has to be taken to ensure compatibility and stability of that integration are maintained so that we don't break applications that depend on our APIs | Builds fail if any tests fail. | Automated scripted tests exist for key journeys and are executed at least before every release. | Postman | [Automate Your API Tests with Postman](https://www.postman.com/use-cases/api-testing-automation/) | -| UI tests | Functionality | Contextual | Check that the user interface components behave as expected, particularly checking the visual elements to verify that they are functioning according to requirements | As the only aspects of software that end users come into contact with, it is essential that these elements behave as expected and allow users to get only what they need from our software applications | Builds fail if any tests fail. | Automated scripted tests exist for key journeys and are executed at least before every release. | | | -| Secret scanning | Security | Universal | Check for secrets (e.g. passwords, IP addresses, etc) accidentally included in software code | This protects us against accidentally leaking secrets (in source code) which could compromise the security of the application | Review the list of patterns and update it as necessary

... then:

Full repository (including history) scan, and all secrets removed

And:

Local (on developer laptop) scanning to block commits containing the patterns

And:

Server-side scanning within the code repository for new commits containing the patterns. | Manual secret scanning can be executed on all repos. Team has ascertained there are no secrets to be removed. | [Security practices](/practices/security.md#secret-scanning) | | -| Security code analysis | Security | Universal | Check for indications of possible security issues (for example injection weaknesses) | This gives fast feedback about security issues: it's not as thorough as security testing, but it's much quicker to execute, so both are important to achieve both rapid and thorough security testing | If using SonarQube, must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met. | Static analysis tool being used and run regularly by engineers locally. | One option is [SonarQube](tools/sonarqube.md). For the purpose of security code analysis, Developer Edition or higher is required as it includes advanced OWASP scanning. | | -| Security testing | Security | Contextual | Check for security issues (for example injection weaknesses) | More thorough than security code scanning, but much slower to execute, so both are important to achieve both rapid and thorough security testing | | Penetration tests are conducted at least annually. | | | -| Dependency scanning | Security | Universal | Check for security issues and vulnerabilities in dependent areas of code that are outside of our direct control | Without this we have no way of knowing of any issues or security vulnerabilities of third party components that we are not responsible for | Automated before every release to a production-like environment - release fails if vulnerable dependencies are found | Manual dependency scans run frequently on an ad-hoc basis. | | | -| Performance tests | Resilience | Contextual | Check whether application performance is acceptable at different levels of load | Without this test, we don't know how load will affect the performance of the application | | | | | -| Capacity tests | Resilience | Contextual | Identify the application's breaking point in terms of heavy load | Without this test, we don't know how much load the application can handle before the application breaks | Routinely as part of part of checks prior to production releases. | At least every 6 months, and whenever usage profile is predicted to change (i.e. the system is not expected to break under load) | | | -| Stress tests | Resilience | Contextual | Check whether sudden spikes in load cause a problem | Without this test, we don't know if the application will fail under a sharp increase in load | Routinely as part of part of checks prior to production releases. | At least every 6 months, and whenever usage profile is predicted to change (i.e. peak load is expected to be satisfied at all times) | | | -| Soak tests | Resilience | Contextual | Check whether sustained heavy load causes a problem | Without this test, we don't know if application performance will suffer under prolonged heavy load | Routinely as part of part of checks prior to production releases. | At least every 6 months, and whenever usage profile is predicted to change (i.e. sustained load is expected to be satisfied at all times) | | | -| Chaos tests | Resilience | Contextual | Cause failures in a system to test the resiliency of that system and its environment, and our ability to respond to failures | Give the team confidence that failures in a given environment will not lead to unplanned downtime or a negative user experience

Ensures that the team has visibility (e.g. dashboards and alerts) to be able to identify issues

Help the team to understand their mean time to recovery (MTTR) and to build muscle memory & confidence for recovery activities | Regular (at least every couple of months) game days, and:

Builds fail if any test fails - note, these tests are slow, and are likely to be part of an infrequently-triggered (e.g. overnight) build

The tests cover whether the system self-heals, auto-scales, and alerts as expected | Manual chaos days on a quarterly (or more frequent) basis. | | | -| Code coverage | Maintainability | Universal | The proportion of the application code which is executed (in this context: during testing) | The higher the code coverage, the more thorough the testing, and therefore the higher the likelihood of detecting functional issues early | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway test coverage is not met

For [new code](tools/sonarqube.md#definition-of-new-code), must meet coverage specified in default Sonarqube gateway.
For legacy code, coverage amount can be initially lowered to deal with historic tech debt. But must have a plan for increasing coverage over time. | | [SonarQube](tools/sonarqube.md) (in conjunction with testing frameworks) | | -| Duplicate code scan | Maintainability | Universal | Check whether the same code is used in multiple places | Duplicate code increases the cost and risk of making software changes and will increase exponentially over time | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met | | [SonarQube](tools/sonarqube.md) | | -| Code smells scan | Maintainability | Universal | Check whether the software code adheres to the principles, patterns and practices of writing clean code | The code is written once but read and executed many times. If the code is not clean, the cost and risk of making software changes both increase exponentially over time | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met | | [SonarQube](tools/sonarqube.md) | [Clean Code: Smells and Heuristics](https://moderatemisbehaviour.github.io/clean-code-smells-and-heuristics/) | -| Dead code scan | Maintainability | Universal | Detecting unused code and files that are not needed | Code is written once but read and executed many times. The more code you have, the greater the risk of somethign going wrong | | | | | -| Code review | Other | Universal | A second person manually checking a code change | Quality check by a human, as opposed to via a tool | Enforced & audited step within workflow. | No amber tier, given this is a fundamental practice. | TBC | [Code review guidance](./patterns/everything-as-code.md#code-review) | -| Accessibility tests | Other | Universal | The practice of making applications usable by as many people as possible. | It is a regulatory requirement that our applications are accessible by as many people as possible. Catching accessibility failures up front is essential to maximise the accessibility | | | | | -| Tech radar check | Other | Universal | Checking that the tools to be used are in line with organisational / team standards | To prevent the unnecessary proliferation of a wide variety of tools and technologies, which would have a negative impact on overall effectiveness | | | | | +| Quality check | Classification / Applicability | What it means | Why we care | Tolerances for green | Minimum for Amber | Endorsed tools / configuration | Further details | +|:---------------------- |:------------------------------ |:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |:------------------------------------------------------------------------------------------------------------- | +| Unit tests | Functionality - Universal | Logic tests for individual blocks of code, e.g. individual methods | This is the quickest (to execute) type of functional test, so these are essential to achieve both rapid and thorough functional testing | Builds fail if any tests fail. | All new code has relevant unit tests and CI is in the process of implementation. | - | [Test practices](./practices/testing.md) | +| Integration tests | Functionality - Universal | Check interactions with other components and dependant systems. e.g. across microservices, authentication layers, database, third-party systems. Ideally includes full end-to-end testing across all components | When components are developed in isolation, it's vital that we regularly test them working together. Changes in one component can break the whole system | Builds fail if any tests fail. | Automated scripted tests exist for key interfaces and are executed at least before every release. | | | +| API / contract tests | Functionality | Check whether the API interface adheres to the agreed contract | Any API interface is an integration point with another component or a software systems. An extra care has to be taken to ensure compatibility and stability of that integration are maintained so that we don't break applications that depend on our APIs | Builds fail if any tests fail. | Automated scripted tests exist for key journeys and are executed at least before every release. | Postman | [Automate Your API Tests with Postman](https://www.postman.com/use-cases/api-testing-automation/) | +| UI tests | Functionality | Check that the user interface components behave as expected, particularly checking the visual elements to verify that they are functioning according to requirements | As the only aspects of software that end users come into contact with, it is essential that these elements behave as expected and allow users to get only what they need from our software applications | Builds fail if any tests fail. | Automated scripted tests exist for key journeys and are executed at least before every release. | | | +| Secret scanning | Security - Universal | Check for secrets (e.g. passwords, IP addresses, etc) accidentally included in software code | This protects us against accidentally leaking secrets (in source code) which could compromise the security of the application | Review the list of patterns and update it as necessary

... then:

Full repository (including history) scan, and all secrets removed

And:

Local (on developer laptop) scanning to block commits containing the patterns

And:

Server-side scanning within the code repository for new commits containing the patterns. | Manual secret scanning can be executed on all repos. Team has ascertained there are no secrets to be removed. | [Security practices](/practices/security.md#secret-scanning) | | +| Security code analysis | Security - Universal | Check for indications of possible security issues (for example injection weaknesses) | This gives fast feedback about security issues: it's not as thorough as security testing, but it's much quicker to execute, so both are important to achieve both rapid and thorough security testing | If using SonarQube, must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met. | Static analysis tool being used and run regularly by engineers locally. | One option is [SonarQube](tools/sonarqube.md). For the purpose of security code analysis, Developer Edition or higher is required as it includes advanced OWASP scanning. | | +| Security testing | Security | Check for security issues (for example injection weaknesses) | More thorough than security code scanning, but much slower to execute, so both are important to achieve both rapid and thorough security testing | | Penetration tests are conducted at least annually. | | | +| Dependency scanning | Security - Universal | Check for security issues and vulnerabilities in dependent areas of code that are outside of our direct control | Without this we have no way of knowing of any issues or security vulnerabilities of third party components that we are not responsible for | Automated before every release to a production-like environment - release fails if vulnerable dependencies are found | Manual dependency scans run frequently on an ad-hoc basis. | | | +| Performance tests | Resilience | Check whether application performance is acceptable at different levels of load | Without this test, we don't know how load will affect the performance of the application | | | | | +| Capacity tests | Resilience | Identify the application's breaking point in terms of heavy load | Without this test, we don't know how much load the application can handle before the application breaks | Routinely as part of part of checks prior to production releases. | At least every 6 months, and whenever usage profile is predicted to change (i.e. the system is not expected to break under load) | | | +| Stress tests | Resilience | Check whether sudden spikes in load cause a problem | Without this test, we don't know if the application will fail under a sharp increase in load | Routinely as part of part of checks prior to production releases. | At least every 6 months, and whenever usage profile is predicted to change (i.e. peak load is expected to be satisfied at all times) | | | +| Soak tests | Resilience | Check whether sustained heavy load causes a problem | Without this test, we don't know if application performance will suffer under prolonged heavy load | Routinely as part of part of checks prior to production releases. | At least every 6 months, and whenever usage profile is predicted to change (i.e. sustained load is expected to be satisfied at all times) | | | +| Chaos tests | Resilience | Cause failures in a system to test the resiliency of that system and its environment, and our ability to respond to failures | Give the team confidence that failures in a given environment will not lead to unplanned downtime or a negative user experience

Ensures that the team has visibility (e.g. dashboards and alerts) to be able to identify issues

Help the team to understand their mean time to recovery (MTTR) and to build muscle memory & confidence for recovery activities | Regular (at least every couple of months) game days, and:

Builds fail if any test fails - note, these tests are slow, and are likely to be part of an infrequently-triggered (e.g. overnight) build

The tests cover whether the system self-heals, auto-scales, and alerts as expected | Manual chaos days on a quarterly (or more frequent) basis. | | | +| Code coverage | Maintainability - Universal | The proportion of the application code which is executed (in this context: during testing) | The higher the code coverage, the more thorough the testing, and therefore the higher the likelihood of detecting functional issues early | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway test coverage is not met

For [new code](tools/sonarqube.md#definition-of-new-code), must meet coverage specified in default Sonarqube gateway.
For legacy code, coverage amount can be initially lowered to deal with historic tech debt. But must have a plan for increasing coverage over time. | Current coverage is understood and checks can be conducted manually. Criteria exists for thresholds (should account for meaningful coverage, and may also account for legacy code) | [SonarQube](tools/sonarqube.md) (in conjunction with testing frameworks) | | +| Duplicate code scan | Maintainability - Universal | Check whether the same code is used in multiple places | Duplicate code increases the cost and risk of making software changes and will increase exponentially over time | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met | Scans can be executed manually. Code reviews consider duplication of code. | [SonarQube](tools/sonarqube.md) | | +| Code smells scan | Maintainability - Universal | Check whether the software code adheres to the principles, patterns and practices of writing clean code | The code is written once but read and executed many times. If the code is not clean, the cost and risk of making software changes both increase exponentially over time | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met | Scans can be executed manually. Code reviews consider bad code smells. | [SonarQube](tools/sonarqube.md) | [Clean Code: Smells and Heuristics](https://moderatemisbehaviour.github.io/clean-code-smells-and-heuristics/) | +| Dead code scan | Maintainability - Universal | Detecting unused code and files that are not needed | Code is written once but read and executed many times. The more code you have, the greater the risk of somethign going wrong | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met | Scans can be executed manually. Code reviews consider presence of dead code. | | | +| Code review | Other - Universal | A second person manually checking a code change | Quality check by a human, as opposed to via a tool | Enforced & audited step within workflow. | No amber tier, given this is a fundamental practice. | TBC | [Code review guidance](./patterns/everything-as-code.md#code-review) | +| Accessibility tests | Other - Universal | The practice of making applications usable by as many people as possible. | It is a regulatory requirement that our applications are accessible by as many people as possible. Catching accessibility failures up front is essential to maximise the accessibility | All services comply with the [NHS Accessibility Standard](https://service-manual.nhs.uk/accessibility). Websites are WCAG 2.1 compliant and scanned for compliance on a regular basis. Accessibility issues are well understood and documented. | Accessibility is considered as part of design. Manual scenarios are evaluated prior to release of major featurs. | | | +| Tech radar check | Other - Universal | Checking that the tools to be used are in line with organisational / team standards | To prevent the unnecessary proliferation of a wide variety of tools and technologies, which would have a negative impact on overall effectiveness | Reviewed at least every 6 months. | Has been reviewed at least in the past year. | | | From 8177f44923a8bf8f601caf29ddbc4c42df134851 Mon Sep 17 00:00:00 2001 From: iw-ezequiel Date: Fri, 13 Aug 2021 09:55:28 +0100 Subject: [PATCH 5/5] Typos and tooling --- quality-checks.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/quality-checks.md b/quality-checks.md index 82d63267..a6ede05c 100644 --- a/quality-checks.md +++ b/quality-checks.md @@ -59,7 +59,7 @@ We recommend tracking progress on an Engineering Quality dashboard, for example: | Code coverage | Maintainability - Universal | The proportion of the application code which is executed (in this context: during testing) | The higher the code coverage, the more thorough the testing, and therefore the higher the likelihood of detecting functional issues early | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway test coverage is not met

For [new code](tools/sonarqube.md#definition-of-new-code), must meet coverage specified in default Sonarqube gateway.
For legacy code, coverage amount can be initially lowered to deal with historic tech debt. But must have a plan for increasing coverage over time. | Current coverage is understood and checks can be conducted manually. Criteria exists for thresholds (should account for meaningful coverage, and may also account for legacy code) | [SonarQube](tools/sonarqube.md) (in conjunction with testing frameworks) | | | Duplicate code scan | Maintainability - Universal | Check whether the same code is used in multiple places | Duplicate code increases the cost and risk of making software changes and will increase exponentially over time | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met | Scans can be executed manually. Code reviews consider duplication of code. | [SonarQube](tools/sonarqube.md) | | | Code smells scan | Maintainability - Universal | Check whether the software code adheres to the principles, patterns and practices of writing clean code | The code is written once but read and executed many times. If the code is not clean, the cost and risk of making software changes both increase exponentially over time | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met | Scans can be executed manually. Code reviews consider bad code smells. | [SonarQube](tools/sonarqube.md) | [Clean Code: Smells and Heuristics](https://moderatemisbehaviour.github.io/clean-code-smells-and-heuristics/) | -| Dead code scan | Maintainability - Universal | Detecting unused code and files that are not needed | Code is written once but read and executed many times. The more code you have, the greater the risk of somethign going wrong | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met | Scans can be executed manually. Code reviews consider presence of dead code. | | | +| Dead code scan | Maintainability - Universal | Detecting unused code and files that are not needed | Code is written once but read and executed many times. The more code you have, the greater the risk of somethign going wrong | Must use SonarQube's default [rules, profiles and gateways](tools/sonarqube.md#default-quality-gates)

Build pipeline must fail if gateway is not met | Scans can be executed manually. Code reviews consider presence of dead code. | [SonarQube](tools/sonarqube.md) | | | Code review | Other - Universal | A second person manually checking a code change | Quality check by a human, as opposed to via a tool | Enforced & audited step within workflow. | No amber tier, given this is a fundamental practice. | TBC | [Code review guidance](./patterns/everything-as-code.md#code-review) | -| Accessibility tests | Other - Universal | The practice of making applications usable by as many people as possible. | It is a regulatory requirement that our applications are accessible by as many people as possible. Catching accessibility failures up front is essential to maximise the accessibility | All services comply with the [NHS Accessibility Standard](https://service-manual.nhs.uk/accessibility). Websites are WCAG 2.1 compliant and scanned for compliance on a regular basis. Accessibility issues are well understood and documented. | Accessibility is considered as part of design. Manual scenarios are evaluated prior to release of major featurs. | | | +| Accessibility tests | Other - Universal | The practice of making applications usable by as many people as possible. | It is a regulatory requirement that our applications are accessible by as many people as possible. Catching accessibility failures up front is essential to maximise the accessibility | All services comply with the [NHS Accessibility Standard](https://service-manual.nhs.uk/accessibility). Websites are WCAG 2.1 compliant and scanned for compliance on a regular basis. Accessibility issues are well understood and documented. | Accessibility is considered as part of design. Manual scenarios are evaluated prior to release of major features. | | | | Tech radar check | Other - Universal | Checking that the tools to be used are in line with organisational / team standards | To prevent the unnecessary proliferation of a wide variety of tools and technologies, which would have a negative impact on overall effectiveness | Reviewed at least every 6 months. | Has been reviewed at least in the past year. | | |