Merge branch 'master' into python-fix-django-taint-sinks

Author: RasmusWL

.devcontainer/devcontainer.json

@@ -0,0 +1,9 @@
+{
+  "extensions": [
+    "github.vscode-codeql",
+    "slevesque.vscode-zipexplorer"
+  ],
+  "settings": {
+    "codeQL.experimentalBqrsParsing": true
+  }
+}

.github/codeql/codeql-config.yml

@@ -0,0 +1,4 @@
+name: "CodeQL config"
+
+queries:
+  - uses: security-and-quality

.github/workflows/codeql-analysis.yml

@@ -0,0 +1,52 @@
+name: "Code scanning - action"
+
+on:
+  push:
+  pull_request:
+  schedule:
+    - cron: '0 9 * * 1'
+
+jobs:
+  CodeQL-Build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v2
+      with:
+        # We must fetch at least the immediate parents so that if this is
+        # a pull request then we can checkout the head.
+        fetch-depth: 2
+
+    # If this run was triggered by a pull request event, then checkout
+    # the head of the pull request instead of the merge commit.
+    - run: git checkout HEAD^2
+      if: ${{ github.event_name == 'pull_request' }}
+      
+    # Initializes the CodeQL tools for scanning.
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@v1
+      # Override language selection by uncommenting this and choosing your languages
+      with:
+        languages: csharp
+        config-file: ./.github/codeql/codeql-config.yml
+
+    # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
+    # If this step fails, then you should remove it and run the build manually (see below)
+    - name: Autobuild
+      uses: github/codeql-action/autobuild@v1
+
+    # ℹ️ Command-line programs to run using the OS shell.
+    # 📚 https://git.io/JvXDl
+
+    # ✏️ If the Autobuild fails above, remove it and uncomment the following three lines
+    #    and modify them (or add more) to build your code if your project
+    #    uses a compiled language
+
+    #- run: |
+    #   make bootstrap
+    #   make release
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@v1

CONTRIBUTING.md

@@ -2,7 +2,7 @@
 
 We welcome contributions to our CodeQL libraries and queries. Got an idea for a new check, or how to improve an existing query? Then please go ahead and open a pull request! Contributions to this project are [released](https://help.github.com/articles/github-terms-of-service/#6-contributions-under-repository-license) to the public under the [project's open source license](LICENSE).
 
-There is lots of useful documentation to help you write queries, ranging from information about query file structure to tutorials for specific target languages. For more information on the documentation available, see [Writing CodeQL queries](https://help.semmle.com/QL/learn-ql/writing-queries/writing-queries.html) on [help.semmle.com](https://help.semmle.com).
+There is lots of useful documentation to help you write queries, ranging from information about query file structure to tutorials for specific target languages. For more information on the documentation available, see [CodeQL queries](https://help.semmle.com/QL/learn-ql/writing-queries/writing-queries.html) on [help.semmle.com](https://help.semmle.com).
 
 
 ## Submitting a new experimental query
@@ -32,7 +32,7 @@ If you have an idea for a query that you would like to share with other CodeQL u
 
     For details, see the [guide on query metadata](docs/query-metadata-style-guide.md).
 
-    Make sure the `select` statement is compatible with the query `@kind`. See [Introduction to query files](https://help.semmle.com/QL/learn-ql/writing-queries/introduction-to-queries.html#select-clause) on help.semmle.com.
+    Make sure the `select` statement is compatible with the query `@kind`. See [About CodeQL queries](https://help.semmle.com/QL/learn-ql/writing-queries/introduction-to-queries.html#select-clause) on help.semmle.com.
 
 3. **Formatting**
 
@@ -53,14 +53,6 @@ After the experimental query is merged, we welcome pull requests to improve it.
 
 ## Using your personal data
 
-If you contribute to this project, we will record your name and email
-address (as provided by you with your contributions) as part of the code
-repositories, which are public. We might also use this information
-to contact you in relation to your contributions, as well as in the
-normal course of software development. We also store records of your
-CLA agreements. Under GDPR legislation, we do this
-on the basis of our legitimate interest in creating the CodeQL product.
-
-Please do get in touch (privacy@github.com) if you have any questions about
-this or our data protection policies.
+If you contribute to this project, we will record your name and email address (as provided by you with your contributions) as part of the code repositories, which are public. We might also use this information to contact you in relation to your contributions, as well as in the normal course of software development. We also store records of CLA agreements signed in the past, but no longer require contributors to sign a CLA. Under GDPR legislation, we do this on the basis of our legitimate interest in creating the CodeQL product. 
 
+Please do get in touch (privacy@github.com) if you have any questions about this or our data protection policies.

README.md

@@ -1,6 +1,6 @@
 # CodeQL
 
-This open source repository contains the standard CodeQL libraries and queries that power [LGTM](https://lgtm.com) and the other CodeQL products that [GitHub](https://github.com) makes available to its customers worldwide.
+This open source repository contains the standard CodeQL libraries and queries that power [LGTM](https://lgtm.com) and the other CodeQL products that [GitHub](https://github.com) makes available to its customers worldwide. For the queries, libraries, and extractor that power Go analysis, visit the [CodeQL for Go repository](https://github.com/github/codeql-go).
 
 ## How do I learn CodeQL and run queries?
 

change-notes/1.25/analysis-cpp.md

@@ -13,9 +13,12 @@ The following changes in version 1.25 affect C/C++ analysis in all applications.
 
 | **Query**                  | **Expected impact**    | **Change**                                                       |
 |----------------------------|------------------------|------------------------------------------------------------------|
+| Uncontrolled format string (`cpp/tainted-format-string`) |  | This query is now displayed by default on LGTM. |
+| Uncontrolled format string (through global variable) (`cpp/tainted-format-string-through-global`) |  | This query is now displayed by default on LGTM. |
 
 ## Changes to libraries
 
+* The library `VCS.qll` and all queries that imported it have been removed.
 * The data-flow library has been improved, which affects most security queries by potentially
   adding more results. Flow through functions now takes nested field reads/writes into account.
   For example, the library is able to track flow from `taint()` to `sink()` via the method
@@ -39,3 +42,5 @@ The following changes in version 1.25 affect C/C++ analysis in all applications.
       }
   };
   ```
+* The security pack taint tracking library (`semmle.code.cpp.security.TaintTracking`) now considers that equality checks may block the flow of taint.  This results in fewer false positive results from queries that use this library.
+* The length of a tainted string (such as the return value of a call to `strlen` or `strftime` with tainted parameters) is no longer itself considered tainted by the `models` library.  This leads to fewer false positive results in queries that use any of our taint libraries.

change-notes/1.25/analysis-csharp.md

@@ -18,10 +18,13 @@ The following changes in version 1.25 affect C# analysis in all applications.
 
 ## Changes to code extraction
 
+* Index initializers, of the form `{ [1] = "one" }`, are extracted correctly. Previously, the kind of the
+  expression was incorrect, and the index was not extracted.
+
 ## Changes to libraries
 
 * The class `UnboundGeneric` has been refined to only be those declarations that actually
-  have type parameters. This means that non-generic nested types inside construced types,
+  have type parameters. This means that non-generic nested types inside constructed types,
   such as `A<int>.B`, no longer are considered unbound generics. (Such nested types do,
   however, still have relevant `.getSourceDeclaration()`s, for example `A<>.B`.)
 * The data-flow library has been improved, which affects most security queries by potentially

change-notes/1.25/analysis-javascript.md

@@ -3,38 +3,106 @@
 ## General improvements
 
 * Support for the following frameworks and libraries has been improved:
+  - [Promise](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise)
+  - [bluebird](http://bluebirdjs.com/)
   - [express](https://www.npmjs.com/package/express)
+  - [fancy-log](https://www.npmjs.com/package/fancy-log)
+  - [fastify](https://www.npmjs.com/package/fastify)
   - [fstream](https://www.npmjs.com/package/fstream)
   - [jGrowl](https://github.com/stanlemon/jGrowl)
   - [jQuery](https://jquery.com/)
   - [marsdb](https://www.npmjs.com/package/marsdb)
+  - [micro](https://www.npmjs.com/package/micro/)
   - [minimongo](https://www.npmjs.com/package/minimongo/)
+  - [mssql](https://www.npmjs.com/package/mssql)
+  - [mysql](https://www.npmjs.com/package/mysql)
+  - [npmlog](https://www.npmjs.com/package/npmlog)
+  - [pg](https://www.npmjs.com/package/pg)
+  - [sequelize](https://www.npmjs.com/package/sequelize)
+  - [spanner](https://www.npmjs.com/package/spanner)
+  - [sqlite](https://www.npmjs.com/package/sqlite)
+  - [ssh2-streams](https://www.npmjs.com/package/ssh2-streams)
+  - [ssh2](https://www.npmjs.com/package/ssh2)
+  - [vue](https://www.npmjs.com/package/vue)
+  - [yargs](https://www.npmjs.com/package/yargs)
+  - [webpack-dev-server](https://www.npmjs.com/package/webpack-dev-server)
+
+* TypeScript 3.9 is now supported.
+
+* TypeScript code embedded in HTML and Vue files is now extracted and analyzed.
+
+* The analysis of sanitizers has improved, leading to more accurate
+  results from the security queries.
 
 ## New queries
 
 | **Query**                                                                       | **Tags**                                                          | **Purpose**                                                                                                                                                                            |
 |---------------------------------------------------------------------------------|-------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Cross-site scripting through DOM (`js/xss-through-dom`) | security, external/cwe/cwe-079, external/cwe/cwe-116 | Highlights potential XSS vulnerabilities where existing text from the DOM is used as HTML. Results are not shown on LGTM by default. |
+| DOM text reinterpreted as HTML (`js/xss-through-dom`) | security, external/cwe/cwe-079, external/cwe/cwe-116 | Highlights potential XSS vulnerabilities where existing text from the DOM is used as HTML. Results are shown on LGTM by default. |
 | Incomplete HTML attribute sanitization (`js/incomplete-html-attribute-sanitization`) | security, external/cwe/cwe-20, external/cwe/cwe-079, external/cwe/cwe-116 | Highlights potential XSS vulnerabilities due to incomplete sanitization of HTML meta-characters. Results are shown on LGTM by default. |
 | Unsafe expansion of self-closing HTML tag (`js/unsafe-html-expansion`) | security, external/cwe/cwe-079, external/cwe/cwe-116 | Highlights potential XSS vulnerabilities caused by unsafe expansion of self-closing HTML tags. |
+| Unsafe shell command constructed from library input (`js/shell-command-constructed-from-input`) | correctness, security, external/cwe/cwe-078, external/cwe/cwe-088 | Highlights potential command injections due to a shell command being constructed from library inputs. Results are shown on LGTM by default. |
+| Download of sensitive file through insecure connection (`js/insecure-download`) | security, external/cwe/cwe-829 | Highlights downloads of sensitive files through an unencrypted protocol. Results are shown on LGTM by default. |
+| Exposure of private files (`js/exposure-of-private-files`) | security, external/cwe/cwe-200 | Highlights servers that serve private files. Results are shown on LGTM by default. |
+| Creating biased random numbers from a cryptographically secure source (`js/biased-cryptographic-random`) | security, external/cwe/cwe-327 | Highlights mathematical operations on cryptographically secure numbers that can create biased results. Results are shown on LGTM by default. |
+| Storage of sensitive information in build artifact (`js/build-artifact-leak`) | security, external/cwe/cwe-312 | Highlights storage of sensitive information in build artifacts. Results are shown on LGTM by default. |
+| Improper code sanitization (`js/bad-code-sanitization`) | security, external/cwe/cwe-094, external/cwe/cwe-079, external/cwe/cwe-116 | Highlights string concatenation where code is constructed without proper sanitization. Results are shown on LGTM by default. |
+| Disabling certificate validation (`js/disabling-certificate-validation`) | security, external/cwe-295 | Highlights locations where SSL certificate validation is disabled. Results are shown on LGTM by default. | 
+| Incomplete multi-character sanitization (`js/incomplete-multi-character-sanitization`) | correctness, security, external/cwe/cwe-20, external/cwe/cwe-116 | Highlights sanitizers that fail to remove dangerous substrings completely. Results are shown on LGTM by default. |
 
 ## Changes to existing queries
 
 | **Query**                      | **Expected impact**          | **Change**                                                                |
 |--------------------------------|------------------------------|---------------------------------------------------------------------------|
-| Misspelled variable name (`js/misspelled-variable-name`) | Message changed | The message for this query now correctly identifies the misspelled variable in additional cases. |
-| Uncontrolled data used in path expression (`js/path-injection`) | More results | This query now recognizes additional file system calls. |
-| Uncontrolled command line (`js/command-line-injection`) | More results | This query now recognizes additional command execution calls. |
-| Client-side URL redirect (`js/client-side-unvalidated-url-redirection`) | Less results | This query now recognizes additional safe patterns of doing URL redirects. |
-| Client-side cross-site scripting (`js/xss`) | Less results | This query now recognizes additional safe strings based on URLs. |
+| Client-side cross-site scripting (`js/xss`) | Fewer results | This query now recognizes additional safe patterns of constructing HTML. |
+| Client-side URL redirect (`js/client-side-unvalidated-url-redirection`) | Fewer results | This query now recognizes additional safe patterns of doing URL redirects. |
+| Code injection (`js/code-injection`) | More results | More potential vulnerabilities involving NoSQL code operators are now recognized. |
+| Exception text reinterpreted as HTML (`js/exception-xss`) | Rephrased and changed visibility | Rephrased name and alert message. Severity lowered from error to warning. Results are now shown on LGTM by default. |
+| Expression has no effect (`js/useless-expression`) | Fewer results | This query no longer flags an expression when that expression is the only content of the containing file. |
+| Hard-coded credentials (`js/hardcoded-credentials`) | More results | This query now recognizes hard-coded credentials sent via HTTP authorization headers. |
 | Incomplete URL scheme check (`js/incomplete-url-scheme-check`) | More results | This query now recognizes additional url scheme checks. |
+| Insecure randomness (`js/insecure-randomness`) | Fewer results | This query now recognizes when an insecure random value is used as a fallback when secure random values are unsupported. |
+| Misspelled variable name (`js/misspelled-variable-name`) | Message changed | The message for this query now correctly identifies the misspelled variable in additional cases. |
+| Non-linear pattern (`js/non-linear-pattern`) | Fewer duplicates and message changed | This query now generates fewer duplicate alerts and has a clearer explanation in case of type annotations used in a pattern. |
 | Prototype pollution in utility function (`js/prototype-pollution-utility`) | More results | This query now recognizes additional utility functions as vulnerable to prototype polution. |
-| Expression has no effect (`js/useless-expression`) | Less results | This query no longer flags an expression when that expression is the only content of the containing file. |
-| Unknown directive (`js/unknown-directive`) | Less results | This query no longer flags directives generated by the Babel compiler. |
-| Code injection (`js/code-injection`) | More results | More potential vulnerabilities involving NoSQL code operators are now recognized. |
+| Uncontrolled command line (`js/command-line-injection`) | More results | This query now recognizes additional command execution calls. |
+| Uncontrolled data used in path expression (`js/path-injection`) | More results | This query now recognizes additional file system calls. |
+| Uncontrolled data used in path expression (`js/path-injection`) | Fewer results | This query no longer flags paths that have been checked to be part of a collection. |
+| Unknown directive (`js/unknown-directive`) | Fewer results | This query no longer flags directives generated by the Babel compiler. |
+| Unneeded defensive code (`js/unneeded-defensive-code`) | Fewer false-positive results | This query now recognizes checks meant to handle the `document.all` object. |
+| Unused property (`js/unused-property`) | Fewer results | This query no longer flags properties of objects that are operands of `yield` expressions. |
 | Zip Slip (`js/zipslip`) | More results | This query now recognizes additional vulnerabilities. |
 
+The following low-precision queries are no longer run by default on LGTM (their results already were not displayed):
+
+  - `js/angular/dead-event-listener`
+  - `js/angular/unused-dependency`
+  - `js/bitwise-sign-check`
+  - `js/comparison-of-identical-expressions`
+  - `js/conflicting-html-attribute`
+  - `js/ignored-setter-parameter`
+  - `js/jsdoc/malformed-param-tag`
+  - `js/jsdoc/missing-parameter`
+  - `js/jsdoc/unknown-parameter`
+  - `js/json-in-javascript-file`
+  - `js/misspelled-identifier`
+  - `js/nested-loops-with-same-variable`
+  - `js/node/cyclic-import`
+  - `js/node/unused-npm-dependency`
+  - `js/omitted-array-element`
+  - `js/return-outside-function`
+  - `js/single-run-loop`
+  - `js/too-many-parameters`
+  - `js/unused-property`
+  - `js/useless-assignment-to-global`
+
 ## Changes to libraries
 
 * A library `semmle.javascript.explore.CallGraph` has been added to help write queries for exploring the call graph.
 * Added data flow for `Map` and `Set`, and added matching type-tracking steps that can accessed using the `CollectionsTypeTracking` module.
+* The data-flow node representing a parameter or destructuring pattern is now always the `ValueNode` corresponding to that AST node. This has a few consequences:
+  - `Parameter.flow()` now gets the correct data flow node for a parameter. Previously this had a result, but the node was disconnected from the data flow graph.
+  - `ParameterNode.asExpr()` and `.getAstNode()` now gets the parameter's AST node, whereas previously it had no result.
+  - `Expr.flow()` now has a more meaningful result for destructuring patterns. Previously this node was disconnected from the data flow graph. Now it represents the values being destructured by the pattern.
+* The global data-flow and taint-tracking libraries now model indirect parameter accesses through the `arguments` object in some cases, which may lead to additional results from some of the security queries, particularly "Prototype pollution in utility function".
+* The predicates `Type.getProperty()` and variants of `Type.getMethod()` have been deprecated due to lack of use-cases. Looking up a named property of a static type is no longer supported, favoring faster extraction times instead.