Python change notes: Merge in internal change notes.

Author: markshannon

change-notes/1.19/analysis-python.md

@@ -6,11 +6,57 @@
 > Changes that affect alerts in many files or from many queries
 > For example, changes to file classification
 
+### Representation of the control flow graph
+
+The representation of the control flow graph (CFG) has been modified to better reflect the semantics of Python.
+
+The following statement types no longer have a CFG node for the statement itself, as their sub-expressions already contain all the
+semantically significant information:
+
+* `ExprStmt`
+* `If`
+* `Assign`
+* `Import`
+
+For example, the CFG for `if cond: foo else bar` now starts with the CFG node for `cond`.
+
+For the following statement types, the CFG node for the statement now follows the CFG nodes of its sub-expressions to better reflect the semantics:
+
+* `Print`
+* `TemplateWrite`
+* `ImportStar`
+
+For example the CFG for `print foo` (in Python 2) has changed from `print -> foo` to `foo -> print`, better reflecting the runtime behavior.
+
+
+The CFG for the `with` statement has been re-ordered to more closely reflect the semantics.
+For the `with` statement:
+```python
+with cm as var:
+    body
+```
+The order of the CFG changes from:
+
+    <with>
+    cm
+    var
+    body
+
+to:
+
+    cm
+    <with>
+    var
+    body
+
+A new predicate `Stmt.getAnEntryNode()` has been added to make it easier to write reachability queries involving statements.
+
+
 ## New queries
 
 | **Query**                   | **Tags**  | **Purpose**                                                        |
 |-----------------------------|-----------|--------------------------------------------------------------------|
-| *@name of query (Query ID)* | *Tags*    |*Aim of the new query and whether it is enabled by default or not*  |
+| Information exposure through an exception (`py/stack-trace-exposure`) | security, external/cwe/cwe-209, external/cwe/cwe-497 | Finds instances where information about an exception may be leaked to an external user. Enabled on LGTM by default. |
 
 ## Changes to existing queries
 
@@ -19,20 +65,30 @@ Most security alerts are now visible on LGTM by default.
 
 | **Query**                  | **Expected impact**    | **Change**                                                       |
 |----------------------------|------------------------|------------------------------------------------------------------|
+| Assert statement tests the truth value of a literal constant (`py/assert-literal-constant`) | reliability, correctness     | Checks whether an assert statement is testing the truth of a literal constant value. Not shown by default. |
 | Code injection (`py/code-injection`) | Supports path visualization and is now visible on LGTM by default | No change to expected results |
 | Deserializing untrusted input (`py/unsafe-deserialization`) | Supports path visualization | No change to expected results |
-| Information exposure through an exception (`py/stack-trace-exposure`) | Now visible on LGTM by default | No change to expected results |
+| Encoding error (`py/encoding-error`) | Better alert location | Alert is now shown at the position of the first offending character, rather than at the top of the file. |
+| Missing call to \_\_init\_\_ during object initialization (`py/missing-call-to-init`) | Fewer false positive results | Results where it is likely that the full call chain has not been analyzed are no longer reported. |
 | Reflected server-side cross-site scripting (`py/reflective-xss`) | Supports path visualization and is now visible on LGTM by default | No change to expected results |
 | SQL query built from user-controlled sources (`py/sql-injection`) | Supports path visualization and is now visible on LGTM by default | No change to expected results |
 | Uncontrolled data used in path expression (`py/path-injection`) | Supports path visualization and is now visible on LGTM by default | No change to expected results |
 | Uncontrolled command line (`py/command-line-injection`) | Supports path visualization and is now visible on LGTM by default | No change to expected results |
-| URL redirection from remote source (`py/url-redirection`) | Supports path visualization and is now visible on LGTM by default | No change to expected results |
+| URL redirection from remote source (`py/url-redirection`) | Fewer false positive results and now supports path visualization | Taint is no longer tracked from the right hand side of binary expressions. In other words `SAFE + TAINTED` is now treated as safe. |
+
 
 ## Changes to code extraction
 
-* *Series of bullet points*
+* Improved scalability: Scaling is near linear to at least 20 CPU cores.
+* Five levels of logging can be selected: `CRITICAL`, `ERROR`, `WARN`, `INFO` and `DEBUG`. `WARN` is the default.
+* The `-v` flag can be specified twice to increase logging level to `DEBUG`
+* The `-q` flag has been added to reduce the logging level to `ERROR` or `CRITICAL`
+* Log lines are now in the `[SEVERITY] message` style and never overlap.
+* Extractor now outputs the location of the first offending character when an EncodingError is encountered.
 
 ## Changes to QL libraries
 
-* *Series of bullet points*
+* Taint tracking analysis now understands HTTP requests in the `twisted` library.
 
+* The analysis now handles `isinstance` and `issubclass` tests involving the basic abstract base classes better. For example, the test `issubclass(list, collections.Sequence)` is now understood to be `True`
+* Taint tracking automatically tracks tainted mappings and collections, without you having to add additional taint kinds. This means that custom taints are tracked from `x` to `y` in the following flow: `l = [x]; y =l[0]`.