Update site from Jekyll source repo

Author: Exploding Labs Bot

posts/2025/07/15/2017-12-13-airflows-execute-context.html

@@ -0,0 +1,52 @@
+<div class="wide-logos">
+  <p><img src="/posts/assets/airflow.png" alt="airflow" /></p>
+</div>
+
+<p>Airflow’s context dictionary can be found in the <code class="language-plaintext highlighter-rouge">get_template_context</code> method,
+in Airflow’s
+<a href="https://github.com/databricks/incubator-airflow/blob/master/airflow/models.py">models.py</a>.</p>
+
+<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span>
+    <span class="s">'dag'</span><span class="p">:</span> <span class="n">task</span><span class="p">.</span><span class="n">dag</span><span class="p">,</span>
+    <span class="s">'ds'</span><span class="p">:</span> <span class="n">ds</span><span class="p">,</span>
+    <span class="s">'ds_nodash'</span><span class="p">:</span> <span class="n">ds_nodash</span><span class="p">,</span>
+    <span class="s">'ts'</span><span class="p">:</span> <span class="n">ts</span><span class="p">,</span>
+    <span class="s">'ts_nodash'</span><span class="p">:</span> <span class="n">ts_nodash</span><span class="p">,</span>
+    <span class="s">'yesterday_ds'</span><span class="p">:</span> <span class="n">yesterday_ds</span><span class="p">,</span>
+    <span class="s">'yesterday_ds_nodash'</span><span class="p">:</span> <span class="n">yesterday_ds_nodash</span><span class="p">,</span>
+    <span class="s">'tomorrow_ds'</span><span class="p">:</span> <span class="n">tomorrow_ds</span><span class="p">,</span>
+    <span class="s">'tomorrow_ds_nodash'</span><span class="p">:</span> <span class="n">tomorrow_ds_nodash</span><span class="p">,</span>
+    <span class="s">'END_DATE'</span><span class="p">:</span> <span class="n">ds</span><span class="p">,</span>
+    <span class="s">'end_date'</span><span class="p">:</span> <span class="n">ds</span><span class="p">,</span>
+    <span class="s">'dag_run'</span><span class="p">:</span> <span class="n">dag_run</span><span class="p">,</span>
+    <span class="s">'run_id'</span><span class="p">:</span> <span class="n">run_id</span><span class="p">,</span>
+    <span class="s">'execution_date'</span><span class="p">:</span> <span class="bp">self</span><span class="p">.</span><span class="n">execution_date</span><span class="p">,</span>
+    <span class="s">'prev_execution_date'</span><span class="p">:</span> <span class="n">prev_execution_date</span><span class="p">,</span>
+    <span class="s">'next_execution_date'</span><span class="p">:</span> <span class="n">next_execution_date</span><span class="p">,</span>
+    <span class="s">'latest_date'</span><span class="p">:</span> <span class="n">ds</span><span class="p">,</span>
+    <span class="s">'macros'</span><span class="p">:</span> <span class="n">macros</span><span class="p">,</span>
+    <span class="s">'params'</span><span class="p">:</span> <span class="n">params</span><span class="p">,</span>
+    <span class="s">'tables'</span><span class="p">:</span> <span class="n">tables</span><span class="p">,</span>
+    <span class="s">'task'</span><span class="p">:</span> <span class="n">task</span><span class="p">,</span>
+    <span class="s">'task_instance'</span><span class="p">:</span> <span class="bp">self</span><span class="p">,</span>
+    <span class="s">'ti'</span><span class="p">:</span> <span class="bp">self</span><span class="p">,</span>
+    <span class="s">'task_instance_key_str'</span><span class="p">:</span> <span class="n">ti_key_str</span><span class="p">,</span>
+    <span class="s">'conf'</span><span class="p">:</span> <span class="n">configuration</span><span class="p">,</span>
+    <span class="s">'test_mode'</span><span class="p">:</span> <span class="bp">self</span><span class="p">.</span><span class="n">test_mode</span><span class="p">,</span>
+    <span class="s">'var'</span><span class="p">:</span> <span class="p">{</span>
+        <span class="s">'value'</span><span class="p">:</span> <span class="n">VariableAccessor</span><span class="p">(),</span>
+        <span class="s">'json'</span><span class="p">:</span> <span class="n">VariableJsonAccessor</span><span class="p">()</span>
+    <span class="p">}</span>
+<span class="p">}</span>
+</code></pre></div></div>
+
+<p>An explanation of each item is found in the documentation under
+<a href="https://airflow.apache.org/docs/stable/macros-ref.html">Macros</a>.</p>
+
+<p>Incidentally, you can generate the context from a TaskInstance.</p>
+
+<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">context</span> <span class="o">=</span> <span class="n">TaskInstance</span><span class="p">(</span>
+    <span class="n">task</span><span class="o">=</span><span class="n">task</span><span class="p">,</span>
+    <span class="n">execution_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">.</span><span class="n">now</span><span class="p">()</span>
+<span class="p">).</span><span class="n">get_template_context</span><span class="p">()</span>
+</code></pre></div></div>

posts/2025/07/15/2018-01-12-generate-fernet-key.html

@@ -0,0 +1,34 @@
+<div class="wide-logos">
+  <p><img src="/posts/assets/airflow.png" alt="airflow" /></p>
+</div>
+
+<p>Install cryptography:</p>
+
+<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip <span class="nb">install </span>cryptography
+</code></pre></div></div>
+
+<p>Generate a fernet key:</p>
+
+<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>python <span class="nt">-c</span> <span class="s2">"from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"</span>
+<span class="nv">81HqDtbqAywKSOumSha3BhWNOdQ26slT6K0YaZeZyPs</span><span class="o">=</span>
+</code></pre></div></div>
+
+<h2 id="use-a-fernet-key-with-airflow">Use a fernet key with Airflow</h2>
+
+<p>Paste the key into your <code class="language-plaintext highlighter-rouge">airflow.cfg</code>.</p>
+
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>fernet_key = 81HqDtbqAywKSOumSha3BhWNOdQ26slT6K0YaZeZyPs=
+</code></pre></div></div>
+
+<p>Alternatively, set the environment variable.</p>
+
+<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">AIRFLOW__CORE__FERNET_KEY</span><span class="o">=</span><span class="s1">'81HqDtbqAywKSOumSha3BhWNOdQ26slT6K0YaZeZyPs='</span>
+</code></pre></div></div>
+
+<p>Restart Airflow’s webserver.</p>
+
+<div class="warning">
+  <p>For existing connections (the ones that were defined before setting the Fernet
+key), you need to open each connection in the web admin, re-type the password
+and save it.</p>
+</div>

posts/2025/07/15/2018-03-05-run-dag-and-watch-logs.html

@@ -0,0 +1,83 @@
+<div class="wide-logos">
+  <p><img src="/posts/assets/airflow.png" alt="airflow" /></p>
+</div>
+
+<p>The command to trigger an Airflow dag is simply:</p>
+
+<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>airflow trigger_dag my-dag
+</code></pre></div></div>
+
+<p>But I also want to watch the logs in the terminal. Trouble is, each time a task is run a new directory and file is created. Something like:</p>
+
+<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~/airflow/logs/my-dag/my-task/2018-03-06T09:59:10.427477/1.log
+</code></pre></div></div>
+
+<p>This makes it hard to tail-follow the logs. Thankfully, starting from Airflow
+1.9, logging can be configured easily, allowing you to put all of a dag’s logs
+into one file.</p>
+
+<div class="warning">
+  <p>If you make this change, you won’t be able to view task logs in the web UI,
+because the UI expects log filenames to be in the normal format.</p>
+</div>
+
+<div class="warning">
+  <p>Logging to a single file is useful for development (using the
+SequentialExecutor), but it won’t work in production because issues
+will arise when multiple tasks attempt to write to the same log file at once.</p>
+</div>
+
+<h2 id="easy-solution">Easy Solution</h2>
+
+<div class="warning">
+  <p>Requires Airflow 1.10+</p>
+</div>
+
+<p>Set the <code class="language-plaintext highlighter-rouge">FILENAME_TEMPLATE</code> setting.</p>
+
+<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">export </span><span class="nv">AIRFLOW__CORE__LOG_FILENAME_TEMPLATE</span><span class="o">=</span><span class="s2">"{{ ti.dag_id }}.log"</span>
+</code></pre></div></div>
+
+<h2 id="advanced-solution---recommended">Advanced Solution - Recommended</h2>
+
+<div class="warning">
+  <p>Requires Airflow 1.9+</p>
+</div>
+
+<p>Since Airflow 1.9, logging is configured Pythonically.</p>
+
+<p>Grab Airflow’s default log config, <code class="language-plaintext highlighter-rouge">airflow_local_settings.py</code>, and copy it
+somewhere in your <code class="language-plaintext highlighter-rouge">PYTHONPATH</code>.</p>
+
+<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-O</span> https://raw.githubusercontent.com/apache/incubator-airflow/master/airflow/config_templates/airflow_local_settings.py
+<span class="nb">cp </span>airflow_local_settings.py <span class="nv">$AIRFLOW__CORE__DAGS_FOLDER</span>
+</code></pre></div></div>
+
+<p>Set the logging_config_class setting. (Make sure this is set in both your
+scheduler and worker’s environments). (Alternatively set the related setting in
+airflow.cfg.)</p>
+
+<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">export </span><span class="nv">AIRFLOW__CORE__LOGGING_CONFIG_CLASS</span><span class="o">=</span>airflow_local_settings.DEFAULT_LOGGING_CONFIG
+</code></pre></div></div>
+
+<p>Now you can configure logging to your liking.</p>
+
+<p>Edit airflow_local_settings.py, changing <code class="language-plaintext highlighter-rouge">FILENAME_TEMPLATE</code> to:</p>
+
+<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FILENAME_TEMPLATE <span class="o">=</span> <span class="s1">'{{ ti.dag_id }}.log'</span>
+</code></pre></div></div>
+
+<p>You should now get all of a dag log output in a single file.</p>
+
+<h2 id="tailing-the-logs">Tailing the logs</h2>
+
+<p>Start the scheduler and trigger a dag.</p>
+
+<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>airflow scheduler
+airflow trigger_dag my-dag
+</code></pre></div></div>
+
+<p>Watch the output with <code class="language-plaintext highlighter-rouge">tail -f</code>.</p>
+
+<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">tail</span> <span class="nt">-f</span> ~/airflow/logs/my-dag.log
+</code></pre></div></div>

posts/2025/07/15/2018-03-06-hide-globals-in-dag-definition-file.html

@@ -0,0 +1,27 @@
+<div class="wide-logos">
+  <p><img src="/posts/assets/airflow.png" alt="airflow" /></p>
+</div>
+
+<p>Airflow has a fairly strange way of registering DAGs and tasks. They’re put
+into the global namespace of the DAG definition file.</p>
+
+<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s">'foo'</span><span class="p">,</span> <span class="n">start_date</span><span class="o">=</span><span class="n">start_date</span><span class="p">)</span>
+<span class="n">MyOperator</span><span class="p">(</span><span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">,</span> <span class="n">task_id</span><span class="o">=</span><span class="s">'foo'</span><span class="p">)</span>
+</code></pre></div></div>
+
+<p>Airflow then comes along and finds them.</p>
+
+<p>When importing that file however, as you do when unit testing, it’s not ideal
+to have those global objects created.</p>
+
+<p>The solution is to protect that code with an <code class="language-plaintext highlighter-rouge">if</code> statement:</p>
+
+<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="n">__name__</span><span class="p">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">'unusual_prefix'</span><span class="p">):</span>
+    <span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s">'foo'</span><span class="p">,</span> <span class="n">start_date</span><span class="o">=</span><span class="n">start_date</span><span class="p">)</span>
+    <span class="n">MyOperator</span><span class="p">(</span><span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">,</span> <span class="n">task_id</span><span class="o">=</span><span class="s">'foo'</span><span class="p">)</span>
+</code></pre></div></div>
+
+<p>This is Airflow’s equivalent of <a href="http://effbot.org/pyfaq/tutor-what-is-if-name-main-for.htm"><code class="language-plaintext highlighter-rouge">if __name__ == "__main__"</code></a>.</p>
+
+<p>Airflow will still find your DAG as normal, however that code inside the block
+won’t be executed when the module is imported.</p>