Bug: Airflow UI logging is always in DEBUG mode

[rbreejen]

Currently, all logging displayed in the Airflow UI (which is controlled by the airflow.task logger) consistently shows logs in DEBUG mode, regardless of the intended log level.

Root Cause

The root of this behavior can be traced to this line in the Stackable Airflow operator:

for logger_name, logger_config in LOGGING_CONFIG['loggers'].items():
    logger_config['level'] = logging.NOTSET

Setting the log level to NOTSET effectively disables any filtering on the logger level, causing all log messages (including DEBUG) to be emitted.

Additionally, there is a no-op on line 116:

if logger_name != 'airflow.task':
    logger_config['propagate'] == True  # This uses '==' instead of '=' and has no effect.

This is a comparison instead of an assignment, so the intended propagation setting is not applied.

---

Suggested Improvement

Instead of mutating all loggers, we can explicitly override only the airflow.task logger:

LOGGING_CONFIG['loggers']['airflow.task'] = {
    'level': logging.INFO, # <---- this is important
    'handlers': ['task'],
    'filters': ['mask_secrets'],
    'propagate': True,
}

---

Current Workaround

For now, we are overriding the entire log_config.LOGGING_CONFIG in our deployment to enforce the desired logging behavior and levels.

[lfrancke]

Thanks for the report. There's a public holiday here today but we'll take a look next week.

[Maleware]

We found that == true should be = true, however it wouldn't affect current behaviour as the default as of Airflow 2.4.x + is propagate = True. It's a good catch tho, we will address it.

Concerning the log level issue. Further brought to light that it 'should' work as we expect. To enforce the behaviour you want we'd suggest to use in your crds:

spec:
  celeryExecutors:
    ...
    roleGroups:
    ....
          logging:
            containers:
              airflow:
                console:
                  level: ...
                file:
                  level: ...
                loggers:
                  ROOT:
                    level: ...
                  airflow.task:
                    level: INFO     #  <-- which is equivalent to your proposal as code 

[rbreejen]

Nice! Seems like an elegant solution!

By default, Airflow uses the INFO log level. Explicitly setting it works, of course, but would it make sense for Stackable to adopt this as the default behavior to align with Airflow’s native settings?

[Maleware]

Hey,

we think you are correct on this. I will see what I can do and come back to you with further informations ( or an PR ).

[Maleware]

Hello again,

thanks for your patience, we've addressed your issue with this PR. However, the proper solution would be to implement #650.

For now this is the second best solution we can think of. If you like to get in touch with our code base, we believe this to be a good first issue for it ☺️.

If you agree, I'd like to move this issue to resolved?

[rbreejen]

Awesome, thanks! I will keep this in mind if I decide to explore the codebase