Git-sync: Support multiple repositories in one AirflowCluster

[Maleware]

Current Situation

We have in our CRD's the possibility to configure git-sync using:

  clusterConfig:
    dagsGitSync:
      # Define your repository here. Currently, the public stackable
      - repo: https://github.com/stackabletech/airflow-operator
        branch: "main"
        gitFolder: "tests/templates/kuttl/mount-dags-gitsync/dags"
        depth: 10
        wait: 20s
        # You may uncomment this line, if your repository is private
        # In this case, please fill in config/airflow-git-credentials.yaml
        # credentialsSecret: airflow-git-credentials
        gitSyncConf:
          --rev: HEAD

As of now, we only support one git-sync although this is already, thankfully, a list.

Favoured Situation

Currently, customers would need either support multiple repositories or support multiple branches per repository but ideally both. If we'd need to decide, I'd rather take multiple repositories as this would suffice for ( in my opinion ) much more brought audience.

[adwk67]

I've tested the current status with two separate gitsync repos and it is possible to list multiple gitsync repos, but the target folders get sync-ed into separate folders, whereas Airflow expects everything under AIRFLOW__CORE__DAGS_FOLDER.

Two gitsync containers are created:

[stackable@airflow-scheduler-default-0 ~]$ ls -al /stackable/app/
total 1076
...
drwxrwsrwx 4 root      stackable    4096 Dec  5 08:03 git-0
drwxrwsrwx 4 root      stackable    4096 Dec  5 08:03 git-1
...

But only one location is picked up:

[stackable@airflow-webserver-default-0 ~]$ env | grep DAGS
AIRFLOW__CORE__DAGS_FOLDER=/stackable/app/git-0/current/mount-dags-gitsync/dags_airflow3

See here.