Dagster -> Meltano Connector

[prasannarathi]

We have a implemented Dagster -> Meltano integration a few years ago when the quantile.nl provided integration. But its years old and hasn't been upgraded to support latest version.

Why doesn't Dagster own/maintain the connector with Meltano ? Meltano is one of the popular tool thats used by many companies.

[cmpadden]

Hi @prasannarathi, agreed that the existing Meltano hasn't been updated with the latest features of the Dagster framework.

We haven't intentionally avoided improving this integration, it's just something that we haven't been able to prioritize alongside other work streams. I'll raise it internally to see if this is something we can bump up in priority. Thanks!

[prasannarathi]

Thanks, @cmpadden !
We're eager to get this working. With several End-Of-Life deadlines approaching, it's crucial for us to either upgrade or select an alternative tool. We would greatly value the Dagster team's assistance in upgrading the Meltano connector promptly.

[acarter24]

There was some work to allow meltano integration to be used with higher versions of dagster after the deprecation of dagster_shell in favour of pipes. I think it largely vendors the code from dagster_shell but it has allowed us to move onto dagster 1.12.x for which I am very grateful!

https://github.com/camisinc/elt-dagster-meltano

[menzenski]

That's doing the same thing as quantile-development/dagster-meltano#58 regarding vendoring of the dagster-shell code. This PR isn't likely to be merged (see comments) but we've been running this PR's dagster-meltano code in production for a couple weeks now, have gotten up through Dagster 1.11 with no issues (1.12 is in progress).

[menzenski]

Why doesn't Dagster own/maintain the connector with Meltano ? Meltano is one of the popular tool thats used by many companies.

I've occasionally wondered the same thing. The way I see it, Dagster does embedded ELT and Meltano can't really be embedded in the same way as dlt or sling, since it runs in its own process (and since it shells out to run all the plugin executables). If/when Meltano adds a "Python API" maybe it'd be more doable to run it "natively" within Dagster.