Project fails to load when model names contain backticks, asterisk or quotes

[giovannipcarvalho]

First of all, thanks for tackling this integration. This is huge!

--

I'm trying it out with our sqlmesh project and it fails due to some of our (external) models having unusual naming.

The external model is actually a set of tables named:

• analytics_12345.events_20240925
• analytics_12345.events_20240926
• ...

So far, nothing unusual here, but to facilitate local development we mock these tables as a single table in duckdb. And to make our external-model schema work in both cases (multiple tables in bigquery vs a single table in duckdb), we declare it as:

# file: external_models/bigquery_ga4_export_schema.yaml
- name: 'analytics_12345.`events_*`'
  columns:
[...] 

(The table in duckdb actually contains an * in its name, and not as a wildcard.)

That's when things get messy (we also had trouble with sqlmesh initially, but it now supports these).

In short, our external models contain backticks, stars and quotes (when normalized by sqlmesh).

I've checked that replacing these extra characters makes our project load just fine. I'm happy to contribute a patch, but first would like to know how you'd want these handled. Just naively adding more string replacements to sqlmesh_model_name_to_key and key_to_sqlmesh_model_name seems like a recipe for disaster.

--

Error message:

dagster._core.errors.DagsterInvalidDefinitionError: "analytics_12345_dot__`events_*`" is not a valid name in Dagster. Names must be in regex ^[A-Za-z0-9_]+$.

[ravenac95]

@giovannipcarvalho apologies for a very delayed response! I've been so heads down writing things around our own sqlmesh configuration that I didn't notice my notification for an issue on this repository. This is definitely an interestiong problem! I'll have to ponder this one.

Thanks so much for the issue 😄

[ravenac95]

Actually upon looking at this a little more. I realize what I need to do is actually use the provided SQLMeshDagsterTranslator for any final connection from sqlmesh to a dagster asset. This should give the correct flexibility. I think the two functions you mentioned (sqlmesh_model_name_to_key and key_to_sqlmesh_model_name) are actually flawed. What we should be doing is internally using exactly the string we have from sqlmesh, but then allow the user to decide whatever the asset name might be in the dagster representation. I'll need to do some refactoring but I think we can make your issue mostly a non-issue (though to get a nice looking name you would need to override the translator). Right now I think things aren't exactly wired in an ideal way mostly because we spent a bit of time figuring out if we could even make this work properly.

[uptickmetachu]

Is this still being looked into? @ravenac95

[marcounternaehrer-cf]

any update?

The problem might still be that get_asset_key_str from utils.py is used in certain places, and not the SQLMeshDagsterTranslator.

[lucargir]

@giovannipcarvalho @uptickmetachu @marcounternaehrer-cf this should be fixed in the next release, for now you can do pip install https://github.com/opensource-observer/dagster-sqlmesh/archive/refs/heads/main.zip