Repository navigation
Add a dry run flag for developer clarity #3086
Description
I've noticed a couple times people ask how sqlmesh does dry runs.
We have the answer: sqlmesh plan dev --skip-backfill
But it doesn't look and feel obvious from a dbt user for example that's looking for literal "dry run" language.
I recommend adding a flag so people can do: sqlmesh plan dev --dry-run that works the same as --skip-backfill so it's really easy for people to confidently invoke dry runs.
dbt's equivalent command: dbt run --empty
https://docs.getdbt.com/reference/commands/run#the---empty-flag
References:
Activity
tobymao commentedon Sep 4, 2024
what about sqlmesh dry-run dev
z3z1ma commentedon Sep 4, 2024
I would also recommend an obvious way to pretty print the actual DDL/DML that would be executed (vs just the rendered select statements) directly to the console so users don't have to sift logs where whole queries are on 1 line, post-hoc @sungchun12
sungchun12 commentedon Sep 4, 2024
@z3z1ma oh interesting. I see value in those logs so someone doesn't have to break sqlmesh workflow immersion.
You're thinking it should be part of the
--dry-runflag output or something else like--render-models?sungchun12 commentedon Sep 4, 2024
@tobymao I'm open to this
sqlmesh dry-run dev. I'll play around with it and see how other tools do dry run flags/commands before I draft the PR.izeigerman commentedon Sep 4, 2024
I don't like the idea of learning another command for this. Especially provided it actually does apply a plan and changes the environment state. So it's not a true dry run.
tobymao commentedon Sep 4, 2024
what if dry-run dev didn't actually apply the plan or change environment state? this would be a much bigger task
z3z1ma commentedon Sep 4, 2024
@sungchun12 I am suggesting a more verbose output to the terminal for a dry run. I am not saying in lieu of logs (which have their place in debugging), I am saying in addition. Case in point is ask any user if they enjoy scraping the logs to find out the DDL being executed. They can be thousands of characters wide with no syntax highlighting. The best you get is word wrapping it into a condensed wall of text which sucks with larger logs; or your left fishing up the DDL you are interested in and copy pasting it to somewhere with syntax highlighting or auto formatting. I will be the first to say its not an optimal dev experience FWIW.
In my mind, I'm like "just print EXACTLY what your gonna send to the database, assuming success, in order".
If you want to level this up one more time -- add an engine specific hook to optionally validate the SQL.
https://github.com/dbt-labs/dbt-adapters/blob/4fd2d9a06522f0ca0548c82ede844755c227de07/dbt/adapters/base/impl.py#L358-L367
dbt has the same for its dry run. this allows engine specific error messages without sqlglot ever having to be a validator.
bigquery has JobConfig for this, snowflake and other engines use
EXPLAINprefix.So, spitballing, your options which are not mutually exclusive are
dry run [vanilla] (writes to logs and uses standard console output, basically a regular run that assumes successful materializations)
dry run [verbose] (same as vanilla but also prints rendered DDL/DML to console, pre hooks, post hooks, query fully encapsulated in CTAS, the whole shebang)
dry run [validate] (same as vanilla but use engine specific validate_sql func)
sungchun12 commentedon Sep 4, 2024
@z3z1ma So sounds like you want the look and feel like this in the terminal with DDL/DML/Pre/post hooks in there. Got it.
sungchun12 commentedon Sep 5, 2024
Going to make a new/find an existing issue for your DX comments @z3z1ma