Using Databricks CLI to work with Databricks system tables

The Databricks CLI, released last year, is a versatile tool. With it, you can do everything that is possible via Databricks REST APIs, but it's much easier because the CLI handles authentication, pagination of results, and many other things that could be hard to do correctly on the first attempt. The list of available commands grows as soon as new REST APIs are released.

One of the regular tasks that administrators need to do is enable the new schemas for Databricks system tables—new functionality is released constantly, allowing customers to gain more insights.

With Databricks CLI (+jq) it's very easy to automate the automatic discovery of not-enabled system schemas and enable them - you just need a few simple steps:

• Configure authentication - the simplest way is to expose an environment variable with the name of a workspace-level authentication profile in ~/.databrickscfg:

export DATABRICKS_CONFIG_PROFILE=...        

• Retrieve the ID of Unity Catalog metastore attached to the given workspace:

export DB_METASTORE_ID=$(databricks metastores current|jq -r .metastore_id)        

• Check the list of system schemas available for enablement:

databricks system-schemas list $DB_METASTORE_ID|jq -cr '.[] | select( .state == "AVAILABLE" ) | .schema'        

• If the list isn't empty, enable all available schemas:

for schema in $(databricks system-schemas list $DB_METASTORE_ID|jq -cr '.[] | select( .state == "AVAILABLE" ) | .schema'); do
  echo "Enabling $schema"
  databricks system-schemas enable $DB_METASTORE_ID $schema
done        

That's all!

And if necessary, we can disable system schemas that we don't need, but this isn't a very frequent operation...