Azure Data Studio is dead!

I resisted the move to Azure Data Studio for so long. Why would I need a different tool when I have SQL Server Management Studio? SSMS did everything I needed. It was proven. It was solid. But the future, I was told, was Azure Data Studio. After a couple of years I finally decided I was going to jump on the ADS (Azure Data Studio) train during my annaul computer rebuild. I made the decision that I would not be installing SSMS and I would be forcing myself to use ADS.

I typed up about half a blog post, realized it was really more of a complaint post about Azure Data Studio and now I'm starting over. Where were we? Ah yes, I've been using Azure Data Studio as my primary development tool for SQL for several years now. I don't even install SSMS on my computers anymore. It's just not necessary for what I need on a daily basis. When I heard ADS was being retired in favor of the MSSQL extension for VS Code I was sad. I have used the MSSQL extension before and it brought back many of the feelings I had in my early days of switching from SSMS to ADS. It felt unfinished, half baked, and had more rough edges than smooth. It just wasn't ready for prime time.

Over the last year though there have been quite a few major updates to the extension and I am happy to say, I at the point where VS Code is my daily driver rather than Azure Data Studio. I do feel like there are things that need some help. The result grid for example just feels off. It's a strange mix of missing functionality, clunky rendering, and a fonts that don't match the rest of VS Code. But the core of what I need at this point checks two important boxes: available and functional.

Now, on to the main point....

How do I configure VS Code's MSSQL extension to be most optimal for my workflow?What settings am I changing to make the environment more functional vs. the defaults? And how can I make VS Code match what I had in Azure Data Studio?

First, you'll need to add the MSSQL extension to VS Code if you haven't already. Next, you'll find that not all settings from Azure Data Studio exist in the VS Code side.

From there, let's explore my settings file.

{
    "editor.acceptSuggestionOnEnter": "off",
    "editor.fontSize": 14,
    "editor.minimap.enabled": false,
    "editor.mouseWheelZoom": true,
    "markdown.preview.scrollEditorWithPreview": false,
    "markdown.preview.scrollPreviewWithEditor": false,
    "mssql.copyRemoveNewLine": false,
    "mssql.format.alignColumnDefinitionsInColumns": true,
    "mssql.format.datatypeCasing": "uppercase",
    "mssql.format.keywordCasing": "uppercase",
    "mssql.openQueryResultsInTabByDefaultDoNotShowPrompt": true,
    "mssql.persistQueryResultTabs": true,
    "powershell.integratedConsole.focusConsoleOnExecute": false,
    "security.workspace.trust.untrustedFiles": "open",
    "workbench.colorTheme": "Default Light Modern",
    "workbench.editor.pinnedTabsOnSeparateRow": true,
    "workbench.editor.untitled.labelFormat": "name",
    "workbench.startupEditor": "none",
    "workbench.tree.indent": 20,
    "workbench.tree.renderIndentGuides": "onHover",
    "mssql.connectionGroups": [
        // I have root, Azure SQL Database, and Microsoft Fabric
    ],
    "mssql.connections": [
        // You know I can't give you this information!
    ]
}

Of course, your settings will vary from these, but the thing I like about Azure Data Studio and VS Code is everyone can make the editor nearly their ideal editor. For me, I don't really care for intellisense, so turning off accept suggestions on Enter allows me to see intellisense suggestions when I may want to, but doesn't put random items into the editor when I really just want to go to a new line in my code. Here are some thoughts on these settings.

• Accept Suggestions on Enter: See above.
• Font Size: 14 works for me well across all my computers as I sync these settings between my Surface Laptop and my desktop computer.
• Minimap: I don't really find a lot of value in this. You can't really see much of anything in it. It's kind of nice when you are searching to see the highlights a little easier, but generally speaking I find it gets in my way.
• Mouse Wheel Zoom: I like to zoom with my mouse wheel. Enough said.
• Markdown Preview Scroll Editor/Preview: I write my blog posts in markdown, GitHub pages, training docs, and contribute to Microsoft docs. I just don't find the scrolling sync very well and becomes more of a distraction than a help.
• SQL Copy Remove New Line: I want the line feeds to stay in the text I copy.
• SQL Format Options: Just some preference for how I like my SQL code to be formatted.
• SQL Persist Query Results Tab: When I switch between queries and come back, I want to pick up right where I left off rather than needing to scroll back.
• PowerShell focus console on execute: I run a script and I want to be able to make changes. Stop moving me someplace else!
• Untrusted files: Just let me open my files!
• Color Theme: I like light mode. Don't talk to me about it. Get out of here with your dark mode...unless it's late at night and I'm in my office where it's dark and the light hurts my eyes. But that's temporary, LIGHT MODE FOREVER!!
• Pinned tabs on separate row: Just makes it easier to organize the tabs. I go back and forth on this one.
• Tab label format: I like to keep a tidy tab. Just the name please.
• Startup editor: Nothing. Give me a blank canvas.
• Tree indent: I like a little more separation between my indentions.
• Tree indent guides: Like my tabs, I like to keep my tree clean. Just show me the guides when I move the mouse over and need to do some navigating.

I've played around with keyboard shortcuts as well, perhaps I'll go into those details more in another post but the short version is there are some keyboard shortcuts I can't live without (CTRL + E to run a query) but there are some things that I'm trying to rewire my brain to do differently given I use VS Code for more than just SQL development (replacing CRTL + R to hide the results with CTRL + J to toggle the panel).

There are a few more settings and keyboard shortcuts that I had in Azure Data Studio that don't seem to have an equal setting in VS Code or may not be relevant becuase of slight differences in how VS Code is setup. But that's how I have things setup in a nutshell.

What are you doing for development these days?

Are you firmly in camp Azure Data Studio for as long as it will run, SSMS forever, already loving VS Code, forcing yourself down a path to get with the times, or perhaps you run a little bit of everything at different times.

• Creating a hub and spoke architecture (OneLake shortcut)

• Leveraging an existing data lake (ADLS Gen2 shortcut)

• Accessing data prepared in another compute engine (ADLS Gen2 shortcut)

• Using data from other clouds for analysis (AWS S3 shortcut)

• Connecting to on-prem data (S3 compatible shortcut with the on-prem data gateway)

There are two ways to create shortcuts:

• The Fabric UX

• Fabric APIs

Today, focus is on the APIs.

I often build and tear down demos or testing environments on a regular basis which means I like to have code-based configuration options. Many customers want to do similar things or need to create dozens of shortcuts and want to reduce the click tax of the UX.

Like many things in Python, I have to go back and reference how to do them multiple times. I'm a SQL person after all. When I can build a way for me to not need to write extra code or go back to the docs then I'm going to do it. That's exactly what I've done with the shortcut maintenance.

Here is the function I've built. It is just a wrapper for the Get, Create, and Delete APIs where I can pass the relevant items without needing to go rebuild the request URLs each time and also returns the status and any errors, so I don't have to go capture those each time. We'll discuss the parameters for this function shortly.

import json
import requests
from requests import status_codes
import sempy.fabric as fabric
import time

def fn_shortcut(action, shortcut_path, shortcut_name, target = None):

    request_headers = {
        "Authorization": "Bearer " + mssparkutils.credentials.getToken("pbi"),
        "Content-Type": "application/json"
    }

    request_body = {
        "path": shortcut_path,
        "name": shortcut_name,
        "target": target
    }

    lakehouse_id = fabric.get_lakehouse_id()
    workspace_id = fabric.get_workspace_id()

    # Get a shortcut
    if action == 'Get':
        response = requests.request(method = "GET", url = f'https://api.fabric.microsoft.com/v1/workspaces/{workspace_id}/items/{lakehouse_id}/shortcuts/{shortcut_path}/{shortcut_name}', headers = request_headers)

    # Create a shortcut
    if action == 'Create':
        response = requests.request(method = "POST", url = f'https://api.fabric.microsoft.com/v1/workspaces/{workspace_id}/items/{lakehouse_id}/shortcuts?shortcutConflictPolicy=Abort', headers = request_headers, json = request_body)

    # Delete a shortcut
    if action == 'Delete':
        response = requests.request(method = "DELETE", url = f'https://api.fabric.microsoft.com/v1/workspaces/{workspace_id}/items/{lakehouse_id}/shortcuts/{shortcut_path}/{shortcut_name}', headers = request_headers)

        if response.status_code == 200:
            # Wait for the delete operation to fully propogate
            while mssparkutils.fs.exists(f'{shortcut_path}/{shortcut_name}'):
                time.sleep(5)

    # Build the return payload for a success response
    if (response.status_code >= 200 and response.status_code <= 299):
        response_content = {
            "request_url"           : response.url,
            "response_content"      : {} if response.text == '' else json.loads(response.text),
            "status"                : "success",
            "status_code"           : response.status_code,
            "status_description"    : status_codes._codes[response.status_code][0]
            }

    # Build the return payload for a failure response
    if not (response.status_code >= 200 and response.status_code <= 299):
        response_content = {
            "request_body"          : request_body,
            "request_headers"       : request_headers,
            "request_url"           : response.url,
            "response_text"         : json.loads(response.text),
            "status"                : "error",
            "status_code"           : response.status_code,
            "status_description"    : status_codes._codes[response.status_code][0]
        }

    return response_content

Before we look at how I use this function, let's understand the properties for each API call so you have a reference point for the function's parameters. You will need to attach the notebook to a lakehouse and gather a little bit of information about your shortcut depending on what you are trying to accomplish (Get, Create, or Delete).

Information for all API calls

All the API calls need a few pieces of information for assembling the various URIs and authentication while create shortcut needs a few additional pieces of information. For each API call you will need:

• Lakehouse ID (The script will get this for you. Remember to attach a lakehouse.)

• Workspace ID (The script will get this for you.)

• Header which passes the bearer token (The script will get this for you.)

Get shortcut

The Get Shortcut API returns the properties of the shortcut. Different values may be returned for OneLake vs. external shortcuts. You need:

• Shortcut Path (The relative location in the lakehouse like Files/Folder)

• Shortcut Name (The name of the shortcut like MyShortcut in the path Files/Folders/MyShortcut)

Delete shortcut

The Delete Shortcut API deletes an existing shortcut. You probably could have guessed that. For this you need the same properties as Get and, just like using Get, the Lakehouse and Warehouse IDs will be captured for you.

Create shortcut

The Create Shortcut API is where things get a little more complicated because we need to pass a request body which is slightly different for each type of shortcut that is created. The reference for those can be found in the Create Shortcut - Target docs. Because I mainly use ADLS Gen2 and OneLake shortcuts, I will include those here. Check out the docs for the others.

ADLS Gen2

There are three pieces of information to replace in the code below:

• Location is the storage account URI

• Subpath is the directory to which the shortcut will point, including the container name

• Connection ID is the Fabric connection ID to the storage account which includes how you will be authenticating to the storage account (Click the gear icon in the top right corner of the screen. Select Manage connections and gateways. View the settings for the connection from the list. Locate the Connection ID.)

target = {
    "adlsGen2": {
        "location": "https://StorageAccountNameHere.dfs.core.windows.net",
        "subpath": "/container/folder",
        "connectionId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    }
}

OneLake

OneLake is a bit more straightforward because there is no connection involved. Just provide the workspace id, lakehouse id, and folder path where the data resides. The path can be to the Files or Tables section.

target = {
    "OneLake": {
        "workspaceId": workspace_id,
        "itemId": lakehouse_id,
        "path": "Files/SomeFolder/SubFolder'
            }
        }

Bringing it all together

Now that we know what we need to provide getting, deleting, and creating shortcuts, let's circle back to our Python function. For my example, I am using a lakehouse that contains some IMDB data.

You will recall that our function is doing a bit of work for us, so we don't need to provide every single piece of information. The function has 3 required and one optional parameter.

• action is required and accepts Get, Create, and Delete to indicate the operation you want to complete.

• shortcut_path is required and should be the path up to, but not including the name of the shortcut. Where we have a shortcut at Files/MyFolder/MyShortcut this parameter would be Files/MyFolder.

• shortcut_name is required and should be the name of the shortcut itself. Where we have a shortcut at Files/MyFolder/MyShortcut this parameter would be MyShortcut.

• target is optional and only needs to be passed when creating a shortcut. This is the payload that describes where the shortcut points and how to authenticate as shown in the prior two sections for ADLS Gen2 and OneLake.

Let's show the function in action!

Get shortcut

Getting shortcut information is super simple. Just pass the Get action, path, and name.

fn_shortcut("Get", "Files", "IMDB")
# Or make the output pretty with this command
# print(json.dumps(fn_shortcut("Get", "Files", "IMDB"), indent = 4))

Delete shortcut

Deleting a shortcut is just as easy. This command does take a little bit longer to run because I have some code that checks to make sure the background cleanup operations complete before returning. You can remove that from the function is you don't want it.

fn_shortcut("Delete", "Files", "IMDB")
# Or make the output pretty with this command
# print(json.dumps(fn_shortcut("Delete", "Files", "IMDB"), indent = 4))

Create shortcut

For creating shortcuts don't forget about the additional parameter that defines the shortcut's target. I will include two examples, one for OneLake and one for ADLS Gen2.

target = {
    "adlsGen2": {
        "location": f'https://scbradlstorage01.dfs.core.windows.net',
        "subpath": "/sampledata/IMDB",
        "connectionId": 'dd8ae71c-dcd7-4550-88cd-caba775f786b'
    }
}

fn_shortcut(action = 'Create', shortcut_path ='Files', shortcut_name = 'IMDB', target = target)
# Or make the output pretty with this command
# print(json.dumps(fn_shortcut(action = 'Create', shortcut_path ='Files', shortcut_name = 'IMDB', target = target), indent = 4))

target = {
    "OneLake": {
        "workspaceId": "fdddc13a-f994-4b62-85ed-bbf7633ede24",
        "itemId": "c57d0e82-b84a-4844-884c-1d6a629adc2a",
        "path": "Files/IMDB/CastCrew"
    }
}

fn_shortcut(action = 'Create', shortcut_path ='Tables', shortcut_name = 'CastCrew', target = target)
# Or make the output pretty with this command
# print(json.dumps(fn_shortcut(action = 'Create', shortcut_path ='Tables', shortcut_name = 'CastCrew', target = target), indent = 4))

Error handling

There is a little bit of error handling in the function as well. If a shortcut exists and you try to create it or if you attempt to delete or get a shortcut that doesn't exist, the errors will be captured and returned with all the relevant information. For example, if I rerun the previous command it will try to create a shortcut that already exists. Parsing out the error message, we can see that Copy, Rename or Update of shortcuts are not supported by OneLake which indicates the shortcut is there so I can't modify it.

Wrapping it up

There you have it. A single Python function that will allow you to reuse the Fabric Shortcut APIs.

What do you think?

After extensive digging I was able to isolate a few different items that I felt were useful for a variety of use cases. Some of these use the Semantic Link package but the most important ones for what I needed just look at the Spark configuration and require no additional packages.

Here is what I collected from the Spark configuration:

• Executor Cores

• Executor Memory

• Minimum Number of Executors

• Maximum Number of Executors

• Number of Nodes in Spark Pool

• Spark App Name

Here is what I collected from Semantic Link:

• Default Lakehouse ID

• Default Lakehouse Name

• Notebook ID

• Notebook Name

• Workspace ID

• Workspace Name

%pip install semantic-link

default_lakehouse_id    = 'No default lakehouse' if fabric.get_lakehouse_id() == '' else fabric.get_lakehouse_id()
default_lakehouse_name  = 'No default lakehouse' if fabric.get_lakehouse_id() == '' else fabric.resolve_item_name(default_lakehouse_id)
notebook_item_id        = fabric.get_artifact_id()
notebook_item_name      = fabric.resolve_item_name(notebook_item_id)
pool_executor_cores     = spark.sparkContext.getConf().get("spark.executor.cores")
pool_executor_memory    = spark.sparkContext.getConf().get("spark.executor.memory")
pool_min_executors      = spark.sparkContext.getConf().get("spark.dynamicAllocation.minExecutors")
pool_max_executors      = spark.sparkContext.getConf().get("spark.dynamicAllocation.maxExecutors")
pool_number_of_nodes    = len(str(sc._jsc.sc().getExecutorMemoryStatus().keys()).replace("Set(","").replace(")","").split(", "))
spark_app_name          = spark.sparkContext.getConf().get("spark.app.name")[::-1].split("_",1)[0][::-1]
workspace_id            = fabric.get_notebook_workspace_id()
workspace_name          = fabric.resolve_workspace_name(workspace_id)

print(f'default_lakehouse_id:   {default_lakehouse_id}')
print(f'default_lakehouse_name: {default_lakehouse_name}')
print(f'notebook_item_id:       {notebook_item_id}')
print(f'notebook_item_name:     {notebook_item_name}')
print(f'spark_app_name:         {spark_app_name}')
print(f'pool_executor_cores:    {pool_executor_cores}')
print(f'pool_executor_memory:   {pool_executor_memory}')
print(f'pool_min_executors:     {pool_min_executors}')
print(f'pool_max_executors:     {pool_max_executors}')
print(f'pool_number_of_nodes:   {pool_number_of_nodes}')
print(f'workspace_id:           {workspace_id}')
print(f'workspace_name:         {workspace_name}')

Now I can use this in a variety of ways. For my use case, I logged all this information to a Delta table so I could track stats across each execution and tie each record back to a cost using Capacity Metrics (more on that in another post maybe).

A frequent inquiry from customers goes something like this: "I ran a query on the [data warehouse or SQL analytics endpoint] and now I want to know how much that query cost me."

On the surface that seems easy enough. Afterall, the Capacity Metrics app tracks the usage for every operation that occurs on your Fabric capacity right down to the individual T-SQL query.

Before getting into how to get the capacity usage, let's outline the assumptions this post is going to make:

1. You understand the association between workspaces and capacities.

2. You understand the concept of smoothing as it relates to capacity usage.

How would I do this manually?

However, when you try to use the app, you soon realize it's far more cumbersome than you'd like. This is the process you need to go through:

1. Run a query using the SQL endpoint (web browser, SSMS, ADS, etc.).

2. Gather the distributed_statement_id and query execution date and time from Query Insights (or capture it through another method of your choosing).

3. In the capacity metrics app, navigate to a time point that is at least 15 minutes after the query finished running.

4. Filter the background operations table to just the operations you want to see (Operation Id = distributed_statement_id).

It's step number 3 and 4 that cause the most pain and what we will address here today programmatically. Step 3 is just confusing for people because you need to wait for the query to show up in the report, but you need to go to "someplace after the query finishes running", usually 10-15 minutes, and find the query there. Step 4 is the bigger challenge because you need to get the list of IDs in step 2 then search for one of them in the filter in step 4, click the check the box, go back to step 2 to copy the next ID, search for it, and repeat that over for each query's distributed statement id.

The process is cumbersome for one query, it will drain your evening if you need to do this for a lot of queries, and it's just impossible if you need to do this for any number of queries at scale.

How would I do this programmatically?

The overall process is similar: Run a query (or queries), get the distributed_statement_id list, look them up in the capacity metrics app.

To accomplish this programmatically we will need to make a couple of changes to the manual process above. Let's do an overview then we'll dive into details.

1. Run a query using the SQL endpoint (web browser, SSMS, ADS, etc.).

2. Gather the distributed_statement_id and query execution date and time from Query Insights. Turn it into a string in the format of "distributed_statement_id_01", "distributed_statement_id_02", "distributed_statement_id_NN"

3. Enter parameters into data engineering notebook.

4. Run the notebook.

Before you try to use this method, you will need to download the notebook Get SQL query CUs from Capacity Metrics which is hosted on my GitHub account. Then, upload the notebook to a Fabric workspace. The workspace will need capacity to run the notebook. The notebook can be in any Fabric workspace; it does not need to be collocated with the Capacity Metrics app or with the workspace where your lakehouses/warehouses live. In fact, this notebook doesn't even require a lakehouse to be in the workspace to run.

Let's look at a walkthrough of this.

Run queries

I think this is self explanatory. Run your queries however you'd like. Keep in mind that it takes several minutes for the queries to show up in Query Insights and a few more minutes to show up in the Capacity Metrics data. Just assume that you need to wait at least 15 minutes after running a query for this whole process to work (the same is true of the manual process above as we are using the same dataset).

Gather the distributed_statement_id(s)

Here is where things deviate from the manual process. While we are still going to be using Query Insights, we want to combine all the distributed statement ids into a single string, grouped by the date the query was run. This is necessary for the magic of a solution that allows you to get the CUs in bulk. Consider the following query to be a guide on how to do this but by no means the only way. The key is the format: double quotes around each distributed_statement_id forming a comma separated list.

SELECT
    CONVERT(DATE, start_time) ExecutionDate,
    COUNT(*) AS CountOfQueries,
    '"' + STRING_AGG(CONVERT(VARCHAR(MAX), distributed_statement_id), '", "') + '"' AS OperationIDList
FROM
    (
        SELECT
            *
        FROM queryinsights.exec_requests_history
        WHERE
            command LIKE '%%'
            AND start_time >= '2024-05-08 20:53:38.000000'
            AND start_time <= '2024-05-13 20:25:53.000000'
    ) AS queryinsights_requests
GROUP BY
    CONVERT(DATE, start_time)

The output from this query will provide the date the queries were executed, how many matched the subquery criteria, and the complete string of query ids. You will want to copy the ExecutionDate and the corresponding OperationIDList for the next step.

For the screenshots later in this post, I chose the 14 queries that were run on May 13th.

Enter parameters in the notebook

The big departure from the manual process is that we will use a notebook to query the Capacity Metrics dataset to get the data we need. You will need to import the notebook to any Fabric workspace (it requires capacity to use Spark) then fill in the 5 parameters in the first code cell.

• capacity_metrics_workspace

  • The name of the workspace that hosts the Capacity Metrics app.

• capacity_metrics_dataset

  • The name of the dataset used by the Capacity Metrics app.

• capacity_id

  • The capacity which hosted the workspace where user query were run.

• date_of_operations

  • This is the ExecutionDate column from the prior step.

  • The date the operations were run.

• operation_id_list

  • This is the OperationIDList column from the prior step.

  • The list of operations of which you want to collect the capacity unit usage.

  • This should be a comma separate list and each operation should be enclosed in double quotes

  • For example: "AAAAAAAA-BBBB-CCCC-DDDD-1234567890AB", "EEEEEEEE-FFFF-GGGG-HHHH-1234567890AB"

Run the notebook

That's it. Run it!

After the notebook runs, scroll down to the very last cell where the dataframe will display each distributed_statement_id (this is the OperationID field in Capacity Metrics) the total CUs consumed by that query and some additional information.

Here you can see the total CUs for each of the 14 queries from step 2.

Did you also notice that we didn't have to deal with a timepoint? We only had to know what date the SQL query ran. You didn't even have to know the time that the query ran. The amount of time that I spend waiting for time period to show up in the capacity metrics report just so I can search for my data is wild. And it's all gone!

The magic that makes this happen is Semantic Link which provides a bridge between the data engineering and Power BI experiences in Fabric. It contains a variety of functionality including refreshing semantic models, reading tables from a semantic model, listing workspaces, getting workspace information, and executing DAX statements to return a dataset from a semantic model which is what this notebook uses.

Wrapping it up

There you have it! Now instead of looking up each query individually, you can just run a simple notebook to look up the CU usage in bulk! Let's wrap up with some key things to be aware of.

1. Here is a link to get a copy of the notebook (Get SQL CUs from Capacity Metrics) from my GitHub account.

2. You still need Capacity Metrics. Make note of the workspace name and semantic model name. Those are needed.

3. This notebook assumes that your Capacity Metrics semantic model is refreshed. Perhaps I will extend this in the future to add model refresh.

4. You will need the ID for the capacity that runs your workspace, not just the capacity name. This is the one part that's a little more difficult than the manual approach.

5. You do need to run this notebook for each date individually. I wanted to make this as easy as possible to use, so I started simple. It's still far faster than looking up individual queries manually. The nice thing though, is you can take this notebook and wrap it in another process that can be much more flexible.

   For me, I have another version of this project where I persist the data from queryinsights.exec_requests_history into a user table, run this notebook to get the CU information, and then update the user table with the CU usage. The next time the process runs it only looks up the queries from query insights that do NOT have any CU usage already.

   You could also do this across many workspaces to centralize the data. You could parameterize this with a pipeline instead of calling it from another notebook. The possibilities are endless! Would love to hear what you have built with it though!

6. I believe there is a limit of 10,000 or 100,000 records that can be returned at any one time. Yes, those numbers are very different. I don't know this for a fact though, but I'm fairly certain that limit is in place. Which means if you have a highly used system, you may need to break the query list into chunks and feed them through. Another great reason to parameterize this and build a framework around it.

What do you think? Time saver? Did I waste my time building this?

Let me know in the comments what you think!

Which leads me to today's question...how do I get a list of all the parameters for a Python function? In today's case: mssparkutils.credentials.getSecret.

You could go to the documentation, but that's often not complete (I'm looking at you, my fellow Microsoft employees on the documentation team!). Usually, I just want to know what my possibilities are so I can at least throw some options in there to see what happens. Maybe I'll get lucky. That's where inspect.signature comes into play.

help

The first thing I could try is using Python's built in "help" method. By simply passing a module, method, function, keyword, or other object the help information from PyDoc.

Python

help(mssparkutils.credentials.getSecret)

Easy enough. What if I want a little more detail?

inspect

The inspect module can help get some useful information for us in this case as well. Using my new friend help, I can see this will give me a look at useful information from Python objects. Here we will look at two functions: inspect.signature() and inspect.getfullargspec().

To begin, we will import the inspect module.

inspect.signature()

As you can see, running inspect.signature(msparkutils.credentials.getSecret) returns what I need in a simple output which is very similar to what we say with help(mssparkutils.credentials.getSecret) earlier. It gives me the list of inputs which are akvName, secret, and linkedService which is optional as it has a default value assigned to it.

But I am a simple person. I need things spelled out a bit more explicitly sometimes.

inspect.getfullargspec()

When I run inspect.getfullargspec(mssparkutils.credentials.getSecret) I am given a much more detailed breakdown of the output. While help and signature() will usually get the job done, it is nice knowing that I can get a full breakdown of the inputs and outputs.

I am sure there are reasons to use signature over getfullargspec and reasons to use getfullargspec over signature. I saw something about a decorated function being handled properly by signature but not getfullargspec. Since the only possible time I can think of to use a decorator function is when I'm putting up Christmas decor and it is currently May, I think that is a deep dive topic for future Brad.

As you may (or may not) know, I work on the Fabric product team at Microsoft. I am a principal program manager on the Fabric CAT (Customer Advisory Team) team. I always feel strange saying CAT team because then it's team team. It's like trying to explain Microsoft Teams to people. "Just post it on our Teams...uh...team?" or "I need to go to the ATM machine to get cash." You see cash is what people used to use to pay for things before...ok, we are wayyyy off topic. Where was I?

Ah yes, I work on the Fabric CAT team, and I often go talk to people about Fabric as a whole. However, most of the demos I do are just touching on one piece of the puzzle. What I'm setting out to do with this series is show the process of building an end-to-end solution in Microsoft Fabric from data source to visualization showing various options along the way. My goal is to be able to walk into any room and be able to say "let me pull up a workspace a show you what this looks like" for most of the major Fabric functionality.

Let's cover some of the basics...

What this will not be

Let's get this out of the way first. What will this demo not be, or what purpose will it not be designed to serve.

• This will not be comprehensive of every single piece of functionality in Microsoft Fabric. We are sticking to major themes, the Heading 1 topics if you will.

• This will not include every experience, just the ones I deal with on a regular basis. Think data science or private endpoints. Maybe it will expand in the future, but we'll see where it goes. The idea is big picture, make the concepts simple.

• This will not be complicated where it can be simple. Again, let's not boil the ocean. We aren't hitting every security feature. I'll leave that for individual posts or other people to cover.

• This will not be smoke and mirrors. I will only use what everyone has access to, and I won't hide any code from you to make things look easier than they are or fake any data along the way.

• This will not show every architecture option. I honestly don't know if I am going to show loading data using Dataflows Gen2 since I can do everything I need in a pipeline copy activity. Again, we shall see what happens. Maybe I'll have secondary posts that cover alternative methods or design patterns.

What this will be

Now that we have some ground rules for what this project will not be, let's go over what it will be.

• This will walk you through every step of building an end-to-end analytics solution on Microsoft Fabric.

• This will be something you can build in your environment as well. I'll make everything available in one form or another, be it screenshots, step-by-step clicks instructions, code on GitHub, or videos on YouTube.

• This will be using the Microsoft Wide World Importers dataset. It's easy to understand (retail sales), has all the translation code to go from the OLTP to DW, and is easily available to everyone.

• This will be as representative of the real world as possible while still keeping the concepts simple. For example, we will use an Azure SQL DB for the source data, but we aren't going to be simulating transactions on the source system and build slowly changing dimension logic. It's easy enough to extend the solution to include those items if you really want to.

• This will be built out over the course of N weeks. That is not a typo, that's not a placeholder I forgot to update, that is the truth. I don't know how long this is going to take. I've thought about doing one post a week, breaking up into approximately 30 minute building sessions, covering one step at a time no matter how long it takes. This is going to change as we go along, I just don't know how long it's going to take. I know I can build everything from scratch in a few hours, but teaching/blogging time is a different animal.

• This will be somewhat sanitized but, as I mentioned above, I won't hide things from you. What I mean by that is that to covert the Wide World Importers into Wide World Importers DW there are a series of views and stored procedures that need to be built/converted. I'm not going to cover every step of that code building/conversion process because it's frankly a bit boring and doesn't serve the purpose of this series, but I will give you all the real code. That means some of the steps you have to do in a real-world project will be happening behind the scenes or will be already done for you. Again, our focus is Fabric not project management or BI development.

What the architecture will look like

This is still to be defined, but here's what I'm thinking:

Azure SQL Database --- Data Factory Pipeline ---> Lakehouse Files --- Notebook ---> Lakehouse Tables --- T-SQL ---> Data Warehouse --- DirectLake ---> Power BI report

Be on the lookout for the first post in the series where we will setup the source database soon!

I hope to see you there!

Date: Wednesday July 19, 2023
Time: 6:00 - 8:00 PM Eastern
Where: Keiser University (6430 Southpoint Parkway Suite #100, Jacksonville, FL 32216
Cost: Free!
Registration: Meetup

Microsoft Fabric 2023 Year in Review and 2024 Roadmap

Less than one year ago Microsoft Fabric was announced at the Build 2023 conference. Since then the team has been hard at work rolling out new features and functionality each month. From small improvements like the ability to convert entire folders to Delta with a right-click rather than going file by file or background improvements like faster CSV loading to marquee functionality like the launch of multiple Copilots and the announcement of general availability or the new database mirroring, 2023 was a huge year for Fabric. We will run down the list of some of the biggest announcements and show demos along the way so you can see the exciting functionality that is being delivered each and every month. Then, we will take a look at the year ahead with a sneak peak into the 2024 roadmap. If you're using Fabric or considering using Fabric for a project, you'll want to join this session to see what all the buzz is about!

As the Christmas season closes and we move deeper into winter there is now a new year to look forward to which means it's time for new year's resolutions.

Resolution

A firm decision to do or not to do something.

I'm not a big fan of resolutions. They are often lofty, unattainable (realistically anyway), and leave little room for error which means people abandon them as soon as things start to go off the rails even a little. Saying "I resolve to go to the gym 3 times a week" psychologically gives you an out as soon as you miss that third day. It creates the feeling of "I said I was going to go this, I messed it up, now my streak is done, and I don't need to keep trying." In fact, almost 25% of people give up on their resolutions in the first week, and more than 2/3 give up before the end of January.

Goal

The object of a person's ambition or effort; an aim or desired result.

Instead, I like to set goals. Goals give you something to aim for but allow for the inevitable error and adjustment that comes along with being human and needing to respond to life events. Someone with the goal of exercising 3 times a week that has a lot of travel coming up would have the freedom to exchange the gym with a jog around the park or a workout in their hotel room before an early morning flight. And if they don't hit the goal that week, then they can evaluate what went wrong, how they can do better and get back to it the next week. Or if they set a goal of 5 days at the gym, they may have a baby and realize that's no longer viable and adjust to 2 gym days with walks on the other days.

Are those two things really all that different. No, of course not. But the mindset that comes with resolutions vs. goals tends to be very different.

What then are my goals for the new year? Some of these are aimed at being healthier so I can live a long, fulfilling life for my family. Some of these are centered around being more intentional with my time. And likely the most important are about deepening my relationship with Jesus Christ. Some of these I already do and want to make sure I don't lose focus, some are things I don't do well and want to improve, and others are areas I haven't started yet.

1. Read through the entire Bible this year (The Bible Recap)

2. Pray every day with the family and on my own

3. Listen to 1 audiobook each month (I'm not a big reader, but I'll listen!)

4. Dedicate as much time between 5:00 PM and the boys bedtime to my family as possible (less screen time/work and more play time with the kids and talk with Nichole)

5. Spend more quality time with Nichole after the boys go to bed (puzzles, crafts, just chatting)

6. Get more sleep (fewer work until 2:00 - 3:00 AM nights)

7. Daily walk and/or bike ride with the family

8. Do something outside with the family every weekend (bike trail, kayak, beach, park trip, etc.)

9. Eat better (less Dr Pepper, more meals at home, fewer carbs, more veggies, all that good stuff)

10. Reduce and shift the kind of content I consume (more history, Bible commentary, educational material and fewer TV shows/movies)

11. Create more content than I did last year (more blogs, more videos)

What do you think?
Am I nitpicking about settings resolutions vs. goals?
What are you resolving to do this year?
What goals are you setting for yourself?

I hope to see you there!

Date: Thursday July 20, 2023
Time: 7:00 - 9:00 PM Eastern
Where: Online (Register to receive the link)
Cost: Free!
Registration: Meetup

Introducing Microsoft Fabric: The All-in-One Analytics Solution

The world is awash with data and it is up to data professionals to make sense of it all. Just a few weeks ago Microsoft announced the next generation of analytics at Build: Microsoft Fabric!

Fabric is a complete end-to-end analytics platform bringing together Azure Data Factory, Azure Synapse, and Power BI in a single location and more deeply integrated than ever. It’s built on a SaaS platform which helps deliver innovation more quickly and allowing users to get up and running in seconds. At the core Fabric provides a lake-centric and open data hub using the popular Delta format with built-in security, governance, and compliance throughout. At the edge, Fabric delivers flexibility for data scientists, data warehouse developers, and Power BI users to build and analyze using their unique skill sets.

During this session we will discuss the guiding principles and architecture behind Fabric and show a lot of demos! Whether you’re a data engineer, data scientist, data warehouse developer, or an existing Power BI user, Fabric has something for everyone. You won’t want to miss this session!

On the surface Fabric looks like an iteration on the work that was done when Azure Synapse Analytics was launched a few years ago. The reality is that this is not just a step forward but a leap. Many of the components of Fabric do look similar to Synapse, including carrying forward the brand in several of the workloads like Synapse Data Warehousing. However, the architecture, integration, and delivery are vastly different.

OneLake

We have to start with OneLake because that is the thread that holds this Fabric together. Everyone is going to write, Tweet, or say that so I'm just going to get it out of the way. Please don't hold it against me. I know, low hanging fruit and all, but it is what it is. Moving on... At the core, OneLake is Azure Data Lake Storage but it's so much more than that. Gone are the days of provisioning storage accounts, setting up private endpoints between storage and compute, copying data so different compute engines can access it, managing security in multiple locations, duplicating data when you want to create development environments, and so much more.

You don't create OneLake. You just have OneLake. It's described as OneDrive for your data, which now that I think about it, I already store data in OneDrive...I'll figure that all out another time. The point is, you interact with OneLake just as easily as you interact with OneDrive. It's driven by your Azure AD account by default. It's easy to share with others. You don't have to worry about how to tune to get the necessary IOPS. It's just there.

The best part is that all of the compute engines in Fabric store their data in OneLake, which means all the compute engines can access each other's data. All the architects out there are probably thinking "Does this mean I don't have to do all that data copying to give my SQL people access to the data in my lake? And does that mean I don't have to copy the data from my warehouse to a dataset in Power BI to be able to report on it?" That is absolutely correct. Along with OneLake comes "One Copy" which means every engine can see and interact with the all the data (with proper security of course).

The other best part...or maybe the equally best part, however you'd say that...is OneLake acts just like Azure Data Lake Storage. That means my data is not locked inside OneLake never to be seen again by any of the other Azure services. Therefore, all the Databricks users out there can read and write data to OneLake meaning you can continue to leverage all of those investments and integrations with other services that you've spent years building.

The other, other best part (this is getting out of hand) is if you have an existing Azure Data Lake Storage account you can create a shortcut and bring that data right into OneLake without even having to move it! So everything in your data estate can then be seen and analyzed through OneLake. More on all of this in a future post. As you can see, there is a huge amount of integration that OneLake unlocks and it's arguably the most important piece of the entire puzzle.

One format

In addition to OneLake there is one other key change that enables all the functionality in Fabric: standardizing on Delta. Every compute engine in Fabric now reads and writes Delta format. This is how we prevent the need to copy data from the lakehouse into the warehouse or into the lakehouse from Kusto. Why Delta? At this point, it's the industry standard. Sure, there is some secret sauce going on under the covers that enables all kinds of nifty functionality but that's for another time...

Workloads

Fabric supports a variety of workloads that are more deeply integrated than ever before. We will go into details of each of these workloads in separate posts, but for now let's talk about what is built into the platform.

• Data Factory - Easily integrate and transform data from any source. Provides no-code and low-code transformation experiences and continues to be the orchestration hub of the data solution.

• Synapse Data Engineering - Provides data engineers a familiar, notebook-based experience for transforming data at scale using Spark.

• Synapse Data Science - The place to go for managing, training, and deploying machine learning models.

• Synapse Data Warehousing - A relational data warehouse with truly independent compute and storage that provides industry-leading performance. This is NOT Synapse dedicated SQL pools and it is NOT Synapse serverless SQL pools.

• Synapse Real Time Analytics - Dive into large volumes of data from apps, websites, IoT devices and any other time series data you can get your hands on.

• Power BI - I feel like this doesn't even need an introduction. Create interactive reports with stunning visuals (I'm not that creative, but I know some people that are) to gain insights from data across your organization.

• Data Activator - Coming soon! - A detection system that provides alerting and monitoring, so you know exactly when important information is changing.

Experiences

The experience in Fabric line up with the workloads to reduce the noise and quickly surface the most relevant information for what you need to get done.

Selecting an experience, such as Synapse Data Warehouse, will bring you to a screen that shows common tasks and resources that a user working on a data warehouse would find useful. In this case, the ability to create a new warehouse, create a new data pipeline, or link directly to the getting started documentation.

Software-as-a-Service

Fabric will be delivered like no analytics platform at Microsoft before. Fabric is leveraging a Software as a Service (SaaS) model. This doesn't mean that you have any less functionality, you will still have complete control over your data and experience like you have in all Azure PaaS services. However, it does mean some things are changing for the better including:

• It just works.

• One capacity that will be shared between all the workloads. No more paying for each individual dedicated SQL pool and worrying about paying for dedicated capacity in multiple databases that may sit unused.

• Updates delivered weekly! That means the platform can be updated with the latest and greatest functionality all the time rather than waiting for less-frequent "big bang" releases.

• Trials to get started in seconds if you just want to kick the tires.

• Fast provisioning and automatic scaling. A data warehouse takes about 10-20 seconds to spin up rather than the 10+ minutes that it takes today. Spark pools come online in less than 15 seconds rather than 5+ minutes today.

• Success by default, meaning fewer knobs to tune because the best practices are implemented automatically (for SQL think things like stats always being up to date) and the workloads are all integrated seamlessly.

Wrapping up

This is the biggest advance in analytics at Microsoft since I started building data warehouses on SQL Server back in 2009. Full disclosure, I may be a little biased since I work on the Fabric product team, as of the time of writing this post I am a member of Fabric CAT (Customer Advisory Team). Sure, all the core components of this platform existed in one shape or another before, but the level of integration, ease of use, and new functionality that is unlocked with Fabric is quite amazing.

If you want to hear directly from the leadership and feature PM teams about all the different workloads, you can head over to the Microsoft Fabric Launch Event page to find a list of sessions that were presented on May 24 and 25 showcasing all the functionality that was announced. If you want to just jump right in and watch the full 6+ hours of content, you can catch the on-demand versions for Day 1 and Day 2 over on YouTube.

In a livestream I did earlier this week someone said Fabric was just Synapse repackaged. I can assure you that is not the case. If you have any doubts, just go try the 60-day free trial and see for yourself.

I hope to see you there!

Date: Wednesday July 19, 2023
Time: 6:00 - 8:00 PM Eastern
Where: Keiser University (6430 Southpoint Parkway Suite #100, Jacksonville, FL 32216
Cost: Free!
Registration: Meetup

Introducing Microsoft Fabric: The All-in-One Analytics Solution

The world is awash with data and it is up to data professionals to make sense of it all. Just a few weeks ago Microsoft announced the next generation of analytics at Build: Microsoft Fabric!

Fabric is a complete end-to-end analytics platform bringing together Azure Data Factory, Azure Synapse, and Power BI in a single location and more deeply integrated than ever. It’s built on a SaaS platform which helps deliver innovation more quickly and allowing users to get up and running in seconds. At the core Fabric provides a lake-centric and open data hub using the popular Delta format with built-in security, governance, and compliance throughout. At the edge, Fabric delivers flexibility for data scientists, data warehouse developers, and Power BI users to build and analyze using their unique skill sets.

During this session we will discuss the guiding principles and architecture behind Fabric and show a lot of demos! Whether you’re a data engineer, data scientist, data warehouse developer, or an existing Power BI user, Fabric has something for everyone. You won’t want to miss this session!

I hope to see you there!

Date: Thursday June 15, 2023
Time: 6:00 - 7:30 PM Eastern
Where: Online (Microsoft Teams, register to receive the link)
Cost: Free!
Registration: Meetup

Introducing Microsoft Fabric: The All-in-One Analytics Solution

The world is awash with data and it is up to data professionals to make sense of it all. Just a few weeks ago Microsoft announced the next generation of analytics at Build: Microsoft Fabric!

Fabric is a complete end-to-end analytics platform bringing together Azure Data Factory, Azure Synapse, and Power BI in a single location and more deeply integrated than ever. It’s built on a SaaS platform which helps deliver innovation more quickly and allowing users to get up and running in seconds. At the core Fabric provides a lake-centric and open data hub using the popular Delta format with built-in security, governance, and compliance throughout. At the edge, Fabric delivers flexibility for data scientists, data warehouse developers, and Power BI users to build and analyze using their unique skill sets.

During this session we will discuss the guiding principles and architecture behind Fabric and show a lot of demos! Whether you’re a data engineer, data scientist, data warehouse developer, or an existing Power BI user, Fabric has something for everyone. You won’t want to miss this session!

The default is 1,000 of them and they aren't cheap.

Do you need 1,000? 100? 8,278?

These are all good questions, but let's try to address the "What is a DWU?" question first.

What is a DWU?

A DWU, or Data Warehouse Unit, is the Synapse dedicated SQL pool's blended representation of compute power. Similar to the early days of Azure SQL Database where we only had DTUs, you won't find vCores or memory listed anywhere on the page or documentation.

This number drives a number of things that we will cover in separate posts at a later date, but at the core it combines CPU, memory, and IO. The used DWU percentage will therefore be represented by the maximum of those three metrics. For example, a workload that is very CPU intensive using 60% of the available CPU for the currently selected DWU, but uses very little memory at 10% and a small amount of IO at 15% would show as using 60% of the available DWU. Similarly, an IO intensive workload using 5% CPU, 15% memory, and 40% of the provisioned IO would show as using 40% of the available DWU.

How do I choose the right number of DWUs?

To oversimplify things, if you need less overall performance then go with a lower DWU. If you need more overall performance, then go with a higher DWU.

I'm a former consultant, so I am well trained in saying "it depends". To that end, in reality, the number of required DWUs is very workload specific. Here are a few guiding principals for helping land on the right setting:

• Do you initial development on a very small number of DWUs. Start with 100 DWUs just get the DDL deployed even.

• DW100 to DW500 are all a single compute node with varying levels of compute power. That means you are really running a single node server with the overhead of an MPP engine. Not until DW1000 do you get a second compute node.

• When doing load tests, use at least DW1000. Anything lower shouldn't be considered for production and should only be used for development purposes.

• Use a tool like JMeter to simulate a workload. Be sure to look at the whole workload.

  • Build a simulation for your ETL.

  • Build another for ad hoc user queries.

  • Build another for reporting tool queries.

  • Run them in series or parallel depending on expected loading and query patterns.

  • Use realistic data sizes. Don't build the simulation on 100k records when you're going to have 100 billion records. It doesn't need to be a 100% match on data size, but it should be representative.

• Scale up or down based on the performance observed in your test.

• Don't forget to account for bursts of activity and plan to scale appropriately. A monthly or quarterly load may mean you scale up a few hours in advance or at some logical point where you can take a few minutes of downtime for the scaling operation to complete.

• DWUs drive concurrency. Maximum concurrency, 128 queries, in not unlocked until DW6000c.

• The more CCI rebuilds you want to run in parallel, the more memory you will need, the more DWUs you will need.

• The moral of the story is this...test, test, test.

How do I know what DWU I'm running?

The current SQL pool service level will be shown in the Azure Portal, Synapse Studio, PowerShell, and through the T-SQL query below when connected to the master database.

SQL

SELECT
    d.name AS DatabaseName,
    Edition AS DatabaseEdition,
    service_objective AS ServiceObjective
FROM sys.database_service_objectives AS dso
INNER JOIN sys.databases AS d
    ON dso.database_id = d.database_id

There you have it. Data warehouse units simplified!

It's been a few years since there was an in-person PASS Summit. Some major changes and a couple of years later we are excited to be back in person for what is now the "PASS Data Community Summit".

If SQL Saturday and Data Saturday event sizes are any indication, this conference will be significantly smaller than past years when it was the PASS Summit but with just as much great content and many great networking opportunities. I can't wait to make my way out to Seattle, meet some new people, and present on Synapse + Power BI.

This year I will be presenting a session called "Better Together: Power BI and Azure Synapse Analytics". If you have ever attended a session of mine in the past, you will know I don't like slides. This session will walk through the adventure of creating a Power BI report on Synapse serverless, eventually hitting performance challenges as the dataset gets really big, tuning the report and Synapse layers, and ultimately making the end users happy again.

If you're attending the Data Community Summit and are looking for a fun place to go at 2:30 PM, come on by! I'll also be around on Wednesday at the Microsoft booth and other places around the conference. If you have Synapse questions or just want to meet to talk, come find me!

Ok, I'm back. It is confirmed. I am in fact officially old. I really did write that blog 12 years ago on June 23, 2010. Wow. Anyway...

12 years later that post has received dozens of views from me trying to remember how I did that. Well, I'm here to tell you there is a better way! I didn't discover this because I wanted to improve my code or because I was scanning the release notes for SQL Server to see what new T-SQL functionality has been released in the last few versions. No, I had to find a better way because my previous method which used a variable and the COALESCE function does not work on Azure Synapse Analytics.

The new method: STRING_AGG()

Let's get started with some sample data:

CREATE TABLE dbo.State
    (
        StateID [int],
        StateName [varchar](50)
    )

INSERT INTO dbo.State
SELECT 1, 'Florida'
UNION ALL
SELECT 2, 'Tennessee'
UNION ALL
SELECT 3, 'Georgia'
UNION ALL
SELECT 4, NULL
UNION ALL
SELECT 5, 'Texas'

SELECT * FROM dbo.State

Now, a simple STRING_AGG(expression, separator) and we are good to go!

Quick and simple!

A couple quick notes in closing.

1. Notice that NULL values are ignored and the separator is not added. If you want to include NULL values you'll need to wrap the column name in this example with ISNULL so it would look something like SELECT STRING_AGG(ISNULL(StateName, 'No Name Provided'), ',') FROM dbo.State
2. There is an optional order by operator that can be used to order the list.
3. The data type is determined by the expression. That means if the column is a string it will retain the string properties. MAX fields will result in a MAX data type. Non-max fields will result in the largest possible non-MAX value (VARCHAR(8000) or NVARCHAR(4000)). All other data types result in an NVARCHAR(4000) result.
4. You aren't limited to a literal string like a comma or a pipe for the separator. That's just my most common request. CHAR(13), CHAR(9), and just about any other expression are all valid as well.

May 2022

Read about the May 2022 updates on the Azure Synapse Blog.

• General
  • [See Details] Get connected with the Azure Synapse Influencer program
• SQL
  • [See Details] Data Warehouse Migration guide for Dedicated SQL Pools in Azure Synapse Analytics
  • [See Details] Specify character column lengths... Not anymore!
• Apache Spark for Synapse
  • [See Details] Azure Synapse Dedicated SQL Pool Connector for Apache Spark Now Available in Python
  • [See Details] Manage Azure Synapse Apache Spark configuration
• Synapse Data Explorer
  • [See Details] Synapse Data Explorer live query in Excel
  • [See Details] Use Managed Identities for External SQL Server Tables
  • [See Details] New KQL Learn module (2 out of 3) is live!
  • [See Details] Azure Synapse Data Explorer connector for Microsoft Power Automate, Logic Apps, and Power Apps [Gen...
  • [See Details] Dynamic events routing from event hub to multiple databases
  • [See Details] Configure a database using a KQL inline script as part of JSON ARM deployment template
• Data Integration
  • [See Details] Export pipeline monitoring as a CSV
  • [See Details] Incremental data loading made easy for Synapse and Azure Database for PostgreSQL and MySQL
  • [See Details] User-Defined Functions for Mapping Data Flows [Public Preview]
  • [See Details] Assert Error Handling
  • [See Details] Mapping data flows projection editing
• Synapse Link
  • [See Details] Azure Synapse Link for SQL [Public Preview]

April 2022

Read about the April 2022 updates on the Azure Synapse Blog.

• SQL
  • [See Details] Cross-subscription restore for Azure Synapse SQL [Generally Available]
  • [See Details] Recover SQL pool from dropped server or workspace
• Synapse Database Templates & Database Designer
  • [See Details] Revamped exploration experience
  • [See Details] Clone lake database
  • [See Details] Use wildcards to specify custom folder hierarchies
• Apache Spark for Synapse
  • [See Details] Apache Spark 3.2 [Public Preview]
  • [See Details] Parameterization for Spark job definition
  • [See Details] Notebook snapshot
• Security
  • [See Details] Synapse Monitoring Operator RBAC role [Generally Available]
• Data Integration
  • [See Details] Dataverse connector added to Synapse Data Flows
  • [See Details] Synapse Pipelines Web activity response timeout improvement
• Developer Experience
  • [See Details] Reference unpublished notebooks

March 2022

Read about the March 2022 updates on the Azure Synapse Blog.

• Developer Experience
  • [See Details] Synapse notebooks: Code cells with exception to show standard output
  • [See Details] Synapse notebooks: Partial output is available for running notebook code cells
  • [See Details] Synapse notebooks: Dynamically control your Spark session configuration with pipeline parameters
  • [See Details] Synapse notebooks: Reuse and manage notebook sessions
  • [See Details] Synapse notebooks: Support for Python logging
• SQL
  • [See Details] Column Level Encryption for Azure Synapse dedicated SQL Pools [Generally Available]
  • [See Details] Better performance for CETAS and subsequent SELECTs
• Apache Spark for Synapse
  • [See Details] Synapse Spark Common Data Model (CDM) Connector [Generally Available]
  • [[](https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/azure-synapse-analytics-march-update-2022/ba-p/3269194#TOCREF_10)See Details] Spark Dedicated SQL Pool (DW) Connector: Performance Improvements
  • [See Details] Synapse Spark Dedicated SQL Pool (DW) Connector: Support for all Spark Dataframe SaveMode choices (...
  • [See Details] Accelerate Spark execution speed using the new Intelligent Cache feature [Public Preview]
• Security
  • [See Details] Azure Synapse Analytics now supports Azure Active Directory (Azure AD) only authentication
  • [See Details] API support to raise or lower Workspace Managed SQL Server Dedicated SQL Minimal TLS version
• Data Integration
  • [See Details] Flowlets and CDC Connectors [Generally Available]
  • [See Details] sFTP connector for Synapse data flows
  • [See Details] Data flow improvements to Data Preview
  • [See Details] Pipeline script activity

February 2022

Read about the February 2022 updates on the Azure Synapse Blog.

• SQL
  • [See Details] More consistent query execution times for Serverless SQL Pools
  • [See Details] The OPENJSON function makes it easy to get array element indexes
• Data Integration
  • [See Details] Upsert supported by Copy Activity
  • [See Details] Transform Dynamics Data Visually in Synapse Data Flows
  • [See Details] Connect to your SQL sources in data flows using Always Encrypted
  • [See Details] Capture descriptions from Asserts in Data Flows
  • [See Details] Easily define schemas for complex type fields

January 2022

Read about the January 2022 updates on the Azure Synapse Blog.

• Apache Spark for Synapse
  • [See Details] 4 New database templates [Public Preview]
• Machine Learning
  • [See Details] Improvements to the SynapseML library
• Security
  • [See Details] Azure Synapse Analytics security overview
  • [See Details] TLS 1.2 required for new workspaces
• Data Integration
  • [See Details] Data quality validation rules using Assert transformation
  • [See Details] Native data flow connector for Dynamics
  • [See Details] IntelliSense and auto-complete added to pipeline expressions
• Synapse SQL
  • [See Details] COPY schema discovery for complex data ingestion
  • [See Details] HASHBYTES easily generates hashes in Serverless SQL

December 2021

Read about the December 2021 updates on the Azure Synapse Blog.

• [See Details] Finding Synapse Monthly Update blogs
• Apache Spark in Synapse
  • [See Details] Additional notebook export formats
  • [See Details] Three new chart types in notebooks
  • [See Details] Reconnect notebooks to Spark sessions
• Data Integration
  • [See Details] Map Data tool [Public Preview], a no-code guided ETL experience
  • [See Details] Quick Reuse of Spark clusters
  • [See Details] External call transformation
  • [See Details] Flowlets [Public Preview]

This post isn’t about Azure Monitor though. Not directly anyway.

Azure Synapse is no different than any other application or service in your environment. It serves a purpose in the architecture and as a result needs special attention from people with a specific skillset, in this case DBAs and developers. That’s where Azure Monitor comes into the picture and what we will be discussing today: how to make sense of the integration between auditing in Azure Synapse and Azure Monitor. The focus will be on Azure Synapse Analytics Dedicated SQL Pools (Gen 2) specifically, but there is integration for monitoring serverless SQL pools, Spark pools, pipelines, and general workspace operations.

First, those with an eye for detail will notice that this post has Log Analytics in the title and not Azure Monitor. That’s true. A very brief explanation so everyone has things straight in their heads. I put Log Analytics in the title because when you set up auditing or diagnostics the option is labeled “Send to Log Analytics workspace” not “Send to Azure Monitor” and I wanted people to be able to find this post. Azure Monitor is a set of functionalities for collecting and analyzing logs. There are some prebuilt integrations and visualizations with some Azure services like Key Vault or Storage Accounts Then there is Log Analytics, a functionality inside Azure Monitor which also happens to be where Synapse writes its logs, for storing and querying all log data. Queries are written in a Kusto Query Language or KQL.

Second, let’s be sure a few very important details are clear. When referring to Azure Synapse there are two distinct but very similar services. They are both the same Microsoft cloud MPP database engine, both cover the same general T-SQL surface area, both scale in terms of DWUs, both talk about “Azure Synapse Analytics” in the documentation and description, but one is deployed to an Azure SQL Server, and one is deployed to a Workspace.

We need to be very clear about this because they both operate slightly different. Again, this post is focused on Azure Synapse Analytics Dedicated SQL Pools (Gen 2). An alternate post is available for Dedicate SQL Pools (Formerly SQL DW) if you need information about that deployment option.

Let’s get started.

What are my options?

There are two types of logging that can be enabled: Auditing and Diagnostics. Auditing tracks a set of database events like queries and login attempts. Diagnostics will make copies of a specific set of system DMVs periodically which can be helpful because Synapse only stores a few thousand records depending on the DMV.

Everything from here assumes that you have a Log Analytics workspace created and a Synapse Workspace with a Gen 2 dedicated SQL pool deployed.

Enabling Auditing

Auditing can be enabled at the workspace level, which will cover all databases on the workspace automatically, or at the individual database level.

First, navigate to your Synapse Analytics Workspace or dedicated SQL pool in the Azure Portal. My screenshots will show the configuration from the dedicated SQL pool, but the same setting can be found under the label of Azure SQL Auditing at the workspace level. From here, select Auditing from the Security section.

Next, toggle the Enable Azure SQL Auditing to the on position. Next, check the boxes for the locations where you would like the log to be written, in this example we are going to focus on Log Analytics. Select a log analytics workspace to which the data will be written. Click Save once complete.

Enabling Diagnostics

While Auditing can be turned on for a database by either enabling it at the workspace or database level, diagnostics must be enabled at the database level. There are diagnostics at the workspace level, but those are different metrics and do not apply to monitoring the dedicated SQL pool.

If you’re not there already, navigate in the Azure Portal to your dedicated SQL pool. Select Diagnostic Settings from the Monitoring section.

Next, click the Add diagnostic setting button.

Provide a name for this diagnostic setting. Check the box next to Send to Log Analytics workspace. Select the subscription and workspace to which the DMVs will be written. Select which DMVs you want to log. Finally, click Save.

Remember, the diagnostic settings each correspond to a different DMV that will be copied to your Log Analytics workspace. If you need additional DMVs you’ll need to roll your own auditing solution at this time.

Reading the Logs

Data will take several minutes to show up in Log Analytics. I’ve had the initial population take up to 30 minutes. After that the data should show up much more quickly, within several minutes. Connect to your database and run a query or two to ensure we have some user queries show up in the logs.

For my example I have run the following query:
SELECT COUNT(*), GETDATE(), DB_NAME() FROM IMDB.stage_Person

To view the logs, you can navigate to your log analytics workspace, or select Logs from your dedicated SQL pool’s Monitoring section. This can be found directly beneath the Diagnostic Settings option used in the last section.

Viewing the Audit Log

Within the Log Analytics Workspace you will see a group of tables with the prefix Synapse* and at least one table prefixed with SQL*. More tables may appear if you are using auditing for Azure SQL Database. The audit setup in the first section of this post will be logged to the SQLSecurityAuditEvents table.

This sample query pulls a few of the relevant columns from the SQLSecurityAuditEvents table.

SQLSecurityAuditEvents
| project
    ResourceGroup,
    LogicalServerName,
    DatabaseName,
    EventTime,
    Category,
    ActionName,
    IsServerLevelAudit,
    Succeeded,
    tostring(SessionId),
    ClientIp,
    HostName,
    ServerPrincipalName,
    DurationMs,
    Statement,
    TenantId,
    _ResourceId

Here you can see the query I ran in the log with an action of BATCH COMPLETED.

Viewing the Diagnostic Logs (DMVs)

Where all audit events are logged to the same table, regardless of the event type, each of the DMVs enabled in Diagnostic settings gets logged to its own table. Expanding on the list of DMVs from earlier in the post we can see the Log Analytics table closely matches the diagnostic setting name with a prefix of SynapseSqlPool.

This sample query pulls a few of the relevant columns from the SynapseSqlPoolExecRequests table.

SynapseSqlPoolExecRequests
| summarize
    TimeGenerated = max(TimeGenerated),
    ResourceGroup = any(ResourceGroup),
    LogicalServerName = any(LogicalServerName),
    DatabaseId = any(toreal(DatabaseId)),
    StartTime = max(StartTime),
    EndCompileTime = max(EndCompileTime),
    Category = any(Category),
    Status = min(Status),
    Command = any(Command)
    by RequestId

Here you can see the query I ran in the SynapseSqlPoolExecRequests table.

Wrapping it up

There is a lot more that can be accomplished with KQL queries in Log Analytics. You may even take the queries found in the accompanying Dedicate SQL Pools (Formerly SQL DW) post and combine them to give a single view of all your dedicate SQL pools.

The key take learning from this walkthrough is:

1. Auditing and Diagnostics are different options and enabled in different locations.
2. Auditing writes all the data to a single table no matter the action.
3. Diagnostics pull a few key DMVs from Synapse and write them to the log. Each DMV is written to its own table in Log Analytics.
4. Something not discussed is that there is the potential on highly used systems for some audit or diagnostic records could be missed.

This post isn’t about Azure Monitor though. Not directly anyway.

Azure Synapse is no different than any other application or service in your environment. It serves a purpose in the architecture and as a result needs special attention from people with a specific skillset, in this case DBAs and developers. That’s where Azure Monitor comes into the picture and what we will be discussing today: how to make sense of the integration between auditing in Dedicated SQL Pools (Formerly SQL DW) and Azure Monitor. The focus will be on Dedicated SQL Pools (Formerly SQL DW) not Azure Synapse Analytics, but more on that in a moment.

First, those with an eye for detail will notice that this post has Log Analytics in the title and not Azure Monitor. That’s true. A very brief explanation so everyone has things straight in their heads. I put Log Analytics in the title because when you set up auditing or diagnostics the option is labeled “Send to Log Analytics workspace” not “Send to Azure Monitor” and I wanted people to be able to find this post. Azure Monitor is a set of functionalities for collecting and analyzing logs. There are some prebuilt integrations and visualizations with some Azure services like Key Vault or Storage Accounts Then there is Log Analytics, a functionality inside Azure Monitor which also happens to be where Synapse writes its logs, for storing and querying all log data. Queries are written in a Kusto Query Language or KQL.

Second, let’s be sure a few very important details are clear. When referring to Azure Synapse there are two distinct but very similar services. They are both the same Microsoft cloud MPP database engine, both cover the same general T-SQL surface area, both scale in terms of DWUs, both talk about “Azure Synapse Analytics” in the documentation and description, but one is deployed to an Azure SQL Server, and one is deployed to a Workspace.

We need to be very clear about this because they both operate slightly different. Again, this post is focused on Dedicated SQL Pools (Formerly SQL DW). An alternate post is available for Azure Synapse Analytics Dedicated SQL Pools (Gen 2) if you need information about that deployment option.

Let’s get started.

What are my options?

There are two types of logging that can be enabled: Auditing and Diagnostics. Auditing tracks a set of database events like queries and login attempts. Diagnostics will make copies of a specific set of system DMVs periodically which can be helpful because Synapse only stores a few thousand records depending on the DMV.

Everything from here assumes that you have a Log Analytics workspace created and a Dedicated SQL Pools (Formerly SQL DW) deployed.

Enabling Auditing

Auditing can be enabled at the logical server level, which will cover all databases on the logical server automatically, or at the individual database level.

First, navigate to your logical SQL server or dedicated SQL pool in the Azure Portal. My screenshots will show the configuration from the dedicated SQL pool, but the same setting can be found under the label of Auditing at the logical SQL server level. From here, select Auditing from the Security section.

Next, toggle the Enable Azure SQL Auditing to the on position. Next, check the boxes for the locations where you would like the log to be written, in this example we are going to focus on Log Analytics. Select a log analytics workspace to which the data will be written. Click Save once complete.

Enabling Diagnostics

While Auditing can be turned on for a database by either enabling it at the logical SQL server or database level, diagnostics must be enabled at the database level. There are no diagnostics at the logical SQL server level.

If you’re not there already, navigate in the Azure Portal to your dedicated SQL pool. Select Diagnostic Settings from the Monitoring section.

Next, click the Add diagnostic setting button.

Provide a name for this diagnostic setting. Check the box next to Send to Log Analytics workspace. Select the subscription and workspace to which the DMVs will be written. Select which DMVs you want to log. There is an additional SQLSecurityAuditEvents entry in my screenshot which is there because I already enabled the database audit. We'll ignore that for the time being. Finally, click Save.

Remember, the diagnostic settings each correspond to a different DMV that will be copied to your Log Analytics workspace. If you need additional DMVs you’ll need to roll your own auditing solution at this time.

Reading the Logs

Data will take several minutes to show up in Log Analytics. I’ve had the initial population take up to 30 minutes. After that the data should show up much more quickly, within several minutes. Connect to your database and run a query or two to ensure we have some user queries show up in the logs.

For my example I have run the following query:
SELECT *, GETDATE(), DB_NAME() FROM dbo.MyTable

To view the logs, you can navigate to your log analytics workspace, or select Logs from your dedicated SQL pool’s Monitoring section. This can be found directly beneath the Diagnostic Settings option used in the last section.

Viewing the Audit Log

Within the Log Analytics Workspace you may see a group of tables with the prefix Synapse*. These tables correspond to diagnostic settings configured on the Synapse Analytics Workspaces. If you do not have any Synapse Workspaces these tables will likely not show up. More tables may appear if you are using auditing for Azure SQL Database. The audit setup in the first section of this post will be logged to the AzureDiagnostics table.

This sample query pulls a few of the relevant columns from the AzureDiagnostics table by narrowing down the data to the SQLSecurityAuditEvents category which is used for the Audit log. For Synapse Workspace Dedicated SQL Pools, as opposed to the "formerly SQL DW" SQL Pools that are the focus of this post, the data is written to a separate table called SQLSecurityAuditEvents rather than being a category of log within the AzureDiagnostics table.

AzureDiagnostics
| where Category == "SQLSecurityAuditEvents"
| project
        ResourceGroup,
        LogicalServerName_s,
        database_name_s,
        event_time_t,
        Category,
        action_name_s,
        tobool(is_server_level_audit_s),
        tobool(succeeded_s),
        tostring(session_id_d),
        client_ip_s,
        host_name_s,
        server_principal_name_s,
        tolong(duration_milliseconds_d),
        statement_s,
        TenantId,
        _ResourceId
| project-rename
        ResourceGroup,
        LogicalServerName=LogicalServerName_s,
        DatabaseName=database_name_s,
        EventTime=event_time_t,
        Category,
        ActionName=action_name_s,
        IsServerLevelAudit=is_server_level_audit_s,
        Succeeded=succeeded_s,
        SessionId=session_id_d,
        ClientIp=client_ip_s,
        HostName=host_name_s,
        ServerPrincipalName=server_principal_name_s,
        DurationMs=duration_milliseconds_d,
        Statement=statement_s,
        TenantId,
        _ResourceId

Here you can see the query I ran in the log with an action of BATCH COMPLETED.

Viewing the Diagnostic Logs (DMVs)

Where Synapse Workspaces write DMV data to separate tables, for Dedicated SQL Pools (Formerly SQL DW) each of the DMVs enabled in Diagnostic settings gets logged to the same AzureDiagnostics table used in the Auditing section above, but with a category that corresponds to the diagnostic setting/System DMV.

This sample query pulls a few of the relevant columns from the AzureDiagnostics table by narrowing the data down to the ExecRequests Category. This data corresponds to the system DMV sys.dm_pdw_exec_requests.

AzureDiagnostics
| where Category == "ExecRequests"
| project
    ResourceGroup,
    LogicalServerName_s,
    DatabaseId_d,
    StartTime_t,
    EndCompileTime_t,
    Category,
    Status_s,
    RequestId_s,
    Command_s
| summarize
    ResourceGroup = any(ResourceGroup),
    LogicalServerName = any(LogicalServerName_s),
    DatabaseId = any(DatabaseId_d),
    StartTime = max(StartTime_t),
    EndCompileTime = max(EndCompileTime_t),
    Category = any(Category),
    Status = min(Status_s),
    Command = any(Command_s),
    record_count = count()
    by RequestId_s

Here you can see the query I ran focused on the ExecRequests category.

Wrapping it up

There is a lot more that can be accomplished with KQL queries in Log Analytics. You may even take the queries found in the accompanying Azure Synapse Analytics Dedicated SQL Pools (Gen 2) post and combine them to give a single view of all your dedicate SQL pools.

The key take learning from this walkthrough is:

1. Auditing and Diagnostics are different options and enabled in different locations.
2. Auditing writes all the data to a single table no matter the action.
3. Diagnostics pull a few key DMVs from Synapse and write them to the log. Each DMV is written to the same table in Log Analytics each with a different category.
4. Something not discussed is that there is the potential on highly used systems for some audit or diagnostic records could be missed.

Maybe my issue isn't with Key Vault so much as it is with soft-delete.

While doing some testing for TDE protectors on Managed Instance I had to remove a key and recreate it for part of my test setup. The key I deleted earlier in the day was named ManagedInstance. So when I went to create a new key called ManagedInstance in the Azure Portal I received an error message. No big deal, I had this one in the bag. Go to PowerShell, find the key, remove the key, go back to the portal and try again. Unfortunately, I hit a snag. Let's walk through the code and see where I went wrong.

To remove a soft-deleted key you must first locate the key's name or Id. To do this we will use the Get-AzKeyVaultKey cmdlet with the parameter -InRemovedState to show all the deleted, but not really deleted (aka soft-deleted) keys.

Get-AzKeyVaultKey -VaultName "bschacht-kv" -InRemovedState

Make note of the name or Id and move on to the next cmdlet and we will be done...or we should have been done anyway. We will use Remove-AzKeyVaultKey, specify the key name to be removed, make note that this key is in a removed state and run the cmdlet. Except here is where I ran into the somewhat unhelpful error message Remove-AzKeyVaultKey : Operation returned an invalid status code 'Forbidden'.

Remove-AzKeyVaultKey -VaultName "bschacht-kv" -Name "ManagedInstance" -InRemovedState

It turns out this error message stems from the user running the command (myself in this case) doesn't not have purge permissions explicitly granted in the Key Vault. Running this cmdlet requires the purge permission, no inherited RBAC or explicitly granted RBAC roles no matter how high up the food chain (Contributor, Owner, etc.) will give you the ability to remove a soft-deleted key. You must go to the Azure Portal, navigate to your Key Vault, go to Access Policies, and grant the user running this cmdlet the Purge permission for keys which is in a special section on the dropdown menu called "Privileged Key Operations".

Now, let's run the Remove-AzKeyVaultKey cmdlet again, verify that we are sure we want to remove the key permanently, and no response means success!

There you have it. In the end it was a very simple fix. Unfortunately that error message seems to be less than well documented. Thankfully, if you go look at the documentation for the cmdlet under example 3 (which, let's be honest, who does that unless they run into a problem and have to go looking for an answer) you will in fact see the purge permission noted. Too bad I don't read instructions. I just go run things and see what happens. It's a great way to learn...as long as you aren't doing it in production.

A few prerequisites for today:

• Azure SQL Managed Instance (any service tier, any size)
• A database (any user database will do, even an empty one)
• Azure Key Vault (with soft-delete enabled)
• Storage Account (optional, if you want to test the backup/restore process)

Switching to Bring Your Own Key

Bringing your own key for TDE is actually not a requirement to use Managed Instance. By default, the service will manage its own TDE key. However, that comes with a few limitations. The main one being you cannot create user initiated backups when using the service-managed key. The built-in backup/restore functionality still works just fine with service-managed keys though.

The key settings can be found by navigating the Azure Portal to your Managed Instance and clicking on the Transparent Data Encryption option in the service navigation panel. Then change the

The next step is to identify the key that will be used for encrypting the databases. This can be done in one of two ways: choosing a Key Vault and Key from inside the tenant or specifying a key identifier URL for a particular key. The second option is particularly useful if the user doing the configuration does not have access to the list of Key Vaults/Keys but is given a key identifier that should be used. The two configuration options can be seen in the following screenshot.

Important Notes

A few important pieces of information to note on from these settings.

First, the Managed Instance must be granted access through a Key Vault access policy. If that has not already been completed an attempt will be made to add the Managed Instance with the appropriate permissions (Get, Wrap Key, Unwrap Key). The access policy is defined and enforced at the vault level, not the individual key, secret, or certificate level. So an instance with Get access to one key in this vault will have Get access to all the keys in this vault.

The second piece of important information is the check box labeled Make the selected key the default TDE protector. By checking that box, all databases will be encrypted using the selected key. The situation may arise where you would like to simply restore a database protected with a particular key, but you do not wish to encrypt the databases on the server with that key. In that scenario, you will want to add the key to the Managed Instance, but NOT select the option to make it the default TDE protector.

Next, the Managed Instance and Key Vault must be in the same Active Directory tenant. At this time it is not supported to backup keys from one subscription and move them to another subscription, the keys must remain in the same subscription but can be moved to another vault someplace else in the same geography (moving between Key Vaults in North America that reside in the same subscription is supported, moving between a Key Vault in North America and one in Europe is not supported even if they are in the same subscription.

Finally, changing the default TDE protector does not encrypt existing backups with the new key. So it is important for business continuity that you retain the prior TDE key(s) in order to be able to restore backups. If you switch from customer-managed key back to service-managed key you must retain the copy of your key in Key Vault at least until the system's point-in-time restore window has rolled off for the customer-managed keys (i.e. retain the customer-managed key for 7-35 days depending on your configuration after switching back to service-managed key).

There is one and only one key that will protect the databases on an instance.

• Key marked as "Make the selected key the default TDE protector."
  • Can restore databases encrypted with this key.
  • Will be the key used to encrypt the user databases on the instance.
  • A database restored with this key will also be encrypted with this key.
• Key NOT marked as "Make the selected key the default TDE protector."
  • Can restored databases encrypted with this key.
  • A database restored with this key will be encrypted with the key marked as the protector after the restore process is completed.

See it in Action - Changing to Customer-Managed Key

As a starting point I have a Managed Instance with a single database, TDETesting, that is encrypted with the service-managed key.

We can use a simple T-SQL query to get the encryptor thumbprint. This same query will help us identify when the key has changed.

SELECT
    DB_NAME(database_id) AS database_name,
    encryption_state,
    CASE encryption_state
        WHEN 0 THEN 'No Encryption Key Present. Database Not Encrypted.'
        WHEN 1 THEN 'Database Unencrypted'
        WHEN 2 THEN 'Database Encryption in Progress'
        WHEN 3 THEN 'Database Encrypted'
        WHEN 4 THEN 'Encryption Key Change in Progress'
        WHEN 5 THEN 'Database Decryption in Progress'
        WHEN 6 THEN 'Certificate or Key Change in Progress'
        ELSE 'Unknown Status'
        END AS encryption_state_descriptoin,
    encryptor_thumbprint
FROM sys.dm_database_encryption_keys

We can see that based on the query results the current thumbprint is 0x051082BA88D2882F551C530B74EB6F4380843029.

Before the switching to the customer-managed key, this is what the access policy settings look like on my Key Vault. The only user that has rights to do anything is myself.

From my Managed Instance I will switch to customer-managed key.

Select my Key Vault.

And on the select a key blade, I will choose create a new key as I do not already have a key to use.

I will keep the option on generate to create a key and provide a name. I'm going to use ManagedInstance, but there is not a specific name that is required. Then Click create.

With all the options configured click Save.

After the settings are applied we can see the change in access policy on the Key Vault has added my Managed Instance with Get, Wrap, and Unwrap permissions.

Additionally, running our T-SQL script we can see the encryptor thumbprint has now changed from 0x051082BA88D2882F551C530B74EB6F4380843029 to 0x71ABFFF1EAA10687BD43878C75E7F2D1744E285C.

With the customer-managed key in place, we can now run a create a user backup of the database.

BACKUP DATABASE TDETesting TO URL = 'https://bschachtstor.blob.core.windows.net/backup/TDETesting.bak' WITH COPY_ONLY

If we then look at the files in that backup, we will see that it is encrypted with the same thumbprint as our database showed after switching to customer-managed keys.

RESTORE FILELISTONLY FROM URL = 'https://bschachtstor.blob.core.windows.net/backup/TDETesting.bak'

See it in Action - Restoring a Backup

What happens when we need to restore a database that has been backed up under a different key? In this particular case I have a backup that was created under a different customer-managed key. This screenshot shows a different thumbprint than the system or current customer-managed key, 0x190256228610BBE409C0345597D49ABB5A40EFA6.

System key thumbprint: 0x051082BA88D2882F551C530B74EB6F4380843029
Current key thumbprint: 0x71ABFFF1EAA10687BD43878C75E7F2D1744E285C
Backup key thumbprint: 0x190256228610BBE409C0345597D49ABB5A40EFA6

In the Azure Portal, I changed the key to the new ManagedInstance-AlternateKey that I added to my Key Vault. However, I did not choose the Make the selected key the default TDE protector option as I want to continue to encrypt my databases with the key used previously in this example but restore databases that are encrypted with the ManagedInstance-AlternateKey.

After saving the change I can now run the restore of my database that is encrypted with the ManagedInstance-AlternateKey (thumbprint 0x190256228610BBE409C0345597D49ABB5A40EFA6). After the restore completes we can see that the database encryptor has been switched to the thumbprint 0x71ABFFF1EAA10687BD43878C75E7F2D1744E285C that is used on my existing database. As mentioned previously, the current implementation is such that all databases are encrypted with the same key. So while the key ending in EFA6 is required for the restore process the database is then encrypted with the TDE protector key ending in 285C.

What Keys Are Associated with my Managed Instance?

We have seen how we can assign a customer-managed key to Managed Instance, add an additional key for restore purposes, how to see which key is encrypting the database, and how to restore backups encrypted with non-TDE Protector keys. There can be confusion as to what keys are available though because the Azure Portal only shows the last key assigned. In our demo walkthrough that wasn't even the TDE Protector key. How then do we find what keys are available to our managed instance for restore commands and what key is the current TDE Protector?

For that we go to PowerShell.

Get-AzSqlInstanceKeyVaultKey will give a list of all keys currently associated with your managed instance. In order for a backup file to be restored to your Managed Instance the key with which the backup was encrypted must be listed. In my example there is a service-managed key and two customer-managed keys with the thumbprints from earlier in this post.

Get-AzSqlInstanceKeyVaultKey -ResourceGroupName "ManagedInstance-RG" -InstanceName "bschacht-sqlmi01"

However, there is no indication of the current TDE Protector. For that we will go to a different PowerShell cmdlet, Get-AzSqlInstanceTransparentDataEncryptionProtector.

Get-AzSqlInstanceTransparentDataEncryptionProtector -ResourceGroupName "ManagedInstance-RG" -InstanceName "bschacht-sqlmi01"

Removing a key is also a simple operation with the cmdlet Remove-AzSqlInstanceKeyVaultKey.

Remove-AzSqlInstanceKeyVaultKey -ResourceGroupName "ManagedInstance-RG" -InstanceName "bschacht-sqlmi01" -KeyId https://bschacht-kv.vault.azure.net/keys/ManagedInstance-AlternateKey/2603353efae04fbab840385dd43a590b

Wrapping Up

What other questions do you have about using TDE with Azure SQL Database Managed Instance? Let me know in the comments and I'll see if I can work through some examples with you.