This video explains how to use custom parameter definition files in Azure Synapse Analytics to customize the ARM templates generated for CI/CD pipelines, enabling more flexible deployments across different environments.
Mind Map
Click to expand
Click to explore the full interactive mind map • Zoom, pan, and navigate
[Music]
Hello Synaps CI/CD fans. My name is Huna
and I'm a support escalation engineer on
the Azure Synapse Analytics team.
Welcome to this video series on Synapse CI/CD.
CI/CD.
In today's video, we will be talking
about custom parameterization of ARM
templates in your Synapse workspace and
how this parameterization can impact
your CI/CD workflow for these artifacts.
One of the advantages of having your
Synapse workspace integrated with Git is
to generate these ARM templates
containing all your workspace artifact properties.
properties.
These templates will be required in
scenarios where you need to deploy these
artifacts to a different environment or
in disaster recovery scenarios where you
need to recover all your workspace artifacts.
artifacts.
By default, these ARM templates are
generated based on the workspace default
or built-in parameter template. However,
when this default template is not
generating all the artifact properties
that you need, you are required to use
your own or custom parameter template.
And this next demo will show you how you
can achieve this. Here's my workspace,
my dev environment, and I want to deploy
my workspace artifacts to my UAT environment.
environment.
When you publish your changes from your
collaboration branch in your dev
environment, the publishing operation
will persist your pending changes to
live mode and then it will start
generating the ARM templates.
These ARM templates will be generated
based on the built-in and default
workspace template. This template will
expose only the necessary
properties or parameters for you to
deploy your artifacts to a target
environment. For example, if you go here
to the workspace publish where these ARM
templates are saved and you hit the
template parameters for workspace, you
will see here all the properties being
exposed by the default template. And
these properties of course they will
depend on the number of artifacts that
you have in your workspace. I have only
a linkage service here, a custom linkage
service. So the URL property is being
exposed for any link service. The
connection string is also a property
that is being exposed to this default
link service. We know that at least
these properties are being exposed and
you by the default um parameters template.
template.
You have other artifacts, other
properties that you would like to to
expose. For example, if you go to your notebooks,
notebooks,
you will see
here this reference name.
This notebook is using my spark pool. In
my dev environment, I have a single
spark pool. But in my UAT environment,
I have a couple of spark pools. And in
this workspace, I would like to my
notebooks to run using this UAT spark
pool. Here is a scenario where you need
to override your notebook
here. So you need to expose this
reference name property.
So you can override this spark pool
name. But this reference name property
is not being exposed by default by your
u default uh template. How you can
expose these properties that are not
being exposed by the default template?
So you need to use the custom parameters
template. So I'm here referencing the
Microsoft official docs and for you to
create this
template definition
file you need to pick this name and this
must be the exact naming that you will
provide to your file.
And what I'm going to do here, I'm going
to rename these files and call it default.
And what you need to do is to go to your
collaboration branch and in the root
root folder you need to create this file
with this exact name template parameters definition.json
definition.json
and you can copy this sample file from
the Microsoft documentation and paste it here.
here.
If you look at this file, the template
is exposing several properties under
each artifact category. Notebooks, SQL
script, pipelines, integration runtimes,
and so on.
And this definition file, we have a
specific syntax that you can
check here. the custom parameter syntax.
Every parameter will have the parameter
name, the property name and then uh you
can rename the file. For example, let me
see here in this connection string
I'm renaming this parameter and I'll
provide the data type. So this is this
syntax here. the action that can be
equal sign to keep the current value
hyphen don't keep the value for the
parameter and this pipe that is a
special case for connection strings
azure key volt and so on
so this is how you provide here the
syntax for example in this value in this
case for example I'm just renaming the
frequency property and call it fck
short name, but I will keep the default
data type and the default value.
You can add as much properties as you
need. For example, in the notebooks, I'm
just having here the reference name, but
I could have any other property that I
would need to expose.
How can you get these properties? For
example, if you go to one of these notebooks
notebooks
here, as you can see, note count,
course, memory, our example. Here it is
the one the property that we are
exposing reference name. So you need to
check the path to your property. For
example, in the notebooks, the reference
name is under the properties
big data pool reference name. This is
your path properties big data pool reference
reference
name. So if you go now to the workspace publish
publish
and not to the workspace publish to the
template parameters definition it is
notebooks properties big data pool
reference name. Now let me go to my dev
do this. I'm doing just this commit for
the sake of time here directly in the
main branch. I would need to as a best
practice feature branch pull request and
all that stuff. But I just for the sake
of this demonstration I just want to
save some time and to generate the new
ARM templates.
Now this will persist my change and
generate the two ARM templates. This is
just to compare the old ARM templates
that were generated using the default
workspace template versus the ARM
templates generated with the template
definition file.
Now back to the workspace publish.
Here it is. look at the default template
parameters with just three parameters
here to be exposed and I'll look at the
contents of this new file
where I'm exposing connection string but
now I'm exposing also the compute
properties for my integration runtime.
So now I'm exposing all these properties
for the integration runtime and I can
override any of these properties
including database name for SQL scripts
if I want to override the SQL pool name
that my script is using and it is our
spark pool reference that I have just
exposed. Right. So this is the logic
around the custom parameter template.
How can we use now this these parameters here?
here?
Now in our release pipeline
we are going to
let me just close this.
We are going to use this override
parameters section here in the task.
And what we need to do is to have the
and expose these parameters.
What I want to override is my sparkle name.
Copy this.
Put it here. And this will be my
And another example and we don't have I
was just thinking we don't have the URL
for our storage account that I wanted to
get the URL for my storage account. So I
was looking for that. So let's go here
to the main branch template parameters. So
So
link services
add it just to show you an example how
you can add a new property
to the template parameters definition
file. Let me go to my link service here
because I'm not getting the URL and this
should be this one. First thing the path
the URL belongs to the type properties
which in turn belongs to the properties.
So the path is properties type
properties URL.
Now let's go to our parameter definition.
definition.
The path properties type properties but
no sign of the URL. So I'm going to put
it here. Give it URL.
And now what I'm going to do equal sign
keep the same value then
this will tell okay let me call it just
for this demonstration how to rename
this my URL
and then
I will keep the data type as is. If I
want to change the data type, I would do
something like this. For example, commit.
commit.
Commit. And now what I need to do again just
just
Let's do a commit and a publish. Again,
this is just to save time
doing a
commit directly in the collaboration
branch is no good. First
First
and after generating this ARM templates
once this is finished we are going to
the workspace publish
and let me just expand this. Let's go to
the template parameters. It it is see
and let's compare with the default look
So I have renamed it instead of using
properties type properties. So, so this
is the path for the property. Instead of
having such a big name for this
parameter, I have just renamed to my URL.
URL.
property and provide the
Now I'm going to
create this release and I will deploy.
Let me see here. Something wrong with my
ARM templates. Let me check what's going on.
Okay, let me just delete this. It was
this was a previous
setup that I had. Now it's different.
Let me grab the workspace publish.
Add let me just reconfigure this to
point to the right location
template for workspace. This one not the
default. I don't want the default. I
want the custom templates here. Save.
Save.
Now I'm expecting my notebook to be
attached to the UAT spark pool here. So
there is no notebooks and no linkage
service. So I'm expecting my linkage
service now to use the UAT storage
not this storage. This is using a
different storage, but I want to attach
this linkage service to my UAT storage.
Let's go back to the deployment task
and wait for the task to complete.
Let's go to the UAD environment. Refresh
Refresh
and link service. hit it or ADLS to link
it service using the UAT staging. So
this was an example
on how you can use this
template parameters definition to
customize your ARM template parameters.
So you can override using your DevOps
deployment task
to deploy your Synapse artifacts to a
target environment using
different properties setting different
values. This is how you can use this
custom parameter templates. Let us know
your thoughts and share your ideas, tips
and tricks in the comments below. And
don't forget to like, comment, and
subscribe our channel. Thanks for
watching and I'll catch you on the next one.
Click on any text or timestamp to jump to that moment in the video
Share:
Most transcripts ready in under 5 seconds
One-Click Copy125+ LanguagesSearch ContentJump to Timestamps
Paste YouTube URL
Enter any YouTube video link to get the full transcript
Transcript Extraction Form
Most transcripts ready in under 5 seconds
Get Our Chrome Extension
Get transcripts instantly without leaving YouTube. Install our Chrome extension for one-click access to any video's transcript directly on the watch page.
