This video outlines essential best practices for Azure Synapse CI/CD pipelines to ensure efficient, reliable, and secure deployments across different environments.
Mind Map
Click to expand
Click to explore the full interactive mind map • Zoom, pan, and navigate
[Music] foreign
cicd fans welcome to this video series
on synapse ICD my name is rikunya and
I'm a support escalation engineer on the
Azure synapse analytics team in
Microsoft's public documentation for
synapse ICD you will find one section
fully dedicated to the best practices
in this video we will walk you through
some of the best practices in synapse
ICD like integrating only your
development workspace with your kit repo
and preparing your pools before
deploying your artifacts across the
environments so let's watch this video
and see some of the best practices in
synapse acidity in action starting with
this first best practice git integration
when working on a multi-environment
scenario you have the workspace in Dev
uat pre-prod prod
you should all only connect
The Source environment your Dev
environment with your git repository no
other environment should be connected
all other environments should be
live mode and the reason for this is
that you don't want your git repo to get
out of sync you don't want any changes
to occur outside your Dev environment so
this is where your development happens
anything that is detected in your uat
pre-prod prod should be fixed in this
lower environment this should be a rule
of thumb in synapse ICD to have your
only your Dev environment integrated
with your git repo another best practice
your branching strategy this branching
strategy should be as simple as possible
and you should use feature branches that
will allow your developers to create new
features and Bug fixes you should use a
consistent naming convention for your
feature branches so you can easily
identify the work that is being done in
the branch if we are talking about the
new feature or a bug fix and some suggestions
suggestions
for example users
then your username
then new feature or add fixes and then
your add fix
on a multi-project environment project
project a
a
my username
new feature
one creating this based on your main
here's an example another one project B
then John
new feature
in Project B then
then
project a at
at fixes
and you should keep this simple
structure here let me show you when developing
developing
SQL script
try to organize your artifacts as much
as possible for example project
a I mean project a right
I will go for project a
keep my artifacts organized same
same
for example
project a
bring this here
and if you go
your branches in devops you should see
this organization reflected here
project a hot fixes developers new
features Project B the same thing keep this
this
folder structure this branching strategy
simple another best practice you should
keep the main branch foreign
foreign
Ty Branch this is your collaboration branch
branch
the code in this Branch should build
cleanly and should always be current why
do you need these qualities in your main
branch because you want to make sure
that any feature branch that is created
by your developers it will start from a
good version of code how can you achieve
this you can set up Branch policies in
the main branch go here to your main branch
branch
and set Branch policies here set a
minimum number of reviewers for example
any change that lands in the main branch
should come from a pull request
now if I try to do
something like this here create an artifact
artifact print
print
this will fail to commit
I want to keep my main branch clean not
allowing any developer to make direct
changes in this branch if I want to
create something here
suppose that I was fixing this notebook great
this is my fix
notebook doesn't matter here the branch
just for demonstration purpose
I'm going to commit all
merging
my art fix
with Main
I will take this three files these three artifacts
I will create my pull request now I have
two reviewers to approve this
let me just quickly
turn this revert this setting for a One
reviewer just for this demonstration
and let me
was missing this option request is to
approve this my own changes Let me refresh
refresh okay
okay
I will go now to my main branch
and we'll see my changes reflected here
another best practice
before deploying your artifacts
make sure that you have your spark pools
compute pools in place with the same
naming to avoid the override
make sure that in uad in the Target
environments your spark pools
will also have the same naming as in
your Dev environment although this is
not mandatory it is suggested just to
avoid the override parameterization in
devops in case you need to keep the
compute pools
the names different across the environment
environment
you need to you must use the
custom parameter template foreign
this template parameters definition Json
where you override the reference name
for the Big Data pool if you keep the
the compute pools with the same naming
across the environments you don't need
to use this custom parameter template
and use the default if you don't do this
when you
try to deploy the
the workspace artifacts
you will hit one error
with a bad request because there is an
invalid reference name okay so your
deployment will fail with a bad request
when trying to deploy The Notebook another
another
best practice when you're setting up
use for example the account key
just an example and it is recommended
to keep separate key vaults for
different environments and why because
you don't want your developers to have
access to production secrets for example
in the same key vault
so the development key Vault should have
the secrets for the development
environment the uat key Vault with the
secrets for the uat environment and so
on and to avoid
um parameterizing each key Vault Secret
you can name the secrets with the same
naming across these environments so
having a a key Vault secret with the
same naming
for example
my storage Secret
and this storage secret name should
exist in all key vaults across all the
environments Dev key Vault uat pre-prod prod
prod
if the storage secret is different
across these key vaults you need to
artifact naming we recommend artifact
names to not have blank spaces you see
had an underscore separate the keywords
and the reason for this is that due to
arm template constraints you can have
issues when using the workspace
deployment test when resources contain
blank spaces in the in their names let
me show you here what I'm talking about foreign
go to the synapse workspace deployment
task when we are overriding the
look what will happen when I have my pipeline
pipeline
1 underscore type property suppose properties
properties
then reference name whatever
it's just to demonstrate a parameter
name with a blank space because our
artifact name here is the first keyword
and we have a blank space now let me set
here the the value
you will see this parameter here
with the corresponding value if you hit
this edit override parameters button
here you will see everything is
incorrectly formatted so for the task
this is the parameter name and this is
the value and then the value that you
have can break the the configuration the
fact that you that you use blank spaces
in your artifact names
last and if you are using a custom
parameter template make sure that the
properties that you have here under each
artifact category you really need to
expose these properties and generate the
parameters if not your arm template can
grow a lot in size depending on the
number of artifacts that you have and if
you were talking about thousands of
artifacts these Farm templates that are
generated to the workspace publish branch
branch
can grow a lot in file size and in git
we have a file size limitation that is
the maximum file size of 20 megabytes if
your work stamp template exceeds this
this limitation you will get an error
your publishing will fail due to this
limitation and git that will not allow
you to generate so you need to revise
the template to reduce as much as
possible the properties that you want to
expose for example suppose here under
the linked services
that you don't want to parameterize the
secret access key for example or the
account name doesn't matter so you
should remove these properties from the
definition file I hope you have enjoyed
this video
don't forget to share your thoughts and
ideas we would love to hear from you
don't forget to like comment and
subscribe to our Channel
I'll catch you up on the next one [Music]
Click on any text or timestamp to jump to that moment in the video
Share:
Most transcripts ready in under 5 seconds
One-Click Copy125+ LanguagesSearch ContentJump to Timestamps
Paste YouTube URL
Enter any YouTube video link to get the full transcript
Transcript Extraction Form
Most transcripts ready in under 5 seconds
Get Our Chrome Extension
Get transcripts instantly without leaving YouTube. Install our Chrome extension for one-click access to any video's transcript directly on the watch page.
