Azure Data Factory Dynamic Content

In my work for the UConn AIMS health-data project (supported by Charter Solutions) we make frequent use of Azure Data Factory (ADF). One of the most useful features in ADF is "dynamic content”, which is a string or field that is determined at pipeline run-time. There is a mini-language for doing this and it is quite powerful. You can include static strings, ADF pipeline parameters, ADF variables, and ADF system/pipeline meta-data.

The syntax for creating dynamic content is tricky, especially if you are combining the different elements mentioned. In my case I was creating a JSON object, so there was additional complexity with embedded double-quotes and curly braces.

But once you get the syntax details correct, you can create any string you want, including a JSON object. The basic rules are:

• Any time you are combining elements, encase them in @ { concat ( item1, item2, item3, etc ) }. This pattern will solve 90% of your problems.
• Literal strings go between single quotes. Inside there, you can freely put double quotes and more curly braces.
• Note the different syntax for ADF parameters versus variables.

Here is a sample that uses literals, parameters and variables to create a valid JSON object.

@{concat(
'{"grouping_period_start":"', 
pipeline().parameters.grouping_period_start,
'", "grouping_period_end":"',
pipeline().parameters.grouping_period_end,
'", "adls_output_dir":"',
variables('adls_output_dir'),
'"}'
)}

The result of this dynamic content (with parameters and variables filled in as they were at run-time) is:

{
"grouping_period_start": "201701",
"grouping_period_end": "201806",
"adls_output_dir": "cdasfilesystem/feature/claims/3m/outputs/"
}