Parameters are a powerful way to make tasks and project flexible.
Parameters can be added at the project and task level. Project parameters are passed down to all child tasks, task parameters will override project parameters with the same name.
A parameter is a key value pair, separated with a
Parameters can include date/time variables, wrapped in a
Values using inside a
parse tag should be a strftime pattern.
Parameters can be accessed in python scripts as environment variables:
import os print(os.environ)
Parameters are inserted into sql queries in the
SET statements. For example, if you have a query to print the date:
DECLARE @my_date datetime = getdate() select @my_date
You could add a task or project parameter to override the value:
@my_date = getdate()-1
The sql run, and show in the sql preview, as:
DECLARE @my_date datetime = getdate()-1 select @my_date
Atlas Hub has a few options for filename parameters as well.
Filenames for a data source can use Unix shell-style wildcards to allow groups of files, or a file with an unknown name, to be found. Options are:
|?||matches any single character|
|[seq]||matches any character in seq|
|[!seq]||matches any character not in seq|
Here’s a few examples.
folder/*/sub_sub_folder/file_*.csv would match
These patterns are most useful for picking up files with date patterns in them.
Filename can also include all strftime patterns.
my_file_%d_%Y.blah is a good name, and will pick up a file with the same day and year.
But, if you need a file from yesterday? That is also possible. Adding a
+100, or any number, to most
strftime options will do date math.
%d-1 will be yesterday day number.
%y-1 will be last year.
If today is the 1st day of the month,
%d-1 will give you the last day of last month, and so on.
The last day and first day of the month are also available, using
Parsed parameters can be used in filenames as well.
For example, and parameter of
#cool=parse(%d_%Y) can be used in a filename
my_date_#cool.csv. The output will be something like