Parameters

Parameters are a powerful way to make tasks and project flexible.

Parameters can be added at the project and task level. Project parameters are passed down to all child tasks, task parameters will override project parameters with the same name.

∞ Creating Parameters

A parameter is a key value pair, separated with a = or :.

key=value
key:value

Parameters can include date/time variables, wrapped in a parse(...) tag.

@my_param=nice_day-parse(%d)

Values using inside a parse tag should be a strftime pattern.

∞ Parameters in Python

Parameters can be accessed in python scripts as environment variables:

import os
print(os.environ)

∞ Parameters in SQL

Parameters are inserted into sql queries in the DECLARE or SET statements. For example, if you have a query to print the date:

DECLARE @my_date datetime = getdate()
select @my_date

You could add a task or project parameter to override the value:

@my_date = getdate()-1

The sql run, and show in the sql preview, as:

DECLARE @my_date datetime = getdate()-1
select @my_date

∞ Parameters in Filenames

Atlas Hub has a few options for filename parameters as well.

∞ Wildcards

Filenames for a data source can use Unix shell-style wildcards to allow groups of files, or a file with an unknown name, to be found. Options are:

Here’s a few examples.

folder/*/sub_sub_folder/file_*.csv would match folder/sub_folder/sub_sub_folder/file_01_02_91.csv.

These patterns are most useful for picking up files with date patterns in them.

∞ Dates

Filename can also include all strftime patterns.

For example, my_file_%d_%Y.blah is a good name, and will pick up a file with the same day and year.

But, if you need a file from yesterday? That is also possible. Adding a -1, or +1, or +100, or any number, to most strftime options will do date math.

%d-1 will be yesterday day number.

%y-1 will be last year.

If today is the 1st day of the month, %d-1 will give you the last day of last month, and so on.

The last day and first day of the month are also available, using lastday, firstday, and firstday0.

∞ Parse Parameters

Parsed parameters can be used in filenames as well.

For example, and parameter of #cool=parse(%d_%Y) can be used in a filename my_date_#cool.csv. The output will be something like my_date_01_21.csv.