Dynamic Parametrization: Template Literals

Template literals are string literals allowing embedded expressions. Template literals (in addition to Javascript computed parameters) are used to dynamically parametrize operation’s parameters.

Evaluation of a template literal is the process of expending (aka substituting) embedded expressions with the results of their evaluation. There are three types of embedded expressions supported:

• Variable reference refers to application global variable or scenario parameter. Syntax of the variable reference is as follows: ${variable_name}.

• Source query is an query to be evaluated by a source. The first column of the first row of the query result substitutes the source query when template literal is evaluated. Syntax of the source query is as follows: $[source_name]{query}. For example, you can use a PostgreSQL source to generate a file name with the current date in it: companies-$[PgSQL]{select to_char(now(), 'YYYY-MM-DD')}.csv

• Expression is an expression in JavaScript. The expression is evaluated and the result is stringified using JSON.stringify. Expressions can refer to scenario parameters and variables as follows: $il.variables.varname. Syntax of expression is as follows: $((expr)). In expressions you do not add return statement as you do in scripts. For example:

  • you can increment loopCount variable: $((parseInt($il.variables.loopCount) + 1)). The following expression will be replaced with true if loopCount is greater than 5: $((parseInt($il.variables.loopCount) > 5)).
  • generate timestamp: $((new Date().toISOString() )) will be replaced with, for example, 2023-03-10T17:10:59.066Z

All parameter (such as source name, space name, table name, query, etc) of all operations (such as Load, Transform, etc) are evaluated as template literals before passing to the operations. Thus template literals is a powerful mechanism to parameterize operations with dynamic and context aware values.