LoadDelta

LoadDelta implements incremental load from a set of files.

Source (including Source, Space, Table name pattern, and Format Parameters) defines the input set of files. Table name pattern is a SQL LIKE expression: only file names matching the pattern will participate in the load.

Target defines the result tables that are used for the load. All target tables share the same prefix defined in Target tables prefix. The target tables are:

• <prefix>_master: is full data set that is the result of incremental load.

• <prefix>_delta: contains delta records loaded from the last incremental file.

• <prefix>_list_in: list of all files in the source (Source -> Space).

• <prefix>_list_in_filtered: is <prefix>_list_in filtered by Table name pattern and the date of the last LoadDelta execution, which is stored in <prefix>_last_modified application variable. So <prefix>_list_in_filtered contains a list of files that are loaded on the current execution of LoadDelta.

Key columns defines columns used to identify delta records in the master file.

LoadDelta has several execution modes:

• Create tables creates <prefix>_master and <prefix>_delta tables using the schema derived from a sample file, which one of the input file set. The sample file used for the schema is specified in the result of the execution in the Activity Log. Create tables mode also removes the <prefix>_last_modified application variable if it was ever created before.

• Load all loads all files matching the source pattern and sets the <prefix>_last_modified application variable to the time of the execution.

• Load delta loads all files matching the source pattern that were created since the time stored in the <prefix>_last_modified application variable. It sets the variable to the time of the execution.