Skip to Content
DocsOperationsHTTPRequest

HTTPRequest

HTTPRequest operation performs HTTP requests, parses the results, and stores the parsed result into one or several target tables. Optionally, if a source table is provided, HTTP request will be performed for each record of the source table. HTTPRequest supports all popular authentication methods such as OAuth, Basic Authentication, API key authentication, etc.

HTTPReqest is very powerful and flexible so that you can implement a number of use cases:

  • Download all (or subset) of records from a source object/table. HTTPRequest has a built-in mechnism to get the next batch of records until all records are downloaded - you do not have to implement such iteration in your scenario.

  • Insert/update/delete records in the source system. In this case you use the source table that contains records to be inserted/updated/deleted. HTTPRequest iterates over the source records, tranform each record into HTTP parameters or HTTP body (eg in JSON format), and perform the HTTP request. If the source being inserted/updated/deleted accepts a batch of records in one call, you can enable batching of the source records. The target table will contain the result of the HTTP requests. For example, in case of insert, the result usually contain ids genereted by the system for the inserted records.

  • Look up information from a data provider such as Dun & Bradstreet or Melissa Data. In this case you use the source table that contains records you want to look up. HTTPRequest iterates over the source records, tranform each record into HTTP parameters or HTTP body (eg in JSON format), and perform the HTTP request. The target table will contain the result of the HTTP requests. As in case of inserting/updating/deleting data you can benifit from batching of the source records is the data provider supports it.

Mapping Data Using Scripts

Many properties of the HTTPRequest operation are specified in Javascript. Javascript gives you a great flexibility in mapping request record to the HTTP request properties and parsing the HTTP response. Script-defined properties are marked with (script) in the title and usually have the editor area with black background. Scripts can consist of many statements but the last statement in the script must be a return statement with the result of the script. Scripts are preloaded with the Underscore.js library version 1.8.3. You can find documentation on Underscore.js at http://underscorejs.org .

Scripts can access parameters via a predefined $il variable: $il refers to a Javascript object that contains the parameters as its properties. For example, you can access the current source record via $il.data. Depending on the property defined by the script, the script can access different set of parameters. Here are a common set of parameters passed to all scripts:

  • $il.variables is an object where property name is variable name and the property value is the value of the variable. Variables include application variables, overwritten (in case of conflict) with the current execution context variables. As always the execution context variables include scenario parameters, but it is also extended with the HTTP source properties (if the source is specified).

  • $il.data represent record(s) from the source table. It is an object containing the current input record when bacthing is disabled and an array of objects when batching is enables. The source table: property name is column name and property value is the value of the column.

Configuring the Source Table

Select a source, space, and table. You can also enable batching of the records and specify the batch size (1000 by default).

Configuring HTTP Request

Source

Source must be of a source of the HTTPSource type. It specifies credentials and configures the authentication mechanism. For more information, see HTTPSource. Source is optional. You can omit it if the HTTP call does not require authentication. You can also just put authentication parameters explicitly as part of HTTP request configuration, which is useful for a quick test.

URL

URL is mandatory. There is a separate Parameter section for specifying URL parameters but you can include parameters in the URL as well. Parameters included in the URL will be overwritten by the parameters defined in the Parameter section.

Method

Method is mandatory. You can select from: GET, POST, DELETE, or PATCH.

Parameters

Parameters is a Javascript script with parameters passed via $il as described in the Mapping Data Using Scripts section. The script must return an object that represents HTTP request parameters and its values. Parameters returned in this object overwrites parameters specified in the URL if there is a conflict.

Headers

Headers is a Javascript script with parameters passed via $il as described in the Mapping Data Using Scripts section. The script must return an object that represents HTTP request headers and its values. If you specified an HTTPSource-type source, the headers returned by this script will be automatically extended with Authorization header depending on the source specification so you do not need to add them explicitly. If you do add Authorization header explicitly, it will overwrite the source Authorization header.

Body Type

Body type can be one of the following. When we refer to Body in this subsection, we mean the content of the Body text box when the script check box is unchecked and the value of the JavaScript return statement when the script check box is checked.

  • string: Body is a string that will be transmitted.
  • JSON -> stringify(): Body is JSON/JavaScript object that is serialized to a string that will be transmitted.
  • JSON -> form-data: Body is JSON/JavaScript object that represent form data: object property name is the form field name and the value of the object property is the form field value. You can transmit the content of a source table as a form field as follows: 
    {
 "filename": "<name of the file>",
 "Content-Type": "<content type>",
 "file":    
    {
     "sourceName": "<source name>",
     "spaceName": "<space name>",
     "tableName": "<table name>"
     }   
 }
    Content-Type above is used to construct the body, not added to the headers, text/plain by default.
  • JSON -> x-www-form-urlencoded: Body is JSON/JavaScript object that represent URL properties to be encoded: object property name is the URL property name and the value of the object property is the URL property value.
  • JSON -> octet-stream: Not supported anymore. Use JSON -> transmitSourceTable() instead. When you switch to JSON -> transmitSourceTable() you should add "Content-Type":"application/octet-stream" header explicitly if it is needed.
  • JSON -> transmitSourceTable(): Body is JSON/JavaScript object that specifies a source table the content of which will be transmitted. 
    {
 "sourceName": "<source name>",
 "spaceName": "<space name>",
 "tableName": "<table name>"
 "Content-Type": "<(optional) content type used for construct the body, not added to the header>"  
}

Body

By default, Body is text that corresponds to the selected body type. You can activate the check box in the title to evaluate it as a script as described the Mapping Data Using Scripts section. The result of the script is serialized to a string using JSON.stringify so the script usually used with Body type set to JSON

Interceptors

Interceptors allows you to post process the request to modify and extend it. Intercepters are defined as a Javascript that return an array of objects that represent interceptors. The interceptors are executed in the order they specified. The following intercepters are supported.

Amazon AWS Signature V4 (signWithAWSSigV4)

It adds “Authorization” header with the string constructed according to Amazon AWS Signature V4 . The signing algorithm requires that the host and the x-amz-date headers be present in the request prior to signing. The x-amz-date header must be formatted with the pattern “yyyyMMdd’T’HHmmss’Z’” (e.g. 20221101T233200Z).

Here is an example of the interceptor definition:

return [
{
    name: "signWithAWSSigV4",
    params: {    
        "region": "us-east-1",
        "serviceName": "execute-api",
        "accessKeyId": "AKIA57EN79UV2XXM5QMT",
        "accessKey": "eCnSR2lSvGsgNXlejg36Jjnl0SdwlNskKSSZ8N3N"
    }    
}
]

Request Extender (extendRequest)

It extends (or overwrites if already exists) the request with extra URL parameters and headers. It is usually used to implement custom request signature algorithm.

return [
{
    name: "extendRequest",
    params: {
        parameters: {
            "api_signature": "7a661860f2b67494a78493a09dba19b5d685447bf4ae9124"
        },
        headers: {
             "api_signature": "7a661860f2b67494a78493a09dba19b5d685447bf4ae9124"
        }
    }    
}
]

Options

Optional parameters of the HTTP request specified in the format of JSON object. For example:

{"readTimeout": "700"}

The following parameters are supported:

  • connectTimeout (type: string; default value: “600”) - connect timeout for the request in seconds
  • readTimeout (type: string; default value: “600”) - read timeout for the request in seconds
  • writeTimeout (type: string; default value: “600”) - write timeout for the request in seconds

Configuring Target

First you select Action, which can be save to file or parse JSON.

Save to File

If save to file is selected in Action, then the response body of the HTTP request will be saved in a file specified by Target source, Target space, Target table.

Parse JSON

If parse JSON is selected in Action, then the HTTP response body is assumed to be JSON that is parsed and passed to Parsing script. The script gets parameters via $il variable as described in the Mapping Data Using Scripts section with additional $il.response property. $il.response is an object that represents the HTTP response and has the following properties:

  • statusCode - integer that is the HTTP status code.
  • statusMessage - string that is the HTTP status message.
  • headers - object representing response headers.
  • body - object or array representing JSON response body.
  • bodyStr - string that is the response body before parsing to JSON

The script is evaluated and its result determines action. The script must return an object that has the following properties (all are optional):

  • data (optional) contains result of mapping the HTTP response onto one or several tables. It is an array of objects where each object contains data for a table and has the following properties:

    • data (mandatory) contains an array of objects representing table records. The object structure should match the data model. Object properties not matching any column in the model are ignored. Columns in the model that do not have corresponding properties in the model are set to null.

    • format(optional) defines the format of the data. Values are objects (the mapping will be applied that returns format: "table" or format: "JSON" then the loading mechanism is applied to the result), table, JSON. table is the default value.

    • sourceName (optional) - source name of the table. If not specified, then the Target source is used.

    • spaceName (optional) - space name of the table. If not specified, then default name space is used if SourceName is specified. If neither sourceName nor spaceName is specified, then the Target space is used.

    • tableName (optional) - name of the table. If not specified, then the Target table is used.

    • tableNameSuffix (optional) - name of the table constructed by adding the suffix to the Target table.

    • modelName (optional) - name of the model. If not specified, then the Model is used.

    • model (optional) - model in the following format

          model: {
          columns: [
              { name: "order_id", type: "text", position: 0 },
              { name: "quantity", type: "decimal", position: 1 } 	                
          ]
      },

    • mode (optional) can have of the following string values: "createThenAppend" - the table is created on the first iteration of downloading the HTTP result and then records are appended to the table on subsequent iterations (iterations are determined by nextUrl property as described below), "append"

      • records are inserted to the existing table.

    • tmpTable (optional) can have one the following string values: "true", "false". If "true", the table is created as temparary.

  • variables (optional) contains an object of variable names and values to be set. If the value is a string, the variable is set as application variable. If the value is an object, the object must have value property and scope property. The value of the scope property can be "executionContext" (in this case the variable is set in the execution context) or "application" (in this case the variable is set as application variable). Here is an example of setting two variables: one as application variable and another in the execution context.

    variables: {
    serverSourceName: "PostgreSQLSever",
    count: {
        value: "5",
        scope: "executionContext"
    }
}

  • doWhile (optional) allows to iterate over the result returned in batches. If the value of doWhile is boolean true value or an object then the HTTP request will be executed again. In case the value is an object it can have url property to set the url for the next HTTP request execution (useful when the source returns an URL for the next batch of data).

  • doWhileDelay (optional) - integer value in milliseconds that defines a period to sleep after each call in the doWhile iteration.

Last updated on
FormulaIfThenElse