Defining Automatic Block-Retry Policies

In order for a submission’s processing to be completed, each block in its flow must finish its task successfully. If a block fails, the submission becomes halted, which may cause the submission to breach its SLA. When you define automatic retry policies for blocks, you minimize the chances of submissions being halted due to temporary outages, which prevents processing delays from occurring. For each retry policy, you can specify the number of times blocks should be retried, along with the amount of time that should pass between retry attempts.

Defining block-retry policies

You can define block-retry policies at the following levels:

  • The system level
    • This policy is the default for all blocks. 
  • The flow level
    • Each flow-level policy overrides the system-level policy for the blocks in the flow.
  • The block level
    • Each block-level policy overrides the system-level and flow-level policy for the block.

When defining these policies, note that each retry may not occur after the exact amount of time specified for the retry interval. Retries are scheduled to occur after their retry intervals pass, but depending on the priority of the blocks’ tasks, they may not be executed at their scheduled times.

This article describes how to define block-retry policies in the application. If you’re creating custom flows, see the Flows SDK documentation to learn how to define these policies with the Flows SDK.

Defining the default block-retry policy

To define the system-level block-retry policy: 

1.  Go to <instance_url>/admin/hyperflow/wfeconfig/ , and click the number in the ID column.

2.  In the Default error handling policy field, enter the details of your retry policy in JSON format:

{
 "block_error_retry_policy": {
"method": "<method_name>",
"retry_count": <number_retries_before_failure>,
"retry_interval_seconds": <base_number_of_seconds_between_retries>
 }
}

The table below describes the valid method values and their effects on retry-attempt intervals:

method value Effect on retry-attempt intervals
FIXED The number of seconds specified in retry_interval_seconds is used as the retry-attempt interval for all retry attempts.
LINEAR_BACKOFF

The number of seconds between retry attempts is calculated as follows:

retry_interval_seconds * the number of the attempt

Example

If retry_interval_seconds = 5, the system waits 5 seconds to initiate the first retry attempt (5 * 1). If that attempt fails, the system waits 10 seconds to initiate the second attempt (5 * 2). If that attempt fails, the system waits 15 seconds to initiate the third attempt (5 * 3), and so on.

EXPONENTIAL_BACKOFF

The number of seconds between retry attempts is calculated as follows:

retry_interval_seconds * 2^(the number of the attempt - 1)

Example

If retry_interval_seconds = 5, the system waits 5 seconds to initiate the first retry attempt (5 * 2^(1-1) = 5 * 1 = 5). If that attempt fails, the system waits 10 seconds to initiate the second attempt (5 * 2^(2-1) = 5* 2 = 10). If that attempt fails, the system waits 20 seconds to initiate the third attempt (5 * 2^(3-1) = 5 * 2^2 = 20), and so on.

Note that the difference between LINEAR_BACKOFF and EXPONENTAL_BACKOFF only takes effect from the third attempt onward.

 

Example policies

A policy where up to 3 retry attempts will be made with 60-second intervals between each attempt

{
 "block_error_retry_policy": {
"method": "FIXED",
"retry_count": 3,
"retry_interval_seconds": 60
 }
}

A policy where up to 4 retry attempts will be made. The system waits 30 seconds to make the first retry attempt, then 60 seconds to make the second attempt, then 90 seconds to make the first attempt, and so on following the LINEAR_BACKOFF formula.

{
 "block_error_retry_policy": {
"method": "LINEAR_BACKOFF",
"retry_count": 4,
"retry_interval_seconds": 30
 }
}

A policy where up to 5 retry attempts will be made. The system waits 4 seconds to make the first retry attempt, then 8 seconds to make the second attempt, then 16 seconds to make the third attempt, and so on following the EXPONENTIAL_BACKOFF formula.

{
 "block_error_retry_policy": {
"method": "EXPONENTIAL_BACKOFF",
"retry_count": 5,
"retry_interval_seconds": 4
 }
}

Do not edit the default value of Retry logic, Payload threshold soft barrier in bytes, or Payload threshold hard barrier in bytes.

3.  Click Save.

Defining a block-retry policy at the flow level

To define a block-retry policy at the flow level:

  1. Go to the Flows page, and click on the name of the flow you want to define a block-retry policy for.

  2. In the flow settings, find the Retry Failed Blocks section, and click Edit.

  3. Select one of the following options:
    1. System Default
    2. Do not retry failed blocks in this flow
    3. Custom rule for this flow

  4. If you selected Custom rule for this flow, click More Options to reveal the options available to you. Then:
    1. In the first text box, enter the number of times blocks should be retried after they fail.
    2. In the second and third text boxes, enter the number of minutes and seconds, respectively, that should pass before the initial retry attempt.
    3. To determine how much time should pass between subsequent retry attempts, select one of the following options:
      • Linear Function — initial retry-attempt interval * the number of the attempt
        • Same as the LINEAR_BACKOFF method described for system-level retry policies
      • Exponential Function — initial retry-attempt interval * 2^(the number of the attempt - 1) 
        • Same as the EXPONENTIAL_BACKOFF method described for system-level retry policies
      • Keep wait time constant — The same retry-attempt interval is used between all attempts
        • Same as the FIXED method described for system-level retry policies.

  5. Click Confirm, then click Save to save your changes.

Defining a block-retry policy for an individual block

To define a block-retry policy for a specific block:

  1. Go to the Flows page, and click on the name of the flow containing the block you want to define a retry policy for.

  2. Click on the block in Flow Studio, find the Retry Failed Blocks section in its settings, and click Edit.

    RetryFailedBlocksBlockSettings.png

  3. Select one of the following options:
    1. Inherit from flow retry policy
    2. Do not retry when failed
    3. Custom rule for this block

  4. If you selected Custom rule for this block, click More Options to reveal the options available to you. Then:
    1. In the first text box, enter the number of times the block should be retried after it fails.
    2. In the second and third text boxes, enter the number of minutes and seconds, respectively, that should pass before the initial retry attempt.
    3. To determine how much time should pass between subsequent retry attempts, select one of the following options:
      • Linear Function — initial retry-attempt interval * the number of the attempt
        • Same as the LINEAR_BACKOFF method described for system-level retry policies
      • Exponential Function — initial retry-attempt interval * 2^(the number of the attempt - 1) 
        • Same as the EXPONENTIAL_BACKOFF method described for system-level retry policies
      • Keep wait time constant — The same retry-attempt interval is used between all attempts
        • Same as the FIXED method described for system-level retry policies

  5. Click Confirm, then click Save to save your changes.

Applying retry policies to subflows

If you’ve connected a flow block to another flow, and that flow is not a notification flow, the block’s retry policy applies to the connected flow and its subflows. 

If you’ve connected a flow blow to a notification flow, you need to define a separate retry policy for the connected flow.   

Best practices for retry policies

When determining the length of your retry-attempt intervals, keep the following factors in mind:

  • Your retry intervals should allow enough time for potential issues to be resolved. For example, if your retry interval is very short (e.g., a few seconds), retry attempts are unlikely to be successful. The intervals should also be long enough to account for the rate limiting of any services that may be connected to your blocks (e.g., API connections).
  • However, long retry intervals may impact submission-processing times, affecting your ability to meet any SLAs you may have in place for submissions.
Was this article helpful?
0 out of 0 found this helpful