Message Reprocessing

Overview

When a message from the downstream system is invalid or having issues with data, functional support teams will have two choices. Either they can request the source system to resend the message, or they can reprocess it by resubmitting the inbound message. The farmer option is time-consuming as it involves communication between source system support teams and integration functional support teams. The latter option is much more convenient and time-saving, especially when addressing real-time mission-critical issues.
Atomic Scope brings the ability to correct and reprocess messages by submitting to following endpoints.

1. WCF-SQL receive locations (BizTalk): In BizTalk integration implementations, it is common that resubmissions happen through SQL Receive locations. A database table acts as a queue to hold resubmitted messages and WCF-SQL receive locations poll them and processes them into the message box.

2. Http Endpoints: Atomic Scope can resubmit the message to Http Endpoints such as BizTalk WCF-WebHttp , BizTalk Http, logic apps, azure functions, and APIM endpoints. The solutions need to have some design considerations to utilize AtomicScope’s reprocessing capability.

3. Azure Service Bus Queues and Topics: User should be able to edit and resubmit the message body to service bus queues and topics. You can also mention the predefined system properties of queues and topics and also you should be able to mention the custom user properties.

4. Azure Event Grid: Atomic Scope allows you to resubmit messages to Azure Event Grid. During resubmitting the message, you shall aslo provide the Subject, Data Version and Event Type.

5. File Location: The message body can we wrapped up in a file based upon the file name given in the Atomic Scope portal. The file name also supports some of the BizTalk Macros.

Reprocessing using WCF-SQL receive location.

The reprocessing of the message using WCF-SQl receive locations involve following steps

Enable archiving

Enabling archiving on a stage is mandatory for the reprocessing. The archiving can be enabled at different processing elements as below

1. BizTalk Pipeline components: In BizTalk receive locations and send ports, Archive property of Atomic Scope pipeline component should be set to true.
   

2. Orchestrations: In orchestrations, orchestration activity logger component can be used to archive the messages.

varOrchestrationActivityLogger.Archive(msgOutboundOrder);

Create Reprocessing WCF-SQL Receive location

To poll the messages from the Atomic Scope database, we need to create a receive location with WCF-SQL receive location with below pipeline configuration.

The Atomic Scope Activity Logger pipeline component before the disassembly is left unconfigured, as the message will be Atomic Scope envelope containing multiple messages to be reprocessed.
The xml disassembler creates the individual messages from envelope schema.
The Atomic Scope logging component after the XML disassembler must be configured with Additional property value “ReprocessedMessage=true”. This gives clue to the component that the message is a reprocessed message. The pipeline component extracts the actual body of the message and resubmits it to BizTalk message box.
The WCF-SQL adapter for the receive location should have following configurations.

XmlStoredProcedureRootNodeName: ASReprocessRecords
XmlStoredProcedureRootNodeNamespace: http://AtomicScope.Schemas.Reprocess/Records 

PolledDataAvailableStatement:
select  count(*) from [dbo].[Tracking_Reprocessing] where [ReceiveLocationName] ='<Receive Location Name>'   and PollingState=0

PollingStatement:
exec Tracking_GetReprocessingMessages '<Receive Location Name>'

Orchestrations picking up single reprocessed messages via a WCF-SQL Receive Location

In case an orchestration needs to pickup reprocessed messages, the configuration of your WCF-SQL receive location looks slightly different.

To poll messages from the Atomic Scope database, we need to create a receive location with WCF-SQL receive location with below pipeline configuration.

Note that in contrast with messaging-only scenarios, the AdditionalProperties attribute contains the value ReprocessedSingleMessage=true.

Also, the configuration of the WCF binding looks a bit different. Under the Binding settings of the WCF-SQL receive location, provide the following values:

• XmlStoredProcedureRootNodeName: ASReprocessRecords
• XmlStoredProcedureRootNodeNamespace: http://AtomicScope.Schemas.Reprocess/Records
• PolledDataAvailableStatement: select count(*) from [dbo].[Tracking_Reprocessing] where [ReceiveLocationName] ='' and PollingState=0
• PollingStatement: exec Tracking_GetReprocessingMessage ''

Next, when developing the orchestration which will pick up the reprocessed messages, make sure that the following conditions are met:

• the orchestration should subscribe to the messagetype of the message that will be sent to the just created Receive Location
• the first shape of the orchestration must be a logical receive port, which is configure with Specify Later
• after the orchestation has been deployed, bind to the logical receive port to the newly created Receive Location

Configure Reprocessing

The reprocess configuration specifies the Biztalk WCF-SQL receive location to which the message will be resubmitted. The AtomicScope portal automatically lists all the receive locations with WCF-SQL adapter configuration.

Please refer the section “How to configure stage for reprocessing” for more information.
Note: Each transaction stage can be configured to reprocess the message to a WCF-SQL receive location. The routing and further processing needs to be handled by the solution implementation.

Reprocessing to Http Endpoints

AtomicScope brings ability to reprocess the messages to Http Endpoints. These Http Endpoints can be Logic Apps, Azure functions, BizTalk Http Receive locations or BizTalk WebHttp receive locations. below is the architenctural diagram for Reprocessing web end points.

Enable archiving

Enabling archiving for services such as Logic Apps, Azure functions, etc will be done in Atomic Scope Portal. Please refer to the section “How to enable archiving for Logic Apps.”

Enabling archiving for BizTalk web receive locations is same as the configuration discussed WCF-SQl reprocessing section.

Configure Reprocess under Stage Level

In AtomicScope portal, the failed transactions can be reprocessed. This can be achieved by configuring reprocess at a stage. The following steps must be followed to configure reprocess at stage level.

1. Click the Configure Reprocess button which will be present at a respective stage.

2. Click Select to select either of the two Endpoint types.

3. Selecting BizTalk Receive Location as Endpoint Type, it will show the configured receive locations with WCF-SQL adapters.

4. Select any of the receive locations and click Done and click Save to save the reprocess configuration for that particular stage.

5. Selecting Http Endpoint as Endpoint Type, it will prompt you to enter the Uri and you have select the respective method type as either PUT or POST. You can also able to Add or Remove headers.

6. Toggling eye icon near respective header will mark the particular header Sensitive, and that confidential header will not be shown to other users except Admin or Product Owner.

7. Enter the appropriate details and click Done and click Save to save the reprocess configuration for that stage.

Reprocessing to Azure Service Bus Queues and Topics

The user should be able to edit and reprocess the message body to Serivce Bus Queues and Topics with the default System properties and also with the customizable user properties.

Configure Reprocess

In AtomicScope portal, the failed transactions can be reprocessed. This can be achieved by configuring reprocess at a stage. The following steps must be followed to configure reprocess at stage level.

1. Click the Configure Reprocess button which will be present at a respective stage.

2. Click Select to select either Azure Service Bus Queue or Azure Service Bus Topic as Endpoint Type.

3. On Selecting Azure Service Queue, you need to mention the Service Bus Connection String and the Service Bus Queue Name. And also you should be able to configure the system properties and also user properties for Serivce Bus Queue.

4. On Selecting Azure Service Bus Topic, you need to mention the Service Bus Connection String and the Service Bus Topic Name. And also you should be able to configure the system properties and also user properties for Service Bus Topic.

5. After mentioning the respective details, click Save to save the reprocess configuration. The configured reprocess settings is editable and also it can be deleted at any point of time.

Reprocessing to Azure Event Grid

The user should be able to edit and reprocess the message body to Event Grid with the Subject, Date Version and also you can mention the Event Type while reprocessing.

Configure Reprocess

In AtomicScope portal, the failed transactions can be reprocessed. This can be achieved by configuring reprocess at a stage. The following steps must be followed to configure reprocess at stage level.

1. Click the Configure Reprocess button which will be present at a respective stage.

2. Click Select to select Azure Event Grid Endpoint Type.

3. You should enter the Topic Endpoint, Topic Key, Subject and Data Version which are mandatory paramters. And also if you have any event type filters to capture Atomic Scope events, you shall also mention the Event Type if it's necessary.
4. After entering the appropriate details, Click Save to save the Reprocess Configuration. The configured Reprocess Configuratoin is editable and also it can be deleted at any point of time.

Reprocessing to File Location

Atomic Scope supports the capablity of reprocessing to file location. User should be able to configure reprocess for file location in Business Process Configuration Page. In the tracking grid, using the configuraiton user should be able to reprocess to repsective file location.

Configure Reprocess for File Location

In AtomicScope portal, the failed transactions can be reprocessed. This can be achieved by configuring reprocess at a stage. The following steps must be followed to configure reprocess at stage level.

1. Click the Configure Reprocess button which will be present at a respective stage.

2. Click Select to select File Location Endpoint Type.

3. Enter the appropriate file path and this file path can also be an UNC Share path. Once you have entered the file path, you can test connection whether you have access to the respective file location.

4. And then enter the file name with the proper extension(.xml , .json) and the file name can also contain BizTalk Macros. The following are the supported Macros,

   • %SourceFileName%
   • %MessageID%
   • %time%
   • %datetime%
   • %datetime_bts2000%

5. Once you have entered the file name and file path, you should be able to save the reprocess configuration.

File overwrite option for File Location reprocess

You can overwrite the file content if file already exist in the configured path. You have the switch button for enabling the overwrite opiton. If its enabled, the file automatically overwrite the content. If not, it will throw the warning as file already exist while reprocessing.

Reprocess Failed Transaction

Once all the configuration is done in the tracking UI, select a failed transaction as shown below.

On clicking the tracking data, you will see a graphical flow in a blade as shown below.

Reprocess

When you click on the button reprocess, you will be directed to a blade in which you will be allowed to edit the archived message and reprocess based upon your reprocess configuration.