Message Reprocessing

Overview

When a message from the downstream system is invalid or having issues with data, functional support teams will have two choices. Either they can request the source system to resend the message, or they can reprocess it by resubmitting the inbound message. The farmer option is time-consuming as it involves communication between source system support teams and integration functional support teams. The latter option is much more convenient and time-saving, especially when addressing real-time mission-critical issues.
Atomic Scope brings the ability to correct and reprocess messages by submitting to following endpoints.

Different types of message reprocessing

1. WCF-SQL receive locations (BizTalk): In BizTalk integration implementations, it is common that resubmissions happen through SQL Receive locations. A database table acts as a queue to hold resubmitted messages and WCF-SQL receive locations poll them and processes them into the message box.

2. Http Endpoints: Atomic Scope can resubmit the message to Http Endpoints such as BizTalk WCF-WebHttp , BizTalk Http, logic apps, azure functions, and APIM endpoints. The solutions need to have some design considerations to utilize AtomicScope’s reprocessing capability.

3. Azure Service Bus Queues and Topics: User should be able to edit and resubmit the message body to service bus queues and topics. You can also mention the predefined system properties of queues and topics and also you should be able to mention the custom user properties.

4. Azure Event Grid: Atomic Scope allows you to resubmit messages to Azure Event Grid. During resubmitting the message, you shall aslo provide the Subject, Data Version and Event Type.

5. File Location: The message body can we wrapped up in a file based upon the file name given in the Atomic Scope portal. The file name also supports some of the BizTalk Macros.

Reprocessing using WCF-SQL receive location.

The reprocessing of the message using WCF-SQl receive locations involve following steps

Enable archiving

Enabling archiving on a stage is mandatory for the reprocessing. The archiving can be enabled at different processing elements as below

1. BizTalk Pipeline components: In BizTalk receive locations and send ports, Archive property of Atomic Scope pipeline component should be set to true.
   

2. Orchestrations: In orchestrations, orchestration activity logger component can be used to archive the messages.

varOrchestrationActivityLogger.Archive(msgOutboundOrder);

Create Reprocessing WCF-SQL Receive location

To poll the messages from the Atomic Scope database, we need to create a receive location with WCF-SQL receive location with below pipeline configuration.

The Atomic Scope Activity Logger pipeline component before the disassembly is left unconfigured, as the message will be Atomic Scope envelope containing multiple messages to be reprocessed.
The xml disassembler creates the individual messages from envelope schema.
The Atomic Scope logging component after the XML disassembler must be configured with Additional property value “ReprocessedMessage=true”. This gives clue to the component that the message is a reprocessed message. The pipeline component extracts the actual body of the message and resubmits it to BizTalk message box.
The WCF-SQL adapter for the receive location should have following configurations.

XmlStoredProcedureRootNodeName: ASReprocessRecords
XmlStoredProcedureRootNodeNamespace: http://AtomicScope.Schemas.Reprocess/Records 

PolledDataAvailableStatement:
select  count(*) from [dbo].[Tracking_Reprocessing] where [ReceiveLocationName] ='<Receive Location Name>'   and PollingState=0

PollingStatement:
exec Tracking_GetReprocessingMessages '<Receive Location Name>'

Orchestrations picking up single reprocessed messages via a WCF-SQL Receive Location

In case an orchestration needs to pickup reprocessed messages, the configuration of your WCF-SQL receive location looks slightly different.

To poll messages from the Atomic Scope database, we need to create a receive location with WCF-SQL receive location with below pipeline configuration.

Note that in contrast with messaging-only scenarios, the AdditionalProperties attribute contains the value ReprocessedSingleMessage=true.

Also, the configuration of the WCF binding looks a bit different. Under the Binding settings of the WCF-SQL receive location, provide the following values:

• XmlStoredProcedureRootNodeName: ASReprocessRecords
• XmlStoredProcedureRootNodeNamespace: http://AtomicScope.Schemas.Reprocess/Records
• PolledDataAvailableStatement: select count(*) from [dbo].[Tracking_Reprocessing] where [ReceiveLocationName] ='' and PollingState=0
• PollingStatement: exec Tracking_GetReprocessingMessage ''

Next, when developing the orchestration which will pick up the reprocessed messages, make sure that the following conditions are met:

• the orchestration should subscribe to the messagetype of the message that will be sent to the just created Receive Location
• the first shape of the orchestration must be a logical receive port, which is configure with Specify Later
• after the orchestation has been deployed, bind to the logical receive port to the newly created Receive Location

Configure Reprocessing

The reprocess configuration specifies the Biztalk WCF-SQL receive location to which the message will be resubmitted. The AtomicScope portal automatically lists all the receive locations with WCF-SQL adapter configuration.

Please refer the section “How to configure stage for reprocessing” for more information.
Note: Each transaction stage can be configured to reprocess the message to a WCF-SQL receive location. The routing and further processing needs to be handled by the solution implementation.

Reprocessing to Http Endpoints

AtomicScope brings ability to reprocess the messages to Http Endpoints. These Http Endpoints can be Logic Apps, Azure functions, BizTalk Http Receive locations or BizTalk WebHttp receive locations. below is the architenctural diagram for Reprocessing web end points.

Enable archiving

Enabling archiving for services such as Logic Apps, Azure functions, etc will be done in Atomic Scope Portal. Please refer to the section “How to enable archiving for Logic Apps.”

Enabling archiving for BizTalk web receive locations is same as the configuration discussed WCF-SQl reprocessing section.

Configure Reprocess under Stage Level

In AtomicScope portal, the failed transactions can be reprocessed. This can be achieved by configuring reprocess at a stage. The following steps must be followed to configure reprocess at stage level.

1. Click the Configure Reprocess button which will be present at a respective stage.

2. Click Select to select either of the two Endpoint types.

3. Selecting BizTalk Receive Location as Endpoint Type, it will show the configured receive locations with WCF-SQL adapters.

4. Select any of the receive locations and click Done and click Save to save the reprocess configuration for that particular stage.

5. Selecting Http Endpoint as Endpoint Type, it will prompt you to enter the Uri and you have select the respective method type as either PUT or POST. You can also able to Add or Remove headers.

6. Toggling eye icon near respective header will mark the particular header Sensitive, and that confidential header will not be shown to other users except Admin or Product Owner.

7. Enter the appropriate details and click Done and click Save to save the reprocess configuration for that stage.

Reprocessing to Azure Service Bus Queues and Topics

The user should be able to edit and reprocess the message body to Serivce Bus Queues and Topics with the default System properties and also with the customizable user properties.

Configure Reprocess

In AtomicScope portal, the failed transactions can be reprocessed. This can be achieved by configuring reprocess at a stage. The following steps must be followed to configure reprocess at stage level.

1. Click the Configure Reprocess button which will be present at a respective stage.

2. Click Select to select either Azure Service Bus Queue or Azure Service Bus Topic as Endpoint Type.

3. On Selecting Azure Service Queue, you need to mention the Service Bus Connection String and the Service Bus Queue Name. And also you should be able to configure the system properties and also user properties for Serivce Bus Queue.

4. On Selecting Azure Service Bus Topic, you need to mention the Service Bus Connection String and the Service Bus Topic Name. And also you should be able to configure the system properties and also user properties for Service Bus Topic.

5. After mentioning the respective details, click Save to save the reprocess configuration. The configured reprocess settings is editable and also it can be deleted at any point of time.

Reprocessing to Azure Event Grid

The user should be able to edit and reprocess the message body to Event Grid with the Subject, Date Version and also you can mention the Event Type while reprocessing.

Configure Reprocess

In AtomicScope portal, the failed transactions can be reprocessed. This can be achieved by configuring reprocess at a stage. The following steps must be followed to configure reprocess at stage level.

1. Click the Configure Reprocess button which will be present at a respective stage.

2. Click Select to select Azure Event Grid Endpoint Type.

3. You should enter the Topic Endpoint, Topic Key, Subject and Data Version which are mandatory paramters. And also if you have any event type filters to capture Atomic Scope events, you shall also mention the Event Type if it's necessary.
4. After entering the appropriate details, Click Save to save the Reprocess Configuration. The configured Reprocess Configuratoin is editable and also it can be deleted at any point of time.

Reprocessing to File Location

Atomic Scope supports the capablity of reprocessing to file location. User should be able to configure reprocess for file location in Business Process Configuration Page. In the tracking grid, using the configuraiton user should be able to reprocess to repsective file location.

Configure Reprocess for File Location

In AtomicScope portal, the failed transactions can be reprocessed. This can be achieved by configuring reprocess at a stage. The following steps must be followed to configure reprocess at stage level.

1. Click the Configure Reprocess button which will be present at a respective stage.

2. Click Select to select File Location Endpoint Type.

3. Enter the appropriate file path and this file path can also be an UNC Share path. Once you have entered the file path, you can test connection whether you have access to the respective file location.

4. And then enter the file name with the proper extension(.xml , .json) and the file name can also contain BizTalk Macros. The following are the supported Macros,

   • %SourceFileName%
   • %MessageID%
   • %time%
   • %datetime%
   • %datetime_bts2000%

5. Once you have entered the file name and file path, you should be able to save the reprocess configuration.

File overwrite option for File Location reprocess

You can overwrite the file content if file already exist in the configured path. You have the switch button for enabling the overwrite opiton. If its enabled, the file automatically overwrite the content. If not, it will throw the warning as file already exist while reprocessing.

Reprocess Failed Transaction

Once all the configuration is done in the tracking UI, select a failed transaction as shown below.

On clicking the tracking data, you will see a graphical flow in a blade as shown below.

Reprocess

When you click on the button reprocess, you will be directed to a blade in which you will be allowed to edit the archived message and reprocess based upon your reprocess configuration.

Bulk Reprocessing of Messages

Overview

In previous version of Atomic scope, user can able to reprocess a particular transaction to various endpoints such as,

• BizTalk Receive Location
• Http Endpoint
• Azure Service Bus Queue
• Azure Service Bus Topic
• Azure Event Grid

To make this feature more intuitive, we have introduced bulk reprocessing of message where a user can able to select multiple transactions and reprocess to various endpoints. And also it was one of the favourite feature for Atomic Scope customers.

Configure Reprocessing

For Bulk reprocessing of messages, you need to map a reprocess setting of a particular stage to that respective transaction.

Reprocess Multiple Transactions

After mapping the reprocess settings, you can able to reprocess multiple transactions from Tracking UI by selecting the required transactions and click Reprocess.

If all the selected transactions contain reprocess setting, it will reprocess and it provides us the respective result. If some of the selected transactions, does not contain reprocess setting there will be a prompt available like below:

Click Yes to proceed for reprocessing. And it reprocess the respective transaction and it provides us the reprocess result like below:

Dynamic Reprocessing

In the Stage configuration, a new section has been introduced. This section is named Dynamic Reprocess Settings and will become visible once a Reprocessing Endpoint has been configured. In this section, users can configure a list of properties to be used while reprocessing.

These properties can be of the following categories:

BizTalk Context Properties
Stage Properties
Global Properties
Macros

The main purpose of this dynamic reprocess properties is to use the value of a tracked property or a macro during reprocessing.

Maintaining Dynamic Reprocess Settings and Property promotion

Using Dynamic Properties, Atomic Scope allows you to define properties which can be used for reprocessing. For example, in case of HTTP reprocess endpoints, it will enable you to provide dynamic (REST-based) urls. It also allows you to provide BizTalk Promoted Properties to reprocessed messages, thereby allowing routing of these messages to take place, based on these messages.

This part of the documentation describes the following:

• how to maintain Dynamic Reprocessing settings

• how to maintain Property promotion

How to maintain Dynamic Reprocessing settings

Once a Reprocessing Endpoint has been configured, the Dynamic Reprocess Properties section becomes visible.

Dynamic Reprocessing Properties can be defined, by starting with clicking the Add Dynamic Property button in the upper-right area of the section. By doing so, the below blade appears.

The blade contains the following fields:

Property Name - The name of the dynamic property. It defines how it will become used during reprocessing
Property Type - The source of the dynamic property. The following types can be used:

• BizTalk Context Property - A context property from the message which needs to be reprocessed
• Stage Property - A tracked Stage property which has been defined in any of the stages of the transaction
• Global Property - A tracked Global property which has been defined in any of the stages of the transaction
• Macros - Predefined macros. The following macros are available:

1. %SourceFileName% - The file name of the incoming message (this option requires the ReceivedFileName promoted property to be available within the message which is being reprocessed)
2. %MessageID% - The ID of the message which needs to be reprocessed
3. %time% - The current time
4. %datetime% - The current date and time
5. %datetime_bts2000% - The current date and time
6. %NewGuid% - The value of a newly generated GUID

• Promote to BizTalk - If this switch is turned on, the following fields become visible:

1. Destination Context Property Name - Name of the Context Property in the message which will be reprocessed
2. **Destination Context Property Namespace **- Namespace of the Context Property in the message which will be reprocessed

• Discard - Cancel the creation of the Dynamic Property
  *** Done **- Save the Dynamic Property

HTTP Reprocessing Channel

From the above configuration you can see that the Uri has a value called {InvoiceId} between curly braces. So to make use of the dynamic reprocess properties, users can enter any {Dynamic Reprocessing Property Name} in the Uri and the actual value will be substituted from the Dynamic Reprocess Property configuration.

Example:
https://baseurl.kovai.co/test/api/financialmodel/{InvoiceId}

For example, if you have a property called InvoiceId in the dynamic reprocess properties, the actual value of it will be retrieved from the tracked Global Property in this example and will be substituted in the Uri while you reprocessing the activity. During reprocessing the Uri will become:

https://baseurl.kovai.co/test/api/financialmodel/I1234567

The same will be true for HTTP headers:

ServiceBus Queue/Topic Reprocessing

While reprocessing for ServiceBus Queues or Topics, there are fields available for the user to enter user properties or system properties. In some customer scenarios, these fields might be needed for correlating a transaction with some other flow in customer's environment. So, until this version it has not possible to dynamically pass CorrelationId or MessageId while reprocessing.

Using the help of Dynamic Reprocess Properties this can be achieved now. During reprocessing the user can enter any {DynamicPropertyName} and it will be substituted from the runtime.

File Location Reprocessing

When reprocessing to a file location, it was already possible to use a MACRO for the fileName. Eg : %SourceFileName%.xml. With the help of dynamic properties you can extend it a bit more.
It is also possible to do {SenderId}_%SourceFileName%

After the resubmission, the file name will look like "Contoso_POsample01.xml".

Message reprocessing permissions

Atomic Scope allows you to assign a single transaction or multiple transactions to a user or a group of users for reprocessing.

How to assign Users/Groups to single transaction for Reprocessing

From the Atomic Scope portal, navigate to the Tracking page

1. Select a transaction which you want to assign to a user or a group

2. Now click the Assign Users button to select the user/group which you want to assign
3. While assigning a user/group, you can also leave them a note to indicate why this transaction has been assigned to them and/or provide any other instructions

4. Once you have completed the assignment, it will be shown like this, including the message that you provided

How to assign Users/Groups to multiple transactions for Reprocessing

** Assigning Users/Groups to transactions**

1. From the Atomic Scope portal, navigate to the Tracking page
2. Select Assign To Reprocess in Tracking screen to open Assign User/Groups for Reprocessing blade.

3. Now select the users from the userlist to assign them to multiple transactions

4. Click Edit Global Intimation message option to Edit Global Initimation message that applies to all transactions and users. This step is optional.
5. Click Edit Intimation message option in Grid to Add/Edit initmation message per transaction to all users. This step is optional.
6. Select transactions from the New Transactions for Configuration Grid to assign selected Users/Groups to transactions for Reprocessing
7. Click Assign to assign multiple users/groups to multiple transactions.

:::(Info) (Selected Users will be added to the existing Assigned users list for the selected transactions)

:::
Removing Users from Transactions

• Remove User Mappings button in Previously Configured Transactions Grid to open Remove User Mappings blade
  

• Now select the users from the userlist to remove it from the transactions

• Select transactions from the Previously Configured Transactions Grid to remove selected Users/Groups from transactions (or) click Delete icon in the specific record in the grid to remove the selected users/groups from that specific transaction.

• Click Remove All User Mappings button in Previously Configured Transactions Grid to remove selected Users/Groups from multiple transactions.