Service Bus Tracking

From Version 7.5, we are changing the Azure Architecture of Atomic Scope to improve performance and avoid any inconsisties in tracking. In this article, we explain the issues we found with the existing EventHub service that we were using.

Inconsistencies in Tracking

With EventHub, it was not possible to support ordered delivery of messages. This caused a problem when the service was exposed to heavy load and the service started to throttle. From customers, we were getting feedback that some of their events were missing. As a tracking & monitoring tool we need to ensure that it is highly available 24 * 7, even when exposed to very high load.

Solution

After doing some R&D, we ended up in Service Bus Sessions and it looked promising. An amazing article from ordered-queue-processing-in-azure-functions which was exactly the problem we ran into.

The benefit of sessions is it enables you to preserve order even when processing at high scale across multiple instances. Sessions also ensure that messages from a session are processed in order

There's also a limit here as well. Due to the Messaging Service Quota Limits each namespace can have only 5000 concurrent connections opened. This limit would be easily exhausted by our customers when they span a logic app activity to span across 1000 records processing with 4 Atomic Scope connectors in a single run.

So how did we overcome this problem? We brought the ability to add x number of Service Bus Namespaces from Atomic Scope itself and can be used for scalability purpose. Which means for example, there can be x number of transactions in Atomic Scope. But not all receive that high volume of messages. Customer's can prioritize which transactions need additional queues to handle such load.

Changes in Atomic Scope's Architecture

• EventHub is removed and we've converted it to ServiceBus
• Atomic Scope connector will have a new optional input to accept QueueMappingId
• Azure Functions will now inject data to ServiceBus
• Added Azure Keyvault to store and use QueueMappings

Overall Architecture view

New Resources Added

• Service Bus Namespace with a session enabled queue
• Azure Keyvault

Security

Since messages that flow through Atomic Scope are highly sensitive and should be viewed by only people who have privildges to do so and also they should not get hold of Queue connection string to hack messages or something. Our Azure functions needs to know where to send messages to and it's tightly integrated into Key Vault when deployed.

By default, even the person who deployed Azure Infrastructure will not be able to view the secrets from the keyvault. Only the Azure function has access to view the connection strings from the vault.

Deployment

When you freshly install or migrate from existing version of AtomicScope we will deploy these new required resources automatically. You can either use the powershell scripts to upgrade or make use of the Azure Deployments page to upgrade or fresh install things.

Whether you choose to fresh install or upgrade that's completely upto the user's preference. Whatever choice is made, Atomic Scope will deploy 1 Queue Namespace with a session enabled queue by default and will be used for messaging. Incase if the user prefers he receive's more load on say on a transaction called Shipment Orders user can add more Queue from the atomic scope portal and pass the id in the connector to be able to use that queue for that particular transaction.

Accessing Keyvault

When you add more queues or map existing one's the details will be stored on the keyvault secrets and consumed by the Azure Function to inject data. Hence therefore you will need to create a Service Principal and provide access to accessing keyvault secrets. To do so

• After you installed Atomic Scope and deployed Azure infrastructure find and open the Azure Keyvault
• In the left pane find the Access policies tab under settings.
• Follow the below screenshots to map a service principal.

Adding / Mapping Queues

User can add new queue or map existing queues to be able to consume them for messaging.

There's also no restriction on the resourcegroup or subscription, user can pretty much map or deploy queues from any resource group. But we recommend you to tie all Atomic Scope resources on a single resource group for maintenace incase if necessary.

Example

Once you have mapped another queue you will get a QueueMappingId shown in the portal which should be passed in the connector for any transaction that you think that might want to use the newly added queue insted of the default one. Example, here i have a newly added queue called tooling31 and i have copied it's queueMappingId

Now I head straight to the transaction that might send high amount of load and i'll pass this id in the connector so that the other transactions will use the default queue and this particular transaction needs a separate queue to be able to handle the load.