- 22 Aug 2023
- 4 Minutes to read
Service Bus Tracking
- Updated on 22 Aug 2023
- 4 Minutes to read
From Version 7.5, we are changing the Azure Architecture of Atomic Scope to improve performance and avoid any inconsisties in tracking. In this article, we explain the issues we found with the existing
EventHub service that we were using.
Inconsistencies in Tracking
With EventHub, it was not possible to support ordered delivery of messages. This caused a problem when the service was exposed to heavy load and the service started to throttle. From customers, we were getting feedback that some of their events were missing. As a tracking & monitoring tool we need to ensure that it is highly available 24 * 7, even when exposed to very high load.
After doing some R&D, we ended up in Service Bus Sessions and it looked promising. An amazing article from ordered-queue-processing-in-azure-functions which was exactly the problem we ran into.
The benefit of sessions is it enables you to preserve order even when processing at high scale across multiple instances. Sessions also ensure that messages from a session are processed in order
There's also a limit here as well. Due to the Messaging Service Quota Limits each namespace can have only 5000 concurrent connections opened. This limit would be easily exhausted by our customers when they span a logic app activity to span across 1000 records processing with 4 Atomic Scope connectors in a single run.
So how did we overcome this problem? We brought the ability to add
x number of
Service Bus Namespaces from Atomic Scope itself and can be used for scalability purpose. Which means for example, there can be
x number of transactions in Atomic Scope. But not all receive that high volume of messages. Customer's can prioritize which transactions need additional queues to handle such load.
Changes in Atomic Scope's Architecture
- EventHub is removed and we've converted it to ServiceBus
- Atomic Scope connector will have a new optional input to accept
- Azure Functions will now inject data to ServiceBus
- Added Azure Keyvault to store and use QueueMappings
Overall Architecture view
New Resources Added
- Service Bus Namespace with a session enabled queue
- Azure Keyvault
Since messages that flow through Atomic Scope are highly sensitive and should be viewed by only people who have privildges to do so and also they should not get hold of Queue connection string to hack messages or something. Our Azure functions needs to know where to send messages to and it's tightly integrated into Key Vault when deployed.
By default, even the person who deployed Azure Infrastructure will not be able to view the secrets from the keyvault. Only the Azure function has access to view the connection strings from the vault.
When you freshly install or migrate from existing version of AtomicScope we will deploy these new required resources automatically. You can either use the powershell scripts to upgrade or make use of the
Azure Deployments page to upgrade or fresh install things.
Whether you choose to fresh install or upgrade that's completely upto the user's preference. Whatever choice is made, Atomic Scope will deploy 1 Queue Namespace with a session enabled queue by default and will be used for messaging. Incase if the user prefers he receive's more load on say on a transaction called
Shipment Orders user can add more Queue from the atomic scope portal and pass the id in the connector to be able to use that queue for that particular transaction.
When you add more queues or map existing one's the details will be stored on the keyvault secrets and consumed by the Azure Function to inject data. Hence therefore you will need to create a Service Principal and provide access to accessing keyvault secrets. To do so
- After you installed Atomic Scope and deployed Azure infrastructure find and open the Azure Keyvault
- In the left pane find the
Access policiestab under settings.
- Follow the below screenshots to map a service principal.
Adding / Mapping Queues
User can add new queue or map existing queues to be able to consume them for messaging.
- When you choose to map existing Queue please ensure that the queue inside the namespace is created with
SessionEnabledelse it will not be considered as a valid mapping.
- You should also use the same ServicePrincipal credentails like
ClientSecretetc which you have mapped to the vault in previous step.
There's also no restriction on the resourcegroup or subscription, user can pretty much map or deploy queues from any resource group. But we recommend you to tie all Atomic Scope resources on a single resource group for maintenace incase if necessary.
Once you have mapped another queue you will get a
QueueMappingId shown in the portal which should be passed in the connector for any transaction that you think that might want to use the newly added queue insted of the default one. Example, here i have a newly added queue called
tooling31 and i have copied it's
Now I head straight to the transaction that might send high amount of load and i'll pass this id in the connector so that the other transactions will use the default queue and this particular transaction needs a separate queue to be able to handle the load.