Tracking Data purging

Purging is an essential component for Atomic Scope because it deals with large amounts of data.

The purging process has been improved with each version of Atomic Scope. As a result, it will be more reliable and better performing. So every time, we perform performance and load testing on this new purging implementation and established threshold values.

The primary focus of this analysis and testing is

How effectively can the user use this feature?

The customer has to first comprehend the level of purging recommended by their requirements. So they can select the purging options as per environment.

• Some transactions have multiple stages with an archived message body. Other transactions in the business process have limited data. Therefore, it is attainable to select to remove certain transactions for the required number of days.
• A few business processes have multiple transactions with extensive activity tracking.Therefore, it is feasible to choose to purge at the business process level.
• Global purging will be performed based on overall data count and time duration.

How can we establish a benchmark?

Continuous load testing will help to understand the feature's reliability and scalability as well as the overall application.

In this testing, the minimum configuration and data were used. According to the data flow in their environment, the customer can then set the benchmark.

Analyzied and tested metrics are given below:

1. Total size of one message is 4 KB
2. 4 stages in one transaction.
3. 5 messages per second for processing
4. Total data available in Database is 8 lakhs.

Below are the result of continuous load testing

The purging process will done with every 4 minutes and each cycle will delete 5000 Main activity (Transactions). It will delete related stage activity, archive message body and exception.

Observation of version 8.3 implementation of purging process

• Total data available in Database id 10 lakhs
• Version 8.3
• Total message size is 4MB
• 4 stages in 1 transaction

Observation of version 9.0 implementation of purging process

• Total data available in Database id 10 lakhs
• Version 9.0
• Total message size is 4MB
• 4 stages in 1 transaction

How can we improve the infrastructure to make the purging process feasible?

• The execution time will vary depending on the size of the message and the number of associated activities. If the message size is increased, the execution time increases, resulting in a time out exception.

• To overcome this time-out exception, the customer can upgrade the infrastructure, and Atomic Scope will notify them to know that the purging process hasn't actually taken place. Therefore, upgrading the infrastructure will be simple.

• The processing will speed up if we increase the infrastructure to handle the massive amounts of data. We can overcome the time out and other purging exceptions.

• Infrastructure of SQL server includes the Size of RAM, Capacity of the hard drive to store the business transaction and the number of CPU core.

• According to the message size and data flow, upgrade the infrastructure.

• If the message body is larger, it can lower the data count for purging. As a result, some data will be deleted. Then it will be easy to manage and purge the data.