Saltar al contenido

LongBOARD: Real-time visualization of QA performance

Introduction

In any ecommerce, it is necessary to ensure that navigational and transactional services work as expected under high traffic load (this refers to thousands of simultaneous site transactions and visits). This is an important aspect to consider especially during sales periods such as Black Friday, Cyber Monday and Christmas.

The way to guarantee that an ecommerce site performs as expected in these circumstances, is to simulate the concurrent access of visitors in QA or Preprod environments and measure the results. For this purpose, performance tests are carried out.

Pain points in QA testing

One of the tools that allows us to simulate load and perform load testing on an ecommerce system is JMeter. The execution of test sessions provide a log with the results obtained from each round of testing. This can be for example: the response time, number of concurrent users, response code, URL to which the request has been made, etc.

As such, how can this results log be analyzed effectively to obtain a conclusion about the results? The output of results is presented in plain text and can therefore be very difficult to analyze, which is why we have created LongBOARD.

LongBOARD features

So what exactly is LongBOARD?

LongBOARD is a set of dashboards powered and developed by Orienteed; they provide real-time visualizations of the results obtained during the execution of the performance and load tests.

LongBOARD is in the cloud. This means that you have a large margin to increase or decrease the computing capacity as needed, since the client does not have to download any type of local software or invest in infrastructure to carry out their performance tests.

The platform behind the dashboard (Grafana + ElasticSearch + LogStash) is available and deployable with a single click through the AWS Marketplace.

LongBOARD consists of 2 main parts:

  • Logstash: This is a pipeline for collecting, filtering and sending the results generated from the execution of the performance/load test.
  • 3 dashboards in Grafana where you can visualize, in real time:
    • A summary of the results.
    • Statistics of the requests.
    • An analysis of the errors.

In the image below, we can see the Request Statistics dashboard. In it, we can visualize a large amount of statistical data about the requests that arrive at the server. It shows data such as the most common requests, the response codes and the percentage they represent of the total, the average time in milliseconds, among others.

longBoard logo
Request Statistics Dashboard
Request Statistics Dashboard

As mentioned above, there are also two additional dashboards that report the requests statistics and the analysis of the errors found.

These dashboards are made up of dynamic graphs. This means that through the use of dynamic filters, you can select the transactions that you want to see. For example, you can filter to view only login requests, requests with a 503 response code, as well as to filter results by the error message, etc.

Using LongBOARD

In this section, we will go into some technical details about the specific steps to configure LongBOARD and the steps to start using it. So if you are interested in understanding how LongBOARD is set up, be sure to read the below.

How to configure the tool?

First of all, the following requirements must be met:

  • JMeter: This will be the application through which the tests will be run, and where the results will be generated and collected for LongBoard’s use.
  • OpenSearch: This tool must be available in order to store the results and make them persistent. In Grafana, the data will be read from OpenSearch.

What elements do we need?

For successful testing, it is important to provide our system with a set of necessary documents. Among these documents, we can find the following:

  • Input CSV (datapools): Input CSV files, also known as datapools, are data files that are used by JMeter to provide input values to tests. These files contain structured data sets, such as usernames, passwords, or any other information relevant to testing.
  • JMeter Scripts: JMeter scripts are files that contain the settings and actions needed to perform tests. These scripts are created in JMeter and are used to define the behavior of virtual users, such as the HTTP requests they make, navigation sequences, and any other interactions with the application being tested.

Understanding the structure of the JMeter script

In LongBOARD, the tests run automatically, but it is important to understand the parameters that can be set. Among the values that we must consider, we can find the following:

  • Number of threads: The number of threads, also known as threads or virtual users, is the number of concurrent users that will be simulated during the test. Each thread represents a virtual user that executes the defined script. A higher number of threads implies a higher load on the system.
  • Ramp time: The ramp time or ramp time is the period in which the threads are gradually added to the test scenario. Defines the time it takes to reach the maximum thread load.
  • Duration: Duration is the total time for which the test will run. Defines the period of time in which the threads will perform the requests defined in the script.
  • Initialization variables: Initialization variables in JMeter are variables that are defined before the start of the test and are used to store values that will be shared between different threads or sections of the script. These variables can be used to pass dynamic data, such as session ids or values generated during testing, between requests.

How to configure the tool?

1. Create the EC2 instance from the AMI.

Once the instance is created, we will have:

  • Logstash: A tool for ingesting, filtering and sending data to OpenSearch.
  • Logstash configuration file
  • Index Template: This template will tell OpenSearch the type of data to be entered and which indexes it has to create in the database.
  • Grafana docker image ready to run.
  • Executable file: This file will automatically run Logstash with the necessary configuration parameters.

4. Open Grafana in the browser: http://public_instance_ip:3000

  • User: admin
  • Password: <id_instancia_AWS>

3. Configure the data source, via Configuration > Data sources > OpenSearch. Configure the following fields:

  • URL: OpenSearch_domain_endpoint:443
  • User: openSearch_user
  • Password: openSearch_password
  • Index name: (the name of the index you put in line 30 of the configuration file to be mentioned later (logstash_csv.fonf)
    For example: stress-myWeb-logstash
    .

4. Click on Save & test. Below are some of the possible errors you may find:

  • No index exists: this is because logstash has not yet been run and has not created the index.OpenSearch error: no such index
  • The date field called timeStamp does not exist: this is because the index does not have an associated template or is empty.
    No date field named timeStamp found

NOTE: When creating the data source and not having the index created in Opensearch, these errors are normal. It is OK if they appear, as soon as you insert data they will be fixed automatically.

If everything is OK and there are no errors, the following message will appear:Index OK

5. Once all the previous steps have been completed, we can go to the "Scripts" board and start testing.

New dashboard “Scripts”

In this new version we have a new dashboard called “Scripts”. From it, we can do everything necessary so that the tests are executed automatically and the files are loaded. Within "Scripts", we have the following components: 

  • Data pools panel: A panel that consists of a Drag & Drop in which we can upload the csv files we want. Next to it, we have a list that provides us with the name of the loaded files and their weight in bytes. Thanks to this component we can avoid errors when uploading duplicate elements or invalid files.
  • JMX Script panel: This component is really useful for testing, since it is not only limited to loading the script, but also allows you to modify parameters such as its duration, number of users or its ramp time. In addition, we have an additional tab that allows us to view the initialization variables and modify them whenever we need it.
  • Control Panel: It consists of two simple buttons that allow us to start and/or stop the execution of our tests in JMeter. Above the Start button we have information about the server on which the tests will be executed and loaded, as well as the route that the data-pools must have configured within the script to avoid errors during execution.
  • Status panel: This component is a table that allows us to receive messages about the execution status and data loading. It provides us with information about the duration of the tests, the errors or when we can begin to visualize the results in the rest of our dashboards in Grafana.

Basic use guide

For the correct functioning of the system, we must carry out a series of very simple steps, which we will show below.

1. To upload the csv files, it is enough to select one or more of them at the same time, drop them in the area marked in blue and it will verify that it is not already among the loaded files and that it is valid. From the list on the right side of the image, we can make sure that there are no missing or excess files among the files to be sent. If so, we can select a file in the table and delete it using the delete button. When everything is ready, click on "Send" and the server will take care of the rest.

Panel Data-pools plugin.
Panel Data-pools plugin.

2. If we want to carry out loading and modifying JMX scripts, we can drop them in the designated area of the panel, just like in the previous plugin. Once our desired JMX file is loaded, it will show us the current parameters that the test script has and we can modify them in the entries on the right whenever we want. It is important to note that the csv files defined in the script must have the following path set:

/home/ec2-user/jmeter/apache-jmeter-5.4.3/bin/DataPools/

In addition, we have a variables tab in the upper left that shows us all the initialization variables that are defined in the script, allowing us to rewrite their identifying values or names.

When we have everything ready, we press the "Update" button to apply the new changes and continue with the execution.

Script JMX Panel  structure
Script JMX Panel structure

3. When we already have all the necessary files prepared, all that remains is to run the tests. For this we have a very simple interface with two buttons. With them we can start the execution or stop it.

We also have relevant data about the url where we find the server or the folder where the data-pools are going to be stored.

Control Panel structure
Control Panel structure

4. In this new version, we have included a state table which provides real-time information on the progress of actions. The table consists of two main columns, the first one shows the status, it can have two values: "Success" which indicates that the action has been carried out successfully, and "Error" which indicates that some problem has occurred and the action has failed. has been completed successfully.

In the second column, an informational message is displayed providing relevant details about the ongoing process. The message includes information about the estimated duration of the tests, confirming whether the execution process has been carried out correctly, and provides information about when the obtained results can be fully viewed and uploaded. This message provides an overview of key timings and events while the execution and loading processes last.

When the system notifies us that the tests will begin to load, we have 2 shortcuts that will allow us to quickly change the dashboard in a new tab. Thanks to it we can see the progress in real time.

Shortcuts to Error Review and Request Statistics
Shortcuts to Error Review and Request Statistics

How to use it?

Once everything is configured, you only need to perform 3 steps to start using LongBOARD.

  1. Load the files in the corresponding "Drag & Drop" and modify all those parameters that we want for our tests.
  2. Press the “Start Test” button.
  3. We can navigate to the rest of the dashboards once the tests have finished running and see the data being loaded in real time. To do this, we have shortcuts at the top right of our screen, which allow you to navigate between boards quickly and easily.

Conclusion

It is a well-known fact that Quality Assurance helps ecommerce stores create and provide online services that meet the needs, expectations and requirements of customers. QA practices help online businesses sell better and create a better online purchase experience, and also builds trust and loyalty with customers during significant campaigns such as Black Friday.

Hence, it is vital not only to optimize one’s ecommerce in preparation of such important sales events, but also to be able to effectively analyze the data from stress & performance testing. The actionable insight that LongBOARD provides with its dashboard visualizations, allows both business & technology leaders to understand bottlenecks and make the necessary improvements for their business.

Do you want to know more about LongBOARD? Get in touch with us!