What is Cloud Testing ? A Beginner’s Guide

What is Cloud Testing: A Beginner's Guide 

When choosing a cloud to host infrastructure, users often ask for a "test drive" - to evaluate the usability and performance of leased resources for themselves. Many, even fairly experienced professionals, imagine the testing process in a simplistic way, limiting themselves to plans to run a set of synthetic or performance tests (similar to those used for desktops) on a virtual machine.

In practice, these researchers face great difficulties because of the relative novelty of cloud computing technology and the differences in the principle of virtual machines from the usual "iron" servers to cloud testing as a service.

The main reason for this is clear- the performance of a classic software system is evaluated at a certain workload for a fixed configuration. But providing a stable workload and configuration of resources in a public cloud is not possible in principle - to dynamically adapt to changing tenant conditions, cloud providers automatically allocate and release resources (compute nodes) on the fly.

Recommended Read: How to install Google Cloud SDK in Linux (Ubuntu, CentOS)

Also Read: Use TERRAFORM for Google Cloud (GCP) – Create a VPC

It turns out that instead of measuring the average performance of a static system at maximum load, it is more logical to evaluate other characteristics - the ability of cloud services to adapt to changing loads (performance/cost ratio). An additional metric should also cover the reliability of these services during individual node failures (as well as data center outages entirely).

The next problem arises when it is necessary to compare the results obtained. Different providers offer services that do not match in content:

•       with a variety of possibilities;
•       with different guarantees;
•       with its own service parameters, spelled out in the SLA contract.

 What's important to remember before Cloud Testing begins

•       distributed computing technologies are rapidly evolving, and there are many incompatible implementations;
•       The cloud IaaS market is young and at the stage of development and experimentation;
•       the user has no direct contact with the provider's equipment and interacts with the leased system resources remotely, via the web interface;
•       Professional software packages for testing cost a lot of money.

The results of the work/test can:

•       depend on the quality of Internet communication channels;
•       differ in geographically different connection points;
•       change throughout the day as the load on the public cloud fluctuates (changes in the number of users and applications they run).

Is it possible to apply "Iron" tests to the cloud?

It is well known that to evaluate the performance of a desktop computer, and it is customary to use several sets of tests - separately CPU and graphics processors, RAM and disk memory, network adapters. The test results of the latest processors and graphics cards have become the object of discussion on the net. Those who did not get high enough test results of their equipment ask themselves a question:

Such skepticism makes sense because, in order to get record results, the equipment is often put in a mode that is far from average (acceptable for daily work). Before testing, overclocking of processors and memory with additional cooling, risky experiments with fine BIOS settings, beta versions of device drivers, etc., are often performed. Probably, the obtained figures will make the winners happy, but will they be able to continue working stably with such hardware and software settings? As for virtual environments, it is impossible to "overclock" components there.

Obviously, the principle of VMs is quite different from the physical, real hardware that users are accustomed to. If the processor core of a physical computer is unambiguously localized on a CPU chip, in IaaS, it is a hypervisor-organized imitation of a quota allocated from the total CPU capacity of the cloud. Despite the fact that the provider provides each user with the ordered amount of resources, the real state and status of the cloud infrastructure changes all the time:

•       when adding and removing running "instances" - runtime abstractions, virtual machines;
•       after starting and stopping running applications;
•       due to fluctuations in user load;
•       during backups, data recovery.

Thus, there is a high probability that when you run the same test repeatedly at different times, the results will not be identical (the status of the infrastructure and the state of communication channels have changed).

Given the above, the answer to the question about the applicability of tests for physical hardware to testing cloud infrastructures seems to be: applying "iron" tests to the cloud, in principle, is possible - but the practical use of it is small.

Practices and tools for testing Cloud Infrastructures

The basic requirements for cloud tests are as follows: they must take into account the current specifics of distributed computing and remote user work via the Internet on resources that are provided on-demand in an automatic mode, scalable in both directions.

Performance tests of individual subsystems (processors, memory, disks) are clearly not enough for a comprehensive assessment of the cloud (unlike similar tests of desktops). But they are still used for the very initial verification of IaaS performance.

Economic viability for business is one of the main factors driving the development of cloud computing. And this includes the availability of the service, stable performance under load fluctuations, security in the public cloud (including backups), and much more. Things that are not always evaluated by performance numbers but really have a big impact on the comfortable and predictable use of the service by the user. Thus, when assessing cloud parameters, it is not worth limiting yourself to a formal test using even the most powerful software.

In conclusion, briefly about the main points (when testing the public cloud)

When preparing for testing, discuss with the technical support team of your cloud provider your intention to run this or that test on their resources (hardware and software). This will allow you to get expert advice in advance and avoid misunderstandings due to possible incompatibility of the provider's hardware resources/virtualization system with the test software package you have selected for your project.