With the advent of DevOps, there is an increasing trend of projects trying to adopt the agile model for their delivery. This article aims at performing a critical analysis of agile performance testing and tries to answer the question: ‘How agile performance testing can fit into the current software delivery world?’
The article is divided into two parts:
- The Pūrvapakṣa – the general myths prevalent about performance testing in agile
- The Uttara-pakṣa – a response to the general myths.
In the Pūrvapakṣa section, we will discuss the top five myths associated with agile and performance testing.
Myth 1: Agile is cost effective
One of the common denominations that stands as an influencer for project managers taking a decision to move from waterfall, or any other delivery model, to agile is the cost effectiveness card.
This cost effectiveness card arises from the notion that agile squads are generally a smaller team comprising of 12-14 members. This notion is derived from George A Miller’s psychological review which is often interpreted to argue that the number of objects an average human can hold in working memory is 7 ± 2.
So, a 9-member development team with a 25% top-up of testing team (functional and non-functional testing) and an addition of operation team member(s) to the squad, roughly becomes the squad size and hence the cost of the team becomes almost fixed. In some cases, the total squad size would be only 9 or even less.
Myth 2: Agile helps to reduce delays in time to market
The early waterfall model demanded the completion of the entire modules before it could be released to consumers. The agile model provides sprint-based developments where the functionalities developed and tested in the earlier sprints could be released to customers and add-on features incrementally released thereby helping consumers to get an early feel of the application.
This also helps to increase the feedback loops from the end-user, thereby helping to fine tune the application.
Myth 3: SDET (software development engineer in test)
With the smaller team size, multi-tasking becomes the norm. This drives the need to have squad team members who are exposed to the full stack, like development (front-end/middleware/database), testing (functional and non-functional testing) and environment/operations management. In a sportive cricketing term, it can be termed as the day of ‘all-rounders’.
The team that has more all-rounders is expected to be the most economical and fruitful team.
Myth 4: Co-location, the characteristics of agile
The other key characteristic of the agile mode of delivery is the co-location of team members, mostly in the same conference room. This helps to reduce the feedback loops and in-turn helps to increase the co-bonding among the team members.
Myth 5: Reduced documentation – storyboards, epics and cards
Agile delivery is based on daily stand-ups where epics, story cards, issues and risks are discussed daily across the detailed story boards, and actions are assigned immediately, thereby reducing the need for detailed documentations. The queries are addressed in the stand-ups, be it requirement clarifications, design clarifications, scope clarifications and so on.
The reduced documentation is perceived to help teams to concentrate on the core delivery, thereby improving the turnaround time for the deliverables.
The Uttara-pakṣa – This section discusses the above myths and how they fare in the practical implementation.
Myth 1: Agile is cost effective
Though it is considered that agile is cost effective, what one can observe is that the cost overruns are higher in terms of agile. With agile, we are trying to achieve a sophisticated squad per module.
Though the per-module squad team members are within controllable units, the number of squads has to be increased later to cover the scope changes that come from early delivery feedback, missed functionalities, better performance improvements and so on.
This would finally take the total number of resources to equal or more than the traditional delivery models. A quicker release cycle also demands the availability of applications across different environments simultaneously thereby the need for additional environment resources.
In the traditional model, the application build moves from development environment (Dev) to system test (ST)/ system integration test (SIT) / user acceptance test (UAT) to performance test environment (PTE) to pre-production environment and then to production environment in a sequential manner thereby the same environment team could be re-used.
With agile and DevOps delivery, the movement is faster, thus the need for additional support from environments becomes a norm, especially the performance test environment that is production-like and needs production-like configuration to be implemented in a shorter period.
Though there are continuous integration/ continuous delivery and continuous deployment (CI/CD) and Kubernetes, the environment support is something that cannot be ruled out. One of the major flaws of the agile model is that it is a developer-friendly model. If we look at the developer to quality assurance (QA) team ratio, the proportion of QA team (both functional and non-functional) is very small compared to the overall team size.
The team by its design has lesser priority for quality, leading to an expectation that the small testing team should cater to both functional and non-functional testing. In practice, functional and non-functional testing are two different spheres altogether; the result being defect slippages that get added to the next iteration storyboards.
The key impacted area is the scope that can be effectively covered by the performance testing team, considering the shorter time duration and lesser resources available.
Myth 2: Agile helps to reduce delays in time-to-market
What is perceived as a reduction in time-to-market is a decrease in scope of delivery. This decrease in the scope of delivery during the initial sprints marked against the timeline, practically provides less opportunity for the development team to concentrate on the overall application performance in the final integrated world of the interconnected modules and application program interfaces (APIs).
The delivery model becomes more focused on functionality being delivered per cycle, thereby the application performance activity gets a back seat – and mostly are exception approved on the basis that the application would have a lesser user base at the time of initial roll-out.
However, when ‘down the lane’ the application user base is expanded, and it becomes very difficult to fix the application performance problems.
The common misunderstanding is that the performance testing defects are like functional testing defects. Whereas in actuality, performance testing and their defect fixes are complicated and require more time and effort. Even if we shift-left and modularise the performance testing – where application performance related defects are identified early – the development team is unable to concentrate, or have enough bandwidth, on performance defect fixes.
This is because the key carrot for driving delivery is a working functionality. This doesn’t mean the developers are voluntarily avoiding application performance considerations. However, it is the nature of the agile model, whose focus is more towards functional delivery in a shorter time and gives less importance to performance testing.
We know, when the time (schedule) is reduced, the principle of project management states: either there is increase in cost or reduction in scope. In either case, it is the ‘quality’ that gets compromised.
Myth 3: SDET (Software Development Engineer in Test)
Over the past decade the world was moving towards expertise in individual components: front-end, back-end, functional testing, performance testing, security testing, operation acceptance testing, and so on. Now, if there is a sudden reversal in the flow of direction and we need a full stack knowledgeable resource, it is going to be difficult to find them in market. Moreover, considering the testing realm, the functional testing and non-functional testing are different spheres altogether.
The functional testing aims at the client side, whereas the performance testing aims at the server side of activities. The thought process required to do functional and non-functional are entirely different.
The performance testing world by itself has its own set of deliverables that need to be addressed in a shorter timespan in the agile world of delivery: test environment, test data, script development, scenario shakeout, performance test executions/re-executions, client and server-side analysis, network and infrastructure configuration support and so on.
Hence the team members are going to get less time to concentrate on other fields.
For a successful agile implementation, it would be more appropriate to have an individual team members who are experts in an area; to have tasks allocated to him/her according to the area of expertise and committed to fulfil their allocated task. Trespassing on different areas would only complicate things.
Myth 4: Co-location, the characteristics of agile
To increase the feedback loops, reduce time to market and bring in the squad-sprint-spirit, co-location becomes the norm. This would mean all team members should be ideally located onsite. An all onsite team would have significant increase in the overall project costing structure.
Though there are concepts like distributed agile and video conferencing facilities, it doesn’t get the same one-team feeling as with teams that are physically present together. Especially in the performance testing world, the test executions, defects and analysis need constant feedback from various stakeholders: development team, backend team, infrastructure, environment etc.
Even in the traditional waterfall model, co-location for performance testing execution and analysis is the norm.
In reality, what one could see is that, due to the adoption of agility which is marketed to be a cost saver, the performance test team, who are last in the delivery cycle, tend to get a lesser share of the number of resources than required, with the work getting pushed to be delivered from offshore to save on overall cost.
The results of which means communication gaps, delays in data availability and lesser knowledgeable information about the challenges faced by performance testing teams reaching the higher management. This creates a false notion that performance testing is the bottleneck for delivery, however the real cause would be upstream delays and other factors that did not get their due attention.
Myth 5: Reduced documentation – storyboards, epics and cards.
The current delivery model includes multi-vendor and multi-parties. With reduced documentation and shorter cycles, ambiguity on scope and requirements remains. One of the major challenges faced by performance testing teams is on the availability of non-functional requirements.
With business analysts’ time being mostly taken over by the support of development and functional testing streams, performance testing queries get swept under the carpet.
Here is a quick summary of the arguments of the Pūrvapakṣa and Uttara-pakṣa.
|Myth 1: Agile is cost effective||With small teams and fixed squad size, the cost is effective.||
|Myth 2: Agile helps to reduce delays in time to market||Sprint-based developments where the functionalities developed could be early released to customers.||Reduced scope, application functionality driven, thereby leading to less attention to fix performance related defects.|
|Myth 3: SDET (Software Development Engineer in Test)||All-rounders improve team effectiveness.||Should be more of collaboration of individual team members with responsible delivery on his/her field.|
|Myth 4: Co-location, the characteristics of Agile||Co-location facilitates co-bonding.||With onshore-offshore model, co-location becomes a passé impacting the One Team effect.|
|Myth 5: Reduced documentation – Storyboards, Epics and Cards||Reduced documentation is perceived to help teams to concentrate on the core delivery.||With reduced documentation and shorter cycles there remains ambiguity on scope and requirements. With business analysts’ time being mostly spent on clearing the application development queries and less time for the performance testing team.|
With all due considerations, it can be evidently seen that the concept of agile might work efficiently for a product-based organisation where incremental changes are being made to an already established and quality-proven product. Trying to implement an agile model in a multi-party application delivery model is only trying to fit a square peg in a round hole.
The notion seems to be, by early integration of performance testing tools on application, development IDEs (Integrated Development Environment) and a performance test execution would uncover all performance testing defects. This notion is flawed because the application performance not only gets affected by the application code, the application code is one of the factors that affects application performance.
The other factors include – but are not limited to – bandwidth allocation, environment configurations, CPU and memory capacity, web and application server configurations, database indexing, database configurations, SQL execution plan, storage limitation, and so-on, that could only be detected in a full load performance testing, conducted in a production-like integrated performance test environment.
Solution for new product/application:
For multi-vendor, application development and testing engagements and new product development, an agile and traditional combined performance testing model co-existing together would be the right approach that should be applied like this:
|Early performance testing||In sprint Agile/ DevOps at API level or individual component level.||Performance test team member part of the squad to early detect performance defects in the code.
Executed in N-1 sprint cycle.
|End-to-End performance testing||At the end of all sprints before delivery of full functionality or before full user base throttling load in the system.||Separate team conducting end-to-end integrated Performance Test on a production like environment.|
The delivery partners have to understand the need for this two-phased approach for performance testing and have enough budget and resource allocated so that we get short-term benefits and long-term benefits without compromising on ‘quality’.
V.M. Guruprasath, Senior Manager, Capgemini Hyderabad