How much like production is my Test Environment? This is a question that is posed inside many organisations and online test and tech forums.
The question, in itself, suggests there is a difference, so I’d like to explore what the underlying reason might be and why the question in itself might be the wrong one to be asking in your organisation at this point in time.
This is a question we as testers ask and get asked frequently, to which without the context of the type of testing, the purpose of the test and the detail of the hypothesis being really understood, then the answer can never be provided, yet we carry on this merry-go-round of questioning.
In my experience, the question is posed most often in the context of trying to recreate the monolithic estate of production in the name of the anti-pattern of the Testing Pyramid.
The dreaded “Ice Cream” of too much e2e testing neglecting the other layers of the solution which could be tested and breaking the solution into managed and understood ‘chunks’ of the system and/or IT estate.
If common consensus, is that the most effective way to prevent defects getting into production is to consign ourselves to testing e2e, we may as well give up on anything being continuous and kick delivery timelines into the long grass.
We are shifting right (but not in a good way) to the slowest, most costly place and any hope of articulating any type of accurate test coverage is false confidence – you are probably just proving a narrow scope of happy paths and very slowly – manually!
I do find it rather odd that we talk about test environments in the context of a comparison with, or deviation from production.
Hasn’t our whole IT change activity from ideation, through architecture, design, build and test had to consistently understand both the baseline of production in their context and how they are changing it?
You have to assume given how rife test environment conversations are on forums and analysts report that this is broken in organisation after organisation.
Even if test teams are doing their best to get away from the “ice cream”, their amigo colleagues to the left of whatever development methodology they follow are consigning them to this fate.
We are doing exploratory, somehow the SDLC has been flipped to Test > Analyse > Design > Build > Test
Management and external consultancies will say “we need to implement Service Virtualisation tooling (aka Stubs)”, but is it really that simple, are they missing the obvious and preventing real progress by keeping it a test-centric problem?
If you had little or no confidence in the understanding you were going to get from a connected system would you replace that system with a virtual substitute?
Sounds crazy right?
How can you create a virtual copy of something you don’t understand and expect to have any confidence that it’s a good replacement?
A virtualised service will only ever be as good as your understanding of an interface or more importantly, the confidence that your understanding of the interface covers all of the important aspects of how the solution is expected to work – if you carry knowledge debt in an interface, at this point in time Service Virtualisation is not for you.
You need to implement a “Strangler” type strategy to isolate these interfaces to better understand them, implement a contract style development approach to decouple your e2e estate, which should start from an architecture perspective and eventually flow through to Test,
They are now able to implement an API centric integration test strategy which in turn will lead to the test team requiring test environments that are very focussed on the purpose of the test they are performing and because the strangler patterns have come all the way through the development lifecycle.
We have collective confidence where we can isolate impact of change and virtualise out interfaces we are confident through service virtualisation – but we also understand through a continual, healthy and transparent narrative through the development lifecycle where unknowns still exist and we can concentrate our next strangler activities in this area.
Configuration management is key, not only in the code but the design assets that underpin the contract – Service virtualisation defects don’t really exist, although if you have experience in implementing this technology, it is the first thing to be blamed should a defect gets detected later in an e2e test.
Yes, we still need to do e2e, but as per the “Volcano Test Pyramid”, that needs to be reduced as we do much of this proving at the API level – and sometimes we might have a gap in our interface understanding, for which we shouldn’t abandon our strategy, rather embrace the fact that we have learned something more about the interface which improves our interface contract, our virtualised service and ultimately allows us to further decouple our once tightly coupled estate
If you are in a place of working with mostly e2e test environments as a source of truth for your test activities, there is probably a more pertinent question to ask which is not how like production is my Test environment, rather do we understand our production environment.
After all, the best copy of production is production itself, if I don’t understand something, it’s the best place to understand, analyse and use this information to feedback into development process – shift right to shift left – striving for continual quality, through learning more about what quality really means, which is everybody’s responsibility.
Organisations are embracing new technologies to face into the environment problem, but despite bringing in these new technologies in the name of Agile and DevOps and probably buying space on the cloud, chances are you won’t see the benefits these technologies enable until you get some of these fundamental issues resolved,
Asking how like production is my Test environment is the wrong question, but it seems to be the same question being asked regardless of them changing delivery methodologies – are they really doing it different though?
A task board and a daily stand-up alone aren’t going to change the architecture of my technology estate and still the same question.
To quote Einstein, the definition of insanity is doing the same thing over and over and expecting a different outcome – so, change the question and see what answer you get, do you really understand your production environment?
Written by Richard Jordan, Test Engineering Manager at Nationwide