Thoughts on test automation

Having worked on projects that are introducing test automation, projects where test automation is almost the only means to test and in projects trying to develop test automation, here's my two cents on the subject. All the projects have had the common definition of test automation, which is what I call "Press the button and go home" test automation. No human interaction required. This means that the tool should setup the test environment, perform the action part of the scripted test and then perform the result observation and validation of the scripted test. I say scripted test since no other kind of test fits this test automation practice. James Bach discussed this topic during his course and without trying to write to much of what he thinks regarding this his definition is more in the line of "anything that does something automatic with regards to testing" and by a rule of thumb he would not bother if the tool required took more than 48 hours to produce. The key here is not the 48 hour rule but the wider term which can lead to a more effective use of test automation. Before I continue with that, I just want to point out the reason for why you would want to do any kind of automation, and the main reason is that you want to speed up the test process. This is why the wider term is so effective because that includes all efforts made to by automatic means speeds up the process, i.e. a tool to calculate all different combinations of given set of variables can speed up the test design time, or a tool to compare log files speeds up the validation, all tools related to create load automatically on the system are also test automation tools.

Having said that, and you still feel like to focus on "Press the button and go home" test automation, at least consider the following. A scripted test generally consists of some test preparation, some action steps and some expected result parts. If you separate the different activities in your automation effort you will both be more resilient to change, and can help to speed up manual testing as well. The proposal is to create at least three different tools and make sure they work together as well as independently. The three tools are the following:

• Test environment control
• Test data generation
• Test data analysis

Test environment control is a tool that make sure that the test environment is in the required state for the test case. I.e. install and configure the correct software, bring the system to a known and desired state, configure and start other tools that are needed for the specific test etc.

The test data generation is a tool that generate actions in the system, i.e. simulates a user, simulate different failures. The generation tool should not validate the system but record all the actions it has performed in the log.

The data analysis tool is the tool which validates that the performed actions produced the desired results.

What I like about this is also that it can support any sapient test activities and you can work on realizing the parts which you need the most first.

Course with James Bach

I've just finished a great course with James Bach where many of my ideas and thoughts about testing got very much clearer. My thoughts and insights, among many are the following: Think for yourself! Define what you do and evolve that into something better. Learn from your and others mistakes.

My problem with testing so long has been the lack of common definitions, the lack of arguments and the unreasonable faith in things. If one tester says "System Testing" one tester will think non-functional testing such as stress testing, performance testing, robustness testing. Another tester will think functional testing of the entire system. Or as the ISTQB defines: "The process of testing an integrated system to verify that it meets specified requirements.". Which actually does not say anything. As for the arguments I have often heard things like "You should do like this." and upon the the question why you get something like "Because it is best." or "Because we have always done this." or similar. Which leads to the blind faith thing, which of course goes against what Mr Bach tries to teach, to think for yourself.

Currently I feel that this is still not something you can be told but something you have to experience, but what I ask is that when you do. Please try to learn from it.

Restart

The questions and misunderstandings regarding tests are spread far and wide. On the other hand the common definitions, methods, tools, techniques are almost tangible in their absence. Just ask two different persons that have been working with testing for a definition and I almost guarantee that you will get answers that can be so different that you would not even believe that you asked for the same thing. I'm not an authority on the subject but after over 10 years in the business I feel that an effort should be made towards clarifying the world of testing a little more.

So here goes, a journey has begun, where it will take me is still unknown, but hopefully the testing community will benefit in some way from it.