Monday, October 21, 2013

How to Take the Decision-Build or Buy a Software Testing Tool

After the software testing manager has completed his/her evaluation of the commercial market, it can be found that there are no tools that meet the requirements within all constraints. It is worth considering whether it might be better to build their own (or wait for the market to catch up with you).

If you build your own tool:

#It will be most suitable for your own needs;

#You may be able to compensate in the tool itself for a lack of testability in the software under test;

#The software testing tool may be able to assume knowledge of your own applications, thereby reducing the work necessary to implement automated tests;

#It will probably not be very well supported in terms of documentation, help, and training;

#It may suffer from 'image' problems ('something developed by Joe in the next office can't possibly be as good as the tool described in this glossy color brochure from a tool vendor');

#The user interface may leave something to be desired (a software testing tool developed by technical people often considers ease of use to be unimportant).

If you buy a commercially available tool:

# It will probably be considerably cheaper to achieve a given level of features and quality (such as usability) than the cost of developing the software testing tool yourself (the cost of commercial tool development is spread over a large number of users; you have to bear all of that cost yourself if you develop your own);

# It should be well supported in terms of documentation, help, training, etc.

@ It is often seen as 'attractive' - something people want to be involved with;

@You will not entirely avoid building your own even if your basic engine is a commercial tool; you will still need to build support for your test automation regime.

If you do build your own, do not attempt to produce a Software Testing tool on the same scale as the commercial tools. Remember in most cases they have been under development for many years, funded by numerous users in many organizations. Build the smallest and simplest software testing tools that will give you immediate and real benefit. You might start by building some filters to use with your existing comparison tools, for example. You could collect information about any existing tools or utilities that different people or groups in your organization have already developed to meet some need within their own scope; these starting points can often be developed at minimal cost to give more general benefits to the organization.


Important Activities involved in Organizing the Test Procedures

The low-level test cases should now be organized and assembled in test procedures and / or test scripts.

The term “procedure” is mostly used by the software testing engineers when they are prepared for manual test execution, while the term “script” is mostly used for automatically executable procedures.

The degree of detail in the procedures depends on who will be executing the test. They should therefore always be written with the intended audience in mind. Experienced software testing engineers and people with domain knowledge and knowledge about how the system works will need far less details than “ignorant” software testing engineers. What we need to specify here is the actual sequence in which the test cases should be executed.

The documentation of a test procedure must at least include:    

#Unique identification
# Description
# References to high-level test cases and/or to test conditions and/or directly to basis documentation to be covered by the procedure
# An explicit description of the preconditions to be fulfilled before the actual test execution can start
# Included low-level test cases

Intelligent software testing engineers organize their test procedures in such a way that the execution of one test procedure sets up the prerequisites for the following. It must, however, also be possible to execute a test procedure in isolation for the purpose of confirmation testing and regression testing. The prerequisites for a test procedure must therefore always be described explicitly.

Test procedures may be hierarchical, that is “call others,” for example, generic test cases.

The test groups and the specification of their test procedures must be revisited to ensure that they are organized to give a natural flow in the test execution. Remember that the production of the test specification is an iterative process. We need to keep on designing and organizing test cases, test procedures, and test groups until everything falls into place and we think we have achieved the required coverage.

The organization in test procedures could be looked at as the execution schedule. It could be fixed, but it could also be dynamic. For specific purposes, especially for regression testing, some of the test procedures may be selected and reorganized in other execution schedules that fit the specific software testing purpose. A test procedure should not include too many or too few test cases - a maximum of 20 test cases and a minimum of 2 to 4 test cases is a good rule of thumb.

The test procedure may also include facilities for logging the actual execution of the procedure.

There are many ways to lay out the specification of test procedures and test cases. It is a good idea to set up a template in the Software Testing and development organization.

Problems Faced by Software Testing Engineers in implementation of test tools

The implementation of development tools tends to be planned when it should be, at the start of a project. Unfortunately, this is not always the case for software testing tools. This presents some additional problems.

Interfaces to other tools and systems:

The software testing tools may need to work with or interface to other tools that are already in place, such as configuration management, development tools, or project management tools.

The software testing tools will also have to interface to the software under test. There can be significant problems in synchronizing tests with the software, which may or may not be easy to overcome.

Time for the learning curve:

The learning time for a software testing tool seems to be more critical from a psychological point of view than the same learning time for a development tool.

Although a week of training near the end of a project takes the same amount of time as a week of training at the beginning of a project, somehow it seems longer. Actually this is one of the many reasons why a test automation tool should never be introduced to a critical project that is running late.

In a maintenance environment:

It is in maintenance that the greatest benefits can be gained from test automation, by automating regression tests that can then be run every time the system is changed. However, if the starting point is a completely manual test environment, where should testers start to automate?

Benefits can be achieved by automating only a small proportion of the tests - for example, the current regression test bottlenecks or those tests which are the most tedious or error prone to run manually, the bug fixes, or just the most critical core business functions.

Software Testing engineers need to be careful & need not to automate everything - in fact, it can not be. Aim for a gradual build-up of tests over time, with the tests being automated giving the greatest benefit for the least effort.

Over time, testers will become better and better at building automated tests, particularly in a maintenance environment. This should lead to a cycle of continuous improvement in the test automation regime.

Understanding of Improvements in Software Testing Process

Maturity is as important for software development as it is for people. When we are immature we can easily find ourselves in a situation where we lose control and are unable to solve the problems - problems we might even have created ourselves.

The demands on the software testing & development industry are growing as pervasive software thunders ahead. More and more products include software, and both embedded software and pure software products are becoming more and more complex. The potential number of faults in the software is hence increasing and so is the cost of finding and removing them through specialized software testing effort - not least keeping in mind that the cost of fault correction increases by a factor 10 for each phase the faults “survive” in the work products.

The solution to the growing demands is more professional development with focus on the entire product and hence the entire development process. The development needs to be able to stay in control, foresee problems, and prevent them or mitigate them in a mature way. Software development needs to grow up, improve, and thereby become a mature industry - and so does the software testing.

Process improvement is based on the understanding that software development is a process and that processes can be managed, measured, and continuously improved.

An important assumption is that the quality of the software produced using a specific process is closely related to the quality of the process. This does not mean that it is impossible to produce excellent software using a useless procedure - or indeed the other way around - but the probability of producing good software rises significantly with the quality of the process.

The urge for improvement can come from many places both outside and inside the organization, and both from below and above.

#Customers or suppliers may push or even demand proof of maturity and ongoing process improvement directly. More indirectly they may express requirements in terms of quality criteria and time-to-market, whose fulfillment requires a certain maturity in the software testing and development organization.

#within the organization the Software Testing managers are pressed to obey constraints and to provide growth in the organization.

#Finally employees may well be fed up with constant firefighting and impossible deadlines requiring them to work overtime and cut corners.

Friday, October 4, 2013

Design of Software Testing Strategy to Suit Project Related Factors

A good software testing strategy is shaped not only by product risks, but also by factors within the project. Technical experts with advanced level ISTQB certification prescribe following project-based strategic principles:

1) We should not lose our bugs in the cracks between the software testing engineers. Unless we apply diverse half measures or overlapping software testing assignments, we face a real risk of not testing something because it lies on the boundary between the assignments of two testers (or teams).

2) We should frequently test what we have been asked to test. We should remember that we are testing on behalf of many clients. What do they think we should test? We should find out and make sure that we are doing at least some of it.

3) Occasionally we should test what we are asked not to test. Sometimes we are asked not to test certain parts of the product. This can be a delicate matter, and it is difficult to tell as to what to do. However, sometimes the things we are asked not to test are the things that need it most.

4) We should test the confusion and conflict. Wherever there is confusion and conflict, bugs will certainly come. If the programmer seems not quite sure what a feature is supposed to do, we should test it. If he is new to the technology, we should test it. If two programmers are building units that interface with each other, we should test the interface. This way we will not be disappointed.

5) We should not beat a dead feature. When it is clear that a feature seems full of bugs, we should not continue to test it unless we check in with the developer. It may be a bad build or a bad configuration. Also, if a component is so bad that it is going to be replaced rather than revised, any bugs we find will be summarily closed, so we should not bother software testing.

6) More change means more Software Testing. Theoretically, the smallest change in the product can create large and non-local effects. That means any change can potentially invalidate all the software testing we have ever done on the product. In reality, most changes have a fairly localized effect. However, it is certainly true that we must follow the changes with our software testing. The more change in the product, the more software testing we must do. This becomes a big deal in the end game of a project.

These principles have been pointed out in the syllabus for ISTQB Certification CTFL as well.