Root Cause "Trend" Analysis

In the software project that I work, we have a requirement to do a Root Cause Analysis for the bugs that are released to the production (Residual bugs) which needs to be done in order to fulfill the ISO quality standards that our company is certified with, as well as to improve our quality assurance process in the project. 

When it comes to root cause analysis of the issues, we can consider them in 2 aspects. 

1. Bugs found by the development team during their testing cycles
2. Bugs found and reported by the customers after product is released to the production.

Root causes analysis of the bugs in these 2 types would be different. 

Root causes in the 1st, would reveal the issues related to requirement, design and coding phases of the development life cycle. Root causes analysis of the latter will reveal the issues regarding the quality assurance process that a team has adopted.

Both types of root cause analysis is very important in order to improve the overall quality of the product being developed as well as improve skills of the human resources involved

There can be 2 ways of doing a root cause analysis

1. Analyse root cause for each bug individually, getting input from each stakeholder (may be a coder and tester) and write it down in the bug itself, and discuss the leanings at each review meeting ex: if you are in agile scrum environment, you can take them to the retrospective meeting and do a 'learnings' session, which we can consider a more active process

2. Do a 'Trend analysis' instead of analyzing each issue individually. You look for a pattern , which areas need attention more , and then take necessary actions for them which we can cosider more of a passive process.

You can decide on either way depending on the requirement that is asked by your project. In my case, I adopted the second option - that is to go for a trend analysis which was more appropriate in our case. 

This is something I tried for the first time, which goes like following.

Firstly, I decided that I would start this for the last major release we did few months ago , instead of going back to more previous releases, because the data I will have to go through will be in manageable level and be more accurate.  
We use Team Foundation Server (TFS) as our main data management system and I created a search query to get the relevant data for the last release. I filtered out the severity medium and low issues and focused on severity high issues 

Secondly I designed a set of root causes, that I thought can be use againt the issues. Following is the list.

1. This area is already tested earlier so done a smoke test
2. Scenario not tested - due to lack of time 
3. Scenario not tested - too many combinations to test so missed this
4. Scenario not identified - this was not identified as a test case at design phase
5. Dependencies not tested properly - effect to and from other systems - impact analysis not  done
6. Lack of valid/invalid input data usage while testing
7. Not tested this particular scenario in particular environment
8. Edge case scenario - very unlikely to test
9. Requirement not understood correctly
10. Specification not studied properly
11. Scenario not covered during testing
12. Customer specific modification- not standard
13. Not existing feature- But customer expects to have
14. Not a bug- Customer mis-configuration
15. Known issue - May not be fixed either due to technical difficulty or not prioritised
16. Difficult to test due to non-existing resources/ skills /environments (ex: language expertise, lotus notes)
17. Bug already added to TFS but not prioritized for fix 

Thirdly, I imported the issues list to an excel sheet and try to guess root cause for each issues from the list i created about. 

Fourthly, I tried to plot the above information in to a bar chart. it looks like following.

Based on the graph, I could see some area that I can improve on. So I thought this is a good learning experience and worth giving it a try next time as well :)

Unit Tests With Telerik- A Simple End-To-End UI Test Automation Solution

Thinking about automating functional tests (UI) for the web application that I work with, and having the idea to make it an "End-to-end" solution where tests are written and pushed to the test runner, are then picked up and run against SUT automatically, and finally a result of the test status is sent to user in the form of a mail, following was an attempt that I made during last couple of days which I guess works fine at the moment. Sharing the experience hoping it will shed some light to others

If you have a working CI environment like Jenkins or TFS, it may be handy as it  will do the most of the things out of the box, but in my case I had to come up with my own way of achieving it.

The entire project can be found here .

Following technology/ tools stack was used to achieve this

 1. Telerik Testing Framework - This is the tool used for writing UI tests. You might probably be using Selenium webdriver over Telerik as your preference  
 2. Visual Studio 2013/15- 'VSUnit' unit-test framework with C#  
 3. Python scripting language - To automate some background tasks like sending mail, reading log files, achieving files etc.  
 4. DOS batch files- To initiate the test building and running  
 5. SMTP Server - This is to send out the status mail that is composed at the end of the test run. I use a free version of an easy to setup mail server called MailEnable for this purpose.  
 6 . Windows Task Scheduler -To schedule test runs  
 7. Command line tools - MSBuild.exe, MSTest.exe - This is to build the visual studio project and run the tests in 'Test Runner' machine  
 8. Dedicated VM for running the tests- Which I call it the 'Test Runner'. Test Runner will run the UI tests at scheduled time without user-intervention  
 9. Github repository for version controlling and pushing tests to 'Test Runner' after they are being developed and tested locally.  
 10. Git command "git pull" to pull new changes back to the test runner  

Telerik Test Studio is a commercial grade UI Test automation platform that helps QA professionals to automate their functional test in Web , Silverlight and WPF applications. 

Telerik Test Framework is their .net library that the Test Studio is built upon which can be freely downloaded from there site. Its free to use under certain conditions. It contain automation infrastructure library + various other libraries to support browsers, proxy , Silverlight etc. We will have to write code to accomplish what is expected from our tests.

We adopt the 'Page object model" when handling locators which could reduce potential maintenance overhead in the long run. If you want to know what it is, click here and here

Once the Telerik Framework is installed in the PC, we can start using it via Visual studio.

We start with a new project in VS, and select "Unit Test Project" from "Test" under "Templates"

 We will need to add references to the solution to make it work

Then use "VsUnit" test under "Telerik TestingFramework" . VsUnit is the Visual studio team test with unit testing framework. Since we develop our tests using Visual studio its a better option to use VsUnit rather than other unit test frameworks lik MBUnit or NUnit

In VsUnit test template, you get several methods and you can write your tests under [TestMethod] section

Once tests are built, they will appear in "Test explorer" in VS

we can manually execute tests from the "Test explorer"

The project  structure looks like following

I have developed some utility methods that can be reused in tests which ease the development efforts. You will find them under 'common methods' folder

Following methods are available.

 1. A separate class has been written to handle database interactions. It has methods to connect, execute and retrieve result from the database.  
 2. A method is implemented to capture screenshots and error logs, which can be used for error handling purposes.  
 3. An email sending method has been added which can be called in a test when you need to send a mail to a recipient.  
 4. A method to provide keyboard inputs to a text field in your application. It has been overloaded so that it can handle text fields in both documents and iframes.  
 5. A random data generating method which can be used as an input.  

Tests can be scheduled to run automatically against a SUT in the following manner.

We can create a batch scripts to initiate the process, first by pulling the new changes from the remote repository to the test runner, and then run MSBuild.exe to rebuild the project and run the test using MSTest.exe. Here the result output is directed to a log file instead of them being printed in the console. This log file later be fed to the email body.

 @echo off 
 "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\MSBuild.exe" "C:\GIT\Telerik\CS.sln" /p:configuration=debug  
 call pull.bat
 "C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\MSTest.exe" /testcontainer:"C:\GIT\Telerik\CS\bin\Debug\CS.dll" /testsettings:"C:\GIT\Telerik\Settings.testsettings" >>"C:\GIT\Telerik\CS\TestResults\Summary.log"  
 exit  

Also add a testsettings file to the project where we can invoke some actions before and after the test execution

For example, here I call a script to delete files in the result log before a new test run begins and call a batch script to send out the mail after the test run is finished. 

How the mail is composed (by reading log files and attaching error-dumps), is done using a python script which can be found in the git repo.

 https://github.com/tharinda2012/Telerik/blob/master/CS/SupportingScripts/SendSummaryMail.py  
 https://github.com/tharinda2012/Telerik/blob/master/CS/SupportingScripts/utils.py  

The parameters that must be fed to the tests can be given in a config like following

The Status of each test run will be notified to the user via an email as below

In case you use mouse or key board actions inside the test scripts to achieve certain tasks, you might encounter issues when running tests if the running machines are locked (no active session). In this case following approach can be used to remedy it.

http://docs.telerik.com/teststudio/knowledge-base/scheduling-kb/keep-active-session

Future of Software Tester

A way of looking at it...

Future of tester is challenged both from 'outside-world' as well as from 'Customer within'. Customer is exploring new ways of doing business and they come up with multitude of products/ projects in their portfolio to cater to the needs of their customers.

Agile methodologies also have changed the way Customer delivers products to their end customers : The release frequency has changed from 'once-a-year' to 'once-in-2 weeks'. This huge change itself exerts enormous challenge on the shoulders of the testers.

Outside world, the testing landscape is also changing with new trends. Emergence of Cloud, Mobile, IOT (including wearables), and Big data applications require new testing paradigms.

Skills in new testing tools, virtualization technologies (including Service, OS etc), managing test data, device simulation (specially for mobile, ‘Last-mile testing’, ‘In the wild testing’ are some of the buzz words that are being echoing in the testing community

This has rendered that a software tester cannot be just a manual function tester, but he needs to be transformed to a skilled and competent tester who is able to take the future challenges.

Specialization for testers

•   Test Automation
•   Performance testing
•   User Experience testing
•   Security Testing

Testers can master any of the above areas they are passionate about Becoming an expert in each of them will provide him/her with added advantage in surviving/ excelling.

Also Proper education/training and QA certifications (ISTQB, CSTE, CEH) are also important to have a better personal profile and development.

Challenges from Customer

Frequent release model – products are being released frequently and how testers manage it in sustainable way without being over-stressed and without compromising the quality

Development team members participate in testing. – How can the tester add value to the team while developers do the functional testing?

Plethora of products- New products are being added to the portfolio which needs testing, this increases the complexity of testing environments/ scenarios etc.

Future of agile

Will the agile era be over in the next decade? Not really. Therefore the agile tester will still be alive and Customer will continue be in agile business. We need to have the role of a tester to be ‘Full stack agile tester’. 

A full stack agile tester is a generalizing QA specialist who is technically competent in all test levels, types and strategies who is also adaptive/responsive, collaborative and culturally fit as an agile team member.

What it takes to be a full-stack agile tester?

• Notion of ‘Pair’. This includes

o   Pair negotiation- healthy debate

o   Pair strength – cross skilled

o   Pair understanding- less surprises

o   Seamless workflow- less handoff

o   Pair accountability- the notion of team entails shared accountability

•  Become a USER PROXY- be in user's shoes and suggest improvements to the team. How to improve UX, challenge the initial requirements- help in fine tuning them.

•  Work the extra mile in time of need

•  Keep the team’s continuous attention to technical excellence

• Have a test automation suite and handover repetitive tasks to it while saving time and effort in exploratory testing

• Use ‘Swarming’ technique to finish a task in case of urgency with in the sprint

• Build an ultra-collaborative relationship with the team by being genuine, honest , open and by being professionally pessimistic. Ask smart questions, make suggestions

•  Play the dev/quality advocate role

o   Be a requirement champion, teach Devs how to do effective functional testing. Have good test cases created and share with the devs.

o   Remind maintainability/scalability/ other non-functional aspects during sprint planning/ other meetings

•  Become T shaped skilled tester

o   While having a thorough knowledge about the product/ project you work, have the ability to work outside of core product. Have a sense of other products. When given quickly adapt to the situation and carry out testing 

•  Finally get trained for what is lacking.

Setup IIS web server with HTTPS and enable/disable TLS

For Testing web applications over https and TLS , a tester might need to setup his web server (IIS) to support the above security protocols.

Following is a set of instructions that you can follow to achieve this.

You can secure your web server/application either by purchasing a security certificate from a vendor or by creating a self-signed certificate that is generated by the web server (for testing purposes).

1. Generate a self-signed certificate in IIS by going to 'Server Certificates'

2. Select the site in IIS that you need to secure (ex: "Default web site"). Then select ‘Bindings’, add https entry and choose the certificate that you created above.

3. You might need to enable/disable specific TLS versions in the server and it can be done like following

Enable TLS 1.1 and TLS 1.2
=======================
Open registry on your server by running ‘regedit‘ in run window and navigate to below location.
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols
Add the TLS 1.1 and TLS 1.2 keys under Protocols. It will looks like directories.

Now create two keys Client and Server under both TLS keys.

Now create the DWORD Values under Server and Client key as following

DisabledByDefault [Value = 0]
Enabled [Value = 1

Disable SSLv3 and Older Version
=================================
Open registry on your server by running ‘regedit’ in run window and navigate to below location.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols

Now change DWORD Values under Server and Client under SSLv3 and Older SSL version keys.

DisabledByDefault [Value = 0]
Enabled  [Value = 0]

After making all above setting reboot your server.

4. You can now verify that the web server TLS protocols are disabled and https works by using TestSSLServer executable from http://www.bolet.org/TestSSLServer/

TestSSLServer.exe SERVERNAME  443

5. Finally to make the certificate work, you will need to imported certificate to client machine 'Trusted root certificate authorities' using MMC.

Performance Testing challenges

#1 Create a valid load (Load validity)

The first challenge is the selection of a subset
of different transactions from the set of all
transactions that can be executed by clients.
The spread of transactions, and the percentage
of each transaction in relation to each other is
of the essence here. Whenever there are backoffice
transactions that have to be taken into
account, it might be necessary to verify the
impact of their execution upon the response
time experienced by users (more on this in the
second article in this series).

#2 Create realistic users (User Realism)

The second challenge is to have the users behaving
as realistically as possible, and thus
generating as realistic a load as possible in
terms of spread of data within a single transaction
(i.e. not always looking for the same set of
www.testingexperience.com The Magazine for Professional Testers 25
data in a table).


#3 Validate the responses (Response validity)

A standard practice for testing is to test the
actual results against expected results. In performance
tests, what often happens is that the
transaction responses are timed. This relies on
the responses being of adequate quality (functionally).
This is contrary to a basic tenet of
software testing that implies that you cannot
rely on an answer to remain the same if conditions
change. When one is stressing a system,
such a reliance is unwarranted, and the
response validity is to be verified.

#4 Reduce the number of injectors 1(Injector
reduction)

Simulating a large number of clients is the
fourth challenge. Here a fine balance exists
between adding more machines and simulating
more clients per machine. The first option
has an impact in terms of hardware (adding
injectors, connecting and synchronizing them
during campaigns), while the second has an
impact in terms of load on the client (injector)
machine, as simulating a large number of
transactions and validating the responses obtained
creates a load on the client hardware.

#5 Transactions coherence (Transaction realism)

A fifth challenge comes in terms of the validity
of the stream of transactions coming from
a client. This is the case when the transactions
involve different screens linked to each other,
such as when purchasing an item from an online
store, or processing banking transactions.
This requires the server to keep data linked to
the opened session through means other than
cookies on the client side or inclusion of the
full information in the request 2.

#6 Encryption (Encryption impact)

The use of virtual private network (VPN) and
dynamic encryption methodology to ascertain
that the transactions are secure can have an impact
on the ability of capture replay to create
transactions that will be correctly interpreted
by the server and not rejected due to invalid
encryption keys (that would happen in case
of encryption based on keys synchronized between
server and client).

Extracted from testingexperience magazine "Agile testing" Sept,2009 http://www.testingexperience.com/

Introduction to Performance Testing

Introduction to Performance Testing from Tharinda Liyanage

Coded UI : Hand-Coding based on "Page Object Model"

Coded UI is a User Interface (UI) Test Automation Tool that comes with Microsoft Visual Studio which can be used to automate our manual, repetitive and regression tests thereby increase productivity of Testing teams.

Last few weeks I was able to work with Coded UI with VS 2012 and here I'd like to share few experience I gathered during the project.

Generating code with Test builder

Coded UI comes with built-in Test building features where a tester just have to invoke the test builder in back ground and execute the testing scenarios in the application under test. The test builder is capable of generating the code for the recorded scenario and you are done. Now its ready to play it back. Quite simple and convenient for a non-coding tester. But the main drawback of this method is that the maintainability and readability of the code is extremely difficult due to reasons like, huge number of code being generated (most of which are redundant), the naming convention is quite complex, if you have to modify your test steps, then it becomes a nightmare for the tester.

Hand-Coding

This is the approach That I was interested in when the above factors are taken in to account. Even though this method is time consuming and initially difficult, I believe its a good investment for creating a sustainable test suite which is more flexible and maintainable.

 following is an extract of a presentation created by me based on the experience I gathered during this period

 A word on CUITe...

Experiences working with an agile team as a tester

I had an opportunity of working in an agile software team as a software tester for last two and half month. Even though the time I spent there was small, I believe I could learn a lot for that time period. 

It was September 1st, 2011 that I was first introduced to the team. I used to work in traditional software projects and this was a new experience for me. I was quite new to the team, quite new to the product domain, quite new to the agile methodology. So I had to face those three challenges in order to fulfill the expectations of the team.

The project team comprised of 4 developers, BA and a tester. BA is also the project manager and the scrum master since we used SCRUM as the development methodology. The customer also played a significant role in the project. The customer comprised of company's CTO, Product owner,Project Manager cum QA manager, an Architect and 2 more developers. All work as a single team during the project. The teams were geographically dispersed.

Even though the team is diversified in terms of culture and geography, I found one common attribute that the entire team posses, that is the easy going and flexibility to the change. Specially the customer had a good understanding as to how a new project would evolve and he was expecting both best and worst case scenarios. We, as a team had ups and downs during first few weeks with regard to sprint/task estimations, communications, technical issues, team velocity issues, lack of domain knowledge issues and many more. And I was quite surprised to see that the customer was in better understanding and actually he was foreseeing them before us. 

The first task that I had to do as the QA person is to come up with a testing strategy for the project. The team had no prior knowledge of proper testing before and no process was followed before. I used to refer the following text book "Agile Testing: A Practical Guide for Testers and Agile Teams"  and found very useful and I recommend that every tester in agile teams should read this book.

to be continued...

Formulating a testing strategy for an agile software project

Here is a power point presentation that I conducted recently with regarding 'Formulating a testing strategy for an agile software project.

http://www.slideshare.net/tharindakasun/agile-testing-strategy

Agile Testing Strategy

View more presentations from tharindakasun.

Importance of software testing matrics

I found some nice articles on using 'Matrics' in software testing practices. The 'Testing experience (te)-The magazine for professional testers' (http://www.testingexperience.com/) is one of the most useful magizines published on software testing has devoted its september volume talking about software matrics.

One of the articles in this volume named as 'Go Lean on Your Software Testing Metrics' by Jayakrishnan Nair grabbed my attention a lot and I found its very useful. And I hope some points there worth noting here for reading in case if someone does not have the access to the magazine.
All credit should go to the magazine and the original authors


A Basic Set of Software Testing Metrics
A small set of basic metrics for software testing that are useful for
many organizations is listed below.
• Test Coverage
• Productivity
• Defect Detection Effectiveness
• Defect Acceptance Ratio
• Estimation Accuracy

Test Coverage

The Test Coverage metric measures the extent to which testing
has covered the requirements of the application under test. If this
is not close to 100%, it means that portions of the application remain
untested and therefore undetected defects may be present
in the product. Therefore it is a leading indicator of quality of the
final deliverable and therefore on customer satisfaction (which
is very likely a business goal for the company). Additional time
spent fixing defects in production can impact time to market
the product. Poor customer satisfaction due to quality issues or
a delayed product launch can adversely impact future revenues

Productivity

Productivity is a fundamental metric in software engineering; it
tells you how fast work gets done. In the context of testing, Productivity
can be defined for test design and execution as shown
in the table below. (It can also be defined similarly for other testing
tasks such as test data set up).

Defect Detection Effectiveness
(Percentage of the total number of defects reported for the application that are reported during the testing stage)
is a measure of how effective the testing process is in detecting defects and not letting them
pass through to the next stage in the software lifecycle.


Defect Acceptance Ratio(Percentage of defects reported that are accepted as valid)

If the DAR value is too low, it means that testers are reporting too
many defects that are invalid. This has a direct impact on Productivity.
When an invalid defect is recorded, effort is wasted not only
by the testing team but also by the development team as they
have to process the defect record anyway and prove that it is invalid.
(If very frequent, this may negatively affect the relationship
between testers and developers, again adversely impacting the
project.) A large number of invalid defects may clog the defect
tracking system, which makes it hard to locate useful information

Estimation Accuracy

Estimation Accuracy measures how closely the actual effort spent
in testing tracks the effort estimated in the beginning:
and may add to the maintenance costs of the tool.

Software Testing Strategies

I found a nice set of articles on formulating a test strategy for a software product. It is presented in 'tell a story' way which I hope is much more eye-catching than any other methods. All credits to www.edistalearning.com

1. Opening scenario

• Click this link

2. Planing software testing

• Click on this link
• Hover mouse pointer over each box that appears to view detailed description of each item

3. The testing strategy- Your testing strategy will be based on how your system is built. The software process should be examined to understand how a testing strategy can be designed.

Analysis->design->code->test


4. Roles and Organization for Testing -Who is going to implement the strategy

• Click on the link