Thinking about automating functional tests (UI) for the web application that I work with, and having the idea to make it an "End-to-end" solution where tests are written and pushed to the test runner, are then picked up and run against SUT automatically, and finally a result of the test status is sent to user in the form of a mail, following was an attempt that I made during last couple of days which I guess works fine at the moment. Sharing the experience hoping it will shed some light to others
If you have a working CI environment like Jenkins or TFS, it may be handy as it will do the most of the things out of the box, but in my case I had to come up with my own way of achieving it.
The entire project can be found here .
Following technology/ tools stack was used to achieve this
1. Telerik Testing Framework - This is the tool used for writing UI tests. You might probably be using Selenium webdriver over Telerik as your preference
2. Visual Studio 2013/15- 'VSUnit' unit-test framework with C#
3. Python scripting language - To automate some background tasks like sending mail, reading log files, achieving files etc.
4. DOS batch files- To initiate the test building and running
5. SMTP Server - This is to send out the status mail that is composed at the end of the test run. I use a free version of an easy to setup mail server called MailEnable for this purpose.
6 . Windows Task Scheduler -To schedule test runs
7. Command line tools - MSBuild.exe, MSTest.exe - This is to build the visual studio project and run the tests in 'Test Runner' machine
8. Dedicated VM for running the tests- Which I call it the 'Test Runner'. Test Runner will run the UI tests at scheduled time without user-intervention
9. Github repository for version controlling and pushing tests to 'Test Runner' after they are being developed and tested locally.
10. Git command "git pull" to pull new changes back to the test runner
Telerik Test Studio is a commercial grade UI Test automation platform that helps QA professionals to automate their functional test in Web , Silverlight and WPF applications.
Telerik Test Framework is their .net library that the Test Studio is built upon which can be freely downloaded from there site. Its free to use under certain conditions. It contain automation infrastructure library + various other libraries to support browsers, proxy , Silverlight etc. We will have to write code to accomplish what is expected from our tests.
We adopt the 'Page object model" when handling locators which could reduce potential maintenance overhead in the long run. If you want to know what it is, click here and here
Once the Telerik Framework is installed in the PC, we can start using it via Visual studio.
We start with a new project in VS, and select "Unit Test Project" from "Test" under "Templates"
Then use "VsUnit" test under "Telerik TestingFramework" . VsUnit is the Visual studio team test with unit testing framework. Since we develop our tests using Visual studio its a better option to use VsUnit rather than other unit test frameworks lik MBUnit or NUnit
In VsUnit test template, you get several methods and you can write your tests under [TestMethod] section
Once tests are built, they will appear in "Test explorer" in VS
we can manually execute tests from the "Test explorer"
The project structure looks like following
I have developed some utility methods that can be reused in tests which ease the development efforts. You will find them under 'common methods' folder
Following methods are available.
1. A separate class has been written to handle database interactions. It has methods to connect, execute and retrieve result from the database.
2. A method is implemented to capture screenshots and error logs, which can be used for error handling purposes.
3. An email sending method has been added which can be called in a test when you need to send a mail to a recipient.
4. A method to provide keyboard inputs to a text field in your application. It has been overloaded so that it can handle text fields in both documents and iframes.
5. A random data generating method which can be used as an input.
Tests can be scheduled to run automatically against a SUT in the following manner.
We can create a batch scripts to initiate the process, first by pulling the new changes from the remote repository to the test runner, and then run MSBuild.exe to rebuild the project and run the test using MSTest.exe. Here the result output is directed to a log file instead of them being printed in the console. This log file later be fed to the email body.
@echo off
"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\MSBuild.exe" "C:\GIT\Telerik\CS.sln" /p:configuration=debug
call pull.bat
"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\MSTest.exe" /testcontainer:"C:\GIT\Telerik\CS\bin\Debug\CS.dll" /testsettings:"C:\GIT\Telerik\Settings.testsettings" >>"C:\GIT\Telerik\CS\TestResults\Summary.log"
exit
Also add a testsettings file to the project where we can invoke some actions before and after the test execution
For example, here I call a script to delete files in the result log before a new test run begins and call a batch script to send out the mail after the test run is finished.
How the mail is composed (by reading log files and attaching error-dumps), is done using a python script which can be found in the git repo.
https://github.com/tharinda2012/Telerik/blob/master/CS/SupportingScripts/SendSummaryMail.py
https://github.com/tharinda2012/Telerik/blob/master/CS/SupportingScripts/utils.py
The parameters that must be fed to the tests can be given in a config like following
The Status of each test run will be notified to the user via an email as below
In case you use mouse or key board actions inside the test scripts to achieve certain tasks, you might encounter issues when running tests if the running machines are locked (no active session). In this case following approach can be used to remedy it.
