Welcome!

Wearables Authors: Pat Romanski, Liz McMillan, Elizabeth White, Dana Gardner, Peter Silva

Blog Feed Post

Is It Fair to Compare Manual and Automated Test Results?

​The article below was originally published by Rajini Padmanaban on the fantastic QAInfoTech blog.

Manual and automated testing have long co-existed and have contributed significantly in successfully signing off on a product’s quality. Over the years test teams have talked about the value of one over another, how testers need to groom their test automation skills and so on. A tester who really understands software quality and the intricacies it entails, will be one that appreciates the value of both these testing approaches and that one cannot be under or over-estimated over the other. With more organizations embracing the Agile style of development, test automation’s scope has definitely gone up. There is an increasing push for newer areas to be automated, newer test engineers to be trained to take on automation, newer tools to help with the process etc. Test managers and leads are being cautious in doing so, because an unplanned test automation effort can soon become a mammoth suite that is overwhelming to manage and maintain.

This is where an objective comparison between manual and test automation efforts is becoming important to determine their true value and make any adjustments that may be needed to further optimize them for the product’s benefit. You may ask – is it fair to compare the manual and automated test efforts and this is a valid question. Often times, it may not even be an apple to apple comparison. For example, the manual test focus may have been on the UI intensive portions of the application, which have been buggier. As a result, the test team may have reported a lot of valid defects through their manual efforts. On the other hand, test automation’s focus may have been API intensive areas, application’s performance, database level functionality etc. where defects are more difficult to find. It could also be the case, that APIs have been re-used from the past and are very stable that the tester has not reported as many bugs through test automation. Does this mean that the automation was not effective?  

While this argument holds weight, and that manual and automated test results comparison is not always apples to apples, it does not mean they cannot be compared. They need to compared on the right grounds to help the team arrive at the right balance between the two test approaches and also what, when, how and how much to automate. If so, how can they be objectively compared? Here are some tips:

  1. Pick areas that have been currently automated, see what bugs are reported through them and also have some of those areas tested manually, in parallel. Compare the results to see if the automation scenarios need any modification. This can also help add the manual tester’s instinctive creativity into the automated suite
  2. Understand the % of valid defects reported by manual and automated test efforts. This will help understand if automation has a lot of test/data/configuration issues that need to be attended to and fixed  
  3. After an exploratory round of testing or a bug bash, compare manual test results with the corresponding automation that may exist in those areas to see how the test automation suite can further be strengthened
  4. Look at overall numbers to a certain extent. If manual tests are yielding way too many bugs but hardly any are coming in through automation, it may be an indication for further analysis. This may end up being a false alarm, but is at least worth a quick look
  5. Compare the kind of defects reported by the two test approaches. If manual is reporting more UI and functional defects in a certain area, while automation is not catching them, this is a good checkpoint to see how the automation’s can be enhanced
  6. Use automation to also evaluate the manual testing effort’s efficiency. The comparison is not a one sided story. Sometimes the manual testers’ on the team may not be very efficient and may be careless in their test effort. Automation, once tested for its reliability and consistency in results, is a great approach to minimize human errors. So, say for instance a certain regression has been automated and the tester also happens to play around with that area manually, a manager can compare the results to understand the tester’s efficiency and areas of improvement

The above is certainly not an exhaustive list. However, when this mindset is established in the team, they will begin to appreciate the need for such an objective comparison between the manual and automated test approaches. This will be a new category that can be incorporated in the metrics that they use in line with the need of the current day, helping the two test approaches complement each other.

About the author:

Rajini Padmanaban is a Sr. Director of Testing Engagements at QA InfoTech and an active software testing evangelist. She has more than twelve years of professional experience, primarily in the software quality assurance space.

Read the original blog entry...

More Stories By Skytap Blog

Author: Noel Wurst is the managing content editor at Skytap. Skytap provides SaaS-based dev/test environments to the enterprise. Skytap solution removes the inefficiencies and constraints that companies have within their software development lifecycle. As a result, customers release better software faster. In this blog, we publish engaging, thought provoking stories that revolve around agile enterprise applications and cloud-based development and testing.

@ThingsExpo Stories
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, will provide an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life ...
Join us at Cloud Expo | @ThingsExpo 2016 – June 7-9 at the Javits Center in New York City and November 1-3 at the Santa Clara Convention Center in Santa Clara, CA – and deliver your unique message in a way that is striking and unforgettable by taking advantage of SYS-CON's unmatched high-impact, result-driven event / media packages.
Internet of @ThingsExpo, taking place November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with the 19th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world and ThingsExpo New York Call for Papers is now open.
SYS-CON Events announced today that Commvault, a global leader in enterprise data protection and information management, has been named “Bronze Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY, and the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Commvault is a leading provider of data protection and information management...
18th Cloud Expo, taking place June 7-9, 2016, at the Javits Center in New York City, NY, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Meanwhile, 94% of enterprises are using some...
The IoTs will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform. In his session at @ThingsExpo, Craig Sproule, CEO of Metavine, will demonstrate how to move beyond today's coding paradigm and share the must-have mindsets for removing complexity from the development proc...
SYS-CON Events announced today that Tintri Inc., a leading producer of VM-aware storage (VAS) for virtualization and cloud environments, will exhibit at the 18th International CloudExpo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, New York, and the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that Enzu, a leading provider of cloud hosting solutions, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Enzu’s mission is to be the leading provider of enterprise cloud solutions worldwide. Enzu enables online businesses to use its IT infrastructure to their competitive advantage. By offering a suite of proven hosting and management services, Enzu wants companies to foc...
SoftLayer operates a global cloud infrastructure platform built for Internet scale. With a global footprint of data centers and network points of presence, SoftLayer provides infrastructure as a service to leading-edge customers ranging from Web startups to global enterprises. SoftLayer's modular architecture, full-featured API, and sophisticated automation provide unparalleled performance and control. Its flexible unified platform seamlessly spans physical and virtual devices linked via a world...
The IoT is changing the way enterprises conduct business. In his session at @ThingsExpo, Eric Hoffman, Vice President at EastBanc Technologies, discuss how businesses can gain an edge over competitors by empowering consumers to take control through IoT. We'll cite examples such as a Washington, D.C.-based sports club that leveraged IoT and the cloud to develop a comprehensive booking system. He'll also highlight how IoT can revitalize and restore outdated business models, making them profitable...
SYS-CON Events announced today that BMC Software has been named "Siver Sponsor" of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2015 at the Javits Center in New York, New York. BMC is a global leader in innovative software solutions that help businesses transform into digital enterprises for the ultimate competitive advantage. BMC Digital Enterprise Management is a set of innovative IT solutions designed to make digital business fast, seamless, and optimized from mainframe to mo...
As cloud and storage projections continue to rise, the number of organizations moving to the cloud is escalating and it is clear cloud storage is here to stay. However, is it secure? Data is the lifeblood for government entities, countries, cloud service providers and enterprises alike and losing or exposing that data can have disastrous results. There are new concepts for data storage on the horizon that will deliver secure solutions for storing and moving sensitive data around the world. ...
Customer experience has become a competitive differentiator for companies, and it’s imperative that brands seamlessly connect the customer journey across all platforms. With the continued explosion of IoT, join us for a look at how to build a winning digital foundation in the connected era – today and in the future. In his session at @ThingsExpo, Chris Nguyen, Group Product Marketing Manager at Adobe, will discuss how to successfully leverage mobile, rapidly deploy content, capture real-time d...
SYS-CON Events announced today that ContentMX, the marketing technology and services company with a singular mission to increase engagement and drive more conversations for enterprise, channel and SMB technology marketers, has been named “Sponsor & Exhibitor Lounge Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York City, New York. “CloudExpo is a great opportunity to start a conversation with new prospects, but what happens after the...
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo 2016 in New York and Silicon Valley. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place Nov 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 17th Cloud Expo and will feature technical sessions from a rock star conference faculty ...
SYS-CON Events announced today the How to Create Angular 2 Clients for the Cloud Workshop, being held June 7, 2016, in conjunction with 18th Cloud Expo | @ThingsExpo, at the Javits Center in New York, NY. Angular 2 is a complete re-write of the popular framework AngularJS. Programming in Angular 2 is greatly simplified. Now it’s a component-based well-performing framework. The immersive one-day workshop led by Yakov Fain, a Java Champion and a co-founder of the IT consultancy Farata Systems and...
IoT generates lots of temporal data. But how do you unlock its value? How do you coordinate the diverse moving parts that must come together when developing your IoT product? What are the key challenges addressed by Data as a Service? How does cloud computing underlie and connect the notions of Digital and DevOps What is the impact of the API economy? What is the business imperative for Cognitive Computing? Get all these questions and hundreds more like them answered at the 18th Cloud Expo...
SYS-CON Events announced today that EastBanc Technologies will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. EastBanc Technologies has been working at the frontier of technology since 1999. Today, the firm provides full-lifecycle software development delivering flexible technology solutions that seamlessly integrate with existing systems – whether on premise or cloud. EastBanc Technologies partners with p...
Companies can harness IoT and predictive analytics to sustain business continuity; predict and manage site performance during emergencies; minimize expensive reactive maintenance; and forecast equipment and maintenance budgets and expenditures. Providing cost-effective, uninterrupted service is challenging, particularly for organizations with geographically dispersed operations.
The essence of data analysis involves setting up data pipelines that consist of several operations that are chained together – starting from data collection, data quality checks, data integration, data analysis and data visualization (including the setting up of interaction paths in that visualization). In our opinion, the challenges stem from the technology diversity at each stage of the data pipeline as well as the lack of process around the analysis.