RBCS Blog
December 26th, 2011 by Rex Black
I received the following message from Helen Huang. My comments are found below, inline…
Dear Rex ,
You are my idol, I am your fan
Thanks, Helen. I appreciate your trust.
I am tester .I has work this job of 5 years on China. Now I meet some problem in my working.
1. In china , I find many company always talk the tester value ,what is the tester value ? . eg: Find the number of bug Or find a bug problem ?
This is a common problem. Test organizations often do not have clearly defined objectives. This makes it very hard to demonstrate value. The first step to demonstrating the value of testing is to work with stakeholders to define what testing should contribute. I wrote about this in Chapter 2 of Beautiful Testing, which I’d encourage you to use as a way to making the value of testing measurable.
2. In China, many companies will consider automation and performance testing as a standard test KPI?
Performance is often a key value for testing. It’s important to understand the need for performance testing, as it is quite expensive to do well. I’d encourage you to check the RBCS Library, especially the Digital Library, for my thoughts on performance testing.
Automated regression testing is also a major potential source of value. It’s very hard to do this well, so you need to understand how to make this effort success. About half of automation efforts fail, often due to poor planning.
3. Now many Chinese tests were halfway decent, there is no specific test theories for the current lot of people think that the ability of the test failed. Test is not taken seriously in the project
Again, I think this is a matter of not having clearly defined objectives. I’d encourage you to work with stakeholders to understand what testing can contribute.
Please help me to work out this question.?
I hope this is helpful, Helen. Please feel free to respond to this post to continue the discussion.
Thanks
Helen .w. Huang
Tags: value of testing
Posted in best practices, software testing | No Comments »
December 21st, 2011 by Rex Black
E-learning Advanced Test Analyst student Donna Hungarter asks:
[An exam question] shows a state transition diagram, sets some criteria, and then asks how many tests are needed for this level of coverage. If we were looking for the fewest number of tests needed to cover all states and transitions, I believe the answer is 6 (not one of the answer options). If all 4 different credit cards need to be tested for each state and transition, I can get to 24, but I do not understand how the correct answer is 26. Can someone please explain? Thank you, Donna
Here’s the question:
You are testing a computerized gas pump that allows users to pay using credit cards. The pump is initially in the waiting-for-customer state. A transaction is initiated when a customer first inserts a credit card. Four types of cards are accepted: Visa, MasterCard, Discover, and American Express. The pump will reject any other type of card. If given an accepted credit card, the pump validates the card. If the card is valid, the pump is turned on and the customer is told to begin pumping gas. The pump remains on until the transaction ends. The transaction ends when one of the following events occurs:
• Pump handle is returned to pump
• Amount reaches transaction limit for card
• No gas is pumped within two minutes of validation of card
• The station attendant throws the emergency shut-off switch
Once the transaction ends, the pump processes the transaction, charging the credit card and producing a receipt. It then returns to the waiting-for-customer state.
Assume a test always starts and may only end in the waiting-for-customer state, a test must end after a transaction-ending event, and a test input consists of (initial state, event[condition], next state, event[condition], …, event[condition], initial state). Design tests covering every unique sequence of up to four states/three events. How many tests are needed for this level of coverage?
The state transition diagram is shown here:
 Gas Pump State Transition Diagram The correct answer is 26. The reason is because the question is asking for two-switch coverage. You have to create the two-switch table. You can short-cut the work by noticing that many of the two-switch elements can’t occur in a test, because they involve returning to the initial state as an intermediary step. You can see the answer in the image below:
 Switch Table and Tests You might need to expand the image to read it. This picture was taken in Kuala Lumpur, Malaysia, after a long 15 minute session to solve the problem. It’s a true brain-twister, and almost certainly harder than anything you’d see on a real exam.
Posted in ISTQB certification, software tester certification, software testing | No Comments »
December 14th, 2011 by Rex Black
Long time reader Gianni Pucciani commented on today’s webinar on Agile testing opportunites:
Hello Rex,
Today I followed your webinar at SmartBear, and I was quite surprised when you mentioned that testers, especially for automation purposes, should do their tasks outside the sprint. Following the agile methods like Scrum, the team produces a potentially shippable product at the end of each sprint, hence tested. I think with agile processes, one should strive to complete automation scripts (I am talking about functional tests, at the system level) inside the team, within the sprint schedule, to cover the stories implemented in the given sprint. Maybe I misunderstood your point, could you please clarify your thoughts on this subject? Thanks a lot!
Best regards,
Gianni
Perhaps I mis-spoke, or maybe didn’t make the point clearly enough. Yes, I agree that testers with responsibilities for testing the user stories in the Agile sprints need to be embedded with the Agile teams. However, that doesn’t mean that all testers should be embedded. For example, when Agile teams are working on systems that will be integrated with other systems (as many of our clients are), system integration testing needs to be separated from the Agile teams, as this role spans multiple systems and requires a perspective different from that needed to system test the content produced by any of the Agile teams. In addition, long-term projects such as building maintainable test automation frameworks should also span individual sprints. These roles need to be part of the test organization, but not embedded within a sprint.
The best approach that I’ve seen is that a central test team continues to exist. Testers are delegated–with dotted-line reporting structures–to the Agile teams where appropriate, at least one tester per team. (Having a tester work on multiple teams creates problems with focus, as I discussed today.) Testers that have roles that span sprints or systems should remain in the central team, providing a supporting role.
Tags: agile testing, software testing best practices
Posted in software testing | 4 Comments »
November 16th, 2011 by Rex Black
Reader and RBCS licensee student Sandor sent me the following question:
Hello Rex,
In the training material of the Advanced Technical Test Analyst Course (V1.1) on the slide 268 I found a misleading example. There is a code fragment
…
if (n<0) {
…
n = -1;
} else {
…
for (i = 1; i <= n; i++)
{
…
}
…
}
...
There is a comment under this code:
“Statement coverage: We tested(n < 0) and (n > 0). To get decision coverage,we still need (n == 0).”
I think the n = 0 test case is not needed because with n > 0 the false branch of the if is fully covered decision point of view (the for cycle is evaluated to true and false). What do you think? Am I correct?
Thanks,
Sandor
So, I think that depends on how you interpret it. If you interpret decision coverage as narrowly meaning that each decision evaluates at least once true and at least once false, then you would be right.
However, let’s think about what we’re trying to test with decision coverage. We are not only testing for bugs in the decision code itself. We are also testing what happens when alternative paths are taken in the code. Certainly, doing the loop body code one or more times, then exiting to the next block of code, is a different path than not doing the loop body code at all and proceeding directly to the next block of code.
Therefore, my interpretation is that decision coverage (or branch coverage, if you prefer), applied to loops, must include not only testing that the decision can be made correctly both true and false, but also that the loop body is executed and is not executed. Not everyone agrees with that, so you’re likely to encounter other opinions in books and articles.
Tags: decision coverage, structural testing, white box testing
Posted in ISTQB certification, software tester certification, software testing | No Comments »
November 11th, 2011 by Rex Black
I recently flew to China on a 747. The entertainment system was a complete failure, and no one in the upper deck cabin (where I sat) had any access to it. They gave us all free frequent flier miles to make up for it. When they gave me the voucher, I said, “I hope the avionics software is better tested than the entertainment system.”
Yes, that was a somewhat rhetorical comment, as I know about the DO-178B standard and how that would affect testing of avionics software versus entertainment system software. However, I have some concerns about that standard and what compliance to it really means. For example, that standard, being white-box based, only requires that you test code that’s there to some degree. What about code that shouldn’t be there, or code that should be there but isn’t (due to bad requirements or design)?
Shortly after I arrived, I received this link via e-mail: http://bit.ly/sNYQBa
Now, this is alarming. Suppose, in all of the resetting of the entertainment systems that was done to try to restore working order, some bad signals had gotten sent to the avionics system through leakage over network? How could that happen, you might say, as this link talks about deliberate hacking into the network? Well, if it can be done deliberately, and you have dozens of in-seat computer systems booting over the network from the storage system, isn’t it possible that all of that network traffic could leak due to a bug in the way the entertainment system is implemented? Even if the traffic doesn’t interfere with the avionics directly, if there’s a lot of it, you have the possibility of denial of service effects.
Of course, this can’t happen with newer planes, you might say. Oh, really? I believe I read articles when the A380 and the B787 were being built that both used a single network infrastructure. I can only assume they use this same “virtual network” approach to try to keep traffic separated.
Tags: aircraft software, design bugs
Posted in best practices, security testing, software testing | No Comments »
October 14th, 2011 by Rex Black
Long time reader ML Gregory wrote me with the following concern:
Subject: To QA or not to QA, that is the question
I was looking for an article about QA in an agile world. I’m uneasy about a change our company is doing – eliminating QA. Testers who choose to stay must work in development as a developer writing tests. No test cases will be written, no bugs will be written. I’ve seen it before and was hoping to find some authoritative material on this type of change.
I’ve written about various ways Agile methodologies can challenge testing (most recently in the blog, but also in an article and a webinar). What you’re describing here, though, is a variant of the problem which is actually independent on Agile per se.
What you are describing is a problem I refer to as the Christmas Tree Ornament problem. I would be willing to bet that these three changes–no independent testers, no bug reports, no written tests cases–have been changes that people have wanted for some time. When the change to an Agile methodology came along, people hung these wishes onto Agile like a Christmas tree ornament.
It’s not unusual for these particular problems to occur with Agile, especially the “no bug reports” problem. Some Agile advocates do push for no tracking of bugs found during the iteration, which of course has the effect of making it impossible to gather metrics about the software process’s quality capability. However, I’ve seen other organizations that are definitely not using Agile have this same problem.
There’s not necessarily a lot you can do to resist these changes. They usually come with a lot of momentum. Standing in the way of an approaching train is not a good career move. Perhaps passing on the links I provided above, along with this blog post, can help.
I’d be interested in other thoughts from you and from other readers.
Tags: agile testing, software testing
Posted in best practices, software testing | 2 Comments »
October 13th, 2011 by Rex Black
Last week, Steve Jobs died, and I published an encomium to him in this blog. Today, I was saddened to read of the death of Dennis Ritchie, one of the greatest software innovators that most people have never heard of, but who had arguably as large an impact on our world as Steve Jobs did. I learned much of what I know about how software works by reading his book on C programming and by applying what I learned from that book to programming Unix systems in the 1980s and 1990s.
Shortly after I read about his death, I received an excellent post from Lauren Weinstein, relating his own thoughts and experiences about Dennis Ritchie’s inestimable contributions to the software world. I asked Lauren if I could quote his post, and he graciously agreed. Here follow Lauren’s fine remarks:
A Few Brief Thoughts on the Death of Computer Science Titan Dennis Ritchie http://lauren.vortex.com/archive/000903.html
Dennis Ritchie has died after a long illness ( http://j.mp/pQfU8J [CNET]).
He was only the relatively young age of 70. Unlike Steve Jobs, Dennis was not a household name. He should have been, but he was a quiet and private person, and would have hated the publicity.
I’ve known Dennis since the early days of the UNIX operating system at Bell Labs in the 70s, where he created the ubiquitous “C” programming language, and co-created UNIX with Ken Thompson.
It is impossible to overstate the importance of this work to today’s world. UNIX was almost entirely written in “C” — and UNIX’s direct descendants like Linux power Google, the Web at large (the vast majority of Web servers run Linux), and likely multiple devices in your home and office right now. The current Mac OS. Android. TiVo. The impact of C and Linux are everywhere. Your ability to read these words rests directly to a major degree on Dennis’ work.
But beyond the nitty-gritty of software design, the creation of UNIX was also the inception of what would ultimately become the open source community, and the vast collaborations of the ARPANET/Internet that have led to the global phenomenon we see today.
Such goals were explicit in the design of UNIX. In this 1980s Bell Labs video from my collection, featuring Ken and Dennis explaining the origins of UNIX, Dennis explains how they wanted the system to specifically foster community and fellowship.
I said this would be brief. I’ll close with one personal story. Back in the early UNIX days, on one of my visits to Bell Labs’ main facilities in Murray Hill, New Jersey, I was sitting at a terminal in the 1127 Labs where UNIX was developed, logged in over the nascent Net
back to UCLA. As usual, I was trying to get a bit of coding done even on this trip.
My rapid typing suddenly stopped as I puzzled over a particularly complex C language declaration in the program, that definitely was incorrect as it stood.
Sitting at the terminal next to me was Dennis. So I asked his advice on the declaration — after all, who better to know than the creator of the language?
He thought about it for no more than a few seconds, then immediately (and graciously) provided an elegant solution that, frankly, would not have occurred to me. I got on with my programming.
In later years, I realized that this exchange was probably the closest I’d ever come in my life to asking a question directly to, and receiving a detailed answer from, a true god.
Rest in peace, Dennis.
–Lauren–
Founder:
Member: ACM Committee on Computers and Public Policy
Tel: +1 (818) 225-2800 / Skype: vortex.com
Lauren, thanks for permission to re-post this. Dennis, thank you very much for what you taught me through your book and your work, and for what you did to advance software engineering. May we always be so fortunate as to work in an industry with such generous hearts and brilliant minds as Dennis had.
Tags: Dennis Ritchie
Posted in software quality | No Comments »
October 12th, 2011 by Rex Black
Long time reader Patricia Osorio wrote to ask about the sample exam questions at the end of Chapter 7 in Advanced Software Testing: Volume 2. She’d like me to explain the answers.
Here are the questions:
Assume you are a test manager working on a project to create a programmable thermostat for home use to control central heating, ventilation, and air conditioning (HVAC) systems. In addition to the normal HVAC control functions, the thermostat also has the ability to download data to a browser-based application that runs on PCs for further analysis. During quality risk analysis, you identify compatibility problems between the browser-based application and the different PC configurations that can host that application as a quality risk item with a high level of likelihood. Your test team is currently executing compatibility tests.
Consider the following excerpt from the failure description of a compatibility bug report:
1. Connect the thermostat to a Windows Vista PC.
2. Start the thermostat analysis application on the PC. Application starts normally and recognizes connected thermostat.
3. Attempt to download the data from the thermostat.
4. Data does not download.
5. Attempt to download the data three times. Data will not download.
Based on this information alone, which of the following is a problem that exists with this bug report?
A. Lack of structured testing
B. Inadequate classification information
C. Insufficient isolation
D. Poorly documented steps to reproduce
The answer is C. We would expect, given the bug that’s being reported, that the tester would try other operating sytems.
The questions continue:
Continue with the previous scenario. Your test team is still executing compatibility tests. Consider the following excerpt from the failure description of a compatibility bug report:
1. Install the thermostat analysis application on a Windows XP PC.
2. Attempt to start the thermostat analysis application.
3. Thermostat analysis application does not start.
4. Re-install the thermostat analysis application three times. Thermostat analysis application does not start after any re-installation.
5. This test passed on the previous test release.
Based on this information alone, which of the following is the most reasonable hypothesis about this bug?
A. The bug might be a regression.
B. The bug might be intermittent.
C. The application didn’t install on Windows XP PCs before.
D. The bug might be a duplicate.
The answer is A. The bug is very likely a regression, because the test worked on the previous version of the software installed in the test environment.
Tags: software bugs, software defects, software testing
Posted in ISTQB certification, software testing | No Comments »
October 7th, 2011 by Rex Black
I received a request from Martin for help on how to apply orthogonal arrays. He wrote:
Hello,
I am working on putting together testing at work using an orthogonal array. This is the first time I’ve used this method. I read about it in detail in Rex’s books. But I’m having difficulty remembering all the steps. I need a little help to jog my memory. Could you please send me the high-level, bulleted, steps for creating the orthogonal array with my test case data?Thank you.
Sincerely,
Martin
Sure, Martin, you can take a quick listen to the key parts of my webinar on pairwise testing techniques. This webinar addresses how to use orthogonal arrays and other pairwise tools. After you listen to that, I’d suggest you go to www.pairwise.org to find a free downloadable tool. Last I heard, the US Department of Commerce had funded a project to create a pairwise tool (your tax dollars at work!) that will actually do classification trees as well, and I think you might find it there.
In general, Martin, if you are looking for a refresher on test design techniques, a good first stop is the RBCS Digital Library.
Tags: software test design, software testing
Posted in best practices, software test skills, software testing | No Comments »
October 7th, 2011 by Rex Black
Long time reader, Patricia Osorio Aristizabal, asked me to clarify the following question:
During a walkthrough meeting for a high-level design, a tester who is participating as a reviewer asks the system architect who created the document why only physical network connections are supported in the design, rather than also supporting wireless connections. The designer replies that they did not have time to address the security concerns raised by wireless access to the system. The tester accepts that answer and the review proceeds.
Which of the following is an objective of a review that was achieved in that exchange?
A. Increased reviewer understanding
B.Early detection of a defect
C. Correction of a defect
D. Analysis of system design
The correct answer is A. The tester now has a better understanding of why the system is built the way it is. The design of the system is not defective, as there is a deliberate reason for not having WiFi support, so C is not the right answer.
Tags: software reviews
Posted in software quality, software testing | No Comments »
|
Rex Black is President of RBCS (
