Sun, 02 Nov 2003

Rules Engine Problems

Discovering and programming the rules is a process requiring a very significant investment of time and effort, including both technical and business experts.

Rules engines can't handle exceptions that are not programmed in.

The current assertion that the improvements to rules engines are in "better interfaces" means only that we can get to the functions from across heterogeneous systems.

In most cases, experts don't in fact use the rules that might be discovered by interview and reflection.

Rules are always context free and cannot take into account the relevance of facts.

Ongoing maintenance of rules.

Deskilling

ceteris paribus

Expert human collaboration and judgment is required to correct and update rules engine errors.

References

• Duda and Shortliffe, from Dreyfus p109.
• CiteSeer search for expert systems research
• From Socrates to Expert Systems (PDF)
• Philosophy 24-1 - What is a skill? The Phenomenology of Learning
• No Silver Bullet: . . . continued, Expert Systems: "more difficult and important is the twofold task of knowledge acquisition: finding articulate, self-analytical experts who know why they do things, and developing efficient techniques for extracting what they know and distilling it into rule bases. The essential prerequisite for building an expert system is to have an expert."
• Rules Engine Links

permanent link

Build Machine Justification

The automated daily build and smoketest is one of the easiest and most cost-effective ways to increase software development productivity and quality. It is practiced by the most proficient software development shops.

• Slashdot article
• Microsoft does it
• Windows 2003 Development

High-level overview of the build process, from http://codefeed.com/tutorial/ant_intro.html

1. Get the source. You may need to download or fetch the source from a source code repository. For this, you might need to know the tag or version of the source code you want to build.
2. Prepare a build area. You will probably want to create a set of directories, perhaps according to some standardized directory layout.
3. Configure the build. In this step, you will determine what optional components can be built based on the current environment. You might want to set build numbers and version numbers to be included in the build.
4. Validate the source code. You may have a standard style guide and you wish to ensure all code conforms to this before you build a release.
5. Compile the source code
6. Build the compiled code into libraries potentially including non-code resources such as properties, images and sound files.
7. Run the system's tests to validate the build.
8. Build the documentation for the software. This may range from something as simple as collecting text files up to processing content through some form of publishing system to produce the documentation in its final form
9. Package up all of the components of the software - code, resources, images, documentation, etc. - into a deployable package. You might need to produce several packages in different formats for different target users
10. Deploy the software to some standard location for use or distribution

permanent link

Offshore Outsourcing

permanent link

Cem Kaner at PNSQC 2003

On October 14 I had the opportunity to hear Cem Kaner talk at the Pacifc Northwest Software Quality Conference in Portland, Oregon. The title of his talk was How Many Lightbulbs Does It Take To Change A Tester? These are my notes and impressions from that talk.

Kaner's opening remarks focused on his views of where software testing has been and is going. After 20 years of practices, he said, little has changed, and the common practices have their problems. Compared to the programming community, testers have changed very little in their self-image.

• Old attitude: Defensive, fearful, hierarchical, grasping for control, procedural. The goal was to "reduce testing to a set of steps even dummies can follow".
• New attitude: The tester as technical investigator: Find stuff that other people don't know how to find. Often these are bugs, but not always. Generally the work is focused around the quality of the product.

One of the recommended practices (from Kaner's own work) is to only do black-box testing. Kaner question that and asked why it was considered good. What value is there in working without knowledge of the underlying code? The answer is that it allows the test, instead of learning about the product from the code, to learn about the product from the stakeholders and see how well the code does that. Tester gathers information the programmer didn't or couldn't consider. The more recent discovery is that sometimes the best source of that information is the code.

He then discussed the wisdom that black-box testing and external testing is necessary because "programmers can't catch their own bugs". Kaner drew a distinction between a private bug, one the programmer makes, sees, and fixes before anyone else sees it. Rate: 15-150 per 100 statements, and public bugs: all bugs left in code when programmer passes it to another person. Rate: 1-3 per 100 statements. In other words: 90% of the bugs are private, so programmers are actually pretty good at finding bugs. The insight here is that programmers find one kind of bugs, but it needs someone else to test other things to find the remaining bugs.

In the next part of his talk, Kaner discussed the view that testers and programmers should be kept independent. He brought up several areas where close collaboration actually is more beneficial than separation.

Para-functional Testing What used to be called Non-Functional testing, but Kaner discovered that too often mean "disfunctional testing". For good testing of things like security, usability, accessibility, supportability, localizability, interoperability, installability, performance, scalability, and the various other "ilities", testers must work with programmers because the stakeholders don't have rules.

Preventative testing. The best kind of preventative testing -- testing that prevents bugs before they happen, is Test-driven Development, said Kaner. It implies tight collaboration between the testing role and the programming role (sometimes so tight, they are the same person) Fast, efficient feedback is possible, and fast is good for learning. Communications cost is low.

Kaner's talk then covered the value of fast and efficient feedback in software development. Traditional external testing is say, 10 calendar days or more, plus all the processing, cycle time, said Kaner. In hours, cycle time for efficient companies, it's 4 hours for a simple defect. Up to 20 hours for a less-efficient companies. Filing lots of bugs is a form of denial of service attack to a programming team, because the processing time of dealing with the bug reports leaves no time to actually fix bugs.

The purpose of testing is to find bugs, right? Well, said Kaner, it depends. In discussing some of the points raised by the paper, A Manager's Guide to Evaluating Test Suites, Kaner examined how test results can guide a project manager in determining the status of the system. Do you want to find and eliminate the most bugs before a fixed date, Kaner asked, or do you want to develop an overview of the state of the system to allow the project manager to decide if it's ready to ship? One area where the testing is not done to find bugs, but to protect the company from risk of lawsuits -- do we test enough and well enough to accept the risk that our system may fail (and kill someone) but know we did the best we could?

Testers in Agile Teams

One of the facts of software development, said Kaner, is that pulling features out late will not save money. In the tradeoff between features, reliability, cost and time in a standard waterfall process, features and cost are used up by the end, so the only thing left is reliability vs. time. Thus testing is often compressed or cut. In evolutionary development where we have a limited set of features that work, then towards the end, tradeoff is do we have enough features to sell it or do we need to spend more time adding features?

Test then code	Test-first development
Big Tests Up Front	One test, one feature, iterate

The lesson for testing is that test code and testers must be able to cope with simple changes. This is where IEEE Std 829 Testing Practices is, according to Kaner, an industry worst practice.

A Final Note

Outsourcing, said Kaner, is a by-product of making the process rigid and "fully documented". It makes software into a manufacturing process, something that is easily outsourced. This is a fatally flawed approach for most commercial software.

permanent link