| Summary | Test Me -- is a suite of programs which
claim to conform to various contracts. It is up to the student to
determine whether or not these claims are justified. |
| Topics |
Software Testing, programming by contract |
| Audience |
CS0 to CS2, depending upon the topic of
the program; most in this suite are CS0-CS1ish |
| Difficulty |
Fairly easy, but difficulty can trivially
be ramped up. |
| Strengths |
Students do NO CODING! Forces
students to read documentation and build test cases in a creative manner.
The assignment can be used to provide "introductions" or "pre-labs" for
many traditional lab assignments. Can serve as an introduction to
testing through JUnit, etc. |
| Weaknesses |
It is easy for students to do a poor job.
It can also be frustrating to find obscure bugs. |
| Dependencies |
Requires an understanding of English and
the problem domain. For students to do well on this type of
assignment, they need to be exposed to the ideas of formal testing. |
| Variants |
Several variants are provided. Once
the concept is clear, it is reasonably simple for instructors to build
their own. Note that the materials at this site use graphical
interfaces, but text-based versions can be generated more easily and have
many of the same benefits. |
The student is provided with a specification for a program and a link to a web page containing an applet that purports to meet the specifications. The student is to run the program and find the conditions, if any, under which the program fails. (The student is not allowed to inspect any source code.) Depending upon the desires of the instructor, the student can be required to identify/design the test cases in advance or the design can be done "on the fly". The student turns in a report describing the testing that was done as well as the results of that testing.
This web site contains an index page referencing six programs. Each of the programs has a specification (written in "programming contest style") and one to four additional web pages containing applets that purport to meet those specifications. One of the applets does meet the specifications; the other applets have some bug or set of bugs. A second link (labeled "description") with each version will open a panel explaining the bugs, if any, in that version. (Note: As of 9/9/04, the "bogus" applets for the APFT program are not submitted; they are done, but are currently "misplaced". It is possible/probable that there will be a seventh program before the first of the year.)
Normally, an instructor would take only one of the versions and the description and link them off of a web page explaining the deliverables. For our first assignment, we tell the students that there are indeed bugs so that they don't run it once and certify it; after that first experience, they are warned that it is possible that the program in question might be correct!
The assignment is simple enough that it can (and should be) repeated. Programmers learn to test code thoroughly by testing code often, under different circumstances. The first assignment can be given with no preparation, and then discussions can be used to help students learn the concept more thoroughly.
The use of GUIs in the suite means that it is difficult to do the type of automated testing that JUnit and its ilk support so well. Students who will eventually use such a tool can still learn the principles of testing through this suite. When they are ready to graduate to a more sophisticated tool, the concepts learned here will transfer directly.
Extra info about this assignment: