tcltest Part 9: Provides Exit Code

6 Replies

The Problem

One of the complains against tcltest is the fact that it does not return an exit code to communicate test pass/fail status.

Consider a simple scenario. In a directory, we have two files: all.tcl, which drives all the tests, and test1.test, which contains the actual tests. The contents of all.tcl is:

package require tcltest
tcltest::runAllTests

And the contents of test1.test is:

package require tcltest
namespace import ::tcltest::*

test fail_me {} -body { set foo 1 } -result 0
test pass_me {} -body { set foo 1 } -result 1

cleanupTests

In the scenario above, one of the tests will fail, but all.tcl will always return with exit code 0. It would be more useful if it returns 0 when all tests passed and 1 when not all tests passed.

The Solution

We looked into the tcltest source code and found an array, tcltest::numTests, which looks what we need. So, we modified all.tcl to see if we can use it:

package require tcltest
tcltest::runAllTests
parray tcltest::numTests

The result is disappointing:

(irrelevant output omitted)
tcltest::numTests(Failed)  = 0
tcltest::numTests(Passed)  = 0
tcltest::numTests(Skipped) = 0
tcltest::numTests(Total)   = 0

With one failed tests, we thought tcltest::numTests(Failed) should be 1, but it was not. we went back to the source code and dug a little deeper and found tcltest::cleanupTests resets these numbers after it finishes reporting.

Just when we was about to give up, we thought of asking Mr. Google for help. Sure enough, tcltest provides a hook into its tcltest::cleanupTests, which gives us access to the statistics variables before it resets them. Here is the final all.tcl:

package require tcltest

# Hook to determine if any of the tests failed. Then we can exit with
# proper exit code: 0=all passed, 1=one or more failed
proc tcltest::cleanupTestsHook {} {
    variable numTests
    set ::exitCode [expr {$numTests(Failed) > 0}]
}

tcltest::runAllTests
exit $exitCode

Discussion

The trick is to write a hook, tcltest::cleanupTestsHook. Before resetting the test statistics, tcltest::cleanupTests calls the hook function, which gives us the opportunity to access these statistics, and use them to determine the exit code.

Now that all.tcl returns the proper exit code, we can use it to determine pass/fail. Here is an example using the bash shell in Unix-like systems:

#!/bin/bash
if tclsh all.tcl; then
    echo Tests passed
else
    echo One or more tests failed
fi

For Windows environment:

tclsh all.tcl
if errorlevel 1 echo One or more tests failed

Conclusion

We wish tcltest automatically returns the correct exit code, but it does not. For now, we will stick with this solution and hope that whoever working on tcltest won’t make our code obsolete.

6 thoughts on “tcltest Part 9: Provides Exit Code”

  2. Pawel

    I have one more question, what if there will be an “error” inside the script? Because this solution is “catching” only “failing” tests, not errored scripts

  3. Hai Post author

    I did some poking around and it looks like tcltest treats error within the script as failed case. I am still thinking of a way to differentiate between the two. On the other hand, you might want to view part 6 if you want to test for error condition.

  5. Gerald Williams

    Thank you for this great series on tcltest. I just came across it and it’s quite useful.
    I encountered a slight issue with this solution as written. If you have a recursive test suite, the lower-level suite can reset its counters, suppressing previously-detected errors. I was surprised that this wasn’t a potential issue when every test case calls cleanupTests: it apparently defers clearing counters until the last test in a directory.

    One possible solution (which can be used in recursive tests also):

    …
    proc tcltest::cleanupTestsHook {} {
    variable numTests
    if {$numTests(Failed)} {set ::testFailed 1}
    }
    tcltest::runAllTests
    exit [info exists ::testFailed]

  6. jlinkels58

    This is really an excellent tutorial. Most Tcl wiki pages and howto’s are extremely terse – to a point where it is uncomprehensible. This was very useful. Thank you so much.

Leave a comment