Customizing TTS Behaviour

Table of Contents

• Tests Driver
• Crashed and Stuck Runs
• Data display
• Equality and Ordering
• Precision Measurement
  • ULP Distance
  • IEEE Comparison
  • Relative Comparison
  • Absolute Comparison
• Data Generator
  • Producing values of a user-defined type

Tests Driver

By default, TTS provides an entry point function for the listed tests. However, it may be required to handle such an entry point. In this case, one can define the TTS_CUSTOM_DRIVER_FUNCTION preprocessor symbol to a name of their own entry-point function as shown below.

After defining the TTS_CUSTOM_DRIVER_FUNCTION symbol, tests can be added as usual. Then, a regular main function is to be defined. This function will then perform any special operations required, then call the specified entry point function. Finally, the main function will call tts::report which will aggregate test results and validate the whole tests with respect to expected numbers of failures and invalid tests.

#define TTS_MAIN
#define TTS_CUSTOM_DRIVER_FUNCTION custom_entry_point
#include <tts/tts.hpp>
#include <iostream>
 
TTS_CASE("Tautological test")
{
  TTS_EXPECT_NOT(false == true);
};
 
int main(int argc, char const** argv)
{
  std::cout << "Welcome to a special test driver !\n";
  custom_entry_point(argc, argv);
  return tts::report(0, 0);
}

Crashed and Stuck Runs

A crashed test case cannot resume, so TTS names it, reports it through the sinks and exits with 1. What the run already printed stays, the cases left are counted as not run, and the summary is written. A case tagged TTS_XFAIL or TTS_MAYFAIL turns that end into the expected one, and the run exits 0.

TTS_CASE("a case that runs to the end")
{
  TTS_EQUAL(1 + 1, 2);
};
 
#if !defined(__EMSCRIPTEN__) // no signal delivery there, so nothing would name the case below
// A crash ends the run, so the tag turns that end into the expected one and the exit code into 0.
TTS_CASE(TTS_XFAIL("a case that brings the run down"))
{
  // A raised signal keeps memcheck and the sanitizers silent: no invalid access happens.
  std::raise(SIGSEGV);
};
#endif
 
TTS_CASE("a case the run never reaches")
{
  TTS_EXPECT(true);
};

tts::set_abort_handler runs just before the exit, with the signal that brought the run down or 0 when nothing did. A test binary that has to handle an abrupt stop in its own way installs one.

TTS_CASE("Choosing what a crashed or stuck run does before exiting")
{
  auto previous = tts::set_abort_handler(
  [](int reason) { std::cout << "tearing down after signal " << reason << "\n"; });
 
  TTS_EXPECT_NOT(previous);
};

The guard covers SIGSEGV, SIGBUS, SIGFPE, SIGILL and SIGABRT, that last one catching an assert in the code under test; under Windows, access violations, stack overflows, integer divisions by zero and illegal instructions. Under emscripten it is compiled out.

--no-crash-guard, or TTS_NO_CRASH_GUARD in the environment, leaves those signals to the system: that is what a debugger, or a sanitizer stack trace, needs.

A case that hangs cannot resume either, and nothing outside the process can name it. --timeout=<ms>, or TTS_TIMEOUT in the environment, gives every case a deadline; past it the case is named the same way a crash is, and the run ends the same way. TTS_TIMEOUT sets one case's deadline, and 0 exempts that case from the global one. The handler receives 0 rather than a signal number, nothing having been raised.

TTS_CASE(TTS_TIMEOUT(5000, "a collective that answers in time"))
{
  TTS_EXPECT(true);
};
 
TTS_CASE(TTS_XFAIL(TTS_TIMEOUT(0, "known to sit there, and exempt from --timeout")))
{
  TTS_EXPECT(1 == 2);
};
 
#if !defined(__EMSCRIPTEN__) // no process timer there, so nothing would end the case below
// The deadline ends the run, so the tag turns that end into the expected one and the exit code
// into 0.
TTS_CASE(TTS_XFAIL(TTS_TIMEOUT(250, "a case that outlives its own deadline")))
{
  std::this_thread::sleep_for(std::chrono::milliseconds(500));
};
#endif

Under POSIX the deadline is an interval timer, so a case blocked in a system call is still interrupted; code under test that uses SIGALRM itself conflicts with it. Under Windows it is a timer queue. Under emscripten it is compiled out.

Data display

By default, whenever TTS needs to display a value in a report, it uses std::to_string or, in the case of sequence-like types, a sequence of calls to std::to_string. In case no overload for std::to_string exists for a given type, a string will be built from the type name and its byte sequence.

#define TTS_MAIN                   // No need for main()
#include <tts/tts.hpp>
 
struct payload
{
  double         d;
  unsigned int   i, j;
  constexpr bool operator==(payload const&) const = default;
};
 
TTS_CASE("Display an unknown type")
{
  payload p {1.5, 0xAABBCCDD, 0x11223344};
  payload q {1.5, 0xFFEEDDCC, 0x99887766};
 
  TTS_EQUAL(p, q);
};

In the case a given type needs to be displayed in a specific manner, specialize tts::display for it and TTS will use it when necessary. The to_text overload this replaces is gone: one left behind is reported where it would have been used.

#define TTS_MAIN                   // No need for main()
#include <tts/tts.hpp>
 
struct payload
{
  double         d;
  unsigned int   i, j;
  constexpr bool operator==(payload const&) const = default;
};
 
namespace tts
{
  template<> struct display<payload>
  {
    static text render(payload const& p)
    {
      return "payload(" + tts::as_text(p.d) + ")[" + tts::as_text(p.i) + "][" + tts::as_text(p.j) +
             "]";
    }
  };
}
 
TTS_CASE("Display a type with a tts::display specialization")
{
  payload p {1.5, 0xAABBCCDD, 0x11223344};
  payload q {1.5, 0xFFEEDDCC, 0x99887766};
 
  TTS_EQUAL(p, q);
};

If needed, one can delegate a part of this string construction to the TTS internal string conversion function tts::as_text that will use all runtime options for display. tts::text can also be constructed from a formatting specification and other similar setup.

#define TTS_MAIN // No need for main()
#include <tts/tts.hpp>
 
namespace sample
{
  struct payload
  {
    double         d;
    unsigned int   i, j;
    constexpr bool operator==(payload const&) const = default;
  };
 
}
 
namespace tts
{
  template<> struct display<sample::payload>
  {
    static text render(sample::payload const& p)
    {
      return text("payload(%f)[%d][%d]", p.d, p.i, p.j);
    }
  };
}
 
TTS_CASE("Display another type with a tts::display specialization")
{
  sample::payload p {1.5, 0xAABBCCDD, 0x11223344};
  sample::payload q {1.5, 0xFFEEDDCC, 0x99887766};
 
  TTS_EQUAL(p, q);
};

Beware that, in this situation, command-line arguments controlling value display like -x or -s will not be applied to the formatted string.

Equality and Ordering

All equality-based checks in TTS use the compared value operator==. If needed, specialize tts::comparison to let TTS use a special comparison scheme. The trait takes both operand types, in the order they are written, and the second one defaults to the first, so a homogeneous comparison names a single type.

#define TTS_MAIN                   // No need for main()
#include <tts/tts.hpp>
 
namespace sample
{
  template<typename T> struct box
  {
    T value;
  };
 
}
 
namespace tts
{
  template<typename T> struct comparison<sample::box<T>, sample::box<T>>
  {
    static bool equal(sample::box<T> const& l, sample::box<T> const& r)
    {
      return l.value == r.value;
    }
  };
}
 
TTS_CASE("Compare values with custom equality")
{
  sample::box<int> a {42};
  sample::box<int> b {13};
 
  TTS_EQUAL(a, a);
  TTS_NOT_EQUAL(a, b);
};

Similarly, TTS uses operator< to build all its ordering-based checks. The less member of the same trait covers those, and a member the specialization leaves out keeps its default. The compare_equal and compare_less overloads these replace are gone: one left behind is reported where it would have been used.

#define TTS_MAIN // No need for main()
#include <tts/tts.hpp>
#include <cmath>
 
namespace sample
{
  template<typename T> struct absolute
  {
    T value;
  };
 
}
 
namespace tts
{
  template<typename T> struct comparison<sample::absolute<T>, sample::absolute<T>>
  {
    static bool equal(sample::absolute<T> const& l, sample::absolute<T> const& r)
    {
      return std::abs(l.value) == std::abs(r.value);
    }
 
    static bool less(sample::absolute<T> const& l, sample::absolute<T> const& r)
    {
      return std::abs(l.value) < std::abs(r.value);
    }
  };
}
 
TTS_CASE("Compare values with custom comparisons")
{
  sample::absolute<int> a {42};
  sample::absolute<int> b {-42};
  sample::absolute<int> c {-13};
 
  TTS_LESS(c, b);
  TTS_LESS_EQUAL(a, a);
  TTS_GREATER(b, c);
  TTS_GREATER_EQUAL(b, a);
};

TTS_BIT_EQUAL and TTS_BIT_NOT_EQUAL use the bit_equal member of the same trait. It defaults to a memcmp over the whole object, so a type whose storage holds padding, such as a SIMD register wider than the lanes it carries, compares bytes that no operation ever writes and has to specialize it.

Precision Measurement

ULP Distance

When dealing with floating point values, TTS measures all ULP checks through the ulp member of tts::precision. Specialize the trait to let TTS use a special ULP comparison scheme; the three members left out keep their defaults. One can also reuse the pre-existing tts::ulp_check to implement their own.

#define TTS_MAIN // No need for main()
#include <tts/tts.hpp>
 
namespace sample
{
  struct ratio
  {
    int n, d;
  };
 
}
 
namespace tts
{
  template<> struct precision<sample::ratio> : _::builtin_precision<sample::ratio>
  {
    static double ulp(sample::ratio const& a, sample::ratio const& b)
    {
      auto ra = static_cast<float>(a.n) / static_cast<float>(a.d);
      auto rb = static_cast<float>(b.n) / static_cast<float>(b.d);
 
      return tts::ulp_check(ra, rb);
    }
  };
}
 
TTS_CASE("Compare values with custom ULP distance computation")
{
  sample::ratio a {8388608, 8388608};
  sample::ratio b {8388618, 8388608};
 
  TTS_ULP_EQUAL(a, b, 10);
};

IEEE Comparison

IEEE comparison consists in checking for exact equality while considering all NaN/Invalid values of floating point values. It is the ieee member of tts::precision. One can also reuse the pre-existing tts::ieee_check to implement their own.

#define TTS_MAIN // No need for main()
#include <tts/tts.hpp>
 
namespace sample
{
  struct ratio
  {
    int n, d;
  };
 
}
 
namespace tts
{
  template<> struct precision<sample::ratio> : _::builtin_precision<sample::ratio>
  {
    static bool ieee(sample::ratio const& a, sample::ratio const& b)
    {
      auto ra = static_cast<float>(a.n) / static_cast<float>(a.d);
      auto rb = static_cast<float>(b.n) / static_cast<float>(b.d);
 
      return tts::ieee_check(ra, rb);
    }
  };
}
 
TTS_CASE("Compare values with custom IEEE distance computation")
{
  sample::ratio a {189, 265};
  sample::ratio b {0, 0};
 
  TTS_IEEE_EQUAL(a, a);
  TTS_IEEE_EQUAL(b, b);
};

Relative Comparison

Relative precision checks within TTS are done through the relative member of tts::precision, which reports a ratio rather than a percentage. One can also reuse the pre-existing tts::relative_check to implement their own.

#define TTS_MAIN // No need for main()
#include <tts/tts.hpp>
 
namespace sample
{
  struct ratio
  {
    int n, d;
  };
 
}
 
namespace tts
{
  template<> struct precision<sample::ratio> : _::builtin_precision<sample::ratio>
  {
    static double relative(sample::ratio const& a, sample::ratio const& b)
    {
      auto ra = static_cast<float>(a.n) / static_cast<float>(a.d);
      auto rb = static_cast<float>(b.n) / static_cast<float>(b.d);
 
      return tts::relative_check(ra, rb);
    }
  };
}
 
TTS_CASE("Compare values with custom relative distance computation")
{
  sample::ratio a {1, 77};
  sample::ratio b {3, 85};
 
  TTS_RELATIVE_EQUAL(a, b, 0.0225);
};

Absolute Comparison

Absolute precision checks within TTS are done through the absolute member of tts::precision. One can also reuse the pre-existing tts::absolute_check to implement their own.

The four free functions ulp_distance, ieee_equal, relative_distance and absolute_distance are gone. One left behind is reported where it would have been used, rather than ignored in silence, so a suite carrying them says so at the first check that needs it.

#define TTS_MAIN // No need for main()
#include <tts/tts.hpp>
 
namespace sample
{
  struct ratio
  {
    int n, d;
  };
 
}
 
namespace tts
{
  template<> struct precision<sample::ratio> : _::builtin_precision<sample::ratio>
  {
    static double absolute(sample::ratio const& a, sample::ratio const& b)
    {
      auto ra = static_cast<float>(a.n) / static_cast<float>(a.d);
      auto rb = static_cast<float>(b.n) / static_cast<float>(b.d);
 
      return tts::absolute_check(ra, rb);
    }
  };
}
 
TTS_CASE("Compare values with custom absolute distance computation")
{
  sample::ratio a {115, 77};
  sample::ratio b {397, 85};
 
  TTS_ABSOLUTE_EQUAL(a, b, 4);
};

Data Generator

Test cases based on data sets and range checks require one or more data generators to perform. If the pre-existing data generators are not suitable, define your own by providing a constexpr callable object with the following signature:

template<typename T>
T operator()(tts::type<T> target, auto index, auto count);

where:

• target is an instance of tts::type<T> representing the type of value to be generated.
• index is an integral value representing the index of the generated value.
• count is an integral value representing the total number of values to be generated.

For example, the following code defines a generator that will generate values alternating between -1 and 1 every n iterations.

#define TTS_MAIN // No need for main()
#include <tts/tts.hpp>
#include <array>
 
struct flip_values
{
  constexpr explicit flip_values(int p = 1)
      : period_(p)
      , value_(-1)
  {
  }
 
  template<typename T> auto operator()(tts::type<T>, auto i, auto)
  {
    if((i % period_) == 0) value_ = -value_;
    return static_cast<T>(value_);
  }
 
  int period_, value_;
};
 
TTS_CASE_WITH("Test custom generator", (std::array<float, 10>, std::array<int, 4>), flip_values {3})
<typename T, std::size_t N>(std::array<T, N> const& args)
{
  for(std::size_t i = 0; i < args.size(); ++i)
  {
    TTS_EQUAL(args[ i ], ((i % 3 == 0) ? T {1} : T {-1}));
  }
};

Producing values of a user-defined type

A generator answers for the type it is handed, so a type TTS knows nothing about needs to say how it is built. Specialize tts::generation for it. A type without a specialization keeps the default way of building, which hands a scalar straight to the generator and fills a sequence element by element.

Generator bounds go through tts::conversion, which casts a plain value to the type under test. Specialize it when a bound is a recipe rather than a value, so that a bound written once in a case answers for each type the case runs on.

produce and convert_as are the dispatchers rather than customization points, and TTS reaches them qualified, so an overload of either name in another namespace is never found. They carry a generic default, which is why a leftover cannot be reported the way the other traits report theirs.

A specialization of tts::generation keys on the type being built and cannot see the generator in an overload set. When one has to be treated apart, tts::is_randoms names the range generator by its type.