We need to design automated unit tests, platform tests and performance tests both so we can monitor our development progress and so we can evaluate a given hardware platform setup using a set of performance scores.
Every object or unit of code will include a Unit Test function [E: in the fashion of real time application (cross)development I suggest to de-couple testing code and release code, and instead build separate objects that implement unit tests and invoke functions on target instances. This can be applied to runtime probes as well, potentially running on remote profiling machines]. This function can be called on an isolated and instantiated unit with no parameters. The return value will be a pointer to a TestResult structure that holds the following data:
If Unit Test fails completely, a NULL pointer will be returned.
A series of standard tests will be run on an instantiated set of Nodes. One Node will be started last as the Test Master Node [E: in line with the comment above, I would rather build a separate, dedicated, test / profiling node running on a remote machine / separate CPU or CPU core] with a list of participating already running Test Participant Nodes. The Test Master will conduct a number of standard tests by creating sets of Test Modules on the Nodes in the setup. For each test conducted a TestResult structure (see Unit Test above) will be returned.
The target performance goal is to consistently achieve 1 message/module every 10msec for 100 modules. A standard test (much like the Platform Test above) will be run on either one computer (single Node Test) or a network of computers (multi Node Test). The test returns a single TestResult structure (see Unit Test above) containing a full set of performance scores, called the Platform Performance Vector.
The Performance Test will also compute a single overall Platform Performance Score which can be used to directly compare either different platforms or different versions of the code. Of course, the Platform Performance Vector will provide better and more detailed performance characteristics.