Getting Started

Define Requirements

First, we must define a set of requirements for our production test, which has nothing to do with the code itself. This will assist us in developing each test and, finally, defining the test sequence.

The first test is a simple communication test. Is the device communicating, True or False?

The second test is a flow test in which the test communicates with an instrument and reads the flow, applying a minimum and maximum flow.

In addition to the test requirements, it turns out that most hardware requires a power supply or other physical component to be turned on or initialized. We will deal with this in our setup and teardown.

Develop Hardware API

Based on the test requirements, it is useful to create some hardware abstractions which expose some of the functionality that we are interested in. The hardware API development effort is independent of the test, but will be necessary to most testing efforts and, thus, we will cover a basic implementation.

Our first object will be a power supply. During the test, we will wish to control a power supply on and off, perhaps even setting different voltages and current limits, depending on the test. For this example, we will only wish to turn on the power supply at the beginning of the test and turn it off at the end of the test or if there is some test fault. We will call our power supply class PowerSupply and we will store the PowerSupply class within power_supply.py.

The next abstraction communicates with the device and provides a property exposing whether the device is communicating or not. The development of this abstraction is beyond the scope of this document; however, it is possible to assume a simple API has been provided using a class called Device which has the property is_communicating.

The last abstraction communicates with data acquisition hardware to determine the flow of the system in liters / minute. The simple api that we are assuming is embodied in a class called Daq and exposed on a property flow.

At this point, our directory structure has a couple of files in it which contain the abstractions. We will call these daq.py and device.py:

root/
  daq.py
  device.py
  power_supply.py

Create Automated Test File

We will now create an automated_tests.py which will contain all of the automated tests that we wish to execute, setup function, and teardown function.

 root/
   automated_tests.py
   daq.py
   device.py
   power_supply.py

Within automated_tests.py, we create our setup, teardown, and individual tests.

setup() and teardown()

For each test, we want to start in a known condition. Use setup and teardown functions supplied to execute commands without saving any data. The primary difference between these functions and a Test is that the setup and teardown functions do not save data.

In our case, the setup() function will turn on the power supply and the teardown() funciton will turn the power supply off. If there is a critical error during the test, the teardown() function will be executed, making it especially convenient for test environments in which the safe must end in a guaranteed safe condition.

from time import sleep
from power_supply import PowerSupply

def setup_hardware(psu: PowerSupply):
    psu.set_voltage(12.0)
    psu.set_current(3.0)
    psu.on()
    sleep(1.0)  # allow power to stabilize

def teardown_hardware(psu: PowerSupply):
   psu.off()
   sleep(0.1)

Note that our setup and teardown functions accept instances of hardware classes. This method makes it fairly easy to modularize and develop each function and Test class efficiently.

Develop Individual Tests

Communications Test

Within automated_tests.py, import mats.Test and subclass our custom tests.

 from time import sleep
 from mats import Test
 from power_supply import PowerSupply

 def setup_hardware(psu: PowerSupply):
     psu.set_voltage(12.0)
     psu.set_current(3.0)
     psu.on()
     sleep(1.0)  # allow power to stabilize

 def teardown_hardware(psu: PowerSupply):
     psu.off()
     sleep(0.1)

 class CommunicationTest(Test):
     def __init__(self):
         super().__init__(moniker='communications')

     def execute(self, is_passing):
         return None

 class FlowTest(Test):
     def __init__(self):
         super().__init__(moniker='flow')

     def execute(self, is_passing):
         return None

At this point, our tests don’t do anything but implement the test class. If this test were executed within a sequence that saved data, it would end up applying no pass/fail criteria and would save None to the headers fields communications and flow.

Note

The moniker of all test sequences must be unique or the test sequence will raise an error!

First, we will focus on the communications test. We will modify our imports to add from device import Device which gives us access to the device class. In some cases, the device will be instantiated already, in which case it might be more appropriate to import the instance of the class rather than the class itself. In most cases, it is worth it to externally allocate hardware and execute the test by passing the class instance.

from time import sleep
from mats import Test
from power_supply import PowerSupply
from device import Device
...

Next, we will store an instance of the hardware within the CommunicationTest and so that we can utilize it during development.

 class CommunicationTest(Test):
     def __init__(self, device: Device):
         super().__init__(moniker='communications')
         self._device = device

     def execute(self, is_passing):
         return None

Now, it is time to acquire a bit of data.

 class CommunicationTest(Test):
     def __init__(self, device: Device):
         super().__init__(moniker='communications')
         self._device = device

     def execute(self, is_passing):
         return self._device.is_communicating

If the test sequence were executed at this point, there would be no pass/fail criteria applied, but a True/False value would be saved under the communications header in the data file.

In order to apply criteria, we will use the pass_if parameter of Test.__init__()

 class CommunicationTest(Test):
     def __init__(self, device: Device):
         super().__init__(moniker='communications', pass_if=True)
         self._device = device

     def execute(self, is_passing):
         return self._device.is_communicating

Flow Test

The development of the flow test will proceed similarly to the previous test.

from time import sleep
from mats import Test
from power_supply import PowerSupply
from daq import daq
from device import Device

...

class FlowTest(Test):
   def __init__(self, daq: Daq):
       super().__init__(moniker='flow')
       self._daq = daq

   def execute(self, is_passing):
       return self._daq.flow

Next, we will apply minimum and maximum pass/fail criteria to the test:

 class FlowTest(Test):
     def __init__(self, daq: Daq):
         super().__init__(moniker='flow',
                          min_value=5.8, max_value=6.2)
         self._daq = daq

     def execute(self, is_passing):
         return self._daq.flow

Using the min_value and max_value parameters allows us to apply quantitative pass/fail criteria to the results of the execution step.

Complete Test Definition

Finally, we have our complete test definition!

from time import sleep
from mats import Test
from power_supply import PowerSupply
from daq import daq
from device import Device

def setup_hardware(psu: PowerSupply):
    psu.set_voltage(12.0)
    psu.set_current(3.0)
    psu.on()
    sleep(1.0)  # allow power to stabilize

def teardown_hardware(psu: PowerSupply):
    psu.off()
    sleep(0.1)

class CommunicationTest(Test):
    def __init__(self, device: Device):
        super().__init__(moniker='communications', pass_if=True)
        self._device = device

    def execute(self, is_passing):
        return self._device.is_communicating

class FlowTest(Test):
    def __init__(self, daq: Daq):
        super().__init__(moniker='flow',
                         min_value=5.8, max_value=6.2)
        self._daq = daq

    def execute(self, is_passing):
        return self._daq.flow

Create Test Sequence

Up to this point, we have created some tests using the framework, but we haven’t actually done anything with them. It would be wise to creat the TestSequence instance earlier than this point in most development processes; however, we have chosen the order in order to better organize the presentation.

We will create the TestSequence within its own file, making our new file structure:

 root/
   automated_tests.py
   daq.py
   device.py
   power_supply.py
   test_sequence.py

Allocate Hardware

Allocate some hardware within test_sequence.py.

from power_supply import PowerSupply
from daq import Daq
from device import Device

# allocate the hardware
psu = PowerSupply()
daq = Daq()
device = Device()

Within test_sequence.py, we will import our mats.TestSequence() along with the CommunicationsTest() and FlowTest() that we previously defined:

 from mats import TestSequence
 from automated_tests import FlowTest, CommunicationsTest,\
     setup_hardware, teardown_hardware
 from power_supply import PowerSupply
 from daq import Daq
 from device import Device

 # allocate the hardware
 psu = PowerSupply()
 daq = Daq()
 device = Device()

Now, we create our sequence as the instantiation of the test objects into a list:

sequence = [
    CommunicationsTest(device),
    FlowTest(daq)
]

It is common to forget to instantiate the objects, so be sure that you include the () so that you are using instances of the test and not the test class. The order of the test sequence is defined by the order of the list, so a re-ordering of this list is all that is required to refactor the order of the tests.

Now we create the TestSequence instance, supplying the sequence of tests, the setup, and the teardown functions, being sure to capture an object handle for the test sequence:

ts = TestSequence(
        sequence=sequence,
        setup=lambda: setup_hardware(psu),
        teardown=lambda: teardown_hardware(psu)
    )

Finally, we run the test sequence one time:

ts.start()

The test will run to completion and output log data to the terminal.

The final full form of test_sequence.py:

from mats import TestSequence
from automated_tests import FlowTest, CommunicationsTest,\
    setup_hardware, teardown_hardware
from power_supply import PowerSupply
from daq import Daq
from device import Device

# allocate the hardware
psu = PowerSupply()
daq = Daq()
device = Device()

sequence = [
    CommunicationsTest(device),
    FlowTest(daq)
]

ts = TestSequence(
    sequence=sequence,
    setup=lambda: setup_hardware(psu),
    teardown=lambda: teardown_hardware(psu)
)

ts.start()

# wait for sequence to complete before exiting this thread
while ts.in_progress:
    sleep(0.1)

Save the Data

At this point, we run the test and collect the data, but do not save it anywhere. There are a couple of options for saving. The first - and easiest - is to use the built-in ArchiveManager, which creates the most common formats of csv and csv-like files common in manufacturing environments. It also does some basic test change detection in order to keep file headers separated as the test evolves over the life of the project.

The most basic implementation of the ArchiveManager can be easily added to the sequence:

 from mats import TestSequence, ArchiveManager
 from automated_tests import FlowTest, CommunicationsTest,\
     setup_hardware, teardown_hardware
 from power_supply import PowerSupply
 from daq import Daq
 from device import Device

 # allocate the hardware
 psu = PowerSupply()
 daq = Daq()
 device = Device()

 sequence = [
     CommunicationsTest(device),
     FlowTest(daq)
 ]

 am = ArchiveManager()

 ts = TestSequence(
     sequence=sequence,
     setup=lambda: setup_hardware(psu),
     teardown=lambda: teardown_hardware(psu)
     archive_manager=am
 )

 ts.start()

 # wait for sequence to complete before exiting this thread
 while ts.in_progress:
     sleep(0.1)

The only requirement for the object instance supplied to archive_manager is to implement the save() method which will accept a dict containing the key: value pairs on test completion. In this way, it is very easy to supply custom archive manager objects to extend the functionality of the archiving process.

See Saving Data for more information about the archive manager.