Testing

Serac has two levels of tests, unit and integration. Unit tests are used to test individual components of code, such as a class or function. While integration tests are for testing the code as a whole. For example, testing the serac driver with an input file against blessed answers.

Unit Tests

Unit Tests can be ran via the build target test after building the code.

Integration Tests

Note

Integration testing is in development and not fully featured.

Requirements:

• Installed ATS

• ATS_EXECUTABLE defined in the host-config (added automatically to Spack generated host-configs) or on command line via -DATS_EXECUTABLE=/path/to/ats.

• If using a personal machine, check the ats-config directory in the serac repo and create a json file <your_machine_name>.json if you haven't already. Your machine's name can be found by running the following lines of code:

  $ python3
  >>> import socket
  >>> socket.gethostname().rstrip('1234567890')
  >>> exit()
  

  Currently, there are configuration json files for Toss3 and BlueOS which can be used as reference.

1. Build the code. Build code with the normal steps. More info in the Quickstart Guide. This generates a script in the build directory called ats.sh.

2. Run integration tests. Run the corresponding command for the system you are on:

   # BlueOS
   $ lalloc 2 ./ats.sh
   
   # Toss3
   $ salloc -N2 ./ats.sh
   
   # Personal Machine (currently runs subset of tests)
   $ ./ats.sh
   

   Append --help to the command to see the current options.

3. View results. ATS gives a running summary and the final results. ATS also outputs the following helpful files in the platform and timestamp specific created log directory:

   • ats.log - All output of ATS

   • atss.log - Short summary of the run

   • atsr.xml - JUnit test summary

   ATS also outputs both a .log and .log.err for each test and checker that is run.

Installing ATS

ATS can be installed via the normal devtools install process. More info on Building Serac's Developer Tools. This method is useful because it builds all development tools in one process.

If you want to install ATS by itself, ATS provides multiple methods to install in their Getting Started section.

ATS Test Helper Functions

We provide the following test helper functions to make defining integration tests easier in tests/test.ats.

• tolerance_test

  Adds a test and dependent tolerance check for the given arguments
  
  Parameters
  ----------
  name : str
      The name of the test
  input_file : str
      Path to the input file
  num_mpi_tasks : int, optional
      The number of MPI tasks (default is 1)
  tolerance : str, optional
      The allowed tolerance in test results (default is 0.00001)
      Note: If using options 2 or 3 a "default" value is required unless you've
            specified all parameters
      Three options:
          1.) Either single value such as 0.00001
          2.) Multiple values in this format: default:0.1,velocity:0.22
          3.) Path to a tolerance JSON file
  restart_cycle: int, optional
      The cycle to run from and test
      Note: If given a value, it will perform a tolerance test for both the normal
            serac run and also the restart serac run using the specified cycle
      Note: If ats.sh has the baseline option set, it will copy the restart's summary
            file to the restart baseline location.
  
  Note: Paths in this function are relative to where the function is called from
        not this file