Profiling Serac using Adiak and Caliper

Introduction to Adiak

Adiak is a library developed at LLNL for collecting metadata that can be used to compare multiple runs across programs. For more information, read Adiak's documentation. Note that Serac provides some wrapper functions to initialize and finalize Adiak metadata collection.

Introduction to Caliper

Caliper is a framework developed at LLNL for measuring the performance of programs. To find out more, read Caliper's documentation. Serac also provides convenient macros that make it easy to instrument and assess the performance of simulation code.

Introduction to SPOT

SPOT is a framework developed at LLNL for vizualizing performance data. SPOT is an external tool and does not need to be linked into Serac.

Build Instructions

To use Adiak and Caliper with Serac, install the profiling variant of serac with Spack, i.e., serac+profiling. Note that these libraries are pre-built as part of the installed set of libraries on LC.

Instrumenting Code

To use the functions and macros described in the remainder of this section, the serac/infrastructure/profiling.hpp header must be included.

To enable Adiak and Caliper for a program, call serac::profiling::initialize(). This will begin the collection of metadata and performance data. Optionally, an MPI communicator can be passed to configure Adiak and a Caliper ConfigManager configuration string can be passed to configure Caliper. Note that you must still annotate regions to be profiled and provide any custom metadata.

Call serac::profiling::finalize() to conclude metadata and performance monitoring and to write the data to a .cali file.

To provide custom metadata for comparing program runs, call SERAC_SET_METADATA(name, data) after serac::profiling::initialize() and before serac::profiling::finalize. This will add extra metadata into the .cali file. Supported metadata types are integrals, floating points, and strings. Note that this macro is a no-op if the profiling variant is not used.

SERAC_SET_METADATA("dimensions", 2);
SERAC_SET_METADATA("mesh", "../data/star.mesh");

To add profile regions and ensure that Caliper is only used when it has been enabled through Spack, only use the macros described below to instrument your code:

Use SERAC_MARK_FUNCTION at the very top of a function to mark it for profiling.

Use SERAC_MARK_BEGIN(name) at the beginning of a region and SERAC_MARK_END(name) at the end of the region.

Use SERAC_MARK_LOOP_BEGIN(id, name) before a loop to mark it for profiling, SERAC_MARK_LOOP_ITER(id, i) at the beginning of the i th iteration of a loop, and SERAC_MARK_LOOP_END(id) immediately after the loop ends:

SERAC_MARK_BEGIN("region_name");

SERAC_MARK_LOOP_BEGIN(doubling_loop, "doubling_loop");
for (int i = 0; i < input.size(); i++)
{
  SERAC_MARK_LOOP_ITER(doubling_loop, i);
  output[i] = input[i] * 2;
}
SERAC_MARK_LOOP_END(doubling_loop);

SERAC_MARK_END("region_name");

Note that the id argument to the SERAC_MARK_LOOP_* macros can be any identifier as long as it is consistent between all uses of SERAC_MARK_LOOP_* for a given loop.

To reduce the amount of annotation for regions bounded by a particular scope, use SERAC_PROFILE_SCOPE(name). This will follow RAII and works with graceful exception handling. When SERAC_PROFILE_SCOPE is instantiated, profiling of this region starts, and when the scope exits, profiling of this region will end.

 // Refine once more and utilize SERAC_PROFILE_SCOPE
{
  SERAC_PROFILE_SCOPE("RefineOnceMore");
  pmesh->UniformRefinement();
}

Alternatively, for single line expressions, use SERAC_PROFILE_EXPR(name, expr). In the following example, only the call to buildMeshFromFile will be profiled (tag = "LOAD_MESH").

auto pmesh = mesh::refineAndDistribute(SERAC_PROFILE_EXPR("LOAD_MESH", buildMeshFromFile(mesh_file)), 0, 0);

Note

SERAC_PROFILE_EXPR creates a lambda and the expression is evaluated within that scope, and then the result is returned.

Performance Data

The metadata and performance data are output to a .cali file. To analyze the contents of this file, use cali-query.

To view this data with SPOT, open a browser, navigate to the SPOT server (e.g. LC), and open the directory containing one or more .cali files. For more information, watch this recorded tutorial.