Introduction to OpenMP

Module 12

From Pthreads to OpenMP

We spent four modules with Pthreads. We created workers, passed each one a stable ID, joined them, and protected shared updates ourselves. OpenMP uses the same shared-memory model of several threads in one process. The difference is that we mark which part of the program should run in parallel, and the compiler and the OpenMP runtime create, number, and wait for the threads.

That is the idea of Chapter 5 of Pacheco and Malensek, §§5.1–5.4: start from a working serial program and add a small number of directives to parallelize it. Our three programs are:

NoteCompile these on PicoCluster

OpenMP needs compiler support, turned on with -fopenmp. The gcc on PicoCluster has it. Apple’s clang on macOS does not support -fopenmp without extra installation, so use PicoCluster for this module. As with Pthreads, an OpenMP program runs as one process on one node, and each PicoCluster node has four cores.

First program: a parallel region

Create omp_hello.c. The complete file is available here.

#include <stdio.h>
#include <stdlib.h>
#include <omp.h>

int main(int argc, char *argv[]) {
    int thread_count = 4;
    if (argc > 1) {
        thread_count = (int)strtol(argv[1], NULL, 10);
    }

    printf("Before the parallel region: one thread\n");

#   pragma omp parallel num_threads(thread_count)
    {
        int my_rank = omp_get_thread_num();
        int team_size = omp_get_num_threads();
        printf("Hello from thread %d of %d\n", my_rank, team_size);
    }

    printf("After the parallel region: one thread again\n");
    return 0;
}

Compile with -fopenmp and give the number of threads on the command line:

gcc -Wall -O2 -fopenmp -o omp_hello omp_hello.c
./omp_hello 4

Predict: How many “Hello” lines will appear? In what order? Run it several times, then try ./omp_hello 1 and ./omp_hello 8.

Reading the program

Piece What it does
#include <omp.h> Declares OpenMP library functions such as omp_get_thread_num.
# pragma omp parallel Runs the structured block that follows, here the { … }, on a team of threads.
num_threads(thread_count) A clause that requests a team size.
omp_get_thread_num() Returns this thread’s number in the team, from 0 to team size − 1.
omp_get_num_threads() Returns the number of threads in the current team.

A pragma is an instruction to the compiler. Compilers ignore a pragma they do not recognize. Without -fopenmp, gcc skips the directive, and the program would also fail to link because it calls OpenMP functions. Pacheco writes # and pragma with spaces between them so the directives stand out in the code. #pragma omp parallel works the same way.

The thread that reaches the directive is the parent of the team. In OpenMP terms, the parent and its new child threads form the team, and the parent becomes thread 0. So ./omp_hello 4 creates three new threads, not four. When the block ends, there is an implicit barrier. Every thread in the team must finish the block before the parent continues alone.

Compare this with Module 8:

Pthreads OpenMP
Write a separate thread function. Write the parallel work as a block in place.
pthread_create in a loop # pragma omp parallel
Pass each worker the address of its own ID. Call omp_get_thread_num().
pthread_join in a loop Implicit barrier at the end of the block
Compile with -pthread. Compile with -fopenmp.

The greetings can still appear in any order. All threads compete for the same stdout, just as in our Pthreads programs. The two lines printed by main, however, always come first and last.

TipAnother way to choose the team size

If a program has no num_threads clause, OpenMP checks the OMP_NUM_THREADS environment variable, for example OMP_NUM_THREADS=2 ./a.out. Without either one, it usually chooses one thread per core. We will use the command-line argument so that each run shows how many threads it requested.

Shared and private variables

Inside a parallel region, each variable is either shared or private (§5.3):

  • A variable declared before the parallel region, such as thread_count, is shared. There is one copy, and every thread in the team can access it.
  • A variable declared inside the block, such as my_rank and team_size, is private. Each thread has its own copy on its own stack.

This is the same distinction as globals versus a worker function’s locals in Pthreads. The rule from Module 9 also stays the same: if two threads update one shared variable, we must coordinate those updates.

Discuss: If my_rank were declared before the parallel region and each thread assigned it inside the block, what could go wrong with the greetings?

Second program: the trapezoid rule with a critical section

In Module 3, each MPI process estimated part of the area under \(f(x) = x^2\) from 0 to 1, using a cyclic loop, and MPI_Reduce combined the parts. We will do the same with OpenMP threads, using the thread number in place of the MPI rank.

The core of omp_trap_critical.c is:

double total_area = 0.0;

double start = omp_get_wtime();
#   pragma omp parallel num_threads(thread_count)
{
    int my_rank = omp_get_thread_num();
    int team_size = omp_get_num_threads();
    double my_area = 0.0;

    for (int i = my_rank; i < n; i += team_size) {
        double left = a + i * h;
        double right = a + (i + 1) * h;
        my_area += h * (f(left) + f(right)) / 2.0;
    }

#   pragma omp critical
    total_area += my_area;
}
double elapsed = omp_get_wtime() - start;

Compare the loop with mpi_trap.c: only the source of my_rank and team_size changed. a, h, n, and total_area are declared before the region, so they are shared. The threads only read a, h, and n. Each thread accumulates into its private my_area, then adds it to the shared total once.

# pragma omp critical makes the next statement a critical section: only one thread at a time may execute it. This is the same job as the mutex lock and unlock around our merge in Module 9, written in one line. Here it applies to just the one statement after it. Use braces to protect several statements.

omp_get_wtime() returns wall-clock time in seconds, like MPI_Wtime(). Subtracting two readings gives the elapsed time of the parallel region.

gcc -Wall -O2 -fopenmp -o omp_trap_critical omp_trap_critical.c
./omp_trap_critical 1
./omp_trap_critical 2
./omp_trap_critical 4

The estimated area should agree with \(1/3\) to about twelve decimal places for every thread count. Record the elapsed times:

Threads Elapsed time (s) Speedup over 1 thread
1 1.00
2
4
8

Discuss: What speedup would be ideal with four threads? Why might eight threads on a four-core node not help?

Third program: let OpenMP do the reduction

Combining private results into one total is so common that OpenMP has a clause for it (§5.4). In omp_trap_reduction.c, only two lines differ from the critical version: the clause on the directive, and the removed critical pragma.

#   pragma omp parallel num_threads(thread_count) reduction(+: total_area)
{
    ...
    total_area += my_area;
}

reduction(+: total_area) tells OpenMP that:

  1. Each thread gets a private copy of total_area, starting at 0, the identity value for +.
  2. Inside the block, total_area refers to that thread’s private copy. No other thread can touch it, so no critical section is needed.
  3. At the end of the region, OpenMP adds the private copies and the original value into the shared total_area.

This is OpenMP’s version of MPI_Reduce with MPI_SUM. Other operators include *, max, min, &&, and ||.

gcc -Wall -O2 -fopenmp -o omp_trap_reduction omp_trap_reduction.c
./omp_trap_reduction 4

The answer and timing should match the critical version closely. Each version does only one combining step per thread. The reduction version is shorter, and it lets the OpenMP implementation choose how to combine the values.

NoteFloating-point sums can differ slightly

The threads add the same trapezoids, but in a different grouping for each team size. Floating-point addition is not exactly associative, so the last few digits can change with the number of threads. With %.12f, you will probably not see a difference.

Experiment: where does the critical section go?

For these experiments, change n to 10000000 (ten million) so that the slow versions finish quickly. Run each version with 1 and 4 threads, a few times each. Predict what will happen before you run it.

  1. Remove the critical section. In omp_trap_critical.c, comment out # pragma omp critical. Is the answer wrong every time? Is the program correct?
  2. Update the shared total in the loop without protection. Restore the pragma, then replace my_area += … with total_area += …. Remove the my_area lines and the protected update after the loop. How often is the answer wrong now?
  3. Protect every update in the loop. Put # pragma omp critical on the line just before the total_area += … statement inside the loop. Is the answer right? Compare the 1-thread and 4-thread times.
ImportantA correct-looking run is not proof

In step 1, there are only a few unprotected updates, so most runs probably look correct. It is still a data race, and the behavior is undefined. Step 2 makes millions of racing updates, so lost updates become easy to see. Step 3 is correct but serializes the main work: only one thread at a time can add to the total, and the threads spend their time waiting for one another.

The fast and correct pattern is the one we used with Pthreads: do the bulk of the work in private variables, then combine once per thread. A reduction clause expresses that pattern directly. Restore the original program and n = 500000000 when you finish.

Check your understanding

  1. What does -fopenmp do? What would happen if you forgot it?
  2. With ./omp_hello 4, how many threads are created by the parallel directive, and which thread is thread 0?
  3. Where does an OpenMP program do the equivalent of pthread_join?
  4. In the trapezoid programs, which variables are shared and which are private?
  5. Why does reduction(+: total_area) remove the need for critical?
  6. Why is putting the critical section inside the loop correct but slow?
  1. It turns on OpenMP: the compiler acts on the omp pragmas and links the OpenMP library. Without it, the pragmas are ignored and the calls to omp_ functions fail to link.
  2. Three new threads. The parent that reached the directive joins the team as thread 0.
  3. At the implicit barrier at the end of the parallel region.
  4. Shared: a, b, n, h, thread_count, and total_area (except that the reduction gives each thread a private copy of total_area). Private: my_rank, team_size, my_area, i, left, and right.
  5. Each thread updates its own private copy, and OpenMP combines the copies safely at the end of the region.
  6. Only one thread at a time can run the update, so the threads take turns for millions of additions instead of working at the same time.
Back to top