Strange Behavior While Adding Coarray Variables

1

I’ve noticed that, while using the OpenCoarray Coarray Fortran Compiler, adding integers, which come from other images, yields incorrect results. Here’s my simple program:

program enigma2
	use iso_fortran_env
				
	implicit none
	
	integer, codimension[*], save :: imc
	integer :: r
	
	imc = 0
	if (.true.) then
		imc = 1
	end if
	sync all
	
	if (this_image() == 1) then
		do r = 2,num_images()				
			print *, r, ': ', imc, ' and ', imc[r]
			imc = imc[r] + imc
		end do
	end if
	sync all
	
	print *, 'inc is ', imc	
	
end program enigma2

Here are the results that are given, after running the program:

evansste@Computer5:$ cafrun -np 5 enigma2
           2 :            1  and            1
           3 :            2  and            2
           4 :            4  and            4
           5 :            8  and            1
 inc is            9
 inc is            1
 inc is            1
 inc is            1
 inc is            1

Shouldn’t the resulting total for inc be equal to the number of images that are used while running this program; which, in this case, is five?

I’m not an expert on Fortran or coarrays. So, I want to make sure I’m not making some silly mistake in how I’ve written this program. Am I doing this right, or is there something wrong with my software installation?

I appreciate any insight that anyone is willing to provide.

2

That looks like a compiler bug. I can’t see any undefined behaviour in your program. imc[r], wherever it is referenced in your code, must have the value given in the first segment, which is 1.

3

I tried your program with ifx on Windows and got the expected result. So I’d say @themos is correct: this exposes a compiler bug. Can you report it on the GCC/gfortran bug report system?

4

On my machine (M1, macOS 15.6, gfortran-14.3.0, opencoarrays-2.10.2 installed via homebrew), the code gives the expected result:

$ caf test1.f90         
$ cafrun -np 5 ./a.out
           2 :            1  and            1
           3 :            2  and            1
           4 :            3  and            1
           5 :            4  and            1
 inc is            5
 inc is            1
 inc is            1
 inc is            1
 inc is            1

More info about my installation:

$ caf -w
caf wraps /opt/homebrew/opt/gcc@14/bin/gfortran-14
$ caf -s
/opt/homebrew/opt/gcc@14/bin/gfortran-14 -I/opt/homebrew/Cellar/opencoarrays/2.10.2_5/include/OpenCoarrays-2.10.2_GNU-14.3.0 -fcoarray=lib -Wl,-flat_namespace -L/opt/homebrew/opt/open-mpi/lib ${@} /opt/homebrew/Cellar/opencoarrays/2.10.2_5/lib/libcaf_mpi.a /opt/homebrew/lib/libmpi_usempif08.dylib /opt/homebrew/lib/libmpi_usempi_ignore_tkr.dylib /opt/homebrew/lib/libmpi_mpifh.dylib /opt/homebrew/lib/libmpi.dylib
$ cafrun -w
mpiexec (Open MPI) 5.0.8
$ cafrun -s
/opt/homebrew/bin/mpiexec -n <number_of_images> /path/to/coarray_Fortran_program [arg4 [arg5 [...]]]
$ otool -L ./a.out
./a.out:
	/opt/homebrew/opt/open-mpi/lib/libmpi_usempif08.40.dylib (compatibility version 81.0.0, current version 81.3.0)
	/opt/homebrew/opt/open-mpi/lib/libmpi_usempi_ignore_tkr.40.dylib (compatibility version 81.0.0, current version 81.1.0)
	/opt/homebrew/opt/open-mpi/lib/libmpi_mpifh.40.dylib (compatibility version 81.0.0, current version 81.1.0)
	/opt/homebrew/opt/open-mpi/lib/libmpi.40.dylib (compatibility version 81.0.0, current version 81.7.0)
	/opt/homebrew/opt/gcc@14/lib/gcc/14/libgfortran.5.dylib (compatibility version 6.0.0, current version 6.0.0)
	/opt/homebrew/opt/gcc@14/lib/gcc/14/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0

So, if the versions of gfortran and opencoarrays are the same as above, there might be some installation problem…? (and it might be a compiler or library bug if the versions are different…)

5

Thanks, so much, for all of your helpful responses. I suspected that it may have been due to a bad installation. Now, I know that it, certainly, is.

My operating system is Debian 12, and I installed from source. I discovered that OpenCoarrays has different bugs, if Octave is installed along side it. Now, I know that, even with Octave uninstalled, it still has bugs.

I’m using the latest version of OpenCoarrays; which is version 2.10.3. I typed gfortran --version, at the command line, and it shows GNU Fortran (Debian 12.2.0-14+deb12u1) 12.2.0.

Here’s how I installed OpenCoarrays, specifically.

After a fresh installation of Debian 12, and getting sudo and apt to work, I typed all of the following commands at the command line:

sudo apt update
sudo apt install build-essential cmake
sudo apt install mpich
sudo apt install git-all
git clone https://github.com/sourceryinstitute/opencoarrays.git
cd opencoarrays
mkdir build
cd build
cmake -DMPI_Fortran_COMPILER=mpifort ..
cmake --build .
sudo cmake --build . --target install

After doing all of that, I opened the .bashrc file, and added the following line at the end of the file:

export PATH=$PATH:/home/evansste/opencoarrays/build/bin

I, then, rebooted the machine.

The good news is that I’ve now been able to verify that the problem is with my installation. It’s a great help to know that many of you were able to run the program, and get correct results, on your machines. I appreciate that many of you took the time to do that.

6

I have little experience with MPICH (and I would like to learn more about the difference from OpenMPI), but LLM seems to indicate some potential issues (?) about the combination of OpenCoarrays + MPICH, so that might be another point to check…? (because my homebrew installation uses OpenMPI) (The following page has some links to the original webpages).

7

I started off by not using MPICH. I only chose to use it because of seeming compatibility issues with Debian 12. This is according to a Google AI search.

I, first, started using OpenCoarrays, several years ago. Back then, I was using Lubuntu 19.04, and was able to get OpenCoarrays to install, and work, fairly well.

After, later, running into bugs that emerged while installing Octave on Lubuntu, I started using Debian; which solved this problem. Now that I’m getting back into Fortran programming, I’ve had to look into installing OpenCoarrays on Debian, rather than Lubuntu.

Google searches, with AI results, have made the process appear easy; in terms of learning how to install. However, it’s only after a successful installation, that I’ve started to notice these bugs.

When I performed a Google search, asking whether or not anyone has successfully installed OpenCoarrays on Debian 12, in 2025, AI responded by stating that there have been compatibility issues with Debian and OpenMPI. It suggested that I use MPICH, instead, and provided a way to install, with that. So, that’s what I did.

I’m a bit of a novice, when it comes to computer programming. I’m one of those people who loves the power of computing, and how computers are able to manipulate massive amounts of data. However, programming is a necessary evil. It’s one of the reasons that I fell in love with MATLAB/Octave, and now, Fortran. They’re extremely powerful programming languages, that are relatively light on syntax.

All of that is to say that, most of my experience is in using these higher-level programming languages, and not in how to install/maintain them. Google and AI, have been very helpful, in terms of explaining how to install. However, it’s not the best at offering experience, or to know how well these installations actually work. For this reason, I truly value the insight that I’m getting, here.

Thanks, again, to all of you.

1 Like
8

It’s interesting that different LLMs give different replies :grin: (I’ve also tried asking ChatGPT for known issues of OpenMPI + OpenCoarrays, and then it gave a lot of remarks / pitfalls…) I will also try Google AI or other LLMs for similar questions.

FWIW, I’ve tested the code with Ubuntu-22 + default gfortran-11 + opencoarrays-openmpi (all installed via apt) on an AMD (Ryzen) machine, which gave the expected result. (I will try opencoarrays-mpich later.) So, if you are not restricted to MPICH for some reason (e.g., other applications require it), it might be an option to try the apt packages based on OpenMPI (but it may be necessary to avoid interference with existing MPI installations via search path etc). The related package names are like:

libcoarrays-dev
libcoarrays-openmpi-dev (or libcoarrays-mpich-dev)
($ apt info libcoarrays* gives more info.)

And the output from Ubuntu22 is like this:

$ caf --version
OpenCoarrays Coarray Fortran Compiler Wrapper (caf version 2.9.2)

$ caf -w
caf wraps /usr/bin/mpif90.openmpi

$ /usr/bin/mpif90.openmpi -v
...
gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04.2) 

$ caf -s
/usr/bin/mpif90.openmpi -I/usr/lib/x86_64-linux-gnu/fortran/ -fcoarray=lib ${@} /usr/lib/x86_64-linux-gnu/open-coarrays/openmpi/lib/libcaf_openmpi.a

$ cafrun -w
mpiexec (OpenRTE) 4.1.2

Report bugs to http://www.open-mpi.org/community/help/
$ cafrun -s
/usr/bin/mpiexec -n <number_of_images> /path/to/coarray_Fortran_program [arg4 [arg5 [...]]]

$ ldd ./a.out
	linux-vdso.so.1 (0x00007ffd5650a000)
	libmpi.so.40 => /lib/x86_64-linux-gnu/libmpi.so.40 (0x00007a557c007000)
	libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007a557bc00000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007a557b800000)
	libopen-rte.so.40 => /lib/x86_64-linux-gnu/libopen-rte.so.40 (0x00007a557bf4a000)
	libopen-pal.so.40 => /lib/x86_64-linux-gnu/libopen-pal.so.40 (0x00007a557bb4d000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007a557ba66000)
	libhwloc.so.15 => /lib/x86_64-linux-gnu/libhwloc.so.15 (0x00007a557beec000)
	libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007a557b7b8000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007a557ba46000)
	/lib64/ld-linux-x86-64.so.2 (0x00007a557c16a000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007a557ba2a000)
	libevent_core-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.7 (0x00007a557b783000)
	libevent_pthreads-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007a557bee5000)
	libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007a557b759000)
9

We had success using spack to install opencoarrays Spack Packages. This could be a quick way for you to deploy different combinations of mpi/opencoarrays quickly.