CI tests fail when compiled with OneAPI compilers

[Madu86]

I've created a CI workflow file for OneAPI compilers: https://github.com/Madu86/QUICK/blob/intel-ci/.github/workflows/serial-intel.yml. We have two problems: 1) CMake doesn't recognize compiler names, 2) tests fail after compilation: https://github.com/Madu86/QUICK/actions/runs/8198338051/job/22421701533. I got around 1 by creating symlinks but we have to fix both.

[agoetz]

Guoquan Chen from Intel has compiled AmberTools 22 (works also with AmberTools 23) with Intel oneAPI LLVM based compilers. All codes compile, including QUICK. See branch amber20-sycl-cmake in the Amber Gitlab repo. He implemented this without CMake smarts (I think starting with CMake 3.27 there is built-in oneAPI support, but we should not rely on users having very recent CMake versions). Compiler is called IntelLLVM (-DCOMPILER=IntelLLVM) to distinguish from old Intel compilers.

Scott Brozell is also working on getting Amber/AmberTools compiled with Intel LLVM compilers. There is an issue / discussion on Amber Gitlab.

Dave Case recently merged support into Amber master. Take a look how this works (compilers called IntelLLVM or ONEAPI).

[ohearnk]

I pulled in the most recent build support in AmberTools in #348. Using this, I was able to successfully build standalone QUICK with OneAPI on the High Performance Computing Center at MSU via the following:

module purge && module load OneAPI/2021.3.0 CMake/3.18.4
cd QUICK
mkdir build_mpi_broadwell_intel_oneapi2021.3.0 && cd build_mpi_broadwell_intel_oneapi2021.3.0
cmake .. -DCOMPILER=ONEAPI -DMPI=TRUE -DCUDA= -DENABLEF= -DCMAKE_INSTALL_PREFIX=${PWD}/../install_mpi_broadwell_intel_oneapi2021.3.0

That said, some gradient and optimization tests are failing for both the serial and MPI implementations (same test failures for both). For the MPI test summary (with 1 process), see below.

======================== Test Summary ========================

mpi version:
    Energy tests: 0/87 (failed/total)
    Gradient tests: 3/71 (failed/total)
    Optimization tests: 4/10 (failed/total)
    API tests: 0/1 (failed/total)
    Total tests: 7/169 (failed/total)

==============================================================

[ohearnk]

Full diffs for the MPI tests:

test/runs/mpi/grad_naip_meoh_libxc_pbeh_sto3g.grad.dif:

704c704
<   193X     0.     0.
>   193X    27.9500174444     0.
705c705
<   193Y     0.     0.
>   193Y    27.9500174444     0.
706c706
<   193Z     0.     0.
>   193Z    27.9500174444     0.
### Maximum absolute error in matching lines = 2.80e+01 at line 704 field 2
### Maximum relative error in matching lines = 1.00e+00 at line 704 field 2

test/runs/mpi/grad_ncnaip_b3lyp_sto3g.grad.dif:

671c671
<   181X     0.     0.
>   181X    26.0000019855     0.
672c672
<   181Y     0.     0.
>   181Y    26.0000019855     0.
673c673
<   181Z     0.     0.
>   181Z    26.0000019855     0.
### Maximum absolute error in matching lines = 2.60e+01 at line 671 field 2
### Maximum relative error in matching lines = 1.00e+00 at line 671 field 2

test/runs/mpi/grad_rpsb_meoh_rhf_sto3g.grad.dif:

911c911
<   247X     0.     0.
>   247X    24.2576170724     0.
912c912
<   247Y     0.     0.
>   247Y    24.2576170724     0.
913c913
<   247Z     0.     0.
>   247Z    24.2576170724     0.
### Maximum absolute error in matching lines = 2.43e+01 at line 911 field 2
### Maximum relative error in matching lines = 1.00e+00 at line 911 field 2

test/runs/mpi/opt_merad_ub3lyp_631gss.dipole.dif:

4c4
<    C             -0.3681            -0.3472
>    C             -0.3448            -0.3278
5c5
<    H              0.1227             0.1165
>    H              0.1150             0.1099
6c6
<    H              0.1227             0.1166
>    H              0.1150             0.1099
7c7
<    H              0.1227             0.1141
>    H              0.1149             0.1080
11c11
<     0.    0.    0.0002    0.0002
>     0.0017    0.0010   -0.6364    0.6364
### Maximum absolute error in matching lines = 6.37e-01 at line 11 field 3
### Maximum relative error in matching lines = 3.18e+03 at line 11 field 3

test/runs/mpi/opt_merad_ub3lyp_631gss.opt.dif:

4c4
<   C       -0.0040   -0.0020   -0.3547
>   C       -0.0048   -0.0024   -0.0748
5c5
<   H       -0.9414    0.5381   -0.3564
>   H       -0.9064    0.5182   -0.4497
6c6
<   H        0.9324    0.5397   -0.3514
>   H        0.8980    0.5198   -0.4449
7c7
<   H       -0.0031   -1.0838   -0.3562
>   H       -0.0029   -1.0436   -0.4494
10c10
<   C        0.    0.    0.
>   C        0.0001    0.   -0.0261
11c11
<   H        0.    0.    0.
>   H        0.0018   -0.0011    0.0087
12c12
<   H        0.    0.    0.
>   H       -0.0019   -0.0011    0.0087
13c13
<   H        0.    0.    0.
>   H        0.    0.0021    0.0087
14c14
<  MINIMIZED ENERGY =    -39.842890550
>  MINIMIZED ENERGY =    -39.831748778
### Maximum absolute error in matching lines = 2.80e-01 at line 4 field 4
### Maximum relative error in matching lines = 3.74e+00 at line 4 field 4

test/runs/mpi/opt_nh4_pbe0_def2svp.dipole.dif:

4c4
<    N              0.0189             0.1411
>    N             -0.0562             0.1049
5c5
<    H              0.2453             0.2142
>    H              0.2635             0.2229
6c6
<    H              0.2453             0.2157
>    H              0.2643             0.2249
7c7
<    H              0.2453             0.2157
>    H              0.2645             0.2250
8c8
<    H              0.2453             0.2131
>    H              0.2640             0.2224
### Maximum absolute error in matching lines = 7.51e-02 at line 4 field 2
### Maximum relative error in matching lines = 3.97e+00 at line 4 field 2

test/runs/mpi/opt_nh4_pbe0_def2svp.opt.dif:

5c5
<   H        0.3432   -0.9711    0.
>   H        0.3535   -1.0008    0.0001
6c6
<   H        0.3434    0.4855    0.8412
>   H        0.3542    0.5006    0.8679
7c7
<   H        0.3434    0.4855   -0.8413
>   H        0.3543    0.5008   -0.8681
8c8
<   H       -1.0301   -0.0002    0.
>   H       -1.0622   -0.0004    0.0001
12c12
<   H        0.    0.    0.
>   H       -0.0041    0.0117    0.
13c13
<   H        0.    0.0001    0.0001
>   H       -0.0043   -0.0060   -0.0105
14c14
<   H        0.    0.0001   -0.0001
>   H       -0.0043   -0.0061    0.0106
15c15
<   H       -0.0001    0.    0.
>   H        0.0127    0.    0.
### Maximum absolute error in matching lines = 3.21e-02 at line 8 field 2
### Maximum relative error in matching lines = 1.28e+02 at line 15 field 2

test/runs/mpi/opt_wat_rhf_631g.dipole.dif:

4c4
<    O             -0.8106            -0.5921
>    O             -0.8710            -0.6417
5c5
<    H              0.4051             0.2955
>    H              0.4354             0.3206
6c6
<    H              0.4055             0.2966
>    H              0.4356             0.3211
10c10
<    -1.6865   -1.4057   -1.1610    2.4836
>    -1.3794   -1.2282   -1.1453    2.1732
### Maximum absolute error in matching lines = 3.07e-01 at line 10 field 1
### Maximum relative error in matching lines = 2.23e-01 at line 10 field 1

test/runs/mpi/opt_wat_rhf_631g.opt.dif:

1c1
< ============= GEOMETRY INFORMATION (NOT OPTIMIZED) ===========
> ================ OPTIMIZED GEOMETRY INFORMATION ==============
4c4
<   O       -0.6831    1.8145    2.9291
>   O       -0.7415    1.7521    2.8963
5c5
<   H       -1.1103    1.0513    3.3055
>   H       -1.1112    0.9798    3.3523
6c6
<   H       -0.9798    1.9792    2.0462
>   H       -0.9205    2.0365    1.9840
9c9
<   O       -0.0012    0.0001   -0.0027
>   O        0.0043    0.0043    0.0027
10c10
<   H        0.0002    0.0001    0.0002
>   H       -0.0001    0.0099   -0.0173
11c11
<   H        0.0010   -0.0003    0.0026
>   H       -0.0042   -0.0142    0.0146
12c12
<  MINIMIZED ENERGY =    -75.985328359
>  MINIMIZED ENERGY =    -75.977981250
### Maximum absolute error in matching lines = 7.15e-02 at line 5 field 3
### Maximum relative error in matching lines = 9.80e+01 at line 10 field 3

test/runs/mpi/opt_wat_rhf_ccpvdz.dipole.dif:

4c4
<    O             -0.3182            -0.5873
>    O             -0.3852            -0.6105
5c5
<    H              0.1611             0.2953
>    H              0.1920             0.3057
6c6
<    H              0.1571             0.2920
>    H              0.1932             0.3048
10c10
<    -1.3627   -1.0935   -0.9504    1.9890
>    -1.0537   -0.9377   -0.8758    1.6603
### Maximum absolute error in matching lines = 3.09e-01 at line 10 field 1
### Maximum relative error in matching lines = 2.93e-01 at line 10 field 1

test/runs/mpi/opt_wat_rhf_ccpvdz.opt.dif:

1c1
< ============= GEOMETRY INFORMATION (NOT OPTIMIZED) ===========
> ================ OPTIMIZED GEOMETRY INFORMATION ==============
4c4
<   O       -0.6648    1.8366    2.9551
>   O       -0.7415    1.7521    2.8963
5c5
<   H       -1.1259    1.0729    3.2792
>   H       -1.1112    0.9798    3.3523
6c6
<   H       -0.9965    1.9589    2.0806
>   H       -0.9205    2.0365    1.9840
9c9
<   O        0.0007   -0.0015    0.0038
>   O        0.0080    0.0076    0.0058
10c10
<   H        0.0014    0.0071   -0.0089
>   H       -0.0015    0.0106   -0.0218
11c11
<   H       -0.0021   -0.0057    0.0051
>   H       -0.0065   -0.0182    0.0161
12c12
<  MINIMIZED ENERGY =    -76.025871001
>  MINIMIZED ENERGY =    -76.013294758
### Maximum absolute error in matching lines = 7.67e-02 at line 4 field 2
### Maximum relative error in matching lines = 1.04e+01 at line 9 field 2