Skip to content

Commit

Permalink
Adding GPU poisson Ctests and changes
Browse files Browse the repository at this point in the history
  • Loading branch information
kartickr committed Mar 11, 2024
1 parent 50cdd0e commit 43061d7
Show file tree
Hide file tree
Showing 11 changed files with 453 additions and 0 deletions.
103 changes: 103 additions & 0 deletions testsGPU/pseudopotential/real/accuracyBenchmarks/outputMg2x_15
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
number of atoms: 31
number of atoms types: 1
Z:12
=============================================================================================================================
number of electrons: 310
number of eigen values: 180
=============================================================================================================================
-----------Simulation Domain bounding vectors (lattice vectors in fully periodic case)-------------
v1 : 1.176399999999999935e+01 0.000000000000000000e+00 0.000000000000000000e+00
v2 : 0.000000000000000000e+00 1.917200000000000060e+01 0.000000000000000000e+00
v3 : 0.000000000000000000e+00 0.000000000000000000e+00 2.037584570023999930e+01
-----------------------------------------------------------------------------------------
-----Fractional coordinates of atoms------
AtomId 0: 2.500000000000000000e-01 2.500000000000000000e-01 4.166666666669999741e-01
AtomId 1: 2.500000000000000000e-01 0.000000000000000000e+00 2.500000000000000000e-01
AtomId 2: 0.000000000000000000e+00 2.500000000000000000e-01 1.666666666670000019e-01
AtomId 3: 0.000000000000000000e+00 0.000000000000000000e+00 5.000000000000000000e-01
AtomId 4: 2.500000000000000000e-01 2.500000000000000000e-01 9.166666666670000296e-01
AtomId 5: 2.500000000000000000e-01 0.000000000000000000e+00 7.500000000000000000e-01
AtomId 6: 0.000000000000000000e+00 2.500000000000000000e-01 6.666666666670000296e-01
AtomId 7: 0.000000000000000000e+00 5.000000000000000000e-01 0.000000000000000000e+00
AtomId 8: 2.500000000000000000e-01 7.500000000000000000e-01 4.166666666669999741e-01
AtomId 9: 2.500000000000000000e-01 5.000000000000000000e-01 2.500000000000000000e-01
AtomId 10: 0.000000000000000000e+00 7.500000000000000000e-01 1.666666666670000019e-01
AtomId 11: 0.000000000000000000e+00 5.000000000000000000e-01 5.000000000000000000e-01
AtomId 12: 2.500000000000000000e-01 7.500000000000000000e-01 9.166666666670000296e-01
AtomId 13: 2.500000000000000000e-01 5.000000000000000000e-01 7.500000000000000000e-01
AtomId 14: 0.000000000000000000e+00 7.500000000000000000e-01 6.666666666670000296e-01
AtomId 15: 5.000000000000000000e-01 0.000000000000000000e+00 0.000000000000000000e+00
AtomId 16: 7.500000000000000000e-01 2.500000000000000000e-01 4.166666666669999741e-01
AtomId 17: 7.500000000000000000e-01 0.000000000000000000e+00 2.500000000000000000e-01
AtomId 18: 5.000000000000000000e-01 2.500000000000000000e-01 1.666666666670000019e-01
AtomId 19: 5.000000000000000000e-01 0.000000000000000000e+00 5.000000000000000000e-01
AtomId 20: 7.500000000000000000e-01 2.500000000000000000e-01 9.166666666670000296e-01
AtomId 21: 7.500000000000000000e-01 0.000000000000000000e+00 7.500000000000000000e-01
AtomId 22: 5.000000000000000000e-01 2.500000000000000000e-01 6.666666666670000296e-01
AtomId 23: 5.000000000000000000e-01 5.000000000000000000e-01 0.000000000000000000e+00
AtomId 24: 7.500000000000000000e-01 7.500000000000000000e-01 4.166666666669999741e-01
AtomId 25: 7.500000000000000000e-01 5.000000000000000000e-01 2.500000000000000000e-01
AtomId 26: 5.000000000000000000e-01 7.500000000000000000e-01 1.666666666670000019e-01
AtomId 27: 5.000000000000000000e-01 5.000000000000000000e-01 5.000000000000000000e-01
AtomId 28: 7.500000000000000000e-01 7.500000000000000000e-01 9.166666666670000296e-01
AtomId 29: 7.500000000000000000e-01 5.000000000000000000e-01 7.500000000000000000e-01
AtomId 30: 5.000000000000000000e-01 7.500000000000000000e-01 6.666666666670000296e-01
-----------------------------------------------------------------------------------------
Number Image Charges 2094

Finite element mesh information
-------------------------------------------------
FE interpolating polynomial order for Kohn-Sham eigenvalue problem: 3
FE interpolating polynomial order for electrostatics solve: 3
FE interpolating polynomial order for nodal electron density computation: 5
number of elements: 1440
number of degrees of freedom for the Kohn-Sham eigenvalue problem : 52791
-------------------------------------------------

Setting initial guess for wavefunctions....

Reading initial guess for electron-density.....

Pseudopotential initalization....

Starting SCF iterations....
SCF iterations converged to the specified tolerance after: 15 iterations.

Energy computations (Hartree)
-------------------
Total energy: -1673.62180253

Absolute values of ion forces (Hartree/Bohr)
--------------------------------------------------------------------------------------------
AtomId 0: 0.001805,0.001878,0.013126
AtomId 1: 0.002319,0.000000,0.003373
AtomId 2: 0.000233,0.001707,0.140373
AtomId 3: 0.000194,0.000000,0.004429
AtomId 4: 0.001175,0.001333,0.013673
AtomId 5: 0.001964,0.000000,0.000305
AtomId 6: 0.000233,0.000011,0.138744
AtomId 7: 0.000191,0.000000,0.000954
AtomId 8: 0.001805,0.001877,0.013126
AtomId 9: 0.003732,0.000000,0.001505
AtomId 10: 0.000233,0.001707,0.140373
AtomId 11: 0.000192,0.000000,0.000588
AtomId 12: 0.001175,0.001333,0.013673
AtomId 13: 0.000157,0.000000,0.001220
AtomId 14: 0.000233,0.000011,0.138744
AtomId 15: 0.000004,0.000000,0.001199
AtomId 16: 0.001805,0.001878,0.013126
AtomId 17: 0.002319,0.000000,0.003373
AtomId 18: 0.000000,0.001220,0.140761
AtomId 19: 0.000096,0.000000,0.000014
AtomId 20: 0.001175,0.001333,0.013673
AtomId 21: 0.001964,0.000000,0.000305
AtomId 22: 0.000000,0.002938,0.140140
AtomId 23: 0.000093,0.000000,0.004499
AtomId 24: 0.001805,0.001877,0.013126
AtomId 25: 0.003732,0.000000,0.001505
AtomId 26: 0.000000,0.001220,0.140761
AtomId 27: 0.000000,0.000000,0.000041
AtomId 28: 0.001175,0.001333,0.013673
AtomId 29: 0.000157,0.000000,0.001220
AtomId 30: 0.000000,0.002938,0.140140
--------------------------------------------------------------------------------------------
68 changes: 68 additions & 0 deletions testsGPU/pseudopotential/real/accuracyBenchmarks/outputMg2x_16
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
number of atoms: 31
number of atoms types: 1
Z:12
=============================================================================================================================
number of electrons: 310
number of eigen values: 180
=============================================================================================================================
-----------Simulation Domain bounding vectors (lattice vectors in fully periodic case)-------------
v1 : 1.176399999999999935e+01 0.000000000000000000e+00 0.000000000000000000e+00
v2 : 0.000000000000000000e+00 1.917200000000000060e+01 0.000000000000000000e+00
v3 : 0.000000000000000000e+00 0.000000000000000000e+00 2.037584570023999930e+01
-----------------------------------------------------------------------------------------
-----Fractional coordinates of atoms------
AtomId 0: 2.500000000000000000e-01 2.500000000000000000e-01 4.166666666669999741e-01
AtomId 1: 2.500000000000000000e-01 0.000000000000000000e+00 2.500000000000000000e-01
AtomId 2: 0.000000000000000000e+00 2.500000000000000000e-01 1.666666666670000019e-01
AtomId 3: 0.000000000000000000e+00 0.000000000000000000e+00 5.000000000000000000e-01
AtomId 4: 2.500000000000000000e-01 2.500000000000000000e-01 9.166666666670000296e-01
AtomId 5: 2.500000000000000000e-01 0.000000000000000000e+00 7.500000000000000000e-01
AtomId 6: 0.000000000000000000e+00 2.500000000000000000e-01 6.666666666670000296e-01
AtomId 7: 0.000000000000000000e+00 5.000000000000000000e-01 0.000000000000000000e+00
AtomId 8: 2.500000000000000000e-01 7.500000000000000000e-01 4.166666666669999741e-01
AtomId 9: 2.500000000000000000e-01 5.000000000000000000e-01 2.500000000000000000e-01
AtomId 10: 0.000000000000000000e+00 7.500000000000000000e-01 1.666666666670000019e-01
AtomId 11: 0.000000000000000000e+00 5.000000000000000000e-01 5.000000000000000000e-01
AtomId 12: 2.500000000000000000e-01 7.500000000000000000e-01 9.166666666670000296e-01
AtomId 13: 2.500000000000000000e-01 5.000000000000000000e-01 7.500000000000000000e-01
AtomId 14: 0.000000000000000000e+00 7.500000000000000000e-01 6.666666666670000296e-01
AtomId 15: 5.000000000000000000e-01 0.000000000000000000e+00 0.000000000000000000e+00
AtomId 16: 7.500000000000000000e-01 2.500000000000000000e-01 4.166666666669999741e-01
AtomId 17: 7.500000000000000000e-01 0.000000000000000000e+00 2.500000000000000000e-01
AtomId 18: 5.000000000000000000e-01 2.500000000000000000e-01 1.666666666670000019e-01
AtomId 19: 5.000000000000000000e-01 0.000000000000000000e+00 5.000000000000000000e-01
AtomId 20: 7.500000000000000000e-01 2.500000000000000000e-01 9.166666666670000296e-01
AtomId 21: 7.500000000000000000e-01 0.000000000000000000e+00 7.500000000000000000e-01
AtomId 22: 5.000000000000000000e-01 2.500000000000000000e-01 6.666666666670000296e-01
AtomId 23: 5.000000000000000000e-01 5.000000000000000000e-01 0.000000000000000000e+00
AtomId 24: 7.500000000000000000e-01 7.500000000000000000e-01 4.166666666669999741e-01
AtomId 25: 7.500000000000000000e-01 5.000000000000000000e-01 2.500000000000000000e-01
AtomId 26: 5.000000000000000000e-01 7.500000000000000000e-01 1.666666666670000019e-01
AtomId 27: 5.000000000000000000e-01 5.000000000000000000e-01 5.000000000000000000e-01
AtomId 28: 7.500000000000000000e-01 7.500000000000000000e-01 9.166666666670000296e-01
AtomId 29: 7.500000000000000000e-01 5.000000000000000000e-01 7.500000000000000000e-01
AtomId 30: 5.000000000000000000e-01 7.500000000000000000e-01 6.666666666670000296e-01
-----------------------------------------------------------------------------------------
Number Image Charges 2094

Finite element mesh information
-------------------------------------------------
FE interpolating polynomial order for Kohn-Sham eigenvalue problem: 3
FE interpolating polynomial order for electrostatics solve: 3
FE interpolating polynomial order for nodal electron density computation: 5
number of elements: 1440
number of degrees of freedom for the Kohn-Sham eigenvalue problem : 52791
-------------------------------------------------

Setting initial guess for wavefunctions....

Reading initial guess for electron-density.....

Pseudopotential initalization....

Starting SCF iterations....
SCF iterations converged to the specified tolerance after: 26 iterations.

Energy computations (Hartree)
-------------------
Total energy: -1666.902177
44 changes: 44 additions & 0 deletions testsGPU/pseudopotential/real/accuracyBenchmarks/outputN2_5
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
number of atoms: 2
number of atoms types: 1
Z:7
=============================================================================================================================
number of electrons: 10
number of eigen values: 8
=============================================================================================================================
-----------Simulation Domain bounding vectors (lattice vectors in fully periodic case)-------------
v1 : 2.000000000000000000e+01 0.000000000000000000e+00 0.000000000000000000e+00
v2 : 0.000000000000000000e+00 2.000000000000000000e+01 0.000000000000000000e+00
v3 : 0.000000000000000000e+00 0.000000000000000000e+00 2.000000000000000000e+01
-----------------------------------------------------------------------------------------
------------Cartesian coordinates of atoms (origin at center of domain)------------------
AtomId 0: -1.300000000000000044e+00 0.000000000000000000e+00 0.000000000000000000e+00
AtomId 1: 1.300000000000000044e+00 0.000000000000000000e+00 0.000000000000000000e+00
-----------------------------------------------------------------------------------------

Finite element mesh information
-------------------------------------------------
FE interpolating polynomial order for Kohn-Sham eigenvalue problem: 2
FE interpolating polynomial order for electrostatics solve: 2
FE interpolating polynomial order for nodal electron density computation: 4
number of elements: 400
number of degrees of freedom for the Kohn-Sham eigenvalue problem : 4529
-------------------------------------------------

Setting initial guess for wavefunctions....

Reading initial guess for electron-density.....

Pseudopotential initalization....

Starting SCF iterations....
SCF iterations converged to the specified tolerance after: 9 iterations.

Energy computations (Hartree)
-------------------
Total energy: -19.66693754

Absolute values of ion forces (Hartree/Bohr)
--------------------------------------------------------------------------------------------
AtomId 0: 0.323342,0.000000,0.000000
AtomId 1: 0.323342,0.000000,0.000000
--------------------------------------------------------------------------------------------
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,12 @@ srun -n 6 -c 1 ./dftfe parameterFileN2_1.prm > outputN2_1
srun -n 6 -c 1 ./dftfe parameterFileN2_2.prm > outputN2_2
srun -n 6 -c 1 ./dftfe parameterFileN2_3.prm > outputN2_3
srun -n 6 -c 1 ./dftfe parameterFileN2_4.prm > outputN2_4
srun -n 6 -c 1 ./dftfe parameterFileN2_5.prm > outputN2_5
srun -n 6 -c 1 ./dftfe parameterFileMg2x_8.prm > outputMg2x_8
srun -n 6 -c 1 ./dftfe parameterFileMg2x_9.prm > outputMg2x_9
srun -n 6 -c 1 ./dftfe parameterFileMg2x_10.prm > outputMg2x_10
srun -n 6 -c 1 ./dftfe parameterFileMg2x_11.prm > outputMg2x_11
srun -n 6 -c 1 ./dftfe parameterFileMg2x_14.prm > outputMg2x_14
srun -n 6 -c 1 ./dftfe parameterFileMg2x_15.prm > outputMg2x_15
srun -n 6 -c 1 ./dftfe parameterFileMg2x_16.prm > outputMg2x_16

Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,11 @@ srun -n 6 -c 7 --gpu-bind closest $BASE/dftfe parameterFileN2_1.prm > outputN2_1
srun -n 6 -c 7 --gpu-bind closest $BASE/dftfe parameterFileN2_2.prm > outputN2_2
srun -n 6 -c 7 --gpu-bind closest $BASE/dftfe parameterFileN2_3.prm > outputN2_3
srun -n 6 -c 7 --gpu-bind closest $BASE/dftfe parameterFileN2_4.prm > outputN2_4
srun -n 6 -c 7 --gpu-bind closest $BASE/dftfe parameterFileN2_5.prm > outputN2_5
srun -n 6 -c 7 --gpu-bind closest $BASE/dftfe parameterFileMg2x_8.prm > outputMg2x_8
srun -n 6 -c 7 --gpu-bind closest $BASE/dftfe parameterFileMg2x_9.prm > outputMg2x_9
srun -n 6 -c 7 --gpu-bind closest $BASE/dftfe parameterFileMg2x_10.prm > outputMg2x_10
srun -n 6 -c 7 --gpu-bind closest $BASE/dftfe parameterFileMg2x_11.prm > outputMg2x_11
srun -n 6 -c 7 --gpu-bind closest $BASE/dftfe parameterFileMg2x_14.prm > outputMg2x_14
srun -n 6 -c 7 --gpu-bind closest $BASE/dftfe parameterFileMg2x_15.prm > outputMg2x_15
srun -n 6 -c 7 --gpu-bind closest $BASE/dftfe parameterFileMg2x_16.prm > outputMg2x_16
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,12 @@ srun -n 6 --mpi=pmi2 ./dftfe parameterFileN2_1.prm > outputN2_1
srun -n 6 --mpi=pmi2 ./dftfe parameterFileN2_2.prm > outputN2_2
srun -n 6 --mpi=pmi2 ./dftfe parameterFileN2_3.prm > outputN2_3
srun -n 6 --mpi=pmi2 ./dftfe parameterFileN2_4.prm > outputN2_4
srun -n 6 --mpi=pmi2 ./dftfe parameterFileN2_5.prm > outputN2_5
srun -n 6 --mpi=pmi2 ./dftfe parameterFileMg2x_8.prm > outputMg2x_8
srun -n 6 --mpi=pmi2 ./dftfe parameterFileMg2x_9.prm > outputMg2x_9
srun -n 6 --mpi=pmi2 ./dftfe parameterFileMg2x_10.prm > outputMg2x_10
srun -n 6 --mpi=pmi2 ./dftfe parameterFileMg2x_11.prm > outputMg2x_11
srun -n 6 --mpi=pmi2 ./dftfe parameterFileMg2x_14.prm > outputMg2x_14
srun -n 6 --mpi=pmi2 ./dftfe parameterFileMg2x_15.prm > outputMg2x_15
srun -n 6 --mpi=pmi2 ./dftfe parameterFileMg2x_16.prm > outputMg2x_16

Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,11 @@ srun ./dftfe parameterFileN2_1.prm > outputN2_1
srun ./dftfe parameterFileN2_2.prm > outputN2_2
srun ./dftfe parameterFileN2_3.prm > outputN2_3
srun ./dftfe parameterFileN2_4.prm > outputN2_4
srun ./dftfe parameterFileN2_5.prm > outputN2_5
srun ./dftfe parameterFileMg2x_8.prm > outputMg2x_8
srun ./dftfe parameterFileMg2x_9.prm > outputMg2x_9
srun ./dftfe parameterFileMg2x_10.prm > outputMg2x_10
srun ./dftfe parameterFileMg2x_11.prm > outputMg2x_11
srun ./dftfe parameterFileMg2x_14.prm > outputMg2x_14
srun ./dftfe parameterFileMg2x_15.prm > outputMg2x_15
srun ./dftfe parameterFileMg2x_16.prm > outputMg2x_16
3 changes: 3 additions & 0 deletions testsGPU/pseudopotential/real/jobscripts/summit.lsf
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,14 @@ jsrun --smpiargs "-gpu" -n 6 -a 1 -c 7 -g 1 -r 6 -d packed -b packed:7 ./dftfe p
jsrun --smpiargs "-gpu" -n 6 -a 1 -c 7 -g 1 -r 6 -d packed -b packed:7 ./dftfe parameterFileN2_2.prm > outputN2_2
jsrun --smpiargs "-gpu" -n 6 -a 1 -c 7 -g 1 -r 6 -d packed -b packed:7 ./dftfe parameterFileN2_3.prm > outputN2_3
jsrun --smpiargs "-gpu" -n 6 -a 1 -c 7 -g 1 -r 6 -d packed -b packed:7 ./dftfe parameterFileN2_4.prm > outputN2_4
jsrun --smpiargs "-gpu" -n 6 -a 1 -c 7 -g 1 -r 6 -d packed -b packed:7 ./dftfe parameterFileN2_5.prm > outputN2_5
jsrun --smpiargs "-gpu" -n 6 -a 1 -c 7 -g 1 -r 6 -d packed -b packed:7 ./dftfe parameterFileMg2x_8.prm > outputMg2x_8
jsrun --smpiargs "-gpu" -n 6 -a 1 -c 7 -g 1 -r 6 -d packed -b packed:7 ./dftfe parameterFileMg2x_9.prm > outputMg2x_9
jsrun --smpiargs "-gpu" -n 6 -a 1 -c 7 -g 1 -r 6 -d packed -b packed:7 ./dftfe parameterFileMg2x_10.prm > outputMg2x_10
jsrun --smpiargs "-gpu" -n 6 -a 1 -c 7 -g 1 -r 6 -d packed -b packed:7 ./dftfe parameterFileMg2x_11.prm > outputMg2x_11
jsrun --smpiargs "-gpu" -n 6 -a 1 -c 7 -g 1 -r 6 -d packed -b packed:7 ./dftfe parameterFileMg2x_14.prm > outputMg2x_14
jsrun --smpiargs "-gpu" -n 6 -a 1 -c 7 -g 1 -r 6 -d packed -b packed:7 ./dftfe parameterFileMg2x_15.prm > outputMg2x_15
jsrun --smpiargs "-gpu" -n 6 -a 1 -c 7 -g 1 -r 6 -d packed -b packed:7 ./dftfe parameterFileMg2x_16.prm > outputMg2x_16

jsrun --smpiargs "-gpu" -n 6 -a 3 -c 7 -g 1 -r 6 -d packed -b packed:2 ./dftfe parameterFileMg2x_1.prm > outputMg2x_1
jsrun --smpiargs "-gpu" -n 6 -a 3 -c 7 -g 1 -r 6 -d packed -b packed:2 ./dftfe parameterFileMg2x_1_spingpu.prm > outputMg2x_1_spin_gpu
Expand Down
70 changes: 70 additions & 0 deletions testsGPU/pseudopotential/real/parameterFileMg2x_15.prm
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
set VERBOSITY = 0
set REPRODUCIBLE OUTPUT=true

subsection GPU
set USE GPU=true
set AUTO GPU BLOCK SIZES=false
end

subsection Boundary conditions
set PERIODIC1 = true
set PERIODIC2 = true
set PERIODIC3 = true
end


subsection DFT functional parameters
set EXCHANGE CORRELATION TYPE = 4
set PSEUDOPOTENTIAL CALCULATION = true
set PSEUDOPOTENTIAL FILE NAMES LIST = pseudoMg.inp
end


subsection Finite element mesh parameters
set POLYNOMIAL ORDER = 3
subsection Auto mesh generation parameters
set AUTO ADAPT BASE MESH SIZE=false
set ATOM BALL RADIUS = 2.0
set BASE MESH SIZE = 4.0
set MESH SIZE AROUND ATOM = 1.0
set MESH SIZE AT ATOM = 1.0
end
end

subsection Geometry
set NATOMS=31
set NATOM TYPES=1
set ATOMIC COORDINATES FILE = coordinatesMg2x.inp
set DOMAIN VECTORS FILE = domainVectorsMg2x.inp
subsection Optimization
set ION FORCE = true
end
end

subsection SCF parameters
set COMPUTE ENERGY EACH ITER = false
set MIXING PARAMETER =0.7
set MIXING METHOD=LOW_RANK_DIELECM_PRECOND
set MAXIMUM ITERATIONS = 100
set TEMPERATURE = 500
set TOLERANCE = 1e-6
set STARTING WFC=ATOMIC
subsection LOW RANK DIELECM PRECOND
set METHOD SUB TYPE=ADAPTIVE
set STARTING NORM LARGE DAMPING=2.0
set ADAPTIVE RANK REL TOL=0.1
set POISSON SOLVER ABS TOL=1.0e-10
end
subsection Eigen-solver parameters
set CHEBYSHEV POLYNOMIAL DEGREE = 20
set NUMBER OF KOHN-SHAM WAVEFUNCTIONS = 180
set CHEBY WFC BLOCK SIZE=45
set WFC BLOCK SIZE=45
set CHEBYSHEV FILTER TOLERANCE=1e-5
end
end
subsection Poisson problem parameters
set GPU MODE = true
set MAXIMUM ITERATIONS = 20000
set TOLERANCE = 1e-10
end
Loading

0 comments on commit 43061d7

Please sign in to comment.