large sloped ceiling - efficient and robust modelling? #13034

stephanB79 · 2024-06-07T13:09:54Z

stephanB79
Jun 7, 2024

Hello, I´m trying to model a large, sloped ceiling in a building (60 x 20 x 20m). The challenge is as follows:

For modelling the slope, I cannot use "Repeated Objects: The MULTNamelist Group" as the items which are placed inside the building are using unstructured Geometry &GEOM. Modelling these items without the sloped ceiling works great. The grid cell site is 0.125m which results in 12 Mio Cells with approx. 16 GB of RAM usage. Such a "fine" grid is required to model light gas dispersion.

For modelling the sloped ceiling, I used "Geometries Extruded From Simple Planar Polygon". This works great for the case in setup only mode T_END=0.0. Upon changing T_END to calculate time steps, the RAM usage explodes to approx. 128GB (which is my current limit) even with zero unstructured geometry placed inside. I tried many configurations for the extruded geometries, but none is solving this issue.

Together with the unstructured geometry, which is located at the floor, the simulation reports an error prior to the 1st timestep at a RAM usage of 104GB:

ULMAT_H_MATRIX_LUDCMP PARDISO Num Factor: The following ERROR was detected: -2
Insufficient Memory for Poisson Matrix Factorization.
ERROR: FDS was improperly set-up - FDS stopped (CHID: hydrogendis)

Are there any ideas for a more efficient, more robust or better way for modelling of the sloped ceiling?

I have attached the fds-code of the test case, in case it is needed. I´m running it with fds_local -p 12 -o 1.

Thanks in advance!

Stephan

__&HEAD CHID='sloped_roof',TITLE='testfile'
&TIME T_END=0.1/

#12,5cm grid, overall dimensions: length = 60m, width = 20m, height = 20m
&MESH ID='Mesh 1' ,IJK=80, 80, 160, XB=0, 10, 0, 10, 0, 20/ 1.024.000 Cells
&MESH ID='Mesh 2' ,IJK=80, 80, 160, XB=0, 10, 10, 20, 0, 20/ 1.024.000 Cells
&MESH ID='Mesh 3' ,IJK=80, 80, 160, XB=10, 20, 0, 10, 0, 20/ 1.024.000 Cells
&MESH ID='Mesh 4' ,IJK=80, 80, 160, XB=10, 20, 10, 20, 0, 20/ 1.024.000 Cells
&MESH ID='Mesh 5' ,IJK=80, 80, 160, XB=20, 30, 0, 10, 0, 20/ 1.024.000 Cells
&MESH ID='Mesh 6' ,IJK=80, 80, 160, XB=20, 30, 10, 20, 0, 20/ 1.024.000 Cells
&MESH ID='Mesh 7' ,IJK=80, 80, 160, XB=30, 40, 0, 10, 0, 20/ 1.024.000 Cells
&MESH ID='Mesh 8' ,IJK=80, 80, 160, XB=30, 40, 10, 20, 0, 20/ 1.024.000 Cells
&MESH ID='Mesh 9' ,IJK=80, 80, 160, XB=40, 50, 0, 10, 0, 20/ 1.024.000 Cells
&MESH ID='Mesh 10' ,IJK=80, 80, 160, XB=40, 50, 10, 20, 0, 20/ 1.024.000 Cells
&MESH ID='Mesh 11' ,IJK=80, 80, 160, XB=50, 60, 0, 10, 0, 20/ 1.024.000 Cells
&MESH ID='Mesh 12' ,IJK=80, 80, 160, XB=50, 60, 10, 20, 0, 20/ 1.024.000 Cells

#sloped roof
&GEOM ID='Roof1', SURF_ID='Poly1',
VERTS =
0, 0, 18,
60, 0, 18,
0, 9, 19.8,
60, 9, 19.8,
POLY = 1, 2, 4, 3,
EXTRUDE=0.2/
&SURF ID='Poly1', COLOR='STEEL BLUE'/

&GEOM ID='Roof2', SURF_ID='Poly2',
VERTS =
0, 11, 19.8,
60, 11, 19.8,
0, 20, 18,_
60,20, 18,
POLY = 1, 2, 4, 3,
EXTRUDE=0.2 /
&SURF ID='Poly2', COLOR='STEEL BLUE'/

#not used:
#CELL_BLOCK_IOR= 3,

&TAIL /_

Answered by marcosvanella

Jun 10, 2024

Hi Stephan, I assume you are still using the single workstation? Also, is the memory usage you report the total memory usage for the run (as what you would gather using the top command in linux)?
Please do me a favor, comment the geom lines and run your case with the line:
&PRES SOLVER='ULMAT'/
And run both sets of meshes, also reporting the memory usage. We are thinking of moving to a different library for unstructured solvers that would provide a lower memory intensive option. But this will take some time as re are tied up for the next few weeks.
In terms of using GEOM, using the extruding polygon constructor should not make much difference memory-wise respect to just loading them from …

View full answer

drjfloyd · 2024-06-07T13:56:22Z

drjfloyd
Jun 7, 2024
Collaborator

GEOM requires the use of the optional pressure solvers which are memory intensive as they are full-matrix inversions rather than FFT. Target ~150,000 cells per mesh (See 21.1.1 in the User's Guide, GEOM invokes ULMAT).

0 replies

marcosvanella · 2024-06-07T14:04:45Z

marcosvanella
Jun 7, 2024
Collaborator

Hi Stephan, as Jason commented, the high memory usage is related to GEOM using ULMAT as Poisson solver. This is because we use an unstructured solver for geometries. ULMAT does a Cholesky factorization per mesh, and the factorization matrix uses a substantial amount of memory (worst case, scales like the square of the number of cells). In your case you have a million cells per mesh, you want to be using ~50^3 cells per mesh to make memory usage manageable.

…

________________________________ From: stephanB79 ***@***.***> Sent: Friday, June 7, 2024 09:10 AM To: firemodels/fds ***@***.***> Cc: Subscribed ***@***.***> Subject: [firemodels/fds] large sloped ceiling - efficient and robust modelling? (Discussion #13034) Hello, I´m trying to model a large, sloped ceiling in a building (60 x 20 x 20m). The challenge is as follows: For modelling the slope, I cannot use "Repeated Objects: The MULTNamelist Group" as the items which are placed inside the building are using unstructured Geometry &GEOM. Modelling these items without the sloped ceiling works great. The grid cell site is 0.125m which results in 12 Mio Cells with approx. 16 GB of RAM usage. Such a "fine" grid is required to model light gas dispersion. For modelling the sloped ceiling, I used "Geometries Extruded From Simple Planar Polygon". This works great for the case in setup only mode T_END=0.0. Upon changing T_END to calculate time steps, the RAM usage explodes to approx. 128GB (which is my current limit) even with zero unstructured geometry placed inside. I tried many configurations for the extruded geometries, but none is solving this issue. sloped.ceiling.png (view on web)<https://github.com/firemodels/fds/assets/172036230/1ddb8f07-6bcf-4f80-b414-ff2c7a9a9a2b> Together with the unstructured geometry, which is located at the floor, the simulation reports an error prior to the 1st timestep at a RAM usage of 104GB: ULMAT_H_MATRIX_LUDCMP PARDISO Num Factor: The following ERROR was detected: -2 Insufficient Memory for Poisson Matrix Factorization. ERROR: FDS was improperly set-up - FDS stopped (CHID: hydrogendis) Are there any ideas for a more efficient, more robust or better way for modelling of the sloped ceiling? I have attached the fds-code of the test case, in case it is needed. I´m running it with fds_local -p 12 -o 1. Thanks in advance! Stephan

__&HEAD CHID='sloped_roof',TITLE='testfile' &TIME T_END=0.1/ #12<#12>,5cm grid, overall dimensions: length = 60m, width = 20m, height = 20m &MESH ID='Mesh 1' ,IJK=80, 80, 160, XB=0, 10, 0, 10, 0, 20/ 1.024.000 Cells &MESH ID='Mesh 2' ,IJK=80, 80, 160, XB=0, 10, 10, 20, 0, 20/ 1.024.000 Cells &MESH ID='Mesh 3' ,IJK=80, 80, 160, XB=10, 20, 0, 10, 0, 20/ 1.024.000 Cells &MESH ID='Mesh 4' ,IJK=80, 80, 160, XB=10, 20, 10, 20, 0, 20/ 1.024.000 Cells &MESH ID='Mesh 5' ,IJK=80, 80, 160, XB=20, 30, 0, 10, 0, 20/ 1.024.000 Cells &MESH ID='Mesh 6' ,IJK=80, 80, 160, XB=20, 30, 10, 20, 0, 20/ 1.024.000 Cells &MESH ID='Mesh 7' ,IJK=80, 80, 160, XB=30, 40, 0, 10, 0, 20/ 1.024.000 Cells &MESH ID='Mesh 8' ,IJK=80, 80, 160, XB=30, 40, 10, 20, 0, 20/ 1.024.000 Cells &MESH ID='Mesh 9' ,IJK=80, 80, 160, XB=40, 50, 0, 10, 0, 20/ 1.024.000 Cells &MESH ID='Mesh 10' ,IJK=80, 80, 160, XB=40, 50, 10, 20, 0, 20/ 1.024.000 Cells &MESH ID='Mesh 11' ,IJK=80, 80, 160, XB=50, 60, 0, 10, 0, 20/ 1.024.000 Cells &MESH ID='Mesh 12' ,IJK=80, 80, 160, XB=50, 60, 10, 20, 0, 20/ 1.024.000 Cells #sloped roof &GEOM ID='Roof1', SURF_ID='Poly1', VERTS = 0, 0, 18, 60, 0, 18, 0, 9, 19.8, 60, 9, 19.8, POLY = 1, 2, 4, 3, EXTRUDE=0.2/ &SURF ID='Poly1', COLOR='STEEL BLUE'/ &GEOM ID='Roof2', SURF_ID='Poly2', VERTS = 0, 11, 19.8, 60, 11, 19.8, 0, 20, 18,_ 60,20, 18, POLY = 1, 2, 4, 3, EXTRUDE=0.2 / &SURF ID='Poly2', COLOR='STEEL BLUE'/ #not used: #CELL_BLOCK_IOR= 3, &TAIL /_ — Reply to this email directly, view it on GitHub<#13034>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABY23VO3LEQ6OFHGWKK2TRLZGGWLVAVCNFSM6AAAAABI6UTAYGVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZWG44TANBXG4>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

1 reply

stephanB79 Jun 10, 2024
Author

Many thanks for your quick reply. I´m now aware that the downside of GEOM is using ULMAT as Poisson solver.
For verifying the influence, I tried to reduce the number cells per mesh (by increasing the number of meshes).

Unfortunately, this does not help with high RAM usage. For a comparison it run a simulation with 128.000 cells per mesh and another one with 1.024.000 cells per mesh on a building with 30 meters length only.

6 Meshes = 1.024.000 cells per mesh = 58GB RAM usage at start of iteration
48 Meshes = 128.000 cells per mesh = 66GB RAM usage at start of iteration

I assume that the additional meshes internally create some "overhead" as there more connecting surfaces to be managed.

Next, I will
a) try to get more RAM
b) try to avoid "Geometries Extruded From Simple Planar Polygon" for the sloped ceiling
c) try to avoid GEOM completely

Any other suggestion is highly welcome. Thanks in advance!

Stephan

I case somebody wants to test, here are the FDS content:

&MESH ID='Mesh 1' ,IJK=80, 80, 160, XB=30, 40, 0, 10, 0, 20/ 1.024.000 Cells
&MESH ID='Mesh 2' ,IJK=80, 80, 160, XB=30, 40, 10, 20, 0, 20/ 1.024.000 Cells
&MESH ID='Mesh 3' ,IJK=80, 80, 160, XB=40, 50, 0, 10, 0, 20/ 1.024.000 Cells
&MESH ID='Mesh 4' ,IJK=80, 80, 160, XB=40, 50, 10, 20, 0, 20/ 1.024.000 Cells
&MESH ID='Mesh 5' ,IJK=80, 80, 160, XB=50, 60, 0, 10, 0, 20/ 1.024.000 Cells
&MESH ID='Mesh 6' ,IJK=80, 80, 160, XB=50, 60, 10, 20, 0, 20/ 1.024.000 Cells

#sloped roof
&GEOM ID='Roof1', SURF_ID='Poly1',
VERTS =
30, 0, 18,
60, 0, 18,
30, 9, 19.8,
60, 9, 19.8,
POLY = 1, 2, 4, 3,
EXTRUDE=0.2/
&SURF ID='Poly1', COLOR='STEEL BLUE'/

&GEOM ID='Roof2', SURF_ID='Poly2',
VERTS =
30, 11, 19.8,
60, 11, 19.8,
30, 20, 18,
60, 20, 18,
POLY = 1, 2, 4, 3,
EXTRUDE=0.2 /
&SURF ID='Poly2', COLOR='STEEL BLUE'/

and for 48 meshes:

&MESH ID='Mesh 1' ,IJK=80, 40, 40, XB=30, 40, 0, 5, 0, 5/ 128.000 Cells
&MESH ID='Mesh 2' ,IJK=80, 40, 40, XB=30, 40, 0, 5, 5, 10/ 128.000 Cells
&MESH ID='Mesh 3' ,IJK=80, 40, 40, XB=30, 40, 0, 5, 10, 15/ 128.000 Cells
&MESH ID='Mesh 4' ,IJK=80, 40, 40, XB=30, 40, 0, 5, 15, 20/ 128.000 Cells
&MESH ID='Mesh 5' ,IJK=80, 40, 40, XB=30, 40, 5, 10, 0, 5/ 128.000 Cells
&MESH ID='Mesh 6' ,IJK=80, 40, 40, XB=30, 40, 5, 10, 5, 10/ 128.000 Cells
&MESH ID='Mesh 7' ,IJK=80, 40, 40, XB=30, 40, 5, 10, 10, 15/ 128.000 Cells
&MESH ID='Mesh 8' ,IJK=80, 40, 40, XB=30, 40, 5, 10, 15, 20/ 128.000 Cells
&MESH ID='Mesh 9' ,IJK=80, 40, 40, XB=30, 40, 10, 15, 0, 5/ 128.000 Cells
&MESH ID='Mesh 10' ,IJK=80, 40, 40, XB=30, 40, 10, 15, 5, 10/ 128.000 Cells
&MESH ID='Mesh 11' ,IJK=80, 40, 40, XB=30, 40, 10, 15, 10, 15/ 128.000 Cells
&MESH ID='Mesh 12' ,IJK=80, 40, 40, XB=30, 40, 10, 15, 15, 20/ 128.000 Cells
&MESH ID='Mesh 13' ,IJK=80, 40, 40, XB=30, 40, 15, 20, 0, 5/ 128.000 Cells
&MESH ID='Mesh 14' ,IJK=80, 40, 40, XB=30, 40, 15, 20, 5, 10/ 128.000 Cells
&MESH ID='Mesh 15' ,IJK=80, 40, 40, XB=30, 40, 15, 20, 10, 15/ 128.000 Cells
&MESH ID='Mesh 16' ,IJK=80, 40, 40, XB=30, 40, 15, 20, 15, 20/ 128.000 Cells
&MESH ID='Mesh 17' ,IJK=80, 40, 40, XB=40, 50, 0, 5, 0, 5/ 128.000 Cells
&MESH ID='Mesh 18' ,IJK=80, 40, 40, XB=40, 50, 0, 5, 5, 10/ 128.000 Cells
&MESH ID='Mesh 19' ,IJK=80, 40, 40, XB=40, 50, 0, 5, 10, 15/ 128.000 Cells
&MESH ID='Mesh 20' ,IJK=80, 40, 40, XB=40, 50, 0, 5, 15, 20/ 128.000 Cells
&MESH ID='Mesh 21' ,IJK=80, 40, 40, XB=40, 50, 5, 10, 0, 5/ 128.000 Cells
&MESH ID='Mesh 22' ,IJK=80, 40, 40, XB=40, 50, 5, 10, 5, 10/ 128.000 Cells
&MESH ID='Mesh 23' ,IJK=80, 40, 40, XB=40, 50, 5, 10, 10, 15/ 128.000 Cells
&MESH ID='Mesh 24' ,IJK=80, 40, 40, XB=40, 50, 5, 10, 15, 20/ 128.000 Cells
&MESH ID='Mesh 25' ,IJK=80, 40, 40, XB=40, 50, 10, 15, 0, 5/ 128.000 Cells
&MESH ID='Mesh 26' ,IJK=80, 40, 40, XB=40, 50, 10, 15, 5, 10/ 128.000 Cells
&MESH ID='Mesh 27' ,IJK=80, 40, 40, XB=40, 50, 10, 15, 10, 15/ 128.000 Cells
&MESH ID='Mesh 28' ,IJK=80, 40, 40, XB=40, 50, 10, 15, 15, 20/ 128.000 Cells
&MESH ID='Mesh 29' ,IJK=80, 40, 40, XB=40, 50, 15, 20, 0, 5/ 128.000 Cells
&MESH ID='Mesh 30' ,IJK=80, 40, 40, XB=40, 50, 15, 20, 5, 10/ 128.000 Cells
&MESH ID='Mesh 31' ,IJK=80, 40, 40, XB=40, 50, 15, 20, 10, 15/ 128.000 Cells
&MESH ID='Mesh 32' ,IJK=80, 40, 40, XB=40, 50, 15, 20, 15, 20/ 128.000 Cells
&MESH ID='Mesh 33' ,IJK=80, 40, 40, XB=50, 60, 0, 5, 0, 5/ 128.000 Cells
&MESH ID='Mesh 34' ,IJK=80, 40, 40, XB=50, 60, 0, 5, 5, 10/ 128.000 Cells
&MESH ID='Mesh 35' ,IJK=80, 40, 40, XB=50, 60, 0, 5, 10, 15/ 128.000 Cells
&MESH ID='Mesh 36' ,IJK=80, 40, 40, XB=50, 60, 0, 5, 15, 20/ 128.000 Cells
&MESH ID='Mesh 37' ,IJK=80, 40, 40, XB=50, 60, 5, 10, 0, 5/ 128.000 Cells
&MESH ID='Mesh 38' ,IJK=80, 40, 40, XB=50, 60, 5, 10, 5, 10/ 128.000 Cells
&MESH ID='Mesh 39' ,IJK=80, 40, 40, XB=50, 60, 5, 10, 10, 15/ 128.000 Cells
&MESH ID='Mesh 40' ,IJK=80, 40, 40, XB=50, 60, 5, 10, 15, 20/ 128.000 Cells
&MESH ID='Mesh 41' ,IJK=80, 40, 40, XB=50, 60, 10, 15, 0, 5/ 128.000 Cells
&MESH ID='Mesh 42' ,IJK=80, 40, 40, XB=50, 60, 10, 15, 5, 10/ 128.000 Cells
&MESH ID='Mesh 43' ,IJK=80, 40, 40, XB=50, 60, 10, 15, 10, 15/ 128.000 Cells
&MESH ID='Mesh 44' ,IJK=80, 40, 40, XB=50, 60, 10, 15, 15, 20/ 128.000 Cells
&MESH ID='Mesh 45' ,IJK=80, 40, 40, XB=50, 60, 15, 20, 0, 5/ 128.000 Cells
&MESH ID='Mesh 46' ,IJK=80, 40, 40, XB=50, 60, 15, 20, 5, 10/ 128.000 Cells
&MESH ID='Mesh 47' ,IJK=80, 40, 40, XB=50, 60, 15, 20, 10, 15/ 128.000 Cells
&MESH ID='Mesh 48' ,IJK=80, 40, 40, XB=50, 60, 15, 20, 15, 20/ 128.000 Cells

marcosvanella · 2024-06-10T15:25:58Z

marcosvanella
Jun 10, 2024
Collaborator

Hi Stephan, I assume you are still using the single workstation? Also, is the memory usage you report the total memory usage for the run (as what you would gather using the top command in linux)?
Please do me a favor, comment the geom lines and run your case with the line:
&PRES SOLVER='ULMAT'/
And run both sets of meshes, also reporting the memory usage. We are thinking of moving to a different library for unstructured solvers that would provide a lower memory intensive option. But this will take some time as re are tied up for the next few weeks.
In terms of using GEOM, using the extruding polygon constructor should not make much difference memory-wise respect to just loading them from BlenderFDS or Pyrosim.

1 reply

stephanB79 Jun 11, 2024
Author

Hello Marcos,

yes, I´m currently still using a single windows workstation with FDS 6.9.
the reported RAM usage is the total of the run. FDS is the only application on the workstation and from the peak usage at start of run I subtract the base usage which is identical before and after running FDS. As I'm not close to memory limit, this should approximately be identical to top command in linux.
my current workaround is to manually "hardcode" the sloped ceiling to the mesh, which works for me for now. Nevertheless, reducing the RAM usage for ULMAT would be a great additional feature.
I run the comparison as proposed by you with following results. Hopefully these numbers make sense?:

marcosvanella · 2024-06-11T14:18:35Z

marcosvanella
Jun 11, 2024
Collaborator

Thank you Stephan, so, when running with 48 meshes, how many meshes per mpi process are you using and how many cpu cores do you have?
As you noted previously, seems what you gain reducing the memory of the ULMAT factorization matrix is now spent in other data structures associated to mesh boundaries and mesh exchanges. When using 1 million cell meshes switching on ULMAT was increasing the memory requirement by 45-9=36 GB, using more (smaller) meshes and ULMAT increases your memory usage by 45-27=18 GB.
GEOM increases the usage substantially also, this is related to data structures used to store cut-cell and boundary face information. It is more so for more meshes as neighbor mesh information is also required. I think there is fat to trim here.
It would be informative to know what is the memory increase when using a voxelized roof (with OBSTs) and compare. I you are planning on doing this let me know.

2 replies

stephanB79 Jun 12, 2024
Author

Hello Marcos,

the 48 meshes were for scientific testing only, as the current workstation has 24 cores / 36 threads only. Nevertheless for the reported RAM usage it should not make a difference as FDS does not know that the cores are not existing.
I assigned the 48 meshes to 48 mpi processes fds_local -p 48 -o 1 ... this is slow but it was only for checking the RAM usage
my current roof modelling does not include obstructions. Instead it really defines the grid stepwise - which is most likely not smart, using:

&MESH ID='Mesh A4' ,IJK=480, 5, 3, XB=-10, 50, -14.125, -13.5, 18, 18.375, MPI_PROCESS=13/ 7200.000 Cells
&MESH ID='Mesh A5' ,IJK=480, 5, 4, XB=-10, 50, -13.5, -12.875, 18, 18.5, MPI_PROCESS=13/ 9600.000 Cells
&MESH ID='Mesh A6' ,IJK=480, 5, 5, XB=-10, 50, -12.875, -12.25, 18, 18.625, MPI_PROCESS=13/ 12000.000 Cells
etc....

mid of next week I can try an alternative using a voxelized roof (with OBSTs) following the test scheme above. I will let you know the results.

Thanks so far!

marcosvanella Jun 12, 2024
Collaborator

Ok, sounds good. Do you have access to a computing cluster?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

large sloped ceiling - efficient and robust modelling? #13034

{{title}}

Replies: 4 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

large sloped ceiling - efficient and robust modelling? #13034

stephanB79 Jun 7, 2024

Replies: 4 comments · 4 replies

drjfloyd Jun 7, 2024 Collaborator

marcosvanella Jun 7, 2024 Collaborator

stephanB79 Jun 10, 2024 Author

marcosvanella Jun 10, 2024 Collaborator

stephanB79 Jun 11, 2024 Author

marcosvanella Jun 11, 2024 Collaborator

stephanB79 Jun 12, 2024 Author

marcosvanella Jun 12, 2024 Collaborator

stephanB79
Jun 7, 2024

Replies: 4 comments 4 replies

drjfloyd
Jun 7, 2024
Collaborator

marcosvanella
Jun 7, 2024
Collaborator

stephanB79 Jun 10, 2024
Author

marcosvanella
Jun 10, 2024
Collaborator

stephanB79 Jun 11, 2024
Author

marcosvanella
Jun 11, 2024
Collaborator

stephanB79 Jun 12, 2024
Author

marcosvanella Jun 12, 2024
Collaborator