Simulation crashes on a second computer whereas she runs on a first

Something strange in your ENVI-met? Please post here...
Post Reply
nathphil
Posts: 22
Joined: Wed Oct 02, 2019 7:58 am

Simulation crashes on a second computer whereas she runs on a first

Post by nathphil »

Hi ENVI-Met team,

I've a strange issue with one of my simulation files. I've run a simulation in April on a first PC. The simulation is indeed very, very slow (I think it is not surprising given the size of the huge area and its high resolution...), but it is still working without any problem for the moment.

In order to check if it was possible to go a little faster, I've run exactly the same simulation with the same area on a second computer, whose processor is supposed to be more efficient (Intel I9 vs intel I7 for the first PC). However, the simulation crashes straight away after the initialisation phase ("the temperature,the humidity or the co2 field has become unstable"). Considering that this type of errors can occur randomly, I've rerun three times the simulation, with no success. Every time, it was another grid which was concerned by the instability (neither captured grid nor particular grid, just some ordinary grid that is lost inside a building). All this part concerned the 4.4.4 version.

After having installed the 4.4.5 version of ENVI-Met, I've run the simulation once again. It worked during one hour of simulation and then unfortunately crashed again (started at 6 am and stopped at 7.15 am). But this time, the concerned grid was right in the empty area !

Do you think that it could be a randomly error ? I'm aware that this simumation with a so pretty huge area is "ambitious", but I don't understand why it is working on a PC and not on another, which is supposed to be more powerful and with the same RAM (64 GB)...

In case of need, here are the concerned files : https://we.tl/t-OWXHjbMzHc

Thank you very much in advance for your reply and your patience.

Best regards,

Nathalia

Online
Tim
Posts: 2042
Joined: Mon May 15, 2017 3:46 pm

Re: Simulation crashes on a second computer whereas she runs on a first

Post by Tim »

Hi Nathalia,

Actually we are very much into that topic right now and made a huge amount of test simulations within the last months. Several times we were at the point to say that we finally fixed it since no more of those random crashes occurred. However, other simulations on other PCs then could not show those positive results.
I don't think that it is connected to your model area but I cannot tell you what reason else it might be.
However, we always thought about issues in the code or the compiler but it could also be connected to the processor as you mentioned. We e.g. had several issues on an AMD processor. We will further investigate that now! Thank you very much for the hint.

Best regards,
Tim

weatherpark
Posts: 15
Joined: Thu Sep 20, 2018 9:12 am

Re: Simulation crashes on a second computer whereas she runs on a first

Post by weatherpark »

Dear Envimet Team!

unfortunately we are experiencing a similar problem with one of our simulations.
It ran on one computer without any problems.
However, the same simulation (identical .sim & .inx, .fox file) crashed on two other computers with the error messages "one ore more errors occured" several times.
It crashed each time at different simulation points (after 3, 5, ..or after >12 hours of simulation).
One example of the error message
error.PNG
The processor is the same on all three computers that we tested.
Here are the simulation files that we use
envimet_forum.zip
(710.79 KiB) Downloaded 1 time
Do you have any ideas/new findings about this problem?

As we want to start a similar simulation with another scenario (buildings instead of a field in parts of the simulation area) we fear that this simulation will crash also randomly and we do not know how we can avoid this.

Thank you in advance.

Isabel

Online
Tim
Posts: 2042
Joined: Mon May 15, 2017 3:46 pm

Re: Simulation crashes on a second computer whereas she runs on a first

Post by Tim »

Hi Isabel,

We indeed could have found a solution in the meantime but are not sure if it applies to your situation as well. Since the crashes seem to appear randomly in your case and don't have any meaningful content, I would be very sure that is not connected to your simulation files. The wind however is very low during several hours of your simulation and forcing without humidity as well as radiation/clouds might give you rather rough results but it should definitely not cause that error.

So, now about this solution (hopefully):
Do you use AMD processors? We have some and experienced those crashes there as well. Then, we disabled SMT mode in the BIOS settings and the crashes did not appear anymore. Hopefully, that applies to your computers as well!

Best regards,
Tim

weatherpark
Posts: 15
Joined: Thu Sep 20, 2018 9:12 am

Re: Simulation crashes on a second computer whereas she runs on a first

Post by weatherpark »

Hi Tim,

the wind is indeed quite low at several hours. However, the crashes always happened in the first 13 hours of simulation. And in these hours the wind speed is always above 2 m/s. And most of the time it crashed around 10 / 11 a.m when the wind speed is above 3 m/s according to the forcing file.

Unfortunately we are not using AMD processors. The three workstations that we used for this simulation all have the Intel Core i9-9900K processor.

Do you have also experiences with intel processors?

Best regars,
Isabel

Online
Tim
Posts: 2042
Joined: Mon May 15, 2017 3:46 pm

Re: Simulation crashes on a second computer whereas she runs on a first

Post by Tim »

Hi Isabel,

Okay, unfortunately we have not tested it for Intel processors, yet. In a quick search, I found that the SMT mode of AMD processors can roughly be compared with the Hyperthreading (HT) of Intel processors. Maybe disabling that on your simulation PCs also helps to stabilize the simulation.

Since we cannot be 100 % sure that it is not connected to your model area, you might also modify some things there and see if that has a positive effect: You placed a lot of high dense trees which feature a high LAD. I would reduce the amount of trees in total and replace them with a low LAD tree (the high LAD's are rather unrealistic).

Best regards,
Tim

weatherpark
Posts: 15
Joined: Thu Sep 20, 2018 9:12 am

Re: Simulation crashes on a second computer whereas she runs on a first

Post by weatherpark »

Hi Tim,

ok, we will try disabling HT.

I would really like to avoid making changes in the model area. Is it really possible that this could be the problem even if one simulation already run through smoothly (and the results look fine)?

Best regards,
Isabel

Online
Tim
Posts: 2042
Joined: Mon May 15, 2017 3:46 pm

Re: Simulation crashes on a second computer whereas she runs on a first

Post by Tim »

Hi Isabel,

No, if one simulation already finished without any errors, it is most probably not a reason for the crashes.

Best regards,
Tim

Post Reply