SPHInX

Dear Noam,

I have tried to understand what might have happened, but I can only offer speculation.

According to the output, everything looks perfectly fine until the last scf iteration, and then within that, the loop that optimize the Lagrangian multipliers (nu) look ok until the line where Gamma jumos to 5.66314e+08. This is a Conjugate Gradient; kappa is the trial step length for the line minimization and kappa opt the optimized step length. The "gradient" for the search direction is the deviation in atomic spins, and gamma is the ratio of the new to previous gradient (squared). That should not be able to jump by 8 orders of magnitude upon a small step in the previous iteration.

I have the suspicion that this results from a synchronization error between the MPI tasks, that can happen if numerical noise accumulates for (conceptually global) quantities that are computed independently for each MPI tasks. What then happens is that the MPI tasks disagree whether the iterations have converged, and they no longer work on the same part of the code. Then some MPI tasks send completely different data, the eigenvalues pick up random values, and the Fermi-level searching algorithm fails to find a reasonable value, and you have the symptoms I see in the log file.

One frequent symptom of MPI mismatch is that the behavior is not reproducible, i.e., the same calculation stops at different moments, or runs in some cases and stops in others. One could also try to set the environment SX_LOG_STDOUT=ALL - then all MPI tasks produce output (tasks 1 - N to hidden files .stdoutDump_<i>, and one can compare if they show different output in the end that shows that some tasks continue with the next step, while others continue to iterate the nu's. Also, if affordable, a non-MPI run should not show such issues.

I had similar things long time ago with the scf iterations, and I ended up broadcasting the convergence decision. Before I implement a corresponding fix for the nu optimization loop, could you try to check if my suspicion is true by rerunning the same calculation with SX_LOG_STDOUT=ALL, and check if the final lines of output show any signs of discrepancies?

If that is the case, I apologize for the inconvenience. MPI parallelization was implemented in the second round, using simple test cases to ensure that I can run them in serial. These simple test cases may be more well-behaved.

I don't see any differences in the last 100 or so lines (although there are differences near machine precision level in some quantities earlier in the process). I've uploaded the output files. One thing I also have, which I'll inspect, is the corresponding set of files from a run of supposedly the same inputs that didn't blow up. If I see anything interesting I'll upload those as well.