Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AmberFF_test fails intermittently #584

Open
dstoeckel opened this issue Apr 28, 2016 · 4 comments
Open

AmberFF_test fails intermittently #584

dstoeckel opened this issue Apr 28, 2016 · 4 comments
Labels
C: BALL Core T: defect

Comments

@dstoeckel
Copy link
Contributor

The AmberFF_test sometimes fails with the following error message (the numbers are always the same):

checking [EXTRA] Additivity of energies w/ selection... 
    (line 511 TEST_REAL_EQUAL(r4_r1 - r4_i, r1_r4 - r1_i): got -145.471, expected -103.957)  - 
    (line 513 TEST_REAL_EQUAL(r1_r4 - r1_i + r1_tpl + r4_tpl + tpl_i, total_energy): got 1680.36, expected 1638.84)  - 
    FAILED

This hints at some non-determinism in the Amber code, that might be triggered by something like pointers in a HashMap, etc.

Tested on Ubuntu 14.04.
@tkemmer
Copy link
Contributor

tkemmer commented Apr 28, 2016

I had the same problem yesterday under Gentoo (same numbers):

checking [EXTRA] Additivity of energies w/ selection... 
    (line 511 TEST_REAL_EQUAL(r4_r1 - r4_i, r1_r4 - r1_i): got -145.471, expected -103.957)  - 
    (line 513 TEST_REAL_EQUAL(r1_r4 - r1_i + r1_tpl + r4_tpl + tpl_i, total_energy): got 1680.36, expected 1638.84)  - 
    FAILED
FAILED

So far, this only happened once and only with my current working copy of the master branch (incremental build processes), not with clean builds. The latter might be a coincidence, though.

@anhi
Copy link
Member

anhi commented Apr 29, 2016

I think this is a rather bad one – not sure yet, but I believe that it is an effect of the setup-handling. Selection is handled differently based on whether it was switched on before or after the call to setup. Calling select before setup will remove all non-selected atoms from the force field entirely. Calling it after setup will compute the energies and forces only on selected atoms, but based also on non-selected atoms. This mechanism is, unfortunately, based on time stamps. If the clock rewinds after calling setup, the logic screws up. We need to fix this, but I don’t think it is a regression, so should not block 1.5.

Von: Thomas Kemmer <[email protected]mailto:[email protected]>
Antworten an: BALL-Project/ball <[email protected]mailto:[email protected]>
Datum: Donnerstag, 28. April 2016 um 18:02
An: BALL-Project/ball <[email protected]mailto:[email protected]>
Betreff: Re: [BALL-Project/ball] AmberFF_test fails intermittently (#584)

I had the same problem yesterday under Gentoo (same numbers):

checking [EXTRA] Additivity of energies w/ selection...
(line 511 TEST_REAL_EQUAL(r4_r1 - r4_i, r1_r4 - r1_i): got -145.471, expected -103.957) -
(line 513 TEST_REAL_EQUAL(r1_r4 - r1_i + r1_tpl + r4_tpl + tpl_i, total_energy): got 1680.36, expected 1638.84) -
FAILED
FAILED

So far, this only happened once and only with my current working copy of the master branch (incremental build processes), not with clean builds. The latter might be a coincidence, though.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHubhttps://github.com//issues/584#issuecomment-215477776

@dstoeckel
Copy link
Contributor Author

Nevertheless we should probably fix the test. I am not sure if this is an issue on the CI servers, but I imagine that randomly failing builds are no fun.

If it is really a timestamp issue we could add short sleeps between select and setup to decrease the likelyhood of things breaking. In Amber we could check the timestamps for non-monotonicity and at least issue a warning.

philthiel added a commit that referenced this issue Jan 20, 2017
@philthiel
Fix for AmberFF_test (issue #584)
d5c8f85 
This fix is only for the test itself.
Please read the discussion of issue 584 for more details on the actual problem
(selection changes followed by FF setup / updateEnergy).
In order to reduce the occurrence of test failures, according to Daniels suggestion,
I interspersed sleep time before calls to FF setup() or updateEnergy(). This reduces
the number of test failures roughly ten times.
@philthiel
Copy link
Contributor

According to Daniels suggestion I added a fix that reduces the number of test failures roughly about ten times (d5c8f85).

tkemmer pushed a commit that referenced this issue Mar 16, 2017
@philthiel @tkemmer
Fix for AmberFF_test (issue #584)
4868144 
This fix is only for the test itself.
Please read the discussion of issue 584 for more details on the actual problem
(selection changes followed by FF setup / updateEnergy).
In order to reduce the occurrence of test failures, according to Daniels suggestion,
I interspersed sleep time before calls to FF setup() or updateEnergy(). This reduces
the number of test failures roughly ten times.
@tkemmer tkemmer added T: defect and removed bug labels Mar 18, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C: BALL Core T: defect
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dstoeckel @anhi @tkemmer @philthiel