[Gate-users] Gate v8.1 failing validation tests

[Boersma David]

Hi Luke,

Yes, we are aware of this issue. The benchmark tests are very rigid, they require the output to be identical to the reference result. This does not make a lot of sense, physics-wise. The benchmark directories also contain analysis scripts with which you can investigate the results yourself, and you'll see that at least for the RT cases the simulation results are statistically compatible with the reference.

I think that at the recent Gate collaboration meeting(s) there was a consensus to replace these identity tests with statistical tests, or at least tests that are more physically meaningful. E.g. for the proton and carbon beam tests, we could determine the range (and related observables) from the simulated dose distribution and check that it is statistically compatible with the range obtained from the reference dose distribution. And/or we could run a gamma index test. We could include tests that mimic the typical validation/commissioning tests that are done at clinics; such tests may take some time to run, but for validating a GATE release it should be OK to run some thorough tests that may last a couple of hours (or maybe even days!).

There are many more problems with the current benchmark tests. For starters, there should be *many* more tests, in order to offer a minimal validation of a good portion (ideally *all*) of the functionality of Gate. I am working on this issue, and I hope that in a few weeks I can commit some prototype of a new testing suite to the develop branch. Expanding the testing suite to perform meaningful tests of a large part of Gate will be a pretty big undertaking, I hope that there more users/developers who also care about this, enough to join the effort and develop & contribute some good tests.

Kind regards,
David


From: Gate-users [mailto:gate-users-bounces at lists.opengatecollaboration.org] On Behalf Of Luke Gilling
Sent: Tuesday, June 12, 2018 02:26
To: gate-users at lists.opengatecollaboration.org
Subject: [Gate-users] Gate v8.1 failing validation tests


Hello everyone,



I have just finished installing gate v 8.1 and the command 'Gate' runs the software. However when I run the benchmark tests via the 'make test' command I get the following output:



Running tests...
Test project /home/physics/Progs/gate_v8.1-build
    Start 1: benchRT_gamma
1/7 Test #1: benchRT_gamma ....................***Failed   36.44 sec
    Start 2: benchRT_proton
2/7 Test #2: benchRT_proton ...................***Failed    2.59 sec
    Start 3: benchRT_carbon
3/7 Test #3: benchRT_carbon ...................***Failed   15.95 sec
    Start 4: benchImaging_ct
4/7 Test #4: benchImaging_ct ..................***Failed   49.31 sec
    Start 5: benchImaging_optical
5/7 Test #5: benchImaging_optical .............***Failed    6.35 sec
    Start 6: benchImaging_spect
6/7 Test #6: benchImaging_spect ...............***Failed   70.11 sec
    Start 7: benchImaging_pet
7/7 Test #7: benchImaging_pet .................***Failed    0.97 sec

0% tests passed, 7 tests failed out of 7

Total Test time (real) = 181.77 sec



Any ideas on what I can do from here? Is this a problem with the physics data?



Kind regards,

Luke Gilling
This electronic message, together with any attachments is confidential and may be privileged. If you are not the intended recipient: 1.do not copy, disclose or use the contents in any way. 2.please let me know by return email immediately and then destroy the message. Waikato DHB is not responsible for any changes made to this message and/or any attachments after sending by Waikato DHB. Before opening or using attachments, check them for viruses and effects. Waikato DHB takes no responsibility for affected attachments. Click on link www.waikatodhb.health.nz/disclaimer<http://www.waikatodhb.health.nz/disclaimer> to view the company policy website. If you are not redirected to the company policy website then copy and paste the URL into a new browser window.
This email has been filtered by SMX. For more information visit smxemail.com<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsmxemail.com%2F&data=02%7C01%7Cdavid.boersma%40medaustron.at%7C0fbed36fc0854fc6ed7e08d5cffb54e0%7Cadb749e06dfd4f6fbea8cd9f8e0c9b89%7C0%7C0%7C636643600733921199&sdata=m90mds%2Bng5BtJHChgKLHOLlJYBLPISjQVNwCBLxedSU%3D&reserved=0>
Disclaimer:
Please notice our E-Mail Disclaimer http://www.medaustron.at/email-disclaimer/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opengatecollaboration.org/pipermail/gate-users/attachments/20180612/1a412421/attachment.html>