[Gate-users] Benchmark disagreements - The End !
Sebastien JAN
sebastien.jan at cea.fr
Mon Sep 17 18:25:04 CEST 2007
Dear Gaters,
According to the discution with Antigoni and Maria at the beginning of
July, we performed a new set of benchmark values (for PET and SPECT
applications).
These values will be published with the next official release - expected
for this week !
Regards
Sebastien
> Dear all,
> about a month ago I posted a message regarding some discrepancies found
> in the results of the SPECTbenchmark when compared with the suggested
> results in the gate web-page. Some people reported similar
> discrepancies. We took this issue a bit further 'privately' with
> Sebastien and the end point was that could be due to different version
> of G4 used by 'us' and that used to obtain the recommended results.
>
> Attached you can find our conversation which might be of interest to
> you.
> If you have any comments/suggestions will be highly appreciated.
>
> Regards
>
> Antigoni
>
> The Institute of Cancer Research: Royal Cancer Hospital, a charitable
> Company Limited by Guarantee, Registered in England under Company No.
> 534147 with its Registered Office at 123 Old Brompton Road, London SW7
> 3RP.
>
> This e-mail message is confidential and for use by the addressee
> only. If the message is received by anyone other than the addressee,
> please return the message to the sender by replying to it and then
> delete the message from your computer and network.
>
>
> ------------------------------------------------------------------------
>
> Subject:
> Fwd: Benchmark disagreements
> From:
> "Antigoni Divoli" <antigoni.divoli at univ-nantes.fr>
> Date:
> Tue, 3 Jul 2007 15:15:32 +0200
> To:
> "Antigoni Divoli" <Antigoni.Divoli at icr.ac.uk>
>
> To:
> "Antigoni Divoli" <Antigoni.Divoli at icr.ac.uk>
>
>
>
>
> Begin forwarded message:
>
>> *From: *Sebastien JAN <sebastien.jan at cea.fr
>> <mailto:sebastien.jan at cea.fr>>
>> *Date: *July 2, 2007 6:35:30 PM CEDT
>> *To: *Antigoni Divoli <antigoni.divoli at univ-nantesfr
>> <mailto:antigoni.divoli at univ-nantes.fr>>
>> *Cc: *Maria Holstensson <Maria.Holstensson at icr.ac.uk
>> <mailto:Maria.Holstensson at icr.ac.uk>>, Patrice Descourt
>> <Patrice.Descourt at univ-brestfr
>> <mailto:Patrice.Descourt at univ-brest.fr>>, Glenn Flux
>> <Glenn.Flux at icr.ac.uk <mailto:Glenn.Flux at icr.ac.uk>>, Mike Partridge
>> <Mike.Partridge at icr.ac.uk <mailto:Mike.Partridge at icr.ac.uk>>, Susan
>> Buckley <Susan.Buckley at icr.ac.uk <mailto:Susan.Buckley at icr.ac.uk>>
>> *Subject: **Re: Benchmark disagreements*
>>
>> All differences concern the scatter fractions inside different materials.
>> Our benchmark results are based on the G4 versions 4 and 5 (and the
>> physic data bases which are associated).
>> Actually, with the current G4 version, the electromagnetic physic
>> list at low energy (< 100 kev) is quite different.
>> This point could explain your differences :
>>
>> 0.35 against 2.5 inside the collim
>> 6.5 against 8.5 inside the crystal
>>
>> So, this is why our benchmark results (specially @ low energy when
>> the physical processes are more tricky) could not be define as a pure
>> gold standard.
>>
>> Sebastien
>>
>>> Dear Sebastian,
>>> thanks for your earlier reply to start with.
>>> I just run the same famous benchmark on a mac cluster with version
>>> 3.0. I can add another line in the table sent by Maria. See values
>>> added (I haven't changed the mean values though)
>>>
>>> I agree with Maria that are too many different platforms and too
>>> many independent tests to blame the random number generator, and if
>>> so how much can we really trust our results if 80% differences are
>>> found.
>>>
>>> I'm happy to run the PET benchmark too, however, could it be that if
>>> the gate developers rerun the benchmark on a different machine might
>>> get different results? In other words how standard are the published
>>> expected results?
>>>
>>> Sorry to bother you with this, but it is quite important in my
>>> opinion to get in the bottom of it in order to trust our results and
>>> use the with confidence.
>>>
>>> Thanks for your time
>>>
>>> Antigoni
>>>
>>> On Jul 2, 2007, at 6:02 PM, Maria Holstensson wrote:
>>>
>>>> Hi again,
>>>>
>>>> I have copied in the exact results from my, Antigonis, Suhailis,
>>>> Daniels and Jonathans simulations below. As you can see, all 5 of
>>>> us get almost the exact same result. The relative standard
>>>> deviation is really small! So the problem is not statistical
>>>> fluctuations.
>>>> You say that the differences could be a cause of the random engine
>>>> on our specific architectures. But all 5 of us have independent
>>>> installations of Gate versions 3.0.0 and 3.1.0, on different
>>>> platforms. I'm running Gate on Linux using Suse, Antigoni is using
>>>> a Mac Intel Duo platform, Suhaili is using a Ubuntu Linux platform
>>>> and Jonathan is using a Linux RHEL4 AMD x86_64 platform.
>>>> I am a bit worried about this. In my research I am doing I am going
>>>> to study the amount of scatter in the collimator, among other
>>>> things. The benchmark discrepency for this is 84%, which makes me
>>>> doubt if I can trust my values.
>>>> I am only going to study SPECT in my research, but if you want me
>>>> to run the PET benchmark and send you the results I will.
>>>>
>>>>
>>>> Kind regards,
>>>> Maria
>>>>
>>>>
>>>> Percentage of unscattered photons:
>>>> Maria: 35.66%, 35.83%
>>>> Antigoni: 35.75%, 35.94%
>>>> Suhaili: 35.54%, 35.73%
>>>> Jonathan: 35.66%
>>>
>>>
>>> Antigoni Cluster: 36.1726%
>>>
>>>>
>>>> Our mean value: 35.73%
>>>> Our standard deviation: 0.13%
>>>> Benchmark value: 32.90%
>>>> Benchmark relative standard deviation: 1.0%
>>>> Discrepancy: 9%
>>>>
>>>> Percentage of photons whose last scattered event occured in the
>>>> collimator:
>>>> Maria: 0.35%, 0.36%
>>>> Antigoni: 0.35%, 0.31%
>>>> Suhaili: 0.30%, 0.35%
>>>> Daniel: 0.35%, 0.36%, 0.35%
>>>> Jonathan: 0.35%
>>>
>>>
>>> Antigoni Cluster: 0.323219%
>>>
>>>>
>>>> Our mean value: 0.34%
>>>> Our standard deviation: 0.02%
>>>> Benchmark value: 2.1%
>>>> Benchmark relative standard deviation: 2.4%
>>>> Discrepancy: 84%
>>>>
>>>>
>>>> Percentage of photons whose last scattered event occured in the
>>>> crystal:
>>>> Maria: 6.43%, 6.39%
>>>> Antigoni: 6.54%, 6.63%
>>>> Suhaili: 6.64%, 6.53%
>>>> Daniel: 6.43%, 6.62%, 6.62%
>>>> Jonathan: 6.42%
>>>
>>>
>>> Antigoni Cluster: 6.40586%
>>>
>>>>
>>>> Our mean value: 6.53%
>>>> Our standard deviation: 0.1%
>>>> Benchmark value: 8.5%
>>>> Benchmark standard deviation: 1.2%
>>>> Discrepancy: 23%
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>>> Sebastien JAN <sebastien.jan at cea.fr
>>>>>>> <mailto:sebastien.jan at cea.fr>> 07/02/07 3:39 pm >>>
>>>>>>
>>>>>>
>>>> Hi,
>>>>
>>>> It's not really easy to give you a definitive answer about this
>>>> problem.
>>>> Your results seem to be close to the right benchmark values. These
>>>> differences could be provide by the random engine on your specific
>>>> architecture.
>>>> Try to launch the PET benchmark to have an other evaluation on this
>>>> situation.
>>>>
>>>> Sebastien
>>>>
>>>>> Dear Sébastien & Patrice,
>>>>>
>>>>> As you may have seen on the Gate-users emailing list, the SPECT
>>>>> users of Gate have problems with benchmark results. It has been
>>>>> issued on the e-mailing list by several independent users, running
>>>>> Gate on different platforms. Please see Gate-users Digest:
>>>>> Vol 12, Issue 22 (23/05/2007) by Suhaili Zakaria
>>>>> Vol 12, Issue 23 (23/05/2007) by Antigoni Divoli
>>>>> Vol 12, Issue 24 (23/05/2007) by Daniel Woodsworth
>>>>> Vol 13, Issue 2 (02/06/2007) by Maria Holstensson
>>>>> Vol 13, Issue 6 (07/06/2007) by Jonathan Churchill
>>>>>
>>>>> It seems like we all have the same problem, and all of us seem to
>>>>> get the same result. The problem is that we get discrepancies
>>>>> between our outputs and the benchmark values on the Gate website.
>>>>> The biggest discrepancies are these:
>>>>>
>>>>> Percentage of unscattered photons
>>>>> My average value: 35.72% , Benchmark value: 32.90% , Out by 9%
>>>>>
>>>>> Percentage of photons whose last scattered event occured in the
>>>>> collimator:
>>>>> My average value: 0.34% , Benchmark value: 2.10% , Out by 84%
>>>>>
>>>>> Percentage of photons whose last scattered event occured in the
>>>>> crystal:
>>>>> My average value: 6.55% , Benchmark value: 8.50% , Out by 23%
>>>>>
>>>>> But some values agree well! My average number of detected counts
>>>>> between 20 keV and 190 keV is 36066 and is only out by 0.4%!
>>>>>
>>>>>
>>>>> I am a bit reluctant to trust my output values from my other Gate
>>>>> simulations before I'm confident that my version of Gate gives me
>>>>> benchmark values which agrees with those stated on the website.
>>>>> Do you have any idea what the problem might be? Do you have any
>>>>> suggestions for what we can do to sort this problem out? Any
>>>>> comments much appreciated!
>>>>>
>>>>>
>>>>> Kindest regards,
>>>>> Maria Holstensson
>>>>>
>>>>>
>>>>> The Institute of Cancer Research: Royal Cancer Hospital, a
>>>>> charitable Company Limited by Guarantee, Registered in England
>>>>> under Company No. 534147 with its Registered Office at 123 Old
>>>>> Brompton Road, London SW7 3RP.
>>>>>
>>>>> This e-mail message is confidential and for use by the addressee
>>>>> only. If the message is received by anyone other than the
>>>>> addressee, please return the message to the sender by replying to
>>>>> it and then delete the message from your computer and network.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> The Institute of Cancer Research: Royal Cancer Hospital, a
>>>> charitable Company Limited by Guarantee, Registered in England
>>>> under Company No. 534147 with its Registered Office at 123 Old
>>>> Brompton Road, London SW7 3RP.
>>>>
>>>> This e-mail message is confidential and for use by the addressee
>>>> only. If the message is received by anyone other than the
>>>> addressee, please return the message to the sender by replying to
>>>> it and then delete the message from your computer and network.
>>>
>>>
>>>
>>> Antigoni Divoli
>>> INSERM U601
>>> 9 Quai Moncousu
>>> 44093 cedex, Nantes
>>> +30 (0) 24041 2821
>>>
>>>
>>>
>>>
>>
>>
>
> Antigoni Divoli
> INSERM U601
> 9 Quai Moncousu
> 44093 cedex, Nantes
> +30 (0) 24041 2821
>
>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Gate-users mailing list
>Gate-users at lists.healthgrid.org
>http://lists.healthgrid.org/mailman/listinfo/gate-users
>
>
More information about the Gate-users
mailing list