|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
ITEP Trials for Detection Reliability Assessment of Metal Detectors (Continued)
The lessons learned from the first two trials were applied to the third trial set in Oberjettenberg in November 2003, with the intention of creating conditions likely to yield better performance. Three new lanes were set up (in addition to the ones available from the previous trial in May) and carefully cleaned of any metal fragments. Mines with large to medium and small metal content were selected and distributed systematically at a depth ranging from zero to 20 cm. The operators, who were inexperienced, were trained carefully in open and blind exercises until they were confident concerning the reaction of each detector to each mine in each soil at different depths. To avoid confusion among the different detector operating procedures, the operators were assigned detectors belonging to one class both during the training and during the first week of the trial only (double-D coil, static mode or single coil, dynamic mode). In the second week, they changed to the second class of detectors. Results of the Trials Figure 4 shows the overall results of each trial set, in ROC diagrams. These diagrams illustrate the influence of the factors (application factor and human factor) degrading the performance of all the detectors, without distinguishing among individual detectors. The result of inexperienced operators with a short training on metal-contaminated ground shows a mean detection rate of 70 percent and 0.3 false alarms per square meter. The artificial uncooperativeness reduces the performance to a 60-percent detection rate and almost one false alarm per square meter, which is surprisingly poor. Even more surprising are the total overall results for Benkovac in June 2003, where the operators consisted of eight experienced Croatian deminers. The detection rate of about 65 percent in neutral soil decreases to almost 50 percent in a real, local, uncooperative soil with frequency-dependent susceptibility. The FAR grows from 0.5 false alarms per square meter to almost 0.6. Possible reasons for this extremely poor result are as follows:
Heterogeneous soil with strong frequency-dependent magnetic susceptibility is a challenge for all detectors, especially in combination with minimum metal mines, since the soil signals often mask the mine signal. The performance in the third trial is much better than in the first two, as expected from the conditions of the test with respect to the human factors and application factors. In Figure 4c, the upper left corner of the ROC point is 90-percent detection rate and false alarms below 0.1 per square meter. The "secret" is in carefully conducted and longer training, reduced workload, neutral and very clean soil, and targets that are easier to detect. If we want to estimate a realistic POD, it is therefore necessary to ask "What is the appropriate scenario of application and human factors for the situation we want to investigate?" Full Process Simulation In Oberjettenberg in November, one additional test was conducted, on the advice of Dieter Guelle.7 The test simulated the full manual demining process, including prodding and excavation. Since the statistical basis was too small to be representative, results of this test must be considered indicative only and any conclusions provisional. The detection rate of the manual clearance process appeared to be higher than that of the detection process without excavation, probably due to instances where a minimum-metal mine was hidden by a larger false-alarm item. Indications that could be assigned to identifiable metal fragments were excluded (according to a "metal-free" approach), so the FAR is lower. The latter is, of course, a matter of definition rather than performance. A more detailed investigation is planned within the GICHD program for improvement of the manual demining methods mentioned above. Example of a Set of Resulting Curves: Detection Rates as Function of Depth and False Alarms for the PMA-2 in Different Soils Figure 5 gives an overview of all the soils in the three trials.
In the following figures, the individual detector results are illustrated for the PMA-2 minimum metal mine under ideal conditions (i.e., neutral soil without metal contamination, well-trained operators and optimized working hours.) Figures 6a–d show the detection rates as functions of the burial depth for each device separately and Figure 6e shows the ROC points of all devices together.
Figures 7a–d and Figure 7e present the same results for the most difficult soil. The anomalous result for detector Y is due to a high FAR in the uncooperative soil, up to one false alarm per square meter and the spuriously higher detection rate at large depth. The latter phenomenon can be explained by the fact that some of the "true" positive indications appear to be signals from the soil that happened to fall within the halo of a target, so that the apparent POD does not approach zero at large depth. To avoid this type of anomaly, the soil compensation and sensitivity of the detector should be adjusted to produce an acceptable low FAR prior to starting the blind trial. CWA 14747: 2003 section 8.1.5 specifies a procedure for checking the adjustment of a metal detector to the soil under test. The test is only to be considered valid if the detector can be adjusted in a representative one-meter by one-meter setup area so that no false alarms are given when it is placed on the soil surface and then raised 30 mm above it. It seems likely that detector Y was not adjusted (or not adjustable) according to this procedure.
In the opinion of the authors, these combinations of ROC curves provide the information that the end-user ought to know about the device that he/she is going to operate in the field. It is therefore recommended that receiver operating characteristic curves, with appropriate explanation and interpretation, be included in device catalogues for the main categories of soils encountered in mine-affected areas. Conclusions and Outlook For detection reliability field tests, the combined scenario of soil type, soil metal contamination and the human factor has to be set up with care and must be appropriate for the local field situation. The characteristics of one detector should be determined in terms of the detection rate as a function of depth in each soil for each mine type and completed with the information about the correspondingFAR. An expected mean value of the performance of a detector in a certain region can then be determined from these basic curves, knowing the local mine distribution. The full demining process should be simulated to assess true clearance performance and might be introduced as a correction factor within a modular reliability model. *All figures courtesy of the author. References
Contact Information Christina Mueller |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
