Seizure detection/SDA Performance Assessment

SDA Performance Assessement

SDA performance is conventionally assessed using sensitivity, specificity and speed of detection in comparison to the “gold standard” of expert visual analysis of EEG and patient video by a trained electroencephalographer/epileptologist. The inter-relationship between sensitivity and specificity is often analyzed using Receiver Operating Characteristic (ROC) curves. However, the results obtained/reported using each of these standard statistics must be critically and carefully interpreted to be meaningful and are far from complete in terms of describing SDA performance. This may be illustrated with a few simple examples (#F10-#F11). Extensive prospective validation of any SDA is required to properly assess its performance (see, e.g., Osorio et al. 2001).

Figure 1: Example illustrating some of the difficulties in interpreting sensitivity and specificity. If a patient had 100 seizures in a validation study and every one of them was immediately preceded by an SDA detection beginning 5s before EO and ending 1s before EO, then the SDA would in actuality accurately predict 100% of seizures an average (and min and max) of 5s prior to EO. Yet, each could be considered as an FP because the detections occur outside the expert-scored EO-to-EE intervals, and each seizure could be considered to be an FN for the same reason. By comparison, had the algorithm detected each seizure 1s before its scored end time (EE), and had no other detections, it would have perfect sensitivity and perfect specificity, but have an average detection delay from EO of 1s less than the average seizure duration.

Figure 2: Some simple examples illustrating why simply reporting (#FP/hr) can be misleading. All plots show the indicator function of whether or not the SDA output, denoted by \(I_{SDA}(t)\ ,\) is in a state of detection (\(I_{SDA}(t)=1\)) as a function of time of day. (A) 100 FP/24hr, but all occur within a single 10s period (which can happen, e.g., when a feature is rapidly alternating above and below a particular threshold). (B) Same as (A), but zoomed in to 12s window containing the time interval during which the FPs occur. Considering the multiple individual detections as a single “grouped FP” (or “FP cluster”) lowers the rate to 1.0 FP/24hr. (C) An example of “just” 1.0 FP/24hr, but 90% of the day is spent under the false positive (which can happen, e.g., in the case of a threshold set much too low, or a broken sensor producing nearly continuous artifact that is being erroneously detected).