Introduction: Nearly all brain-computer interface experiments have been focused on the identification of classes of stimuli or behavior where the timing of experimental parameters is known or pre- designated. Real world experience, however, is spontaneous. In order to capture the spontaneous nature of experience, we performed an experiment predicting the occurrence, timing, and types of visual stimuli perceived by human subjects.
Methods: Electrocorticographic arrays were placed on the subtemporal cortical surface of 7 epilepsy patients. Simple, luminance and contrast matched, grayscale faces and houses were displayed rapidly, in random sequence. We developed a novel template-projection method, where event-averaged broadband timecourse (BB) and event-averaged raw potential (ERP) from training periods were used as projection filters for testing periods. The output of these projections was mapped into a common feature space, and a simple linear classifier was used to predict what the patient saw.
Results: We found that ERP and BB contribute independently to classifiability and prediction of the stimuli. If both broadband and raw potential features were used, and the timing of testing stimuli was defined ahead of time, 97% of stimuli could accurately be classified as face or house. More interestingly, when the spontaneous testing data stream was examined, 96% of all stimuli were captured spontaneously (more than 90% in every subject), with approximately 20ms error. Only 4% of the spontaneous predictions were incorrect (i.e. predicted stimuli at the wrong time, or as the wrong class).
Conclusions: This demonstrates, for the first time, decoding of the human perceptual experience at the speed over which decisions and actions are made.
Patient Care: It will facilitate development of neural prosthetics.
Learning Objectives: By the conclusion of this session, participants should be able to: 1) Describe the category specific organization of inferotemporal regions, 2) Understand how template projection can be used to bring together different aspects of physiology for robust real-time decoding from the spontaneous stream of data from the brain.