We propose a robust physiological model for the recognition of human emotions, which is based on a Convolutional Long Short-Term Memory network and a temporal margin-based loss function. Formulating the emotion recognition problem as a spectral-temporal sequence classification problem of bipolar EEG signals underlying brain lateralization and photoplethysmogram signals, the proposed model improves the performance of emotion recognition. Specifically, the new loss function is designed to result in penalties for the violation of the confidence of specific feelings. Our deep physiological learning technology enables recognition rates that significantly outperform state-of-the-art techniques. An extensive analysis of the relationship between participants’ emotion ratings and physiological changes in brain lateralization function during the experiment is also attempted.