In this talk, I will give a brief overview of recurrent networks and their applications. I will then present several extensions that aim to help these powerful models to learn more patterns from training data. This will include a simple modification of the architecture that allows to capture longer context information, and an architecture that allows to learn complex algorithmic patterns. The talk will be concluded with a discussion of a long term research plan on how to advance machine learning techniques towards development of artificial intelligence.

The term prosody refers to certain inherent suprasegmental properties that carry melodic, timing, and pragmatic information of continuous speech, encompassing accentuation, intonation, rhythm, speaking rate, prominences, pauses, and attitudes or emotions intended to express. Prosodic features are physically encoded in the variations in pitch contour, energy level, duration, and silence of spoken utterances. Prosodic studies have indicated that these prosodic features are not produced arbitrarily, but rather realized after a hierarchically organized structure which demarcates speech flows into domains of varying lengths by boundary or break cues. It is also known that the hierarchical prosodic structures are highly correlated with information sources of the linguistic features (lexical, syntactic, semantic, and pragmatic), the para-linguistic features (intentional, attitudinal, and stylistic), and the non-linguistic features (physical and emotional). Therefore, we can regard prosodic information as an interface between messages generated by humans and realized speech acoustic features. We may also regard prosody as a communication protocol between speakers. This talk will introduce some advances of prosody modeling jointly developed by the Speech and Multimedia Signal Processing Lab, NTPU, and the Speech Processing Lab, NCTU. The applications of the prosody modeling to automatic speech recognition (ASR) and text-to-speech system (TTS) will be also addressed.

Using sound event recognition in home environments has become a new research issue in home automation or smart homes. Identifying sound classes can significantly help home environmental monitoring. Predefined home automation services can be triggered by associated sound classes. However, variant noises or interferences always make an impact on the recognition performance. These problems are unsolved and researches to tackle them are greatly needed. In this talk, we will present several robust sound event recognition techniques such as front-end processes to filter out noises or interferences, and an approach to extract robust audio features.

Recent advances in deep and weakly supervised learning helped to lend computers new socio-affective skills. Focusing on human speech analysis, this talk highlights current abilities and potential in automatic characterisation of speakers in rich ways. This includes acquisition of information on speakers’ sincerity, deception, native language and degree of nativeness, cognitive and physical load, emotion and personality, or health diagnostics – just to name a few. An according modern architecture for holistic speech analysis will be shown including cooperative on-line learning by efficient crowd-sourcing. Further, an approach for end-to-end learning will be featured aiming at seamless speech modelling. Then, a low-resource implementation based on the openSMILE toolkit co-developed by the presenter is demonstrated considering real-time on device processing on smart phones and alike. Examples of application use-cases stem from a number of ongoing European projects – these will show-case the potential, but also current shortcomings. In an outlook, future avenues are laid out to best overcome these.

The abstraction of humans with a signals and systems framework naturally brings a synergy between engineering and behavioral sciences. Behavioral signal processing (BSP) offers a new frontier of interdisciplinary research between these communities. The core research in BSP is to model human behaviors, internal states, and perceptual judgements with observational data by using computational methods grounded in signal processing and machine learning. The outcome of BSP offers novel informatics for enhancing the capabilities of domain experts in facilitating better decision making

In this talk, we will demonstrate the use of BSP techniques in various application domains: affective computing, mental health, and educational research. The heterogeneity in human behavior expression, the subjectivity in human perceptual judgement, and the complex non-linear interplay of multiple influencing factors require not only an advancement in algorithmic development but also a closer collaboration with domain experts. With this emerging effort of BSP, we strive not only to provide engineering solutions to domain experts but also to open up potential opportunities of novel insights in the applications with broad societal impact.

Cochlear implants (CIs) are surgically implanted electronic devices that provide a sense of sound in patients with profound to severe hearing loss. The considerable progress of CI technologies in the past three decades has enabled many CI users to enjoy a high level of speech understanding in quiet. For most CI users, however, understanding speech in noisy environments remains a challenge. In this talk, I will present two approaches to address this important issue to further improve the benefits of speech intelligibility for CI recipients under noisy conditions. First, I will describe the proposed adaptive envelope compression (AEC) strategy, which is effective at enhancing the modulation depth of envelope waveform by making best use of its dynamic range and thus improving the intelligibility for CI recipients compared with the traditional static envelope compression. Second, I will introduce the deep learning based NR approach (deep denoising autoencoder, DDAE), which has been investigated its effectiveness for improving the speech intelligibility for CI recipients. Experimental results indicated that, under challenging noisy listening conditions, the AEC strategy and DDAE NR yields higher intelligibility scores than conventional approaches for Mandarin-speaking listeners, suggesting that AEC and DDAE NR could potentially be integrated into a CI processor to overcome speech perception degradation caused by noise.