DLF-EEND: Dynamic Layer Fusion for End-to-End Speaker Diarization

In end-to-end speaker diarization, the layers of a self-attentive encoder learn different things: lower layers pick up acoustic detail, upper layers separate speakers. Training only on the final layer’s output leaves that hierarchy unbalanced.

DLF-EEND adds an auxiliary branch during training that uses dynamic routing to adaptively fuse multi-resolution representations at each time step, and applies the Permutation-Invariant Training (PIT) loss to the fused features. This keeps layer-specific information distinct and improves accuracy where diarization is hardest – overlapping speech and speaker transitions.

At inference only the main branch runs, so the fusion benefit comes without the extra computation.

Results. 59.18% relative DER reduction on simulated data, and a 14.26% improvement on CALLHOME, over SA-EEND.

Recommended citation: Bongsu Jung and Wooil Kim. (2025). "DLF-EEND: Dynamic Layer Fusion for End-to-End Speaker Diarization." Interspeech 2025, pp. 1688–1692. doi:10.21437/Interspeech.2025-1059
Download Paper | Download Bibtex