Hierarchical Permutation Consistency Learning for Self-Conditioned End-to-End Speaker Diarization

Self-conditioned (SC) end-to-end neural diarization refines its speaker predictions layer by layer, feeding each layer’s output back as conditioning for the next. But conventional layer-wise Permutation-Invariant Training (PIT) solves for the optimal speaker permutation independently at every layer.

That independence is the problem. Different layers settle on different permutations — hierarchical permutation inconsistency (HPI) — and the resulting permutation noise propagates straight into the self-conditioning path, destabilising training.

We propose two frameworks that constrain permutation across the hierarchy rather than per layer:

• Group-Wise PIT (GW-PIT) enforces a hard global permutation constraint.
• Continuous Influence PIT (CI-PIT) applies soft regularisation instead, aggregating layer-wise cost matrices with Gaussian weighting. This pulls permutations toward consistency while still leaving each layer room to form its own local representation.

Results. On CALLHOME, CI-PIT suppresses permutation noise, stabilises the training dynamics, and improves DER for both 2-speaker and 3-speaker scenarios.

Recommended citation: Bongsu Jung and Wooil Kim. (2026). "Hierarchical Permutation Consistency Learning for Self-Conditioned End-to-End Speaker Diarization." Interspeech 2026, pp. 5307–5311. doi:10.21437/Interspeech.2026-1198
Download Paper | Download Bibtex