Evaluating Subtitle Segmentation for End-to-end Generation Systems - Traitement du Langage Parlé Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Evaluating Subtitle Segmentation for End-to-end Generation Systems

Alina Karakanta
  • Fonction : Auteur
  • PersonId : 1166680
François Buet
Mauro Cettolo
  • Fonction : Auteur
  • PersonId : 1166681
François Yvon

Résumé

Subtitles appear on screen as short pieces of text, segmented based on formal constraints (length) and syntactic/semantic criteria. Subtitle segmentation can be evaluated with sequence segmentation metrics against a human reference. However, standard segmentation metrics cannot be applied when systems generate outputs different than the reference, e.g. with end-to-end subtitling systems. In this paper, we study ways to conduct reference-based evaluations of segmentation accuracy irrespective of the textual content. We first conduct a systematic analysis of existing metrics for evaluating subtitle segmentation. We then introduce Sigma, a new Subtitle Segmentation Score derived from an approximate upper-bound of BLEU on segmentation boundaries, which allows us to disentangle the effect of good segmentation from text quality. To compare Sigma with existing metrics, we further propose a boundary projection method from imperfect hypotheses to the true reference. Results show that all metrics are able to reward high quality output but for similar outputs system ranking depends on each metric's sensitivity to error type. Our thorough analyses suggest Sigma is a promising segmentation candidate but its reliability over other segmentation metrics remains to be validated through correlations with human judgements.
Fichier principal
Vignette du fichier
2022.lrec-1.328.pdf (854.74 Ko) Télécharger le fichier
Origine : Fichiers éditeurs autorisés sur une archive ouverte

Dates et versions

hal-03783891 , version 1 (22-09-2022)

Identifiants

  • HAL Id : hal-03783891 , version 1

Citer

Alina Karakanta, François Buet, Mauro Cettolo, François Yvon. Evaluating Subtitle Segmentation for End-to-end Generation Systems. 13th Language Resources and Evaluation Conference (LREC), ELDA, Jun 2022, Marseille, France. ⟨hal-03783891⟩
41 Consultations
94 Téléchargements

Partager

Gmail Facebook X LinkedIn More