Pyramid-based Summary Evaluation Using Abstract Meaning Representation
Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017 (2017)
Abstract
We propose a novel metric for evaluating summary content coverage. The evaluation framework follows the Pyramid approach to measure how many summarization content units, considered important by human annotators, are contained in an automatic summary. Our approach automatizes the evaluation process, which does not need any manual intervention on the evaluated summary side. Our approach compares abstract meaning representations of each content unit mention and each summary sentence. We found that the proposed metric complements well the widely-used ROUGE metrics.
Authors
BibTex
@InProceedings{steinberger-krejzl-brychcin:2017:RANLP, author = {Steinberger, Josef and Krejzl, Peter and Brychc\'{i}n, Tom\'{a}\v{s}}, title = {Pyramid-based Summary Evaluation Using Abstract Meaning Representation}, booktitle = {Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017}, month = {September}, year = {2017}, address = {Varna, Bulgaria}, publisher = {INCOMA Ltd.}, pages = {701--706}, abstract = {We propose a novel metric for evaluating summary content coverage. The evaluation framework follows the Pyramid approach to measure how many summarization content units, considered important by human annotators, are contained in an automatic summary. Our approach automatizes the evaluation process, which does not need any manual intervention on the evaluated summary side. Our approach compares abstract meaning representations of each content unit mention and each summary sentence. We found that the proposed metric complements well the widely-used ROUGE metrics.}, url = {https://doi.org/10.26615/978-954-452-049-6_090} }
