CNNDailymail data: Predicted summaries are lists of single words and lead to rouge score of zero #21
Description
Thank you very much for providing the latest updates to the repo. I am still having trouble training the model on a small subset of CNNDailymail data. Upon inferencing, the model keeps producing predictions that are lists of single words. I am providing more details below:
- How I ran the code:
train_and_eval.py --infer_source_file /home/shan/datasets/NLP/dev_CNNDM_sequenceGGNN/jsonl/test/inputs.jsonl.gz --infer_predictions_file /home/shan/datasets/NLP/dev_CNNDM_sequenceGGNN/jsonl/test/predictions.jsonl- The spurious single-word predictions:
Validation predictions...
[['at'], ['at'], ['at'], ['at'], ['5.3million'], ['5.3million'], ['5.3million'], ['at'], ['at'], ['at'], ['at'], ['at'], ['5.3million'], ['at'], ['at'], ['at'], ['at'], ['at'], ['at'], ['5.3million'], ['at'], ['at'], ['5.3million'], ['at'], ['at'], ['at'], ['at'], ['at'], ['at'], ['at'], ['at'], ['at'], ['at'], ['5.3million'], ['at'], ['5.3million'], ['at'], ['at'], ['at'], ['5.3million'], ['at'], ['5.3million'], ['5.3million'], ['at'], ['rehahn'], ['at'], ['at'], ['at'], ['at'], ['at'], ['at'], ['at'], ['rehahn'], ['at'],
(rest of stdout omitted)Whereas the target summaries should have been parsed properly. For example:
Targets...
[['Lord', 'Mervyn', 'Davies,', '62,', 'was', 'at', 'a', 'Royal', 'Academy', 'of', 'Arts', 'party', 'last', 'night.', 'Singer', 'Usher', 'had', 'been', 'speaking', 'to', 'group', 'of', 'young', 'people', 'at', 'charity', 'event.', 'Labour', 'peer', 'showed', 'off', 'his', 'fancy', 'footwork', 'on', 'the', 'dance', 'floor.', 'Usher', 'will', 'finish', 'his', 'tour', 'with', 'a', 'concert', 'at', 'the', 'O2', 'tonight', '.'], ['Craig', 'MacLean,', '22,', 'was', 'on', 'flight', 'to', 'Abu', 'Dhabi', 'when', 'staff', 'called', 'for', 'doctor.', 'The', 'medical', 'student', 'stepped', 'in', 'to', 'help', 'when', 'man', 'suffered', 'a', 'cardiac', 'arrest.', 'Dundee', 'University', 'student', 'started', 'trying', 'to', 'revive', 'the', 'passenger', 'at', '36,000ft.', 'KLM', 'flight', 'from', 'Scotland', 'diverted', 'to', 'Turkey', 'and', 'man', 'received', 'medical', 'care.'],
(and so on)- The error messages:
The rouge score ends up being zero and the training quickly reports error:
eval loss: 8.41, eval rouge: 0.00
early stopping triggered...
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
~/workspace/GGNN_text_summarizer/train_and_eval.py in <module>
625
626 if __name__ == "__main__":
--> 627 main()
~/workspace/GGNN_text_summarizer/train_and_eval.py in main()
212
213 if args.infer_source_file is not None:
--> 214 infer(model, args)
215
216
~/workspace/GGNN_text_summarizer/train_and_eval.py in infer(model, args)
487 # saver = tf.train.Saver(max_to_keep=100)
488 saver = tf.train.Saver(max_to_keep=1)
--> 489 saver.restore(session, os.path.join(args.checkpoint_dir, "best.ckpt"))
490
491 # build eval graph, loss and prediction ops
~/software/anaconda3/envs/tensorflow/lib/python3.7/site-packages/tensorflow/python/training/saver.py in restore(self, sess, save_path)
1266 if not checkpoint_management.checkpoint_exists(compat.as_text(save_path)):
1267 raise ValueError("The passed save_path is not a valid checkpoint: "
-> 1268 + compat.as_text(save_path))
1269
1270 logging.info("Restoring parameters from %s", compat.as_text(save_path))
ValueError: The passed save_path is not a valid checkpoint: cnndailymail_summarizer/best.ckptWould you mind providing some insights on what might have caused this issue? Thanks!