Nice that you went back to the original FPN paper after the YOLOv3 article, since the neck part is easy to skip over. Building it from scratch and hooking it to a backbone and RPN head is a good way to really get it.