If your SW is buggy, or something changes in a particular browser that causes it to start throwing errors, the user gets nothing until you fix your SW (unless the browser provides a "retry without service worker" feature).

It'd be nice to allow the developer to recover in these cases

In many cases, a .catch() at the end of the respondWith promise lets them do that, but not in the case of a response that fetch deems unacceptable, eg an opaque response to a CORS request.