New blog post about a technique for "deep filtering" of generic foreign keys: https://blog.bmispelon.rocks/articles/2024/2024-11-25-genericforeignkey-deep-filtering.html
blog.bmispelon.rocksGenericForeignKey Deep Filtering | Baptiste Mispelon
@bmispelon great post Baptiste!
I've used a similar approach in a somewhat similar situation with Wagtail pages.
I had a queryset of 'generic' pages that I want to annotate a field to. Say 'author'. Not all page models have an author text field, so I used a similar When construct to only annotate this field for the page types that have it. Works well! Should probably document it in a blog post too... 
The contenttypes app is a hidden gem if you ask me.