It’s really quite simple: If someone doesn’t share what you want to consume, don’t follow.
You can still be friends with each other. Following doesn’t need to be mutual.
Following someone doesn’t mean you’re friends, and being friends with someone doesn’t mean you must follow.
There’s nothing wrong with a context-based relationship. You’ll both get more value that way.