I recommend clearing all caches on deploy via
Caches should not be used to mask performance issues, they are used to solve concurrency & scaling, by holding onto data that doesn't need to be recomputed each time.
As such, caches should be ephemeral, and the cost of clearing a cache should not be so expensive that it drastically impacts the performance of the site.
If clearing all of your caches detrimentally impacts the site, probably the caching layer is masking issues with the site that should be addressed.
Important to note: Clearing all caches in Craft CMS does not cause image transforms (which are very expensive) to be deleted on disk.
It's not a matter of if, but when you will spend hours debugging an issue only to find that it's the result of a stale cache. Or conversely, being notified in the middle of the night that there is an issue with the latest deployment that was not discovered until a cache clears.
Web servers are highly complex state machines; caching introduces yet another layer of state that adds more complexity that you don't want to spend your time diagnosing and debugging.
Selectively clearing caches means you have to fully understand what each cache does, and all of the potential impacts and interplay involved in not clearing it. This can be more complex than it seems at first blush.
Selectively clearing caches also means that you need to ensure any new caches introduced by Craft or plugins are evaluated, and then you may have to go back and retroactively add to your list of caches to clear for each site you maintain.
It's probably not the most effective use of your resources to do so; better to spend your time optimizing your site, and clear all caches on deploy.
You cannot do truly atomic deployments while keeping stale cache data intact. ref: Atomic Deployments Without Tears