Is is possible to enable certain plugins only in devMode? (or via some other configuration option)?
I just was pointed to the correct solution (shame on me I did not find it myself):
There already is a config for this: https://docs.craftcms.com/v3/config/config-settings.html#disabledplugins
To be used like this:
'production' => [ 'disabledPlugins' => ['cloner', 'cp-field-inspect', 'dumper', 'migration-manager'] ],
Thanks gtettelaar for dropping me a hint! (https://github.com/craftcms/cms/issues/5107)
However if you explain more, and the reasons why you want to do this, there might be a solution.
EDIT: Based on comment
As part of my deployment, I would have a diiferent composer.json file for the production environment which would only contain the plugins you want.
composer update on production. Only what you want will be included.