I finaly found a way to do that using the "twigpcre" craft plugin:

|preg_replace('/&(?!(?:[a-z]+|#x?\d+);)/', '&')|raw

This will replace all unescaped & characters which are not part of something else.