Order Entries by Category Fields (with fallback)

Question

This is almost a duplicate of this question

I have a section which I want to sort by category field. But the categories are not mandatory.

When I apply |group('refSorter.first().title') it will throw an error:

Key "type" for array with keys "0, 1, 2, 3, and so on" does not exist.

I did a workaround with a conditional statement but I guess there is a DRYer solution:

{% set refList = craft.entries.section('referenz') %}
{% set refCategories = craft.categories.group('referenzRubrik').relatedTo(refList) %}

 {% for category in refCategories %}

    <h2>{{ category.title }}</h2>

    {# testing if there is a category slug for preventing a error when no category is asigned #}

    {% if category.slug == 'kantone' %}

      {% for item in refList.relatedTo(category).order('title')|group('refSorter.first().title') %}
          {# this is beeing sorted via category title fields #}
      {% endfor %}

    {% else %}

      {% for item in refList.relatedTo(category).order('title') %}
        {# this is beeing sorted via entry title fields #}
      {% endfor %}

    {% endif %}

{% endfor %}

The must be a better way right?