The problem is that you are doing .relatedTo(category) but the variable you are passing is called relatedTo, not category.

Just like with limit, based on your code, it should be .relatedTo(relatedTo). Also, it should be part of the same hash as your limit.

However, you can't pass a category object to a component and will need to pass its id instead (docs).

Depending on what category is in your case (and I guess it's an object), you could do:

{{ sprig('_components/load-more', {
    'limit': 6,
    'categoryId': category.id
}) }}

Then in your component:

{% set offset = offset ?? 0 %}

{% set entries = craft.entries.section('caseFiles').offset(offset).limit(limit) %}

{# Make sure a categoryId is being passed #}
{% if categoryId %}
    {% set entries = entries.relatedTo(categoryId) %}
{% endif %}

{% for entry in entries.all() %}
   ------Entry Content-----
{% endfor %}

{% if entries %}
  {# Increments `offset` by the value of `limit` and swaps itself out on click #}
  <button class="orange-button" sprig s-vars="offset: {{ offset + limit }}" 
    s-target="this" s-swap="outerHTML">
    <p>Load More</p>
  </button>
{% endif %}

The problem is that you are doing .relatedTo(category) but the variable you are passing is called relatedTo, not category.

Just like with limit, based on your code, it should be .relatedTo(relatedTo). Also, it should be part of the same hash as your limit.

However, you can't pass a category object to a component and will need to pass its id instead (docs).

Depending on what category is in your case (and I guess it's an object), you could do:

{{ sprig('_components/load-more', {
    'limit': 6,
    'categoryId': category.id
}) }}

Then in your component:

{% set offset = offset ?? 0 %}

{% set query = craft.entries().section('caseFiles').offset(offset).limit(limit) %}

{# Make sure a categoryId is being passed #}
{% if categoryId %}
    {% set query = query.relatedTo(categoryId) %}
{% endif %}

{% set entries = query.all() %}

{% for entry in entries %}
   ------Entry Content-----
{% endfor %}

{% if entries %}
  {# Increments `offset` by the value of `limit` and swaps itself out on click #}
  <button class="orange-button" sprig s-vars="offset: {{ offset + limit }}" 
    s-target="this" s-swap="outerHTML">
    <p>Load More</p>
  </button>
{% endif %}