What are your 4 sources used for? Depending on your content you might be able to work around it.
There's no exact science to this but in my experience doing IA, here's a few thoughts on category sources:
I like to set mine up so they kind of match the sections I'm posting in. If you have to reach across multiple category sources for a field—say you only have one section throughout the site—it's possible your top level sections aren't specific enough.
You might be using categories that really shouldn't be categories. Like on a news site, Breaking News could very well span across everything. In that case, you'd be better off setting up a new field, say a checkbox, and allowing your authors to tick it as appropriate.
If you're coming from something like WordPress where you only had
categories and tags to use out of the box, you can fall in that trap.
Along the same lines, if you wanted to put something on the home page, you might have used a "featured" category that only shows up on the home page; a better, Craftier way is to use its relational magic and setup a field on the home page entry itself.
Category sources are hierarchical so you could merge the ones you need into some kind of catch-all category as needed. Categories should be relatively shallow. If you have like 50 categories for a smaller site, you're doing it wrong. It sounds like you have categories that are of varying topics. That can be a nightmare anyway you slice it.
As Luke mentioned, you can set it up so only 1 category can be selected. I would also look at how your 4 category fields are labeled. If you have labels like "Main Category" and then "Sub Category", it's probably OK that someone leaves the second and 3rd fields blank, for example. So the first could be required as a general "gist" of what the topic is, more less drilling down specificity.