Idea: Replacing GPs with Specialists

I've been mulling over the idea of replacing GPs with specialists as the bulk of patient to doctor relationships.

It strikes me that when people get ill, they normally know where the problem is. So, why are people not going straight to a specialist who knows about their area?

This isn't to say that you don't need GPs, but that GPs would be more like "holistic specialists"*. That is to say, when there's a problem which a specialist can't diagnose which is about pain in one part of the body being affected by another, these people would then come into effect, perhaps creating a team of specialists to analyse the problem.

Am I missing something in this?

Because what strikes me is that GPs come from olden times, when people were less mobile so that you had to have 1 jack-of-all-trades to treat everything. When you've got towns of 200,000 or more people, I see no reason that you couldn't have a number of specialists.

* holistic as defined by Aristotle, not the new age ecobollocks use of it.