I have a theory about how we see colors; I came up with it a while ago, and I've got absolutely no evidence. But it's sort of one of those things that can never really be proven either way. Actually... we'll get to that.
I'm going to start my explanation with an assumption: that you have a favorite color. For the purposes of this post, I'm going to say that my favorite color is green (it's not as simple as just "green" but it would take a lot of space to explain and you'd grumble and say "aw, why are you wasting my time with this?" so we won't go into it). So my question is: why does everyone have a favorite color? As humans we like and dislike the same basic things, or at least as inexperienced children we do; warm is good but hot is bad, bitter is yucky and sweet yummy, and loud sounds hurt our ears but soft music is pretty. As we grow our tastes change and evolve, and we learn to like bitter things like coffee and loud things like rock music, but we all pretty much start off with the same hardwired instincts, bred into us by millions of years of natural selection. So why shouldn't it be the same with colors? What says we don't all like basically the same colors to start with?
The only way this works is if we see different colors when we believe we are seeing the same colors. What I mean is that when I see what we all call green (we'll call this aw_green), you actually see what I would call blue (aw_blue). But you call it green. When we both look at a color and call it red, I see something different than you, but we've both grown up seeing that color and calling it red so we don't realize we're seeing different things.
"Okay, but how do you explain the fact that individual colors are created by measurable wavelengths of light?" Color is, at the root, about neurological perception. Perhaps there is even a psychological element, with experience influencing our perceptions. But the point is, color isn't something our eyes see, it's something our brains create from the signals they're sent. In effect, color is a fabrication of our minds.
The only way I can think of to test this is with some kind of neural scan, maybe an fMRI. But if different colors don't produce visibly (to an MRI) unique responses in our brains, we're sunk. If they do, then they could set baselines for a bunch of different people and then show them different colors and watch the magic. Of course, there's all sorts of other variables too; someone might always think of dear old Grams who died last year when they see that shade of purple, and someone else might remember that time they were almost hit by a car that was that exact navy blue.
Also, studies have shown that certain colors can semi-reliably elicit certain emotional responses, varying even more with your cultural and personal experiences. (See this abstract at The Journal of Cross-Cultural Psychology)
Maybe the best way to test this would be with filters. I mean, you could put a subject in an MRI and show him color blocks (just a square of color, not an object of that color), then put different-colored lenses in front of the image and see if anything on the fMRI matches up. Maybe it could work. Somehow, you'd have to influence the wavelength of the light getting to the subject's retina...