Adjectival or Possessive Use of "Color?"

While the term has shown up occasionally over my last three years of teaching in the Deep South (though, never when I taught previously in the Midwest to my recollection), this semester, several students (8/64) have been non-critically referencing "colored people" in their papers. It's quite off-putting. I explained in my comments in their essays that the term is generally considered offensive and that "people of color" is preferred. Admittedly, it's an odd distinction. Unlike other terms that are preferred over alternatives because they are endonyms (e.g. Native American vs. Indian), there doesn't seem to be an obvious reason to favor "of color" over "colored" aside from convention. It got me thinking about usage. Here are the related ngrams:

"Colored" starts to fall out of favor around 1970 and is replaced with "of color" beginning around 1990. Initially, though, "of color" declined with "colored." It was only later that it gained acceptance and became a replacement. While "of color" solves some problems when trying to reference those who are not white (as if white were the absence of color), it is relatively imprecise. I encourage students to say "black" when they mean black. If they mean "those who are not white," non-white is fine.