During a recent interview, Golden State Warriors Draymond Green discussed why players around the league have long doubted or dismissed the talents of his superstar teammate, Stephen Curry. But it was Green’s last point, mentioned almost as an aside—“And of course, Steph is light-skinned so [players] want to make him out to be soft”—that got the most attention.

To white Americans, the relationship between skin color and toughness or masculinity might not be obvious. They might associate skin color with race or with attractiveness. But toughness? Not so much.

My first book, published in 1992, referred to skin color as The Last Taboo Among African Americans. It explored how African-Americans, within their community, grapple with prejudices that stem from their various shades of skin colors. If you’re black, depending on the shade of your skin, other black people might think of you as “high yella” or “red-boned,” a “white wanna-be” or just not “black enough.”

But because it’s almost never discussed in public and seldom in private, many might not realize that skin color is still a big deal in the black community. For example, black athletes might treat Curry differently on the court because of his light skin, guarding him more physically or trying to goad him into fighting, but they’ll rarely acknowledge the issue in public, as Draymond Green did.

Yet occasionally we see glimpses of the prevailing attitude. In 2013, Jason Terry called his former light-skinned teammate Kris Humphries “soft,” a player who “pretends” to be tough. A couple of years later, Los Angeles Lakers All-Star Kobe Bryant instructed his lighter-skinned teammate, Jordan Clarkson, to “go to the hole like a dark-skinned dude.”

This unspoken belief about skin color and toughness has historical roots.

After the first African slaves arrived at Jamestown, Virginia, a population of mixed-race blacks emerged. Their masters and fellow slaves celebrated their exotic features—not quite African, but not exactly white. The women were called “fancy girls” and paraded at quadroon balls, events for wealthy white men to meet and mingle with them. Lighter-skinned black men, meanwhile, were dubbed “run ’round men” because, with their fairer skin, they could supposedly have their pick of any woman in the black community.

They also ended up being more privileged than their darker-skinned counterparts. Called “house Negroes,” they often lived in the master’s residence, ate better food, and wore better clothes. They moved up the social hierarchy with relative ease, given whites’ belief in the “mulatto hypothesis”—the idea that mixed-race blacks were superior to their darker-skinned brethren. This history may have created an element of resentment toward lighter-skinned blacks in the black community, something the community is still grappling with today.

Throughout American history, darker skin has also been associated with masculinity. In sports, dark-skinned black athletes, from Jack Johnson to Mike Tyson, have long been described in masculine terms: intimidating, fierce, hard.

Meanwhile, studies have shown that in many cultures around the world, America included, lighter skin is equated with femininity. In 1937, Breck shampoo, as part of a marketing campaign, sought out the perfect embodiment of feminine beauty—and settled on a 17-year-old who had an alabaster complexion.

This feminine ideal might serve lighter-skinned black women well. Unfortunately it serves as a handicap for black athletes, who, by virtue of their skin color, are assumed to be soft.

And for a light-skinned, green-eyed, black basketball player like Steph Curry, it means having to prove himself each time he steps onto the court in ways darker-skinned athletes don’t.