Soon to be former Los Angeles Clippers owner Donald Sterling’s recent racist statements—made in a private setting—have brought about a discussion about the nature of freedom of speech.

One misconception that has entered discussions on this topic is that freedom of speech means that no one can be punished for what they say—specifically in private conversations. There is a difference between freedom of speech, freedom from the backlash of such speech, and freedom of privacy.

There is no explicit “freedom of privacy” in the Constitution. People are free to have their own religious beliefs, are protected from unreasonable searches of property, and are protected from self-incrimination. All of these rights to privacy can be a bit blurry.

For example, if people wanted to claim that their racism, sexism or other bigotry is being protected as a religious principle, this could be perceived as a way to protect such bigotry. Also, the protection from self-incrimination only means that people have the right to be silent and not incriminate themselves. If they were to speak and admit guilt, those people wouldn’t utilize their rights.

However, reality conflicts with these views on privacy. Once you say something and it is recorded and distributed across the media or web, your private thoughts are no longer private. If you were recorded without your consent, it can’t be used in a courtroom against you. However, if you’re rich and powerful, the mob of angry people outside will consider it.

This is where the idea of being punished conflicts with public backlash. It may hurt someone’s ability to make money if they say something racist and their business is based on serving customers of many races. This isn’t a punishment. It’s simply cause and effect. We don’t need to look at rights to know that language has meaning, and unpopular statements create resistance—they should in a healthy, literate society.

Donald Sterling alienated most of his employees, coworkers, and customers. His coworkers decided to remove him. If I were to tell my employer, “Hey, I hate you, and you disgust me,” it would be fair to assume I would be fired. And I wouldn’t be able to sue my now former employer for infringing on my free speech because I was free to say that.

No one can make the argument that bosses can’t fire their employees for expressing hatred and disgust for their superiors. We all have the right to say anything—thanks to our freedom of speech. So we can’t go to jail for what we say, but we don’t have the right to employment.

The Donald Sterling predicament is unique, though. He bought the Clippers in 1981 for $12.5 million and, through a forced sale, is set to receive $2 billion for the franchise in a sale to former Microsoft CEO Steve Ballmer.

Sterling will pay an accountant or lawyer to work out how much of the $2 billion will go to his estranged wife (Shelly Sterling) and to the IRS, but it’s hard for me to see billions of dollars in profit as a punishment. It’s safe to assume that Sterling will be okay through all of this.