Lesson 5 of 6 · 6 min read
Bias
How unfair patterns get into the output, and why a chatbot must not be the only voice in hiring, lending, or punishment.
The data was already a point of view
Models learn from what people wrote and labeled. That material includes history, habits, and prejudice. The tool does not have a personal grudge. It can still repeat a stereotype because that pattern was common in the examples.
Design choices matter too: which data was kept, which behavior was rewarded, and which languages or dialects were treated as the default. “The computer said so” is not a neutral fact.
What it looks like
Ask an image tool for “a CEO” or “a nurse” and you may get a narrow picture of who does that job. Ask a chatbot to judge names, neighborhoods, or schools and you may get a ranking that tracks status, not the skill you care about.
A résumé sorter can bury qualified people because their school, gap, or wording was rare in the training examples. A fluent paragraph about a group of people can be confident and still be a stereotype.
Decisions that need a human standard
Do not let a chatbot be the only reason you hire, fire, lend, diagnose, arrest, or punish. Those choices need criteria you can explain, a person who is accountable, and a way for someone to challenge the result.
If you use a tool for a first pass, read the underlying material yourself. Throw out a ranking you cannot justify. Ask who is missing from the output, not only whether the sentences sound fair.
Keep these
- Bias here means unfair patterns from the data and the product design, not a machine with a private motive.
- Stereotypes can arrive in a polite paragraph or a tidy ranking.
- Hiring, lending, and punishment are not chatbot-only decisions.