In 2008, new Duke head football coach David Cutcliffe was watching his squad workout for the first time. After 30 minutes, he abruptly cut the session short, gleefully announcing to the team and staff that he was watching the fattest football players he’d ever seen. “This is fantastic. We can add four wins just by getting you in shape.” Over the next three months, the Blue Devils lost a cumulative 500 pounds.

That quarter-ton wasn’t spread equally across 90 players. Far from it. The vast majority of that weight was lost by only a handful of players: the fat ones.

I thought about that story as I signed my tax return this past week. I think about it every time I hear anyone demagogue changes in the tax code, complaining that changes unequally benefit the wealthiest taxpayers. However, the vast majority of any change in the country’s tax burden – either an increase or decrease – will always be borne by the same small group of people: those who pay most of the taxes.

There are plenty of obvious inequities in the tax code. It favors homeowners, Mormons and investors over renters, childless couples and wage earners. It also favors the unemployed and underemployed over the employed. That’s not a political statement or moral judgement; it’s a statement of fact. If a couple with two children makes below $45,000 their income tax rate is zero.

Maybe that figure should be $70,000. Or $25,000. Maybe the highest rate should be 50% or 25%, instead of 37%.

Whatever the structure of the code, tax cuts will only be enjoyed by people who pay taxes. Tax increases will only be suffered by people who pay taxes.

So who pays taxes?

According to the latest IRS data, it takes $481,000 of adjusted gross income (AGI) to place a taxpayer in the top one percent of taxpayers. This group pays about 39% of all federal income taxes. “So what,” you say? “Those people are rich. They can afford it.”

Let’s see if we can hit a little closer to home. The top ten percent of taxpayers – those with AGI in excess of $138,000 – pay 70.6% of all income taxes. The Tax Policy Center estimates that 45.5% of households don’t pay any income tax, so fewer than five percent of Americans pay 70% of all federal income taxes.

It costs money to run the government, and we only collect enough taxes to fund about 80% of our federal expenditures each year. Whether higher or lower rates maximize tax collections is a question of politics and economic theory. But any change in rates is felt by a small group of people – those who pay the taxes. That’s simple math.

David Moon, pictured above with Johnny Majors, is president of Moon Capital Management. A version of this piece originally appeared in the USA TODAY NETWORK.