Tens of millions of Americans collected unemployment benefits for at least some period of time last year due to the pandemic’s hit on the economy.

After the stress of losing their job and figuring out how much their unemployment check would cover, chances are good taxes weren’t top of mind for too many people.

Nevertheless, unemployment compensation is treated as taxable income both by the IRS and by most states. (The exceptions are Alabama, Alaska, California, Florida, Montana, Nevada, New Hampshire, New Jersey, Pennsylvania, South Dakota, Tennessee, Texas, Virginia, Washington and Wyoming.)

But it’s up to you to decide whether to pay taxes on the money upfront or file estimated taxes on it quarterly.

“Tax withholding from unemployment compensation is voluntary at both the federal and state level,” said Julie Sforza-Smith, program manager at The Tax Institute of H&R Block.

Typically, when you apply for jobless benefits, states will give you a form to fill out if you want taxes withheld. But, according to the IRS, many people don’t opt for tax withholding, nor do they pay quarterly.

In that case, the income taxes owed will be assessed at tax time.

So some people may be in for a “negative surprise” when they do their taxes this year, said Elaine Maag, a principal research associate of the Urban-Brookings Tax Policy Center.

That may be especially true for anyone who ended up making more on unemployment than they did while working, thanks to the $600 per week federal supplement to unemployment benefits that was in effect from April through July. That was paid on top of one’s state unemployment benefit, which averages $380 a week. In those cases, especially if someone had other sources of household income, such as a spouse’s paycheck, their total tax liability may be higher than normal.

“It’s going to be a big surprise for many people,” Maag said.

But the net effect may not mean they have to write a check to the IRS when they file their tax return. Instead, they may just see a smaller refund than they expected because their increased income may reduce their eligibility for large tax credits, such as the refundable Earned Income Tax Credit.

Last year, 77% of filers got a refund from the IRS, according to data on returns processed by the end of November. And the average refund paid out was more than $2,500.