He told the News & Record Greensboro
that the North Carolina real estate firm he had been working for informed him that he was "in the middle of a news story that we don't want anything to do with."
Varner was criticized for outing Smith as transgender during an episode of "Survivor: Game Changers" which aired last week on CBS.
Other contestants reacted negatively, telling Varner that was personal and he shouldn't have said it. Varner insisted he did it to show that Smith was deceptive during the game.
He apologized to Smith for doing it, but was still voted off the show.
Varner told "Entertainment Tonight"
he believed his firing was done in "an ugly, knee-jerk reaction kind of way."
"I wasn't even given the chance to explain or right the wrong," he said. "In the real estate world, buyers and sellers want to know they're signing up with a company that won't dump them or turn their backs on them in time of trial."
A former anchor for Fox and CBS affiliates, Varner said he was doing press for the episode when he realized he couldn't access his work e-mail. He said he was "devastated."
Varner is now looking to get picked up by another real estate agency.
"So I'm talking to several firms now that I know will care about and believe in their employees," he said. "I have had several reach out [and] I'm confident I'll find a better home."
In a statement, Allen Tate's chief executive officer Pat Riley said, "The Allen Tate Companies were built on core values of honesty, integrity and respect. Those fundamental beliefs led us to end our relationship with Mr. Varner, a real estate agent who had become affiliated with our firm just 17 days earlier."
"To be clear, at no time before the airing of the Survivor episode on Wednesday evening was our management aware of his actions on that episode," Riley added in the statement. "We make all decisions such as this one with careful consideration of our clients, employees and our agents."