Chrisley, 55, is currently serving a 12-year sentence at Federal Prison Camp Pensacola in Florida.
He was recently dismissed from his role as assistant to the chaplain, where he was helping set up church services for inmates of different faiths, his attorney, Jay Surgent, told FOX Television Stations.
Chrisley was removed from his position due to "associating with inmates" who are part of the prison's Residential Drug Abuse Program, Surgent said. He added that Chrisley also lost access to an office in the chapel where he did his volunteer work and was "sad" about his dismissal from the role.
"He's a very religious person himself, and he's able to keep himself strong with the belief that he has that God in the end will work things out," Surgent told FOX Television Stations about his client.
For its part, the Federal Bureau of Prisons told TMZ that it does not comment "on the conditions of confinement for any incarcerated individual, including their work detail assignments" for privacy reasons.
Todd Chrisley, and his wife, Julie, gained fame for their show, "Chrisley Knows Best," which followed their tight-knit family and extravagant lifestyle. A jury in 2022 found them guilty of conspiring to defraud community banks out of more than $30 million in fraudulent loans. The Chrisleys were also found guilty of tax evasion by hiding their earnings.
Before the Chrisleys became reality television stars, they and a former business partner submitted false documents to banks in the Atlanta area to obtain fraudulent loans, prosecutors said during their trial. They accused the couple of spending lavishly on luxury cars, designer clothes, real estate and travel, and using new fraudulent loans to pay off old ones. Todd Chrisley then filed for bankruptcy, according to prosecutors, walking away from more than $20 million in unpaid loans.
Julie Chrisley was sentenced to seven years in prison for her convictions on bank fraud and tax evasion charges and has been held at a facility in Lexington, Kentucky. In September, U.S. District Judge Eleanor L. Ross denied her request for a shorter sentence.
The couple was initially ordered to pay $17.8 million in restitution, but Ross said that the amount now stands at $4.7 million.