Matt Smith said his costar is "the funniest man on the planet ever"

House of the Dragon may be a drama, but there's plenty of laughter on the set of the fantasy series.

During a New York Comic Con panel with cast members Matt Smith, Fabien Frankel and Tom Glynn-Carney, the actors said that Rhys Ifans, who portrays Otto Hightower, is the cast member most likely to not know what is going on -- or break character the most -- causing hilarity to ensue.

"Like, he'd know exactly what's happening with his character, but he'd be like, 'No idea what's going on, no idea what's going on today,' " Frankel, 30, said during the Oct. 20 conversation moderated by Josh Horowitz.

Related: House of the Dragon Season 3: Everything We Know About the 'Total War' to Come Next Season

Smith, 41, meanwhile, called Ifans, 57, "the funniest man on the planet ever," as Frankel added, "It's unbearable to work with him, actually. Like, every day is an uphill struggle to try and do something professional,"

"I feel like I literally can feel HBO behind the monitor going, like, 'Jesus f---ing Christ, Rhys.' Fabian and Olivia Cook in a f---ing scene together. Good luck with this man," he continued.

Never miss a story -- sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from celebrity news to compelling human interest stories.

House of the Dragon showrunner and co-creator Ryan Condal told Variety in August that fans can expect Ifans' character to have an important role in the series' next installment, after season 2 ended with the former Hand of the King appearing to be imprisoned.

"We were delighted to have Rhys reenter the story, and I will just say that the Otto Hightower tale has not yet come to an end," Condal said.

According to Variety, the show will resume production for season 3 in 2025 and it is set to end after four seasons total.