Robbie Williams, 45, and his wife Ayda Field, 39, were new to the X Factor judging panel last year.
But after just one series, the pair have announced they are departing the ITV show.
Robbie and Ayda said they made the decision as they are unable to do “everything”.
The Angels hitmaker wrote beside a black-and-white snap of him and his spouse on Instagram: “A quick update from Team Williams… it’s been a very busy year so far. Vegas has been incredible and we cannot wait for June and July to do it all again.
“I’ve got the massive @BSTHydePark show on July 14 and I’ve written the lyrics and music for the #RSCBoyInDress show with @guyachambers – opening on November 8.”
Robbie went on to tease they have more exciting news on its way and are “gutted” they’re not returning to The X Factor.
He continued: “As well as all this we’ve got some very big announcements coming soon – but we can’t reveal all yet.
“Sadly though it’s impossible to do everything … and we’re gutted to tell you we won’t be able to return to The X Factor this year.
“We will, however, be working on projects with Simon going forward – and as huge fans of the show, we will still be tuning into The X Factor, voting and winding Simon up…x”
Ayda, who tied the knot to Robbie in 2010, shared a similar statement to her Instagram account.
The couple were joined by One Direction singer, Louis Tomlinson, and Simon Cowell on the judging panel last year.
At the end of 2018, Robbie confirmed his summer residency at Wynn Las Vegas along with some June tour dates in the United States.
Appearin on Lorraine last year, the singer revealed he joined The X Factor after producers asked his spouse to take a seat on the panel.
“Genuinely it was the Ayda card, genuinely,” Robbie explained.
He continued: “You know, I thought about doing it often, but I have been really, really busy with my own career.
“Then Ayda got a call in the car from Nigel the producer of the show and said do you think Rob will do it if you do it too?
He added: “Instantly, I was like that makes complete sense – I’m totally in.”
The music icon also revealed: “Do you know the amazing thing is, since starting to do the show, I can’t believe I haven’t done it before!”
Robbie previously admitted money had played a huge part in convincing him to join the series, saying: “There are literally millions of reasons to be on this panel for this season.
“I have got five albums with Take That and 12 of my own, and now it is time to branch out into something different.”
The X Factor 2019 returns to ITV later this year.