Ianto saw more about Torchwood than anyone would give him credit. And since Jack was Torchwood, that meant Ianto saw more than Jack wanted him to.
After Owen died, and came back but was dead, Ianto saw what Jack was doing. Holding back, letting Gwen slip back into the role of leader, letting the others take hold of the situation that Jack had screwed up beyond measure, letting himself slip into the background.
And in those actions, Ianto saw something that he never wanted to accept.
Sometimes, there was no fixing things.
Jack knew this.
And now, so did Ianto.