Well, because X is about *graphical components* of which 'windows' are just one tiny, tiny subset
Like you might want your program to accept input from and send data to buttons? Grids? Charts? Images? Textboxes?
None of that is even really possible with the 'a simulated VT-100 running just inside a window' use case of X.
So CLI is not interfacing with X 'as X', just X 'as a VT-100'.