Arbitrary-Order Time-Accurate Semi-Lagrangian Spectral Approximations of the Vlasov-Poisson System