Motivated by the results of Lemou, Méhats, and Räphael and Lemou concerning the quantitative stability of some suitable steady states for the Vlasov-Poisson system, we investigate the local uniqueness of steady states near these ones. This is inspired by analogous results of Choffrut and Šverák in the context of the 2D Euler equations