Random matrix theory (RMT) universality is the defining property of quantum mechanical chaotic systems, and can be probed by observables like the spectral form factor (SFF). In this paper, we describe systematic deviations from RMT behaviour at intermediate time scales in systems with approximate symmetries. At early times, the symmetries allow us to organize the Hilbert space into approximately decoupled sectors, each of which contributes independently to the SFF. At late times, the SFF transitions into the final ramp of the fully mixed chaotic Hamiltonian. For approximate continuous symmetries, the transitional behaviour is governed by a universal process that we call Hilbert space diffusion. The diffusion constant corresponding to this process is related to the relaxation rate of the associated nearly conserved charge. By implementing a chaotic sigma model for Hilbert-space diffusion, we formulate an analytic theory of this process which agrees quantitatively with our numerical results for different examples.
