The geometric optics approximation traditionally used to study the propagation of gravitational waves on a curved background, breaks down in the vicinity of compact and extended astrophysical objects, where wave-like effects like diffusion and generation of polarization occur. We provide a framework to study the generation of polarization of a stochastic background of gravitational waves propagating in an inhomogeneous universe. The framework is general and can be applied to both cosmological and astrophysical gravitational wave backgrounds in any frequency range. We derive an order of magnitude estimate of the amount of polarization generated for cosmological and astrophysical backgrounds, in the frequency range covered by present and planned gravitational wave experiments. For an astrophysical background in the PTA and LISA band, the amount of polarization generated is suppressed by a factor 10−4 (10−5) with respect to anisotropies. For a cosmological background we get an additional 10−2 suppression. We speculate on using our approach to map the distribution of (unresolvable) structures in the Universe.