We develop the general theory of the angular N-point spectra and derive the cosmic variance on the light cone. While the angular bispectrum and the trispectrum are well developed in literature, these higher point angular spectra in general are only shown to be the solutions under the symmetry requirements, rather than uniquely constructed from the N-point orthonormal harmonic components, rendering it difficult to go beyond N=4. Here we extend the Wigner 3-j symbols of triangles to construct the multilateral Wigner symbols of polygons and compute the angular N-point spectra directly from cosmological observables. We apply the Ergodic hypothesis to cosmological observations on a single light-cone and derive the cosmic variance of the angular N-point spectra.