We present a framework for general relativistic N-body simulations in the regime of weak gravitational fields. In this approach, Einstein's equations are expanded in terms of metric perturbations about a Friedmann-Lema\^itre background, which are assumed to remain small. The metric perturbations themselves are only kept to linear order, but we keep their first spatial derivatives to second order and treat their second spatial derivatives as well as sources of stress-energy fully non-perturbatively. The evolution of matter is modelled by an N-body ensemble which can consist of free-streaming nonrelativistic (e.g. cold dark matter) or relativistic particle species (e.g. cosmic neutrinos), but the framework is fully general and also allows for other sources of stress-energy, in particular additional relativistic sources like modified-gravity models or topological defects. We compare our method with the traditional Newtonian approach and argue that relativistic methods are conceptually more robust and flexible, at the cost of a moderate increase of numerical difficulty. However, for a LambdaCDM cosmology, where nonrelativistic matter is the only source of perturbations, the relativistic corrections are expected to be small. We quantify this statement by extracting post-Newtonian estimates from Newtonian N-body simulations.