The interaction of a gravitational wave (GW) with an elastic body is usually described in terms of a GW "force" driving the oscillations of the body's normal modes. However, this description is only possible for GW frequencies for which the response of the elastic body is dominated by a few normal modes. At higher frequencies the normal modes blend into a quasi-continuum and a field-theoretical description, as pioneered by Dyson already in 1969, becomes necessary. However, since the metric perturbation h_{\mu\nu} is an intrinsically relativistic object, a consistent coupling to GWs can only be obtained within a relativistic (and, in fact generally covariant) theory of elasticity. We develop such a formalism using the methods of modern effective field theories, and we use it to provide a derivation of the interaction of elastic bodies with GWs valid also in the high-frequency regime, providing a first-principle derivation of Dyson's result (and partially correcting it). We also stress that the field-theoretical results are obtained working in the TT frame, while the description in terms of a force driving the normal modes is only valid in the proper detector frame. We show how to transform the results between the two frames. Beside an intrinsic conceptual interest, these results are relevant to the computation of the sensitivity of the recently proposed Lunar Gravitational Wave Antenna.