The processing of quantum information always has a cost in terms of physical resources such as energy or time. Determining the resource requirements is not only an indispensable step in the design of practical devices - the resources need to be actually provided - but may also yield fundamental constraints on the class of processes that are physically possible. Here we study how much energy is required to implement a desired unitary gate on a quantum system with a non-trivial energy spectrum. We derive a general lower bound on the energy requirement, extending the main result of Ref. [1] from finite dimensional systems to systems with unbounded Hamiltonians. Such an extension has immediate applications in quantum information processing with optical systems, and allows us to provide bounds on the energy requirement of continuous variable quantum gates, such as displacement and squeezing gates.