Firmware is the program that lives in a device rather than being loaded from a disk. It's stored in non-volatile memory, so it's there the moment the power comes on.
That timing is the point. At power-on, the RAM is volatile and holds nothing useful. The processor starts fetching instructions from a fixed address, and that address must be in non-volatile memory. The firmware found there tests the hardware, sets it up, and (in a computer) loads the operating system into RAM.
In small devices, the firmware is the whole program: a microwave's controller, a keyboard, a thermostat.
Where it's kept has changed over time: