Think of the Blockchain as a giant post office. Your "wallet" is a po box, and the hardware device is the key. The money is never in the device, but the device has the only key in the world that can get anything out of the box.
But you don't need the app or the device at all when coin is sent to one of your addresses. It's just sitting there on the Blockchain until you use your key to move it.
Stepping out of the post office analogy, the Ledger is also nice because it can give you a new recipient address whenever you want one (like for every transaction)