Why does a patent not give us the right to use our invention?

A patent gives the patentee (the owner of the patent) the right to exclude another from making, using or selling the claimed invention. If you were to build your invention and some subsystem or subcomponent in your invention was protected by someone else's patent, then the device you built may be infringing someone else's patent(s). So filing a patent application is one thing; building a device is another. Before building your device, you should conduct an infringement search to make sure there are no other patents that may cover subsystems/components of your device.