Capturing Boos are introduced as early as Gloomy Manor's "Visual Tricks" mission. However, the game does not explicitly tell you what is the reward for capturing them all.
You can obtain golden medals with E. Gadd's face on them that appear on your save file if specific requirements are met. There are three to collect.
This option creates a Scarescraper with infinite floors as well as its own difficulty, also named "Endless".
As you find and obtain gold in this game, you will periodically unlock upgrades to the Poltergust 5000, your primary weapon against the ghosts infesting Evershade Valley. These unlockables can result in the ability to deal stronger attacks to ghosts or to allow your Dark-Light Device to have more use.