"If he lost his wand, he can just use his hands, right?'

No, he cannot. In ancient times, some magicians could channel their abilities through their hands, but often produced poor results if not done correctly. Hand magic was unpredictable and near uncontrollable. The invention of the wand was to benefit magicians and make it safer and easier to perform feats of magic. Witches and wizards have to channel their magic through a wand, as the wand itself is not magical.