When they rescued Harry, he was severely abused and starved. When they tried to help him, they couldn't. So Dumbledore wiped his memory. The only problem is, he didn't lose his memories completely.

Harry's twin is believed to be the Boy-Who-Lived. James has survived to raise the twins on his own. Dumbledore believes that Henry needs to be trained on his own, so he takes Harry away. James and Sirius get the best of him though. Manipulative Dumbledore