Voltron was doing fine without him, they didn't seem to miss him, or care about him. So why should he care about them?