In Defense of A Muggle

It's been said that Harry's horcrux/scar could have negatively affected the Dursley family, causing them to mistreat Harry. What if it was wizard-kind as a whole that affected them, however? One tiny difference in a conversation that almost never happened will change the fate of the wizarding world as a whole. The real question is, is the change for the better, or worse?