Biggest difference; a public company has a responsibility towards it's shareholders. It has to give periodical updates on how it's performing and disclose their books for the benefit of the public. They have to hold the shareholders' interests utmost and make a profit for them.
A private company on the other hand doesn't have to disclose their books to the general public, and it's basically run by the boss/bosses alone and he/she/they has/have no obligation to make profit for their shareholders per se, which gives them the freedom to steer the company as they see fit.
A good example would be Tesla and SpaceX. Founded and managed by the same person, Tesla is a public company, which means it has to make considered decisions and make sure the decision will benefit the company and shareholders financially. SpaceX on the other hand is private. And there's a reason for that; Elon Musk wants to go to Mars eventually and he knows that going public means he can't take risks or decisions that may be financially unproductive. He would have to ask for the majority shareholders permission whenever he wants to do something.