To “throw out the baby with the bathwater” is a German idiom dating back to at least the 1500s (originally das Kind mit dem Bade ausschütten). It’s a caution to not discard something good in your haste to discard something bad.
Its first well-known use in English is in an essay by the Scottish philosopher Thomas Carlyle, addressing the slave trade that ran through England in the 1800s. However, Carlyle was using it as an excuse to not abolish slavery, or at least transition it to serfdom. If only the phrase had a more positive introduction to the English language, perhaps it would have caught on quicker.
The idiom has also famously been used by Martin Luther, Johannes Kepler, Goethe, Otto von Bismarck, and George Bernard Shaw.