Industrial fishing is the main culprit.
Most of the remaining penguin colonies are on offshore islands, and I saw them off Luderitz in Namibia when I visited there in 1991 and 1992. At one time, however, there was a large mainland breeding colony in German Southwest Africa at Cape Cross, north of Swakopmund, as the above photograph from a book published in 1904 by Franz Seiner clearly illustrates. Today, Cape Cross hosts a very large (and very pungent) Cape Fur seal colony, but the penguins that were once there in vast numbers have vanished. There may still be a few that come ashore, but their last strongholds are far to the south along the Namibia Coast. Cape Fur seals compete with the penguins for food and sometimes consider them as prey as well.
Alas, there is no historic scenario from the German-Herero war that would call for penguins on the tabletop. Sadder still is the thought that these marvelous birds may soon have no place in the wild either.