Most of the guys who do the grinding jobs at The Spice House have one spice they hate grinding more than any other. I HATED grinding white pepper more than any other spice. Luckily, the least favorite grinding jobs are rarely the same across the staff, so that if you can't stand one thing, you can probably trade off for someone else's least favorite spice.
The weird smell/funk really does come in during the soaking of pepper in water process, and this has actually gotten much better and more hygienic over the years. If you saw photos of this process 30 years ago, you would see black pepper soaked in a small river or steam in India, which does not exactly bring cleanliness to mind. After soaking a few days, the shells softened and then the laborers would walk over this in their bare feet to loosen up the outer shells of the black pepper. So yuk, dirty water, dirty feet. Luckily there are cleaner procedures in place today. I think the factor most undesirable now is more a mold factor where the pepper is not dried enough after the water process. If the burlap sacks are filled while the product is still damp the scents from the sturdy cloth, along with mold, will enter into the equation. This happens sporadically in spice crops that are not dried properly, as the farmer is trying to get the crop out to the cooperative as fast as possible to see his income quickly. Vanilla beans are another really good example of this, if they are not properly cured, they come to us still wet, which is bad for the beans.
Each crop we import can vary drastically in flavor. When we get one where people noticeably do not like the flavor, this batch tends to be of the super fermented flavor. If only the farmers just could direct their batches to the end user, the Japanese LOVE the flavor of these super fermented berries. Americans, not so much!