Is the selection 16 bits per channel as well as the RGB? In my ancient PS 5.1 I can definitely see the artifacts you mention if I feather a selection sufficiently, fill it with white, and then emboss it. One thing to try is to use a few small Gaussian blur operation (0.5 pixel, 4 or 5 repeats of that tiny blur). If you're not already doing it, I recommend putting your terrain elevation on a separate layer from the color texture, embossing that layer, and using Multiply blend mode to get the final effect. That way, you can do awful things to the elevation without worrying about the underlying coloring. Similarly, you can put the noise on one layer and the gentle elevations on another and then emboss the multiplied result.