We study particle-scale motion in sheared highly polydisperse amorphous materials, in which the largest particles are as much as ten times the size of the smallest. We find strikingly different behavior from the more commonly studied amorphous systems with low polydispersity. In particular, an analysis of the nonaffine motion of particles reveals qualitative differences between large and small particles: The smaller particles have dramatically more nonaffine motion, which is induced by the presence of the large particles. We characterize how the nonaffine motion changes from the low- to high-polydispersity regimes. We further demonstrate a quantitative way to distinguish between “large” and “small” particles in systems with broad distributions of particle sizes. A macroscopic consequence of the nonaffine motion is a decrease in the energy dissipation rate for highly polydisperse samples, which is due both to a geometric consequence of the changing jamming conditions for higher polydispersity and to the changing character of nonaffine motion.