We simulate binary soft disk systems in two dimensions, and investigate how the dynamics slow as the area fraction is increased toward the glass transition. The "fragility" quantifies how sensitively the relaxation time scale depends on the area fraction, and the fragility strongly depends on the composition of the mixture. We confirm prior results for mixtures of particles with similar sizes, where the ability to form small crystalline regions correlates with fragility. However, for mixtures with particle size ratios above 1.4, we find that the fragility is not correlated with structural ordering, but rather with the spatial distribution of large particles. The large particles have slower motion than the small particles, and act as confining "walls" which slow the motion of nearby small particles. The rearrangement of these confining structures governs the lifetime of dynamical heterogeneity, that is, how long local regions exhibit anomalously fast or slow behavior. The strength of the confinement effect is correlated with the fragility and also influences the aging behavior of glassy systems.