In short, only one small subset of Terrigen Crystals, the ones artificially created in Afterlife from a melted down Diviner harm humans, as a result of being contaminated with the metal of the Diviner (which is fatal to humans). All other Terrigen Crystals you will see on the show are COMPLETELY HARMLESS to humans.

If you pay attention, there is also a distinct visual difference between standard and contaminated Crystals.