It is indeed a chemistry thing. It has to do with the way the atomic orbitals on the iron ion of hemoglobin interact with the molecular orbitals of the CO molecule.
Basically, the carbon atom of CO donates its lone pair of electrons (the pair not involved in the triple bond to oxygen) to the iron ion, which is typical of ligands bonding to metal ions. However, due to their arrangement in space, d orbitals from the iron ion are able to donate electrons back to the CO molecule due to overlap with its pi antibonding orbitals. This diminishes the strength of the C-O bond, but strengthens the bond of the carbon atom to the metal.
Diatomic oxygen doesn't have this same arrangement. The two lone pairs on each oxygen atom point out at 120 degrees to each other and to the double bond. One of these pairs has to be donated to the metal ion for attachment, but the d orbitals on the iron ion can't donate electrons back nearly as well due to the orientation of diatomic oxygen's molecular orbitals. As a result, the bond to the iron ion is weaker than that of CO.
For a better visual of this bonding, you might check out the Wikipedia link below.
This strong bonding is precisely why CO is so dangerous. It can displace oxygen bound to hemoglobin and won't let go due to how tightly it binds. Any oxygen you previously breathed or subsequently breathe won't be able to bind hemoglobin, and thus can't be transported to the rest of the body's cells.