quantum chemistry - Physical reason for why delocalisation leads to stability?

Why does delocalization of electrons generally make compounds more stable (e.g. in carboxylate anions, where the lone pair on the negatively charged oxygen is delocalised into the C=O π* orbital)? Is there any proper explanation involving quantum mechanics?

Answer

A simple QM explanation I can think of would invoke Hückel theory. It would go this way:

Suppose you have a two-state system (analogous to having two orbitals) with eigenvalues of the Hamiltonian $\lambda_1$ and $\lambda_2$. These correspond to the orbital energies. So just for argument's sake, let's say that

$$\begin{equation} \mathbf{H} = \begin{pmatrix}1 & 0 \\ 0 & 2\end{pmatrix} \end{equation}$$

then obviously you have $\lambda_1 = 1, \lambda_2 = 2$. Let's say this system has two electrons. They would both go into the orbital with the lower energy, i.e. $\lambda_1$. In the context of organic chemistry, this would typically mean that the first orbital would be bonding in nature (hence filled), and the second orbital would be antibonding (hence empty), although that's not always the case.

Now, you perturb the system by making the non-diagonal elements of the Hamiltonian negative (analogous to allowing the orbitals to overlap, i.e. allowing delocalisation of electron density between the two). Just for argument's sake, let's change $\mathbf{H}_{12}$ and $\mathbf{H}_{21}$ to both be $-0.1$. If you recalculate the eigenvalues you get

$$\lambda_1 = 0.99, \lambda_2 = 2.01$$

The new eigenvectors will be linear combinations of the old eigenvectors – one in-phase and one out-of-phase combination. The "bonding" or in-phase combination of the two orbitals will have decreased in energy, whereas the out-of-phase combination of the two orbitals will have an increased energy. If you put both electrons into the bonding orbital, you get a net stabilisation.

In general, interactions between filled and unfilled orbitals are stabilising in nature, since both electrons occupy a lower-energy bonding orbital.

---

On the other hand, interactions between two filled orbitals are destabilising, because you would have four electrons and you would have to fill both of the new orbitals. If you add up the two new eigenvalues, the total energy is greater than it was before when $\mathbf{H}$ was unperturbed.

This is not evident in my example above. The reason is that I made the assumption that $\langle i | j \rangle = \mathbf{S}_{ij} = \delta_{ij}$. In extended Huckel theory you would solve the generalised eigenvalue equation, $\mathbf{Hc} = E\mathbf{Sc}$. However with my assumption, $\mathbf{S}$ is simply the identity matrix and this reduces to $\mathbf{Hc} = E\mathbf{c}$. A theorem from linear algebra states that the sum of the eigenvalues of $\mathbf{H}$ is simply equal to the trace of $\mathbf{H}$. Therefore, as long as we assume $\mathbf{S}_{ij} = \delta_{ij}$, merely changing the off-diagonal values will not change the sum of the orbital energies.

In general, the basis vectors are not orthogonal. If you solved the full equation, $\mathbf{Hc} = E\mathbf{Sc}$ then you would find that the sum of the new eigenvalues is greater than the sum of the old, unperturbed eigenvalues.

Obviously this is ignoring other important aspects, e.g. electron-electron repulsions (that is a problem intrinsic to Huckel theory), but it's a start.