Why does a weak acid and a salt containing its conjugate base form an acidic buffer?

I read this today:

If you mix equal amounts of $1 M$ $\ce{H_3PO_4}$ and $1 M$ $\ce{NaH_2PO_4}$ you get a buffer at $pH<6$.

The reason given is that a weak acid plus a salt containing the conjugate base forms a acidic buffer.

If you mix equal amounts of $1 M$ $\ce{NH_3}$ and $1 M$ $\ce{NH_4Cl}$, you get a buffer at $pH>8$.

The reason given is that a weak base plus a salt containing the conjugate acid forms a basic buffer.

I don't understand why this is true. Let's take the first statement for example. If you have equal amounts of a weak acid and its conjugate base, then by definition you have a buffer. Whether the buffer is acidic or basic shouldn't depend on whether the salt contains the conjugate acid or the conjugate base, right?