In chapter 1, when computing histogram results for the traces of MCMC on the example there, the y-axis is labeled probabilities.

This is not how histograms really work in numpy, however. From the documentation:

numpy.histogram(a, bins=10, range=None, normed=False, weights=None, density=None)[source]
Compute the histogram of a set of data.
...
density : bool, optional
If False, the result will contain the number of samples in each bin. If True, the result is the value of the probability density function at the bin, normalized such that the integral over the range is 1. Note that the sum of the histogram values will not be equal to 1 unless bins of unity width are chosen; it is not a probability mass function. Overrides the normed keyword if given.

http://docs.scipy.org/doc/numpy/reference/generated/numpy.histogram.html

It is correct to write probability density, not probability.

Interpreting probability density is not as straight-forward, however.

In the ideal, limiting case for a Poisson distribution, MCMC should yield a posterior distribution approximating a Dirac delta. Higher variance distributions will be broader and look more uniform. The example in the first chapter has low variance indeed compared to Poisson statistics that tend to arise in nature.