Concentration of Measure I

Wed, 27 May 2009 05:20:00 -0700

The concept of Concentration of Measure (CoM) quantifies the tendency of an $latex N$-dimensional probability distribution to "lump" or concentrate around an $latex (N-1)$-dimensional submanifold. The phenomenon is that this tendency is especially large in high dimensions. Viewed another way, this is about the interaction between probability and distance in high dimensions.
This is (hopefully) the start of a series of posts that will explain this concept. I would like to review a few concepts and definitions first though.
We denote by $latex X$ a metric measure space of metric $latex d$ and probability measure $latex \mu$. A probability measure is in effect a normalised measure, such that the measure of the whole space is 1. In such a space, the $latex \epsilon$-extension of a set $latex A$ is denoted by $latex A_\epsilon$ and is defined as $latex A_\epsilon=\left\{ x|d(x,A)\lt\epsilon \right\}$. Of course, the $latex \epsilon$-extension of $latex A$ includes all of $latex A$. What it does more than that is that it fattens $latex A$ by width $latex \epsilon$. Among other things, this means that $latex A_\epsilon \setminus A$ is a shell enveloping $latex A$, and that for small values of $latex \epsilon$, this volume (substitute measure if you want to be pedantic) is approximately the surface area (substitute surface measure) of the set $latex A$ multiplied by $latex \epsilon$.
Now we can define the concentration rate of a space. The basic idea is: of all the sets $latex A$ that span half of the total volume of the space, find the one whose $latex \epsilon$ extension spans the smallest volume. We then declare the concentration rate of the space to be the volume outside that extension. In math terms: $latex \alpha(\epsilon)=\sup\left\{ \mu\left( X \setminus A_\epsilon \right) | \mu(A)=\frac{1}{2} \right\}$.
The more the measure is concentrated around a half-space, the smaller the concentration rate is. What we really want, and what CoM is about, is a very quick decay of the concentration rate as $latex \epsilon$ increases. This will be useful for us to decide in a learning algorithm which value of $latex \epsilon$ to cut off learning at with small probable error. But that is for another day.
And yes, it would be good if Posterous supported Latex..

Permalink | Leave a comment »

Testing Posterous -- and the ratio of the volumes of hyperballs and hypercubes in high dimensions

Sat, 09 May 2009 04:45:00 -0700

This Posterous service looks like a great thing.
The picture attached is the ratio of the volume of a hyperball to its
circumscribing hypercube in N dimensions.
The mathematical relation is actually $latex \frac{V_b}{V_c}(N)=\frac{\pi^\frac{N}{2}}{2^N \Gamma(\frac{N}{2}+1)}$.
I need to review both the relation and the latex text as I am writing
from memory, but this should be good enough for a test post.

Permalink | Leave a comment »

Muhammad's posterous

Concentration of Measure I

Testing Posterous -- and the ratio of the volumes of hyperballs and hypercubes in high dimensions