2.1.1 Nonparametric Estimation of Renyi's Entropy
Renyi's entropy is a parametric family defined by [Ren70]
1 o
H(X) log f (x)dx (2.1)
-0-a 0
where X is a random variable with the designated marginal and joint pdf, and ac is the
order parameter. The entropy definition in (2.1) can also be written as,
HaX)= 1 logE[f-1(X)] (2.2)
1-a
To find the pdf f (X) in the above equation, the famous non-parametric
estimation using Parzen windowing is utilized. The Parzen window estimator [Par67] for
the pdf of the data samples p,, i=1,2,...N, is evaluated using a kernel function C, (.),
where oy is a parameter that controls the width of the kernel function.
1 N
fX(x) C Z= K(x- pi) (2.3)
N.1
In the multidimensional pdf estimation case, this can be a vector or the covariance
matrix of the kernel function. In general, use of joint kernels of the type
n
IC(x) = fKo (x) (2.4)
o=1
where xo is the o component of the input vector has been suggested. Replacing the
expected value in (2.2) by the sample mean, the following nonparametric estimator for
Renyi's entropy [Erd02] becomes,
1 N 12-1
Ha(X) a log- ICr(P Pi) (2.5)
)1- N j=l i=1