**Introduction**

In response to my Nov-10-2006 posting, Introduction To Entropy, an anonymous reader asked:

*Can we use entropy for distinguishing random signals and deterministic signal? Lets say i generate two signals in matlab. First signal using sin function and second using randn function. Can we use entropy to distinguish between these two signal?*

The short answer is: Yes, we can use entropy for this purpose, although even simpler summary statistics would reveal that the normally distributed

*randn*data included values outside of -1..+1, while the

*sin*data did not.

In this article, I will be using my own entropy calculating routines, which can be found on MATLAB Central: Entropy, JointEntropy, ConditionalEntropy and MutualInformation.

**A Slightly Harder Problem**

To illustrate this application of entropy, I propose a slightly different problem, in which the sine data and the random data share the same distribution. To achieve this, the "random" data will be a random sample from the sine function:

>> X = [1:1000]';

>> Sine = sin(0.05 * X);

>> RandomData = sin(2 * pi * rand(size(X)));

>> X = [1:1000]';

>> Sine = sin(0.05 * X);

>> RandomData = sin(2 * pi * rand(size(X)));

As a quick check on the distributions, we will examine their respective histograms:

>> figure

>> subplot(2,1,1), hist(Sine), xlabel('Sine Value'), ylabel('Frequency'), grid on

>> subplot(2,1,2), hist(RandomData), xlabel('RandomData Value'), ylabel('Frequency'), grid on

>> figure

>> subplot(2,1,1), hist(Sine), xlabel('Sine Value'), ylabel('Frequency'), grid on

>> subplot(2,1,2), hist(RandomData), xlabel('RandomData Value'), ylabel('Frequency'), grid on

*Click image to enlarge.*

More or less, they appear to match.

**A First Look, Using Entropy**

At this point, the reader may be tempted to calculate the entropies of the two distributions, and compare them. Since their distributions (as per the histograms) are similar, we should expect their entropies to also be similar.

To date, this Web log has only dealt with

*discrete entropy*, yet our data is continuous. While there is a

*continuous entropy*, we will stick with the simpler (in my opinion) discrete entropy for now. This requires that the real-valued numbers of our data be converted to symbols. We will accomplish this via quantization ("binning") to 10 levels:

>> Sine10 = ceil(10 * (Sine + 1) / 2);

>> RandomData10 = ceil(10 * (RandomData + 1) / 2);

>> Sine10 = ceil(10 * (Sine + 1) / 2);

>> RandomData10 = ceil(10 * (RandomData + 1) / 2);

If the MATLAB Statistics Toolbox is installed, one can check the resulting frequencies thus (I apologize for Blogger's butchering of the text formatting):

>> tabulate(Sine10)

Value Count Percent

1 205 20.50%

2 91 9.10%

3 75 7.50%

4 66 6.60%

5 60 6.00%

6 66 6.60%

7 66 6.60%

8 75 7.50%

9 91 9.10%

10 205 20.50%

>> tabulate(RandomData10)

Value Count Percent

1 197 19.70%

2 99 9.90%

3 84 8.40%

4 68 6.80%

5 66 6.60%

6 55 5.50%

7 68 6.80%

8 67 6.70%

9 82 8.20%

10 214 21.40%

>> tabulate(Sine10)

Value Count Percent

1 205 20.50%

2 91 9.10%

3 75 7.50%

4 66 6.60%

5 60 6.00%

6 66 6.60%

7 66 6.60%

8 75 7.50%

9 91 9.10%

10 205 20.50%

>> tabulate(RandomData10)

Value Count Percent

1 197 19.70%

2 99 9.90%

3 84 8.40%

4 68 6.80%

5 66 6.60%

6 55 5.50%

7 68 6.80%

8 67 6.70%

9 82 8.20%

10 214 21.40%

It should be noted that other procedures could have been used for the signal-to-symbol conversion. For example, bin frequencies could have been made equal. The above method was selected because it is simple and requires no Toolbox functions. Also, other numbers of bins could have been utilized.

Now that the data is represented by symbols, we may check the earlier assertion regarding similar distributions yielding similar entropies (measured in bits per observation):

>> Entropy(Sine10)

ans =

3.1473

>> Entropy(RandomData10)

ans =

3.1418

>> Entropy(Sine10)

ans =

3.1473

>> Entropy(RandomData10)

ans =

3.1418

As these are sample statistics, we would not expect them to match exactly, but these are very close.

**Another Perspective**

One important aspect of the structure of a sine curve is that it varies over time (or whatever the domain is). This means that any given sine value is typically very similar to those on either side. With this in mind, we will investigate the conditional entropy of each of these two signals versus themselves, lagged by one observation:

>> ConditionalEntropy(Sine10(2:end),Sine10(1:end-1))

ans =

0.6631

>> ConditionalEntropy(RandomData10(2:end),RandomData10(1:end-1))

ans =

3.0519

>> ConditionalEntropy(Sine10(2:end),Sine10(1:end-1))

ans =

0.6631

>> ConditionalEntropy(RandomData10(2:end),RandomData10(1:end-1))

ans =

3.0519

Ah! Notice that the entropy of the sine data, given knowledge of its immediate predecessor is much lower than the entropy of the random data, given its immediate predecessor. These data are indeed demonstrably different insofar as they behave over time, despite sharing the same distribution.

An astute reader may at this point notice that the conditional entropy of the random data, given 1 lagged value, is less than the entropy of the raw random data. This is an artifact of the finite number of samples and the quantization process. Given more observations and a finer quantization, this discrepancy between sample statistics and population statistics will shrink.

Entropy could have been applied to this problem other ways, too. For instance, one might calculate entropy for short time windows. I would point out that other, more traditional procedures might be used instead, such as calculating the auto-correlation for lag 1. It is worth seeing how entropy adds to the analyst's toolbox, though.

**Further Reading**

See also the Apr-01-2009 posting, Introduction to Conditional Entropy.

Print:

*The Mathematical Theory of Communication*by Claude Shannon (ISBN 0-252-72548-4)

*Elements of Information Theory*by Cover and Thomas (ISBN 0-471-06259)

## 3 comments:

I wonder if entropy here is better than any other statistic measuring spread of empirical distribution, like sum of squares of probabilities

Hi,

Doing my research I found great books about Data mining.

This books intends to bring together the most recent advances and applications of data mining research in the promising areas of medicine and biology, in real life applications, web applications etc. The readers will benefit from this books and consider it as an excellent way to keep pace with the vast and diverse advances of new research efforts.

This is link where you can find it: http://www.intechopen.com/search?q=Data+mining

They are free to download!!

Hi.

Great blog.

Do you have a reference, or could you explain the steps for how you calculate joint entropy in you JointEntropy.m matlab function? I don't quite follow what is going on.

Thanks.

Post a Comment