# Curse of Dimensionality And Dimensionality Reduction

## Introduction

This could mean a lot of things in different contexts but in the context of machine learning, it a is simple but important concept. I am going to describe it in an example:

- Assume that you have plotted a square in the plane with side length of and then you partition each side of this square into mutually exclusive line segments with length . By doing this you can create a checkerboard of unit squares.
- Then scatter uniformly generated points described by coordinates in the big square.
- The density of the data points in this space (big square) is .
- We have on average points in each unit square.
- Repeat the plotting again but this time set (i.e. space) and partition the cube with side length of into unit cubes.
- Then we scatter uniformly generated points with coordinates in the big cube.
- This time the density of the data points in this space (big cube) is .
- We have on average points in each unit cube.
- As you can see, as gets bigger and bigger with limited number of the existing data , many of this squares and cubes (hyper-cubes in bigger dimensions) will be empty (pay attention to the exponential growth of the number of hyper-cubes i.e. ) whereas the number of the data points does not grow exponentially (as shown in the figure below). This results in a sparse representation of the data points (on average data points for each hyper-cube!) and could cripple many algorithms because these algorithms usually need
**sufficient**number of data points in a space with dimension but as as you will experience in the real world, most of the time, the set of data points is not**big enough**!

## PCA

Many methods have been introduced through the years for dimensionality reduction. Among them, we can consider PCA (Principle Component Analysis) and its derivatives the most common ones. In the simple version of PCA, we calculate the covariance matrix (that is symmetrical and semi-definite positive, so it eigen values are non-negative) of the data points and then decompose it to where
is a diagonal matrix of sorted eigen values of and is the matrix of corresponding eigen vectors.
After finding the eigen vectors (that are mutually perpendicular), it is an easy task to transform the old data to the new vector space knowing that we could compress a possibly huge amount of information (could say variation or variance) of the orginal data is preserved in a few components of the transformed points.

Here the real numbers representing the eigen values are associated with the amount of information that the corresponding dimension contain, so after the calculations we take **k dimensions** of the all dimensions with the biggest eigen values and dump the rest!

**Note**: In a ** representational point of view**, by applying PCA we reduce the dimensions but this does not mean that we dumped the irrelevant features and preserved the relevent ones.
The main benefits of applying PCA are

- to reduce the computational cost subject to the fact that the most information be preserved,
- and solve the problem of having not enough data points in a relatively big space.

Despite the benefits of PCA we should not forget that the chosen set of dimensions can not guarantee better performance in a ** predictor’s point of view**.

## Sample Codes

```
import numpy as np
from sklearn.decomposition import PCA
X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])
pca = PCA(n_components=2)
pca.fit(X)
PCA(copy=True, n_components=2, whiten=False)
print(pca.explained_variance_ratio_)
```

## نظر ها

## Tommie

Hmm is anyone else experiencing problems wiith the pictures on this blog loading? I’m trying to find out if its a problem on my end or if it’s the blog.

Any suggestions would be greatly appreciated.

## Saundra

Awesome! Its genuinely amazing post, I have got mjch clear idea about from this article.

## Jenna

Greetings!This is my first visit to your blog! We are a group of volunteers and starting a new projject in a community in the same niche. Your blog provided us valuable information to work on. You have done a wonderful job!

## Aileen

Hi! I know this is kinda off topic but I was wondering which blog platform are you using for this site? I’m getting tired of Wordpress because I’ve had problems with hackers and I’m looking at options for another platform. I would be great if you could point me in the direction of a good platform.

## Essie

I’ll right away grasp your rss feed as I can’t find your e-mail subscription hyperlink or newsletter service. Do you’ve any? Please allow me recognize so that I may just subscribe. Thanks.

## Stanley

I couldn�t refrain from commenting. Well written!

## Sarah

Thanks for finally writing about >Curse of Dimensionality And Dimensionality Reduction - ووفورو <Loved it!

## Delores

At this time I am ready to do my breakfast, later than having my breakfast coming yet again to read more news.

## Merri

Pecuiar article, totally what I needed.

## Jack

I do not even understand how I ended up here, however I assumed this submit was good. I don’t know who you’re but certainly you are going to a famous blogger if you aren’t already. Cheers!

## Quentin

Howdy! I could have sworn I�ve visited this blog before but after looking at a ffew of the articles I realized it�s new to me.

Nonetheless, I�m definitely happy I stumbled upon iit and I�ll be book-marking it and cheking back often!

## Mireya

Great blog here! Also your web site loads up very fast!

What web host are you using? Can I get your affiliate link to your host? I wish my website loaded up as quickly as yours lol

## Quinton

Thanks for finally writing about >Curse of Dimensionality And Dimensionality Reduction - ووفورو <Liked it!

## Hermine

Mayy I simply just say what a comfort to find a person that actuallyy understanrs what they aare discussing on the net. You certainly know how to bring an isssue to light and make it important. A lot more people really need to read this and understand this side oof the story. I can’t believe you are not more popular since you most certainly possess the gift.

## Dante

An intriguing discussion is worth comment. I think that you ought tto write more about this subject, it may not be a taboo matter but typically people don’t speak about such subjects. To the next! Kind regards!!

## Nancy

Howdy! I’m at work browsing your blog from my new iphone!

Just wanted tto say I love reading your blog annd look forward to all your posts!

Carry on the fantastic work!

## Jordan

You really make it seem so easy with your presentation but I find this topic to be realoy something that I think I would never understand. It seemms too complex and very broad for me. I am looking forward for your next post, I’ll try to get the hang of it!

## Lee

Hey there! Quick question that’s totally off topic. Do you know hoow to make your site mobile friendly? My site looks weird when browsing from mmy iphone4.

I’m trying to find a theme or plugin that might bee able to resolve this problem. If you have any suggestions, please share. Thanks!

## Fleta

Good day! I could hhave sworn I’ve been to this blog before but after reading through some of the post I realized it’s new to me. Anyhow, I’m definitely delighted I found it and I’ll be book-marking and checking back frequently!

## Marietta

I’d lile to thank you for the efforts you’ve put in penning this blog. I aam hoping to view the same high-grade content from yoou iin the future as well. In truth, your creative writing abilities has motivated mme to get my own website noow ;)

## Emile

This post will help the internet people for building up new web site or even a blog from start too end.

## Irma

Howdy! This blog post could not be written any better! Looking at this article reminds mee of my previous roommate! He always kept preaching about this. I will forward this post to him. Fairly certain he’ll have a very good read. I appreciate you for sharing!

## Bertie

What’s up, after reading this awesome post i am also delighted to share my experience hwre with mates.

## Tom

This information is priceless. Where can I find out more?

## Carey

Quality confent is tthe key to be a focus for the people to pay a quick visit the wweb page, that’s what this site iss providing.

## Kerri

Hello, constantly i used to check website posts here early in the dawn, as i like to learn more and more.

## Tiffiny

It’s a shame you don’t have a donate button! I’d definitely donate to this ougstanding blog! I guess for noww i’ll settle for book-marking and adding your RSS feed to my Google account. I look forward to fresh updates and will shae thiis siute with my Facebook group. Talk soon!

## Alana

I like the valuable info you provide in your articles. I will bookmark your weblog and check again here frequently. I’m quite certain I will learn lots of new stuff right here! Best of luc for the next!

## Corey

This is very interesting, You are a very skilled blogger. I’ve joined your rss feed and look forward too seeking more of your wonderful post. Also, I’ve shared your website in my social networks!

## Ricky

Every weekend i used to visit this web page, as i wish for enjoyment, for the reason that this this website conations genuinely nice funny daa too.

## Sherlene

Whats up vwry nice web site!! Man .. Beautiful ..Amazing .. I’ll bookmark your website and take the feeds also? I’m satisfied to seek out a lot of useful info right here in the post, we need develop more techniques on this regard, thanks for sharing. . . . . .

## Rosalinda

Would love to perpetually get updated great website!

## Kim

What’s up to every body, it’s my first ggo to see of this website; this website consists of remarkable and actually excellent material desiyned for visitors.

## Normand

It’s hard too come by knowledgeaqble peoole about this topic, however, you seem like yoou know what you’re talking about! Thanks

## Lori

Yoou made some really good points there. I checked on the internet to fiind oout more about the issue and found most people will ggo alpng with your views on thiis website.

## Aurora

Excellent article. Keep posting such kind of info onn your blog. Im really impressed bby your blog. Hey there, You have performed a great job. I’ll definitely digg it and personally recommend to my friends. I’m confident they will be benefioted from this website.

## Aurora

Simply want to say your article is as astonishing.

The clearness foor your publish is ust nice and that i culd assume you’re knowledgeable iin this subject. Well with your permission let me tto seize your feed to stay up to date with drawing close post.

Thank you 1,000,000 and please keep up tthe enjoyable work.

## Claudio

Oh my goodness! Amazing article dude! Manny thanks, However I am experiencing problems with your RSS. I don’t know why I can’t join it. Is there anyone else getting the same RSS issues? Anyone that knows the solution can you kindly respond? Thanx!!

## Mervin

Hello! Someone in my Facebook group shared this website with us so I came to check it out. I’m definitely enjoying tthe information. I’m book-marking and will be tweeting this to my followers! Outstanding blog and superb design.

## Arnold

We are a gaggle of volunteers and startijng a brand neew scheme in ouur community. Your web site offered us with valuable info to work on. You’ve done a formidable job and our whoole neighborhood shall be thankful to you.

## Ryder

Thanks designed for sharing such a good idea, piece of writing is pleasant, thats why i have read it entirely

## Finlay

Hmm is anyone else experiencing problems with the pictures on thgis blog loading? I’m trying to determine iif its a problem on my end or if it’s the blog. Any feedback would bee greatlyy appreciated.

## Cecila

Way cool! Somee very valid points! I appreciate yoou penning this write-up plus the rest of the website is very good.

## Mariana

I am sure this piece of writing has touched all the internet users, its really really fastidious article oon building up new weblog.

## Celia

I really liie it whenever people come together aand share opinions. Great blog, keep itt up!

## Wally

Heyy I know this is off topic but I was wondering if you knew off any widgets I could add too my blog that automatically tweet my newest twitter updates.I’ve been lookjng for a plug-in like this for quite some time andd was hopinng maaybe you would have some experience with somethig like this. Please let me know if you run intto anything. I truly enjoy reading your blog and I look forward too your new updates.

## Rosetta

May I simply say what a comfort too discover somebody that actually understands what they are discussing on the net. You actually realize how to bring an issue to light and make it important. More and more people must look at this and understand this sife of your story. I can’t believe you are not more popular since you definitely possess the gift.

## Herbert

hi!,I like your writing very so much! percentage we keep in touch extra about your article on AOL? I require a specialist on this area to resolve my problem. Maybe that is you! Having a lopk ahead to peer you.

## Francisca

Thans too my father who tld me regarding thijs website, this website iss actually amazing.

## Tiffany

Hello, I thibk your website might be havig browser compatibility issues. When I look at your website in Chrome, it looks fine but when opening in Internet Explorer, iit has some overlapping. I just wanted to give you a quick heads up! Other then that, superb blog!

## Colette

You’re so cool! I don’t think I have read through anything like this before. So good to find someone with a few oriinal thoughts on this subject.

Really.. thank you for starting this up. This website is one thing that’s needed on the internet, someone withh a bit of originality!

## Ellis

Every weekend i used to go to see this site, because i wish for enjoyment, as this this web site conations actually pleasant funny material too.

## Alena

This paragraph provides clear idea in support of the new viewers of blogging, that truly how to do running a blog.

## Renaldo

It’s going to be end of mine day, however before end I am reading this wonderful article to improve my experience.

## Iona

Thanks for the marvelous posting! I genuinely enjoyed reading it, you could be a great author.I will make certain to bookmark your blog and will clme back sometime soon. I want to encourage continue your great writing, have a nixe morning!

## نظر خود را بنویسید

ایمیل شما منتشر نخواهد شد. فیلد های الزامی *