Search
rss Logo
If you can read this, you are still awesome
5/11/2012
Hello there! you seem in a good mood. Any reason? Nah, I mean other than it being Friday. Any new breakthroughs with data mining? Not really. I've been going back and forth between looking for optimizations to my random forest implementation and trying out some new ideas of mine. So far nothing has really panned out.
For the last few days, I have been watching all of the foamy (the squirrel) shortsThere are a LOT of them 300+. Some of them are really good but most are kinda light on the laughshe lies they are awesome. Maybe I'm just saturated. Either way it's something to do while running tests and I enjoy them on the whole.
For some odd reason, today I find myself dwelling on one of my favorite "tricks" in compression. So for no particular reason other than that, I'll share Burrows–Wheeler transform. It occurs to me just now, (related to compression, not data mining) that maybe I could redefine what "sorting" is slow down there cowboy and far improve the transformation. That is if you take the last column after the transformation you can reverse out the transformation. This would be true for ANY mechanism that reordered as long as the mechanism was defined rigidly. I bet someone already did that. probably. most simple data compression is so thoroughly explored Maybe there is some sort of application in my random forest implementation... hmmm...
Garbage in ... Garbage Out ... same old song and dance
5/8/2012
Yeah.... So, I've been struggling with understanding why my competitive score was so low. That is, the score based on results I submitted to kaggle for the most recent competition I'm in. You see, when I run a test locally my algorithms would be scored and show a result that looked good. However, when I submit my results it would always score fairly badly. Like really badly. And last night I finally said "ok, I'm gonna figure out what the deal is". It turns out while my training data was fine which I use for local testing, my test data was all kinds of messed up. The columns were off by 1 and 1 column was missing entirely. Needless to say, this was the equivalent of sending in random data and trying to score it.hah!
Until today, had you ever sent in good results? Ah.. nope. Does that mean you might have skipped over a better algorithm? Probably not. Since my training data was OK, and I was testing locally with said training data, I got a good idea of how each of my algorithms should perform. The current mechanism I'm using is, by that standard, out performing everything I've done before. So even though I don't know, really, how well my old algorithm were doing, I know they were worse than what I've currently got. So I guess it's all OK.:)
How are you doing? You can see for yourself http://www.kaggle.com/c/bioresponse/leaderboard I'm the user j_scheibel. My current score is 0.46041 which is JUST above the random forest benchmark. Position 197bleh :( still a lot of people ahead of me. but at least now I'm on the positive side of the top benchmark! Really the difference between the top performer and my isn't as much as you think. But like so many other things... it becomes increasingly hard to squeeze out a little more performance when you hit a certain wall. I still have a long time before this contest is over and suddenly being much higher up on the ranking is very motivating.
Another day another line of code
5/4/2012
It's Friday! Doesn't that make you happy?!!? YES! shortest blog ever? Nah... here's what happened last night. I double checked my code, found a few things that could be done far more efficiently. I also double checked that I was using the best values to split on. I also ran a test or 10 to make sure everything was working well. It seems to be hooray! My big test ran over night but it didn't quite finish. So I still don't have a new set of numbers to submit to kaggle. bleh
What will you do next? Well, I actually think there must be lots of improvements to be made still, just in making my program the "stock" version of Random forest. My results from little tests still don't seem good enough to get a really good placement overall say, the top 170 aka the baseline random forest result the show. If I can figure out what I'm doing wrong, I'll start in on my first big modification I want to speed up mechanism (and hopefully accuracy) by selecting multiple attributes instead of one attribute to split on. And I want change the way it works in terms of the smoothness of the generated results. Think in terms of selecting points instead of blocks and then finding the line between two points. That's not how i'll do it per say, but it's an easy was to visualize it.
Pay no attention to the man behind the curtain!
5/3/2012
First a little entertainment.isn't this whole thing supposed to be entertainment???indeed! I've been watching a starcraft 2 video cast series called "when cheese fails" They also have a will cheese fail series. The two casters watch a video sent in by viewers where someone tries to win a game using an attack that is typically either a cannon rush, 6 pools attacks or a proxy base. Thus far I think this is the best one I've seen. If you like starcraft 2, hopefully, you will enjoy it. www.youtube.com/watch?v=0mPGYeuU2Oc
As for the random forests stuff. I finally got it working last night. It definitely looks like the results are going to be way better.=D I'm not 100% sure I've got everything optimized for the best traditional result yet. It's hard to tell because it runs so slow and as it currently operates and I was only able to do a test run with 800 "trees" in the forest. I'm pretty sure i need a couple thousand trees. But before I can do that I need to speed it up. I think I would like to take a crack at explaining random forest as I understand it now that I've got a version working. So here goes...
Random Forest Explained (core concept)

So let's say you are a doctor and you have a bunch of patients. For each patient you have height, weight, gender and age. 1 in 10 of your patients developed the flu last year. You know who they are. Predict who will get the flu this year. To do this you take all the data for last year and randomly select a few properties about the patient technically you randomly select the square root of the number of attributes ... but we have 4 which is 2. Let's say Age and Weight. Then for both of them you find out which has the highest entropy or some system of choosing one with a lot of variation. You process that one. Let's say it was age. Next You find the correct age to use for splitting the group in half. That is you will pick an age that will move half the entropy to one side and the other half to the other.That is, the exact mechanism by which you pick the split is also based on entropy not on an strict average number of occurrences.. You now have a two groups of people.
You then repeat this process on the two groups separately and then again on their groups and then their subgroups etc.. (subdividing them further and further), until you get to either a group with only people who were all sick, healthy or have identical information. You stop dividing at this point. It is completely valid to randomly select the same property again and again. Or possibly once then some other property a few times then come back to it. So you may end up splitting by age 3 times in a row or more... or maybe you wont split by age at all. The property you select to do the division must be random each time. Regardless of the choice, each time you divide you are halving the number of possible results to be processed in the next iteration. Eventually you will get to a scenario for stopping.
You have constructed 1 tree. This can be used to score the patents for this year. Granted, some of the stats for a patient are the same year to year, but age always changes. You might also have lost some patients and gotten some new ones. You can score each patient by starting at the top and following the logic of where they end up. "Oh their age is less than this, so they go to this question." When you reach a stopping spot you have an answer for that patent.
The thing is for a random forest you make 100s and 100s of these trees. Since there are lots and lots of different ways the data can be sliced up you are trying to build a tree for a good selection of those possibilities. Each time you follow the logic of a particular tree you get a vote for a possible answer. When you have take 1 patient down all the trees you will have a number a certain number of votes for will get the flu and won't get the flu. This is that patient's odds of getting the flu.
That doesn't sound so hard... At it's core it isn't. The trick lies in finding the right splitting point. Having properties that are independent of each other (don't for instance take age of the patient and decade they were born in. Those both relate to each other.) Oh and I suppose making the process fast. The last one helps, because the more trees you have the more accurate the results. You can test with 1 tree, but more is always better statistically. There are a world of improvements that can be made but that's outside the scope of that explanation, more on some of those another time.
"Random Forest" your time has come.
4/28/2012
O_o ? Random forest is a data mining technique that has been around quite a few years. To be fair I only first read about it about 6 months ago, but when I saw it I knew someday I would be implementing it. What is random forest? Well, you can look it up on wikipedia but I think you'll be disappointed in the explanation (I was). With a little searching I found this nice explanation. copied in its entirety to preserve it lest the linking page goes away. Random Forest explained
Author:Edwin Chen

Suppose you're very indecisive, so whenever you want to watch a movie, you ask your friend Willow if she thinks you'll like it. In order to answer, Willow first needs to figure out what movies you like, so you give her a bunch of movies and tell her whether you liked each one or not (i.e., you give her a labeled training set). Then, when you ask her if she thinks you'll like movie X or not, she plays a 20 questions-like game with IMDB, asking questions like "Is X a romantic movie?", "Does Johnny Depp star in X?", and so on. She asks more informative questions first (i.e., she maximizes the information gain of each question), and gives you a yes/no answer at the end.

Thus, Willow is a decision tree for your movie preferences.

But Willow is only human, so she doesn't always generalize your preferences very well (i.e., she overfits). In order to get more accurate recommendations, you'd like to ask a bunch of your friends, and watch movie X if most of them say they think you'll like it. That is, instead of asking only Willow, you want to ask Woody, Apple, and Cartman as well, and they vote on whether you'll like a movie (i.e., you build an ensemble classifier, aka a forest in this case).

Now you don't want each of your friends to do the same thing and give you the same answer, so you first give each of them slightly different data. After all, you're not absolutely sure of your preferences yourself -- you told Willow you loved Titanic, but maybe you were just happy that day because it was your birthday, so maybe some of your friends shouldn't use the fact that you liked Titanic in making their recommendations. Or maybe you told her you loved Cinderella, but actually you *really really* loved it, so some of your friends should give Cinderella more weight. So instead of giving your friends the same data you gave Willow, you give them slightly perturbed versions. You don't change your love/hate decisions, you just say you love/hate some movies a little more or less (you give each of your friends a bootstrapped version of your original training data). For example, whereas you told Willow that you liked Black Swan and Harry Potter and disliked Avatar, you tell Woody that you liked Black Swan so much you watched it twice, you disliked Avatar, and don't mention Harry Potter at all.

By using this ensemble, you hope that while each of your friends gives somewhat idiosyncratic recommendations (Willow thinks you like vampire movies more than you do, Woody thinks you like Pixar movies, and Cartman thinks you just hate everything), the errors get canceled out in the majority. Thus, your friends now form a bagged (bootstrap aggregated) forest of your movie preferences.

There's still one problem with your data, however. While you loved both Titanic and Inception, it wasn't because you like movies that star Leonardio DiCaprio. Maybe you liked both movies for other reasons. Thus, you don't want your friends to all base their recommendations on whether Leo is in a movie or not. So when each friend asks IMDB a question, only a random subset of the possible questions is allowed (i.e., when you're building a decision tree, at each node you use some randomness in selecting the attribute to split on, say by randomly selecting an attribute or by selecting an attribute from a random subset). This means your friends aren't allowed to ask whether Leonardo DiCaprio is in the movie whenever they want. So whereas previously you injected randomness at the data level, by perturbing your movie preferences slightly, now you're injecting randomness at the model level, by making your friends ask different questions at different times.

And so your friends now form a random forest.

So you are gonna implement that? But use what you've learned up to this point to try and improve it? Yes. Yes I am and it will be glorious Damn Straight.
a new contest and more scoring stuff
4/27/2012
I went and had a look at kaggle.com to see if I had missed any new contests starting. I had!wooo wooo! They have a new contest which is right up my ally, Predicting a Biological Response. This contest is setup exactly like the previous. That is, I have a training set of data and test set of data. All attributes for each datum are provided. The datum is classified either as being responsive "1" or not responsive "0". I can for my purposes consider that a single group that you are either in or not. Since they are polar opposites and you can never be in both groups, if I did consider it two groups I end up with the same result in reverse on each line (which would score to a one or a zero any way).
Great, so have you submit a result? It sounds like other than loading the data there is no work to do here Yes, I've loaded the data and submitted a result. But unfortunately there is work to do here. my initial submission put me dead last in 338 teams.hahahhahah loser but there is a very good reason.we already covered that. loser The way they are scoring isn't built for predicting a 1 or a 0. they are using a log loss model (you can read up on it here), which basically produces an infinity when you guess wrong with a 100% certainty. Now my score wasn't infinity, so they are clearly considering an infinity to be a very large number but not infinity. The moral of the story is you can dramatically improve your score just by moving the score over a tiny bit off of 1 or 0 (assuming you guessed at least 1 wrong).
And, how did you score? Initially, I came in dead last of 338 teams with a score of 14.+++ the number one score is currently around .40++ . When I moved my predictions over a tiny bit my score dropped to 4.+++ . any other details?There are still 48 days left in the contest it's about half over. The top prize is $10,000 dollars and they expect to see how you did it to award the prize (as well as an explanation).
What I need to get busy doing is figuring out a good way to turn my predictions in to a percentage chance. I think I know how I'm going to do it at first at least. I'll build a normal curve of the prediction's distance from 0 typically I score each group in terms of positive and negative distance from zero. positive is a success, negative is not. Then turn that in to a percentage. So zero would be a 50% which is exactly right. and -100 would be closer to 0 and +100 would be closer to 1. I'll probably do all my calculations then artificially scale the number down to 0.02 to 0.98 just to make sure no infinities crop in to the scoring.