I went and had a look at kaggle.com to see if I had missed any new contests starting. I had!^{wooo wooo!} They have a new contest which is right up my ally, Predicting a Biological Response. This contest is setup exactly like the previous. That is, I have a training set of data and test set of data. All attributes for each datum are provided. The datum is classified either as being responsive "1" or not responsive "0". I can for my purposes consider that a single group that you are either in or not. Since they are polar opposites and you can never be in both groups, if I did consider it two groups I end up with the same result in reverse on each line (which would score to a one or a zero any way).

Great, so have you submit a result? It sounds like other than loading the data there is no work to do here Yes, I've loaded the data and submitted a result. But unfortunately there is work to do here. my initial submission put me dead last in 338 teams._{hahahhahah loser} but there is a very good reason._{we already covered that. loser} The way they are scoring isn't built for predicting a 1 or a 0. they are using a log loss model (you can read up on it here), which basically produces an infinity when you guess wrong with a 100% certainty. Now my score wasn't infinity, so they are clearly considering an infinity to be a very large number but not infinity. The moral of the story is you can dramatically improve your score just by moving the score over a tiny bit off of 1 or 0 (assuming you guessed at least 1 wrong).

And, how did you score? Initially, I came in dead last of 338 teams with a score of 14.+++ the number one score is currently around .40++ . When I moved my predictions over a tiny bit my score dropped to 4.+++ . any other details?There are still 48 days left in the contest ^{it's about half over}. The top prize is $10,000 dollars and they expect to see how you did it to award the prize (as well as an explanation).

_{Dear lord, this is about data mining again isn't it? And I bet it's more crap without pictures}It so is!!^:DAll right, what innocuous thing have use done this time? Well, I've made further improvements, and my locally tested score is getting better. I'll share some numbers but first a word about scoring...._joy.

there are really 3 scores I look at. They all approach 1 one as the algorithm get's closer and closer to the perfect prediction but each score measures different things so, depending on what you are looking for 1 may be preferable over the other. They are: accuracy, F-Score average and F-Score total.

Accuracy is simply the percentage correct over the total possible answers. This may seem like the best way to score, but it's not. Here's why, if you are picking yes or no for each item and 90% of the answers are no. Then naively picking no every time will produce a 90% result. The thing is the value is in picking out the Yeses. not the Nos. I list it mainly for insight into, well, overall accuracy. if my program is not naively picking all one or the other there is going to be a uniformity of error. So, when I see 90% (in the above example) it's generally going to be more like 90% of the yeses were right and 90% of the nos were right. not always mind you but generally.

F-Test scores are more complicated but are essentially a measure of explained variability divided by unexplained variability ... times 2. if you don't double the score it's actually a 0 to 0.5 score 0 to 1 is what I want. So what is variability? Wow, ok... I don't want to give a long winded explanation here that no one will care about,read or understand. So I'm going to jump to the chase and show the final result and let you stew on it. Then explain a little.

Consider That userLabels is the set of Yeses we selected. Also consider trueLabels to be the real set of answers (the correct yeses). commonSize is the overlap (what we got right). Or, another way to think about it, really there are 4 possible out comes. We predict yes, and the answer is yes ^yea!_{group a}. We predict yes and the answer is no^boo!_{group b}. we predict no and the answer yes^boo!_{group c}. we predict no and the answer is no ^yea!_{group d}. commonSize is group a. userLabelsSize is group a and group b. trueLabelsSize is group a and group c. Using these values we get our measure.

But you have two measures for f-test? I measure two different sets of solutions. There are many different groups being predicted. I measure each result and average them for one score and for the other I measure results of everything all at once.

So...? when the contest ended I had a best score (f-test average) of 0.439 the winning score was 0.535 _{the score to beat!}. I don't have the answer set so I can't continue to do the exact same scoring ^{this changes at the end of the month when they give us the answers}. To continue working on this, I split my training data in half and use half to do testing with and have to train with. This made my score worse but at least I now have a mechanism to self score. my base method (which scores 0.43 not 0.439. 0.439 has an improvement on the base method) scores 0.954139 (accuracy) 0.38439 (f-test avg) 0.40644 (f-test overall). My most recent best method has since score 0.943880 (accuracy) 0.41057543 (f-test avg) 0.41424 (f-test overall). which is a nice improvement! note that the actual accuracy has dropped and the f-test score has gone up. this is due to me actually picking out more correct answers but getting more of the "no"s wrong, but since yeses are so much more rare it's a gain. I think that's enough for today!

Not to much to say in terms interesting news. I've been quietly working on improving the DMT (data mining tool) ^{acronyms for the win!} _{and for the lazy}. Thus far I have nothing to show for it. I could go on about what I'm doing exactly but really it would be far more easily understood with pictures. So at some point I'll throw them up for you all to see. Just not right now.

I did have one idea that is worth sharing. Once I get this thing working well, I think I'm going to apply it to financial analysis. How? First, I'll grab a list of stocks ^{probably everything from AMEX, NYSE and NASDAQ} and their daily stock prices for a couple years. Then I'll setup attributes for each stock representing key financial statistics. I think those statistics all have to be relative to stock price or possibly market capitalization. This is done so that there is some reference point day to day on how good that attribute is. I'll probably also have to organize the data by industry. For example, I won't want to compare agricultural to pharmaceutical. They may be similar but one industry works off of a difference set of ideals in valuation than the other. I can try mushing it all together to see if it does perform similarly but in case it doesn't they will be stored separately. Finally, I setup some sort of criteria for groupings of buy and sell (hold will be everything else) based on how a stock ended up performing in the following months. Once that's all setup, I run the tool over the data and presto _{if you did everything right} stock picks.

Aren't other people doing that sort of thing? Yep! It'll be my take on it. I'll just be one of many... many people doing that sort of thing. I need to get my algorithm in the same league as top performers in the last contest first though. And it'll never be perfect because the world is full of random events that make the best analysis fail. Hurricanes, early freezes, earthquakes, war... etc. all of those things can throw a monkey wrench in to the predictions. Regardless, once everything is performing really well, I'll attack that stock data and see what I can do.

I put together an RSS feed for the site. No idea if it works. I'm not much of an RSS guy. I've had to work with them before but I don't actively use RSS for notifications or reading articles. So, if there are any problem with it, sorry in advance. If not well great! :D

So, I was in Vegas for 4 days. And it was glorious! I went for a wedding and that would have been awesome by itself. but ya know, I always have a good time there and that made it that much more fun. I have to admit though, I could do with about a 10 year break from the place._blasphemy! Give it some time to change and not be so fresh in the memory. I've been 3 times in the last 10 years now and while I always have a blast, there is less and less to do that is truly new and exciting. I've actually never been there for a convention. So, I suppose I'm due for that. If that happens though, It'll be for work and really I wont be there for "fun".

Other than that not to much to share. I did find out that I will be getting the test data results for the competition I just worked on. ^hooray! With those I can continue my work on improving the methods I was working on using the exact same framework and scoring I previously had. I wont get it till April 30th, but that's OK. I probably could use the separation from it anyway. Also, with regards to my next attack on the data. I'm not entirely sure how I'm going to go about making my decisions on which data goes where my new modeling I have planned. Eh? I'll talk more about it another time.

The bugs are fixed. You can now search and look at old entries_{what few there are} to your hearts content!^hooray!