_{Dear lord, this is about data mining again isn't it? And I bet it's more crap without pictures}It so is!!^:DAll right, what innocuous thing have use done this time? Well, I've made further improvements, and my locally tested score is getting better. I'll share some numbers but first a word about scoring...._joy.

there are really 3 scores I look at. They all approach 1 one as the algorithm get's closer and closer to the perfect prediction but each score measures different things so, depending on what you are looking for 1 may be preferable over the other. They are: accuracy, F-Score average and F-Score total.

Accuracy is simply the percentage correct over the total possible answers. This may seem like the best way to score, but it's not. Here's why, if you are picking yes or no for each item and 90% of the answers are no. Then naively picking no every time will produce a 90% result. The thing is the value is in picking out the Yeses. not the Nos. I list it mainly for insight into, well, overall accuracy. if my program is not naively picking all one or the other there is going to be a uniformity of error. So, when I see 90% (in the above example) it's generally going to be more like 90% of the yeses were right and 90% of the nos were right. not always mind you but generally.

F-Test scores are more complicated but are essentially a measure of explained variability divided by unexplained variability ... times 2. if you don't double the score it's actually a 0 to 0.5 score 0 to 1 is what I want. So what is variability? Wow, ok... I don't want to give a long winded explanation here that no one will care about,read or understand. So I'm going to jump to the chase and show the final result and let you stew on it. Then explain a little.

Consider That userLabels is the set of Yeses we selected. Also consider trueLabels to be the real set of answers (the correct yeses). commonSize is the overlap (what we got right). Or, another way to think about it, really there are 4 possible out comes. We predict yes, and the answer is yes ^yea!_{group a}. We predict yes and the answer is no^boo!_{group b}. we predict no and the answer yes^boo!_{group c}. we predict no and the answer is no ^yea!_{group d}. commonSize is group a. userLabelsSize is group a and group b. trueLabelsSize is group a and group c. Using these values we get our measure.

But you have two measures for f-test? I measure two different sets of solutions. There are many different groups being predicted. I measure each result and average them for one score and for the other I measure results of everything all at once.

So...? when the contest ended I had a best score (f-test average) of 0.439 the winning score was 0.535 _{the score to beat!}. I don't have the answer set so I can't continue to do the exact same scoring ^{this changes at the end of the month when they give us the answers}. To continue working on this, I split my training data in half and use half to do testing with and have to train with. This made my score worse but at least I now have a mechanism to self score. my base method (which scores 0.43 not 0.439. 0.439 has an improvement on the base method) scores 0.954139 (accuracy) 0.38439 (f-test avg) 0.40644 (f-test overall). My most recent best method has since score 0.943880 (accuracy) 0.41057543 (f-test avg) 0.41424 (f-test overall). which is a nice improvement! note that the actual accuracy has dropped and the f-test score has gone up. this is due to me actually picking out more correct answers but getting more of the "no"s wrong, but since yeses are so much more rare it's a gain. I think that's enough for today!

Not to much to say in terms interesting news. I've been quietly working on improving the DMT (data mining tool) ^{acronyms for the win!} _{and for the lazy}. Thus far I have nothing to show for it. I could go on about what I'm doing exactly but really it would be far more easily understood with pictures. So at some point I'll throw them up for you all to see. Just not right now.

I did have one idea that is worth sharing. Once I get this thing working well, I think I'm going to apply it to financial analysis. How? First, I'll grab a list of stocks ^{probably everything from AMEX, NYSE and NASDAQ} and their daily stock prices for a couple years. Then I'll setup attributes for each stock representing key financial statistics. I think those statistics all have to be relative to stock price or possibly market capitalization. This is done so that there is some reference point day to day on how good that attribute is. I'll probably also have to organize the data by industry. For example, I won't want to compare agricultural to pharmaceutical. They may be similar but one industry works off of a difference set of ideals in valuation than the other. I can try mushing it all together to see if it does perform similarly but in case it doesn't they will be stored separately. Finally, I setup some sort of criteria for groupings of buy and sell (hold will be everything else) based on how a stock ended up performing in the following months. Once that's all setup, I run the tool over the data and presto _{if you did everything right} stock picks.

Aren't other people doing that sort of thing? Yep! It'll be my take on it. I'll just be one of many... many people doing that sort of thing. I need to get my algorithm in the same league as top performers in the last contest first though. And it'll never be perfect because the world is full of random events that make the best analysis fail. Hurricanes, early freezes, earthquakes, war... etc. all of those things can throw a monkey wrench in to the predictions. Regardless, once everything is performing really well, I'll attack that stock data and see what I can do.

I put together an RSS feed for the site. No idea if it works. I'm not much of an RSS guy. I've had to work with them before but I don't actively use RSS for notifications or reading articles. So, if there are any problem with it, sorry in advance. If not well great! :D

So, I was in Vegas for 4 days. And it was glorious! I went for a wedding and that would have been awesome by itself. but ya know, I always have a good time there and that made it that much more fun. I have to admit though, I could do with about a 10 year break from the place._blasphemy! Give it some time to change and not be so fresh in the memory. I've been 3 times in the last 10 years now and while I always have a blast, there is less and less to do that is truly new and exciting. I've actually never been there for a convention. So, I suppose I'm due for that. If that happens though, It'll be for work and really I wont be there for "fun".

Other than that not to much to share. I did find out that I will be getting the test data results for the competition I just worked on. ^hooray! With those I can continue my work on improving the methods I was working on using the exact same framework and scoring I previously had. I wont get it till April 30th, but that's OK. I probably could use the separation from it anyway. Also, with regards to my next attack on the data. I'm not entirely sure how I'm going to go about making my decisions on which data goes where my new modeling I have planned. Eh? I'll talk more about it another time.

The bugs are fixed. You can now search and look at old entries_{what few there are} to your hearts content!^hooray!

So what news? The most recent data mining contest ended. I didn't do well at all. 79th out of 126. I ended up slicing the data quite a few different ways but I always seemed to end up sitting with a score between 0.419 and 0.430. My best score of 0.439 was really refining a particular result more than it was finding some great revelation. I did figure a few things out. I'm not sure if it's remarkable or not but I made my best result off only slicing the data in to two subgroups._eh? Each group you are trying solve has two sets of data. 1 set that is in-group and 1 set that is out. So for each of the 83 groups I make 2 subgroups out of the whole thing with respect to any 1 group. I then used that to identify where the groups appear. Through the course of events I also tried some variations of nearest neighbor and few other techniques that involved making many groups. But they always produced nearly the same result and were slower. Typically, they scored slightly worse. In the end my two subgroups method seemed the best. I suppose I learned/figured out quite a bit about that kind of data manipulation (aggregates) which in itself is something.

So now that it's over you have all kinds of new ideas right? You bet! ^*grin* So even though the contest is over I'll probably explore those ideas anyway. Hopefully, the guys running the old contest will release the test data answers and I can test with that data. If not I'll just mock something that isn't quite as good from the data they did provide. If it all works out I'll use the ideas next time I participate in a contest.

Also in the news, I managed to upgrade this site's code to MVC 3. Hopefully, you don't even notice. There are still a few issues. clicking on an entry doesn't take you to the entry, the search still goes to the old page and the calendar still seems to be using the old URLs. Also, unfortunately, I still haven't set up an RSS feed. I'll fix most of that Tuesday night. But I'll probably another week or two getting the RSS finished.