Dim Red Glow

A blog about data mining, games, stocks and adventures.

making data mining better

I've been working on a new version (seems like it's been a while) of my data miner. the last one i worked on /got working used the logistic function to build curves that represented the left and right side of each splitter in the trees I built. This worked really well. I'm not quite sure it was state of the art or not but when combined with either random forests or with gradient boosting the results were better than my previous techniques.

So what's the new one do? well that would be telling :) . maybe a better thing to say is how much better is it? not much but it does seem to be better. some run of the mill tests on the last kaggle contest I worked on (Home Credit Default Risk) show it's producing around .74 auc compared to .73 with the old technique. "That's terrible!" no, it's not :) That is i'm just using 1 table they gave us (the main one). And I'm doing nothing special to it. i only created 1 feature. my results arent going to be really all that with so little work. That is, to me that seems pretty decent considering what it's working with. you can read the winner's solution with auc was .8057 on the leader board  here https://www.kaggle.com/c/home-credit-default-risk/discussion/64821

both with home credit default risk and santander value prediction I didn't really put the effort in I should have to get the training data setup. especially with santander. That one was starting to get interesting then they discovered the leak and i tried naively to implement it with basically 0 success and said. "meh. even if i get this working right.. i'm pretty far behind the curve. this has stopped being a contest I really want to do."

Another thing i've come to realize, the genetics can do the same thing my normal data mining does, but a better way it seems (at least for the run time) to use the genetics is to just make new features. I'm still working on mechanisms (in my head) for deciding how to determine a particular genetic out put is ideal. correlation seems ideal but also ideal is something that correlates highly with the solution but not with any given features. That is maximize one and minimize the other. I'm just not positive on the best fitness test to do that. Once you've done it once, you do it again and again. since each feature is independent they ~should~ add.

Incidentally, I'll be using the new aforementioned technique on my new for-the-public data mining webste ( Https://yourdatamine.com ). I've spent a lot of time cleaning up my code and moving it in to a wrapper so it's easily portable to the website. cleaning it up was good too, i got rid of LOTS of old code that either did nothing, wasn't used or  was being used but shouldn't. I also sped it up the load and made it so you can do new cross validations willy-nilly. previously i setup that in the database.  Now i can just pick a number of CV folds to do and the code will make new onest on the fly (or it can still use the DB). having a consistent cross validation does give a reproducible result but sometimes the random selection happens to be weird. so mixing it up can be good.

The website is going to get a new page that lets you know what kind of results you can expect from a training set but doing analysis on it using the on-the-fly cross validation and giving you expected error margins.

I made a little effort to get all that to run in memory. The processing was always done that way but the datasource used to come from a database (as i was mentioning).when i put the code in to a package I also wrote a class that directly imports the data in to the structures it needs. This has 2 upshots; 1 people can rest assured I dont keep or even care about their data. (i setup https for that very reason too). And 2 the website has no database to maintain. It's hard drive footprint never increases (though memory varies by current usage). if the site becomes popular, i'll make it so the requests will go to new instances/farms/application servers. Those will be able to be spun up easy peasy! so its a super scalable solution. (or should be)


new website on data mining and a neat video on heat death

Hello folks! Things keep moving forward (time is like that). I've been busy today. I've started to put together a website for data mining https://www.yourdatamine.com. This is a site that allows anyone with a few csv files or tab delimited files to do some data mining. I might monetize it later (make it pay per use) but till i gauge the interest I'm going to make it free. I need to finish it too of course. But I think I can have a basic version ready by end of day tomorrow. i only started on it today so that's something.

I also have continued to update projects on http://project106.com. I need to update it tomorrow as well (just to bring the weight loss section up to date and add a project for yourdatamine.com. But really things are going well there, even if updated sporadically.

Finally, I wanted to throw out a link to a video done by PBS space time on the blog's name sake.... the heat death of the universe. What? you didn't realize that's what dim red glow referred to? I .. I don't know you any more. *grin* seriously, the name dim red glow is a reference to the idea that at the very end of the universe all that will really be left are protons zipping around the universe slowly getting stretched out more and more as they grow fainter and fainter due to dark energy stretching them out over an ever increasing area. Eventually leaving nothing but a dim red glow. Anyway here you go, it's a good one  :) enjoy.

Vacations and Project 106

Real quick update.... not a whole lot going on right now.I went to aruba and vegas since i  last posted.

the trip to aruba was the worst vacation i've ever been on. why? well while nothing catastrophic happened lots of really crappy stuff did. here's the list. I hurt my back a week before going... making sitting for long times hard and a lot of things impossible. My connecting flight was canceled. I got stuck over night in chicago (despite it being on a new noon when the flight was canceled) which lost me a day of vacation (thanks American Airlines :( ) my next morning flight was at 5:30am. i usually get up around 11am... and was already on 3-4 hour sleep so I was exhausted in a way i shouldn't have been for a leisure vacation. They lose my bags. lesson learned pickup your bag if you have a canceled flight. dont leave it overnight.

My accommodations weren't a resort. This my fault but was a shock when i got there. I thought it was but it was a condo :( This means no comforts or even a gift shop. everything i need is about a mile walk away. Like you know, clothes i didnt have cause of the lost bag (thanks american airlines). There is no bar (wet or otherwise), no place to get clothes/gift shops, no restaurants, no casino, tv was basic cable, no... well anything except a pool and a nearby small public beach. even people didn't exist. the place was so empty. the pools were nice though. There was 1 near by restaurant that did a special 8 course meal thing i did. but otherwise it was a pretty giant waste/not a vacation.

The island is a desert, so not to much to look at (cactus and everything) and it was so windy! I skipped/missed 2 of the 3 excursions i planned because of my back and the canceled flight. My ear got clogged with wax Saturday night right before the 8 course meal (so i had this delightful ringing muffled sound in right ear for the rest of the trip). Turns out there are no pharmacies open in Aruba Sunday. Not that they really could have helped i didn't get it cleaned out till Wednesday at the doctors office. i had allergies Saturday night... and full on sickness Sunday night (flew back Monday). So yeah. all in all it was terrible/felt like work.

Vegas was the exact opposite. i stayed at the Belagio and while it was really nice they did have a few bad things that surprised me. the sink liked to suck air when someone else in the hotel flushed a toilet which means it would go "glug" in the middle of the night. the hall smelled of mold and clearly had a broken pipe right before i got there... so yeah odd for being a fancy hotel.

The pools were amazing as was the food. i saw 1 show (Michael Jackson one). I think i'm just done with musical shows cause it was kinda boring/not my thing. Also I played in the vegas GP which went terribly but they refunded my money. the did this to compensate us for waiting around. They had technical issues so we sat for like 2.5 hours doing nothing (they also gave me a free entry to another gp later that year. i was done with GPs but... free entry so yeah maybe i'll do one more)

I got pretty badly sunburned at the pool. apparently the shirt i wore down took off a lot of my suntan lotion :( let that be a lesson to you all; reapply at the pool. the pool though, amazing! drinks/good times/books/swimming/lots of people. yeah no complaints there.

Finally (unrelated) i've put together another website to track the various things I've been working on. check out project106.com when you get a chance. The site itself is just a living document about the rest of the projects. Many of which I'll mention here eventually.


wow talk about leaving people hanging

hah, sorry about that. 17 hours to go and i disappear! i did not win *grin* the contest really went all topsy turvy when they released the results. everyones score got worse and the people got rearranged. more than a few (myself included) people said their best result was not selected. I fell to sub 100. The winner was someone who moved up a lot, and i heard from at least 1 person in the forums that they had a better score than the winner but it wasnt selected.

Why did this happen? well, probably due to the size of the data. it was a really small sample. It was also possibly due to some deliberate sampling the contest runners did (where certain things weren't expressed in the training data) but really sometimes it's just bad luck too. But that still leaves me in a great position. i've not rested in the time between then and now. I've been applying the stuff i've learned. honed the code. improved the "Fitness" test.

Ah yes, so i experimented with all kinds of ways to score results.... ways to test the results, fit the results...etc. Right now the thing i think is best is to bag the data. train on the 63% of the data (and do the linear algebra for fitting on that). score on the 32%. use the score to evaluate your accuracy of the that stack and modify the weight the stack as it gets added to the final answer. I also score (fitness test) it by the same method i mentioned previously (self similar scoring) as well as a final modification I made the correlation coefficient's calculation internally.  this last one is the big and since it does no harm to share, i'll do that.

Normal Pearson correlation coefficient https://en.wikipedia.org/wiki/Pearson_correlation_coefficient basically takes a sum of all (x - avg(x))*(y -avg(y)) where x is the prediction and y is the answer. it then divides by the number of elements and divides again by the standard deviation of x and standard deviation of y. (multiplied together) My change is to leave that. then also do a 2nd calculation on the squared of (x - avg(x))*(y -avg(y)). then divide that by the number of elements and divide that by the variance of x and y again multiplied together  (aka standard deviation squared). I take that result and multiply it by the original value. this new value is my "true" coefficient. or at least close to the truth.

I actually left out a step.. i dont allow any value to be over  1. so if it does go over 1 i invert it. dont ask me why but this seems to be best.... that is .5 coefficient is roughly equal to 2 overall (in terms of value for analysis)  .33  = 3 ..etc. so when i multiply the two numbers together they are already in <= 1 state. I also tend to do an Absolute value on the original coefficient since negative or positive will get adjusted by the linear algebra (the squared is always positive).

So that's the skinny. that works best (at least so far). when i dont do that i get random over fittings to the training data.... spikes that 1 value here or there that correlate that make it look better than it actually is. The squaring deals with that. using cubes and higher powers might help, but after trying cubes in there, i saw no real value-add.

what am i using this stuff for? stock analysis :)


17 hours left...

There are 17 hours left in the transparent materials contest. despite lots of changes i havent managed to move my score in a few weeks. i have though made my code more efficient (add a few mechanisms that help it get to the best score faster). And I added a mechanism i think will help a lot when there is a larger data set. I can flag train data features to get passed forward to the linear algebra that fits the correlated value to the actual value. with this tiny dataset though all it does is seriously overfit.  So, I turned it off. I also tweeked the scoring mechanism to work very nearly (only 2 differences the final run splits the data in half and uses 2 different linear algebra computations. and it also dampens the results merging them with the average for each half.) to the final adjustments... basically i use the linear algebra in there too.

So where does that leave me. i'm doing one final run right now. it'll either be done or be close being done by 6 tomorrow. (i can submit it at any point along the stage of evolving to an answer). so maybe it'll give me a better score, but probably not. We'll see! regardless its look like i'll have a top 10% finish which is nice its been gosh 4ish years since i had one.





i guess things are looking up

Work still continues on the client/server code. i had some real issues automating the firewall access, but i think i got that sorted now. i set it aside after that (that was sunday) I'll probably look at it some more tonight.

I've been trying some slow gradients in the boosting part of the algorithm. i think the idea here is less about picking up other signals in the data and more about dampening the noise i introduce. instead of 70% of the answer put in. i'm doing 40% now. this is working pretty well. i get the feeling 20% might work better still. the idea being that if you are introducing noise you want your noise to be same "level" as the background noise, so on the whole you arent adding any (since noise ideally hovers around 0 and the various noises cancel). the signal while only at 20% is still signal and does what it's supposed to do. (identify the answers) the 40% run is still going, i'll probably start on the 20% run before i  get the client/server version done. 

I started looking at the data its using (just for fun) turns out its using a lot atom position data.... which is weird. I read that there is a standard "best" form for describing crystalline structures and these have already been optimized for that. So maybe conceptually certain atoms positions (atom 20's X position for example) are telling to the details of the crystal. but this seems wrong... at least a really terrible way to get at the data you want. I might do my 20% run without the XYZ data turned on just to see if it: 1, produces better results and 2, has results that make more sense.




bias bias bias...

Work continues on the client/server version of the genetic algorithm. but the pressing issue still remains, bias. the results i get locally at the end of a run are usually far better than the number 1 spot on the leader board... but my submissions are never that good. i think i found 2 things that might be causing it.

The first was some code i had in place from when i was trying to do rmse instead of correlation coefficient. I was calculating an average value to be used for "filler". the genetic algorithm has 2 major things going on. 1 is the prediction and the other is when the prediction is used.... when its not used it uses the average value (actually its not a hard fast switch but a weighting based on the a 2nd prediction going on.) the point is the value being used wasnt the average for that stack (its a series of stacks each adjusting for the error of the last) but the average for all stacks. the difference is the average of the current stack is usually 0 (first stack it is not). the average for all stacks is the average value of the training set's average. I dont know how much that matters cause of the linear alegbra at the end adjusted the correlation values to actual values... but it certainly didnt help.

The second was the idea that even with the self similar approach i'm still fitting to my training data... and all of it.... so bias is unavoidable. i might have a way to fix that. basically i'm going to treat the entire stack as one fitted answer on 63.2% of the data. (that's right this is bagging) by using the same exact training data portion over and over again as i fit my genetics to it (its still self similar too). I can at the end, when i'm done improving, take the remaining results and use those to figure out how i should adjust my correlated predictions. in short the hold out data becomes the unbiased mechanism i use to scale the correlated values. I could also use this to get an accuracy if I wanted. I might, but right now I'm just using it to do the scaling in an unbiased way.

So those 2 should help! there is a 3rd thing i thought of too but its unrelated to bias. i'll try it later when the client/server code is working. instead of using least squares to fit the correlated values. i think fitting by the evaluation metric might work better. sqrt(sum((log(prediction +1) - log(actual +1))^2) / n ) see here https://www.kaggle.com/c/nomad2018-predict-transparent-conductors#evaluation . the thing is, to do that i have to implement it myself. i couldnt find any accord.net code to do it

i just can't stop working on genetic algorithm stuff

hello hello hello. i don't even know where to begin. I guess some short stories, the improvements on the genetic algorithm have been steady and successful. I do still struggle with bias some. I also struggle with speed. now a little more on each.

For bias i'm pretty certain my strategy of scale-able scoring (or whatever i'm calling it) is the way to go. that is regardless of the size of the sample the sample scores the same. the more samples of varied size the more accurate the score is (with more certainty). Basically, you use the worst score from the group of samples. you should always include a full 100% sample as well as the various sub samples, but i run that one last and only if the sub-samples indicate the result is still qualifying. in fact i quit at anytime the score drops below a cutoff. this actually makes it faster on the whole too.

For speed, i've managed to get some sample GPU code to work, which is great! but alas, i haven't found time to write client/server code and implement a distributed version of the genetic stuff. I will, i just need more weeks/weekends. this will hopefully give me something like a 50-1000x boost of processing power.

All this work has been on https://www.kaggle.com/c/nomad2018-predict-transparent-conductors which is rather ideal for my purposes here. you can read the indepth on-goings here https://www.kaggle.com/c/nomad2018-predict-transparent-conductors/discussion/46239 . I'm really hoping i end up in the top 10 before its all over.

There also happens to be a new contest https://www.kaggle.com/c/data-science-bowl-2018 which needs image analysis to be completed.... however! I think this might be also be a contender for the genetic algorithm, though maybe a different version. I could certainly load the images in to the database and let the genetic algorithm figure out what is what... but i think there might be a better way. I think it might be better/more fun. to  design a creature (yes creature) that can move over the board and adjust its shape/size, and when it thinks its found a cell it it sends the mask back for a yes/no. after, i dunno 1000 tries it is scored and we make a new creature that does the same thing... breed winners etc etc. or we could go the reinforcement route. whenever it sends back a mistake we tell it "bad". and when it sends back a success we send back "good". in that way there would be only 1 organism and the learning would be at a logic level inside of it instead of having new versions of itself over and over again. I haven't decided which i'll do, but i think its probably something i'd get a kick out of writing.

Still MORE genetic algorithm stuff, running and feathercoin

The short version is I streamlined the mutation process. I did 2 things first I made it very simple either you breed you take 1 of the 2 parent genes or you insert or remove a gene. the chance of doing the insert and remove is totally based on stagnation. (as is the chance of a bad copy when taking the parent genes) stagnation is how many generations we've had since a new best cell was found. since this constantly increases it will pass through whatever sweet spot it needs to get to to maximize a the odds for a new best.... till it finds a new best then it resets. also by starting out baby stepping it, i give the code a really good chance to improve the results.

My old code was far more off the cuff. this new method seems really really focused. unfortunately the new method seems to improve a little slower at first. I think because you dont get the super crazy mutations the old one did. i'm hoping it just doesnt stop though.

The other improvement was i made it so the number of channels and layers dont mutate right now. I think they should be allowed to enter in but the actual process of that is probably in the same vein as gene mutation, super low odds slowly increasing and even if you do add one it should be a small channel (1 or 2 genes) or a layer that comes before the final result that can start being used.

I've started looking in to GPUing the code it looks like amd gpu code choices are kinda limited there is a c++ library that gets wrapped in various c# libraries that i'll probably end up using. the trick will be in finding the one that is easiest use. but... i'm not in a serious hurry there.

The program is currently doing 2 layers with 3 channels  each and is at .22454 gini (which is terrible) its been running for about a day so hopefully this time tomorrow its up to between .235-.255  if it gets over .255 we'll be in a good spot. the contest has a week or so to go. i think it's my best chance to win, though i might go back and fiddle with the normal data mining program over the long weekend. i really want to see if i can maybe hone the GBM using that least squares stuff i did to the gene results to improve them.

Unrelated to any of that... marathon training. i told you i'm going to run in a marathon right? or at least that's the plan. i've been having a real tough time losing weight this time around. and getting the miles in on a treadmill is... hard. i stopped dieting (which was working well) this last week so i could do/enjoy thanksgiving but the weight is trying to come back.

In better news though, I think i've finally fixed my stride as my plantar fasciitis isn't getting worse despite the regular longer runs. which brings me to tonight. tonight i do 18 miles. this is kind of a rubber meets the road moment. Since I started training all attempts to do extended distances (over 13 miles) on the treadmill or otherwise have been cut short by me. usually due to concerns about damaged feet/ligiments or just not being in shape enough. so if this doesnt stick tonight i might put a pin in the planned marathon and do one further out where i can get in better shape/have more time. we will see.

One other thing I'll mention. I've been watching feathercoin do its thing lately. I haven't talked about cryptocurrency since i think before i reset the log... it's been a while. I'm a huge supporter of feathercoin and their  team and want to see them do well. not to mention the coin itself. The point is its nice to see it start to take off. and i mean really take off. it was 10 cents a coin a week ago, its toying between 25 and 30 cents now. it could be some sort of artifical spike but i cant find a source of the news/pumping so i think it might just be a case of some big buyers wanting to hold reserves. I'm hopeful at any rate.

The genetic algorithm delights don’t stop

And I’m back, and so quickly! So the improvements have started....

I tried the least squares weighting of answers ... this works but I think isn’t worth doing while training. I’ll do it on the final submission but train without it.

i've just gotten the layers implemented and that seems to work but it’s too early to know if gains are really there. I think maybe if anything this will allow the code to improve to much greater extents faster instead getting bogged down at a lower score. (Though code without layers may eventually get there too)

The final improvement is essentially heuristic modeling to be applied to the odds of any given thing happening. I did this a little a while back and have rethought what I said since last post. I think the big thing is to just balance the odds of feature/channel selection and function/method/mechanism selection. This should increase speed and accuracy.

I'm still amused by the idea of giving a result to a Kaggle contest without giving the technology. I mean I give the the algorithm it generated but not how you get there. It would be delightful to win a few contests in a row without actually giving the technology away. It would turn Kaggle on its head (especially since its not the sort of thing that translates in to  kernel)

once everything is working the last step is to migrate it to a video card runnable code so I can scale it massively.