Dim Red Glow

A blog about data mining, games, stocks and adventures.

The Legend of Question Six [Solution] - Numberphile

If you are reading this you might be searching for a solution, saw my comment or just be browsing the internet. So here's the thing, I saw a Numberphile and was like "i gotta try that" https://www.youtube.com/watch?v=Y30VF3cSIYQ . I do recreational math, so you know.... i like a challenge. anyway here's the solution I came up with (I circled) the "magic step". Basically the realization is that the left and right sides are in the same form. and since I made "I" up I can choose it to be equal to B which forces the other side to be A which then of course dictates its value.

This was my 2nd attempt I spent a few minutes going down another path to see if i could turn it in to a quadratic and just get solutions... it was a mess so I went back and looked for well, what you see. Oh and above The J substitution isn't necessary. i was just doing it to see if anything made sense as a next step beyond the obvious. it didn't help you could remove it. I just didnt want to rewrite the work and i did it in ink so... there it stands.

Gen Con and the state of magic

I haven't had a blog in such a long time. The last thing i wrote about was the sum of 3 cubes, which while i'm working on is now in a pot with a bunch of other things I'm working on. this post though is all about gen con and magic. hopefully I'll get another up in the next day or so on some other stuff going on.

First, magic, I went Gen Con again this year, and played a lot of magic among other things there. it was great in general but the magic part of it was pretty lack luster. Not much to say here other than my magic days are winding down. Its not that I don't enjoy the game but the discovery and problem solving aspect of it is almost played out for me. It may be counter intuitive but when you get to a point where you can't sabotage your deck with random crazy ideas (cause you feel like you've tried them all) you start to do better but it becomes a lot more boring. That's a long winded way of saying winning with the expected decks is boring. And really winning in general is kind of boring. The struggle in anything is what makes it exciting.

I don't really plan on doing any more mtg grand prix's though I might squeeze in 1 or 2 next year if the event looks like it'll be a ton of fun. I expect this time next year, i'll either be done with it or have like 1 major event.

Also as a note, Gen Con next year is the 50th anniversary so if you aren't planning on going, change your plans! It'll be one not to miss. Assuming I go (you never know for sure) I won't be playing nearly as much magic, and I'll definitely be getting more sleep. being tired the entire con makes it less fun and less memorable. Also it goes by much quicker which stinks!

Besides magic, I played true dungeon (1st dungeon was meh, 2nd was pretty good). I did AEG big game night (fun when you have others with you). Did the keg tapping on Wednesday for the 20 sided rye ale. I did the orc stomp 5k (probably wont do this next year, cause the 6:30am start time... ie 5:45 wake up ruins a lot of the con cause of the lack of sleep i get). And I got a chance to demo a friends game for an hour and half.  https://www.kickstarter.com/projects/1410499285/dragonstone-mine-a-family-friendly-board-game

Back to magic, what am I playing right now? In standard bant humans. In modern The rock (or my version of it). in legacy eldrazi-post (12 post with eldrazi)  and painter. I'm working on a zur's weirdening/life gain deck in modern. And I'm working a black/white control/life gain deck in standard as well.

 

 

 

 

 

Sum of 3 cubes

So I was watching/catching up on numberphile and watched this 74 is cracked - Numberphile . The short version is of all the numbers less than 100 they have shown that only 3 (now 2) can be expressed as the sum of 3 cubes except for a list of a few which are proven to have no solutions. I had watched the original video this one references before and while being entertained at the time, I promptly forgot about it after watching it.

I dont think many people realize or trully care that I've spent long years of my life thinking about Diophantine equations . I did it back when I was looking for new method to methodically factor primes (read 17-32 off and on) . While I came up with some interesting stuff (to me at least) I never really got anything that good. It was always a hobby/fun problem for me. The point is solving the 3 cubes problem is something the "work" i did could probably do relatively quickly if there arent tons of Residue Classes in the various modulos of 33 and 42 (the two remaining numbers). Just saying that makes me think there probably are, but if there arent merging them in to larger equations isn't very hard.

Even if there are bunches you can still get a new set of equations that dramatically cut down your search time. Getting past 10^16 is pretty remarkably easy heck getting past 10^32 might be trivial it just depends on the numbers. I think I'm gonna spend a little time on solving 33 and 42 just for fun and see how hard the problem really is. It would be neat to have my named tied to something in mathematics even if its just a simple solution to some rather mundane puzzle. And I need another hobby/would like to return to the old hobby for a while.

Slow going on the GBM in GBM idea

I've been thinking about this for weeks. I'm normally a jump in there and do it kind of guy but with no contests around to motivate me I've been stewing on it more than writing it. I did implement one version of the GBM in GBM mechanism but it underperforms compared to my current normal tree. There are probably more things to do to it to hone it and make better but this is where I stopped and started stewing.

I've been thinking i'm approaching this the wrong way. I think that trees are great for emulating the process of making decisions based on a current known group of data, but don't we know more? Or rather can't we do better? We actually have the answers so isn’t there a way we can tease out the exact splits we want or get the precise score we want? I think there might be.

I've been looking at build a tree from the bottom up. I'm still thinking about the details but the short version is you start out with all the terminal nodes. You then take them in pairs of 2 and construct the tree. Any odd node sits out and gets put in next go. The "take them in pairs of 2" is the part i of been really thinking hard about. Conventionally going down a tree your splits are done through some system of finding features that cut the data in to pieces you are happy with. I'm going to be doing the same thing but in reverse. I want the 2 data pieces paired together to be as close to each other as possible from a Euclidian distance perspective at least with regards to whatever features I use. But (and this is one of the things I debate on) I also want the scores to be far apart.

When you think about what I’m trying to accomplish putting two items with really far apart scores makes sense. You want to figure out shared qualities in otherwise disjointed items. Similar items could be joined as well if we approach it that way the idea is you are building a tree that hones the answer really quickly and exactly. This however wouldn’t do a good job of producing scores we can further boost... we wouldn’t be finding a gradient. Instead we would be finding 1 point and after 1 or 2 iterations we'd be done.

By taking extreme values the separation would ideally be the difference from the maximum value and the minimum value. If we did that though it would only work for 2 of our data points. The rest would have to be closer each other (unless they all shared those 2 extremes) I think it would be best to match items in the center of the distribution with items on the far extreme. Giving all pairs a similar distance of (max-min)/2 and likely a value that actually is 1/2 the max or the min since it would average to the middle.

In this way we merge up items till we get to a fixed depth from the top (top being a root node). we could keep merging till then and try to climb back down the tree, i might try that at some point but since the splitters you would make won’t work well, i think the better way is to then introduce the test data at the closest terminal node (much like how nodes were merged together) and follow it up the tree till you get to a stopping spot. The average answer there is score you return.

Again I still haven’t implemented it, I’ve been stewing on it. The final piece of the puzzle is exactly how I want to do feature selection for the merging. There has to be some system for maximizing score and minimizing distance so it isn’t all ad-hoc.

 

 

Small GBM in GBM and a newish idea

Things are going okay with my GBM in GBM. I've got the boosting in a tree now but the implementation i've got in there is not quite what I want. It's a little bit too much all or nothing. that is since each tree level is a binary splitter and my method for evaluating splits is also binary, boosting the binary splitter doesn't work real well. there is no gradient to boost persay. Everything is a yes or a no

You can change the mechanism to turn it in to a logic gate and essentially each itteration trys to make the final result all zeros (or ones). but this turns out to be so-so at best. You are missing the real advantage of easing in to a solution and picking up various feature to feature information that is weakly expressed. Don't get me wrong, it DOES work. it's just it's no better than a normal classifing tree.

When I do try and ease in to by turning the shrinkage way up. (the amount you multiply the result by before adding it in to your constructed result.) it ends up producing essentially the same result over and over again till you finially cross the threshold and the side the result wants to go on to changes (for right or wrong). the point is the function in my series of functions that add up to a final answer is producing nearly the same result each time it is called.

I can change this by making it randomly select features each call but it turns out not to make much difference in my particular implementation. What I really need is a way to weight the various responses so the influence varies based on probabilities. I thought it might be good to quantize these and make it more tiered as opposed to trying to build a standard accuracy model of sorts. so instead of 0-100% maybe 10%,20%,30% etc.. or some other measured amount. The idea is that like valued rows will group together. Though I don't know if that's really any better than a smooth accuracy distribution.

Update on the updates

First let me share this 'cause you know, ridiculous.

That's pretty much what I'm working on right now in my data mining. it's about half done.

Also I haven't had any marathon training updates. The short version is I've decided to hold off and only give updates when there is something meaningful to share. I never wanted this blog to be work, not to mention I want it to be enjoyable to read. Doing a daily or weekly update for the sake of the update doesn't in my mind fit that. I'm still running, still putting in around 16 miles a week. The weight hasn't really come off. I'm hovering around 197. I figure in a few more weeks if something doesn't give I'll make a more concerted effort to change my diet.

Magic the gathering continues to be a fixture in my life, legacy play especially. I've been contemplating a couple decks in legacy. One I called fraken-mud which was a tool kit mud deck that used birthing pod and eldrazi instead of the usual suspects (like metalworker and forgemaster). It's been inconsistent. Sometimes its really really good, but last week i went 0 for 4... yeah talk about disappointing. I'm also looking at black-white cloudpost deck I want to try that just runs lots of my favorite things (stoneforge, wurmcoil, deathrite, bitter blossom, ugin ..etc) not sure if it'll work or not. I'll try it this friday. I've also got a white painter's deck in the wings I'm gonna try in a few weeks as well. I think going the all red or red white or red blue route on painter grindstone is fine. but I want to try an all white version.

In standard I've been trying to find something interesting to play but eh... it all seems like rehash. I'm probably going to play green white little creatures/good-stuff next time I play. And as for modern, well i have my abzan eldrazi deck still together. I think that's all you get till i get a wild hair and build ... i dunno a new version of affinity. I'd kind of like to try souping up affinity. making it more mid range/stable. but we'll see.

 

 

 

 

Gradient Boosting (Part 3)

I've been trying to find my next logical step in data mining development. I think I've settled on it. I'm going to make a tree that uses boosting in it's fundamental splitting nature. It would be nice if the final results from the tree produced better results in general (from my current trees), but if it can remove all trace of bias and move towards a good uniformly evaluated answer, it will be a better tree to be used in subsequent ensembles.

Currently my tree building mechanism tries to build trees to get the most exact answer possible by grouping like-scoring training rows. The algorithm looks for features to use as splitters that would move rows down the left hand of the tree when they score closer to negative infinity and down the right side of the tree when they score closer to positive infinity. This allows me to do feature selection using the correlation coefficient instead of random selection.

This turns out to be a great way to select features. Not only does the feature selected work towards getting a good separation, but it also has the next step(s) split built-in to the feature separation. That is it correlates with final score as a whole. Subdividing the training set so part goes left and part goes right might be just as accurate using a different feature that doesn't correlate as well, but the mismatches should be closer to correct in the version with a higher correlation coefficient.

I've been making my trees using some form of this method for years. I do lose the ability to discern good secondary features unless they show up due to bagging. This is something I would like to try and address in the new version. This is also what I was talking about in terms of bias nature in the opening paragraph. My bias is to the highly correlated features.

I also much more recently moved to the "logistic function" (sigmoid) style splitter. so I build a curve for each feature that represents the feature's distribution. It then gets combined with 2nd, 3rd .. etc features I choose to use as well (depending on the number of features or type of data) to determine which path down the tree the data is set to go. This works wonderfully, it gave me my last big boost in Random Forest before I moved on to Gradient boosting which also makes good use of those stub trees.

The new tree I'm going to be making will endeavor to take the best of all worlds as it were. I'll be using every feature and building a splitter from the net total sum of where the data says it should go. I'll be weighting each opinion by it's overall correlation to the final score (this might change, but you should have some system of setting importance otherwise its noise) and finially I'll be doing the process itteratively where like in GBM each pass will attempt to adjust the result of the previous to hone in on the actual correct score.

Once a tree is finished I'll return it out to a normal GBM tree ensemble mechanism which will make a new tree and repeat the process. In the case of a binary classifier as you final score. The first tree will more likely than not be a single depth tree because it will have a binary classifier to deal with. But the trees after that will start to get some depth to them as it finds more and more resolution. I'll need to set a maximum depth allowed much like a normal tree building exercise.

Give me a few days and we'll see how it goes.

Training for a marathon (part 2)

Just a small post today. I think I'm going to do updates on the marathon training once a week. I'll give mileage from the last update and the most recent timed run data I have. I don't time myself most runs as wearing the watch is a bit annoying but i'll try and do it the day before or day of the post.  Originally, I was going to do an update every time i ran but that's just too much/boring. 

But while i'm here I'll say, I did run today and i put in another 4 miles. The way the weather is shaping up I'll probably run some more this weekend. I also weighed myself (i usually do in the morning) and my weight was down to 196.2 i only mention this cause it's important to know that weight tends to vary +-2 pounds around some actual center weight. That's just another reason not to obsess about it. If you are seeing your own weight vary a lot from day to day (well a few pounds anyway) that's just food and water moving through you.

Training for a marathon (part 1)

I enjoy running. Well, most of the time. I've run in some form regularly for most of my life. I started in in track field in jr high and then did it again in high school. I ran for fun and to stay in shape periodically in my 20s and much more so in my 30s. In my mid 30s I decided it was time to try a marathon, partly because I wanted to see what it was like, partly cause it was an 'Everest' sort of thing.

I ended up running 3 marathons in about 2 years.  I did the rock and roll marathon in St. Louis 2 years in a row and the Beat the Blerch one in Seattle the first year they had it. The st. Louis rock and roll marathon stopped having full marathons last year (I guess turn out was too low), so that wasn't an option last year. The Beat the Blerch one, while fun, was never meant to be a reoccurring thing for me. I just wanted to travel and try it.

So, I didn't run a marathon at all last year. I guess I could have found one but I was 'meh' in terms of motivation. Also I have had chronic plantar fasciitis. I think I've finally beaten that thanks to a number of things: stretching, lots of time off, changing my running shoes to newtons (which emphasize toe sticking not heal striking) and changing my walking around shoes to a minimalist cross trainer (over a running shoe). I've always had mild problems with foot pain because I tend to be a heal toe walker/runner. I think however what caused this chronic condition was something else. I trained for a 5k (I was running for time) and at the same time I was training for the 2 marathons i did. When one marathon was done and the 5k was run I was already suffering pretty constant pain, and I didn't give my self any real time off to heal since I had 1 more 2 months later. That I think sealed the condition.

Skipping ahead around a year and half. I'm 40 now and it seems like as good a time as any to try again. I didn't ever really think marathons would be my thing and they probably won't be long term, but I did have a few goals that I haven't accomplished in the 3 i ran. I wanted an under 4 hours time (4:25 was my best) and I actually even wanted to do a 3:30 time once I did the 4. Some day I thought it might be fun to do a biatholon or triathalon as well but I want to make those goals first.

I've spent the last 3 or 4 weeks running periodically and mildly dieting, trying to get my body back in to some sort of shape. I can do my "stay in shape" now with out much though. I did it today here are the stats

distance: 4.02 miles

time: 37:46.6

pace: 9:23 /mile

weight: 197.6

The watch I have that tracks this stuff has heart rate and gps in it too, though i'm not sure if it keeps it all. At some point I'll have to hook it up to my computer and download the data and see what is what. my current weight is a heavy 197.6 (heavy for me anyways). A few things I'd like to point out, in a perfect world I'd weigh about 20 pounds less (i'm 5'10-5'9"). This is especially true when i'm about to run the marathon (lighter means less weight to carry). I've never quite reached it when training previously and always end up around 180-185. So that's a goal of sorts to go along with the 4 hour run. 1 really feeds the other. If you put in the miles the weight goes away and it helps you run faster. I'm not what you call hardcore about this though which is why previous attempts have met with limited success.

The mileage I want to be putting in eventually will be around 40 a week probably 4-5 days a week for 8-10 miles each with maybe a big run every other week of like 15-20. Right now i'm probably doing about 8 (two 4 mile runs), though the weather is changing and i'll be putting in a lot more soon. This weekend I'll probably put in 10 more and I'll probably put in 4 more tomorrow. I wont go too nuts till I pick a marathon and get a timeline. training for a marathon eats up a lot of time.

Diet is something I need to think about too.  I've actually been losing weight, sort of... I started a low carb diet but i havent been running enough or been strict enough to really shed the weight. I was actually around 201 when is started about a month ago.. lost 5 pounds real quick and have floated there for the last month probably taking in around 60 or so grams on average a day. That's going to stop now though. I want the energy to train with and not eating the starches puts undue strain on me. yes you will lose weight faster but this isn't about losing weight over night.

It's worth mentioning having a regular exercise routine is important to a long life, looking and feeling healthy, and once the marathon are over i plan on continue running regularly. I will try and give updates on the days I run so anyone interested can follow along.

Improving t-sne (part 3)

This post will be the last for the immediate future on t-sne and really will just cover what I've done and how it all ended up. I have a new direction to head. And while I may use t-sne , I don't think it will be the main focus (though i guess it could be! surprises do happen).

So without further adieu let me surmise what I've written about so far. T-sne is an algorithm that reduces dimensions of a data set while preserving the statistical probability dictated by distance between rows of data and the features there in. It does this by randomly distributing the points in whatever N dimensional space you choose. Then it slowly moves the points iterativly in such a way as best reflects the same probability distribution as what existed in higher order space. You could i suppose increase dimensions in this way too if you want to.

I covered how I would speed this up at the end of part 1. I use a series of smaller "mini" t-sne's and combine them together. All of those t-snes share a lot a large number of the same data points. these were seeded in to the same position. I use those points to stitch together all the various frames. using the closest point as an anchor to find its final relative position in 1 key frame i use to put all the points. In that way all points from all the frames are mapped to 1 of the frames. This seems to work really well though the results aren't exact but the run times are linear. (i'm calling this a win)

In the 2nd part I covered how one can augment the results to get a clear separation of data points based on a particular score.  In short it's not magical, you build features from a data mining tool and provide the results as new features. you then use only those as the t-sne inputs. In my case my GBM does some sub selecting and bagging internally. because of this, my outputs are never exact. This makes for a very good set of inputs for seeing the separation of the scores.

Unfortunately, the results you get don't provide a way for improving the score. I had hoped they would, but truly it's just another way at looking at the final output. If you train on a sub section of the data and feed the t-sne results back in to the GBM machine and re train using the other part and no overall improvements will be found.  (i'm calling this a loss or at best a draw. yes I did get wonderful separation which was my starting goal, but it's nothing more than a visualization tool and didn't lead to something that actually "improves" anything from a data mining perspective.)

It's worth noting that if you don't remove the bias and go ahead and retrain from the t-sne result using the same data you fed it originally to get scores to build the t-sne results, it will slowly get better and better but this is just a serious case of over-fitting. GBM does such a good job of finding connections between data points it will emphasize any aspects that are represented in the new t-sne feed that allow it to get closer to the right score.

I tried this out and found I could turn a .47 results (log-loss) in to a .43 result by adding in the t-sne data. then I could further reduce it all the way to .388 by letting the algorithm pick and choose the features to use (well if brute force trial and error is pick and choose). The point is it's not useful for improving the results of the test set (which actually score .48 when the cross validation set score .388). 

I'll leave this right there for now.