<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kai Franke on Medium]]></title>
        <description><![CDATA[Stories by Kai Franke on Medium]]></description>
        <link>https://medium.com/@kaifranke3?source=rss-8421ecd35e58------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*x8_ilMEC67rCoJSYIrfryw.jpeg</url>
            <title>Stories by Kai Franke on Medium</title>
            <link>https://medium.com/@kaifranke3?source=rss-8421ecd35e58------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 00:44:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kaifranke3/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Stuff+ Model Using XGBoost]]></title>
            <link>https://medium.com/@kaifranke3/building-a-stuff-model-using-xgboost-8c548fbab8f2?source=rss-8421ecd35e58------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c548fbab8f2</guid>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[baseball]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[xgboost]]></category>
            <dc:creator><![CDATA[Kai Franke]]></dc:creator>
            <pubDate>Mon, 08 Aug 2022 00:33:33 GMT</pubDate>
            <atom:updated>2022-08-09T00:23:13.351Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pKn6Us1ZE5QRf8VpLLfpPg.jpeg" /><figcaption>Cover image from: <a href="https://www.nbcsports.com/bayarea/video/camilo-doval-unleashes-nasty-slider-get-giants-out-jam-vs-tigers">https://www.nbcsports.com/bayarea/video/camilo-doval-unleashes-nasty-slider-get-giants-out-jam-vs-tigers</a></figcaption></figure><p>The code for this project can be found on my GitHub: <a href="https://github.com/kaifranke/StuffPlus/blob/main/code">https://github.com/kaifranke/StuffPlus/blob/main/code</a></p><p><strong>Introduction</strong></p><p>Over the last decade-plus, baseball analytics has grown through pitch-tracking technology to benefit mainly pitchers. Much of this help for them has come through pitch design where pitchers can go and work on developing the best pitches possible for themselves. This is seen throughout the pros, college, and even high school baseball. The goal of pitch design is to help pitchers build the best arsenal possible, a big part of this is how much “stuff” each pitch has.</p><p>Stuff can be described as how “nasty” a pitch is, how much it moves and how fast, or how much it deceives the hitter. A lot of baseball research lately has been developing Stuff+ metrics to evaluate stuff and use it in pitch design. I wanted to take my stab at this and come up with my Stuff+ metric. So let’s get into it!</p><p><strong>Methodology</strong></p><p>The first question I had to ask myself was how I wanted to evaluate stuff from a pitcher. My first thought was to predict the expected change in run value based on the pitcher’s movement profile, but instead, I decided to evaluate whether or not a batter swung and missed at a pitch, given that they swung. I felt this was better in terms of “stuff” as it shows that the hitter was fooled enough to where he could not even touch the ball. While a hitter being frozen up could also be attributed to nastiness, I feel that may be more attributed to sequencing rather than the movement of the pitch.</p><p>I then took that predicted whiff probability and created it into a “+” stat by multiplying the predicted whiff number by 100 and dividing it by the average whiff probability. This makes the average 100 and is easier for the typical fan, coaches, and players to read.</p><p><em>Stuff+ = 100 * (predicted whiff% / league mean predicted whiff%)</em></p><p>To create these predicted probabilities, I had to build a model that can do this. The algorithm I chose to do this is XGBoost is quick to run, has many parameters to change to try to improve the model, and is simply just powerful. Speed was very important to me here as I decided to make two different XGBoost models to represent fastballs (Fourseamers and Sinkers) and offspeed/movement-based pitches (Sliders, Curveballs, Changeups, and Cutters). The model would take the features I select and give a probability of a swing and miss from the batter.</p><p><strong>Feature Selection</strong></p><p>Similar to my last article, I decided to again use Boruta for my feature selection as I wanted to get the best outcome possible. However, this time was a little different as there are not as many variables that I considered. I built two different Borutas, one for the fastballs and offspeed/movement-based pitches once again.</p><p>For the fastball Boruta, I tested the movement of the pitch, the velocity, the release point, and the pitch’s spin rate. It came back and told me that all of the variables were deemed important to building the model, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/0*VNv7ww6ldIq4QbeN" /></figure><p>I decided to take spin rate out though, I did this as the spin rate is how movement is created and I did not feel that I should have two different measurements of movement in my model. Also, surprisingly, the release point was deemed the most important feature of the pitch and not movement or velocity. This could be in part due to the vertical approach angle created by the release height and the horizontal release position getting affected by the vertical position.</p><p>For the offspeed and movement-based pitches, I added a couple of extra variables to be tested: the velocity difference between the fastball and offspeed pitch and the movement difference between the two. Being able to have offspeed pitches be much different compared to the fastball will lead to better results overall, but let’s see that importance plays compared to raw velocity and movement numbers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/0*5U02OqB-ovs-GOl6" /></figure><p>As you can see, the velocity difference is the most important variable. The movement differences seem to have the same importance as the raw numbers. You can also see that all of the variables were once again deemed important. Although the movement differences seemed to be of similar importance to the raw numbers, I decided to stick with the raw numbers as it would be easier to implement in pitch design if the Stuff+ metric were to be used as a tool and the raw numbers are actually what is seen on the pitch. So the metrics on each of the models will be:</p><ul><li>Velocity</li><li>Movement</li><li>Release point (along with extension)</li></ul><p>Let’s get on to building the models!</p><p><strong>Model Tuning and Building</strong></p><p>One of the biggest questions I had to answer for myself was what type of model I wanted to use and what parameters were needed to make the best model possible. The model I decided to choose was an XGBoost.</p><p>The main reason I chose this model was simply that I wanted to learn more about it, however, it can work with large datasets efficiently and accurately as it can build on its predictions. With the dataset that I had of over 1,000,000 observations of swings from 2018 to 2021, I had to have a model that can run fast.</p><p>In the XGBoost, three parameters affected my model the most, those were:</p><ul><li>Max depth: this is how big the decision trees in the model can get up to with how many decisions they can make</li><li>Eta: this is the rate at which the model can learn, lower numbers mean it learns slower, and vice versa</li><li>Number of rounds: this just determined how many times the model would train</li></ul><p>I chose the values for these parameters by seeing how each of these affected the test AUC of the models and pick the values that would increase the accuracy of the algorithm. I wanted to make sure not to overfit, so I wanted to be a bit conservative with each of these metrics, as higher values will lead to overfitting results. I also could not make these values very low and underfit, so I had to find a sweet spot.</p><p>I did this for both fastball and offspeed/movement models and came up with an eta of 0.1 and a number of rounds of 200 for both, with a max depth of 3 for fastballs and 4 for offspeed. These parameters gave me the best overall test AUC for both of the models. Here are the ROC plots for each of them:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/0*2Aee8T_got9zc7xt" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/0*C9NOtzpU7oJi-wGO" /></figure><p>As you can see, the AUCs of each of them are not that great. The fastball AUC is just above 0.6 and the offspeed is not even 0.58. This means that they are only a little bit better than random guessing. However, since this is a stuff model, there is no pitch location, which would lead to a much higher AUC if it were used. So although the AUC scores are low, I feel okay with it as a significant factor in predicting swings and misses is omitted.</p><p>When putting the dataset all together, though, it gave me a final AUC of 0.638 which is even better than the two before! This shows that when putting the models together, we end up creating a better predictor. Here is the ROC plot of all the data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/0*p9R8bRryDplErsAj" /></figure><p>So, as I stated, I am happy overall with the model and its ability to predict whiffs. This gives me a good idea of which pitchers have the best stuff and how it relates to getting hitters out.</p><p>One last thing that I wanted to check out was the year-to-year correlation between the predicted whiff% and the next year’s whiff% (again, Stuff+ is just a function of predicted whiff%). I chose to use 2021’s predicted whiff percentages to see how they relate to this year’s whiff percentage by each pitch by each pitcher. Here is the plot and R² value I got out of it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/0*uqYjZC6674LHGfAA" /></figure><p>I am extremely happy with this. I was able to come up with an R² value of 0.445 which indicates a moderate linear relationship between the two years. This is great as it shows there is some stabilizing power with my model and it can be at least somewhat reliable. Again, if I was able to use pitch location, I expect that number to be higher, but I did not want this in my model.</p><p><strong>Leaderboards</strong></p><p>Now that we have the model building and testing out of the way, we can now run some analysis on it and see what some of the best pitches in baseball are based on just stuff. I think we should take a look at how pitchers are doing in 2022 as well as a sample use case on how this can work to improve a pitcher with a struggling pitch.</p><p>Let’s take a look at the top pitches of each pitch type in 2022 up to August 1st. Here are the leaderboards.</p><p>Fourseam Fastballs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/0*e4cZ7gqvb_F-oxfW" /></figure><p>Curveballs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/0*VcGVGELpB2dDrd-I" /></figure><p>Sliders:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/0*7vS-TG_4QXBxJo2c" /></figure><p>Changeups:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/0*-VcMiLcZsWkXjIcR" /></figure><p>Sinkers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/0*GFoSADH_ojHvsVVe" /></figure><p><em>*You may notice that all of the numbers, except for Hader, are less than 100, we will get to that in a minute</em></p><p>Cutters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/0*detcvhPPiulxVYjw" /></figure><p>A lot of names that you may have recognized are on these lists, such as Ohtani, Cole, Hader, and more. Now one thing that you may have noticed when you looked at these lists is that the sinker numbers are much lower than the rest as well as the four seamers are lower than typical as well. This is because that sinkers and fastballs do not generate swings and misses as much as offspeed or movement pitches. So, generally, they will be lower. You do see that Hader is miles above everybody else in sinkers with 143 compared to 88 at second place. The best way to fix this for individual pitches is to change them for each pitch to have their average moved to 100. This is good to use in pitch design when you are trying to see what pitches are above and below average.</p><p><strong>Pitch Design Example</strong></p><p>So with all of this said, the best usage of this model, in my opinion, is being able to use it to develop pitchers’ arsenals. Whether a player needs a complete overhaul or just a slight change, using this model can help a coach and player see how much needs to change, and in which direction, to be better.</p><p>The pitcher I chose to improve is Twins pitcher Chris Archer. Most of you know him as an all-star of the Tampa Bay Rays as well as the piece going to the Pirates in the terrible Glasnow, Meadows, and Baz trade. Archer had been poor the past few years and has had a little bit of a bounce-back this year, though his underlying metrics have not been great.</p><p>The reason I chose him is that he has a lot of room for improvement just by changing his movement profile. He still has an average fastball velocity at 93.1 mph so he has not completely fallen off with his arm talent. In terms of Stuff+, he has a 96 fastball, 98 slider, 83 changeup, and 99 curveball. Archer has 3 average offerings and 1 below average offering and if he can improve all 4 of his pitches, he could be above average in terms of his overall stuff in his arsenal. The pitch that has the most room for improvement, though, is the changeup, so that is the one I will seek to improve here.</p><p>To find a more optimal movement profile, we must first control for velocity and release point to find pitches with similar characteristics. He throws his changeup at 86.9 mph on average, so I decided to filter to just changeups that are between 86 to 88 mph. I did something similar with his release point as I made the range for his horizontal release point +/- 0.15 and +/- 0.3 for his vertical release point. Now that we have set controls for those 3 variables, we can make a plot to show where Archer can improve:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/0*DGIdFTAyBU1oTahP" /></figure><p>As you can see, blue means a lower Stuff+ and red means better. The black dot in the blue represents Archer’s movement profile. We can see that Archer has a lot of potential going towards the bottom right-hand corner, which means less vertical movement and more arm side break. We see a similar direction in a lot of elite changeups, especially Devin Williams’ pitch.</p><p>Changing the movement to that profile would also make sense as Archer would not have to increase his spin rate much. Instead, you can transfer backspin into more side spin creating the movement profile we want from his pitch. We can see in this plot what my proposed new movement profile for Archer, green represents the new changeup:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/0*XHGvDBYW8sAag14h" /></figure><p>That improves his Stuff+ from 83 to 99, 16% better than what it was before and just barely below league average. It may not necessarily be possible to change it that significantly, however, going in that direction would be beneficial for Archer if he wants to improve the pitch.</p><p>His arm path, the mechanics of his hand, or just simply Archer’s comfort in gripping and throwing the pitch may lower his possibility of creating it. However, if Archer (or any other pitcher for that matter) can’t get to that profile, then he could try to go another way with it and maybe add a less vertical break to get closer to the red or by adding more horizontal to get closer to the white. There are many ways to change the pitch and help him be more successful.</p><p><strong>Pitches With Best Stuff</strong></p><p>Let’s have some fun and see if these numbers match the eye test! We will take a look at the best pitch by each pitch type this season until August 1st; the Stuff+ numbers will be relative to the pitch types themselves.</p><p><strong>Fourseamer: </strong><a href="https://www.mlb.com/video/bal-nyy-d088c4?q=felix%20bautista%20strikes%20out%20gleyber%20torres%20swinging&amp;cp=MIXED&amp;qt=FREETEXT&amp;p=0">Felix Hernandez strikes out Gleyber Torres swinging</a>, Stuff+: 354</p><p>You could tell this pitch was nasty just by watching it, it got a whiff, had 98 mph of velocity, and looked very flat coming in to Torres. This passes the eye test.</p><p><strong>Sinker: </strong><a href="https://www.mlb.com/video/josh-hader-ball-to-patrick-wisdom-ujrpuf?q=PlayerId%20%3D%3D%20%5B623352%2C621550%5D%20AND%20PitchResult%20%3D%20%5B%22ball%22%5D%20AND%20Date%20%3D%20%5B%222022-04-10%22%5D%20AND%20Balls%20%3D%20%5B0%5D%20AND%20Strikes%20%3D%20%5B0%5D%20Order%20By%20Timestamp%20DESC&amp;cp=MIXED&amp;p=0">Josh Hader ball to Patrick Wisdom</a>, Stuff+: 502</p><p>As seen earlier, Hader has a sinker that is way better than everyone else. This one is 400% better than the average sinker. The location wasn’t great as there was no chance of a strike, but you can see what could happen if it was located well.</p><p><strong>Cutter: </strong><a href="https://www.mlb.com/video/edmundo-sosa-strikes-out-swinging-1rslvd?q=jose%20alvarado%20strikes%20out%20edmundo%20sosa%20swinging&amp;cp=MIXED&amp;qt=FREETEXT&amp;p=0">Jose Alvarado strikes out Edmundo Sosa swinging</a>, Stuff+: 257</p><p>This one has all the makings of a nasty cutter. It is thrown hard and has a good amount of horizontal movement to make the hitter look silly. Alvarado led the cutter leaderboard with a total Stuff+ 20% better than the second-best player.</p><p><strong>Changeup: </strong><a href="https://www.mlb.com/video/adam-ottavino-ball-to-mike-ford-jtfuel?q=PlayerId%20%3D%3D%20%5B493603%2C645801%5D%20AND%20PitchResult%20%3D%20%5B%22ball%22%5D%20AND%20Balls%20%3D%20%5B1%5D%20AND%20Strikes%20%3D%20%5B2%5D%20Order%20By%20Timestamp%20DESC&amp;cp=MIXED&amp;p=0">Adam Ottavino ball to Mike Ford</a>, Stuff+: 208</p><p>Typically you would expect Devin Williams here, but it is Adam Ottavino who is more known for his sweeping slider. He has a sweeping changeup too to get a very nasty offering to go opposite of the slider. Again, not a great location, but could be good if it was just below the zone or a low strike.</p><p><strong>Slider: </strong><a href="https://www.mlb.com/video/bruce-zimmermann-ball-to-aaron-judge-qtox61?q=PlayerId%20%3D%3D%20%5B669145%2C592450%5D%20AND%20PitchResult%20%3D%20%5B%22ball%22%5D%20AND%20Date%20%3D%20%5B%222022-04-28%22%5D%20AND%20Balls%20%3D%20%5B1%5D%20AND%20Strikes%20%3D%20%5B0%5D%20AND%20Inning%20%3D%20%5B5%5D%20Order%20By%20Timestamp%20DESC&amp;cp=MIXED&amp;p=0">Bruce Zimmermann ball to Aaron Judge</a>, Stuff+: 193</p><p>This slider has more vertical movement than horizontal and does not have a great location once again. Zimmermann seems to lose control of it and it doesn’t come near the zone. But it moves a ton and will be successful if it is located in or near the zone.</p><p><strong>Curveball: </strong><a href="https://www.mlb.com/video/shane-baz-ball-to-dj-lemahieu?q=PlayerId%20%3D%3D%20%5B669358%2C518934%5D%20AND%20PitchResult%20%3D%20%5B%22ball%22%5D%20AND%20Balls%20%3D%20%5B0%5D%20Order%20By%20Timestamp%20DESC&amp;cp=MIXED&amp;p=0">Shane Baz ball to DJ Lemahieu</a>, Stuff+: 219</p><p>This pitch from Baz could create tons of whiffs, the location could make a lot of swing-happy hitters look foolish as well. It ended up as a ball here, but it has all the makings of being a great pitch. It has a good mix of both horizontal and vertical with more vertical.</p><p><strong>Conclusion</strong></p><p>Overall, I am very pleased with the model I built here. It can help us as analysts and fans understand how to see what pitches have and how their pitches can be improved. This is a metric that can make pitch design more optimized and successful by giving coaches and pitchers numbers on the quality of their current pitch and what they need to do to improve.</p><p>Stuff alone definitely does not mean a pitcher will be successful, as we saw in some of those videos, a pitcher needs command with their pitches to create the swings and misses that they want.</p><p>One thing I am interested in with this model using run value instead of whether or not a hitter whiffed. Whiffs require that the hitter swings but the run value can be modeled with every pitch and would give us a higher sample size and potentially a better model.</p><p>Another thing that could improve this model is being able to optimize parameter tuning to build the best model possible. I felt I guessed and checked with tuning the parameters and if I used a function to tune instead, I feel that the model could be better, even if it was only a little bit.</p><p>Improving pitches using the model also does not figure in a pitcher’s arm slot and the mechanics of a pitcher’s wrist to analyze the capabilities of how much movement they can get. I would be interested in building a pitch design recommendation system based on their arm slot, spin rate, wrist angle, and more; depending on what data is available to me.</p><p>Also in the future, I want to build a pitch design dashboard that can be used to help coaches make decisions on what pitchers should do to get more out of their arsenal. This could be combined with the recommendation system so that there is an unbiased opinion on how a pitcher should improve. Improvement could come through movement, but it could also be from changing the arm slot a little bit or adding or lowering velocity.</p><p>As I said, I am very happy with this model and I think it can be very useful for teams to improve their player development systems to make them both more efficient and successful.</p><p><em>Note: All data from Baseball Savant</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c548fbab8f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Strike Probability Model and Catcher Framing Using Random Forest]]></title>
            <link>https://medium.com/@kaifranke3/strike-probability-model-and-catcher-framing-using-random-forest-7b482d6f159f?source=rss-8421ecd35e58------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b482d6f159f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[random-forest]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[baseball]]></category>
            <category><![CDATA[rstudio]]></category>
            <dc:creator><![CDATA[Kai Franke]]></dc:creator>
            <pubDate>Tue, 24 May 2022 20:38:28 GMT</pubDate>
            <atom:updated>2022-05-24T20:38:28.577Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DACpITCstL2esPxLMSvaSA.jpeg" /><figcaption>From the article: “With James McCann Struggling, Tomas Nido Deserves Additional Playing Time” by Sam Navarro on Metsmerized Online</figcaption></figure><p><em>All of my code for this project can be found on my GitHub: </em><a href="https://github.com/kaifranke/Catcher-Framing-Strike-Probability-Model/blob/main/Model%20Code"><em>Model Code</em></a></p><p><strong>Introduction</strong></p><p>There is a hidden art in the game of baseball that has become more noticed in the past few years; that is pitch framing. Innings can be saved or blown up based on one pitch that is miscalled by the umpire. Catchers that can do this well add much value to their teams and help them win more games by saving runs. Evaluating this and figuring out which catchers are the best will help in overall team building.</p><p>In the past, there have been articles such as <a href="https://medium.com/@ChristianHook/credited-framing-runs-which-catchers-saved-the-most-runs-with-their-receiving-e82ee8e9c546">Christian Hook’s</a> and <a href="https://medium.com/iowabaseballmanagers/an-in-depth-look-into-collegiate-catcher-framing-3e1a140d0db1">Luke Statler’s</a> that have done this pretty well. However, I wanted to attempt to do my research on this by building a model and doing an analysis with it.</p><p><strong>Methodology</strong></p><p>To measure this skill by catchers, I wanted to build a metric that could tell us how much catchers had to do with making each pitch a strike. The way that I did this was by predicting the probabilities of each pitch being called a strike and crediting the catcher with the remaining probability that it was not going to be a strike. For example, if a pitch had a probability of 0.67 to be a strike and it is called a strike, the catcher will have 0.33 strike of a strike added to his name. If it is a ball, the opposite is true, that will take the 0.67 and subtract it from the catcher’s strike probability added. These are the equations:</p><p><strong><em>Strike Probability Added</em></strong></p><p><em>Strike: 1 — Strike Probability</em></p><p><em>Ball: Strike Probability * -1</em></p><p>In both Hook and Statler’s models, they used run value in at least one of their results, however, this assumes context affects the catcher’s ability to frame pitches and I did not want to do this. So I just stuck with strike probability added as the metric to evaluate framing.</p><p>One thing that must be done before this, though, is building a model that can predict those probabilities. The algorithm that I chose to use is a Random Forest model; it takes many decision trees and then aggregates the predictions of each of them to create the model. The data used to train this model is 75% of the data of called strikes and balls from 2021 and the test data is the other 25% from 2021; all of it is from Baseball Savant. In total, there are 352,312 observations in my dataset.</p><p><strong>Feature Selection</strong></p><p>Baseball Savant features over 90 variables in their datasets and these are very difficult to sift through when trying to decide what is important and not important. This is where feature selection comes into play. While we do know the location of the pitch is important in determining a ball or strike by the umpire, there may be other factors like the batter’s handedness, the count, the score of the game, etc. To do this, I used an algorithm called Boruta which does exactly that. I took a sample of 5000 from my dataset to run this as this was more optimal for me in terms of the run time of the code in R.</p><p>Running this gave me 14 important attributes, the two obvious ones are vertical and horizontal location. The others are shown in the plot below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/0*Rfzr_ECaHeVCB4cQ" /></figure><p>As you can see, the vertical movement of the pitch is the next most crucial variable after location, which makes some sense as some pitches dropping down into the zone may not be called as it gets into the zone at the last second, or it could drop out making it called a strike. Then there’s also the count, which shows umpires may change their calls based on the count.</p><p>One variable that was deemed important that I have decided not to use in my model is the spin rate of the pitch, this essentially accounts for the movement of the pitch, however, we already have these variables in there anyways. So that leaves me with 13 variables in my model and those are:</p><ul><li>Pitch location (vertical and horizontal)</li><li>Pitch movement (vertical and horizontal)</li><li>Count</li><li>Pitch velocity</li><li>Release position (Height, side, and extension)</li><li>Heights of the top and bottom of the strike zone</li><li>Batter handedness</li><li>Pitcher handedness</li></ul><p><strong>Running the Random Forest</strong></p><p>In the Random Forest model, I decided to edit one of the parameters, which is the number of trees created for the model. This is set at 500 in default, however, with this being a big model, 500 could not run on my computer, so instead, I set that to 200 trees. This could lead to less accurate results, however, I cannot sacrifice computing time here either.</p><p>Again, I used those 13 variables that I had mentioned earlier to build this model, it would predict the probability of whether or not the pitch was a strike. A strike would have the assigned value of “1” and a ball would be “0”. This variable was created in my dataset.</p><p><strong>Model Evaluation</strong></p><p>Now that, we have come this far, let’s make sure our model is accurate! There are two ways that I decided to do this. The first is to see if the predicted probabilities are calibrated with actual probabilities, meaning that they are similar, and use the area underneath the curve for a ROC plot.</p><p>The way to see if there is calibration between the actual and predicted probabilities is to make a plot that has the average predicted probabilities at every 10 percent mark. For example, if we take the predicted probabilities that round to 70%, the actual combined probability of these events happening should be roughly 70%, if not, then the probabilities are off. Here is the plot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/0*x8A00D67ODb-xWIq" /></figure><p>So looking at this plot, you can see that the predicted probabilities tend to be close to the actual probabilities, meaning that the model is calibrated pretty well. In the top right corner, the probabilities tend to be underpredicted as the actual probability of those points at 80 and 90 percent are a little bit higher. This also can be seen in the lower-left corner as probabilities are being over-predicted for lower actual probabilities. The model seems to be a little bit careful with extreme values as those could hurt its performance, but overall, it is calibrated well.</p><p>Now, for the ROC curve. What it does is it tells us how much our true positive rate changes as our true negative rate decreases. Preferably, we would like a graph that has a sharp increase at the start and then levels off or has a high area underneath the curve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/0*56ggtZUv_flCzSco" /></figure><p>The area underneath the curve is 0.9813, which is very good! That is a high AUC, meaning that the model is pretty accurate! So in terms of model building, this seems like it is a success!</p><p><strong>Catcher Evaluation</strong></p><p>Now that the model is complete, we can use those equations for strike probability added to help us figure out who the best catchers at framing were in 2021! Again, we will continue to use strike probability added, however, since that is a counting stat and will favor catchers that catch more pitches, I wanted to turn this into a rate stat. I took the total number of strikes stolen and divided it by the number of pitches that the catcher caught that were called by the umpire. After that, I multiplied that rate by 1000 to give us the total number of strikes added by the catcher per 1000 pitches. The average is -0.622 per 1000, which is essentially 0, the average catcher loses .622 strikes per 1000 pitches.</p><p>Before we look at the top catchers, though, let’s take a peek at what a pitch that is framed well looks like and what a pitch that is framed badly looks like. Let’s start with a bad framing job.</p><p><em>Follow the link for the video: </em><a href="https://www.mlb.com/video/daniel-bard-ball-to-dansby-swanson-zcqfku?q=PlayerId%20%3D%3D%20%5B453268%2C621020%5D%20AND%20PitchResult%20%3D%20%5B%22ball%22%5D%20AND%20Strikes%20%3D%20%5B0%5D%20AND%20Balls%20%3D%20%5B0%5D%20AND%20Outs%20%3D%20%5B2%5D%20Order%20By%20Timestamp%20DESC&amp;cp=MIXED&amp;p=0"><em>Daniel Bard ball to Dansby Swanson</em></a></p><p>This pitch had a 99% chance of being called a strike despite it being on the edge of the strike zone. It is middle height and starts by being in the strike zone anyways and would typically be called a strike. The catcher, Elias Diaz seems to catch the ball on the outside of the plate, so this is where the umpire probably interpreted it to be and called it a ball. Diaz lost 0.99 of a strike due to this lousy frame job.</p><p>Now for the good frame!</p><p><em>Follow the link for the video: </em><a href="https://www.mlb.com/video/buck-farmer-called-strike-to-ryan-jeffers-dyH7o7?q=PlayerId%20%3D%3D%20%5B571656%2C680777%5D%20AND%20Balls%20%3D%20%5B1%5D%20AND%20Strikes%20%3D%20%5B0%5D%20AND%20Outs%20%3D%20%5B0%5D%20AND%20Inning%20%3D%20%5B4%5D%20Order%20By%20Timestamp%20DESC&amp;cp=MIXED&amp;p=0"><em>Buck Farmer called strike to Ryan Jeffers</em></a></p><p>This pitch ended up way outside of the strike zone and essentially had a 0% chance of being called a strike. Wilson Ramos, however, was able to pull the pitch into the strike zone very quickly to make it look like a strike and got the umpire to call it one. The hitter, Ryan Jeffers, was not happy with this call. Ramos gained a full strike to his cumulative total due to this.</p><p>Now that we have this metric and understand it, we can evaluate to see who the best and worst pitch framers of 2021 are! To start, I wanted to get the best catchers, so I took got the top 5 catchers in terms of pitch framing and they are found in the plot below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/0*_ws2DvTzbjlzYDLJ" /></figure><p>Tomas Nido was far and away the best framer, adding 13.7 strikes per 1000 pitches, the next 4 in Gallagher, Trevino, Higashioka, and Heim were great as well, all adding more than 6 strikes to their teams.</p><p>To see the worst catchers, I did the same thing but grabbed the 5 lowest scoring catchers in the metric. So here they are!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/0*Qoz2iUXc-3ow7Bn8" /></figure><p>Two former Baltimore Orioles make the list in Austin Wynns and Chance Sisco, with Sisco being the worst framer in the league and a per pitch rate of about 16 strikes lost per 1000 pitches. We also have Tomas Nido’s former partner Rene Rivera, so the Mets had two catchers that were on opposite ends of the spectrum. Overall, these guys were not very good in 2021.</p><p>While I did say that I would rather look at this statistic from a rate standpoint, it still is insightful to see it as a counting stat as well to see how teams were affected overall by their catchers. So I made leaderboards for the top 15 and the bottom 15 catchers. Here they are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/0*AjB2VQXclF3QP5ib" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/0*74fYdoLPfaggO_eu" /></figure><p>Tomas Nido is overtaken in this list by Jose Trevino as Trevino caught almost double the amount of called pitches. He was able to add 45 strikes to his team which can be valuable to a pitching staff! They are still very close which shows Nido’s framing ability to be very good. J.T. Realmuto and Austin Hedges, who are always talked about being elite framers, show up at 6th and 14th in this list, respectfully.</p><p>Meanwhile, on the other side, we see Salvador Perez in last place, losing about 67 strikes and this hurt the below-average Royals staff in 2021. Gary Sanchez, who has had much criticism for his framing, is also close to the bottom for his abilities at number 69. There seems to be a massive drop-off after Eric Haase to Zack Collins, and then another from Collins to Pedro Severino. Severino and Perez were able to catch many more pitches than everybody else that was low on the list and so they were able to accumulate more strikes missed.</p><p><strong>Predictive Power</strong></p><p>The final thing that I want to do with this model is to see if it has any predictive power. My way of doing this is to see if framing strikes added per 1000 pitches in the first half of 2021 were able to predict the catchers’ performance in the second half of 2021. To evaluate if there is predictive power, I will use the r-squared value. This plot shows the relationship between the two along with the r-squared value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/0*cH933Eorl5o5Iixm" /></figure><p>There is a positive relationship between the 1st and 2nd half with an r-squared value 0.42, so this means that there is a moderate linear relationship between the two variables. It is great to see that there is some relationship as that means this metric can hold up even throughout a whole season and it does get the general trend correct as well. Overall, I am happy with the statistic.</p><p><strong>Conclusion and Limitations</strong></p><p>Catcher framing will be an important piece to evaluating catchers until robot umpires come into place, when that happens, the art of framing will be insignificant. Using the random forest model that I built can take many different pieces of information to help understand the likelihood of a pitch being a strike. Using that model can help us evaluate whether or not catchers are good at changing balls into strikes.</p><p>However, the model and metric are not perfect. First off, it does not take into account the skill of the umpire in calling strikes. A catcher may have framed a pitch well, however, the umpire could make a bad call, or it could have happened the other way around with a bad frame job. Another thing that the model does not see is if the catcher was crossed up and the catcher framed it badly due to this. There also could be umpire bias that is not accounted for where there could be “makeup” calls for bad calls made earlier in the game.</p><p>There are some important ways to use a metric like this as well. First, the most obvious one, is to use it to see how much value a catcher can add to your team based on how many strikes they add or lose. This can be used to see which catchers should play more or see which catchers should be acquired in trades.</p><p>Another way it can be used is to see where catchers struggle with framing and help them improve. For example, if they lose more strikes than other catchers on low and glove side pitches, they can be specifically trained on working on that spot.</p><p>One final way that I thought of is that it can be used for pitchers to throw to certain spots where catchers will be more likely to frame a pitch for a strike. Say a catcher is good at framing up and away fastballs, a pitcher could gain more value by throwing fastballs in that location more often and perform better throughout a game. <br> As I stated, there still is value in this, and there will also be value in this right away when robot umpires come into play as well. This metric will then be used to be able to take away all of the strikes added from a catcher’s framing. This could make some catchers less valuable like Tomas Nido, or some catchers that are bad framers can become more valuable like Salvador Perez.</p><p>In the future, I would be interested in taking a look at how to have pitchers optimally perform with specific umpires behind the plate based on where they call pitches, or the same thing with catchers. I would also like to add the umpire’s skill to the model to see if that makes any difference at all.</p><p>I felt like this model did its job and I am very happy with how it and the metric that I derived from it turned out!</p><p><em>Note: All data from Baseball Savant</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b482d6f159f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>