Airline Delay Prediction Using Flight and Weather Data

UT Austin - MS Business Analytics Project

Team Members: Mrunal Dalvi, Archie Korale Arachchige Don, Carter St. Geme, Peyton Gibbs, Ryota Yokoyama

Abstract

Flight delays are a consistent operational challenge in the U.S. transportation industry, affecting airline efficiency, and customer experience. Predicting whether a flight will be delayed is challenging because delays can occur due to a combination of logistical problems, airport congestion, and changing weather patterns. This project addresses the problem of forecasting flight delays in real time, with the goal of developing a model that can be generalized across airports in the United States. Our team of Graduate Students at the University of Texas at Austin studying Business Analytics was able to apply the modeling techniques we have learned in the classroom through this impactful project.

To solve this problem, we used an integrated dataset of domestic flight records and hourly weather reports from public agencies. With this data, machine-learning models were developed with the goal of predicting whether a flight will be delayed by more than 15 minutes. We evaluated predictive performance, examined temporal stability, analyzed concept drift from 2018–2025, and studied which operational and weather factors most strongly drive delays. The final LightGBM model achieves strong precision, recall, and ROC-AUC, and reveals key insights into the interplay between route structure, short-term airport congestion, and nonlinear weather effects. The results show that delay prediction is feasible at a certain scale, with airport-level rolling delays (a sign of congestion) emerging as the strongest indicator. The project provides insight for airports, airlines, and passengers in quantifying how logistics and weather impact delay risk.

Introduction & Background

What is the Problem? And why is it important?

According to the Bureau of Transportation of Statistics (a source of this project), 22% of flights in the United States in the last three years have arrived at their destination gate at least 15 minutes late. This statistic affects the around 3 million people (FAA) that fly in the U.S. every day, causing frustration for customers and profit loss for airlines and airports alike. These delays arise from a combination of factors, including adverse weather conditions (such as rain, fog, or snow), air traffic congestion, maintenance issues, and scheduling conflicts. Given the wide range of contributing causes, understanding and predicting delays has become increasingly important for improving operational planning, optimizing resource allocation, and enhancing the passenger experience.

The core problem this project addresses is: Can we accurately predict whether a scheduled flight will experience a delay more than 15 minutes? Can we identify the operational and environmental factors that drive those delays? Finally, can we observe any concept drift in flight delays before COVID and after COVID?

Related Work and Novel Characteristic about Project

There has been extensive prior research in this field due to the importance outlined above. Prior work spans academic literature and operational analytics. Many classical time series models have been created to predict average delays at the airport level. Most using the specific technique of ARIMA. More recent work applies machine learning models like logistic regression, random forests, gradient boosting, and deep learning to predict delays at the flight level. This project aims to add a more robust nature to this existing modeling including multiple weather attributes as well as unique feature engineering with logistical variables. We also extend prior work through rolling-delay features and concept drift analysis across pre and post COVID years. This industry is constantly changing due to climate and socio-economic factors requiring new modeling frequently.

Outline of Approach and Rationale

1. Large-Scale Data Integration: We merge multiple national level datasets in a single flight-level analytic table.
2. Data Cleaning and Feature Engineering: Attributes filtered and aligned to account for noisy dataset, and new feature created
3. Exploratory Data Analysis: Baseline exploration of flight and weather data, providing context for future modeling
4. Model Training and Selection: Several models evaluated using cross-validation and temporal splits
5. Model Analysis: Analyze feature importance and SHAP values, as well as assess year to year stability
6. Insights: Provide actionable conclusions about the airline industry with data driven perspective

Data Collection & Description

In this project, we created a large dataset that combined U.S. flight level statistics and weather observations to predict flight delays. The goal of this combination was to build a comprehensive list of attributes that can be used as explanatory variables for the target variable of flight delays. Whether the delays be from logistics and congestion

Sources

The two sources used come from government agencies that provide public data on airline and weather statistics.

Flight Operation Data Source

For the flight level data, the Bureau of Transportation Statistics within the U.S Department of Transportation was used to supply monthly files. The link https://transtats.bts.gov is where we get the dataset “On-Time Reporting Carrier On-Time Performance” reported by the DOT. This dataset is a compressed monthly CSV file that is downloadable by year and month. This specific source was chosen because it is the official record of all domestic flights with multiple delay attributes and logistical attributes that was used in the modeling portion of the project. It captures the operational side of delays that contribute to real-world performance.

Weather Data Source

For the weather data, we downloaded data from the National Centers for Environmental Information (NCEI) within the National Oceanic and Atmospheric Administration (NOAA). The NOAA is a scientific agency responsible for weather forecasting and climate monitoring through weather stations. A weather station is a physical location equipped with instruments that measure atmospheric conditions in real time. The specific dataset used within this organization is the Integrated Surface Dataset. The website https://www.ncei.noaa.gov/data/global-hourly/access/ has yearly folders containing one file per station per year. Within each file, there is data broken down into hourly weather observations. We chose this source due to the consistent official reporting from the government agency of weather observations as well as the available data from airport stations.

Relevant Characteristics of the Data

The On-Time Performance database supplies us with flight level data that is used for the rest of the project. The key property of this dataset is that one record represents one scheduled flight from an origin airport to a destination airport. These records include flight for all major U.S. commercial airlines with airports across the domestic United States. The database has monthly files dating back to 1987 with around 6 million flight records per year. The timeframe we chose to collect data from was January 2018 to January 2025. The window gave us more than a sufficient amount of data to train our models and do exploratory data analysis while also giving up-to-date statistics. With this timeframe we were also able to look at concept drift with statistics pre-covid and post-covid.

Key attributes of the flight data:

1. FlightDate — Date of flight departure (used for merge with NOAA Weather Data)
2. CRSDepTime — Time of flight departure (used for merge with NOAA Weather Data)
3. Reporting_Airline — Code of commercial airline operating flight
4. Origin — Airport where flight is leaving from
5. Dest — Airport where flight is arriving to
6. DepDelayMinutes — Decimal value representing how many minutes flight is delayed in departure (minimum 0)
7. DepDel15 — Binary Variable, 1 representing a flight delayed for 15 minutes or more in departure
8. ArrDelayMinutes — Decimal value representing how many minutes flight is delayed in arrival (minimum 0)
9. ArrDel15 — Binary Variable, 1 representing a flight delayed for 15 minutes or more in arrival
10. Distance — Length in miles the plane travels in route from origin to destination airport

The NOAA’s Integrated Surface Database aggregates the Meteorological Aerodrome Report (METAR) and other surface observations from thousands of stations domestically. This database is from where we collect our weather and climate observations that are used as key attributes in the rest of the project predicting flight delays. Each file collected contained hourly observations for a specific year and station. Nearly every U.S. airport has a corresponding weather station which was crucial for our data. We collected only necessary airport and yearly files depending on the flight level data which will be explained more in the next section. Weather conditions are a major cause of delays and disruptions, making this dataset crucial for building a predictive model.

Key attributes of the weather data:

1. Date — Timestamp of the weather observation in UTC
2. Latitude — Geographic latitude of the reporting weather station
3. Longitude — Geographic longitude of the station
4. WND — Wind data including direction (degrees), speed (m/s), and quality codes
5. TMP — Air temperature reported in degrees Celsius
6. DEW — Dew point temperature (indicator of humidity) in degrees Celsius
7. VIS — Horizontal visibility distance measured in meters
8. CIG — Cloud ceiling height (the lowest overcast or broken cloud layer), measured in meters
9. SLP — Sea level atmospheric pressure in millibars (hPa)

Methods of Acquisition and Merging

Flight data was acquired programmatically from the Bureau of Transportation Statistics using direct ZIP file downloads from the BTS “PREZIP” archive. BTS does not provide a bulk API, so the dataset had to be retrieved one month at a time. A python script iterated over all months from January 2018 through January 2025, constructing monthly download URLs in the form https://transtats.bts.gov/PREZIP/On_Time_Reporting_Carrier_On_Time_Performance_1987_present_{year}_{month}.zip. Each ZIP file was fetched using requests.get(), read directly into memory, and converted into a Pandas DataFrame. The monthly DataFrames were appended to an in-memory list and concatenated into a single integrated flight dataset containing 46.5 million flight records and 110 flight attributes. The full flight dataset exceeded the practical memory limits for interactive analysis on a typical local environment. To allow efficient exploratory data analysis (EDA) and initial model development, we created a random sample of the full dataset using chunked processing. Instead of loading the full CSV, the data was read in chunks of 500,000 records and for each chunk a 5% random sample of rows was drawn and stored. We had a fixed random state to ensure reproducibility. The resulting sampled dataset preserves the key structure and delay patterns of the full dataset while making it computationally feasible to do all future modeling.

To obtain NOAA weather data aligned with the flight records, we followed a three-step process of identifying all airports in the flight dataset, map each airport to its nearest NOAA weather station, and finally download hourly weather files for those required stations and years. In the first step, we extracted all unique origin and destination airports and attached latitude/longitude coordinates from an external airport reference file. Each airport was then matched to its nearest NOAA station using the Haversine distance calculation. Dummy weather stations were avoided by adding a penalty to the distance calculations. This process produced a mapping from IATA airport codes to station IDs which was merged back into the flight records. Lastly, we iterated over all the years in our time range and all the required stations to download the hourly CSV files from the NOAA archive which were stored into Pandas DataFrame.

Since ISD provides hourly observations, flights were matched to the nearest hourly weather report around their scheduled departure or arrival team within 4 hours and closest station as found before. The weather data was joined twice to the flight data, once for the scheduled departure time and once for the scheduled arrival time. Two datetime columns were added to the flight dataset using the FlightDate and CRSDepTime or CRSArrTime column. The merge was performed using a closest to join (pd.merge_asof) with the keys: stations identifier, scheduled departure/arrival datetime, and weather observation datetime. If a flight could not be matched within 4 hours, the weather columns were filled with N/A values that were dealt with in future data cleaning.

Data Cleaning and Final Dataset before EDA

After creating a merged dataset between the two data sources, careful cleaning was necessary to remove invalid entries, deal with missing values, resolve inconsistencies, and standardize formats. To begin, cancelled and diverted flights were excluded from the predictive modeling dataset since we were focusing purely on flights that were delayed. Flight records missing the target variables ArrDelMinutes and ArrDel15 were removed as these entries could not be labeled in future supervised learning. Airport codes were then validated against IATA codes dataset obtained, excluding any records with airports that had been closed. Mixed-type columns were explicitly cast to strings or numeric datatypes. With the weather data, there were many composite fields that were split into separate attributes. For example, the WND column was split into WND direction, speed, and quality flags. Several NOAA codes also represented missing values, so codes like 9999 were set to NA. Finally, many attributes were removed from the merged dataset as they were irrelevant to delay prediction or redundant. For example, quality codes and marketing statistics were dropped for the final dataset. To the right are the final dataset statistics.

Data Pre-Processing & Exploration

Flight delays are more than just a nuisance; they are a multi-billion dollar problem affecting airline operations and passenger sanity alike. But before we could train any models, we had to dive deep into our data to understand the hidden drivers of disruption.

Defining the Target: The 15-Minute Threshold

Before analyzing the drivers of delays, we had to define exactly what a “delay” means for our model. We chose ArrDel15, a binary indicator used by the FAA that flags a flight as delayed if it arrives more than 15 minutes past its scheduled time. This transformed our problem into a binary classification task: Will this flight be late or not? However, this definition introduced our first major hurdle: Class Imbalance.

Our analysis revealed that only 18.1% of flights in our dataset were actually delayed by more than 15 minutes. While this is good news for passengers, it presents a significant challenge for machine learning. A model could simply predict “On Time” for every single flight and still achieve nearly 82% accuracy, while being completely useless. Overcoming this heavy bias toward on-time flights became a central focus of our modeling strategy.

Exploratory Data Analysis (EDA): Uncovering the “Why” Behind Delays

The Mystery of the Missing Values

Our first roadblock was a massive amount of missing data. Columns like CarrierDelay, WeatherDelay, and NASDelay were null in approximately 81% of the rows. Standard practice might suggest dropping these rows or imputing with the mean, but we dug deeper. We tested a hypothesis: Are these values missing because the flights weren’t delayed?

The chart above confirms our suspicion. The blue bars represent missing delay records for flights under a specific time threshold.

• At the 15-minute mark, the blue bar reaches its maximum height (~46,000 records), capturing 100% of the missing data.
• This proves that missing values don’t exist for flights delayed longer than 15 minutes.
• Conclusion: The data is Missing Not At Random (MNAR). These aren’t errors; they are on-time flights. We imputed them with 0 (no delay) rather than dropping them.

The “Summer Surge” Reality Check

Our ‘Flight Volume by Month’ plot revealed the operational heartbeat of the US aviation network. We observed a distinct peak in the summer months (Months 6–8), aligning perfectly with the ‘Summer Slump’ in delays we identified later. This visual confirms that high traffic volume is a precursor to system stress, validating our decision to use ‘Month’ and ‘Season’ as key temporal features.

Temporal Rhythms: When to Fly

We visualized delay patterns over time and discovered distinct seasonal peaks in the aviation network:

• The “Summer Slump”: July consistently appeared as the worst month for travel, likely a compound effect of high passenger volume and convective summer storms.
• The “Fall Sweet Spot”: September through November showed the highest reliability, a lull between the summer rush and winter holidays.
• Day-of-Week Trends: Mondays often showed higher delay accumulation as the business week kicked off, whereas mid-week flights remained relatively stable.

The Real Drivers of Delay

While our target variable ArrDel15 tells us if a flight was delayed, the DOT data tells us why. We analyzed the specific delay causes reported by airlines to understand the root problems:

• Carrier Delay (10.2% of flights): The most common culprit. These are issues strictly within the airline’s control, such as maintenance, crew scheduling, or cleaning.
• NAS Delay (9.3% of flights): The National Aviation System (NAS) accounts for a massive portion of disruptions. These are often volume-driven (heavy traffic) rather than just weather-driven, highlighting inefficiencies in air traffic control.
• Late Aircraft (9.2% of flights): This represents the “cascading effect.” A delay in a previous leg (e.g., a plane arriving late from NYC) forces a delay in the current leg (departing for LA). This confirmed that our model needed features that could look “backwards” in time to capture network stress.

Operational Hotspots

Not all airports are created equal. By grouping delays by Reporting_Airline and Origin, we identified clear performance tiers:

• Reliability Leaders: Delta (DL) and Republic (YX) consistently maintained the lowest delay rates.
• High Variance: Budget carriers like Frontier (F9) and JetBlue (B6) showed higher average delays and variance.
• The Route Factor: The specific pairing of Origin → Destination proved to be a stronger indicator of delay than either airport individually, capturing the complex congestion dynamics of specific flight corridors.

Feature Engineering: Turning Noise into Signal

While EDA gave us intuition, raw data rarely trains a state-of-the-art model. We had to transform our findings into mathematical signals that a machine learning algorithm could digest.

Refining the Weather Signal

Weather is a critical driver of delays, but raw meteorological data is often too noisy to be useful. We had to refine specific features to extract the real signal:

• Visibility (The Non-Linear Effect): Raw visibility had a negligible correlation (-0.009) with delays. However, we found that low visibility is what matters. We engineered a binary feature, OriginLowVisibility, flagging flights where visibility dropped below 3 miles. This simple flag identified flights with 3.7x higher average delays.
• Wind Speed vs. Gusts: Similarly, general wind speed showed low correlation with delays. Planes handle steady wind well. However, High Wind Gusts (captured in extreme value outliers) correlated strongly with specific delay clusters. We retained wind gust features to capture these specific high-stress events.
• Temperature (Feature Selection): We attempted to engineer a “Temperature Differential” feature (Origin Temp — Destination Temp) to predict weather fronts, but it showed very low predictive power. To keep the model lean and prevent noise, we excluded this feature from the final training set.

Capturing Network & Schedule Dynamics

Beyond weather, we recognized that logistical patterns are massive drivers of delay. A flight’s risk isn’t just about where it is, but when it flies and the path it takes.

• The Route Effect: While knowing the Origin or Destination is helpful, the specific pairing defines the real risk. A flight from SFO to JFK faces fundamentally different congestion pressures than SFO to LAX, even if the weather at SFO is identical. We created a combined Route feature (Origin + Dest) to capture these unique corridor dynamics, effectively allowing the model to “memorize” the congestion profile of specific flight paths.
• The Weekend Shift: Operational pressure changes drastically depending on the day. To help the model distinguish between heavy business travel volumes (Mon-Fri) and leisure capacity, we engineered an IsWeekend feature. This simple binary flag helped isolate the distinct traffic patterns and resource constraints that specifically occur on Saturdays and Sundays.

Preventing Data Leakage

A critical step in feature engineering is knowing what not to include. Our dataset contained columns like CarrierDelay and LateAircraftDelay. These perfectly explain why a delay happened, but they are recorded after the flight lands.

Get MrunalDalvi’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Including them would be “data leakage”, giving the model the answer key. To ensure our model simulated a real-world prediction scenario (predicting before takeoff), we ruthlessly dropped these post-flight metrics.

Final Feature Set

After cleaning, encoding, and engineering, we narrowed our inputs down to 19 core features spanning four categories:

1. Temporal: Month, DayOfWeek, IsWeekend, Season, DepHour
2. Operational: Distance, Reporting_Airline, Origin, Dest, Route
3. Meteorological: OriginLowVisibility, DestLowVisibility, WindGusts (Origin/Dest)
4. Contextual: Rolling_delay_origin, Rolling_delay_dest

Modeling Approach and Results

Model Selection and Rationale

We selected LightGBM (Light Gradient Boosting Machine) as the primary model for predicting whether a flight would be delayed by more than 15 minutes. LightGBM is a gradient-boosted decision tree framework that is particularly well suited for large-scale tabular datasets with both numerical and categorical features. Given the size of our dataset (over 2.2 million observations) and the presence of complex nonlinear interactions between airports, routes, time-of-day effects, and weather variables, LightGBM provided a strong balance of scalability, predictive power, and interpretability.

Compared to alternatives such as Random Forests and CatBoost, LightGBM offered significantly faster training times while maintaining robust performance. Its leaf-wise tree growth strategy enables more precise modeling of high-impact regions in the feature space, which is critical in this application where flight delays are relatively rare but operationally costly events. These characteristics made LightGBM an appropriate choice for both experimentation and potential real-world deployment.

To preserve the underlying class distribution (only ~18% of flights are delayed), the model was trained using a stratified train–test split, ensuring that delayed and non-delayed flights were proportionally represented across training and evaluation data.

Baseline Model Performance

The initial LightGBM model was trained using default parameters and evaluated using a standard decision threshold of 0.50. At this threshold, the model achieved an accuracy of approximately 81%, but this metric proved misleading due to strong class imbalance. While precision was moderate (~0.56), recall was extremely low (~0.05), resulting in a weak F1 score (~0.09). In practice, the model almost never predicted flights as delayed, severely limiting its operational usefulness.

Despite this poor recall, the baseline ROC-AUC of approximately 0.69 indicated that the model did contain meaningful predictive signal. The issue was not the model’s ability to rank flights by delay risk, but rather the default threshold suppressing minority-class predictions. This diagnosis motivated further feature engineering and post-training calibration rather than a complete change in model architecture.

Feature Engineering and Model Optimization

A major improvement resulted from the introduction of rolling delay features for both origin and destination airports. Instead of relying solely on static airport identifiers, the modeling pipeline computed short-term historical delay rates using rolling averages. These features capture real-time operational congestion and cascading delays that are not reflected in long-run averages alone.

Importantly, rolling features were computed in a time-aware manner, ensuring that only information available prior to each flight’s departure was used. Separate rolling averages were generated for origin and destination airports, allowing the model to account for asymmetric congestion patterns along specific routes.

Additional temporal features, including departure hour, peak-hour indicators, and weekend flags, were incorporated to capture travel demand cycles and airline scheduling behavior. Together, these features introduced meaningful temporal dependency into the classification task and substantially improved the model’s sensitivity to delay events.

Rather than treating delay prediction as a fixed binary classification problem, the modeling approach leveraged probability outputs from LightGBM. Predictions were generated as continuous probabilities, enabling post-training optimization of the classification threshold.

Model performance was evaluated across multiple thresholds, and a threshold of 0.30 was selected to maximize the F1 score. This adjustment significantly improved the balance between capturing true delays and avoiding excessive false positives. Notably, this optimization was performed without retraining the model, demonstrating a clear separation between model learning and operational decision rules.

Model Evaluation and Diagnostics

After optimization, model performance improved dramatically:

• Accuracy: ~90%
• Precision: ~78%
• Recall: ~66%
• F1 Score: ~0.71
• ROC-AUC: ~0.89

These results represent a substantial improvement over the baseline and reflect a model that is both predictive and practically actionable.

Performance evaluation extended beyond single-point metrics. ROC curves and Precision–Recall curves were generated to diagnose baseline and optimized behavior. These visual diagnostics confirmed that improvements were driven by genuine learning and better calibration rather than metric manipulation.

The optimized model demonstrated a significantly improved precision–recall tradeoff, indicating that the model was able to identify high-risk flights more reliably while limiting false alarms. This evaluation strategy ensured that gains were meaningful for real-world airline and passenger use cases.

Key Modeling Insights

Beyond overall feature importance rankings, the model reveals several practical insights into how and why delays propagate through the air transportation system. The dominance of route-level features suggests that delays are largely driven by structural characteristics of specific origin–destination pairs, such as hub-to-hub congestion, downstream scheduling dependencies, and systemic bottlenecks that persist regardless of short-term weather conditions. This implies that some routes are inherently more delay-prone, even under nominal operating conditions.

Temporal effects further reinforce this finding. Departure time emerged as a critical driver of delay probability, particularly during evening and peak travel hours, when congestion accumulates and recovery buffers shrink. Flights departing later in the day face a higher risk of inherited delays from earlier operations, making schedule timing a key factor in airline reliability management.

The strong performance gains from rolling airport delay features highlight the importance of short-term operational context. These features capture recent congestion and cascading disruptions at both origin and destination airports, allowing the model to identify compounding effects that static historical averages fail to reflect. This result underscores that airport performance is highly dynamic and that recent conditions are often more predictive than long-term trends.

Weather variables, while weak when used in raw continuous form, became meaningfully predictive once engineered appropriately. In particular, visibility conditions at both origin and destination airports played an outsized role, demonstrating that nonlinear threshold effects are common in aviation operations. Rather than impacting delays gradually, certain weather conditions trigger sharp increases in delay risk once critical operational limits are crossed.

Taken together, these findings confirm that flight delays are systematic rather than random. Delays emerge from an interaction of route structure, temporal demand patterns, evolving airport congestion, and threshold-based environmental constraints. Gradient boosting methods such as LightGBM are especially well suited to this problem because they naturally capture nonlinear interactions and conditional effects across these domains. As a result, the final model not only achieves strong predictive performance, but also provides interpretable insights that can inform airline scheduling, airport operations, and real-time disruption management.

Concept Drift and Temporal Validation

Predictive models trained on operational transportation data must remain robust to temporal shifts. Flight delays are influenced by evolving schedules, seasonal weather, airline operational strategies, and exogenous shocks such as the COVID-19 pandemic. Because our dataset spans from 2018 to 2025, evaluating whether the underlying data-generating process changes over time is essential for ensuring real-world reliability. Our concept drift analysis proceeded in two stages:

1. Distribution shift detection using adversarial validation and statistical drift tests, and
2. Temporal validation using year-over-year retraining.

Together, these analyses assess both whether the data distribution is changing and whether a model trained in one period can generalize to future periods.

Distribution Shift Detection via Adversarial Validation

To detect potential concept drift at the distribution level, we used adversarial validation, a technique that trains a classifier to distinguish between data from two different time periods. If the classifier can reliably tell which year a record comes from, then the feature distribution has changed meaningfully.

For each pair of consecutive years, we:

1. Labeled records from the earlier year as 0 and the later year as 1.
2. Combined the two datasets and trained a Random Forest classifier using the engineered continuous features (e.g., visibility, temperature, distance, and cyclic time features).
3. Measured how well the classifier could distinguish the years using ROC-AUC.

An ROC-AUC close to 0.50 indicates no distinguishable difference (no drift), while values above 0.70 suggest significant distribution shift.

Results showed that there is no strong evidence of year-over-year distribution drift. Across all adjacent year pairs, adversarial AUC values remained between 0.56 and 0.65, consistently below the 0.70 drift threshold.

These values indicate mild but not substantial distribution changes over time. The slightly elevated AUCs around 2019–2021 align with real-world events (COVID-19 and its aftermath) but remain below the threshold that would signal serious drift.

We also examined feature importance from the adversarial classifier to identify which variables contributed most to whatever small differences existed. While certain features (such as visibility, temperature, and congestion-related signals) occasionally appeared as “top drift features,” no single variable consistently dominated across multiple year pairs. This rotation suggests that the observed differences are local and noisy, rather than a systematic structural shift in the underlying process.

Statistical Drift Tests on Individual Features

To complement adversarial validation, we applied Kolmogorov-Smirnov (KS) tests to individual continuous features, comparing each feature’s distribution in year t vs. year t + 1.

For every consecutive year pair and each continuous variable, we:

1. Computed the KS statistic and p-value,
2. Flagged a feature as drifting if p < 0.05, and
3. Checked whether any feature showed persistent drift across multiple transitions.

The results showed that while some features occasionally produced low p-values, indicating local distribution differences (for example, year-specific variability in temperatures or visibility), no feature drifted consistently across multiple consecutive years. The pattern of significant KS results was scattered and non-systematic, consistent with normal year-to-year variability rather than genuine concept drift.

Together, adversarial validation and KS tests strongly suggest that the joint feature distribution is relatively stable over the 2018–2025 period, aside from expected short-term fluctuations.

Temporal Validation Using Year-Over-Year Retraining

While distribution-level tests help determine whether the data distribution changes over time, the more practical question is whether a model trained in one year still performs well in the next. To answer this, we implemented a time-respecting temporal validation scheme. For each year in the dataset, we restricted the data to that single year and then split it into training, validation, and test subsets. The validation set was used for early stopping, while the test set served as a held-out same-year evaluation benchmark. The neural network model was trained using globally consistent categorical encodings (shared across all years) and year-specific scaling of continuous features, where the scaler was fit only on the training subset to avoid information leakage. Early stopping was driven by validation loss, and the final model for each year was evaluated both on same-year test data (in-year generalization) and on the subsequent year’s data (forward-looking generalization). Performance was assessed using Accuracy, Precision, Recall, F1, and ROC-AUC.

Across the 2018–2025 period, the models generalized well year-over-year. Same-year accuracy typically fell between roughly 0.79 and 0.90, while next-year accuracy remained in a similar band between about 0.79 and 0.83. ROC-AUC values were consistently in the 0.63–0.70 range for both same-year and next-year evaluations. Taken together, these results indicate that the model retains most of its predictive power when applied to the following year without retraining, and that any temporal degradation in performance to be generally only a few percentage points.

There are, however, a few notable nuances. A model trained on 2019 data actually generalized better to 2020 than it performed on its own held-out 2019 test set (approximately 90% versus 81% accuracy). This suggests that 2019 may include more noise or irregular operational patterns that are not fully representative of the surrounding years. In contrast, a model trained on 2020 data achieved very high same-year accuracy (around 89.9%), but this performance did not fully carry over to 2021. This is consistent with 2020 being a structurally unusual year due to the pandemic, with reduced flight volumes, altered schedules, and atypical passenger demand.

From 2021 onward, both same-year and next-year metrics exhibit tight clustering, indicating a return to a more stable regime in the flight delay process. In other words, after the disruptions associated with COVID-19, the relationship between inputs and delays appears to have re-stabilized, allowing models trained on one post-2021 year to generalize reliably to the next.

Interpretation and Practical Implications

Taken together, the distribution-level tests and the temporal validation results point to the same conclusion that there is no evidence of persistent, structural concept drift in U.S. domestic flight delays over our 2018–2025 window. Adversarial validation AUCs remain well below the typical drift-flagging threshold of 0.70, and KS tests only pick up scattered, non-repeating differences in individual features. At the same time, both the LightGBM and neural network models maintain similar levels of accuracy and ROC-AUC when applied to future years, with only modest degradation. In other words, a model trained on historical data continues to behave predictably when moved forward in time.

From an operational perspective, this stability is encouraging. A delay prediction model trained on several years of data can be deployed with the expectation that it will perform similarly on future flights, without requiring constant retraining. The inclusion of rolling airport-level delay features further helps the model absorb short-term congestion spikes and local disruptions, reducing the need for frequent structural changes to the model itself. In practice, lightweight periodic monitoring using adversarial validation and simple statistical tests is likely sufficient to flag any emerging structural shifts, while a single, well-regularized model (such as our final LightGBM and neural network pipelines) can support multi-year deployment in production.

Conclusion

This project set out to build a comprehensive, data-driven system for predicting U.S. domestic flight delays by integrating large-scale operational data from the Bureau of Transportation Statistics with high-resolution weather observations from NOAA. We constructed and cleaned a combined dataset spanning more than 2.2 million flights across 2018–2025, engineered meaningful temporal, operational, and weather-based features, and developed a modeling pipeline capable of predicting delays with strong accuracy and interpretability. Our final LightGBM model, enhanced with rolling airport-level delay statistics and optimized decision thresholds, achieved an accuracy of ~90%, precision of ~78%, recall of ~66%, and an F1 score of ~0.71, a substantial improvement over the baseline. Through extensive diagnostic work, including ROC/PR curve analysis, feature importance studies, and operational insights, we demonstrated that delay patterns arise from nonlinear interactions between route structure, short-term congestion, departure timing, and threshold-based weather effects.

Beyond predictive performance, we conducted a detailed concept drift analysis to assess whether the underlying distribution of flight and weather conditions changed over time. Using adversarial validation, Kolmogorov-Smirnov tests, and year-over-year retraining, we found no evidence of persistent structural drift across the 2018–2025 window. While natural variation, particularly around the 2019–2021 COVID period, was observed, models consistently generalized well to subsequent years. This stability suggests that a well-calibrated model trained on multi-year data can remain operationally reliable without constant retraining, especially when supported by rolling features that capture evolving airport congestion.

Lessons Learned

Throughout this project, we gained a deep appreciation for the critical role that feature engineering plays in predictive modeling, especially when dealing with large-scale operational data. Early results demonstrated that even strong models like LightGBM perform poorly when fed raw, unrefined features; however, once we introduced rolling airport-level delay metrics, temporal encodings, and weather threshold transformations, model performance improved dramatically. We also learned the importance of addressing class imbalance thoughtfully, default thresholds severely restricted recall, but tuning the decision threshold allowed us to achieve a meaningful balance between precision and sensitivity. On the data engineering side, merging millions of flight records with hourly weather observations highlighted the complexity and importance of rigorous preprocessing, data cleaning, and careful timestamp alignment. Our concept drift analysis reinforced the necessity of validating model robustness over time, revealing that despite major external disruptions like COVID-19, the underlying structure of delay patterns remained stable. Finally, we discovered that interpretability is key for real-world adoption: understanding why certain routes, times of day, or weather conditions drive delays helps stakeholders translate predictions into actionable operational decisions.

Future Work

There are several potential extensions that could build on the foundation of this project. One natural direction is incorporating real-time or streaming data sources, such as live METAR feeds, air traffic congestion indicators, and real-time runway usage, to improve short-horizon predictions and support operational decision-making. Expanding beyond structured tabular data to include multimodal inputs, such as textual weather reports or radar imagery, could further enhance predictive power. Another improvement would be to develop specialized models for particular airports or routes, especially major hubs where congestion dynamics differ meaningfully from smaller regional airports. Future work could also explore predicting continuous delay durations rather than binary delay outcomes, allowing for more granular applications such as passenger rebooking optimization or resource allocation. From an MLOps perspective, building a monitoring pipeline that tracks model performance, identifies drift, and automatically retrains when necessary would support long-term deployment. Finally, incorporating economic cost modeling would allow airlines to not only predict delays but also prioritize interventions based on operational and financial impact, making the system even more useful for real-world decision-making.

References

Data Sources

1. Bureau of Transportation Statistics (BTS): On-Time Reporting Carrier On-Time Performance. U.S. Department of Transportation. Available at: https://transtats.bts.gov
2. National Oceanic and Atmospheric Administration (NOAA): Integrated Surface Database (ISD/METAR). National Centers for Environmental Information. Available at: https://www.ncei.noaa.gov/data/global-hourly/access/

Link for Code

• GitHub repository https://github.com/gibbs28/flight-delay-prediction