NYC taxi fare prediction and commuting patterns
Smart Commute Analytics: EDA, feature engineering and three model families on NYC TLC trip data — and why the best-scoring model was the one to distrust.
Problem
Three questions from a Smart Commute Analytics brief: when and where are the peak hours in NYC taxi demand, can fares — including the surcharges that make them opaque — be predicted reliably, and can geographic structure be used to predict better than a flat tabular model? The intended consumer is a commuter deciding whether and when to take a taxi, and a planner looking at where congestion comes from.
Data
NYC TLC trip records for July 2025 — Green Taxi (≈48k trips, 21 fields) and High-Volume For-Hire Vehicle data (≈19.7M trips, 25 fields) — plus the TLC taxi-zone lookup (265 zones with borough and service zone) and hourly NYC weather (temperature, humidity, conditions). Pickup and drop-off are zone IDs, not coordinates, which shapes everything downstream.
Cleaning decisions that mattered.
- 5,185 Green Taxi rows had no
RatecodeID,payment_type,trip_typeorstore_and_fwd_flag, and an average trip distance of 174 miles — a different record type (dispatched, unmetered), not missing values. They were dropped rather than imputed. - 762 trips with zero passengers but real fares were kept; passenger count is unreliable on these records and isn’t a fare driver.
- Weather was joined on pickup hour, which was a deliberate simplification for a one-month window.
Feature engineering
From the timestamps: pickup hour, day of week, and trip duration. From the join: hourly conditions and temperature. Categorical fields (vendor, rate code, payment type, trip type, weather condition, weekday) one-hot encoded. Zone IDs kept as integers for the tree models and used as graph nodes for the GNN.
EDA findings
- Demand is bimodal and asymmetric. A morning peak and a larger late-afternoon/evening peak, with the evening peak longer in duration and higher in average fare — the signature of surge conditions.
- Duration, not distance, carries the hour signal. Average trip duration by pickup hour rises through the afternoon while distance stays flat; fares follow duration. Congestion is the mechanism behind “expensive hours.”
- Vendor mix shifts by hour, which matters for any model that uses vendor as a feature.
- Rate-code analysis showed the fixed-fare codes (airport, negotiated) as distinct populations that a single regression shouldn’t be asked to fit alongside metered trips.
Models
Three families, in order of ambition:
| Model | Setup | Result |
|---|---|---|
| Random Forest | 80/20 split, default hyperparameters | RMSE 0.001 · R² 1.000 |
| XGBoost | same split | RMSE 1.56 · R² 0.994 |
| GCN (PyTorch Geometric) | zones as nodes, trips as edges, edge-level regression, 5 epochs | RMSE 30.8 · R² −1.27 |
The finding that matters: a perfect score is a bug
An R² of 1.000 on a held-out set is not a result, it’s a warning. The feature set included fare_amount, extra, mta_tax, tip_amount, tolls_amount, surcharges and improvement_surcharge — and the target, total_amount, is their sum. The Random Forest had learned addition. XGBoost’s 0.994 is the same leak with slightly more regularisation.
The honest version of the task predicts total_amount (or fare_amount) from what is known before the trip: pickup zone, drop-off zone, hour, day, weather, rate code, and the distance an app would estimate from the route. Everything else is a post-hoc component of the answer. Re-framing the problem this way is the single most important step in the project, and it’s the one a scoring rubric won’t catch for you.
Why the GNN underperformed — and what it was for
The graph framing is right for the data: zones are nodes, trips are edges, and fare depends on where in the network a trip happens, not just how far. The implementation was a first pass — a two-layer GCN with edge features averaged per node and a linear head on concatenated endpoint embeddings, trained five epochs, evaluated on the training graph. It scored worse than predicting the mean.
The diagnosis is unglamorous and useful: too few epochs to converge from a loss in the 900s; edge features flattened into node averages, which throws away the per-trip signal the task depends on; no train/test split at the graph level; and a target with the same leakage problem as above. None of those is a reason to abandon the approach. They’re the to-do list for the second version — a model that predicts fare from origin and destination embeddings learned from the trip graph is the version of this project that would beat a tabular baseline on a clean target.
What I’d do differently
- Define the prediction-time feature set first. Decide what a rider’s phone knows at request time, and allow nothing else into the model. The leak was avoidable by writing that sentence down before the first fit.
- Hold out by time, not by row. A random 80/20 split on one month leaks temporal structure. Train on weeks 1–3, test on week 4.
- Run the GNN properly before judging it. Hundreds of epochs, edge features kept on the edges (a GAT or edge-conditioned convolution), a proper split, and the 19.7M-row HVFHV data rather than the 43k-row Green Taxi sample.
The project was scored on EDA, feature engineering, model selection and a dashboard. The part worth keeping is the discipline it forced: a model that scores too well is the one to audit first.