Bike Sharing MLOps Pipeline
ProductionA production-grade MLOps pipeline for bike sharing demand prediction that demonstrates the full machine learning lifecycle: data validation, ETL to star schema, feature engineering, model training with experiment tracking, model registry, REST API serving, and automated drift monitoring. Built entirely with Docker Compose for reproducibility.
Problem
Most student ML projects stop at a Jupyter notebook with a trained model. I wanted to build something that mirrors how ML actually works in industry: orchestrated pipelines, validated data, tracked experiments, versioned models, served APIs, and monitored performance over time. The bike sharing dataset is simple enough to understand quickly but complex enough to justify a full MLOps stack.
Architecture
The pipeline is orchestrated by Apache Airflow with 8 sequential tasks: validate source data with Great Expectations, transform to star schema, load into MariaDB ColumnStore, extract into Redis, engineer features with temporal splitting, verify Redis storage, train XGBoost with MLflow logging, and register the best model. FastAPI serves the registered model via REST API. A standalone monitoring script uses Evidently to detect data drift and model performance degradation.
Challenges
The hardest part was getting Airflow, ColumnStore, and MLflow to communicate correctly inside Docker networks. ColumnStore's CMAPI required manual triggering on startup, which I automated in the start script. Another challenge was designing the Redis storage layer - I used PyArrow for serialization to preserve pandas dtypes across task boundaries. Temporal splitting (not random) was critical for this time-series problem but broke standard sklearn patterns, so I implemented custom split logic.
Lessons Learned
MLOps is about reproducibility and observability, not just automation. Having Great Expectations validation at the start of the pipeline caught schema drift before it corrupted training. MLflow experiment tracking made it trivial to compare hyperparameter runs and select the best model. The biggest lesson: always test the serving layer - I wrote 21 pytest tests covering API prediction and model performance because a model that can't be served is useless.