Public Sector · Advanced
Water Pump Functionality Prediction
Classify 59,400 Tanzanian waterpoints as functional, needing repair or broken, and serve the model from a Flask form.
Open in SQL workspaceThe scenario
The scenarioAccess to clean, working water is a critical challenge across much of Tanzania. Thousands of waterpoints, from wells and boreholes to communal standpipes, have been installed across the country, but many are non-functional or need repair, leaving communities without reliable water. You are a data scientist supporting the Ministry of Water and its NGO partners. They want a model that predicts the operational condition of a waterpoint from its location, management, water source and technical setup, so maintenance crews and budgets can be pointed where they will do the most good. The features and labels arrive in two files that must be joined on id before anything else.
Your role
Data Scientist, Ministry of Water and NGO maintenance partners, Tanzania
The business problem
Maintenance teams have no way to prioritise tens of thousands of waterpoints. Records exist for each point, but nobody can predict which ones are likely broken or about to fail, so repairs are reactive and resources are spread thin. A reliable three-class prediction (functional, functional needs repair, non functional) would let the ministry rank sites for inspection.
Objectives
Objectives- 1Load both files and merge them on the shared id column before any exploration.
- 2Investigate missing values, placeholder zeros and invalid GPS coordinates, and handle each deliberately.
- 3Reduce redundant column groups and high-cardinality categoricals with justification.
- 4Engineer features such as pump age from construction year and the recorded date.
- 5Explore numeric and categorical features against the target, using Cramér's V for categorical associations.
- 6Build ColumnTransformer preprocessing inside scikit-learn Pipelines and train at least four model families with cross-validation.
- 7Evaluate with accuracy, weighted F1 and per-class confusion matrices on a held-out test set, then save the best pipeline.
- 8Deploy the saved model in a single-page Flask app with a form and a prediction result.
Tasks and business questions
Tasks and business questions- 1
Step 1 · Load and explore
Set up a virtual environment and the folder structure. Load both CSV files and merge them on id. Inspect shape, columns and types, preview rows, check the target distribution, find missing values, spot suspicious zeros in coordinates, year and population, and measure the cardinality of every categorical column.
- How many rows and columns do you have after the merge, and did every id match?
- What is the class balance of status_group, and what does it mean for accuracy as a metric?
- Which columns have missing values, which have suspicious zeros, and which have more than 100 unique values?
- 2
Step 2 · Clean and engineer features
Decide per column whether to drop, fill or flag missing values. Treat zeros in construction year, population and GPS as placeholders where appropriate, and handle the 0,0 coordinates. Investigate the column groups that describe the same thing at three levels of detail and keep one with justification. Reduce high-cardinality columns such as funder, installer and subvillage. Parse date_recorded into year and month, derive pump age, encode the target and save data/processed/train_clean.csv.
- Which zeros did you treat as missing, and how did you fill or flag them?
- Within each duplicated column group, which column did you keep and why?
- How did you reduce the cardinality of funder, installer and subvillage?
- How did you compute pump age, and how did you handle unknown construction years?
- 3
Step 3 · EDA and correlation analysis
Univariate distributions of key numeric and categorical features; bivariate views of each feature against the target with stacked bars and grouped box or violin plots; a numeric correlation heatmap for multicollinearity; Cramér's V for every categorical feature against status_group, ranked; and optionally a latitude-longitude scatter coloured by class.
- Which categorical features have the highest Cramér's V with the target?
- Which numeric features are collinear, and which will you drop?
- Do quantity, water quality or extraction type separate the classes clearly?
- Is there a spatial pattern in functionality?
- 4
Step 4 · Pipelines and modelling
Separate X and y, split into training and test sets (there is no external test file), define numeric and categorical feature groups from the EDA, build a ColumnTransformer for imputation, scaling and encoding, wrap it with each classifier in a Pipeline, and train logistic regression, decision tree, random forest and a gradient-boosting model with cross-validation. Optionally tune the best one.
- How is your ColumnTransformer configured for numeric and categorical columns?
- What were the cross-validated scores for each of the four models?
- Which hyperparameters did you tune, and did it help?
- 5
Step 5 · Evaluate and save
Evaluate every model on the held-out test set with accuracy, weighted F1 and the full classification report, plot a confusion matrix for each, select the best model with a written justification, and save the pipeline and target encoder with joblib.
- Which class is hardest to predict, and where do the errors concentrate?
- Which model did you choose, and why?
- 6
Deploy with Flask
Build app/app.py to load the saved model, render a form at GET / and return a prediction at POST /predict. Use one index.html with Jinja2 logic so the result shows after a prediction. Keep the form to the most impactful features from the EDA rather than all 40 columns. Run it locally, then finish the README and push to GitHub.
- Which features did you expose in the form, and why those?
- How does the app map form inputs into the shape the pipeline expects?
- What would you tell a maintenance planner about the model's limits?
Dataset and data dictionary
Dataset and data dictionaryTwo files sharing an id column. The features file has 59,400 waterpoints and 40 columns: static head, date recorded, funder, installer, GPS height, longitude and latitude, waterpoint name, basin, subvillage, region, district, local government area and ward, population, public meeting flag, recording group, scheme management and name, permit flag, construction year, and grouped columns for extraction type, management, payment, water quality, quantity, source and waterpoint type. The labels file holds the status_group target with three classes. Several numeric columns use 0 as a placeholder for missing data, some coordinates sit at 0,0 in the ocean, funder, installer and subvillage have thousands of unique values, and the three classes are imbalanced.
Waterpoint featureswater_pump_features
59,400 waterpoints with 40 location, management, technical and water-source columns. Join to the labels on id. · 59,400 rows
| Column | Type | Description | Example |
|---|---|---|---|
id | integer | Waterpoint identifier shared with the labels file. | 69572 |
amount_tsh | decimal | Total static head: amount of water available to the waterpoint. Often 0. | 6000.0 |
date_recorded | date | Date the row was entered into the system. | 2011-03-14 |
funder | text | Who funded the well. Very high cardinality. | Roman |
gps_height | integer | Altitude of the well. 0 may be a placeholder. | 1390 |
installer | text | Organisation that installed the well. Very high cardinality. | Roman |
longitude | decimal | GPS longitude. 0 marks an invalid location. | 34.938 |
latitude | decimal | GPS latitude. Near 0 marks an invalid location. | -9.856 |
wpt_name | text | Name of the waterpoint if available. | none |
num_private | integer | Usage unspecified; explore during EDA. | 0 |
basin | text | Geographic water basin. | Lake Nyasa |
subvillage | text | Sub-village location. Very high cardinality. | Mnyusi B |
region | text | Region name. | Iringa |
region_code | integer | Region as a numeric code. | 11 |
district_code | integer | District as a numeric code. | 5 |
lga | text | Local Government Area. | Ludewa |
ward | text | Administrative ward. | Mundindi |
population | integer | Population around the well. 0 may be a placeholder. | 109 |
public_meeting | boolean | Whether a public meeting was held. Has missing values. | True |
recorded_by | text | Group that entered the row. Check whether it varies at all. | GeoData Consultants Ltd |
scheme_management | text | Who operates the waterpoint. | VWC |
scheme_name | text | Name of the operating scheme. Many missing values. | Roman |
permit | boolean | Whether the waterpoint is permitted. Has missing values. | False |
construction_year | integer | Year built. 0 means unknown. | 1999 |
extraction_type | text | Extraction mechanism, most detailed level of a three-column group. | gravity |
extraction_type_group | text | Grouped extraction type. | gravity |
extraction_type_class | text | Extraction type class, coarsest level. | gravity |
management | text | How the waterpoint is managed. | vwc |
management_group | text | Grouped management type. | user-group |
payment | text | What the water costs. | pay annually |
payment_type | text | Type of payment; overlaps with payment. | annually |
water_quality | text | Quality of the water. | soft |
quality_group | text | Grouped water quality. | good |
quantity | text | Quantity of water available. | enough |
quantity_group | text | Grouped quantity; nearly identical to quantity. | enough |
source | text | Source of the water. | spring |
source_type | text | Type of water source. | spring |
source_class | text | Class of water source. | groundwater |
waterpoint_type | text | Kind of waterpoint. | communal standpipe |
waterpoint_type_group | text | Grouped waterpoint type. | communal standpipe |
Waterpoint labelswater_pump_labels
The status_group target for each waterpoint id: functional, functional needs repair, or non functional. · 59,400 rows
| Column | Type | Description | Example |
|---|---|---|---|
id | integer | Waterpoint identifier shared with the features file. | 69572 |
status_group | text | Target class: functional, functional needs repair, or non functional. | functional |
Required deliverables
- Notebook (water_pump_analysis.ipynb) with clearly labelled markdown sections for each of the five steps: loading and exploration, cleaning and feature engineering, EDA and correlation, pipelines and modelling, evaluation and saving
- Cleaned dataset saved to data/processed/train_clean.csv
- Saved best pipeline at models/best_model.pkl with any target encoder alongside it
- Flask app (app/app.py and app/templates/index.html) serving a single-page form and prediction result at http://127.0.0.1:5000/
- reports/figures with the saved EDA plots, requirements.txt, .gitignore and README.md, all pushed to GitHub
Notebook (water_pump_analysis.ipynb) with clearly labelled markdown sections for each of the five steps: loading and exploration, cleaning and feature engineering, EDA and correlation, pipelines and modelling, evaluation and savingHow it is marked
- Data loading and cleaning15%
- Exploratory and correlation analysis15%
- Feature engineering15%
- Pipelines and modelling20%
- Evaluation and model selection15%
- Deployment10%
- Documentation and reproducibility10%
- Pass mark60%
Marked with the machine-learning rubric. Expect the two files merged on id first; zeros in construction year, population and coordinates investigated rather than trusted; the duplicated column groups (extraction, management, payment, quality, quantity, source, waterpoint type) reduced with justification; Cramér's V used to rank categorical features; four model families trained through Pipelines with cross-validation; weighted F1 and confusion matrices reported on a held-out set; and a running Flask form that returns one of the three classes.