Public Sector · Advanced
Classify 59,400 Tanzanian waterpoints as functional, needing repair or broken, and serve the model from a Flask form.
Open in SQL workspace
Open in Python Lab
How to do this projectdata/ folder of the Python Lab.Access to clean, working water is a critical challenge across much of Tanzania. Thousands of waterpoints, from wells and boreholes to communal standpipes, have been installed across the country, but many are non-functional or need repair, leaving communities without reliable water. You are a data scientist supporting the Ministry of Water and its NGO partners. They want a model that predicts the operational condition of a waterpoint from its location, management, water source and technical setup, so maintenance crews and budgets can be pointed where they will do the most good. The features and labels arrive in two files that must be joined on id before anything else.
Your role
Data Scientist, Ministry of Water and NGO maintenance partners, Tanzania
Maintenance teams have no way to prioritise tens of thousands of waterpoints. Records exist for each point, but nobody can predict which ones are likely broken or about to fail, so repairs are reactive and resources are spread thin. A reliable three-class prediction (functional, functional needs repair, non functional) would let the ministry rank sites for inspection.
Set up a virtual environment and the folder structure. Load both CSV files and merge them on id. Inspect shape, columns and types, preview rows, check the target distribution, find missing values, spot suspicious zeros in coordinates, year and population, and measure the cardinality of every categorical column.
Decide per column whether to drop, fill or flag missing values. Treat zeros in construction year, population and GPS as placeholders where appropriate, and handle the 0,0 coordinates. Investigate the column groups that describe the same thing at three levels of detail and keep one with justification. Reduce high-cardinality columns such as funder, installer and subvillage. Parse date_recorded into year and month, derive pump age, encode the target and save data/processed/train_clean.csv.
Univariate distributions of key numeric and categorical features; bivariate views of each feature against the target with stacked bars and grouped box or violin plots; a numeric correlation heatmap for multicollinearity; Cramér's V for every categorical feature against status_group, ranked; and optionally a latitude-longitude scatter coloured by class.
Separate X and y, split into training and test sets (there is no external test file), define numeric and categorical feature groups from the EDA, build a ColumnTransformer for imputation, scaling and encoding, wrap it with each classifier in a Pipeline, and train logistic regression, decision tree, random forest and a gradient-boosting model with cross-validation. Optionally tune the best one.
Evaluate every model on the held-out test set with accuracy, weighted F1 and the full classification report, plot a confusion matrix for each, select the best model with a written justification, and save the pipeline and target encoder with joblib.
Build app/app.py to load the saved model, render a form at GET / and return a prediction at POST /predict. Use one index.html with Jinja2 logic so the result shows after a prediction. Keep the form to the most impactful features from the EDA rather than all 40 columns. Run it locally, then finish the README and push to GitHub.
Two files sharing an id column. The features file has 59,400 waterpoints and 40 columns: static head, date recorded, funder, installer, GPS height, longitude and latitude, waterpoint name, basin, subvillage, region, district, local government area and ward, population, public meeting flag, recording group, scheme management and name, permit flag, construction year, and grouped columns for extraction type, management, payment, water quality, quantity, source and waterpoint type. The labels file holds the status_group target with three classes. Several numeric columns use 0 as a placeholder for missing data, some coordinates sit at 0,0 in the ocean, funder, installer and subvillage have thousands of unique values, and the three classes are imbalanced.
The scenario
Objectives
Tasks and business questions
Dataset and data dictionarywater_pump_features59,400 waterpoints with 40 location, management, technical and water-source columns. Join to the labels on id. · 59,400 rows
| Column | Type | Description | Example |
|---|---|---|---|
id | integer | Waterpoint identifier shared with the labels file. | 69572 |
amount_tsh | decimal | Total static head: amount of water available to the waterpoint. Often 0. | 6000.0 |
date_recorded | date | Date the row was entered into the system. | 2011-03-14 |
funder | text | Who funded the well. Very high cardinality. | Roman |
gps_height | integer | Altitude of the well. 0 may be a placeholder. | 1390 |
installer | text | Organisation that installed the well. Very high cardinality. | Roman |
longitude | decimal |
water_pump_labelsThe status_group target for each waterpoint id: functional, functional needs repair, or non functional. · 59,400 rows
| Column | Type | Description | Example |
|---|---|---|---|
id | integer | Waterpoint identifier shared with the features file. | 69572 |
status_group | text | Target class: functional, functional needs repair, or non functional. | functional |
10 exercises worth 200 points. Your query runs against this project’s dataset and is marked automatically.
6 exercises worth 90 points. Your code runs in the Python Lab against this project’s data and is marked automatically.
Marked with the machine-learning rubric. Expect the two files merged on id first; zeros in construction year, population and coordinates investigated rather than trusted; the duplicated column groups (extraction, management, payment, quality, quantity, source, waterpoint type) reduced with justification; Cramér's V used to rank categorical features; four model families trained through Pipelines with cross-validation; weighted F1 and confusion matrices reported on a held-out set; and a running Flask form that returns one of the three classes.
| GPS longitude. 0 marks an invalid location. |
34.938 |
latitude | decimal | GPS latitude. Near 0 marks an invalid location. | -9.856 |
wpt_name | text | Name of the waterpoint if available. | none |
num_private | integer | Usage unspecified; explore during EDA. | 0 |
basin | text | Geographic water basin. | Lake Nyasa |
subvillage | text | Sub-village location. Very high cardinality. | Mnyusi B |
region | text | Region name. | Iringa |
region_code | integer | Region as a numeric code. | 11 |
district_code | integer | District as a numeric code. | 5 |
lga | text | Local Government Area. | Ludewa |
ward | text | Administrative ward. | Mundindi |
population | integer | Population around the well. 0 may be a placeholder. | 109 |
public_meeting | boolean | Whether a public meeting was held. Has missing values. | True |
recorded_by | text | Group that entered the row. Check whether it varies at all. | GeoData Consultants Ltd |
scheme_management | text | Who operates the waterpoint. | VWC |
scheme_name | text | Name of the operating scheme. Many missing values. | Roman |
permit | boolean | Whether the waterpoint is permitted. Has missing values. | False |
construction_year | integer | Year built. 0 means unknown. | 1999 |
extraction_type | text | Extraction mechanism, most detailed level of a three-column group. | gravity |
extraction_type_group | text | Grouped extraction type. | gravity |
extraction_type_class | text | Extraction type class, coarsest level. | gravity |
management | text | How the waterpoint is managed. | vwc |
management_group | text | Grouped management type. | user-group |
payment | text | What the water costs. | pay annually |
payment_type | text | Type of payment; overlaps with payment. | annually |
water_quality | text | Quality of the water. | soft |
quality_group | text | Grouped water quality. | good |
quantity | text | Quantity of water available. | enough |
quantity_group | text | Grouped quantity; nearly identical to quantity. | enough |
source | text | Source of the water. | spring |
source_type | text | Type of water source. | spring |
source_class | text | Class of water source. | groundwater |
waterpoint_type | text | Kind of waterpoint. | communal standpipe |
waterpoint_type_group | text | Grouped waterpoint type. | communal standpipe |
Notebook (water_pump_analysis.ipynb) with clearly labelled markdown sections for each of the five steps: loading and exploration, cleaning and feature engineering, EDA and correlation, pipelines and modelling, evaluation and saving