Finance · Advanced
Credit Risk Prediction
Predict which Kenyan digital-loan customers will default, from 68,000 loan records and FRED macro indicators, and ship the model as an API.
Open in SQL workspaceThe scenario
The scenarioYou are a data scientist at a Kenyan digital lender that funds short-term loans together with partner lenders. The credit team wants a machine-learning system that predicts whether a customer will default, so high-risk borrowers can be priced or declined before money goes out. You have historical loan, customer and lender records: some customers appear many times, and a single loan can appear with several lenders because loans are co-funded. You also have macroeconomic indicators from the FRED portal to enrich the picture. Your job is to explore the data, engineer features, train and compare classifiers, save the best pipeline, and deploy it behind Flask or FastAPI.
Your role
Data Scientist, credit risk team at a Kenyan digital lender
The business problem
Financial institutions lose money when customers fail to repay, and traditional credit scoring does not fully capture borrower behaviour or the economic conditions loans are issued into. The lender needs a model that accurately predicts default from historical loan, customer and economic data, and a way to call it on new applications.
Objectives
Objectives- 1Understand the structure of the dataset, including repeated customers and co-funded loans.
- 2Explore relationships between features and loan default behaviour.
- 3Clean and preprocess the data: missing values, duplicates, data types, dates and outliers.
- 4Engineer date-based, financial and customer-based features, and join the macroeconomic indicators.
- 5Build scikit-learn pipelines and train several classification models.
- 6Compare model performance with accuracy, precision, recall, F1 and ROC-AUC on an imbalanced target.
- 7Save the best-performing model and its preprocessing pipeline.
- 8Deploy the model with Flask or FastAPI so a request with loan details returns a prediction.
Tasks and business questions
Tasks and business questions- 1
Set up and load the data
Create a virtual environment, install the suggested libraries, and lay out the project folders (data/raw, data/processed, notebooks, models, app, src). Load both CSV files, inspect shape and data types, explore missing values and understand the target distribution.
- How many rows, customers, loans and lenders are there, and why do the counts differ?
- What share of loans defaulted, and what does that imply for the metrics you will use?
- Which columns have missing values, and which columns are dates stored as text?
- 2
Clean the data
Handle missing values, duplicate records, incorrect data types, date formatting and outliers. Convert disbursement and due dates to datetime, check for duplicate rows, and decide how to treat the same loan appearing with several lenders.
- Which rows are duplicates, and which only look like duplicates because a loan is co-funded?
- How did you treat outliers in Total_Amount and duration, and why?
- Which missing values did you drop and which did you impute?
- 3
Explore the data
Univariate analysis of loan amounts, durations and the target; bivariate analysis of loan type, new versus repeat and amount against the target; multivariate views with correlation matrices and grouped statistics, using histograms, box plots, count plots, heatmaps and scatter plots.
- How do loan amounts and durations distribute, and do defaulters differ?
- Do repeat customers default more or less than new ones?
- Which loan types carry the highest default rate?
- 4
Correlation and relationship analysis
Investigate numerical features against the target with a correlation matrix, point-biserial correlation and feature importance; categorical features with chi-square tests, crosstabs and grouped target distributions. Check multicollinearity and identify redundant features.
- Which numerical features correlate most with default, and which are collinear with each other?
- What does a chi-square test say about loan_type and New_versus_Repeat against target?
- Which features are redundant and can be dropped?
- 5
Engineer features
Create date-based features (loan month, year, duration category), financial features (interest_amount = Total_Amount_to_Repay − Total_Amount, repayment_ratio = Total_Amount_to_Repay / Total_Amount) and customer-based features (loan frequency, previous borrowing patterns, aggregated customer statistics). Join the FRED indicators to each loan by country and disbursement year.
- How did you reshape the wide economic indicators table so it joins to loans by country and year?
- Which customer-level aggregates did you build, and how did you avoid leaking future loans into past ones?
- Which engineered feature turned out most predictive?
- 6
Preprocess and train models
Encode categoricals, scale numerics and build the preprocessing with ColumnTransformer inside a scikit-learn Pipeline. Train logistic regression, decision tree, random forest and at least one of XGBoost, LightGBM or CatBoost, with cross-validation on a stratified split.
- What does your ColumnTransformer do to each group of columns?
- Which models did you train, and what were their cross-validated scores?
- Did you tune hyperparameters, and what changed?
- 7
Evaluate and save the best model
Compare models on the held-out set with accuracy, precision, recall, F1 and ROC-AUC, plot confusion matrices and ROC curves, choose the final model with a justification, and save both the model and the preprocessing pipeline with joblib.
- Which model did you choose, and why is it better for the lender than the one with the highest accuracy?
- What are its recall and precision on the default class, and what does each error cost the business?
- 8
Deploy the model
Build a Flask or FastAPI app that receives user input, applies the preprocessing pipeline, generates a prediction and returns the result. Write the README, freeze requirements and push the repository.
- What inputs does your endpoint require, and how does it handle a malformed request?
- How would a loan officer use the prediction, and what threshold did you choose?
Dataset and data dictionary
Dataset and data dictionaryTwo files. The loans table has 68,654 rows and 16 columns: row ID, customer, country, loan and lender identifiers, loan type, amount disbursed and amount to repay, disbursement and due dates, duration in days, whether the customer is new or repeat, the lender's funded amount, share and expected repayment, and the binary target (1 = defaulted, 0 = repaid). The economic indicators file is a wide table of 27 country-indicator rows with one column per year from 2001 to 2023, covering inflation, exchange rate, real, deposit and lending interest rates, interest rate spread, precipitation, fossil fuel energy use and unemployment for several African countries.
Loan records (train)credit_loans
68,654 loan-lender rows with customer, loan, lender and repayment details and the default target. · 68,654 rows
| Column | Type | Description | Example |
|---|---|---|---|
ID | text | Unique identifier for each row. | ID_257244237466267278 |
customer_id | integer | Unique customer identifier; a customer can appear on many rows. | 257244 |
country_id | text | Country where the loan was issued. | Kenya |
tbl_loan_id | integer | Unique loan identifier; a loan appears once per funding lender. | 237466 |
lender_id | integer | Identifier of the lender funding this portion. | 267278 |
loan_type | text | Category of loan. | Type_1 |
Total_Amount | decimal | Total loan amount disbursed to the customer. | 16477.0 |
Total_Amount_to_Repay | decimal | Total the customer must repay, including interest and fees. | 16642.0 |
disbursement_date | date | Date the loan was issued. | 2022-08-15 |
due_date | date | Date repayment is due. | 2022-08-22 |
duration | integer | Loan term in days. | 7 |
New_versus_Repeat | text | Whether this is the customer's first loan or a repeat loan. | Repeat Loan |
Amount_Funded_By_Lender | decimal | Portion of the loan funded by this lender. | 2436.0 |
Lender_portion_Funded | decimal | Share of the total loan funded by this lender (0 to 1). | 0.1478 |
Lender_portion_to_be_repaid | decimal | Amount this lender expects to recover. | 2460.0 |
target | integer | 1 = customer defaulted, 0 = customer repaid. | 0 |
FRED macroeconomic indicatorseconomic_indicators
27 country-indicator rows, one column per year 2001 to 2023: inflation, exchange rate, interest rates, precipitation, energy use and unemployment. · 27 rows
| Column | Type | Description | Example |
|---|---|---|---|
Country | text | Country the indicator describes. | Kenya |
Indicator | text | Indicator name, e.g. inflation (FP.CPI.TOTL.ZG), exchange rate (PA.NUS.FCRF), real, deposit and lending interest rates, interest rate spread, precipitation, fossil fuel energy use, unemployment. | Inflation, consumer prices (annual %) |
YR2001 … YR2023 | decimal | One column per year with the indicator's value; unpivot these to join by year. | 7.99 |
Required deliverables
- Clean, labelled notebooks following the expected structure: data loading, cleaning, EDA, feature engineering, model training and model evaluation
- Trained machine-learning model and saved preprocessing pipeline (models/best_model.pkl and models/pipeline.pkl)
- Source modules for preprocessing, training, prediction and utilities under src/
- Deployment application (Flask or FastAPI) that receives loan details, applies the pipeline and returns the prediction
- README.md documenting the approach, results and how to run the app, plus requirements.txt and a pushed GitHub repository
Clean, labelled notebooks following the expected structure: data loading, cleaning, EDA, feature engineering, model training and model evaluationHow it is marked
- Data loading and cleaning15%
- Exploratory and correlation analysis15%
- Feature engineering15%
- Pipelines and modelling20%
- Evaluation and model selection15%
- Deployment10%
- Documentation and reproducibility10%
- Pass mark60%
Marked with the machine-learning rubric. Expect the target imbalance to be measured and to shape the metric choice; repeated customers and co-funded loans to be handled without leaking the target across rows of the same loan; features such as interest amount, repayment ratio and customer loan frequency to be justified; several model families compared on a held-out set with ROC-AUC and recall on the default class reported; and a deployed endpoint that returns a prediction for a new loan.