Subpopulation FPR / TPR

To assess fairness on your predictions, is a good idea to visualize the False Positive Rate (FPR) and True Positive Rate (TPR) for different subgroups. Please refer to [1] to further details on this predictions. In the next post I will show how Kafkanator can be used to acomplish this task:

This time, lets use CreditScore dataset to compute FPR and TPR: This dataset gathers personal data to different customers. At the end it tells us if a given customer will default to pay a credit.

Let's say we train an ensemble predictor on this dataset [INCLUDE DATASET] and it predicts on column Prediction on this dataset.

Now we want to know how fair is this ensemble predictor is performing using FPR and TPR metrics. We can actually do it very easily and quickly with Kafkanator. First the imports :

import matplotlib.pyplot as plt
import pandas as pd
from sklearn.metrics import confusion_matrix

from kafkanator.fairness import fpr_fnr
from kafkanator.util import categorize_interval,rename_column

We will use the fpr_fnr function along with utility functions categorize_interval and rename_column whose main objective is to rename numerical values into string for visualization purposes (next sprint):

df2 =  pd.read_csv('ensemblist.csv')

df2['SEX'] = rename_column(df2,'SEX',{1:"male",2:"female"})
df2['MARRIAGE'] = rename_column(df2,'MARRIAGE',{1:"married",2:"single",3:"others"})
df2["AGE"] = interval_categorize(df2, "AGE", { 'young':(20,30) , 'adult':(31,50), 'elder': (51,70)  })

rename_column changes the numerical values 1,2,3 for string descriptions in order to make easier the visualization part. interval_categorize is a helpful method that collapse numerical values into intervals and assign labels to make easier the manipulation of high diversity metrics such as age. You define a dictionary containing labels and intervals (for example { 'young':(20,30) , 'adult':(31,50), 'elder': (51,70) } ) and kafkanator will perform the renaming for you . This step is only needed if you want to get more meaningful visualizations. Your dataset will be transformed from table 1 to table below :

Once you have your table like you want you can launch the fpr_fnr function :

d1=fpr_fnr(df2,'SEX')
d2=fpr_fnr(df2,'MARRIAGE')
d3=fpr_fnr(df2,'AGE')
print ( ' FALSE POSITIVE AND NEGATIVE RATES :  ', d1 , ' ', d2 , ' ', d3)

The print command will output :

    
{'female': 0.040474528960223306, 'male': 0.05430316490838423}, {'female': 0.6341398564211645, 'male': 0.6012547926106657})   ({'others': 0.06944444444444445, 'single': 0.046648155493851644, 'married': 0.0441486305305497}, {'others': 0.6404494382022472, 'single': 0.620059880239521, 'married': 0.619181505779444})   ({'elder': 0.06480380499405469, 'young': 0.05289081740354169, None: '', 'adult': 0.03880756328148826}, {'elder': 0.5786713286713286, 'young': 0.5931174089068826, None: '', 'adult': 0.6452693273792911}
    

According to this numbers, there is no significant gap in FPR nor FNR between female and male nor single or married. however there is a slightly bias on adults on FPR ( 0.03 vs 0.05 and 0.06 ).[TO BE REVIEWED].

[1] Fairness definitions explained. Verma et al. 2018 ACM International workshop on software fairness.