공지

• 본 포스트는 재직자 교육을 위해 만든 강의안의 일부입니다.

Introduction

대회 개요

Many people struggle to get loans due to insufficient or non-existent credit histories. And, unfortunately, this population is often taken advantage of by untrustworthy lenders. Home Credit strives to broaden financial inclusion for the unbanked population by providing a positive and safe borrowing experience. In order to make sure this underserved population has a positive loan experience, Home Credit makes use of a variety of alternative data–including telco and transactional information–to predict their clients’ repayment abilities. While Home Credit is currently using various statistical and machine learning methods to make these predictions, they’re challenging Kagglers to help them unlock the full potential of their data. Doing so will ensure that clients capable of repayment are not rejected and that loans are given with a principal, maturity, and repayment calendar that will empower their clients to be successful.

데이터 파일 구조

• 데이터 파일 구조는 크게 2가지로 나뉠 수 있다.

• Credit Bureau 파일 구조와 Home Credit 파일 구조로 분리할 수 있음.
• Bureau (External File)
• Home Credit (Internal File)
  • 현재 Home Credit File (Application train/test)
  • 이전 Home Credit File (Other’s)

데이터 설명

• 본 캐글 대회 영문 참조

• application_{train|test}.csv

  • This is the main table, broken into two files for Train (with TARGET) and Test (without TARGET).
  • Static data for all applications. One row represents one loan in our data sample.

• bureau.csv

  • All client’s previous credits provided by other financial institutions that were reported to Credit Bureau (for clients who have a loan in our sample).
  • For every loan in our sample, there are as many rows as number of credits the client had in Credit Bureau before the application date.

• bureau_balance.csv

  • Monthly balances of previous credits in Credit Bureau.
  • This table has one row for each month of history of every previous credit reported to Credit Bureau – i.e the table has (#loans in sample * # of relative previous credits * # of months where we have some history observable for the previous credits) rows.

• POS_CASH_balance.csv

  • Monthly balance snapshots of previous POS (point of sales) and cash loans that the applicant had with Home Credit.
  • This table has one row for each month of history of every previous credit in Home Credit (consumer credit and cash loans) related to loans in our sample – i.e. the table has (#loans in sample * # of relative previous credits * # of months in which we have some history observable for the previous credits) rows.

• credit_card_balance.csv

  • Monthly balance snapshots of previous credit cards that the applicant has with Home Credit.
  • This table has one row for each month of history of every previous credit in Home Credit (consumer credit and cash loans) related to loans in our sample – i.e. the table has (#loans in sample * # of relative previous credit cards * # of months where we have some history observable for the previous credit card) rows.

• previous_application.csv

  • All previous applications for Home Credit loans of clients who have loans in our sample.
  • There is one row for each previous application related to loans in our data sample.

• installments_payments.csv

  • Repayment history for the previously disbursed credits in Home Credit related to the loans in our sample.
  • There is a) one row for every payment that was made plus b) one row each for missed payment.
  • One row is equivalent to one payment of one installment OR one installment corresponding to one payment of one previous Home Credit credit related to loans in our sample.

• HomeCredit_columns_description.csv

  • This file contains descriptions for the columns in the various data files.

평가 지표

• 제출한 파일은 예측 확률과 기존 관측 대상 사이의 ROC Curve로 평가 한다.

분석 준비

패키지 불러오기

• 주요 패키지를 불러오도록 한다.

library(tidyverse)
library(skimr)
library(GGally)
library(plotly)
library(viridis)
library(caret)
library(DT)
library(data.table)
library(lightgbm)
library(kableExtra)

데이터 불러오기

• 이번에는 fread()를 활용하여 데이터를 불러도록 한다.
• skim() 함수를 활용하면 아래와 같은 결과물을 확인할 수 있다.

na_strings = c("NA", "NaN", "?", "")
train = fread('data/home-credit-default-risk/application_train.csv', 
              stringsAsFactors = FALSE, 
              data.table = FALSE, na.strings = na_strings)

train %>% skim()

• 위 코드를 토대로 다른 데이터도 불러와서 확인을 해본다.

주요 변수 시각화

Target

• 우선 Target Variable의 데이터를 확인해본다.
• 먼저 데이터 요약을 해보자.

train %>% 
  group_by(TARGET) %>%
  summarise(Count = n())

## # A tibble: 2 × 2
##   TARGET  Count
##    <int>  <int>
## 1      0 282686
## 2      1  24825

• 전체적인 비율을 구해본다.

# fillColor = "#FFA07A"
# fillColor2 = "#F1C40F"
# fillColorLightCoral = "#F08080"

train %>% 
  group_by(TARGET) %>%
  summarise(Count = n() / nrow(train) * 100) %>% 
  arrange(desc(Count)) %>% 
  ungroup() %>% 
  mutate(TARGET = reorder(TARGET, Count)) %>% 
  ggplot(aes(x = TARGET, y = Count)) + 
    geom_bar(stat = "identity", fill = "#F1C40F") + 
    geom_text(aes(x = TARGET, y = 1, label = paste0(round(Count, 2), " %")), 
              hjust = 0, vjust = .5, size = 3.5, colour = "white", fontface = "bold") + 
    coord_flip() + 
    labs(x = "TARGET", 
         y = "Percentage (%)", 
         title = "Percentage of Target Data (N = 307,511)", 
         subtitle = "0 - Capable to Repay Debt; 1 - Not Capable to Repay Debt", 
         caption = "Created by McBert") + 
    theme_minimal()

Gender

• 이번에는 Gender의 숫자를 구해본다.

train %>% 
  group_by(CODE_GENDER) %>%
  summarise(Count = n())

## # A tibble: 3 × 2
##   CODE_GENDER  Count
##   <chr>        <int>
## 1 F           202448
## 2 M           105059
## 3 XNA              4

• 이번에는 TARGET ~ TARGET 데이터를 요약해본다.

train %>% 
  group_by(CODE_GENDER, TARGET) %>%
  summarise(Count = n())

## `summarise()` has grouped output by 'CODE_GENDER'. You can override using the `.groups` argument.

## # A tibble: 5 × 3
## # Groups:   CODE_GENDER [3]
##   CODE_GENDER TARGET  Count
##   <chr>        <int>  <int>
## 1 F                0 188278
## 2 F                1  14170
## 3 M                0  94404
## 4 M                1  10655
## 5 XNA              0      4

• 이번에는 시각화를 해본다.

train %>% 
  group_by(CODE_GENDER, TARGET) %>%
  summarise(Count = n() / nrow(train) * 100) %>% 
  arrange(desc(Count)) %>% 
  ungroup() %>% 
  mutate(CODE_GENDER = reorder(CODE_GENDER, Count), 
         TARGET = as.factor(TARGET)) %>% 
  ggplot(aes(x = CODE_GENDER, y = Count, fill = TARGET)) + 
    geom_bar(stat = "identity", position = position_dodge(width = 1)) + 
    geom_text(aes(x = CODE_GENDER, y = Count + 2, label = paste0(round(Count, 2), " %"), group = TARGET), 
              position = position_dodge(width = 1), size = 3.5, colour = "black", fontface = "bold") + 
    labs(x = "Gender", 
         y = "Percentage (%)", 
         title = "Percentage of Target ~ Gender Data (N = 307,511)", 
         subtitle = "0 - Capable to Repay Debt; 1 - Not Capable to Repay Debt", 
         caption = "Created by McBert") + 
    scale_fill_manual(values = c("#F5896E", "#7BDCFF")) + 
    theme_minimal()

## `summarise()` has grouped output by 'CODE_GENDER'. You can override using the `.groups` argument.

AMT CREDIT

• 이번에는 대출 잔액을 확인해본다.

summary(train$AMT_CREDIT)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   45000  270000  513531  599026  808650 4050000

• 데이터의 분포도를 그려보도록 한다.

train %>% 
  ggplot(aes(x = AMT_CREDIT)) + 
  geom_histogram(bins = 30, fill = "#F5896E") + 
  labs(x = "Amount Credit", 
       y = "Count", 
       title = "Distribution of Amount Credit", 
       caption = "Created by McBert") + 
  theme_minimal()

• 이상치를 제거하도록 한다.
  • 상위 2.5%는 제거하도록 한다.

upper_threshold = quantile(train$AMT_CREDIT, 0.975)
cat("Top 2.5% threshold is:", upper_threshold)

## Top 2.5% threshold is: 1574532

train %>% 
  filter(AMT_CREDIT <= upper_threshold) %>% 
  ggplot(aes(x = AMT_CREDIT)) + 
  geom_histogram(bins = 30, fill = "#F5896E") + 
  labs(x = "Amount Credit", 
       y = "Count", 
       title = "Distribution of Amount Credit", 
       caption = "Created by McBert") + 
  theme_minimal()

TARGET ~ Age, Gender, Status

• 이번에는 조건문을 활용하여 데이터를 요약하고, 시각화를 진행해본다.

train$DAYS_BIRTH[1:5]

## [1]  -9461 -16765 -19046 -19005 -19932

train$CODE_GENDER[1:5]

## [1] "M" "F" "M" "F" "M"

table(train$CNT_FAM_MEMBERS)

## 
##      1      2      3      4      5      6      7      8      9     10     11 
##  67847 158357  52601  24697   3478    408     81     20      6      3      1 
##     12     13     14     15     16     20 
##      2      1      2      1      2      2

table(train$NAME_FAMILY_STATUS)

## 
##       Civil marriage              Married            Separated 
##                29775               196432                19770 
## Single / not married              Unknown                Widow 
##                45444                    2                16088

train %>% 
  filter(!is.na(TARGET), CODE_GENDER != "XNA", CNT_FAM_MEMBERS <= 4) %>% # nrow() ~ 303498
  group_by(age = -round(DAYS_BIRTH/365, 0), 
           gender = ifelse(CODE_GENDER == "M", "Male", "Female"), 
           status = ifelse(NAME_FAMILY_STATUS == "Civil marriage", "Married", 
                           ifelse(NAME_FAMILY_STATUS == "Single / not married", "Single", as.character(NAME_FAMILY_STATUS)))) %>% # select(NAME_FAMILY_STATUS)
  summarise(count = n(), 
            AVG_CREDIT = mean(AMT_CREDIT), 
            AVG_TARGET = mean(TARGET)) %>% 
  mutate(AVG_TARGET = pmin(pmax(AVG_TARGET, 0.00), 0.20) * 100) %>% 
  ggplot(aes(x = age, y = count, fill = AVG_TARGET)) + 
    geom_histogram(stat = "identity", width = 1) + 
    facet_grid(gender ~ status) + 
    scale_fill_gradient("Avg. Default Rate %", low = "white", high = "blue") + 
    labs(title = "Default Rate of Applicants (N = 303,498)", 
         subtitle = "Age, Gender, And Marriage Status", 
         caption = "Created by McBert") +   
    theme_minimal()

## `summarise()` has grouped output by 'age', 'gender'. You can override using the `.groups` argument.

## Warning: Ignoring unknown parameters: binwidth, bins, pad

• 상대적으로 Single, Male 그룹이 다른 그룹보다 Default 비율이 높았다.
• 또한, 상대적으로 연령대가 낮을수록 Default 비율이 높았다.

Dept/Incomes Ratio %

• Dept의 비율이 높으면 높으수록 Default 비율도 높을 것으로 예상할 수 있다. 실제로 그런지 확인해본다.

library(scales)

## 
## Attaching package: 'scales'

## The following object is masked from 'package:viridis':
## 
##     viridis_pal

## The following object is masked from 'package:purrr':
## 
##     discard

## The following object is masked from 'package:readr':
## 
##     col_factor

train %>% 
  filter(!is.na(TARGET), CODE_GENDER != "XNA", CNT_FAM_MEMBERS <= 4) %>% 
  mutate(CODE_GENDER = ifelse(CODE_GENDER == "M", "Male", "Female"), 
         DEPT_INCOMES_RATIO = AMT_ANNUITY/AMT_INCOME_TOTAL) %>% 
  select(CODE_GENDER, DEPT_INCOMES_RATIO, TARGET) %>%
  ggplot(aes(x = DEPT_INCOMES_RATIO, y = TARGET)) + 
    geom_smooth(se = FALSE) + 
    scale_x_continuous(name = "Debt / Incomes Ratio (%)", 
                       limits = c(0, 0.5), 
                       labels = percent_format(accuracy = 0.1)) + 
    # coord_cartesian(ylim=c(0.00, 0.15)) + 
    scale_y_continuous(name = "Avg. Default Rate (%)", 
                       breaks = seq(0, 1, 0.01), 
                       labels = percent_format(accuracy = 0.1)) + 
    facet_grid(~ CODE_GENDER) + 
    labs(title = "Default Rate by Dept/Incomes Ratio (N = 303,498)", 
         subtitle = "The Comparison between Female and Male", 
         caption = "Created by McBert") +
    theme_minimal()

## `geom_smooth()` using method = 'gam' and formula 'y ~ s(x, bs = "cs")'

## Warning: Removed 2488 rows containing non-finite values (stat_smooth).

Bureau Data

(과제) 개인별 시각화 인사이트 찾기

• 지금까지의 작성방법을 토대로 다양하게 시각화를 작성해본다.

R 강의 소개

• 필자의 강의: 왕초보 데이터 분석 with R
  • 쿠폰 유효일은 2021년 10월 30일까지입니다.
  • 링크: https://www.udemy.com/course/beginner_with_r/?couponCode=5BF397C9A1E46079627D
  • 현재 강의를 계속 찍고 있고, 가격은 한 Section이 끝날 때마다 조금씩 올릴 예정입니다.

Python 강의 홍보

• 취준생을 위한 강의를 제작하였습니다.
• 본 블로그를 통해서 강의를 수강하신 분은 게시글 제목과 링크를 수강하여 인프런 메시지를 통해 보내주시기를 바랍니다.
  • 스타벅스 아이스 아메리카노를 선물로 보내드리겠습니다.
• [비전공자 대환영] 제로베이스도 쉽게 입문하는 파이썬 데이터 분석 - 캐글입문기