Validating Rating Models

After completing this reading, you should be able to:

• Explain the process of model validation and describe the best practices for the roles of internal organizational units in the validation process.
• Compare qualitative and quantitative processes to validate internal ratings and describe elements of each process.
• Describe challenges related to data quality and explain steps that can be taken to validate a model’s data quality.
• Explain how to validate the calibration and the discriminatory power of a rating model.

Definitions

Model Validation

Model validation is defined as the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives, and business uses.

Rating Model

According to the Basel Committee (2004), a rating model “comprises all of the methods, processes, controls, and data collection and IT systems that support the assessment of credit risk, the assignment of internal risk ratings, and the quantification of default and loss estimates.”

The Process of Model Validation

During the validation process, a bank has to verify whether the results generated by the rating system are reliable, taking the existing regulatory requirements and operational needs into account. A bank also periodically reviews validation instruments and methods to ensure that they remain appropriate because market variables and operating conditions are always evolving. According to the ‘proportionality principle’, the scope and depth of quantitative and qualitative validation should be commensurate with the overall complexity of a bank, the type of credit portfolios examined, and the level of market volatility.

Rating systems must undergo a validation process consisting of a set of formal activities, instruments, and procedures for assessing the accuracy of the estimates of all material risk components and the predictive power of the overall performance system.

According to Basel II regulation, “the institution shall have a regular cycle of model validation that includes monitoring of model performance and stability, review of model relationships, and testing of model outputs against outcomes.” (Basel Committee, 2004, §417).

The validation process has both quantitative and qualitative elements.

• As a quantitative process, it lies in the statistical comparison of actual risk measures against the ex-ante estimates, checking of parameter calibrations, and benchmarking and stress tests.
• As a qualitative process, it involves analyses of all the components of the internal rating system. That includes operational processes, controls, documentation, and IT infrastructure.

Therefore, validation also requires the assessment of the model development process, with particular reference to the underlying logical structure and the methodological criteria supporting the risk parameter estimates.

The other important part of the validation involves verifying that the rating system is actually used (and how it is used) in the various areas of bank operations. This is known as the ‘use test’, also required by Basel II. The results of the validation process need to be adequately documented and periodically submitted to the internal control functions and the governing bodies. The reports shall specifically address any problem areas.

Best Practices

Basel II rules serve two main purposes:

1. They lay down essential notions and criteria that banks must adopt in developing their rating systems.
2. They set down the organizational and quantitative requirements banks must comply with for recognition of their methods for capital adequacy purposes.

Specific requirements are set for senior management and those who have roles in corporate governance and oversight:

‘All material aspects of the rating and estimation processes must be approved by the bank’s board of directors or a designated committee thereof and senior management. These parties must possess a general understanding of the bank’s risk rating system and detailed comprehension of its associated management reports. Senior management must provide notice to the board of directors or a designated committee thereof of material changes or exceptions from established policies that will materially impact the operations of the bank’s rating system’

(Basel Committee, 2004, §438)

‘Senior management also must have a good understanding of the rating system’s design and operation and must approve material differences between established procedure and actual practice. Management must also ensure, on an ongoing basis, that the rating system is operating properly. Management and staff in the credit control function must meet regularly to discuss the performance of the rating process, areas needing improvement, and the status of efforts to improve previously identified deficiencies.’

(Basel Committee, 2004, §439)

In line with these requirements, there are certain best practices that can be adopted by internal organizational units:

1. Senior management must consider recommendations produced by the validation process and review reports produced by the internal audit unit.
2. The validation process should be performed by a specific organizational unit that may partially leverage the support of operational units in performing its activities. In smaller banks, a manager should be appointed to coordinate and oversee these activities
3. The validation unit should maintain independence from functions devoted to developing and maintaining model tools
4. The validation unit should also be independent of those involved in assigning ratings and lending. Specifically, persons in charge of the function should not be subordinate to persons responsible for such activities.
5. Specific attention has to be paid to ensure the appropriate skills of human resources are employed.
6. The design and operation of the rating system must be fully understood by the senior management, and material differences between established procedures and actual practice must be approved.
7. The internal audit function should also take part in the validation process by conducting continued analyses of compliance in the application of internal rating systems with internal and regulatory requirements.

$$ \textbf{A summary of the Validation and Control Processes} $$

$$\small{\begin{array}{l|l|l|l|l} {} & \textbf{models} & \textbf{procedures} & \textbf{tools} & {\textbf{Management } \\ \textbf{decision}} \\ \hline { \text{Basic } \\ \text{controls} } & { \textbf{Task} \text{; model} \\ \text{ developing and} \\ \text{back testing} \\ \textbf{Owner} \text{; credit risk} \\ \text{ models development} \\ \text{unit} } & { {\textbf{Task; } \text{credit }\\ \text{ risk procedures} \\ \text{maintenance} } \\ {\textbf{Owner; } \text{lending } \\ \text{units/internal } \\ \text{control units}}} & {{\textbf{Task;} \\ \text{operations} \\ \text{maintenance} } \\ {\textbf{Owner; } \\ \text{lending } \\ \text{units/IT/internal} \\ \text{audit} }} & { { \textbf{Task;} \\ \text{lending} \\ \text{policy} \\ \text{applications} } \\ { \textbf{Owner;} \\ \text{central} \\ \text{and} \\ \text{decentralized} \\ \text{units/internal} \\ \text{control} \\ \text{units} } } \\ \hline {\text{Second } \\ \text{controls} \\ \text{ layer} } & { { \textbf{Task; } \\ \text{continuous} \\ \text{ test of} \\ \text{ models/processes/tools} \\ \text{performance} } \\ { \textbf{Owner;} \text{ lending} \\ \text{unit/internal audit} } } & {{ \textbf{Task;} \\ \text{lending} \\ \text{policy} \\ \text{suitability} } \\ { \textbf{Owner;} \\ \text{validation} \\ \text{unit/internal unit} }} & {} & {} \\ \hline { \text{Third} \\ \text{control layer} } & {} & \text{Risk management/CRO} & \text{Organisation/ COO} & { \text{Lending} \\ \text{unit/CLO/COO} } \\ \hline { \text{Accountability} \\ \text{for supervisory} \\ \text{purposes}} & { \text{Top } \\ \text{management/surveillance} \\ \text{board/board} \\ \text{of directors} } & {} & {} & {} \end{array}}$$

Qualitative and Quantitative Validation

They are the two main areas of validation that basically complement each other: quantitative validation and qualitative validation.

Elements of Qualitative Validation

Rating System Design

The assumptions and evaluations that form the basis of the rating design must be transparent and the rating approach assessed.

The areas listed below have to be investigated:

• Consistency of model development process and methodologies.
• Adequate calibration of model output to default probabilities.
• Proper documentation.
• Analytical description of the rating process.
• Existence of robust procedures for validation and general review.

The main requirements in qualitative validation are:

• Obtaining probabilities to default: Statistical models are developed on the basis of an empirical dataset. This makes it possible to determine the PD for individual rating classes by calibrating results with empirical data.
• Completeness: To ensure completeness, banks need to take all the available information into consideration when assigning ratings to borrowers or transactions. Why is it important to assess completeness? Many default risk models are structured to make use of a small number of borrower characteristics to infer creditworthiness. Since statistical-based models allow for many borrower characteristics to be used, the process of adding variables to a model needs to be validated to have greater coverage of appropriate risk factors.
• Acceptance: Rating systems have to be accepted by users, both internal and external. These include credit analysts, credit officers, and loan officers. In this regard, the rating system should not produce classifications that are very often too far from those expected by bank analysts and officers. Mechanical rating models have been proven to have higher discriminatory power than a poorly structured judgment-based approach developed by credit officers with little or no experience. A certain level of familiarization and education is, therefore, needed so that all users involved do not end up rejecting a model on the basis that they do not understand its working or output.
• Consistency: Models have to be coherent and suitable for the borrowers to which they are applied. What does this mean? When developing a statistical rating model, there may be relationships between indicators that contradict the economic theory. Such contradictory indicators need to be filtered to ensure consistency.
• Objectivity: A good rating system needs procedures that ensure that all factors that affect creditworthiness are captured clearly. The rating system needs to minimize room for interpretation. In order to achieve high discriminatory power, ratings must be assigned as objectively as possible to minimize biases.

Elements of Qualitative Validation

There are four main elements of qualitative validation:

1. Sample representativeness.
2. Discriminatory power.
3. Dynamic properties.
4. Calibration.

Sample Representativeness

A sample is said to be representative of a population when its characteristics match those of the population. Some loan portfolios (in certain industries or business sectors) are characterized by very few defaults. Since risks of these ‘low-default portfolios’ have to be assessed in any case, it is important to develop and validate rating systems

Discriminatory Power

Discriminatory Power is the ability of a rating model to discriminate between defaulting and non-defaulting borrowers over the forecasting horizon.

Dynamic Properties

A rating model’s dynamic properties usually refer to its rating stability and migration matrices. In practice, rating stability can be assessed by observing migration matrices, which can be built once a rating system has been operational for, at least, two years.

What are some of the desirable properties of annual migration matrices?

• Transition rates to default should be in ascending order as rating classes worsen.
• High values should be on the diagonal and low values off-diagonal, something that indicates that ratings are stable over time.
• Rating movements are gradual whereas sudden leaps of many classes at one time are not that frequent. For example, migration rates of plus or minus one class should be higher than migration rates of plus or minus two classes, and so forth.

$$ \textbf{Example of an Annual Migration Matrice Consistent With the Desirable Properties Listed Above} $$

$$\begin{array}{l|cccccccc}
& & & &\textbf{Rating at year end} & & & & \\ \hline
\textbf{Initial Rating} & \textbf{AAA} & \textbf{AA} & \textbf{A} & \textbf{BBB} & \textbf{BB} & \textbf{B} & \textbf{CCC} & \textbf{Default} \\
\textbf{AAA} & 90.81\% & 8.33\% & 0.68\% & 0.06\% & 0.12\% & 0.00\% & 0.00\% & 0.00\% \\
\textbf{AA} & 0.70\% & 90.65\% & 7.79\% & 0.64\% & 0.06\% & 0.14\% & 0.02\% & 0.00\% \\
\textbf{A} & 0.09\% & 2.27\% & 91.05\% & 5.52\% & 0.74\% & 0.26\% & 0.01\% & 0.06\% \\
\textbf{BBB} & 0.02\% & 0.33\% & 5.65\% & 86.93\% & 5.30\% & 1.17\% & 0.12\% & 0.18\% \\
\textbf{BB} & 0.03\% & 0.14\% & 0.67\% & 7.73\% & 80.53\% & 8.84\% & 1.00\% & 1.06\% \\
\textbf{B} & 0.00\% & 0.11\% & 0.24\% & 0.43\% & 6.48\% & 83.46\% & 4.07\% & 5.20\% \\
\textbf{CCC} & 0.22\% & 0.00\% & 0.22\% & 1.30\% & 2.38\% & 11.24\% & 64.86\% & 19.795
\end{array}$$

If the migration matrix is stable, the indication is that the model is mainly centered on the counterparty’s fundamentals (less sensitive to credit cycles and to transitory circumstances). Therefore, the rating system is considered to be forward-looking.

Calibration

Validating calibration is the process of analyzing differences between forecasted probabilities of default (PDs) and realized default rates. Basel guidelines allow financial institutions to validate calibration using a number of tests, including the binomial test, Chi-square test (or Hosmer-Lemeshow), normal test, and traffic lights approach.

Data Quality

A comprehensive dataset is an essential prerequisite for quantitative validation. Good data give outstanding model results, but poor data quality can be problematic, particularly when dealing with advanced models.

In this context, a number of qualitative aspects have to be considered:

• Consistency and integrity of the sources of data.
• The volume of data available.
• Completeness.
• Representativeness of samples used for model development.
• Adequacy of procedures used to ensure data cleansing and, in general, data quality.

Validating a Model’s Data Quality

To validate a model’s data quality, the validation unit has to focus on two critical aspects:

1. Stability of the lending technology behind data.
2. Proper model calibration in order to generalize results from sample to population.

Lending technology can be defined as the rules and regulations used in credit origination and monitoring. It encompasses all the tools used to make lending decisions. In modern banking, banks have developed algorithms and other IT tools that help analyze borrower information fast and can even be programmed to reach a decision by themselves. Validation would be less of a problem if lending technology remained stable for a prolonged period, but in practice, this is almost impossible. Lending technologies rarely remain stable for more than a few years. The world of tech changes constantly, and so do the tools used in credit analysis.

If a model is not re-calibrated after a change in lending technology, the model continues to apply old criteria to new states of business.

If the observed in-sample default rate diverges from the total population, then calibration should reflect this divergence because the sample’s central tendency would be different from the population’s central tendency.

How to Validate the Calibration and the Discriminatory Power of a Rating Model

Validating Calibration

As mentioned earlier, validating calibration is the process of analyzing differences between forecasted PDs and realized default rates.

Basel guidelines allow financial institutions to validate calibration using a number of tests, including:

• The binomial test applied to one rating category at a time.
• The Chi-square test (or Hosmer-Lemeshow), which simultaneously checks several rating categories.
• The normal test, which is applied to a single rating class but is a multi-period test of correctness.
• Default probability forecasts.
• The traffic lights approach – a multi-period backtesting tool for a single rating category. It came with the 1996 Market Risk Amendment as a supervisory evaluation tool of internal market risk models.

Validating the Discriminatory Power

The discriminatory power of a model is assessed ex-post using data on defaulted and non-defaulted cases (backtesting).

Validating discriminatory power can be done using any of the following methods as outlined by the Basel Committee (2005a):

• Statistical tests (e.g., Fisher’s \( { \text{r} }^{2} \), Wilks’ \( { \lambda } \), and Hosmer-Lemeshow).
• Migration matrices.
• Accuracy indices (e.g., Lorentz’s concentration curves and Gini ratios).
• Classification tests (e.g., binomial test, Type I and II errors, chi-squared test, and normality test).

Practice Question

In the qualitative validation process, a rating approach’s general suitability for specific rating segments needs to be assessed. The following are areas that should be investigated. Which of the areas given below is NOT among the ones that should necessarily be addressed?

A. The model development processes and methodologies’ consistencies.

B. Adequate calibration of the model output to default probabilities.

C. Sample representativeness of the population referred to at the estimation time and in subsequent periods.

D. The rating process’ analytical description, with a description of the key personnel’s duties and responsibilities.

The correct answer is C.

In the qualitative validation process, it is a must for options \(A\), \(B\), and \(D\) to be investigated as far as the rating systems design is concerned. However, it is not necessary to investigate the sample representativeness of the population referred to at the time of estimation and in subsequent periods.