Validating Rating Models

By the end of the chapter, the candidate should be able to give an explanation of the model validation process. The procedures for the roles of internal organizational units in the process of validation should be well described by the learner.

Furthermore, in the validation of internal ratings, qualitative and quantitative processes should be compared, with a description of each process’ elements. Data quality associated problems should be described and steps should be taken to validate the quality of data be explained. Finally, the reader should be able to explain how the calibration is validated and a rating model’s discriminatory power.

Validation Profiles

The scope of validation can be generally agreed to be quite wide since the methods, processes, controls, and collection of data. The IT systems supporting the assessment of the credit risk, internal risk ratings’ assignment, and quantification of default and loss estimates should all be comprised in a rating system.

A robust system should be in place for the accuracy and consistency of banks’ internal models and modeling processes to be validated. These internal validation processes should enable the bank to meaningfully and consistently assess the performance of its internal models and processes.

Critical to the validation of a bank’s entire system of credit risk management is the validation of an internal rating system, and this is from both a regulatory and business management points of view.

Rating systems for banks rely upon the capital adequacy for the adoption of Internal Ratings Based Approaches according to the Basel II regulations. The reliability of results generated by rating systems and the regulatory and operational needs should be verified by the banks in the process of validation on an ongoing basis.

The validation process undergone by rating systems should consist of a set of formal activities, instruments, and procedures to assess the estimates’ accuracy for all components of risks and the overall performance system’s power of prediction.

According to the Basel II regulations a regular cycle of model validation should include:

  1. monitoring the performance of the model and its stability;
  2. reviewing the relationships of the model; and
  3. testing the outputs of the model against its outcomes.

As required by the Basel Committee, the use test should be included in the validation process. This is the critical verification that the rating system is actually applied to adequately in the various areas of operations of banks.

Roles of Internal Validation Units

There is a particular innovation in terms of the organizational necessities and internal controls set by the Basel II regulations. Essential notions and criteria adopted in the creation of their rating systems and organizational and quantitative requirements are clearly laid out by these rules.

The board of director (or a designated committee) must approve all the material aspects of the rating and estimation process. All of the parties involved should possess a general understanding of the risk rating system used, including a detailed understanding of its associated reports on management.

The design and operation of the rating system must be fully understood by the senior management, and material differences between established procedures and actual practice must be approved.

A consideration of the recommendations produced by the process of validation is a necessity for the senior management. This includes a review of the reports produced by the internal audit unit. The process of validation is usually done by a specific unit in the organization which may partially leverage the operational units’ support to achieve this task.

This implies that the unit doing the validation has to be independent of other functions. Moreover, the validation unit should not depend on individuals assigning ratings and lending.

Appropriate human resource skills should be applied and, if this is not possible, the unit may be involved in the process of designing and developing the rating system as long as appropriate organizational and procedural precautions are adopted.

The internal audit function should also take part in the validation process by conducting continued analyses of the compliance in the application of internal rating systems with internal and regulatory requirements.

Qualitative and Quantitative Validation

In qualitative validation, quantitative methods are properly applied and ratings are properly used. On the other hand, all rating validation procedures for computing statistical indicators are comprised in quantitative validation and interpreted on the basis of an empirical dataset.

These two methods complement each other; a positive assessment in the area of qualitative should be received for a rating procedure to be applied. Since statistical estimates are subject to random fluctuations, a negative quantitative assessment should be considered decisive, and there should be an allowance of a certain degree of tolerance when interpreting the results.

Qualitative Validation

Rating Systems Design

These designs are about choosing the proper model’s architecture relative to the market segments the designs will be applied to. The assumptions and evaluations which the rating model’s design are based on should be transparent.

For a specific rating segment, the assessment of the general suitability of a rating approach is necessary. The following are the issues that need investigation:

  1. Whether the development methodologies and processes of the model are consistent;
  2. Whether the model output is adequately calibrated to default probabilities;
  3. Whether all model functions are properly documented;
  4. Whether the rating process is analytically described with the duties and responsibilities of key personnel; and
  5. Are there robust procedures to regularly validate and review the model?

Crucial organizational profiles for qualitative validation of the rating systems are about the connection between the model, process, approval powers, controls and procedures. In qualitative validation, assessing the actual application of rating systems in the process of approving credit is critical.

Model building, number crunching, statistical testing, and many more are the greatest time-consumers in the early stages of developing rating systems by credit risk functions. There is often an underestimation of procedural aspects in terms of the necessary time, resources, and investments because they are wrongly considered less challenging.

The following are the five main features that summarize the essential requirements of the rating systems needing to be checked in a qualitative validation:

  1. Obtaining default probabilities: Nearly all risk management applications are based on ratings immediately once default likelihoods are obtained.
  2. Rating system completeness: All available information should be taken into account when ratings are assigned to borrowers or transactions.
  3. Objectivity of the rating system: Creditworthiness factors should be clearly captured by the procedures of a good rating system, and the room for interpretation has to be minimized. Objective assigning is necessary for minimizing biases.
  4. Rating system acceptance: The following are the requirements for the rating system to be accepted:
    1. Classifications produced by the rating system should not be far from the ones expected by the bank’s analysts and officers.
    2. Compared to a poorly structured judgment-based approach that has been developed for credit officers that are not adequately trained or experienced, the discriminatory power of mechanical rating models is often higher.

    The rating models should be well understood and shared by the users and verified by the process of validation. The degrees of acceptability usually differ with the different rating models. Heuristic models are generally accepted and their designs are based on the experts’ lending experience, so the end users consider their credit assessment to be warmer. For fuzzy logistic systems, their acceptance may be lower because a greater degree of technical knowledge is required, attributed to their fuzzy algorithms and changing variables’ weights in different contexts.

  5. Rating system consistency: There should be suitability and coherence in the models. If statistical models and other mechanical methods are applied to assign borrowers ratings or in the approximation of PDs, LGDs, or EADs, the validation requirements are specifically stated in the Basel II regulations.

Data Quality

An essential requirement for a quantitative validation is a dataset that is comprehensive. Therefore, the following qualitative aspects should be considered:

  1. Completeness of the data;
  2. Volume of available data;
  3. Representativeness of applied samples in the validation and development of the model;
  4. The data sources’ consistency and integrity; and
  5. Adequacy of used procedures to ensure data is cleansed and the general quality of the data.

The reliability and completeness of defaulted observations are the actual limits to adequately developing large sets of data when developing the model, quantifying the rating, and validating the said model.

The timespan is also important to the quality of the data. An entire credit cycle should be considered for the dataset to be generated, or else specific favorable and unfavorable cycle stages will be depended on by the estimates.

The preliminary activities of treating data should be paid attention to by the validation process. Since the quality of the data is very relevant, a specific attention should be dedicated to many aspects.

In assessing the stability of the lending technology behind the data, and for proper calibration of the model for results to be generalized from a sample to the population, the role played by the validation unit is essential.

The validation unit should not be ignorant of the consequences of the lending technology’s changes and the misalignments between the profiles of the borrowers in the original sample and the profile of the population.

Finally, the validation unit is key in the verification of the central tendency over time by back testing and stress testing the model.

Quantitative Validation

The following are the four areas covered by the quantitative validation:

  1. The reference population’s sample representativeness at the estimates’ time in subsequent periods;
  2. Discriminatory power: Rating assignments accuracy in terms of the ability to rank borrowers both in the overall samples and in its different breakdowns, by risk levels for the model;
  3. Dynamic properties: The rating systems’ stability and migration matrices’ properties;
  4. Calibration: The power of prediction of the default likelihoods.

Exposures should not be excluded from the rating model’s scope of application due to insufficient data for the validation of the risk parameter estimates on the basis of statistics. Furthermore, the analysis techniques in this process of estimation and their limitations should be keenly watched.

The fundamental ability of a rating model to differentiate performing and defaulting borrowers over the forecasting horizon is referred to as the discriminatory power. In order to validate the discriminatory power, longer forecasting horizons should be applied.

Ex post reviewing of a model’s discriminatory power can only happen by applying the data on defaulted and non-defaulted cases. Using a longer time horizon implies that the observation period used should be more distant from time zero and from the collection time of data feeding the model’s explanatory variables.

Various analyses of the rating discriminatory power are possible, based on the resulting sample. The following is the list of methods from the Basel Committee:

  1. Statistical tests like Fisher’s \({ r }^{ 2 }\), Wilks’ \(\lambda\) and Hosmer-Lemeshow;
  2. Migration matrices;
  3. Accuracy indices, e.g., Lorentz’s concentration curves and Gini ratios;
  4. Classification tests.

An observation of the migration matrices is crucial for the assessment of the stability of ratings. If the rating systems operate for at least two years, they can be built. The following are the desirable properties of annual migration matrices:

  1. As rating classes worsen, there should be an ascending order by transition rates to default;
  2. The stability of the ratings, over time, signaled by high values being on the diagonal and low values off-diagonal; and
  3. When departing from the diagonal, off-diagonal values should be in descending order.

Although a natural reduction in the diagonal values and an increase in off-diagonal values are witnessed, the above properties should hold for time horizons longer than a year, hence the ratings will change over time but not with large lapses.

Since ratings are less sensitive to credit cycles and transitory circumstances, their change is quite slow over time since analyses of companies’ fundamentals dominate the rating assignments.

Calibration is a crucial issue due to the scarcity of available statistical tools. Calibration validation methods are at a much earlier stage, in comparison to the evaluation of the discriminatory powers.

An analysis of the difference between forecasted PDs and default rates realized is validating the calibration process. As indicated by the Basel Committee, calibration is assessed by a few tests, namely: binomial tests, Chi-square tests, normal tests, and traffic lights approaches. Each of these tests bears crucial limitations and, therefore, no powerful tests of adequate calibration are currently available.

Back Testing, Benchmarking, and Stress Testing

These are the three activities for validating rating systems:

  1. Back testing: There must be a regular comparison of the realized default rates against the estimated PD for each rating grade. The deviation reasons should be analyzed by the validation unit in the event that they fail to fall within the expected range.
  2. Benchmarking: Procedures to specify acceptable deviations between internal estimates and benchmark data should be established by the validation unit. When the acceptable limits are surpassed by these deviations, the validation unit should identify the actions to be taken, at least in general terms.
  3. Stress testing: The reliability and robustness of the results should be assessed by the validation unit when their independent variables are set to indicate extreme conditions.

These three activities should be reported to the top management in a way that is transparent and easy to comprehend. This way, the internal communication strategy of the validation unit will be enhanced as clearer communication implies that the top management’s contribution will be more effective.

Practice Questions

1) In the qualitative validation process, a rating approach’s general suitability for specific rating segments needs to be assessed. The following are areas that should be investigated. Which of the areas given below is NOT among the ones that should be necessarily addressed?

  1. The model development processes and methodologies’ consistencies
  2. Adequate calibration of the model output to default probabilities
  3. Sample representativeness of the population referred to at the estimation time and in subsequent periods
  4. The rating process’ analytical description, with a description of the key personnel’s duties and responsibilities

The correct answer is C.

In the qualitative validation process, it is a must for options \(A\), \(B\), and \(D\) to be investigated as far as the rating systems design is concerned. However, it is not necessary to investigate the sample representativeness of the population referred to at the time of estimation and in subsequent periods.


Leave a Comment

X