Explainability and AI: how can a decision be unpicked at an individual level?

In our article exploring safeguards in AI, we considered the benefits of AI generated decisions being “locally explainable”. This worked example shows how this would operate in practice.

The image below shows an extract of an example of what a locally explainable system would generate in respect of an individual candidate. The algorithm will have taken data from a CV or other input data or will have taken data from an assessment (or both) and identified the factors or features which correlate best to success and the appropriate weighting to be given to each.

In the above example, the algorithm has determined that various features contribute to predicting the success of a candidate for a post. These have been categorised and given a label - shorthand for a more readily understandable feature (e.g “Leadership 1” might, for example, be a label used for years of management experience).

The above extract includes five of the features used by this algorithm (such as CogAb 2 (cognitive ability 2) etc.). Each feature is weighted, and this is represented by the size of the bar in column two (i.e. , CogAb2 counts for 31% of the total score). In reality, there would be many more features used by the algorithm but these five represent the most significant features with the highest weighting, accounting for 58% of the total score.

Each feature also has a “par” threshold score. In the above example, the score for this individual for four of the features is above the threshold (orange). However, for this individual their score for CogAb4 is below the threshold (blue).

This individual’s total score is set out in the first column and the 0.91 pass figure could be equated to a 91% score and could be used as a comparator against other candidates’ score.

For a given individual it is possible to run counterfactuals where data about an individual is changed and any difference in the outcome can be seen. For example, if a candidate considered that a career break in their CV had counted against them, this piece of data could be changed, and the outcome observed.

With local explainability, the basis on which a decision has been made can be explained accurately and in precise detail. For example, an unsuccessful candidate can be told that they scored below par for years of management experience and that with two more years of experience their candidacy would have been successful. Such accuracy and precision are not available with most human made decisions.

System developers apparently express concerns that local explainability could lead to the gaming of a system. However, this concern is overblown as algorithms are continually changing through machine learning particularly if deployed using “single tenant” infrastructure separate from similar systems used by other deployers.

Assessing disparate impact

An outcome with local explainability is that each material feature or factor can be assessed transparently for disparate impact. The table below represents an extract of one outcome where each columns represents a factor taken into account in achieving the outcome (in this case the full table shows over 50 columns/factors and over 8,000 candidates). Assuming that the numbers are statistically significant (and the numbers considered can be in the thousands), it is possible to consider the impact of each factor by gender, ethnicity etc.

Where such is impact is identified, human intervention can assess whether this disparate impact is objectively justifiable and lawful. Of course, gender and ethnicity are not amongst the source data or data inputted in respect of the candidates considered by the algorithm. In this case, this information is only supplied for the disparate impact analysis.