Black box effect in ship risk profiling

This article was co-authored by Dr Harvey Maddocks, Data Scientist, London.

In ancient times, we bowed to oracles that interpreted the rustling of oak-leaves, the flights of birds or the murmuring of the waters from a sacred spring. Nowadays, corporations resort to the latest state-of-the-art techniques of ‘data science’. While the benefits of these techniques are wide ranging, there have been questions raised about some potential implications, such as the ‘black box’ effect, when human intervention is removed entirely. Here we consider possible solutions to the unintended consequences of data science in the maritime industry.

What is data science?

Data science is the extraction of useful knowledge, patterns, or regularities from large bodies of data to build ‘predictive models’.

The collection of methods for extracting predictive models from data is known as ‘machine learning’ or ‘predictive analytics’ (a subfield of artificial intelligence).

A ‘predictive model’ is a mathematical or logical formula for estimating an ‘unknown value’ (the ‘target’ of the prediction).

The target defines what is predicted, that is to say, the kind of behaviour, action or event to predict for an individual.

Data science in shipping

In the shipping field, there can be a number of targets. For example, whether a given vessel is kept in a seaworthy and safe condition, or a sub-standard ship, or the likelihood that of an incident in the next 12 months. These are the type of targets currently used by ship vetting services and Port State Controllers.

In order to predict or estimate the value of such a target it is necessary to find, select and weigh the ‘predictors’ that reduce the uncertainty about the same.

Ideally, only the most informative predictors are applied, but selecting and ranking them is not an easy task for the so called ‘learning algorithms’ (learners). These are the core of the machine learning system, which analyses vast quantities of ever-changing data, spots patterns and draws conclusions.

But those learners not only estimate or predict the value of the intended target but are also engaged in ‘automated data-driven decision-making’ (ADM), which refers to the practice of basing automated decisions on the analysis of empirical data, rather than purely on human intuition.

Potential unintended consequences

Inaccurate automated predictions may lead to denial of services/goods or unjustified discrimination. We fear that underrated vessels may sustain damages and/or losses arising out of incorrect automated assessing of insurance, safety or environmental risks, together with inaccurate setting of insurance premiums; while overrated vessels may be posing increased risks to the shipping industry.

All of the above processes, which result in the ship vetting, profiling, inspection and scoring or rating, are performed by sophisticated ADM systems. At Kennedys, we are taking steps to analyse the safeguards against the risk that a potentially damaging decision was taken without any kind of human intervention.

Comparison with GDPR principles

There is no legal framework addressing the risks of ADM systems in connection with ship vetting, which is not surprising since there is no legal framework concerning ADM systems in general.

However, we can take the General Data Protection Regulation (GDPR) as an external paradigm of reference in this matter, since the GDPR specifically addresses the risks of ADM systems (albeit in relation to personal data).

The GDPR contains a general prohibition on full ADM. Human intervention is the key element. The rationale behind such a prohibition is to ensure that no decision is made merely by technological means without any human intervention. In doing so, the GDPR is fighting the so called ‘black box’ effect (a metaphor used to describe a system in which the input data and the results are known but the process that leads from one to the other is opaque, like a black box).

The general prohibition on full ADM has exceptions under the GDPR. However, in all of those excepted cases the controller may undertake full ADM provided that the following data subject’s rights are safeguarded:

1 The right to obtain substantial human intervention from the data controller

2 The right to challenge the decisions made by ADM.

But how can we challenge what we cannot previously understand? This is the scenario when the third right comes into play:

3 The right to obtain an explanation of the decisions made by ADM.

The so called ‘right to explanation’ is meant to enhance the accountability and transparency of ADM, since the latter relies on arcane algorithmic mechanisms.

Issues to consider

It will indeed not always be easy for the data controller to find simple ways to tell the data subject about the criteria relied on in reaching the decision without explaining the underlying algorithms (which are the result of the work of machine learning techniques that are protected trade secrets). This means that the data subject’s right to be informed about the ADM process can be undermined by the growing complexity and opaque nature of the involved machine learning algorithms (something which goes against the so called “algorithmic accountability”).

Therefore, we consider that the same general GDPR principles could also be analogically applied in relation to ship vetting or risk profiling services.
Ship vetting companies or Port State Controllers should also use appropriate mathematical or statistical procedures for the profiling and take measures to prevent discrimination and erroneous or unjustified ratings (which may lead to shipowners facing more inspections, or paying higher hull and machinery or P&I insurance premiums).

In light of this, the maritime industry should be aware of the importance of having effective safeguards against the “black box” effect in ship risk profiling or ship vetting, with a view to guarantee the “algorithmic accountability”.

Otherwise, many of us, like the prehistoric hominids in Arthur C. Clarke’s Space Odyssey, will still be wandering around these Machine Learning’s black Monoliths without a clue about their inner workings.