1.1 The Framework: Market Abuse Detection

Market Abuse Detection (MAD) is the activity of monitoring the data flows of a financial market place (prices, orders and trades) with the aim to find anomalous and suspicious behaviors of market participants. The MAD activity consists of looking for a set of patterns in the data that can suggest that a player tried to manipulate market prices or took advantage in negotiation by exploiting illegal or reserved information.

The anomalous patterns in the data flow are defined by a list of rules settled up by the Regulator in several normative and are organized into six categories (Insider Dealing, Market manipulation, HFT, Cross Product Manipulation, Inter-Trading Venues Manipulation and Bid/Ask Spread).

Each pattern defines an algorithm, i.e. a metric, function of the market dataset (prices, volumes, order frequencies, executed trades, etc…) that has to be monitored and has to be limited to certain specific ranges of values: the overcoming of such thresholds must first be analyzed and, if confirmed, can trigger a reporting to the Regulator. There are at least fifty algorithms, each with its specific thresholds to be set.

The activity of MAD is demanded to the market participants so that each market institutional player has to monitor the data flow and, if the case, report abuses to the Regulator. In this context, institutional players must equip themselves with tools that, implementing the normative patterns, are able to rise alarms in case of threshold overcoming.

Since the quantitative value of a threshold is left to the player, there is always a big effort in fine tuning each algorithm in order to avoid the rising of too many or too few alarms. Because finding the best configuration can be very difficult, the risk is that those who have to monitor the alarms end up losing confidence in the monitoring tool due to presence of too many false positive and false negative.

1.2 Machine Learning Anomaly Detection

Machine Learning (ML) methodologies are therefore a natural direction in which to seek more automated solutions to such a kind of problems: if an unsupervised algorithm was able to autonomously learn the typical behavior of a user, it could be used as an (anomalous) activity monitor, thus saving a lot of time in the management of the alarms raised by the MAD tool and giving focus only to those alarms that are worthy to be analyzed.

The operator can then crosscheck the alarms raised by the normative patterns with those raised by the ML method and analyze the intersection. Moreover the information given by the ML can be used to fine tune the thresholds in order to have a quasi-perfect match of the two subsets of alarms. Consider further that the thresholds of the normative algorithms must be reviewed as time passes because market condition can change over time. The ML methods instead are non-parametric, so in principle they don’t require maintenance over time, giving a static reference for thresholds fine tuning.

The problem of these ML techniques is that they inherently need a very large amount of data to allow the machine to learn users’ patterns, while a financial market place is full of subjects that trade very little, for whom it would be extremely difficult to learn any patterns of behavior.

This is where Recommendation Systems come into play.

1.3 Recommendation Systems as Anomaly Detector

Recommendation Systems (RecSys) are the standard ML methodologies used by content or item retailers like Netflix or Amazon to provide users with personalized recommendation for services or products.

In a nutshell a RecSys is an engine capable of analyzing a dataset of interactions between users and items in order to characterize each user and each item with respect to their mutual interactions, so as to be able to guess what kind of interaction would take place between a user and an item, even if they have never interacted before in the analyzed dataset.

As a key factor, a RecSys is able to profile users and items based on their past interactions not just on a single-user basis, but looking at them as a whole, allowing to infer common patterns from each user even for those who rarely interacts.

The use for which they were designed is to be able to recommend items to a user. But if the RecSys was really able to characterize users and items, we hope to be able to use it in the opposite perspective: to judge, in retrospect, whether a post-training interaction is in line or not with the behavior that the user has maintained in the training interactions.

In the following blog posts we will present the results of our attempt to apply a RecSys to MAD, where the users/items roles are played by the securities and traders, respectively. In this reversed anomaly-detection perspective, deals ill-judged by the RecSys can be considered suspicious at least and worth reporting.

At the time of writing and at our best knowledge, this is the first attempt to use RecSys in an opposite way, not to recommend but instead as an anomaly detector tool.

1.4 RecSys in a nutshell

The Internet has plenty of resources on Recommendation Systems, we don’t want to add here yet another introduction or tutorial.

We will simply provide basic terminology and key concepts required to follow the discussion in the following posts.

The main ingredients of a RecSys are:

• ITEMs, the things to be recommended;
• USERs, who need an item;
• INTERACTIONs, a metric that gives a feedback of the affinity between a user and an item.

In these three abstract concepts it is possible to fit various concrete things in order to set up the context of a specific recommendation system. For instance Netflix items are films or series, the users are people, the interactions are ratings or just ‘watched’ or ‘non watched’. For Amazon, items are products available for sale, the users are people, the interactions are a purchase or just a `search’ of an item.

The numerical methods behind this type of approach are generic enough to be able to examine any dataset, as long as you can find two categorical dimensions to use in the roles of USERs and ITEMs, and a third quantitative metrics to be used in the INTERACTION role, that can be thought as a proxy of some “compatibility score” through which to calibrate the RecSys. After that, the RecSys will try to profile USERs and ITEMs based just on their mutual INTERACTIONs in such a way to: A) gather similar USERs and gather similar ITEMs; B) judge whether a USER and an ITEM fit together or not.

Technically, there are several way to build a recommendation system. If you are interested in further details, please refer to our article, in which we better explain the methods used and the analyses carried out. Here it will suffice to say that we used a collaborative filtering approach with an implicit feedback matrix and a matrix factorization model for the latent factors.

References

Regulation(EU) No. 596/2014 of the European Parliament (MAR)
ESMA 2015/224 Final Report (technical advice on possible delegated acts concerning MAR)
Commission Delegated Regulation (EU) 2016/522 of 17 December 2015