Market Abuse Regulations (MAR) took effect on July 3 2016 across the EU. In order to align with these new set of rules, financial institutions operating in financial markets (stocks, FX, etc.) are obliged to surveil their own activities against market abuse.
Although types of market abuse may vary by the market and institution, some basic scenarios include how insider information can lead to abusive activities such as spoofing, front-running, and the manipulation of fixes. As the project I was previously engaged with was centred around the Foreign Exchange (FX) market, I will predominantly discuss such activities in relation to FX.
There are basically two types of data - transaction, and order. Order data refers to the buy/sell requests made by clients whereas transaction data maps to the completed orders. In other words, order data may not be completed (may be cancelled, expired or timed out).
According to MAR, institutions must leverage both of these data sources to prevent and/or detect abusive activities. For example, an order may be given by phone. In order to cover that case, phone surveillance must be put in place. Although we will discuss bits and bytes types of data here, that is a good example to understand the scope of surveillance requirements. MAR is very broad.
Spoofing is a well-known method to abuse markets. It is essentially a way of creating an artificial fluctuation on the price of traded goods (the price of a currency pair in this context) so that the spoofer can make money within a very short time.
In order to detect spoofing you will need order data. Spoofers may put tens, hundreds or (with the help of high frequency trading) thousands of orders to buy/sell a currency pair (i.e. EUR-USD) within minutes to fake a demand. These orders will need to be cancelled very shortly after they put; otherwise the spoofer will end up with a lot of currency that they do not actually want.
Below chart introduces a new term - flash crash. An extreme devaluation of a market (usually because of algorithmic high frequency trading) is called a flash crash. Below you can see the most recent flash crash of the FX markets, the 6-7 October 2016 GBP/USD flash crash. The reason we are looking at this chart is that, although not every flash crash is necessarily considered as spoofing (there is still a lot of debate around it), the impact is identical. A successful spoofing act would exactly look like this (yet the cycle timeframe would be a bit wider):
In order to detect potential spoofing, one needs to detect/place alerts on orders that are put and cancelled for a specific currency pair and direction within a very short time (minutes).
Front-running is another widely used method for market abuse purposes. It requires insider information. A trader can act swiftly upon a very restricted piece of information (i.e. the devaluation of a currency) or a very large order which can move the market. In other words, the trader front runs the market to make profits.
Front running can potentially be detected using both order and transaction data. First, a very large order can be detected in the order data and then the transaction data can be searched to flag any significant transactions in the same currency pair.
The definitions of a large order and a significant order may vary, but a reasonable approach can be made using a logarithmic distribution for each currency market and standard deviations in these distributions (i.e. orders larger than 2 standard deviations in a currency market being a large order). Another method can be using quantiles.
The FX market usually chooses a reference point for the price of a currency pair to value assets and to make certain transactions. London 4pm is a widely used reference in Europe and it is widely referred as "the fix". On the other hand, "the fix" is not the only one. There are many fixes, such as Tokyo 3pm, New York 5pm, and so on.
Manipulation of fixes is essentially another type of front-running. It again requires a restricted knowledge for abuse. A trader or institution may want to take advantage of a market-moving order which is due to be transacted at 4 pm London time. So what the abuser may try to do would be to sell or buy before the fix time and buy/sell immediately after fix time to make quick profits. For more info see here and here.
The following chart is borrowed from the second link above:
The algorithms needed to deal with manipulation of fixes are very similar to front-running, whereas to combat it you actually know the time you need to focus on (i.e. London 4pm). First, identify a large order (again using statistical methods), and then search the transaction data within a specified time window for possible manipulation.
Sometimes it is possible to observe off market value trades in data. It is important to flag these for further investigation. For that type of query, in addition to transaction data, max and min market prices for each currency pair are needed from a trustworthy reference (i.e. Reuters). And then you simply need to check if there are any trades above or below these reference prices. It may be a good idea to implement a tolerance (i.e. 1-2% below or above), to not be overly strict.
There is also switching and washing of trades. Switching is an attempt to prolong a currency position due to longer term position seeming more profitable. Washing happens when two institutions are not able to make the trade directly (due to amount limits etc.) so that an intermediary institution comes into play in order to complete the intended trade indirectly.
The requirements are essentially to detect these abusive scenarios and report them. There may be several technologies to make use of to achieve this, but the most important thing is to first have the proper data.
Once we have the data (transaction and order), now we can start thinking about technologies to implement. There are some specific market surveillance softwares avaliable (Actico, Ancoa, etc.), but usually these take a long time and are expensive to implement because they require a lot of adaptation, due to each financial institution having very different data patterns. It is still a good choice as a long term solution though. One of the main advantages of such software is the built-in reporting option.
Other approach (tactically or strategically) may be using what most companies would do as of today: relational databases and SQL. SQL is a solid, proven method of dealing with data and companies have been using this technology for decades. It is quite possible that they retain in-house skills for it. An existing data warehouse or data mart can be leveraged or a new data mart can be created just for surveillance purposes. Obviously, this is a mixed approach as we still need a platform to report on results of the queries. Theoretically this can be any BI platform ranging from modern ones such as Tableau and QlikView/QlikSense to more traditional ones such as Business Objects, MicroStrategy, IBM Cognos, etc.
Speaking of QlikView/QlikSense, it is quite possible to build these surveillance queries and at the same time report on them within the platform. With that approach, all you need is one product instead of two or more. If this is meant to be a tactical solution (MAR is already effective so perhaps you need a quick result), this would be my personal choice simply because Qlik tools allow very agile development in addition to being very capable of data manipulation/integration. Needless to say Qlik is a leading technology vendor in data visualisation, so reporting is not a problem at all.
Qlik can very well be a strategic solution for surveillance purposes as well. The only caveat is that it may be argued that all data should be stored in a RDMS environment instead of Qlik's qvd or txt data files. If that is the preference, then Qlik can very well be part of a mixed solution.
Similar to Qlik, Tableau is another good option for data visualisation/reporting. The main difference is that Tableau needs an accompanying solution to build queries, which could be either Alteryx or simply SQL Server. Considering Tableau's solid integration with Alteryx, Alteryx + Tableau can make an unbeatable combination.