The age of data is upon us – and the scale of it is near-incomprehensible. Each day, we send 370 billion-plus emails and over 100 billion WhatsApp messages, while more than 500 minutes of video is uploaded to YouTube every 60 seconds. And this is just the tip of the iceberg.
All the data created by humanity – predicted to hit 175 zettabytes of data created every year by 2025, according to analysis firm IDC – comes from every part of our lives. For financial institutions, traders and commodities firms, the data explosion has provided another asset they can make decisions based upon, combining historical trading and pricing data with information captured in real-time from the world at large.
Information gathered from numerous tiny sensors can be combined with a myriad of other data to expand our understanding of the world. It’s now possible to know not only who is shipping precious metals, but also where the ship transporting them is and the oncoming weather conditions that will potentially slow the ship down.
But the existence of this data is not enough. Traders and financial analysts need to be able to unlock the intelligence it contains and combine it with other sources of information. They cannot do this by themselves. The volume of data being created each day is too much for humans to understand and analyse. Enter the machines: the only way financial services will be able to make sense of the data that exists is by using analytics and visualisation tools to parse the information. Data can be fed through digital infrastructure and the results rapidly presented to analysts in ways they can understand. Refinitiv runs over 200 models over 100,000 times, with 200,000 messages sent to clients’ databases per day. The movement of goods, for example, can be plotted onto real time maps of the world, and vast databases turned into graphs and charts that are easily understood.
Over time, visualisation and analytics processes will become increasingly automated. Businesses will be able to use artificial intelligence and machine learning to help them find patterns in data and present it to analysts. Those analysts will then be able to use their experience to make intelligence-based decisions using the outputs. But for financial groups to get to this stage, they must first understand the data they have, put processes in place to analyse it, and understand what they can do with visualisation.
Most companies understand the value data can provide – however, they may not be set up to properly take advantage of the data that’s available to them. For many businesses, the Covid-19 pandemic has seen an acceleration in their digitisation efforts, and many have begun to rethink how they operate. “As a first step to integrating their data, companies need to map their systems, data, data sources and workflows across the organisation,” says Colin Hayward, the CEO of contract automation firm Chinsay. He adds that, so far, the adoption of digital technologies has resulted in many companies having disjointed workflows. “Most of them have grown their use of systems without a plan, leading to a proliferation of data and systems.”
Hayward says that companies wanting to make the most of their data need to focus on the workflows they use and how they manage the data. Once they do, they will be able to improve their workflows and better understand what the data is telling them. “With the advent of automation and artificial intelligence and machine learning, it’s more important than ever to get the basic processes correct,” Hayward says.
Once companies have the data basics in place, they can start to analyse the data. “We’re building dashboards to show what’s happening,” says Geoffrey Smith, the director of oil and shipping research at Refinitiv. “Then, we can write our reports around that.” These include dashboards that present datasets in ways that people can understand. Dashboards can pull in data, based on sets of rules, and show the results visually. Tools such as Tableau, a data analytics and visualisation platform owned by Salesforce, or Microsoft’s analytics service Power BI, allow companies to better understand their data.
Smith explains that Refinitiv, through its thousands of sources of data, is able to create analytics, such as a European diesel dashboard. To do this, its systems are integrated with tools including Tableau and Power BI. This shows where diesel is moving now, and makes predictions for future movements, for example, how much diesel is expected to be imported from Russia. Once the dashboard has been set up, it can automatically update and provide Refinitiv’s analysts with up to-the-minute views of diesel movements.
Smith says that Refinitiv’s huge collection of data – its Eikon platform has more than 2,000 pricing data sources and historic data going back 65 years – allows the company to conduct deep analysis on the movements of commodities. “You can start to see trends and themes [in the data],” he explains. The insights analysts would previously need years of experience to spot what can now be done automatically, Smith says, and the analysts are instead able to use their knowledge to uncover new information. “You can see things that maybe you wouldn’t have picked up beforehand,” he says.
These processes are being supercharged by the use of artificial intelligence and machine learning, technologies that analyse patterns in huge datasets and make predictions about what will come next. “Many of the dashboards that companies have relied on in the past have been backwards looking and descriptive, rather than predictive and prescriptive,” explains Francois Zimmermann, the chief technology officer of Tableau EMEA.
“The legacy IT-led approach to analytics is already being replaced with one where IT governs and curates data assets, and enables decision making at the frontlines,” Zimmermann says. For instance, Tableau is developing more natural-language processing technology which can understand written words, and plans to use this to allow people to interrogate its visualisations more easily. By typing a question into its systems – such as “show the average of this dataset from 2019” – it will automatically be able to generate the required results. This makes it easier for staff to interact with the data and, at least for some people, requires less technological understanding to implement. “If you are serious about running a truly data-driven business, then it is better to ensure that everyone from front-line staff to people with advanced business domain expertise are able to ask ad-hoc questions of their data and follow these to their conclusion.”
Open the website for the United Nations International Trade Statistics Database, also known as UN Comtrade, and the wealth of official data about the movement of commodities becomes apparent. Since 1962, 170 countries have contributed more than 40 billion data records of their trading statistics to Comtrade.
Comtrade also provides a glimpse of how huge amounts of global commodities data can be more usefully interpreted through visualisation. Graphs, for example, show countries’ foreign trade levels, national-level maps combine economic and trade data, and there’s a 3D map showing $15 trillion of world trade. The public visualisations of Comtrade data demonstrate what is possible and how visualisation can help to make sense of the world.
“With all visualisations you are essentially telling a story,” Smith says. “You can identify trends and differences.” Refinitiv’s visualisation work has multiple levels and is based on the company’s huge volumes of public and commercially available data. Its data management platform allows businesses to access Refinitiv’s proprietary content, bring in their own data and then innovate with it. The systems include real-time interactive maps which show the position of every ship in the world and specific details about commodities businesses. For instance, if you click on an oil storage facility, you can see the latest data about its supplies and oil flows. Click on an agricultural site, and rainfall data can be used to help predict potential crop yields.
These interactive maps are supplemented with graphs and visualisations that can be generated by those with data science skills. Refinitiv staff are able to use the Python coding language to interrogate data and create new applications. Of particular use, Smith says, are interactive Sankey charts. These diagrams can show the movement and amount of a particular commodity from its source to its destination. An oil producer in West Africa, he says, could use one of the charts to see what grade of oil their competition is buying, where it is going in the world and how they stack up against them. This intelligence would give the oil producer a better understanding of the market and the actions of their competition, helping them to make better-informed business decisions.\
This type of analysis and visualisation, which can be updated automatically when the data is refreshed, will become increasingly important as more companies unlock the potential in their data. Many businesses are likely to use predictive analytics to prepare for what may change in the market – if they do, they may be able to get ahead of competitors and move faster on critical decision making. Sonia Ghosh, a business analyst and digital ambassador at energy company VARO Energy, says the analytics being used ahead of trades is becoming increasingly sophisticated.
“Data analytics capabilities with intuitive visualisation are used to draw meaningful market intelligence from raw feeds to unlock new revenue streams and mitigate risks better,” Ghosh says. She adds that visualisation and analytics are becoming more sophisticated as artificial intelligence and machine learning are being more routinely deployed by businesses. “AI is deployed to analyse signals for algorithmic trading strategies,” she says, adding that machine learning is being used to “validate models” by analysing how predictions varied from what actually happened.
Tableau’s Zimmermann agrees that the emerging technologies are fundamentally changing how companies are doing business. Increasingly, artificial intelligence and machine learning will be used within data analysis and visualisation to help traders and financial institutions respond to changes in their data. “Their AI/ML will notify them of which metrics across all their dashboards are unusual, so that they can jump straight to the most relevant views of their business,” Zimmermann says.
To take advantage of this more automated future, companies need to ensure they have robust data management systems in place and are able to effectively unlock the value that’s contained
in the large datasets they have available to them. That begins with getting the basics right: companies need to understand the data they have, what types of additional information they can access, and how they can then build on top of it.
“Technology has arrived and data has been democratised,” says Alessandro Sanos, global director, sales strategy and execution for commodities at Refinitiv. This means that financial organisations and companies need to properly understand how to make the most of
the data they have in their system. If they don’t, they will fall behind. “The competitive advantage is really evolving from being able to access those additional sources of data, to how well companies can integrate it, commingle it with the data that they produce themselves, and then apply technology to generate insights,” Sanos says.
This is the final ebook in a four-part series on how businesses can prepare for the future of commodities trading, created in partnership with Refinitiv. To explore the others, click on the below links:
1.Making big data work for commodities
To find out more about Refinitiv, click here
To find out more about WIRED Consulting, click here
This article was originally published by WIRED UK
