Why data engineers shouldn’t hide away

We know that data is one of the most valuable assets to most businesses in the digital era. One common problem in data engineering, though, is that it doesn’t create clear pathways to action; you get loads of insights and information that don’t actually show you what changes you need to make in order to produce better results. 

Ian Mahoney (Director of Engineering Platforms at New Look) spoke about this recently on the LEAP:In podcast, from the perspective of retail and sales. He uses his engineering background to help retailers make the best possible use of the data available to them – and he shared some tips that we think will be useful to any business. 

Data engineers shouldn’t hide behind closed doors

The role of an engineering team within a retailer like New Look is to create tech solutions to support the needs of the business – not just to develop cool tech. “Everything we do has to be laser-focused on delivering value for our business and our customers,” Mahoney said – which is why the data engineering platform is such a crucial focus. 

“It’s very easy to end up with data or information overload. What we do is ensure that the analysis and insights provided are very clear and actionable. You can have all the data in the world, but unless you’ve got some actionable insight to make then it’s really just a nice read, but you’re not able to do anything off the back of it.”  

Conventionally, it’s pretty standard for an IT department and data engineering team to be segregated from other teams in a company. To get support or information from the IT team, employees in other teams have to put in requests and wait for those requests to be fulfilled. IT staff are doing their own thing and working to their own objectives without enough integration with the wider business goals (like in the British sitcom The IT Crowd that we love to hate). 

But this means other teams are always working on slightly-too-old information, and IT teams aren’t leveraging the information they’re gathering as effectively as they could. 

How can businesses make the most of data insights?

The most powerful data insights are generated with the specific intention of helping a business take action and make decisions. Netflix uses data to create blockbuster hits – including (but not limited to) device and browser data, watch history, search queries, time spent watching a show, and data on where and for how long a user pauses a show. Uber uses predictive analytics to put the right number of drivers in the right area of a city at the right time, to minimise how long customers have to wait for a ride. And retailers use data to generate customer recommendations and personalise the shopping experience, as well as manage stock efficiency and minimise product waste. But none of this is possible if data engineers aren’t collaborating closely with other areas of the business. 

“Don’t keep all of the data and insight to a single insight team,” Mahoney said. “It’s really important that we democratise the data so that the teams needing to make decisions have access to the data, and don’t need to put in a service request to IT to produce a report for them. Because as we know, a report is out of date as soon as it’s been created or printed.” 

For Mahoney, the ultimate objective is to put data at the heart of all decision-making processes; from the way the brand interacts with customers, to the amount of stock that ends up on the shelves. And to do that, he’s introducing real-time data dashboards. Each dashboard is stacked with live data that answer the needs of different areas of the business, so members of each team can go to the relevant dashboard and drill into it to find up-to-date insights that will help them make good decisions. 

The dashboards act as a bridge between data engineering teams and other teams. With this real-time interface in place, different teams are then able to give feedback to the data engineering team about what data is useful to them, and what isn’t – thus enabling engineers to fine-tune the tech that analyses data over time, to produce increasingly valuable insights. 

Put your data engineers at the heart of your business. Enable them to work across teams and establish a multi-way conversation around the data insights they’re producing. In doing so, they’ll gain knowledge about how they could create more value for the company – and you’ll get far more value from the data available to you.