Find Data “Gold” With Process Mining

Blog: Find Data "Gold" With Process Mining -

When thinking about mining, most people think about gold. Miners dig and dig to reach that rich vein or strain through endless pans of water to collect those glistening flecks. The idea is to keep mining until you literally hit pay dirt. That same analogy applies to “Process Mining,” searching through your project until you hit gold.

In this case, the “gold” is discovering the root causes behind our process’s poor performance.

In the last few years, an entire industry has taken off in the digital space around “Process Mining.” It’s a cross between traditional process mapping and the latest big data techniques to “mine” gigabytes of key process data.

When working on a process improvement project, one of the most important, and often time consuming activities, is accurately capturing the process so you can see what’s really happening. This can involve endless meetings with business leaders, subject matter experts, and customers of the process—all with the sole focus of accurately documenting what happens from beginning to end, including measures of cycle time, rework, and cost. These last three often take precious time to collect, analyze, and then populate into dashboards.

Now, imagine you could capture all that information digitally as the process is unfolding in real time. We’re talking about creating a “digital twin” of the process. You may have recently seen General Electric’s “digital twin” version of a jet engine. They have the ability to digitally recreate, using real-time data, all the activities going on inside a complex piece of machinery with the goal of perfecting its operation. That’s what we’re talking about here. But instead of a jet engine, we’re focused on a process.

Process Mining flow from Minit Software

The Importance of Process

As any good student of Continuous Improvement will tell you, a successful Lean Six Sigma project begins with understanding the process. Diagramming the process gives us the ability to see what’s truly happening, step by step. But getting the process down on paper can take time, as mentioned earlier. Process mining, when done right, allows us to open up a web page and see our process rendered in seconds. Plus, because it is software-based, we can see it on any browser, laptop or desktop.

Once you have the process flow on screen, you can break out your process analysis tools and look for bottlenecks, rework loops, cycle-time sinks, and other process symptoms.

Another advantage of process mining is an “unfiltered” view of your operation. By seeing without the bias or lack of information that’s often introduced during stakeholder or subject-matter walkthroughs, you’ll see activities at the system level. This unfiltered data arms you and your business process team with the ability to find opportunities for:

  • Process cost savings
  • Faster throughput times
  • Increased efficiencies

Variation in the Mine

In Lean Six Sigma, you learn that “variation” is the enemy. One process-mining provider, Celonis, has created a process variant tool. It provides a Histogram highlighting the number of variants or unique paths an order or case can take through your process. Many teams are surprised to learn how many variants a single process can have. Thinking the process works a certain way, doesn’t mean that it does. Having a large number of variants means you’ve got a lot of opportunity for improvement.

So, How Do You Process Mine?

Under the hood, process mining is simple. Rather than a pick and shovel, you mine the transactional data from the server log files and load them into the process-mining platform. Let’s take a basic “Quote-to-Cash” process as an example. Each process step is supported by some kind of application which, in turn, writes tons of data to server log files.

Log files are everywhere now due to the number of systems supporting automated work. The example in the graphic below is of a common “Quote-to-Cash” process. The top layer contains the process steps the next layer the supporting applications, and on the bottom, we see the databases containing the logs and other digital artifacts. The process-mining platform takes that information and turns it into informative process flows, visualizations, and dashboards.

Process Mining Flow

Process Mining Flow -

The Event Log

The secret to finding the “vein” of gold deep in the mine is the event log. When teaching Lean Six Sigma, I’ll often coach participants to “become the thing going through the process.” Event logs help us do that. Event logs, at a minimum, contain three types of data.

  • Case ID – The order number or other unique number that “threads” all related activities for that order, purchase order, or invoice, etc., together. Think of it as that one number that would help you find the artifact if it ever got lost in the non-virtual world.
  • Activity Name – A label, name, or word used to describe the event or what is happening to the “case” at a particular point in time. Like a good SIPOC (aka high-level map) or process flow, this should follow the verb-noun format.
  • Time Date Stamp – The date and time when the activity happens.

The Gold Divining Rod

Now, it’s time to put away the gold-seeking divining rod. When you “triangulate” these three types of data, you almost magically begin to see the process. You can even play it back – like a recording. You could probably do the basics in a simple Excel spreadsheet, but you wouldn’t get the rich process flows produced by the process-mining platform. Additionally, due to some of the “low-code” features of the software, anyone with basic Excel or SQL skills can easily build metric performance dashboards. Cycle time, rework counts, and order volumes are popular ones to start.

Getting the Data

Event logs can be downloaded in CSV, Excel, TXT and other formats. You can also devise ways to directly read event-log data live from your production systems. This carries the best promise for capturing real-time data. However, it also carries risk by placing additional burdens on your production servers. Some companies use a data warehouse strategy to get around this. In short, they constantly make a copy of all their data and place it in a widely accessible data warehouse. This allows problem-solving teams to use the “warehouse” data without placing risk on the live production systems.

Attribute Data and Filtering

Now that you have your process map and are collecting data, it makes great sense to capture other characteristics about the order, case, or invoice going through the ordering system. This is where doing a good job on your project’s data stratification can go a long way. For example, you want to compare your process for customers who order less than 10 items at a time with those who order more than 10. A good attribute to capture with your data in this case is order quantity. Let’s say you would also like to compare the cycle times of orders from two different divisions. It would make sense to include a code for the two divisions. In the event log example above, the additional attributes are resource and cost. Including this data allows you to compare performance between resources, focus in on costs, identify golden best practices and some that are, well, lumps of coal.

Like traditional process mapping, you can have an “As-Is” or current state model of the virtual process when doing process mining. This is sometimes referred to as the “read-only” version. But you can also use it to build the foundation for virtual “To-Be” or future state process. Plus, when you implement your process changes, it’s possible to build a digital twin to reflect real-time data. Simply put, you can see how your new process is working as it is happening. You don’t have to wait until the end of the week, month or quarter to collect and package the data. Think of how this can help during the Improve phase of DMAIC.

What are the Benefits?

Process mining tools hold great promise for accelerating process analysis because many systems capture digital fingerprints. With the ability of process mining platforms to read this data, it means any process that’s digitized can be analyzed more quickly and robustly than ever before.
According to a recent Gartner white paper, process mining has been applied to the following processes at large, enterprise-level companies around the globe.

  • Lead-to-Order
  • Order-to-Cash
  • Procure-to-Pay
  • Accounts Payable
  • Accounts Receivable
  • Supply Chain
  • Manufacturing
  • Warehouse Operations
  • Insurance Claims Handling
  • IT Service Management
  • Data Workflow
  • Meter-to-Cash (Energy & Utilities)

As mentioned earlier, the focus for improvement gains will be around process cost savings, faster throughput times, and increased efficiencies. Some process-mining vendors like Celonis, also provide ready-made connectors for some of the most post popular enterprise applications out there, to name a few:

  • SAP
  • Salesforce
  • Service Now
  • Oracle
  • HP
  • Opentext

This means the lead time you’ll need to start producing insights can be dramatically reduced. Imagine being able to see your sales, delivery fulfillment, procure-to-pay, and other processes end-to-end.

Celonis Dashboard with Process, Variants, and key Dashboard Measures for Order-to-Cash Process.

The Process Mining Market

According to that Gartner White Paper, in 2017 the process-mining market reached $120 million for new product licenses and maintenance. Gartner expected to see a big increase in 2018, with the market tripling or quadrupling through 2020.

If you’re interested in pursuing the technology and techniques, here’s a list of vendors highlighted by Gartner as of last year. Gartner says there are about 25 total providers. The Top 15 ranked by client interactions, vendor briefings and academic research, are listed below.

Vendor Product
Celonis Celonis Process Mining version 4.3
Cognitive Technology myInvenio version 1.8
Fluxicon Disco version 2.1
Icaro EverFlow version 1.5
Kofax Kofax Insight version 5.5
Lana Labs LANA Process Mining — Magellanic, version 1.5.4
Minit Minit version 3.0
Process Analytics Factory PAFnow Fall 2017
Process Mining Group ProM version 6.7, ProM Lite version 1.2, and RapidProM version 4.0.1
ProcessGold ProcessGold version 9.0
Puzzle Data ProDiscovery version 2.0
QPR Software QPR ProcessAnalyzer version 2018.1
Signavio Signavio Process Intelligence version 1.9.1
Software AG ARIS Process Mining 10
StereoLOGIC StereoLOGIC Process Analytics version 5.0

Top Capabilities Going Forward

We’re still early in the process mining journey, so it’s likely that more capabilities will emerge. There’s already much discussion about using process mining to lay down the baseline for using Artificial Intelligence, Machine Learning, and Robotic Process Automation.

Gartner also highlighted some of the most promising capabilities for process mining as the technology evolves.

  • Process Models – identifying exceptions and instances
  • Customer Interaction Support and Journey Maps – capturing and improving the customer experience
  • Social and Organizational Mining – looking at process through the lens of those, internally and externally, who make it happen
  • Real-time Dashboards – testing and fixing improvements, then getting feedback ASAP
  • Capability Gap Analysis – quickly understanding and prioritizing needed capabilities
  • Multi-Process Models – truly creating an end-to-end digital twin of all your key processes
  • Process Visualization – seeing the process, including those things happening deep in a company’s systems

These are a few of the exciting areas emerging in the process mining space. If you spend any time on LinkedIn, you’ll see countless webinars associating process mining with the latest big data practices. I’d encourage you to take a look, learn, and see if these are tools you’d like to add to your Lean Six Sigma toolkit.

The next time you decide to search for “process gold,” you’ll have well-reviewed technology to help you separate process dust from gold nuggets. Happy prospecting.

Reference: Gartner, Market Guide for Process Mining – 4/3/2018, by analyst Marc Kerremans


try it


Accredited by PMI. Over 15k glowing reviews.