Data Mining

Data mining is the process of sorting through large data sets to identify patterns and relationships that can help solve business problems. Data mining techniques and tools enable organizations to predict future trends and make better business decisions.

Data mining, which employs advanced analytic techniques to uncover useful information in data sets, is a critical component of data analytics and one of the foundational disciplines in data science. Data mining is a step in the Knowledge Discovery in Databases (KDD) process, a data science methodology for collecting, processing, and analyzing data. Data mining and KDD are sometimes used interchangeably, but they are in fact distinct concepts.

History of Data Mining

The concept of data mining predates the invention of the computer. The statistical origins of data mining began with the discovery of Bayes' Theorem analysis in 1763 and regression analysis in 1805. The Turing Universal Machine (1936), the discovery of neural networks (1943), the development of databases (1970s), genetic algorithms (1975), and Knowledge Discovery in Databases (1989) laid the groundwork for our current today understanding of data mining. During the 1990s and 2000s, with the growth of computer processors, data storage, and technology, together data mining became not only more powerful but also more productive across all types of applications.

Data Mining Process

The data mining process can be divided into four main stages:

1. Data collection: Data relevant to an analytical application is identified and gathered. Data can be stored in a data lake, increasingly popular as a source that combines data from various source systems, a data warehouse, or a mix of structured and unstructured data. External data sources may also be used. Regardless of origin, a data scientist typically moves the data into a data lake for use in subsequent steps of the process.

2. Data preparation: This stage consists of a series of steps that prepare data for mining. It begins with data exploration, profiling, and preprocessing, followed by data cleaning to correct errors and other data quality issues. If a data scientist does not wish to analyze raw, unfiltered data for a specific application, data transformation is performed to make datasets consistent.

3. Data mining: After preparing the data, the data scientist selects an appropriate data mining technique and applies one or more algorithm to perform the mining. In machine learning applications, algorithms are typically trained on sample datasets before being applied to the entire dataset to search for desired information.

4. Data analysis and interpretation: Data mining results are used to generate analytical models that can support decision making and other business actions. The data scientist or another member of the data science team must communicate findings to business managers and users, often using data visualization and storytelling techniques.

Data Mining Techniques

There are various data mining techniques, and the one you use depends on your overall objective. Different data models exist, each based on distinct data mining techniques. The main data models are descriptive, predictive, and prescriptive models.

Descriptive Modeling

This identifies similarities or groupings in data to understand the causes of success or failure (for example, categorizing customers by product preferences or emotions). Example techniques include:

1. Association rules: Also known as basket analysis, this type of data mining examines relationships between variables. For instance, association rules can analyze a company’s sales history to determine which products are most frequently purchased together. The company can use this information for planning, campaign and forecasting.

2. Cluster analysis: Clustering aims to identify similarities within a dataset by dividing data points into subgroups based on shared common characteristics. Clustering is useful for segmenting customers by purchasing behavior, needs, life stage, or preferences in marketing communications.

3. Outlier analysis: This model is used to identify anomalies, or data points that do not conform to expected patterns. Deviant value analysis is particularly useful in fraud detection, network intrusion detection, and crime investigations.

Predictive Modeling

This modeling goes deeper to classify future events or predict unknown outcomes (for example, using credit scores to determine the likelihood of a person repaying a loan). Example techniques include:

1. Decision trees: Used to classify or predict outcomes based on a list of criteria. A decision tree takes input as a series of hierarchical questions that sort the dataset according to given responses. Sometimes visualized as a tree, decision trees allow deeper exploration of data and accommodate user input.

2. Neural networks: These process data through the use of nodes, which consist of inputs, weights, and an output. Data is mapped in a manner similar to how the human brain functions, making this approach suitable for setting threshold values to determine model accuracy.

3. Regression analysis: Regression analysis aims to identify the most significant factors in a dataset, determine which factors can be ignored, and understand how these factors influence each other.

4. Classification: This involves assigning data points to groups or classes based on a specific question or challenge. For example, a retailer seeking to optimize discount strategies for a particular product may examine sales data, inventory levels, coupon usage rates, and consumer behavior data to guide decisions.

Prescriptive Modeling

With the rise of unstructured data from the internet, email, comment sections, books, PDFs, and other text sources, the adoption of text mining as a discipline of data mining has significantly increased. Data analysts require skills to parse, filter, and transform unstructured data to incorporate it into predictive models for enhanced prediction accuracy.

Data mining auxiliary tools (Source: vizyonergenc.com)

Data Types in Data Mining

Data types suitable for mining include:

1. Data stored in a database or data warehouse

2. Transaction data (e.g. flight reservations, website clicks, store purchases, etc.)

3. Engineering design data

4. Sequential data

5. Graph data

6. Spatial data

7. Multimedia data

Applications of Data Mining

Data mining is used for numerous purposes depending on the organization and its needs. Some Possible application areas include:

1. Sales: Data mining can help increase sales. For example, consider a store’s point-of-sale records. The retailer logs the time of each sale, which products are purchased together, and which products are most popular. The retailer can use this information to optimize its product range.

2. Marketing: Businesses can use data mining to improve marketing activities. For instance, insights from data mining can help determine where potential customers see advertisements, which demographics to target, where to place ads, and which marketing strategies are most effective for customers.

3. Production: For companies manufacturing their own goods, data mining can analyze the cost of raw materials, whether materials are used efficiently, how time is spent during the production process, and what obstacles affect production. Data mining can ensure timely fulfillment of needs by predicting what new materials should be ordered or when equipment should be replaced.

4. Fraud detection: The goal of data mining is to find patterns, trends, and correlations among data points. An organization can use data mining to identify anomalies or correlations that should not exist. For example, a business might analyze cash flow and discover recurring payments to an unknown account. If this is unexpected, the company may request research to do to investigate potential fraud.

5. Human resources: HR departments typically have a wide range of data to process, including employee retention, promotions, salary ranges, company benefits and how they are used, and employee satisfaction surveys. Data mining can correlate this data to better understand why employees leave and what motivates new hires to join the organization.

6. Customer service: Customer satisfaction is shaped by various factors. For example, consider a retailer that ships goods. A customer may be dissatisfied with delivery time, delivery quality, or communication regarding delivery expectations. This customer may experience long telephone wait times leading to dream frustration. Data mining collects operational insights about customer interactions and summarizes findings to identify both strong and weak areas of performance.

7. Customer retention: Companies can use data mining to identify characteristics of customers who have switched to competitors and then offer special incentives to other customers with similar traits to retain them.

8. Security: Unauthorized access perception techniques use data mining to detect anomalies such as network outages.

9. Entertainment: Streaming services use data mining to analyze what users watch or listen to and provide personalized recommendations based on their habits.

10. Healthcare: Data mining assists doctors in diagnosing medical conditions, treating patients, and analyzing X-rays and other medical imaging results. Medical research also relies heavily on data mining, machine learning, and other analytical methods.

Data Mining Software and Tools

Data mining tools are available from a variety of vendors, often as part of larger software platforms that include data science and advanced analytics capabilities. Key features of data mining software include:

1. Data preparation capabilities.

2. Built-in algorithms.

3. Support for predictive modeling.

4. GUI-based development environment.

5. Tools for deploying models and evaluating their performance.

Alteryx, Databricks, Dataiku, DataRobot, H2O.ai, Knime, RapidMiner, SAP, SAS Institute, and Tibco Software are among the vendors offering data mining tools.

DataMelt, Elki, Orange, Rattle, scikit-learn, and Weka are free, open-source technologies capable of data mining. Some software vendors also offer open-source options. For example, Knime combines an open-source analytical platform with commercial software for managing data science applications, while Dataiku and H2O.ai provide free versions of their products.