Don’t miss the benefits of “Small Data” in the shade of "Big Data"

Nowadays, everyone is speaking about big data to be used in deep learning, machine learning, and AI applications. However, how many of the existing companies have access to such big data? And even if they have, how ready the data is to be used for decision making purposes or how rich the data is in terms of knowledge and wisdom that can create? (refer to Thick Data concept and DIKW model)

When we talk about small data, we normally talk about hundreds or less of data points, in compare with millions and billions of data points in big data concept. Thus, projects that can be run using small data require much lower resources and budgets and can be done with basic analytical methods that are available for everyone [1]. So, companies normally don’t need a team of experienced data scientists to run such projects. Additionally, although the financial benefits of such small data projects are between 100,000 SEK to 2,500,000 SEK per year [2], when this is compared with the high rates of failure for big data projects [3], it does not sound bad at all. Moreover, such small data projects, will pave the way for the organization towards using the big data in bigger projects, by making the team trained and accustomed to use data, as well as finding the data gaps in terms of data points as well as formats. Such small projects are also beneficial from a change management perspective, as it is not perceived by team members as something that is going to take their job away, and additionally the employees will have the chance to adapt and train themselves for the bigger change that may come in the future with big data projects.

Although performing small data projects may not seem difficult, it requires a well-planned change management process within the organizational thinking to prioritize such small projects with lower financial benefits over bigger ones with more fruitful financial benefits but also higher risk.

In their recent article in HBR, Thomas C. Redman and Roger W. Hoerl [1], have suggested four steps for applying such change in an organization, in order for all to benefit from such small data projects:

1- Get everyone involved — yourself included

Since small data projects can be done with basic analytical methods, everyone, including yourself can be involved in such projects. Thus, they recommend to have at least one of such projects per year done by everyone at the company. There are many areas that such small data projects can be defined and performed, and you will find more areas the more you perform such projects. Examples of areas that you can perform small data projects are:

a- Eliminating date errors: Poor data quality is quite common, and it is highly costly for your business [4]. The goal for such small projects should be to reduce the non-value-added work needed to deal with data errors.

b- Reducing wasted time: Nowadays in any size of corporate (the larger the company the bigger the problem), people waste a lot of time waiting for meetings to start, waste time on meetings that are not relevant or poorly planned, waiting for inputs from a colleague, a shipment to arrive, and so on. The goal here should be to reduce that waiting/wasted time.

c- Simplifying handovers: As work proceeds into, across, and out of your team, non-existence or poor handovers may increase complexity, cost, or time. The goal of such projects is to establish and streamline these handovers.

2- Follow a disciplined approach

While working with small data you may be tempted to jump into the results and the conclusion as soon as possible; however it is beneficial from the change management and training perspective that you define an approach and follow it for all of your projects; first learn the process and then mature the process eventually to make it ready for the time you want to run big data projects. An example of such simple process is: i. define the business problem, ii. gather the needed data, iii. analyze the data, iv. make improvements, v. lock in the gains, vi. identify the next opportunity, and vii. repeat the cycle [1].

3- Provide training

As you and your team start to work with data and data projects, you will find gaps in your skills and knowledge related to data projects in your domain. Performing small data projects gives you the opportunity to recognize those gaps and plan to receive the required training to fill that gap. In this way, the small data projects can also work as case studies in your training process. This is helpful to master the skill that you are learning as you always learn better by doing!

4- Define your unique area of expertise

By performing these small data projects, you should try to find your niche and the domain that you like or have/gained the most competence, so you can make that the area of your expertise. This way, when you move towards big data projects, you can act as the expert in that specific domain in your team for big data projects.

References:

1- Redman, T. C., Hoerl, R. W., 2019, Harvard Business Review, “Most Analytics Projects Don’t Require Much Data” https://hbr.org/2019/10/most-analytics-projects-dont-require-much-data

2- Snee, R. D., Hoerl, R. W., 2018, “Leading Holistic Improvement with Lean Six Sigma 2.0” https://www.amazon.com/Leading-Holistic-Improvement-Lean-Sigma-dp-0134288882/dp/0134288882/

3- Satell, G., 2018, DigitalTonto, “Most AI Projects Fail. Here’s How To Make Yours Successful” https://www.digitaltonto.com/2018/most-ai-projects-fail-heres-how-to-make-yours-successful/

4- Redman, T. C., 2016, Harvard Business Review, “Bad Data Costs the U.S. $3 Trillion Per Year” https://hbr.org/2016/09/bad-data-costs-the-u-s-3-trillion-per-year