Big Data in Construction

These days Data Analytics, Business Intelligence, and Artificial Intelligence are all catch phrases we hear very often in the media.

What these terms all reference, is the ability of machines to process and make sense of lots of data - Big Data – and this is influencing our lives in many ways.

It’s helping in the fields of GPS/navigation, marketing, healthcare, transport, agriculture - the list is extensive.  Big data is transformative, and it is here.

We have always had data - So what’s so special about now?

Two reasons:

  1. Technology platforms - It is much cheaper now to create, store and organise data than it ever has been. And consequently, we are creating more and more.
  2. Software - We now have better tools to manage data and create actionable insights. The data can tell us about what we are doing in ways we haven’t had access to before.

How does big data fit into construction and engineering?

Estimating Costs

With big data, we can quickly access historical cost records to allow us to create budgets and check prices for projects large and small. In addition, databases of supplier and subcontract rates can easily be called upon to build up prices using standardised methodologies and algorithms.

Managing Costs

Getting timely and cost data from the field allows managers and engineers manage construction costs by giving them accurate real-time costs to date. This helps identify problem areas that require urgent attention. Companies like SiteDiary specialise in this (

Internet of Things

Machines are getting smarter. We are now able to install sensors to monitor real-time fuel use, location, and machine efficiency. This leads to better utilisation of plant and equipment across many industries

Health and Safety

By recording and tracking trends in health and safety related events, we are able to create meaningful leading and lagging indicators on the effectiveness of company/industry practises. This greatly benefits society and workers alike


When we can easily and quickly create data about the work we do, it becomes easier to identify, track and deal with quality issues in a timely manner. This is especially true when it comes to setting service standards.

There are so many ways we can use data, both quantitatively and qualitatively to help improve what we do. If it can be measured, it can be managed, and I think we will start to see new and interesting ways companies use data to give them a competitive edge.

One thing you can be sure of is, is that big data is already touching your life. And it will continue to do so in bigger and bigger ways.

“The most valuable commodity I know of is information."  (Gordon Gekko, Wall Street)

Why Excel is still King

Ever needed to:

  • Create a chart showing what’s hot and what’s not?
  • Track your costs within your business?
  • Quickly calculate how much it would cost you to do this or that?

Excel to the rescue!

Besides simple tables, you can build really complex spreadsheets and algorithms and do a huge amount of data analysis. There’s not a lot it can’t do!

Some of the tools that I have found most useful are:

Macros - Automate your repetitive tasks. This can help with things like cleaning up data and carrying out iterative calculations (i.e. calculations that require multiple steps and have if/else logic). More info:

Tables - Tables in excel make life easier by allowing you to add formulas to an entire range of information quickly and also get that information to be used in other areas of the spreadsheet. More info:

Pivot Tables -  This is probably my favourite tool. If you have large amounts of data that you want to get insights from - this is what you want to use! More info:

Data Models - This is relatively new to me but it is turning out to be a really powerful tool for data analysis. If you use databases in excel, then you really need to learn how to use this. More info:

Formulas - This is the most common and used aspect of excel. The formulas are powerful and easy to use. I think most people would only use less than 10% of what is available so there is always something new to learn.

When you combine all the features and flexibility that you get from excel, it is extremely hard to ignore this program as a serious business tool.

Despite being so awesome, there still are some downsides:

Data can and does get corrupted from time to time so it is important to keep back ups! (and backups of backups)

Formulas can be copied incorrectly and cause errors in calculations

Inheriting a complex spreadsheet from another person can become a huge task to breakdown understand the logic behind it.

The good news is that Excel is always getting better!

And, one of the most exciting things is its ability to use and manipulate data. This is important as this is part of the growing emphasis on data analysis and business intelligence that is gaining momentum across the globe.

The ability to manage, interpret, manipulate and visualise data is becoming an indispensable tool for businesses. I think it’s great that our trusty office companion Excel is rising to the task and providing businesses with the means to do this in an affordable way with a tool that is so readily available.

Oh, and SiteDiary works really well with Excel!

And that’s why I think Excel is still King

How it all began...

As an engineer on civil construction sites, I have always been tracking costs and the performance of our projects. A lot of civil construction projects that I have been involved in have been self-performed - meaning that as a company we controlled the labour, equipment, and materials that get the job done. This makes tracking costs harder than if we were managing subcontractors because we need to be able to track where each person and piece of equipment is working every day. The best way to track these type of costs is to keep a daily log of site events that describe where each person has been working, what the plant and equipment have been doing, materials used and what has been achieved. This is all done on a construction site diary. 

75% of all companies I have surveyed still use a paper-based site diary system. This reflects my own experience as all but one company I have worked for have used a paper-based system. Paper-based systems aren't bad and the do work! They provide you with most of the data you need and are flexible enough that if you need to change something you can just scribble down some notes.

The main problem I faced was that over all these years the site supervisors would spend a lot of time writing up these daily site diaries and then I would spend even more time (and I mean a lot!) copying the same information into excel spreadsheets for analysis. This is fine if you have a small project BUT, once you have a fast-paced, reasonably large job, things start getting insane when you try to capture all of the costs!

How I wished there was an easier way to do this!

And that was when I started thinking that maybe there was.

Then along came CCIQ and BlueChilli. Together they had created the Collaborate Program and to start it off they issued their Small Business Challenge. Investment dollars were being offered if you had a good idea to help small businesses improve productivity.

In Feb 2016, SiteDiary made the top 20 and was given the chance to pitch to the judges. Sadly, we didn't win any investment from CCIQ but BlueChilli was keen to give SiteDiary a chance. 

And the rest, they say is history....