Long Live Spreadsheets! A Brief History & Their Modern Foothold

Spreadsheets. The very word can cast a shiver of dread through many a hardened data professional’s spine. They are everywhere, they seem to multiply, and no one can be entirely sure how many their organization may have, nor — perhaps more worryingly — where the data they contain may have originated.

In the past I have listened to BI team members in various organisations bemoan the fact that they are being asked why the data in their BI report does not reconcile with a business user’s spreadsheet. They spend hours trying to figure out the problem… only to discover that there is no provenance for the data they are comparing against. It was manually produced with no audit trail.

Or the ‘source of truth’ spreadsheet was copied out to people – and despite sharing the same source DNA the clones evolved and mutated, resulting not so much in different versions of the truth as the data equivalent of Aesop’s Fables. How did the giraffe get its long neck? How did our product list develop conflicting groupings?

Elsewhere, I have seen a single spreadsheet become responsible for the daily running of an entire company’s business flow, literally on the critical path.That one sheet put terror in the hearts of us assigned to manage it (like a spreadsheet-turned-sentient Skynet right out of Terminator). Every day felt a little nervy in case there was an issue with that sheet. Trying to automate that beast took huge efforts in analysis alone as a formula that seemed fixed would suddenly change part way down a column then return again without warning or signal to do so.

They are often recognised as the bane of a BI person’s life and yet still they exist. A wise person said to me years ago that ‘you will never get rid of spreadsheets’. They were so very right. But then would — and should — that ever be the right thing to do?

Why do spreadsheets have such a foothold? And are they really the scourge some would have them be?

A Brief Synopsis

Some will trace the story of spreadsheets back to the financial ledgers used by businesses, written on sheets and spread across the table, but spreadsheets as we know them really started from the 80s after VisiCalc paved the way in 1979.

My dad used to run his own small business and was quite the pioneer when in the very early 80s he bought a Sharp MZ80B. This was one of the earliest affordable personal computers (see picture below — and yes, that is a cassette deck, which was used to load the operating system every time you switched it on, though Dad added an external floppy disk as soon as they existed!)

Sharp MZ80B Computer

A friend of his was a computer programmer, and together they developed an invoicing system which was groundbreaking stuff for a one-man band selling office equipment. I, on the other hand, used to programme it to repeatedly display the word ‘HELLO’ until someone hit a key to break its friendly if repetitive intentions.

10 PRINT ‘HELLO’20 GOTO 10

Well, I was quite young!

It was inevitable that Dad would be an early adopter of spreadsheets, and boy did it make his life as a business owner so much easier. I think Lotus 1-2-3 was the earliest spreadsheet I can distinctly recall him using. It allowed him to keep track of income vs expenses, track stock and simply keep on top of his business in a fraction of the time. His business could grow and he was no longer the self-employed business man locked in his office doing paperwork.

Just as coal fired the Industrial Revolution, spreadsheets were a key fuel in the digital revolution – and more specifically the revolution they and PCs brought to business operations.

Understanding Spreadsheets’ Modern Foothold

Why do spreadsheets have such a deep foothold-dareIsay-stranglehold? Why does it feel like MDM in some companies is not Master Data Management but the Microsoft Data Matrix? Let’s break this down a little.

Spreadsheet Benefits

Easy to Use

— Spreadsheets can be as complicated as you care to make them, but even the simplest format can add value to a process. The interface is friendly and their use is usually taught in schools, so most people are familiar with them early in life. Anyone can create a spreadsheet.

Readily Shared

— It’s a simple file and anyone with the program can open it. There are even pretty sophisticated free spreadsheet programs out there now too.

Flexible & Adaptable

— Now with a wealth of features, spreadsheets are capable of a wide range of applications. But not only that, there is an immediacy to interactions with spreadsheets that allows quick responses to a problem; e.g., what would happen if we changed our growth forecast from 20% to 25%?

Accessible

— The same spreadsheet can be reviewed and administered by multiple people.

Sense of Understanding & Control

— Like Gollum with his ring, a spreadsheet can become a precious thing to a worker. They saw how it came into being and controlled the process, so they often feel a sense of understanding, ownership, and trust of that data.

But for every yin there is a yang and the advantages above can lead to issues.

Spreadsheet Challenges

Easy to Use

— Anyone and everyone can and will produce spreadsheets, resulting in a proliferation of them. How many spreadsheets do you have in your organisation?

Readily Shared

— Readily shared and readily altered resulting in multiple variations on a theme all independent of each other with potentially conflicting stories to tell.

Flexible & Adaptable

— With flexibility comes great responsibility. It is so very easy to amend a figure, which means it is also so very easy to accidentally mistype a figure or leave a speculative change in place as you did not realise you had auto-save on.

Accessible

— Too many cooks spoil the broth. Changes can be made all too easily without understanding the impact they could make. But of course we could always take a copy of the spreadsheet to make those changes — see above.

Sense of Understanding & Control

— With this comes a reluctance to move to automated methods, even when we know that a manually maintained spreadsheet is simply not scalable.

The issues above probably don’t affect small businesses like Dad’s; it is when you get into larger-scale operations that issues can and will surface. Just as the coal fires of the Industrial Revolution brought about a smog over industry — from which we are now clearing the airs — should ‘smog’ be the collective noun for spreadsheets deemed responsible for a smog of data, with multiple versions & quality issues. Is now the time to clear that air and embrace greener thinking?

Best of Both Worlds

In general I am a big fan of playing to the strengths of a process and managing the shortfalls. But is this possible with spreadsheets? A giant leap for spreadsheet kind would be to improve the provenance of the data they contain. It’s a simple, obvious thought, and one our product team asked in their search for extensions of our platform. They asked, ‘What if we could provide spreadsheets with governed data, the provenance of which is traceable to source?’

And so enters Alation Connected Sheets. I quite often get excited by our product roadmap but this one had me sitting bolt upright and paying close attention. By taking the spreadsheet interface business users know and love, and marrying this to a data catalog platform, business users can curate the data being consumed into the spreadsheet. Could this be a best-of-both-worlds solution to appease both business users and data professionals? I think it shows amazing promise.

When I look back at various spreadsheets I have encountered over the years I can think of numerous cases where the ability to link directly to source data in this manner would have transformed the process while still making the user comfortable and confident with its use. Avoiding the ‘downloaded and copied then manipulated (every time)’ process by creating them from a common source would have relieved substantial effort and reduced risk. All while utilising the already existing skills the business had with spreadsheets.

In Conclusion

Spreadsheets are not the embodiment of evil. They secure the loyalty of their users by being powerful and highly customisable while giving a sense of understanding and control. They need not become instruments of torture for data people. Rather they are valuable tools which can power businesses to success – but this relies upon them being deployed and utilised with an element of structure and dare we say governance. In the wrong hands many innocent things can become weapons of mass data-destruction. But in the right hands such tools can power meaningful growth for a company.

Quite frankly, without the humble spreadsheet, us data professionals may well not have the data industry and careers we know and love. Spreadsheets have made running a business more feasible and scalable but also put data front and centre to all people. Our explanations of how powerful data can be are made so much easier because most people at some point or other have used a spreadsheet. Chances are I am also not the only one who got to see their mum or dad a little more because of them.

Perhaps we of the data set need to respect our ancestry a little more. I for one will on October 17th, designated International Spreadsheet Day, be raising a glass to the oft maligned spreadsheet in recognition of the transformational power they possess and also in anticipation of the new era they may well be entering through giving them a well governed home. I think I may ask Dad to join me (if I can tear him away from Solitaire).