All Posts in executives
February 24, 2021 - Comments Off on Legacy system failures expose the application knowledge gap’s harmful risks
February 24, 2021
by Steve Brothers
Government system failures during the rush to provide public benefits to alleviate the economic effects of the COVID-19 pandemic publicly exposed the mainframe knowledge crisis that also threatens financial institutions, healthcare providers, and many other organizations foundational to the world economy.
Several states discovered the knowledge-gap’s potentially devastating consequences when waves of unemployment-claims poured into their systems as COVID-19 ravaged the economy in early 2020. The states’ unemployment computer systems crashed trying to process the deluge of claims using mainframes and decades-old programming languages.
But it wasn’t the mainframes or the legacy programming languages that failed, despite what you may have read. It was the lack of available expert programmers necessary to maintain and update these systems to handle the voluminous claims.
According to The Verge’s article, “Unemployment checks are being held up by a coding language almost nobody knows,” Colorado employed exactly one full-time programmer to maintain the state’s COBOL system prior to the pandemic. Back then, Colorado processed roughly 2,000 unemployment claims per week. In March and April 2020, that number rocketed to as high as 104,572 claims per week.
Now governments, non-profits, and private organizations are reviewing their systems’ strategies to learn from these mistakes. If your business relies on legacy systems, you probably should keep reading – and schedule some time with your IT people.
Mainframes are cornerstones
Legacy mainframe systems and software bedrock many of our most trusted institutions, including government services, finance and banking, healthcare, and insurance. In a substantial number of cases the expert developers that created and maintained these systems and software are retiring without a supporting workforce to replace them.
Besides the people that make your business run, your software is potentially the most important resource your organization has. Internal applications likely drive your employees' capabilities and productivity. Customer-facing programs attract new customers, close business deals, and increase revenue. New applications and features can open new markets and opportunities.
To maintain and improve your critical applications, your software team relies on individual engineers that developed an expert understanding of your programs through years of experience. They know the applications and all the accumulated system changes and challenges.
When those experienced engineers depart your business, the developers that replace them must acquire the same application knowledge through training, mentorship, and on-the-job programming. This exercise introduces several material business risks.
Learning on the job
Developers new to software applications typically require 6-12 months of on-the-job learning to become productive, depending on the size of the source code base. To become proficient, programmers may need up to 3 years.
Without qualified software developers knowledgeable about your applications, you endanger your business's operations, reputation, and security. You also risk a significant decrease in your software teams' productivity and efficiency.
Consider the monumental task confronting newly hired or transferred IRS developers last March. Congress passed the CARES Act on March 25, 2020, and then Treasury Secretary Steven Mnuchin announced that individual stimulus checks would be mailed in early April.
To assist with the delivery of economic stimulus payments, the new developers were required to immediately start working with the agency's source code base, which, in 2019, was estimated at nearly 20 million lines of code and includes over 60 years of legislative and system changes. As of mid-May 2020, nearly 20 million people had not received their stimulus checks, and some recipients had problems throughout the year.
These engineers didn’t have 6-12 months to become productive. They had to hit the ground running on day one. And without the benefit of weeks or months of training and on-the-job learning, they didn’t have the application knowledge necessary to understand how even simple changes could affect entire applications.
And let's not forget the productivity loss due to the remaining IRS's experienced engineers for training, mentoring, and supervising the new recruits.
What’s your risk?
Your situation may not be as dire as what the IRS faced – for now. But how much time can you really give your new developers to learn the system, and how much productivity can you afford to lose while your experienced programmers train and supervise them?
How much do you trust the developers that have just started working on your critical applications?
How confident will you be when your CEO or Board of Directors asks for assurances that the next customer-facing application update will not result in outages and lost revenue – especially if the update was programmed by a developer you hired just weeks ago?
Your software is a critical part of your organization, especially if you rely on legacy mainframe systems. You must have a plan or tool that keeps the code running and bridges the gap between retiring and departing developers and the people that will replace them.
April 24, 2019
by Greg Brueggeman
COBOL maintenance costs are rising as aging developers leave the workforce and well-trained replacements are in short supply. Learn about how pervasive COBOL-based applications remain, why the supporting developer workforce is shrinking every year, and scrutinize our estimates of how high these applications' maintenance costs are rising.
I began working with mainframe programming languages in 1990 while I was in college. I started with FORTRAN and assembly languages, and I knew about COBOL but wasn’t exposed to it until much later.
While FORTRAN and assembly languages now have their niche uses, COBOL is entrenched as a vital part of the global economy. And now the cost of maintaining the nearly 60-year-old language is rising precipitously.
A little history
The Common Business Oriented Language (COBOL) was developed as a stop-gap measure during the second Eisenhower administration to create a portable mainframe programming language for the Department of Defense. It was based on the FLOW-MATIC compiler, which was recognized as the first English language data-processor compiler and was designed by Rear Adm. Grace Hopper.
I predict that the last mainframe will be unplugged on March 15, 1996.
~Stewart Alsop II, InfoWorld magazine, 1991
The bad news is that some of these applications are nearing 50 years in age. And although COBOL remains vital to many critical systems, this prominence has not resulted in a stable supporting workforce. The population of experienced COBOL developers declines at least 5% every year because of retirement. The population’s average age is roughly 55 years old, and the language’s absolute uncoolness has motivated the majority of university computer science programs to drop their COBOL classes entirely. The trickle of newly trained COBOL programmers are coming from community colleges, technical schools, and programs run by private companies such as IBM.
The continuing importance of COBOL-based applications and the diminishing trained workforce is causing the cost of maintaining COBOL-based applications to rise.
How often do defects arise?
You might ask, “Well, if COBOL is so reliable, why do we need lots of programmers? Fewer bugs naturally means fewer engineers.”
That sound perfectly logical, however, in my experience, due to the lack of a sufficient supporting workforce and the spaghettified nature of today’s COBOL-based applications, they are not modified and supported as well as other applications programmed in more popular languages.
Think about all the patches, additions, and technical debt that have built-up since these applications began surfacing in 1960, as well how many developers contributed to those applications but are no longer around to address issues and answer questions.
Due to the uncertainty of downstream impact and to reduce the risk of breaking critical applications, I believe many COBOL engineers duplicate hundreds or thousands of lines-of-code when making changes or repairs because they do not completely comprehend the COBOL code in their system. As a result, changes are made, tested, and implemented in an environment of uncertainty and heightened risk.
And when the flaw is discovered, the repair cost includes:
- Consultants or outside IT contractors
- The development team’s time away from building new features and products
- Salaries of internal personnel tasked with fixing the bug or supporting the project
- Application downtime
- Lost business opportunities
The Y2K scare is just one example. While it ended up being much less daunting than many estimates, roughly $320 billion was spent worldwide evaluating and fixing systems.
But the price of fixing COBOL code includes more than just repairs and downtime. An organization’s security, reputation and market position can be affected by one time-bomb defect. When an August 1, 2012, software failure caused Knight Capital LLC to create thousands of trades per second on the New York Stock Exchange, the company lost between $440-460 million in 45 minutes. By the next day, Knight’s stock had fallen 75% and within the year the company was acquired by a rival.
What is the cost of fixing COBOL code?
So, what is the actual cost of fixing COBOL code?
For the sake of simplicity, let's analyze the cost based on salaries and time spent supporting COBOL-based applications, but not developing new programs.
Here are our assumptions:
- A 2012 Compuworld survey found that 95% of companies with COBOL-based programs say they are using the programming language to maintain production applications. Of those, 44% are using COBOL solely for system maintenance.
- There are an estimated 2 million COBOL programmers worldwide. In 2003, Gartner estimated there were 90,000 in the U.S., however, based on estimates from my industry contacts, I would guess that about 20,000 of those are still active.
- According to ZipRecruiter.com, the average salary for a COBOL programmer in the U.S. is about $81,000.
If we conservatively estimate the cost of maintaining COBOL-based applications by only including the developers working solely on maintenance, it’s roughly $1.02 billion per year. Here’s how we get there.
20,000 COBOL developers
x 42% solely doing maintenance
x $81,000 per year
= $680.4 million per year
In my experience, you have to add another 50% of the development cost for quality assurance (QA) testing, which would bring the total to $1.02 billion just in the U.S.
+ $340.2 million for QA testing
= $1.02 billion per year
If anything, these numbers are low because we didn’t consider the companies doing both maintenance and new development in COBOL.
The cost of fixing COBOL v. alternatives
Another approach to handling applications written in legacy programming languages is to modernize the code – translate the COBOL source code to a modern language such as Java.
Despite the astronomical maintenance costs, it doesn’t necessarily make sense to modernize COBOL-based applications. Hundreds of millions of dollars are spent each year on COBOL transformation projects, and more than a few have ultimately failed.
The risks associated with a COBOL transformation project are significant. Dave Brown, Systems Architect at The Bank of New York Mellon, said in 2012 that his bank had 343 million lines of COBOL code. Transforming a code base of that size and complexity would take years and hundreds of millions of dollars.
Just ask the executives at the Commonwealth Bank of Australia, which undertook a planned AU $580 million (US $413 million) legacy core banking technology transformation in April 2008. The project was finally completed in August 2013 at a final cost of AU $1.1 billion (US $783.871 million) – almost two years behind schedule and $370 million over budget. And this was a successful legacy system conversion.
The COBOL programming language is not destined for retirement. Organizations can plan on continuing to spend a high percentage of their IT budgets maintaining legacy systems, especially in the financial and government sectors. An April 2017 article by NextGov.com claims that the federal government spent roughly $90 billion in 2017 maintaining legacy systems, which was roughly 80% of its entire IT budget ($90 billion in 2017).
Until new technology comes along to enable organizations to better understand their COBOL-based applications, unravel their complex and spaghettified code, and extract the embedded technical, business, and regulatory knowledge buried within, this trend will continue. Whichever approach you take, right now the only solution is to throw time and money at the problem.
*Greg Brueggeman is the Director of Product Management at Phase Change Software. You can reach him at firstname.lastname@example.org.
January 25, 2018 - Comments Off on Introducing Mia — the first assistive AI for software development
January 25, 2018
by Todd Erickson
Phase Change proudly presents our maiden in-house video, featuring Mia, the first assistive AI for software development.
Mia helps organizations retain the expert knowledge encoded in software. Developers, stakeholders, and executives can use this knowledge to better understand their applications and increase development productivity by a factor of 100.
Learn why founder and CEO Steve Bucuvalas first began to envision the technology and see how Mia collaborates like an expert to help users explore and comprehend their software.
Discover why one executive said, "I never thought I'd see this in my lifetime."
Todd Erickson is a tech writer with Phase Change. You can reach him at email@example.com.
August 4, 2017 - Comments Off on Phase Change enables market adaptability through impact analysis
August 4, 2017
by Todd Erickson
Gary Brach, Ken Hei, and Brad Cleavenger discuss how Phase Change's assistive AI removes the doubt associated with changing software applications.
Changing software is difficult and expensive, and it can be a major stumbling block to business innovation.
Phase Change's assistive AI will enable software teams to quickly and fearlessly address market opportunities by rapidly assessing the scope and viability of proposed software modifications, and then efficiently making changes without adding the technical debt that reduces system performance and application life span.
Todd Erickson is a tech writer with Phase Change. You can reach him at firstname.lastname@example.org.
March 6, 2017 - Comments Off on An Analogy: Software AI and Natural Language — blog
March 6, 2017
Today's AI technology is amazing.
Only a few short years ago, only humans could interpret the meaning of text and speech. Now our cell phones understand our voices and language well enough to distinguish accents, metaphors, and sarcasm.
IBM's Watson supercomputer even understood Alex Trebek well enough to beat some of Jeopardy!'s® best players.
Computers achieve natural-language understanding through a series of logically consistent normalization steps -- starting with the processing of basic sounds to recognizing words and then understanding sentences.
If computers can understand natural language using logically consistent processes, shouldn't we be able to use similar processes to break down and normalize software?
In fact, shouldn't software be easier to normalize than the messy ambiguity of human communication?
Phase Change normalizes software source code into formal data types and organizes them into hierarchical structures that are probabilistically linked (horizontally and vertically). Our technology unlocks the vast domain and system knowledge embedded in software and makes it available to anyone involved in creating and supporting software.
To learn more about how Phase Change's revolutionary technology transforms chaotic code into coherent data and intractable software into artificially intelligent agents, read Steve Bucuvalas' paper: "An Analogy: Software AI and Natural Language."
February 16, 2017 - Comments Off on Leveraging software’s encoded knowledge to create an assistive AI — science podcast 4 of 4
February 16, 2017
This is the fourth and final in a series of practical talks by founder and CEO Steve Bucuvalas about Phase Change Software, what we are developing, the math and science behind our technology, and the impact on the software development process.
Using a whimsical example of dog banking, Steve discusses how the knowledge that’s encoded in software is normalized into a data structure, which enables us to create an assistive AI and solve the learning curve problem.
Podcast Slides and References
February 16, 2017 - Comments Off on Changing the essence of software and creating breakaway efficiency — science podcast 1 of 4
February 16, 2017
This is the first in a series of practical talks by founder and CEO Steve Bucuvalas about Phase Change Software, what we are developing, the math and science behind our technology, and the impact on the software development process.
In keeping with the physics' definition of the term ‘phase change,’ we are changing the essence of software. Taking something that is chaotic and turning it into something coherent. Taking something that is intractable and hard to understand and making it into an AI that actively helps every person in the software development process.
January 5, 2017 - Comments Off on How Phase Change’s AI impacts release management — video
January 5, 2017
Phase Change President Gary Brach leads a practical discussion with Ken Hei, director of engineering, and Brad Cleavenger, senior software architect, about how Phase Change's technology will transform release management.
January 5, 2017 - Comments Off on Math and science make the difference — video
January 5, 2017
Founder and CEO Steve Bucuvalas explains why Phase Change is well-founded in science and how it is overturning historical assumptions about computational theory.