In an Agile class this week, someone asked the question “How can you be sure the sprint testing covers all code changed? Developers often make a change in some different area of code to fix something and testers don’t know to test that area at all since it wasn’t part of the sprint.”

Huh?

That’s a good example of a team that needs a good process. And he wasn’t the only one in the room waiting for an answer. I learned the answer to that question in the early ‘80s. And the lesson was a hard one. ..

I had just been assigned as manager for the Electronic Defense Systems software department for Ford Aerospace. It wasn’t my first management job but was my first position responsible for government customer deliverables rather than internal projects like developing in-house CASE tools or complex algorithms to help the engineers building satellites and ground stations. The five software sections I would manage were each working on separate contracts for different arms of the military. I had met the five supervisors in my new department and their developers but hadn’t officially started work when my boss, Sharon, who headed up all software engineering for Ford Aerospace came to me and said that the customer for one of my new contracts, an Army Lt. Colonel, was going to be giving awards for completing the testing of the release and wanted me to come to the awards ceremony to meet him and see the team get their awards.

It was about 5 PM as we walked across the campus. But as we started up the steps to enter the building the doors opened and young men and women, the developers and testers, came streaming out and down the front steps with tears in their eyes or outright sobs.

“What’s going on?” asked Sharon.

They couldn’t answer and ran by us. The company’s General Manager was hurriedly entering the building. He saw Sharon and said sternly “I was called to a meeting with the Lt. Colonel. You’d better come with me.”
We three went upstairs into a warehouse room where an impromptu meeting was being held. The head of Systems Engineering, Ray (Sharon’s boss) was already there with the Lt. Colonel. The Lt. Colonel was a tall man in full army fatigues. Red flat-top hair with a face as red as his hair. He was livid.
“I normally only give companies one chance. But for you”, he said glaring at our GM, “this is the third strike.”

He saw me and said “Who’s this?”

Sharon told him I was the manager of the software group. He glared at me and said “How are you going to remedy these problems?” Sharon, to my relief, quickly let him know it was my first day as manager. He directed us to find out what was wrong and get it fixed that night or Ford would never have another contract with his division. But, he said, if we want to re-run the test scenarios he was willing to spend the night here but he had to be on a plane, with the verified software by 8 AM in the morning. Sharon said we’d find the problem and run the tests.
We went to the supervisor’s office. The supervisor was a tough, seasoned, ex-Marine used to conflict and discourse, was also crying and said “This is it, I’m through. I quit.”

“What happened?”

“We started the test scenarios and the first test was simply adding a new user. There was a system error. The SETA (Systems Engineering and Technical Assistance) contractor quickly jumped to the conclusion that the code wasn’t complete and we must be hiding problems and issues. The Lt. Colonel blew up, yelled at everyone, and they left in tears. I don’t understand it. We worked long and hard on this release and the tests we ran were perfect.”

A senior programmer, one of Ford’s best, appeared in the doorway. “Excuse me,” he stammered. “I’m afraid I know what is wrong.”

Seems he, at the last minute, had noticed a small issue in the software. Although he was well aware that the process was to first notify the Configuration Manager (CM) before changing anything, and this change wasn’t even needed for this delivery, he thought it was just a simple quick fix. Unfortunately, he failed to realize that that the code was shared by a newly implemented user registration function and his change broke that function. He had reverted the code to the prior version and was sure it would now pass the tests.

The team was rounded back up and were all more than willing to spend the night re-verifying the tests to exonerate themselves. In the morning the tests had all been passed with glowing colors and the Lt. Colonel at 7 AM left the awards to be passed out to the team later in the day.

Ironically, the senior developer who caused the issue was the one who was scheduled to travel with the Lt. Colonel to install the software in Saudi Arabia. The LT. Colonel seemed to take pleasure in telling him before they left that he’d need to shave his beard. “What?” the developer complained. “Well, it’s up to you, but the gas masks don’t fit well over beards.” This was Desert Storm. He shaved.

A good process needs to be both easy to follow and so integrated into the workflow that it’s second nature. It isn’t a burden, just a simple step in the normal activities. And developers have to “buy in” that it’s required, important, and a step that cannot ever, ever be skipped. The issues caused may not be as significant as the story above, but if changes are made to code without a process, there will be issues. This is true for all projects – from major defense software down to little start-ups.

The process we implemented at Azerity was that our SD Tracker tool was the tracking system for every change. Every change to our requirements docs or code needed to be an “SDR” in SD Tracker. Adding an SDR was quick and easy. If a developer wanted to change code for any reason, all they needed to do was enter an SDR saying that. Of course the development managers had guidelines about when and which developers could act immediately on their desire to change other areas of code and when reviews or other conditions were needed to be met. Then QA knew it needed verification and we could confidently tell our clients that our Release Notes listed all areas of code changes which helped reduce their SOX testing on maintenance releases.