What Pixar can teach us about Software Development

I had visions of a career focusing on game development and animation, my thesis was even on the subject, and I spent 2 years working in a game development studio. Even though I changed career paths years ago, I have followed Pixar for years, from before the first Toy Story was released. I remember the 1996 Computer Animation Conference at the Ontario Science Centre where Pixar showed off a blooper reel from Toy Story.

So it was instinct that I grabbed the May 2010 issue of Wired magazine, with the cover story of "How Pixar Works". As I read the article I stumbled into something, software companies have it wrong; the core development lifecycle used for close to 50 years is inheriently flawed. It doesn't matter whether they use waterfall, agile, scrum or any other development methodoligies they are all disruptive and prone to failure.

Pixar's culture is one of simplicity, create an environment of success and ownership through innovation and development. Everyone is invested and owns the final product, and after four years in development no one wants to let go of Woody or Buzz, they want to keep making them better. The article highlights a key principle, "the project team meets each morning to rip apart the two or three seconds of film from the previous day searching for ways to make the shot more expressive". This is done collaboratively, even the newest and most junior team members have input. At the end of the project, the team moves intact to the next project allowing trust, communication and relationships to develop between team members.

So, how did software companies get it so wrong?

Arcane software development lifecycles imprinted over time and transferred between software organizations through staff turnover have choked off thought leadership and preserved the inability to change. As it relates to development process, those who come into organzations want it their way, whether it is a conversion to agile or scrum. As it relates to team organization, the standard lifecycle limits input by those working on a project (a developer or tester) as the product is defined before they see it and it is too late in the product development lifecycle to impact design or feature sets. The standard lifecycle prevents trust between roles as it is inheriently combatively and people are afraid of failure. A last minute issue results in arguments over whether it was valid as it wasn't in a test plan, or whether the quality of testing was high enough etc.

What has to change?

Sofware companies need to throw away what they know and start over. They need to stop being combative and work together. Take away the development lifecycle, split the resources into teams, not verticals or by role/department but rather by project. In the simplist terms, a Director is in charge of only one project. They are given a number of resources, covering project management, development, testing and release.

On day 1, the team goes offsite and flushes out the concept of the project.

On day 2, the entire team flushes out storyboards for the design and user experience.

On day 3, the development starts. Everyone who can code does so, there are no more specific roles anymore. Those who can't code are given marketing and other deliverables; everyone is equal.

On day 4, the team rips apart the previous days work; whether it is code or a user interface design seeking ways to make more impactful.

On day 5,6,7,8 etc - new development occurs and as the project takes shape testing starts

Days 4-8 repeat for the length of the project. At periodic points the project is presented to senior management for review.

Do not get the above workflow confused with agile or scrum as it isn't.

The clear difference is ownership and innovation,they are built into the process, everyone is invested in the final outcome as they provide creative and impactful contributions to the outcome.

It also pulls changes/enhancements/defects up to the front of the process reducing rework towards the end. It removes the known phases of the development lifecycles (requirements dev, test, etc).

Once the product is released the entire team moves onto the next project repeating the cycle above, building trust and synergies between the team members. For executives, the team is a known entity who delivers results.

The next obsolete site - CNNSI

I'm a news junkie, I collect my news from various sources on the web. When it comes to sports, I would visit cnnsi.com and pick up the headlines. That was then, now like other old media companies they haven't stayed ahead of the curve. Other sites feature live streaming comments from the Phoenix court about Balsillie and the NHL (thanks TSN), commenting on each article, and more up to the minute statuses for my fantasy football lineups. CNNSI sources their material from sources such as AP and STATS, providing only limited original material beyond editorials. What happened to the site that was cutting edge?

Not only have other sites such as TSN, Yahoo Sports, CBS Sports and ESPN passed by CNNSI in terms of up to date and immediate content, but the quality of the site has dropped substantially. I have featured the CNNSI site many times in the "Quality Issue of Day" and have started to wonder if they have anyone on staff that validates the quality of the pages as content is streamed from Banks, King, and Trotter. I would understand if the page is not exactly correct in the "low hanging fruit" of the browser world. But for it not to work in Internet Explorer 8? Come on, how many times will badly rendered pages go unnoticed before action is taken to prevent them in the first place?

I have to speculate that CNNSI is using a CMS (content management system) to push the editorials, headlines and new slide shows out to the web, so they must assume that it's just content and the structure of the page will be fine so why test it.

Here is the latest....

I can't blame CNNSI completely, especially when their parent company CNN is struggling in the new media realm. It was the leader in the field, broadcasting news in a 24 hour format long before anyone else (1992), now they struggle to create an identity in the new media universe. They spend most of their time, repeating tweets and comments posted by their viewers and their evenings are filled mostly with repeats of shows aired earlier in the day. Their Headlines News network now offers "News and Views" rather than the 24 hour news format that once was their specialty. Somewhere along the line, they lost their vision that "a viewer could tune in at any time and, in just 30 minutes, receive the most popular national and international stories"; now they only air news during the day (when no one is watching?).

According to Hitwise data in July 2009, CNNSI didn't even register in the listing of top ten sports sites. Until CNNSI steps up and reestablishes itself as a leader in sports arena in both quality and content, count me out.

Quality Assurance

Quality Assurance is a term dropped just about everywhere in the software field now a days. While there are companies that have strong quality assurance departments built upon frameworks that allow for success, other organizations brand basic testing teams as quality assurance but they are often reduced to blindly following step-by-step test plans with no ability to influence quality issues at a corporate level or they have no testing teams what so ever.

Let’s get this out of the way right now, quality assurance is not testing, testing is inherent in quality assurance. Somewhere along the way, this inaccurate perception skewed how QA is perceived and implemented in organizations. Quality Assurance reflects an ongoing commitment to the prevention of quality issues throughout the software development life cycle and beyond, while testing is independent of any process and is solely the examination or evaluation of the degree to which software satisfies the specified requirements after the completion of development. Testing is part of the work a quality assurance team performs.

What is Quality, Anyways?

Quality Assurance is built upon the foundation of ensuring the quality of an organization’s application or service, internal processes and resources. Great, that sounds simple enough, it’s all about quality, but what is quality anyways?

Quality is all about perception. Each individual user has their own definition of quality and they will apply it to your software or service when using it. QA must speak for the quality of the application on behalf of each and every user during the software development lifecycle. Each employee in your organization will have their own definition of quality. What’s yours?

I left you some blank space below to write it down.
------------------------------------------------------------------------------------------------------



------------------------------------------------------------------------------------------------------

Hard to do wasn’t it?

“Quality is hard to define; impossible to measure, but immediately recognized if it is missing.”

Quality Assurance

A survey of three hundred U.S. and European senior information technology executives from large companies showed that eighty five percent of IT executives interviewed indicated “application quality is either critical or very critical to their overall effectiveness in demonstrating business value.”

In another survey of more than 150 software development organizations found that thirty eight percent of developers said their companies do not have an adequate software quality assurance programs and thirty-one percent said their companies had no quality assurance personnel at all.

With such an emphasis placed on quality being so critical to the success of the business, why is quality assurance not emphasized in an organization? In fact, the perception among most developers is that most senior managers are satisfied with the quality of software that their companies are producing. When is the last time you have heard from a developer than the quality of the application they work on is poor, that the code they developed has defects and other issues? Never – why would they admit that their code is not working and what they developed is of poor quality. The perception is ingrained that any negative reflection of the application quality reflects on their performance as a developer rather than a means to strengthen overall product quality. This inherently sets the course where development and quality assurance are on opposite sides of the fence. Finding and resolving defects are ok but question the quality of the underlying application code and up go the defenses. Take the performance testing of a web application, the development team will try and look for every hole in how the test was executed to invalidate the results.

There are ingrained biases surrounding quality assurance. This bias extends from developers through to senior management. Everyone thinks QA is easy, so less-skilled people are hired off the street (with no training) to fill QA positions at much lower salaries. Management can and has demoted underperforming developers into QA.

What is Software Quality Assurance?

All of the quality assurance books on the market today, focus on software quality assurance and provide similar definitions for it. D. Galin wrote software quality assurance is: “A systematic, planned set of actions necessary to provide adequate confidence that the software development process or the maintenance process of a software system product conforms to established functional technical requirements as well as with the managerial requirements of keeping the schedule and operating within the budgetary confines.” The definition seems to leave out critical parts. What about social engineering, user behavior, usability, client expectations and process improvement? As of now, the old definitions no longer exist; I’m throwing them out and writing one that makes sense. You cannot address software quality assurance without dealing with social engineering, human behavior and the processes that developed the software. I consider it to be a vital concept in quality assurance and I will go door to door if I have to, but I’m gonna convince you I’m right.

Without the Why

Quality Assurance has been hopelessly stalled for the last 20 years. Moribund. Ossified. There is a very simple reason for this catastrophic and intractable state we are in. People think they know what QA is and stopped asking why along time ago. Without the why, quality becomes theory at best, not practiced or even of critical concern. No process or methodology, regardless of complexity is going to tell us why quality assurance has stayed the way it is. When the why’s remain unanswered, there is no understanding. When there is no understanding there is no progress. When there is no progress there is no evolution. Without evolution, things just fade away. Welcome to QA.

Quality Issue of the Day - August 17, 2009

The following issue was noted on the Microsoft Visual Basic registration page. The blue text in the image informs the user to "check the first checkbox, above to recieve important information...". On the page, the first checkbox on the page exists below the message.

A Year without the Yankees

If you are a baseball fan, there is a chance you might know the answer to this little piece of trivia. When was the last time the New York Yankees failed to reach the playoffs? The answer, 1993; the same year the Toronto Blue Jays won their second consecutive World Series. There were no wild card teams or Tampa Bay Devil Rays, the Florida Marlins were in their first year of existence and Jim Abbott (born without a right hand) pitched a no-hitter against the Indians.

In 1993, Bill Clinton succeeded George H. Bush as the President of the United States, Unforgiven won best picture at the Academy Awards and the Montreal Canadians won their 24th Stanley Cup beating the Los Angeles Kings in the finals and Seinfeld was TV royalty. On the computer and technology front, it was a year that started the evolution of our lives, The World Wide Web is released by CERN (European Organization for Nuclear Research), Windows NT 3.1, the first version of Microsoft's line of Windows NT operating systems, is released to manufacturing and id Software releases Doom, a first-person shooter that uses advanced 3D graphics for computer games setting the standard for future games.

On the Quality Assurance front, Mercury Interactive (bought by HP) had shipped the first versions of their products two years earlier and wouldn’t hit stride financially for another 6 years. WinRunner had been released, Test Director (now Quality Center) and Load Runner existed in only their first versions. There was no thoughts of any automated web testing tools as the Internet was just released (the first versions were released in 1996). For QA process and methodology, “Testing Computer Software” by C. Kaner, J Falk and H. Nguven was republished as a 2nd edition (verbatim of the original 1998 version). This book was the standard reference for the testing field and included topics on test case design, test planning, project life cycle overview, software errors, boundary conditions, bug reports, regression testing, black box testing, software quality and reliability, managing test teams, printer testing, internationalization, and managing legal risk.

In explicitly, 16 years later this classic book is still used as a reference guide (I have seen it used by non QA resources and executives as a reference and a guide for QA), even though it is out of date and out of print. It uses examples of MS-DOS and testing dot-matrix printers. It even advocates a “wait and see” approach to the “Microsoft Test” application and the “fad” of automated testing. In fact, automated testing is now a staple of any testing strategy.
For all the gains in technology, process, methodology, the changes in types of testing, and use of automated testing, QA is hopelessly deadlocked and stalled because of this standard reference. The technology field has relied so heavily on this book that there is no longer any evolution and development teams have assumed a dominant role in IT organizations, a la Darwin. Books like "What would Google Do?" are swooped off of shelves and read by product managers, search and advertising executives, human resources and of course engineering teams. These books evangelize changes in how to do business in the new media world, yet there is not a matching book for Quality {QA} in the new world. QA needs a new reference, a new standard, a game changer and I'll write it if I have to.

Quality Issue of the Day - August 15th, 2009

The following screenshot is from Facebook's Music/Youtube share panel. "To attach, select one of the songs below and click Post" is the instruction to the user, however there is no Post button.