Friday, December 8, 2017

Continuous Integration: Let's Talk Turkey

Why practice Continuous Integration? What is the Goal? What is the Business goal behind improving engineering practices? What if we were to make this into a user story? What would it look like?  “As a business I want the ability to introduce features and functionality to my customers with very little risk, so that more people will buy and use our products without disruption.”

How would we accomplish this goal? You might start with a shift to agile. This would shorten the feedback loop between customers and the developers of features and products, thus ensuring you are giving them what they actually want. You might design your teams around products instead of subdividing the company by “IT” and “Business” and then further subdividing technical teams around technological silos and businesses around domain silos.

Those would be great steps to take. So how then do we deliver the product? We have product teams of comprised of business people and technical people that have all the knowledge needed to create the product. But how do we get it, and subsequent changes, to market in a world that wants everything right now, without exposing the company to the huge risks currently associated with big bang software releases? 

The silver bullet seems to be “CI/CD”. After all, it is on everybody’s lips. And they may be on to something. But do they understand it? Most people think CI/CD is a hardware/software pipeline where things get checked in to a repo and automagically "delivered/compiled" by Jenkins. But is it just software and hardware with some automation mixed in? When I ask people if they need help with engineering practices or with continuous integration, they often respond with:

“We are already doing it. We have Jenkins. We use git.”

“Hmmmm. So ‘having Jenkins’ and a repo means you are practicing Continuous Integration?”

“Well… we check into git and then Jenkins runs a build.”

“Ah! Cool! So all your code is integrated together and then you execute a test of all your functionality automatically through a test automation suite?”

“Well, no. Not all our functionality. Actually we don’t really have a lot of automated tests.”

“So you are just… verifying the compile??”

“Um… yeah.”

And there are other varied responses to this question. But the bottom line is that when we start probing we find that the practice of CI is not being followed. (And CD? They don’t really distinguish between the two.) So… They have the hardware. They have the software to kick off a build. But they are lacking in test automation or at least lacking in “good tests”.

assertNotNull(Object) does not qualify as a good test.

“Continuous integration is a practice–it is about what people do, not about what tools they use. As a project starts to scale, it is easy to be deceived into thinking that the team is practicing continuous integration just because all of the tools are set up and running. If developers do not have the discipline to integrate their changes on a regular basis or to maintain the integration environment in a good working order they are not practicing continuous integration.”

Giving CI to dev teams without teaching them how to practice it, is like giving a hammer to a six year old. Just because the six year old is able to lift the hammer, doesn’t mean he should. To use a hammer properly one must know its purpose, the right applications for its use, and develop the proper motor skills and coordination to use it effectively. Without that knowledge in place, things just get smashed.

CI is no different. It is a tool used by developers. Developers must know how to employ it as it is a practice. They must change the way they work and change their habits. This is hard enough to do for a small scrum team without the proper coaching. Attempting to implement it at scale without the proper coaching is pretty near impossible.

The six year old may be able to pick up the hammer, but that doesn’t make him or her a carpenter. It just makes them dangerous.

We can talk about CD on another day. I think we should concentrate on nailing CI first.  

Wednesday, October 25, 2017

Sprint Zero - some just can't let go of the Waterfall Mindset

Sprint Zero is NOT a part of Scrum. You will not find it in the Scrum Guide or any Scrum Certification Course. Sprint Zero is an anti-pattern.

So where did it come from? People that have little understanding of Scrum but LOTS of Waterfall experience must have come up with this term.

A “Sprint 0” clearly does not deliver a potentially releasable product. In fact it perpetuates the Waterfall mindset of Big Up Front Analysis and Planning. It is almost identical to what most of us remember as “gathering requirements” before a project.  I cannot emphasize enough how “anti-agile” this concept is.  Does that mean we don’t put together a Product Vision and a backlog before we start Sprinting? No. Of course not. But we do it BEFORE we start Sprinting. If we want to eliminate the waterfall mindset, we need to start by eliminating the concept of a “planning Sprint” or a Sprint 0. Plan all you want. Just don’t call it a “Sprint”. Clearly, it is not. Don’t just take my word for it, you can consult Mike Cohn, the Scrum Alliance,  Alex Ballarin Latre, and the Scrum Guide.

The heart of Scrum is a Sprint, a time-box of one month or less during which a 'Done', usable, and potentially releasable product Increment is created.” - Scrum Guide 

A Sprint 0 is the name often given to a short effort to create a vision and a rough product backlog which allows creating an estimation of a product release… that activity does not meet the definition of a Sprint in Scrum, so it is better not to call it so.” - Alex Ballarin Latre

“Sprint zero is usually claimed as necessary because there are things that need to be done before a Scrum project can start. For example, a team needs to be assembled. That may involve hiring or moving people onto the project. Sometimes there is hardware to acquire or at least set up. Many projects argue for the need to write an initial product backlog (even if just at a high level) during a sprint zero.
One of the biggest problems with having a sprint zero is that it establishes a precedent that there are certain sprints or sprint types that have unique rules. Teams doing a sprint zero, for example, will dispense with the idea of having something potentially shippable at the end of that sprint. How can they have something potentially shippable after all if the goal of the sprint is to assemble the team that will develop the product?” – Mike Cohn,

“[Reason for Sprint 0] Consider planning, product backlog setup, and design. This is my favorite, as it clearly indicates two things:
  1. Someone is finding it hard to move away from Waterfall.
  2. The organization is finding it difficult to give up control and empower the team.

Introducing a long Sprint Zero at the start and a long validation at the end helps keep the love for Waterfall alive. In addition, it hinders the effectiveness of Scrum.
If Waterfall works for you, then by all means, you should use Waterfall. This article focuses on teams that follow the Scrum rituals but keep the Waterfall soul…

Consider planning, product backlog setup, and design is a proxy for "big design up front" (BDF). In Agile we try to stay away from that. Often, an associated symptom will be a very hierarchical organization, thus preventing the Scrum team from being really empowered.
Personnel who are most active in Sprint Zero may be different from those most active in other sprints.
The Sprint Zero leaders will "guide" the team through sprints but may not feel bound by the time and other constraints that the rest of the team has.
This creates a two-level organization instead of the flat one that Scrum promotes. A two-level organization tends to follow a task-assignment and status-reporting culture, instead of a mind-set of ownership and delivering value.
Scrum believes that every sprint should deliver potentially usable value. None of the above Sprint Zero items produce any value for the customer. They cannot be demoed.

In other words, there is no valid reason for a Scrum Team to have a Sprint 0. It is for people who cannot let go of the Waterfall mentality and/or do not fully understand Scrum. As a coach that has been practicing Scrum since 2004, I can tell you there is no need – and plenty of reasons against – creating a “Sprint 0” and calling it Scrum.

Thursday, October 19, 2017

... You Just Might Be Waterfall

  • If you have a "Sprint Zero" that lasts 12 weeks to get a "jump on planning"... You just might be Waterfall.
  • If you have a test team that works a "sprint behind"...
  • If you have a test team that performs all of your regression tests. Manually...
  • If you have an "Agile Project Manager" who drives you to "increase your velocity"...
  • If you have a partially allocated resource on your team...
  • If you have a Technical Lead on your team who is responsible for all decisions regarding software...
  • If your team and the people around your team often use the phrases "but we’re not true agile" or "but we’re not pure agile"...
  • If you plan on completing System, Integration, and Performance Testing in your last four Sprints...
  • If your leadership complains that they need more Status Reporting and they have to know what the "Gap to Green" is...
  • If your daily standup consists of going through all the tickets on your board and giving a progress report on each of them...
  • If your Scrum Master has more than three letters behind his or her name (SPC4, CSP, ICP-ACC, SA4, CSPO, PMPO, PSPO-I, CSM, SASM, PSM-I, PMI-ACP, SSM, TKP, PMP)...
  • If your Release Plan looks like this: Sprint 1-4 Analysis, Sprint 5-8 Design, Sprint 9-15 Coding, Sprint 16 and 17 Test... You just might be Waterfall.
  • If you have a pre-planning meeting to prepare for your planning meeting...  
  • If at any time, anyone ever utters the phrase "steering committee"...  
  • If your teams spends a “Sprint” on estimation...
  • If your “Agile Project Manager” tells you that your estimation (which you wasted a whole "Sprint" developing) “won’t work”...
  • If your dev team members tell you that their stories are done “except for JUNITs”...
  • If your dev team members feel that assertNotNull(someObject) is a comprehensive test...
  • If your testers “have nothing to do” because they are “waiting for the coders to finish” ...
  • If your Product Backlog consists of a long list of technical tasks...
  • If your Product Owner insists on the team performing Release Planning every two weeks...
  • If your “Agile Manager” starts to “grade” individual team members on the number of stories they finish each Sprint...
  • *If your stakeholders start deriding you as a “purist” because you insist they AND the Product Owner attend the Sprint Review...
  • If your PMO…  Wait? You STILL have a PMO..?  You just might be waterfall.
  • But seriously, If your PMO starts looking for “metrics like velocity” to measure teams’ success...
  • If your Enterprise declares that all story points must be normalized... You just might be Waterfall. Or SAFe Certified. (But what’s the difference, really?)
  • Finally, if you think that developing a Big up Front Design, Fixing Scope, Fixing Delivery Date, AND Fixing Dollar cost, to create a software application built by partially allocated resources conducting a series of hand-offs to different teams comprised of geographically dispersed contractors across the globe, and then throwing it all together at the end and believing that it will all work, (pause to breathe here) is a better way of building software... You just might be Waterfall. And live in an alternate reality.

Wednesday, March 5, 2014

Why Adopt Agile?

On a recent discussion board, a fellow LinkedIn member asked the question,
"Why Adopt Agile?" 

Here's his complete question:

"Hey all,
I figure we often get asked the question by executives and engineering teams "Why should we adopt agile?" I wanted to better articulate my argument to my team, and hope that a few of you may bring insight into reasons to adopt Agile that I haven't thought of before."

Since I believe he is probably not the only one with this question, I have decided to share my response here:

Communication.  Communication. Communication.  Everything we do boils down to communicating. In Software development we get requirements from the business and attempt to turn them into software.  In the waterfall world you gather all the requirements in front with the assumption that all the requirements can be gathered and they won't change over time.  If you can get a comprehensive list of requirements and spend enough time up front figuring out how you will build it, if the business approves the design, you'll be set.  The problem is those assumptions are simply wrong.  Requirements will change. The team’s understanding of requirements will change. Problems that could not be foreseen in development will arise and they will impact the design.  These are the main reasons Waterfall projects fail.  Sure they try to manage things like scope and make it very hard to change scope. But if you don't allow the scope to change, you will deliver a product that is no longer what the business wants or needs.  On the other hand we all know that allowing scope to change in a world of Big Up Front Design (BUFD) is extremely expensive.

Agile addresses these issues through effective and frequent communication with the product owners (the business). Short Sprints give short feedback loops.  Constant discussions about changing priorities and requirements can be addressed, entered into a backlog, and tackled with very low cost in the very next Sprint.  Requirements are gathered via user stories.  The reason for having user stories is to facilitate discussions between the developers and the business about what it is they think they need. The requirement itself is captured on a 3x5 card that can be discarded on a moment’s notice if a "story" (requirement) changes.  At the end of a Sprint, the business can see functioning software and can tell immediately if what was delivered by the team, was the thing they intended to be built or something different. Feedback is shared and corrections can be applied in the next Sprint. 

Agile does all the same things as waterfall: requirements gathering, design and release planning, construction and testing. But it mitigates risk in several ways.  First, it shortens the cycle for feedback to the length of your Sprint (usually 2-3 weeks, instead of months) Second, it involves the business intimately with the design, development, and construction through each process rather than at the end of each process making it easier to accommodate change. Third, it delivers function software at the end of each Sprint.  So the business has something that works.  If you halt a waterfall project before the construction phase is completed, you have piles of documentation that can't DO anything. 

Agile is the way to go because it minimizes risk to the business by emphasizing communication and constant feedback while delivering functional software.  This is something the business cannot duplicate with weekly Status Reports.

Agile software development eliminates the unnecessary artifacts that are often required to give some architect or business person a warm fuzzy and replaces it with intimate involvement in the project.  It moves quickly and embraces change. 

In a business world that has become almost wholly dependent on technology and technology that changes at incredible speeds, Agile is simply the safest thing for a business to do.

Monday, February 3, 2014

"The environment that nurtures creative programmers kills management and marketing types - and vice versa."

I saw this while reading another blog: Fuzz Box.  So good I had to repost it:

"The environment that nurtures creative programmers kills management and marketing types - and vice versa. Programming is the Great Game. It consumes you, body and soul. When you're caught up in it, nothing else matters. When you emerge into daylight, you might well discover that you're a hundred pounds overweight, your underwear is older than the average first grader, and judging from the number of pizza boxes lying around, it must be spring already. But you don't care, because your program runs, and the code is fast and clever and tight. You won. You're aware that some people think you're a nerd. So what? They're not players. They've never jousted with Windows or gone hand to hand with DOS. To them C++ is a decent grade, almost a B - not a language. They barely exist. Like soldiers or artists, you don't care about the opinions of civilians. You're building something intricate and fine. They'll never understand it.


Here's the secret that every successful software company is based on: You can domesticate programmers the way beekeepers tame bees. You can't exactly communicate with them, but you can get them to swarm in one place and when they're not looking, you can carry off the honey. You keep these bees from stinging by paying them money. More money than they know what to do with. But that's less than you might think. You see, all these programmers keep hearing their parents' voices in their heads saying "When are you going to join the real world?" All you have to pay them is enough money that they can answer (also in their heads) "Geez, Dad, I'm making more than you." On average, this is cheap. And you get them to stay in the hive by giving them other coders to swarm with. The only person whose praise matters is another programmer. Less-talented programmers will idolize them; evenly matched ones will challenge and goad one another; and if you want to get a good swarm, you make sure that you have at least one certified genius coder that they can all look up to, even if he glances at other people's code only long enough to sneer at it. He's a Player, thinks the junior programmer. He looked at my code. That is enough. If a software company provides such a hive, the coders will give up sleep, love, health, and clean laundry, while the company keeps the bulk of the money.


Here's the problem that ends up killing company after company. All successful software companies had, as their dominant personality, a leader who nurtured programmers. But no company can keep such a leader forever. Either he cashes out, or he brings in management types who end up driving him out, or he changes and becomes a management type himself. One way or another, marketers get control. But...control of what? Instead of finding assembly lines of productive workers, they quickly discover that their product is produced by utterly unpredictable, uncooperative, disobedient, and worst of all, unattractive people who resist all attempts at management. Put them on a time clock, dress them in suits, and they become sullen and start sabotaging the product. Worst of all, you can sense that they are making fun of you with every word they say.


The shock is greater for the coder, though. He suddenly finds that alien creatures control his life. Meetings, Schedules, Reports. And now someone demands that he PLAN all his programming and then stick to the plan, never improving, never tweaking, and never, never touching some other team's code. The lousy young programmer who once worshiped him is now his tyrannical boss, a position he got because he played golf with some sphincter in a suit. The hive has been ruined. The best coders leave. And the marketers, comfortable now because they're surrounded by power neckties and they have things under control, are baffled that each new iteration of their software loses market share as the code bloats and the bugs proliferate. Got to get some better packaging. Yeah, that's it."
- by Orson Scott Card

Wednesday, January 29, 2014

Building Software with Al Capone: It's just not Agile.

In 2000-2002 I worked for a company in the Chicago area with a call center in Phoenix. They sent me out to Phoenix twice for training and to work with our Call Center Reps on the software I was developing.  Each time I stayed over the weekend to golf so I was around from Friday until Sunday both times.  I bring this up because in Chicago we have two seasons: winter and construction.  But construction also takes place in Phoenix.  It's just unobtrusive and efficient. At least it was back then. 
The difference between Agile and Waterfall Development is very similar to the difference in approaches to road construction in Phoenix vs. Chicago.

The Chicago Way

In Chicago, huge plans are drawn up years in advance. Billions in capital are approved and funded. The construction project then starts by blocking off the lanes and tearing up everything for the length of the projects.  Traffic patterns are changed for months.  Temp lanes opened, lanes adjacent to construction shut down and people traveling in both directions are subject to the torture that is Construction Season.  For 6,9, 12 months or more, the entire project under construction is ripped up.  At least the Phase 1 of construction.  Usually multiple phases are involved. Each phase taking a year or so. 

All planned up front.

The end result is that by the time the project is actually finished (usually 4-5 years later) the population density has changed as well the traffic patterns and the project that was planned no longer meets the current needs. So they start the whole process over again.  They never catch up to current needs.  And we live under perpetual construction.  First get the guys to tear up the road.  All of it. Then a few months later bring in the guys to lay dirt and gravel.  Then the guys come to flatten it.  Next they lay down the asphalt.  Finally at the end of the project the guys come by and lay down the paint on top of it all.  The funding needed for the project is in the billions.  Cost overruns abound. And whatever is delivered is already obsolete.  Nice.

Contrast that with...

What I saw in Phoenix:

On Friday night after rush hour a crew comes out and blocks off the highway between two entrances and exits.  Nothing else is touched.  Traffic over the weekend is rerouted off the highway and allowed back on at the next entrance.  A crew tears up a piece of road from between exits on Friday night.  Saturday during the day they lay down the foundation of the dirt and gravel and flatten it.  Saturday night they lay the asphalt and flatten it again.  Sunday during the day they add more and flatten it again and let it set up.  Sunday evening they apply paint, clean up their gear, and let the paint dry.  They are gone before Monday morning rush hour.  Exits opened.  New road laid down.  Finished.  The Following Friday they move down to the next exit/entrance and repeat the process.  Instead of billions of dollars being tied up into the project they have hundreds of thousands.  If anything comes up during the reconstruction of the road it can be halted at any time without impacting travel. 
See the difference? At the end of their weekend sprint, the Phoenix team has delivered a completely functional piece of road.  Sound familiar?  If priorities change, they can simply stop moving forward without any lingering issues or unfinished business with which to deal.  Risks are greatly mitigated.  Now imagine if a highway project in Chicago were cancelled in the middle, the workers just getting up and walking away?  What kind of mess would be left behind and headaches?  None of that is a problem if Phoenix decides to halt the project.  The risk in their approach is incredibly small compared to the Chicago Way.  Fewer people are inconvenienced and only for a very short time – and never during the work week when those road are most critical.  There is less road rage, less stress.  And at the end of every weekend, a functioning piece of newly repaved road is available to the end users.  See where I’m going?

Shops that are using waterfall are trying to build applications the Chicago Way.  The only one who wins are developers and contractors. And the politicians they pay for.  That is the Chicago Way.  Commuters, motorists, taxpayers, businesses, deliverymen,  basically everyone else not connected, loses.  And even when the project ends they still don’t win because the requirements have changed.  We need to scrap that approach. We need to bring in Sean Connery and change the way we build software or we will fail much more often than we win.