Agile Intelligence: Continuous Integration

The philosophy of continuous integration was first brought in the process of Extreme Programming. Subsequently, it been successfully deployed in several Agile development projects. Continuous Integration is a practice where the development team members integrate their work products frequently. This is more of a learned wisdom over the years than something that got developed in a lab or meeting room. Over the years, the software projects started with big promises and landed into endless integration phases with tons of rambles around. Continuous Integration aims to manage the integration mess in a better way.

Traditional development models used to keep the integration at the end of all development work. Quite often such approach turned into the eleventh hour rush of doing everything, and in many cases redoing the development of some of the modules/functionalities. The project managers were scared to death since after all those fancy progress reporting over months they were handed over with a reality of cost and schedule overrun. No matter how well and detailed the planning was, the integration phase was never smooth tide. Rather, those unfortunate fellows (who could not leave the scene before integration) ended the project with experience which they would remember in gory details for a long period.

To tackle this problem, Continuous Integration started with a hard frequency of one day. In other words, the early proponents of continuous integration aimed to make it a daily integration. Everyday, the programmers would contribute their day’s work to the shared location. At night, or on the next morning, one volunteer would take all those codes together and build those into a single application. After that happens, it would remain open to all for viewing. This would help the individual programmer to see the behavior of his/her codes at the integrated environment. This would also help the programmer to fix some (if not all) of the broken elements into his next day’s work. As a result, the next day’s contribution of the programmer would have some new developed codes as well as some fixes on the previously developed codes. Subsequent build would be expected to have fixes for some of the broken elements, and as the days progress the software application would be getting more refined collectively.

The benefit of such a practice is manifold. It provides a tangible work product for showcasing no matter how messy that could be. The project managers can breathe easy since they now can see what’s going on, as well as a big risk of software integration failures are getting mitigated in steps. The other stakeholders who are not involved with the technical stuff would also get an opportunity to see what’s getting cooked in the kitchen. The programmers also get the benefit to self-correct their own mistakes which could have become a Herculean job if accumulated till the end. It also helps sustaining the morale of the team since a lot of eleventh hour bickering can be avoided.

Continuous Integration comes with its own limitations. Since everything in a software engineering project cannot be related to the programming only, it leaves a good part of the team out of the integration practice. Project Managers practicing agile processes should do a careful planning around them to ensure that their work products too move along harmoniously.

Also, the build process can be long and complex depending upon the size and complexity of the software application getting developed. Daily Integration would require some specialized full time staff allocation to do this job, thus raising the cost of project execution. For such cases, the frequency should be rightly defined. If you are a consultant working for a client, you may be asked to prepare daily builds since the client has read it through some magazines on Agile. If you think the software under consideration is large and complex, it may be a good idea to explain the cost impact of daily builds to the client. Agile does not address the project economics well (we will talk about it later) so it’s your responsibility to bring that under client’s radar for a better cause. For most of the software applications, a twice weekly frequency (Monday morning, and Thursday morning) should be enough. For large and complex applications, take it to a weekly frequency (Friday night, or Monday morning).

Introducing Continuous Integration could be a challenge if the integration process eats up a lot of energy and enthusiasm of the team members. The Agile practitioners use and recommend using the process improvement tools for improving the situation. Maintaining a shareable code repository would be one such step. Nowadays, having source management software like PVCS, or VSS are pretty common, so ensure that it’s in place for your project too. The other important improvement would come through automating the build process. The code compilation, build preparation, and deployment should be automated using scripts and templates to the maximum extent possible; this would provide significant incentives for increasing the frequency of integration during the course of development.

Agile Intelligence: Sprint

Those who have already read through the pages of Agile Alliance and Scrum Alliance should have fair familiarity of the cycle of software engineering by now. The diagram is epic, and is widely used anywhere when there is a discussion on Agile Software development. It has been hard, but somehow I resisted myself to upload the same diagram here, but the diagram is still important for everybody to understand the core purpose of being agile.

Those who are already familiar to the diagram should be aware of the term “sprint” as well. For the others, the definition goes like this: Sprint is a time period within which the team completes the development of the backlogs committed by them at the beginning.

Those who are well aware of the software development life cycle should know that the life cycle does not contain the “development” alone. To give a software application a complete shape, development should be preceded by the analysis and design; and should be followed by various flavors of testing, bug fixing, and deployment. The proponents of Agile took a conscious attempt to fuse all these activities into a continuous set of activities, and hence they used the word “development” only to describe the activities within a sprint. However, the sprint activities does not remain confined within development only, it does have analysis, design, and testing.

Those who are well experienced with the commonly used software engineering models like waterfall or iterative should know that the duration of a phase is driven by several constraints like time to market, availability of resources, etc. Sprint, on the other hand, is fixed with duration. Its duration is 30 days. Agile practitioners around the world follow the same guidelines, albeit with some minor changes to fit their purpose. In most of the cases, the sprint starts on the first working day of the calendar month, and ends on the last working day of the same calendar months. Some take it to 30 working days, thus making a 6-week schedule corresponding to every start date. Although, people stretching the duration beyond 4/5 weeks are quite rare, and those who do so, have mastered the software engineering processes before getting into Agile. Most of the Agile development teams are charged up to show something been accomplished at the end of the sprint, and sooner the better. So it’s a self imposed restriction towards stretching or postponing the deadlines, everybody around an Agile development project know that something is getting done at the end of the sprint, which in turn is not far away on the calendar.

A software development project may have many sprints. Since the sprint duration is fixed, depending upon the size and complexity of the software application, the number of sprints varies. The sprints must be sequential, i.e., end of one sprint can only trigger the start of the next sprint. While the same team is expected to continue on the following sprints as well, the Agile methodology does not put any restriction on this area. In fact, at the end of each sprint, the product owner and the sprint master should take a joint review of the resource requirements on the following sprint(s) depending upon how much of the product backlogs should be converted into the sprint backlogs in a single sprint. End of sprint review would also help the sprint master to re-balance the team with the individual team members (e.g. somebody fell sick in the middle, or expected to go on a vacation on the following month, etc.).

Agile Intelligence: Backlogs

The Agile development process talks about two types of backlogs: Product Backlogs, and Sprint Backlogs. We will talk about the specifics a little later; let’s try to understand the fundamental constitution of backlogs first. Backlogs, unlike the word suggests is not about falling behind. It’s about maintaining a to-do list. A backlog is a master list of all functionalities desired in a product (software application).

The proponents of Agile development are of opinion that the details specification of a software application are not known during the start of the project. So, the obvious requirements are listed down as the product backlog. As the project progresses, the requirements get more elaborated and refined, and the product backlog grows accordingly. While such a statement does not provide enough confidence at the investor level, but works pretty nicely to the people who work on the project.

The backlogs are documented in tabular format. Each backlog has a name (or short description) to it, and a priority mark (high/medium/low) with the respect to the critical success factor. The product owner takes the responsibility to maintain and update this backlog. The product owner gets the inputs from the various groups of potential users and beneficiaries of the software application, and converts those inputs into backlogs with due priority. The product backlog does not have any other project planning attributes attached to it.

The product backlog works as the input to the sprint backlog. A sprint backlog is the list of backlog items that the team commits to deliver in a particular sprint. In a way, sprint backlog is a sub-set of the product backlog. However, there may be additional backlogs into a sprint backlog which are normally the supporting work for the sprint. Sometimes, a product backlog is also broken into multiple sprint backlogs for better granularity and distribution of work. As example, the product backlog may specify that only the employees of the organization will be able to access the software application. At the sprint backlog, this may be broken down into the login pages, authentication module, and integration with the enterprise directory. Sprint backlogs are picked up by the team members, which is largely a self-assignment process. The sprint master keeps track of the assigned work through the scrums. Scrum also helps the sprint master to fill up the start and end dates of each sprint backlog.

The webmasters of the Agile Advice website has brought in the Queuing Theory to understand the backlogs better. The theory helps the mathematical analysis and optimization of the work packages flowing from starting point to the finish line. The theory denotes the entire work package completion path using the time interval of having the work packages for start, the time required to complete the work within each package, and the number of resources (in software engineering, it should be the number of team members) available for the work packages. My personal opinion is not to focus too much on the advanced techniques at the beginning. Just apply some common sense if you are a sprint master. In case you have to apply something fancy, first focus on the principles and best practices of software project planning, tracking, and supervision.

Agile Intelligence: Story

There have been multiple traces of origins to this terminology so far the Agile programming goes. By and large the proponents are indebted to the ad makers who traditionally prepare the storyboard while creating an advertisement. The concept of storyboard was eventually borrowed by many streams of content development including those done for digital consumption. Before getting into the definition of Story in Agile Programming, let’s dig out the definition of storyboard first: Storyboards are organizers to depict a series of illustrations or images that would get composed sequentially to display the purpose with better visualization. (Multiple Internet based sources)

Now let’s look into the definition of Story: A story is a high level depiction of the user requirements, which would provide just enough information to the programmer to estimate and implement the requirements. (The Agile Modeling website)

While the similarities are quite obvious, I would like to highlight the words “high level” in the definition of the stories in Agile Programming. Depending upon how the user perceives and composes the requirements, the stories can be of diverse shape and size. However, the objective here is not to write a requirements bible, but to communicate the needs or expectation from the software application. So, the user story may well start with a bulleted list of actions to show how the business process should be. Subsequently, each of the bullet points would be blown out to a set of user screens, business rules, data units, etc. In other words stories are expected to branch out as the requirements evolve, and so is the development work.

Being discreet in providing high-level and just enough information to the programmers would ensure that more iterations happen during the course of development, thus enabling the test driven development more rigorously. Let me explain, how it works. For sake of simplicity, let’s assume that the story under consideration talks about developing one screen. The story provides just enough information so that the developer would be able to develop the UI with all the input/output controls positioned correctly. Immediately after that, the programmer would need to know the input validations to be implemented. The story-writer could have given this in the first cut of the story itself. But there would be some limitations: a) the story-writer might not have thought of all the validations that would be required; b) the programmer would have to wait till the story gets completed with the description of validations. So, the first story came out without any validations requirements. After implementing this, the programmer went back to the story-writer (or user group, or product owner, whoever are in the loop) to show his work as well as to gather some more requirements (validations). Now the story-writer could see how the UI came up, and could provide some upfront review comments on it. At the same time, the story-writer now would be able to provide more crystallized information on what kind of validations would be best fit here. With this second story, the programmer goes back to work and implements the review feedback and the new set of requirements (validations). Then comes the requirements related to the more comprehensive business rules, like how the input data should be saved, how it should be deleted, browsed, and so on. The beauty of this iterative process is it keeps the programmer and the story-writer (normally the user representative) continuously engaged with the happenings around the software development. So the risk of expectation mismatch with the users is minimized to the lowest probability.

However, it comes with a pinch of salt, too. It sets the pre-condition of having the user groups continuously engaged with the programmers. Those who work in the IT consulting space know very well what I am talking about. It’s not easy to get such a high degree of time commitments from the user groups since they too have their daily chores to take care of. To them, contributing time in larger chunks with lesser frequency would be a preferred option than what I have been writing in the paragraphs above. Such a situation creates the weakest link of failure in the Agile programming, as the lack of frequent iterations would eventually result of delayed completion of tasks.

To those who are consultants, and are doing programming for your clients (external organizations), my suggestion would be take a judgment call here. Not too high-level, not too detail-oriented may turn out to be a critical success factor for most of you, specifically those who are competing for your clients’ time as users and trying to make a killing using Agile. Nevertheless, as Agile is gaining popularity, you would also see quite a few organizations are committing larger share of the users’ time during the development of the software application.

We will talk more about such trade-off later in the following posts. Now, I would like to move on to the topic of Backlogs. For those who would like to develop deeper understanding on Stories, may go through one of the early attempts to compile the subject within a book: User Stories Applied - For Agile Software Development (by Mike Cohn)