Shachins Blog

More and more companies are trying to offshore software development to countries in Asia and Eastern Europe. By and large the motivation for these efforts has been to reduce the cost of software development, which is facilitated by the low cost of software professionals in these countries. The mean base salary for a typical software professional with 2-4 years of experience in US (Dallas area) would be around $69K (source salary.com). The fully loaded salary is around $100K once we add the overheads like taxes, health benefits, bonus, vacations, office & equipment costs, etc. Contrast it with the hourly rate of $8 to $25 for the offshore software professionals and one can easily visualize the savings. However, only considering the hourly rate of individual offshore software professionals is a huge mistake. There are several hidden costs associated with this operation:

Client Management Cost

An outsourcing operation requires close and proactive supervision to make sure that there is no communication gap between the client and the offshore team and the offshore team is continuing on the right path. Since the offshore team does not have the advantage of ‘picking up the hallway conversations’, it is all too easy for them to continue with incorrect assumptions. There are three main models to manage the offshore teams:

1. The client directly manages the offshore team. This is one of the more expensive options because a senior member of the client team will be responsible for this supervision. This person will not only have to spend significant time with the offshore team, he /she will also need to be working at odd hours – resulting in the loss of productivity during regular hours. The above is true even if the client only needs to communicate with the offshore team lead.
2. The vendor places offshore team manager at client’s premises. This option is much better than the client managing the offshore team directly. This may also be the most cost-effective option with large offshore teams (5+ offshore members). However, this still suffers from two drawbacks. An on-site manager is like an extra employee – more expensive and with many of the overheads associated with an employee. Moreover, the on-site manager is typically a low to medium level professional with little US experience. The client still needs to invest significant time and effort to make sure that the ‘supervisor’ understands the project and stays on-track.
3. The vendor places a highly skilled US trained professional at client’s disposal. This professional is responsible for all communications with the client’s team and takes overall responsibility for the project success. He or she would spend as much time as required on client’s premises, even full-time for large projects. Though these professionals tend to be very expensive, the total management cost may still be lower.

Offshore Team Training Cost

We can assume that a good offshore team will be already trained in the common technologies (e.g., C#, J2EE, WebSphere, Linux, Oracle, etc.) required for the client’s project. However, there may still be a lot of areas where the offshore team will have training requirements. These include:

1. Less common or cutting edge technologies
2. Client’s development environment
3. Client application architecture, design, tools and API
4. Client specific domain knowledge

The above training is required even for the client’s employees. However, starting an offshore team is like opening an office in a new location. Typically one or more experiences people (trainers) are sent to the new location to manage the training of the new hires. Similarly, optimal training is achieved when client sends senior members of its team to offshore location for extended time. This, however, can be extremely expensive and only feasible for very large projects. Another possibility is to call senior members of the offshore team to US and train them.

Offshore Team Attrition Cost

As offshore development is becoming popular, the demand for trained offshore professionals is also increasing, resulting in heavy attritions (by some statistics as much 20% in India). This results in addition costs, including hiring and training new resources. Even though the vendor takes care of the hiring costs, the new employee training costs are typically born by the client. Worse still, it has the potential of delaying the project resulting in huge opportunity costs.

Some offshore vendors maintain ‘shadow’ resources to counter the attrition related issues. They assign additional resources to the offshore team to work alongside the team and get trained. If and when a team member leaves, these shadow resources – who are already trained – take their place. However, the client will need to pay for the additional resources. Even though the shadow resources are not explicitly billed to the client, their cost is spread across the rest of the team.

Offshore Team Lack of Productivity Cost

It is a myth that the countries like India has no dearth of skilled software professionals. Highly skilled and experienced professionals are very hard to come by. However, there are a lot of fresh graduates with programming knowledge. The skill level of these individuals vary a lot. With this reality, the offshore teams tend to have a pyramid structure. The apex consists of one or two experienced professionals and the bottom consists of 5x fresh graduates. The middle is filled with software engineers with 1-3 years of experience. This pyramid structure keeps the costs low. However, considering that numerous studies have found 10:1 difference in productivity and quality among the individuals; the productivity of the team as a whole suffers.

Conclusion

When thinking about the offshore development, one should not only consider the low rates of the offshore software professionals, he/she should also be aware about many of the hidden costs. Moreover, the economics of offshore development for large projects is very different than the smaller projects. I am definitely not saying that offshore development is not a viable option, but one needs to carefully choose the correct model and the correct offshore partner in order to derive maximum benefits from this endeavor. Otherwise, this may turn out to be a very unpleasant experience with disastrous consequences.

In my earlier posts I talked about the history of outsourcing and how commoditizing software development and offshoring go hand in hand. So if outsourcing (and offshoring in particular) is beneficial to company bottom line, the next question is – “Will it work for me?”. I will try to document some guidelines that will help answer this question.

Type of Development

The offshore development is best suited for the projects that need the vendor to only have technology skills. If the vendor also needs to acquire the client specific domain knowledge, this engagement can become very difficult. Not only the client will need to train the vendor into its domain knowledge, the whole process may need to be repeated as the vendor team members inevitably leave for greener pastures. I am not saying this training and attrition cannot be managed. In fact several offshore vendors are managing these issues very well. However, this quickly becomes expensive for the client.

Having said the above, I strongly believe that every project can be divided into (generic) technology specific and (client) domain specific parts. In fact, most of the projects have 70%-90% technology specific parts. As long as we can partition the project tasks and the technology specific tasks are a significant majority, the project is a good candidate for outsourcing.

Changes in Functional Specifications

I have yet to work for a small or large company where the functional specifications do not change over the duration of the project. In fact, if the specifications do not change, maybe we are working in a dead area anyway:) However, every time the specs change, the offshore team will need to be informed and they will need to change course. Although the same thing is also true with onshore teams, the cost of changing the course the much more with the offshore teams because of the distance, time and cultural differences and scope for miscommunication.

One approach is to initially outsource the parts that do not change with the functional specs. These parts typically are technology specific and include functionality like device drivers, protocol interfaces, framework pieces, utilities and tools, etc. As functional specifications become more stable, more parts may be outsourced.

Management of Offshore Team

Managing an offshore team is no small task. I don’t believe that the offshore team members are incompetent. However, there are huge cultural differences. The client project manager needs to understand these differences and adjust for those. Otherwise there is huge scope for miscommunication and eventual disaster.

Moreover, there is typically a significant time difference between the two teams (onshore & offshore). In some ways you can turn this time difference to your advantage by planning for (almost) 24 hour work day. However, the project manager needs to work in all sorts of odd hours to interact with the offshore team.

Planning and Discipline

Working with offshore teams require significant planning and discipline. The project manager has to divide the project in smaller parts, determine the interdependencies, assign these parts to different team, and then monitor the progress very carefully. The project manager should also get into the habit of documenting the communication with the offshore team. This is extremely important to eliminate misunderstanding.

Trust

One critical ingredient for success is very often missing. The client needs to trust the offshore team. It should realize that both onshore as well as offshore team members sometimes miss the deliverables, don’t understand the requirements completely, and have bugs. There is typically a discomfort simply because the offshore team is not onsite and cannot be monitored all the time. Whereas I advocate close monitoring of the offshore (and onshore) teams, the client project manager should cut the offshore team some slack as well.

A Strategy for Successful Outsourcing

I would like to summarize the strategy for a successful outsourcing experience as:

1. Divide the project into (generic) technology specific and (client) domain specific parts
2. Identify the parts that are fairly independent of changes in the functional specifications and assign those to the offshore teams
3. Find a project manager who understands the cultural differences and account for those in communicating with the offshore teams
4. Document all communications with offshore team
5. Monitor progress very carefully
6. Trust your offshore team

Partnering with Onshore Outsourcers

Another strategy is to partner with onshore outsourcers who manage the offshore teams for the client. Essentially they provide the client with a project manager who understands outsourcing nuances. I advocate finding a highly skilled technology savvy project manager who can help the client divide the project into parts and schedule the development. These outsourcing companies typically have higher hourly rates but the total cost of the project may yet be less once the cost of client’s project manager and other overheads is factored in.

Software Development – Art or Engineering

There are many of us who consider software development an art form. Like an artist, we dream up the design of a software system and keep changing it until it becomes perfect. A well designed software system indeed has a beautiful and elegant design. Then the (software) artist moves on to coding. A well written code definitely stands out. A friend of mine used to say that the code should read like a novel. Not only a well designed system or well written code is easier to understand, it is easier to maintain and change as well. Many of us have tried to teach others the art of building beautiful designs or writing elegant code. However, at best we can come up with a set of guidelines. It is very similar to trying to teach someone to write a fiction. There may be guidelines, but the ‘magic’ has to come from within the author.

The software development managers, however, would like to treat software development as a form of engineering with standards everywhere whereas one piece may be replaced by another easily. In their dream scenario, one should be able to accurately estimate the time and resources required for a software project. Then they should be able to engage a set of software professionals as per the estimate – ‘x’ number of type ‘A’, ‘y’ number of type ‘B’, etc. They should be able to replace a type ‘A’ professional with another type ‘A’ professionals even in the middle of the project with no or minimal impact. This scenario allows them to engage the required type of professionals anywhere in the globe. Utopia? One would think…

Moving Towards Standards

Last several years has seen a massive change in the way we do software development. Let us talk about some of these changes:

1. Inexpensive Infrastructure: We now have the proliferation of inexpensive (or free) infrastructure pieces, e.g., Apache, J2EE, .Net, JBoss, PHP, MySQL, Linux, Eclipse, etc. This has allowed us to concentrate on the application development rather than worry about memory management, inter-process communication, UI engine, etc. Just a few years back when I started a new project, my team would build the framework pieces first – every time. Today I use the already developed software for these standard tasks.
2. Prebuilt Libraries: Now a days you can find a library for almost everything you want to do, whether it is to encrypt data, use a protocol, spell check, compute complex formulas, XML document processing, etc. Most of these libraries are free and supported by a thriving community of developers – leading to their widespread use.
3. Open Source: Arguably open source has had the biggest impact to software development. Not only much of the infrastructure and libraries we talked about are open source, there are a slew of open source applications that could be used as the base to start with. Since source code is available, these applications can easily be modified to suit one’s needs. Just couple of weeks back I tweaked PHPBB to use with an application Gyanee when I needed a bulletin board.

This phenomena has made the application development process easier and faster. Some of the infrastructure software, libraries and applications have become so popular that it has resulted in a form of standardization. Some genre of applications use the same set of technologies, e.g., significant number of web applications use LAMP (Linux, Apache, MySQL, PHP) stack. The structure of these applications tend to be very similar.

An Opportunity for Commoditization

The above standardization presents a unique opportunity for the software managers. It is hence possible to partition applications such that some of the parts (modules) are fairly standard in nature. These use standard technologies, have standard interfaces, and standard structure. The plug-and-play utopia has a chance to become a reality in this case.

It gets only better if we combine the above phenomena with the inexpensive offshore development model. In theory, it is hence possible to farm out these ‘standard’ parts of a product to offshore teams, let them work on these, and then integrate with the rest of the product. This, IMHO, is commoditizing software development.

There is no doubt that this kind of standardization and commoditization is not possible for all the software projects. However, I submit that a large number of projects can benefit from this approach. I am equally certain that the software development process is ready for another revolution to take advantage of standardization and low cost offshore development.

Introduction

A majority of us have come in contact with software outsourcing in the last several years. Some of us have benefited from it whereas some have seen their jobs going off-shore. We may like or detest this phenomenon, but we should be in agreement that outsourcing is here to stay. Hence, we should all strive to understand it and try to benefit from it. And the first step is to look at the history.

The Start of Outsourcing

Back in late 80′s and early 90′s, when I was in India, we rarely talked about software outsourcing. In fact, the only thing we used to talk about was called ‘body shopping’ – where hoards of software professionals were sent to (mainly) US to work on client’s projects. Then came Y2K that took the body shopping to a new level. When the market collapsed in early 2000′s many of these professionals were sent back. This also coincided with a glut of broadband bandwidth across Atlantic – giving rise to cheaper and better telecommunications between the two countries. The CIO’s of US based companies noticed thousands of US trained software professionals (many of whom they themselves trained) willing to work at 1/10th of the US salary. The cash strapped CIO’s could not let go of this opportunity and started moving the software tasks to India. Thus started outsourcing.

Outsourcing Gathers Steam

Initially the maintenance and QA work was moved. As the confidence in Indian teams rose, small development tasks (the ones we used to lovingly call ‘grunt work’) were outsourced. There were hiccups for sure, but gradually the Indian teams started proving their worth. Around the same time the first effects of opening up of Indian economy were becoming visible. Doing business in India was getting (relatively) easier. The foreign companies were allowed to open offices in India without having an Indian partner. Many US companies took advantage of this opportunity and opened offices in India.

Then around 2003 something significant happened. The Indian teams started asking to be involved in complete software development life cycle. In short they wanted to build the entire products in India. Many of us in the US (myself included) came up with 1001 reasons why this would not be successful. To be honest many of the initial ‘projects’ failed. But the pressure to reduce cost was too much. The US companies also had invested too much in Indian teams. Gradually the Indian teams and the US management learned how to work with each other. Success was round the corner.

Everybody Wants to Outsource

Suddenly outsourcing became the silver bullet every CIO loved. They could simply move their software development operations to India, not have to deal with the demanding professionals in US, get the work done in 1/10th of the cost, and be called a hero. There was a rush to open offices in Bangalore, Chennai, Hyderabad, Pune, Mumbai, NOIDA, Gurgaon, etc. The companies planned to hire thousands of professionals every year for their Indian operations.

At the same time in India, there was a huge demand for software professionals. Everybody and his brother wanted to become a software professional. The Indian universities planned to produce more and more IT professionals every year. Many more private institutions started offering a degree in Computer Science or diplomas in various programing languages. India had a billion plus people and she was going to produce enough professionals to quench the thirst of the world for ‘cheap software development’.

Crash and Burn – Not Really

This was not to be. Not all the professionals are created equal. There is a huge difference in productivity of an average developer and a very good one. Many of these (private) institutes which were churning out programmers by thousands were nothing more than money making ventures. Since software industry was so lucrative, there was little incentive for a smart young graduate from a good university to enter academics. Sadly many of the graduates did not have right kind of knowledge or training.

This created a shortage of high quality software professionals – especially the experienced ones. The salaries of experienced software professionals sky-rocketed. It was not uncommon to get a raise of 20-50% every year. With the Indian economy booming, the real-estate costs in the so-called IT hubs went up many-folds. Gradually, the Indian cost advantage started disappearing.

However, the smart people who were running these ‘software shops’ understood the dynamics. They put in place a pyramid like team structure. The top of the pyramid would be an experienced professional and at the bottom would be fresh graduates. The middle would be filled with professionals with 1+ years of industry experience. The average cost of the team was thus kept in check.

Looking at the Future

Though there is no doubt that the outsourcing model has lot of potential, this does not work for everybody. We are, however, at a stage where the availability of quality professionals in not only India but also in other places in the world (Russia, Eastern Europe, China, Far East, and soon in Africa) would force us to find a paradigm that allows us to take advantage of their skills. At the same time I submit that the on-shore professionals have a huge part to play. I have been working on such a model that would accomplish this very goal and have been very successful so far. I will share that model with the community in a post in coming days.