Tuesday, October 4, 2011

How I promoted agile and the cloud a long time ago

The other day I ran into an old article that I wrote in 1997 for the magazine "Informatie" in the Netherlands and for some reason, I also had an English translated version. The contents is still actual today, so I thought I publish this article here. The magazine does not exist anymore, so I hope that the publisher doesn't mind since it is now over 10 years old.

The first thing you will notice is that my English has improved over the years (I hope so!).This English version was never published and it seems to be more like a draft, but I left it as I found it even though it is a bit long winded. I promise you that the Dutch version was much better! A professional editor reviewed it anyway :-)

Click to enlarge
Secondly, the article is written in a period that many organisations still did not believe in the Internet or they wanted to do something but had no clue what. It was also a period that many people did not understand web technology.

The key thing what I promote here is what we call these days an agile development or project management methodology. Though for the purists it is better called an evolutionary or iterative approach. Central to the story is the pilot project which lays the foundation for the solution but is not complete and it might be clumsy with respect to features, commercial offering and technical implementation. But once you've gone live, you will have to gradually improve it through iterations where the IT team needs to work closely with the business team to progressively improve the system. I have limited myself to write this story for online systems but the agile vision tells us that the approach could just as well be taken for any application.

There is much criticism on the traditional waterfall methodology. I don't understand this. You need to use a methodology that is most appropriate for the situation and there are situations that the straightforward waterfall methodology works very well. Each iteration as I describe here goes through its own mini waterfall process.

Regardless how development will be done, I really prefer to have a very sold grasp on the scope of the project before I begin and that means ideally a full set of requirements. Though requirements can change during the project, it really helps to have a solid set of requirements to define which architecture you require and to give some expectations of time and budget. As we all know how it goes with budgets, once you mention one number, that is going to be it.

In her blog, Julie Zimmerman gives some good points regarding agile projects: http://projectmomblog.blogspot.com/2011/01/when-is-agile-not-agile.html

Over time, I have been successful running projects with an agile approach. But I must say that the success depends on the people you have and I am grateful to the fantastic people that I had in my projects. It all comes down to working well in a multi disciplinary team having developers, BA's and end-users working very closely together. In an agile project, they run the project much more than you as project manager.

Coming back to my article: Via an explanation of the technology infrastructure, I already promote the cloud  as an option to consider. At least according to one of the many variations of the definition of "the cloud". I used the old fashioned term outsourcing for it.

It is funny to see how some things have changed (e.g. the uptake of Internet and Mobile technology) and other things not that much (e.g. we're still debating the cloud and agile; and definitely we too often do our first implementation in a clumsy way).

PS: If you don't want to read the whole article, you still might want to scroll down for the larger version of the diagram as depicted above.


____________________________________________________________________


The growing pains of Web-projects


Contents
Introduction
Characterizing Web-systems
Why does a Web-project differ from a client/server-project?
How many Internet-projects started
The forces that act upon and within the project
You have to go Live on the Net with the pilot!
The overall project structure
The pilot project
The system architecture
Outsourcing the project (or rather not?)
Conclusion

Introduction

Internet projects differ from projects that develop internal information systems in a few aspects. Generally they are just system development projects for systems with a constant exposure to (potential) clients. This last element introduces the specific flavor. The Web-site should constantly be attractive and change. Also the business goals and gains are not clear. In this article we discuss the development method for Internet systems that need a database and a Web-application to accesses the database. Internet-sites that serve only static pages or Intranets are excluded. Intranets don’t have the constant exposure to clients outside the company. An Intranet project resembles more to a ‘normal’ client/server project. The same thing counts for Extranets, but everything depends on how much more diverse and big the user group is and the impact on the organization and the business. Basically it covers projects for electronic commerce on the Internet.
This article treats some experiences with starting an Internet project for the first time and what problems and obstacles can be expected. I hope to show that you have to face these problems and should not take an defensive attitude towards Internet activities, but take the time to find your place in cyberspace. You’ve got to play the game if you want to win.

Characterizing Web-systems

Information systems using Web-technology can be divided according to some characteristics. A primary distinction can be made between public and private systems. Intranets and Extranets serve a limited and known group of users. You do not have to attract people to your site by adding new features and changing contents all the time. Intranets serve an inter-organization task and people need to use it according to their work-tasks. For Extranets this counts just as well, but when it serves clients it needs more attention. With an (open) Internet application however, you serve the whole Internet community. People are not forced to access your site and therefore you need to attract people constantly.

A second important distinction has to be made between sites that just display static HTML and images and the ones where data from a database is accessed. The typical Home-pages of companies are in general based upon static HTML-pages with images. It can be characterized as a form of advertising. It is general accessible on the Internet, the data is not based upon a database and it does not serve transactions. A search-engine or a Yellow Pages Web-site is on the other hand based upon structured data in a database and serves a combination of advertising (adverts of companies) and publishing (publishing the directory).

A third distinction is the distinction between just retrieving data from a database (e.g. the Yellow Pages) and supporting transactions. An example of the latter one is a Web-site where people can actually order physical products (Pizza’s, CD’s).
A fourth distinction can be made based upon the amount of data that can be accessed. Choices for investments and technology can differ a lot for cases where you have a large database or a small database. You get quite a different project when you want to make a product-ordering system for a company that sells 150 different products, then when you build a system for a publisher that wants to publish it’s 100 terabyte database with articles and books via the Web.
A last distinction can be made between advertising and actual publishing, although the difference might not always be clear. Most Home-pages can be seen as a form of advert (who are we, what are we doing, how can you reach us). When you start placing complete articles, magazines or books on the net, it will become some kind of publishing.
The method that is discussed here serves public and database driven Web-projects.

Why does a Web-project differ from a client/server-project?

As soon you make an application on the Web that is accessible to anyone in the world, you need to change it continuously without being off-line. Just as with the Intranets and Extranets, the first step is a small project that in only a very short time results in a first prototype. Often this prototype is then placed on the Net. From that moment, you started doing ‘business’ on the Net and so you can’t take it off-line anymore. The fact that it is a prototype and therefore has a limited functionality, means that you need to improve it quickly. This needs to be done with small pieces each time. You are doing maintenance and upgrades almost right from the beginning.  And you do it in a business environment where the goals and gains are not clear. Therefor you need to follow a true evolutionary way of developing. This situation creates a specific project structure. For such a project you need a new methodology and a different way of working.
There are already a couple of methods that describe the evolutionary way of developing. Because they are general development methods, it leaves you also with a lot of questions. Which modeling techniques do I use for Web-pages with frames and Java-applets? How do I control the project planning when priority for new functionality and requirements change every week? And, how do I transform this quickly built prototype into a professional system that can support our business? It is quite a challenge to manage and control the project, that is constantly changing in a constantly changing environment.

How many Internet-projects started

Somewhere around 1995 the Internet started to become the new technology in the IT-world. At least everyone talked about it and in 1997 most bigger companies had their Web-presence with an own domain. But still, for most managers, including IT-managers, it was not clear what you should really do on the Web. The Web and Web-technology evolved in an incredible speed. Although most managers had the feeling that they should get on board of this fast moving train, they did not have a clear idea what to do on the Web and how it would influence their business. Often it was the sales and marketing people who found the first practical uses for it. It was sometimes referred to by them, that this was finally a new technology for which they did not need the IT-department. The first Web-sites of companies were therefore often built by sales and marketing people with some whiz kids. The IT-department was only partly involved.
But as soon the Web-site became a success, it needed a continuation. This meant more hardware and software investments and the playground had to become a project of the IT-department. And now the trouble started. IT-personnel had to learn new technology and had to respond more quickly to the demands of the marketing people. Where before the difficult demands of the users within the organization could be fended off with arguments like the budgets and the ‘choices of the management’. Now serious demands came that needed to be realized instantaneously, because the Web-site influenced directly clients and thus business results.
Although the situation described here is not exactly true for all companies and Web-projects it is true that the takers of the initiative of the Web-site usually have to start with a limited budget and have to show in a very short time what they have in mind. And indeed, with modern technology you can create a Web-application in a few weeks that sometimes leads to completely new businesses or companies.  But the drawback is that this pilot system is often not fit for maintenance and forthcoming functionality. When new changes and developments come not as quickly as the first successes, the clients and the company directors get easily disappointed. However, it is a fact of life you have to deal with as a taker of the initiative. Unless you know exactly what is needed and get all the budget straight from the beginning. But as it happens a lot in the field of information system development, “users only know what they want when they see it”. The pilot system has to evolve gradually, although you would rather rebuild it completely. But you won’t get the time for it and you can’t take the system off line anymore. And more problematic, you don’t know exactly in advance what technology you need and what development effort is involved. So you can not make a reasonable estimate of the project costs.

The forces that act upon and within the project

Besides the fact that you have to find out how to use the new medium for your existing business, the Internet makes it also possible to get into new sorts of business activities. The Internet reduces physical distances and makes it possible, not only to acquire new clients, but also to let them browse through your products and services and let them actually order the product or services and even perform the complete (business) transaction including the payment. This changes the way of doing business radically. Also the Internet reduces differences between business activities. For example the differences between advertising and publishing is reduced.
These fast changing and unclear markets do not make it easier for the company’s management-team to define their new position. For the Web, companies need to redefine their ‘core activities’. What should you do and what should you not do on the Internet. How important is it now and how important will it be in 5 years? How is the world going to be in 5 years?
This unclear business-context of the project makes it, that the management is very reluctant in providing the necessary financial means (this is generally more true in Europe than in the North America). The Web-team needs to make the necessary investments while it might take some years before the Web-site becomes profitable. And there are surely no guarantees! 

 
Because you go live quickly with the pilot, you get besides positive reactions from clients negative reactions because the application does not work as perfect as they might have hoped. You must take account of the fact that your Web-application will be more off-line than you think you can afford. The technology is very new and changes every day. A Web-application uses different sorts of software of different suppliers. The Internet-software industry is the most unpredictable and booming industry of the moment. Suppliers will release software that is not as robust and tested (more bugs then ever before!), just to beat the competition. The project-team does not get the time to build a well defined, tested and modeled application; sales and marketing will have continuously new demands and wishes. It does not get the right means; too small computers, not enough people and not the right software. As far as you make a nice time and resource plan, you will find that it needs to be changed almost every day. You do not deliver the new functionality in time and you will find yourself spending more time in keeping your pilot-application alive, then spending it in rebuilding the system with a better concept and architecture. Although you have learned from the mistakes, you find that you don’t have the time and the means to rebuild the system with your new insights.

You have to go Live onthe Net with the pilot!


All together there are different forces working upon and within the Web-project. You can expect a lot of problems. This foresight might make you come to the conclusion that you better wait with the Web-project or do the pilot but not make it accessible on the Internet. But the contrary is advised. You need to do the pilot. And you need to make it live on the Net. This is the only way to evaluate and learn what kind of business you have to do on the Web. What technology you need, how the technology works and what kind of project-organization you need. If you keep it only local, for example on the Intranet, you will never experience what a real Internet exposure means. And you want to experience and evaluate this. The Web is changing business and markets dramatically. As sure as there are many threats, there are even more opportunities.  

The key factor here is to do some experience with the pilot on the Internet, but you need to have the discipline to introduce a break in the whole process and invest the time and means to rebuild the application according to the new insights. The IT-people as well as the marketing-people in the Web-team must realize that this is really necessary. If you are capable of getting the two disciplines in line even though there are so many uncertain factors, you will find that you get into a challenging market where your personal ideas and actions can have great influence on the companies well being.
The alternative, not doing the Web-project is probably no alternative at all. In that case you will not get the right experience and you will be too late to acquire the right position on the Internet-marketplace.



The overall project structure
Because with the Internet most companies step into a new world and because they have to use new technology and start doing business in a new way, it is wise to find out what is involved and get some experience with it. If it is not for this reason, it is for the earlier mentioned reasons that the Internet project starts in the form of a pilot. The pilot that actually is placed on the Net and that is being accessed by Internet users, is no pilot anymore. It is a live system. From that moment all new developments are in fact maintenance. The ‘owners’ of the system (most often product or marketing managers; from now on they are referred to as the product manager) are demanding new features and changes continuously. It is “not done” to keep a Web-site completely unchanged for a longer period. At least the visual look and feel should change every once in a while. To be able to serve all these requests, you need to treat each request as a little project. The system evolves in a real evolutionary way.

The requests are evaluated and processed by the project manager (the Information Architect). With his knowledge of the technology and the business needs, he can see how and when the requests can be realized. Together with the product manager he analyzes the request. When needed, he asks assistance of project members who can add their specific skills. To realize some requests, new IT-means have to be acquired. When he has a clear idea about all that is involved in realizing the new functionality, he sees which project members can do the job, acquires the optional needed hardware and software, defines a plan for executing the request and adapts the project plan.
The project plan will be changed constantly. For each new request it is decided what priority it has. New high priority request delay other requests. You can expect that there are times that the plan is changed every week.

The pilot project

The first initiative for the Web-application will usually be in the form of a pilot project that has to result into a pilot system. The goal of the pilot is to find out more precisely what functionality is needed, to find out what kind of user interface (layout, look and feel) is appropriate and to get acquainted with the technology.

The pilot will give the product manager an idea how the system will work and look like and gives an opportunity to get some practical experience and input from end-users or clients. Also some experience has to be obtained with merging editorial elements within the application and Web-site. The editorial elements are the ‘static’ text and graphical elements that completes the Web-application. Within a Web-site there will be places that give information that is not obtained from the database per se. This is for example some advertisements, some ‘hot news’ or some articles. They are added, changed and removed by hand by the editor of the Web-site.

An important goal of the pilot is to get acquainted with the technology. Because the technology is very new (and it will always stay new because it changes so quickly) you need to get some experience with it. You have to find out what components (hardware and software) are needed and what actually is involved. Sometimes you need to acquire new hardware systems with which you do not have any experience yet and you might want to obtain a new database management system and programming tools. Also you have to find out how you want to protect your Web-system and your corporate systems from the outside world. All these things will become more clear after the pilot.

You might want to put the pilot system live on the Internet and actually use it commercially. You can do this for various reasons, like being the first on the Net with this kind of service or application. As mentioned before, this has some implications. The system is probably not optimal. The code is probably not very clean, due to the fact that the programmers needed to learn the new programming language and tools and had to build the application as quick as possible. The developers might have lots of problems on maintaining the system. You also might have come to the conclusion that the chosen technology was not the best and maybe you want to use other hardware or software for the next releases of the system. This can mean that in spite of the fact that the pilot was developed in a very short time, improving the system and making a second release will take much more time then first expected. From a technical point of view it is sometimes best to redo just everything. From a product marketing point of view it might be really necessary to spend all the efforts in creating new features and add-ons to the pilot system. The trick is to find a good balance between the two forces and putting all arguments into the right perspective.

The system architecture

The pilot system was build with some technology of which you did not know if it would be the right one. Also the architecture (components; how they are put together; installation and parameters of the hardware and software; database structure; program structures; etc.) of the system is probably not optimal. Most companies do not know in advance what the benefits of the Web-site will be. Usually you will have an idea that it is important or that it will become important in the future, but you do not know exactly what it actually will mean in financial terms. Therefore you start the pilot with a limited budget.
The project starts with the minimal number of components that is needed. But what is minimally required? You need a connection to the Internet. This is usually a rented line with a modem of some kind of some network provider (for example a telephone company). Second you need a router that performs the network switching service. But you can also let your provider perform this task. Besides the router, you need a Domain Name Server (DNS) that translates logical names of nodes in the Internet (like www.antares.nl) to IP-numbers (like 123.456.23.2). The IP-numbers are the actual identifications of computers within the Internet. Then you need at least the Web-server itself. Your Web-application is located on the Web-server. But when you publish a great amount of data on the net, you might need to place all this data on a separate machine: the database-server. And also you need a Mail-server. It is required because your network provider and the domain name registry need to be able to contact you (at least this is the case in the Netherlands). However, all these server-functions can be merged into one single computer. But it very much depends on your specific situation if this is advisable. Altogether you might need to invest in more than one computer.
Then you need some software. The router has its own particular software. The Web-server needs a program that supports HTTP, the protocol for the Web. Then you need some system to manage your application data. For this, it is best to use a commercial available relational database management systems. And at last, you need a programming tool with which you make the application that can be run via the Web and that can perform transactions on the database.


Most often you want to have your Web-application perform on a selection of your corporate data. And you want data obtained by the Web-application to be available for your corporate application (for example to send invoices). But because you don't want to expose your corporate system to the outside world, you will have to secure it. This can be done by a firewall. This means that the Web-application is not completely integrated in your corporate system. It has to be done by a secure interface. The configuration that results will look something like what is shown in the next figure.
Your Web-server, Internet Mail-server and domain name server (DNS) are located in a section outside the secure area (demilitarized zone). The outside world has read and controlled write access to these machines. Behind the firewall is the corporate local network. The firewall protects the corporate network form all access by the outside world. But from within the secure area, you are able to access the Internet. You can send Email to everybody on the Internet and you can access the World Wide Web. You can of course also access your own Web-server.
By means of the Web-application on the Web-server, you can give your colleagues access to the data that is obtained from the outside world. A part of all the functionality that you need on this data can be done with the Web-application. Another part needs an actual integration with your corporate system. For example, orders placed by clients have to be processed by your financial and stock management systems. This can usually be done in the safest way by regular downloads of data from the Web-server to the corporate database inside the firewall.
Another issue to be dealt with, is the fact that you want your Web-server 24 hours up and running. This means that you cannot afford any downtime. Therefore you need to make your Web-server failsave by doubling components. But it is also wise to have a (almost) complete mirror of the system. When you have to perform some maintenance tasks on the Web-server, you can quickly replace it by the mirror system. This mirror system can be the development machine on which you place a copy of the database of the Web-server on a regular basis. Even better is to have a separate test and backup server for this purpose.
In cases where your Web-database contains large amounts of data and you need to update this data on a regular basis, you might get into the situation that you cannot perform these updates on the same database that is accessed by the surfers on the Web. In these situations is the corporate database available for such updates outside the office hours. But because the Web-system is used 24 hours a day, this is not possible. A second server can be used to achieve a 24 hours availability for large databases with large updates. There are different technical solutions to do this. One is to do all the updates on the second machine and to do a backup and restore action to move the database to the Web-server. The Web-application is in that case not available for a short period of time. Another solution is, to perform the updates on the ‘disconnected’ database, then to switch the Internet access to this server and disconnect the not yet updated database. Then you perform the same updates on this database.


Making your Web-site have a good performance is a delicate matter. There are many aspects and components to consider when you want to improve the performance. First of all, there is the experienced performance by the people on the Web. But this is influenced by the performance of the Internet itself.
Second, you need to investigate what type of system you have. Is it a large database through which the Web-surfers browse but do not insert or update data? Or is it a system where people continuously perform database-updates. For example, in the case where people place orders to buy your products. And of course, the most important thing is to use the right hardware. Investing money in hardware is always cheaper then investing it in people (hours). Sometimes installing some extra memory, disk-space or buying a faster computer is the easiest and smartest thing to do. Also you need to configure your operating system and database management system the right way. All efforts have no use when this is not done properly. And then you have of course the application and data structure. With a good designed application and database, one can improve the performance enormously. It is very important to analyze the properties of the database management system and the programming language. Every SQL implementation is different, so study the effect of your queries and indexes. Also study how data manipulation (insert, update, delete) transactions with respect to locking mechanism is organized. When exploiting large databases, one can learn from theories in the field of Data Warehouses. In a Data Warehouse you introduce data redundancy to improve the queries. Because the data in a Data Warehouse is quite static, the extra costs during updates are accepted.
All this analysis must be done with respect to the number of page requests per minute that are performed on the server. Each page view can result into multiple database queries or transactions. What is acceptable in terms of milliseconds that are needed to generate the page, depends very much on the number of requests that are expected on your server. Is your goal to serve the whole Internet community like the popular search-engines such as Alta Vista and Yahoo, then executing a page request must be served more quickly then in the case when you expect about 10 visitors per hour. In the last case, you must place the page generation speed in perspective of the overall Internet performance. In the first case, you must make sure that each page request is dealt with within a fraction of a second, to be able to serve as much requests parallel as possible.
When your server gets bombed with requests, you can duplicate the system and distribute the requests equally over the servers. For really complex Web-sites that server different applications and tasks, it is wise to use dedicated server for each tasks. An advanced implementation of these kinds of Web-sites are called Web farms. A Web lets a corporation use all of its Web-sites as though they were part of a single system while at the same time letting each individual Web-site stay under the control of its local owner (from “Harvest profits from Web farms”, in: Datamation, march 1997).

Outsourcing or insourcing?

Besides new technology, you might be faced with new working hours and aspects in the field of human resources. Due to the fact that the Web-applications have to be up and running outside office hours, you need your system manager and web-master to do work on any time of the day. This means changing labor agreements.
All the new aspects on technology and organization together, make companies feel that they should outsource the technical part of the Internet activities. There are two aspects to this. First, is the outsourcing of the hardware, software and the Internet connectivity. Internet providers seem to be the most appropriate parties for this. This might be very well the best choice if you have a simple Web-application and easily to standardize services like email and newsgroups. When the focus lies on a complex Web-application you must be aware of the fact that the system management of the hardware and software are very intertwined with application development. And application development is again intertwined with business activities. These things have always been the case, but for Web-applications this is even more true. You cannot deny that business on the Internet is very much IT-driven.
An alternative is, to outsource the Internet activities to consultancy companies. They have traditionally focused on application development and business analysis. They can more easily support your specific business needs.
Outsourcing might be a very good option. However, if you do this in an early stage, your organization misses the experience of all that is involved. You get too easily in the situation that your partner needs to spend more money and time to serve you, than you think is necessary. And when you want to change your requirements constantly the partner will protest because it does not fit in the agreement. As long as you do not have your Internet activities well defined, you should be careful with outsourcing it. Insourcing, to get the right expertise in house for the moment, is a better option for the initial phase. If you do want to outsource, make sure that you do not lose your flexibility.

Conclusion

The Web is becoming the new medium for exploiting your business for the near future. Although the Internet is still slow, the necessary bandwidth will become available soon. This means that people and organizations will be surfing the Net daily. You have to prepare yourself now and get the right experience for electronic commerce on the Web. The challenge is to learn to manage the new technology and in the same time to find out what business activities you need to exploit on the Web and how to do this. Because there are too many variable and uncertain factors, you have to accept the growing pains that follow Internet projects. You have to learn while doing. This means that you have to go live with a pilot application on the Web of which you know that is far from optimal and that will raise expectations that you cannot meet. Most often, you can achieve in the beginning great results in a very short time. But the trick is to manage the phases that follow the initial pilot project. You will be faced with more and more investments in hardware, software and people, while the profits of your Web-application are almost zero. A clear view on a strategic level for the companies business on the Web is necessary, but is very hard to achieve because the Web is only there for a short while and nobody exactly knows what it will be and what it will do in a few years time. You need to organize your projects in a flexible and adaptable way. The technical and marketing people have to learn to work together, even though they might feel that they are frustrated in their work every now and then. A good understanding of the technical concepts of Web-applications is essential to be successful. Keep in mind that your business depends more and more on information technology. The (Informix) slogan “If you can imagine IT, you can manage IT” has basically been valid already for the last decades.



The author
Drs. H.D.Bouman RI is a consultant of Antares Informatisering B.V. in the Netherlands. He was and is involved as a project manager in different Web-projects, where amongst the Dutch Yellow Pages (www.goudengids.nl).


References
1.    Bouman, H.D., OTIF - Een ontwerptechniek voor informatiefuncties in een 4GL-omgeving. Antares Informatisering B.V., Nieuwegein, 1996.
2.    Tolido, R.J.H., IAD - Het evolutionair ontwikkelen van informatiesystemen. Academic Service, Schoonhoven, 1996.
3.    Eskrow, D.,Harvest profits from Web farms. Datamation, martch 1997.
4.    Bordeware Firewall  - http://www.kaon.co.nz/border/border.html





No comments:

Post a Comment

You are welcome to leave any response or thoughts that you have as feedback.