The other day I ran into an old article that I wrote in 1997 for the magazine "Informatie" in the Netherlands and for some reason, I also had an English translated version. The contents is still actual today, so I thought I publish this article here. The magazine does not exist anymore, so I hope that the publisher doesn't mind since it is now over 10 years old.
The first thing you will notice is that my English has improved over the years (I hope so!).This English version was never published and it seems to be more like a draft, but I left it as I found it even though it is a bit long winded. I promise you that the Dutch version was much better! A professional editor reviewed it anyway :-)
 |
| Click to enlarge |
Secondly, the article is written in a period that many organisations still did not believe in the Internet or they wanted to do something but had no clue what. It was also a period that many people did not understand web technology.
The key thing what I promote here is what we call these days an agile development or project management methodology. Though for the purists it is better called an evolutionary or iterative approach. Central to the story is the pilot project which lays the foundation for the solution but is not complete and it might be clumsy with respect to features, commercial offering and technical implementation. But once you've gone live, you will have to gradually improve it through iterations where the IT team needs to work closely with the business team to progressively improve the system. I have limited myself to write this story for online systems but the agile vision tells us that the approach could just as well be taken for any application.
There is much criticism on the traditional waterfall methodology. I don't understand this. You need to use a methodology that is most appropriate for the situation and there are situations that the straightforward waterfall methodology works very well. Each iteration as I describe here goes through its own mini waterfall process.
Regardless how development will be done, I really prefer to have a very sold grasp on the scope of the project before I begin and that means ideally a full set of requirements. Though requirements can change during the project, it really helps to have a solid set of requirements to define which architecture you require and to give some expectations of time and budget. As we all know how it goes with budgets, once you mention one number, that is going to be it.
In her blog, Julie Zimmerman gives some good points regarding agile projects:
http://projectmomblog.blogspot.com/2011/01/when-is-agile-not-agile.html
Over time, I have been successful running projects with an agile approach. But I must say that the success depends on the people you have and I am grateful to the fantastic people that I had in my projects. It all comes down to working well in a multi disciplinary team having developers, BA's and end-users working very closely together. In an agile project, they run the project much more than you as project manager.
Coming back to my article: Via an explanation of the technology infrastructure, I already promote the cloud as an option to consider. At least according to one of the many variations of the definition of "the cloud". I used the old fashioned term outsourcing for it.
It is funny to see how some things have changed (e.g. the uptake of Internet and Mobile technology) and other things not that much (e.g. we're still debating the cloud and agile; and definitely we too often do our first implementation in a clumsy way).
PS: If you don't want to read the whole article, you still might want to scroll down for the larger version of the diagram as depicted above.
____________________________________________________________________
The growing pains of Web-projects
Contents
Introduction
Characterizing
Web-systems
Why does a Web-project
differ from a client/server-project?
How many
Internet-projects started
The forces that act
upon and within the project
You have to go Live on
the Net with the pilot!
The overall project
structure
The pilot project
The system architecture
Outsourcing the project
(or rather not?)
Conclusion
Internet
projects differ from projects that develop internal information systems in a
few aspects. Generally they are just system development projects for systems
with a constant exposure to (potential) clients. This last element introduces
the specific flavor. The Web-site should constantly be attractive and change.
Also the business goals and gains are not clear. In this article we discuss the
development method for Internet systems that need a database and a
Web-application to accesses the database. Internet-sites that serve only static
pages or Intranets are excluded. Intranets don’t have the constant exposure to
clients outside the company. An Intranet project resembles more to a ‘normal’
client/server project. The same thing counts for Extranets, but everything
depends on how much more diverse and big the user group is and the impact on
the organization and the business. Basically it covers projects for electronic
commerce on the Internet.
This
article treats some experiences with starting an Internet project for the first
time and what problems and obstacles can be expected. I hope to show that you
have to face these problems and should not take an defensive attitude towards
Internet activities, but take the time to find your place in cyberspace. You’ve
got to play the game if you want to win.
Information
systems using Web-technology can be divided according to some characteristics.
A primary distinction can be made between public and private systems. Intranets
and Extranets serve a limited and known group of users. You do not have to
attract people to your site by adding new features and changing contents all
the time. Intranets serve an inter-organization task and people need to use it
according to their work-tasks. For Extranets this counts just as well, but when
it serves clients it needs more attention. With an (open) Internet application however, you serve the whole Internet community. People are not forced to access your site and therefore you need to attract people constantly.
A second important distinction has to be made between sites that just display static HTML and images and the ones where data from a database is accessed. The typical Home-pages of companies are in general based upon static HTML-pages with images. It can be characterized as a form of advertising. It is general accessible on the Internet, the data is not based upon a database and it does not serve transactions. A search-engine or a Yellow Pages Web-site is on the other hand based upon structured data in a database and serves a combination of advertising (adverts of companies) and publishing (publishing the directory).
A third distinction is the distinction between just retrieving data from a database (e.g. the Yellow Pages) and supporting transactions. An example of the latter
one is a Web-site where people can actually order physical products (Pizza’s,
CD’s).
A fourth
distinction can be made based upon the amount of data that can be accessed.
Choices for investments and technology can differ a lot for cases where you
have a large database or a small database. You get quite a different project
when you want to make a product-ordering system for a company that sells 150
different products, then when you build a system for a publisher that wants to
publish it’s 100 terabyte database with articles and books via the Web.
A last
distinction can be made between advertising and actual publishing, although the
difference might not always be clear. Most Home-pages can be seen as a form of
advert (who are we, what are we doing, how can you reach us). When you start
placing complete articles, magazines or books on the net, it will become some
kind of publishing.
The method
that is discussed here serves public and database driven Web-projects.
As soon you
make an application on the Web that is accessible to anyone in the world, you
need to change it continuously without being off-line. Just as with the
Intranets and Extranets, the first step is a small project that in only a very
short time results in a first prototype. Often this prototype is then placed on
the Net. From that moment, you started doing ‘business’ on the Net and so you
can’t take it off-line anymore. The fact that it is a prototype and therefore
has a limited functionality, means that you need to improve it quickly. This
needs to be done with small pieces each time. You are doing maintenance and
upgrades almost right from the beginning.
And you do it in a business environment where the goals and gains are
not clear. Therefor you need to follow a true evolutionary way of developing.
This situation creates a specific project structure. For such a project you
need a new methodology and a different way of working.
There are
already a couple of methods that describe the evolutionary way of developing.
Because they are general development methods, it leaves you also with a lot of
questions. Which modeling techniques do I use for Web-pages with frames and
Java-applets? How do I control the project planning when priority for new
functionality and requirements change every week? And, how do I transform this
quickly built prototype into a professional system that can support our
business? It is quite a challenge to manage and control the project, that is
constantly changing in a constantly changing environment.
Somewhere
around 1995 the Internet started to become the
new technology in the IT-world. At least everyone talked about it and in 1997
most bigger companies had their Web-presence with an own domain. But still, for
most managers, including IT-managers, it was not clear what you should really
do on the Web. The Web and Web-technology evolved in an incredible speed.
Although most managers had the feeling that they should get on board of this
fast moving train, they did not have a clear idea what to do on the Web and how
it would influence their business. Often it was the sales and marketing people
who found the first practical uses for it. It was sometimes referred to by
them, that this was finally a new technology for which they did not need the
IT-department. The first Web-sites of companies were therefore often built by
sales and marketing people with some whiz kids. The IT-department was only
partly involved.
But as soon
the Web-site became a success, it needed a continuation. This meant more
hardware and software investments and the playground had to become a project of
the IT-department. And now the trouble started. IT-personnel had to learn new
technology and had to respond more quickly to the demands of the marketing
people. Where before the difficult demands of the users within the organization
could be fended off with arguments like the budgets and the ‘choices of the
management’. Now serious demands came that needed to be realized
instantaneously, because the Web-site influenced directly clients and thus
business results.
Although
the situation described here is not exactly true for all companies and
Web-projects it is true that the takers of the initiative of the Web-site
usually have to start with a limited budget and have to show in a very short
time what they have in mind. And indeed, with modern technology you can create
a Web-application in a few weeks that sometimes leads to completely new
businesses or companies. But the
drawback is that this pilot system is often not fit for maintenance and
forthcoming functionality. When new changes and developments come not as
quickly as the first successes, the clients and the company directors get
easily disappointed. However, it is a fact of life you have to deal with as a
taker of the initiative. Unless you know exactly what is needed and get all the
budget straight from the beginning. But as it happens a lot in the field of
information system development, “users only know what they want when they see
it”. The pilot system has to evolve gradually, although you would rather rebuild
it completely. But you won’t get the time for it and you can’t take the system
off line anymore. And more problematic, you don’t know exactly in advance what
technology you need and what development effort is involved. So you can not
make a reasonable estimate of the project costs.
Besides the
fact that you have to find out how to use the new medium for your existing
business, the Internet makes it also possible to get into new sorts of business
activities. The Internet reduces physical distances and makes it possible, not
only to acquire new clients, but also to let them browse through your products
and services and let them actually order the product or services and even
perform the complete (business) transaction including the payment. This changes
the way of doing business radically. Also the Internet reduces differences
between business activities. For example the differences between advertising
and publishing is reduced.
These fast
changing and unclear markets do not make it easier for the company’s
management-team to define their new position. For the Web, companies need to
redefine their ‘core activities’. What should you do and what should you not do
on the Internet. How important is it now and how important will it be in 5
years? How is the world going to be in 5 years?
This
unclear business-context of the project makes it, that the management is very
reluctant in providing the necessary financial means (this is generally more
true in Europe than in the North America). The Web-team needs to make the
necessary investments while it might take some years before the Web-site
becomes profitable. And there are surely no guarantees!
Because you
go live quickly with the pilot, you get besides positive reactions from clients
negative reactions because the application does not work as perfect as they
might have hoped. You must take account of the fact that your Web-application
will be more off-line than you think you can afford. The technology is very new
and changes every day. A Web-application uses different sorts of software of
different suppliers. The Internet-software industry is the most unpredictable
and booming industry of the moment. Suppliers will release software that is not
as robust and tested (more bugs then ever before!), just to beat the
competition. The project-team does not get the time to build a well defined,
tested and modeled application; sales and marketing will have continuously new
demands and wishes. It does not get the right means; too small computers, not
enough people and not the right software. As far as you make a nice time and
resource plan, you will find that it needs to be changed almost every day. You
do not deliver the new functionality in time and you will find yourself
spending more time in keeping your pilot-application alive, then spending it in
rebuilding the system with a better concept and architecture. Although you have
learned from the mistakes, you find that you don’t have the time and the means
to rebuild the system with your new insights.
All together there are different forces working upon and within the Web-project. You can expect a lot of problems. This foresight might make you come to the conclusion that you better wait with the Web-project or do the pilot but not make it accessible on the Internet. But the contrary is advised. You need to do the pilot. And you need to make it live on the Net. This is the only way to evaluate and learn what kind of business you have to do on the Web. What technology you need, how the technology works and what kind of project-organization you need. If you keep it only local, for example on the Intranet, you will never experience what a real Internet exposure means. And you want to experience and evaluate this. The Web is changing business and markets dramatically. As sure as there are many threats, there are even more opportunities.
The key
factor here is to do some experience with the pilot on the Internet, but you
need to have the discipline to introduce a break in the whole process and
invest the time and means to rebuild the application according to the new
insights. The IT-people as well as the marketing-people in the Web-team must
realize that this is really necessary. If you are capable of getting the two
disciplines in line even though there are so many uncertain factors, you will
find that you get into a challenging market where your personal ideas and
actions can have great influence on the companies well being.
The
alternative, not doing the Web-project is probably no alternative at all. In
that case you will not get the right experience and you will be too late to
acquire the right position on the Internet-marketplace.
The overall project structure
Because
with the Internet most companies step into a new world and because they have to
use new technology and start doing business in a new way, it is wise to find
out what is involved and get some experience with it. If it is not for this
reason, it is for the earlier mentioned reasons that the Internet project
starts in the form of a pilot. The pilot that actually is placed on the Net and
that is being accessed by Internet users, is no pilot anymore. It is a live
system. From that moment all new developments are in fact maintenance. The
‘owners’ of the system (most often product or marketing managers; from now on
they are referred to as the product manager) are demanding new features and
changes continuously. It is “not done” to keep a Web-site completely unchanged
for a longer period. At least the visual look and feel should change every once
in a while. To be able to serve all these requests, you need to treat each
request as a little project. The system evolves in a real evolutionary way.
The
requests are evaluated and processed by the project manager (the Information
Architect). With his knowledge of the technology and the business needs, he can
see how and when the requests can be realized. Together with the product
manager he analyzes the request. When needed, he asks assistance of project
members who can add their specific skills. To realize some requests, new
IT-means have to be acquired. When he has a clear idea about all that is
involved in realizing the new functionality, he sees which project members can
do the job, acquires the optional needed hardware and software, defines a plan
for executing the request and adapts the project plan.
The project
plan will be changed constantly. For each new request it is decided what
priority it has. New high priority request delay other requests. You can expect
that there are times that the plan is changed every week.
The first
initiative for the Web-application will usually be in the form of a pilot
project that has to result into a pilot system. The goal of the pilot is to
find out more precisely what functionality is needed, to find out what kind of user interface (layout, look and feel) is appropriate and to get acquainted with the technology.
The pilot will give the product manager an idea how the system will work and look like and gives an opportunity to get some practical experience and input from end-users or clients. Also some experience has to be obtained with merging editorial elements within the application and Web-site. The editorial elements are the ‘static’ text and graphical elements that completes the Web-application. Within a Web-site there will be places that give information that is not obtained from the database per se. This is for example some advertisements, some ‘hot news’ or some articles. They are added, changed and removed by hand by the editor of the Web-site.
An important goal of the pilot is to get acquainted with the technology. Because the technology is very new (and it will always stay new because it changes so
quickly) you need to get some experience with it. You have to find out what
components (hardware and software) are needed and what actually is involved.
Sometimes you need to acquire new hardware systems with which you do not have
any experience yet and you might want to obtain a new database management system
and programming tools. Also you have to find out how you want to protect your
Web-system and your corporate systems from the outside world. All these things
will become more clear after the pilot.
You might
want to put the pilot system live on the Internet and actually use it
commercially. You can do this for various reasons, like being the first on the
Net with this kind of service or application. As mentioned before, this has
some implications. The system is probably not optimal. The code is probably not
very clean, due to the fact that the programmers needed to learn the new
programming language and tools and had to build the application as quick as
possible. The developers might have lots of problems on maintaining the system.
You also might have come to the conclusion that the chosen technology was not
the best and maybe you want to use other hardware or software for the next
releases of the system. This can mean that in spite of the fact that the pilot
was developed in a very short time, improving the system and making a second
release will take much more time then first expected. From a technical point of
view it is sometimes best to redo just everything. From a product marketing
point of view it might be really necessary to spend all the efforts in creating
new features and add-ons to the pilot system. The trick is to find a good
balance between the two forces and putting all arguments into the right
perspective.
The pilot
system was build with some technology of which you did not know if it would be
the right one. Also the architecture (components; how they are put together;
installation and parameters of the hardware and software; database structure;
program structures; etc.) of the system is probably not optimal. Most companies
do not know in advance what the benefits of the Web-site will be. Usually you
will have an idea that it is important or that it will become important in the
future, but you do not know exactly what it actually will mean in financial
terms. Therefore you start the pilot with a limited budget.
The project
starts with the minimal number of components that is needed. But what is
minimally required? You need a connection to the Internet. This is usually a
rented line with a modem of some kind of some network provider (for example a
telephone company). Second you need a router that performs the network
switching service. But you can also let your provider perform this task.
Besides the router, you need a Domain Name Server (DNS) that translates logical
names of nodes in the Internet (like www.antares.nl) to IP-numbers (like
123.456.23.2). The IP-numbers are the actual identifications of computers
within the Internet. Then you need at least the Web-server itself. Your
Web-application is located on the Web-server. But when you publish a great
amount of data on the net, you might need to place all this data on a separate
machine: the database-server. And also you need a Mail-server. It is required
because your network provider and the domain name registry need to be able to
contact you (at least this is the case in the Netherlands). However, all these
server-functions can be merged into one single computer. But it very much
depends on your specific situation if this is advisable. Altogether you might
need to invest in more than one computer.
Then you
need some software. The router has its own particular software. The Web-server
needs a program that supports HTTP, the protocol for the Web. Then you need
some system to manage your application data. For this, it is best to use a
commercial available relational database management systems. And at last, you
need a programming tool with which you make the application that can be run via
the Web and that can perform transactions on the database.
Most often
you want to have your Web-application perform on a selection of your corporate
data. And you want data obtained by the Web-application to be available for
your corporate application (for example to send invoices). But because you
don't want to expose your corporate system to the outside world, you will have
to secure it. This can be done by a firewall. This means that the
Web-application is not completely integrated in your corporate system. It has
to be done by a secure interface. The configuration that results will look
something like what is shown in the next figure.
Your
Web-server, Internet Mail-server and domain name server (DNS) are located in a
section outside the secure area (demilitarized zone). The outside world has
read and controlled write access to these machines. Behind the firewall is the
corporate local network. The firewall protects the corporate network form all
access by the outside world. But from within the secure area, you are able to
access the Internet. You can send Email to everybody on the Internet and you
can access the World Wide Web. You can of course also access your own
Web-server.
By means of
the Web-application on the Web-server, you can give your colleagues access to
the data that is obtained from the outside world. A part of all the
functionality that you need on this data can be done with the Web-application.
Another part needs an actual integration with your corporate system. For
example, orders placed by clients have to be processed by your financial and
stock management systems. This can usually be done in the safest way by regular
downloads of data from the Web-server to the corporate database inside the
firewall.
Another
issue to be dealt with, is the fact that you want your Web-server 24 hours up
and running. This means that you cannot afford any downtime. Therefore you need
to make your Web-server failsave by doubling components. But it is also wise to
have a (almost) complete mirror of the system. When you have to perform some
maintenance tasks on the Web-server, you can quickly replace it by the mirror
system. This mirror system can be the development machine on which you place a
copy of the database of the Web-server on a regular basis. Even better is to
have a separate test and backup server for this purpose.
In cases where your Web-database contains large amounts of data and you need to
update this data on a regular basis, you might get into the situation that you
cannot perform these updates on the same database that is accessed by the
surfers on the Web. In these situations is the corporate database available for
such updates outside the office hours. But because the Web-system is used 24
hours a day, this is not possible. A second server can be used to achieve a 24
hours availability for large databases with large updates. There are different
technical solutions to do this. One is to do all the updates on the second
machine and to do a backup and restore action to move the database to the
Web-server. The Web-application is in that case not available for a short
period of time. Another solution is, to perform the updates on the
‘disconnected’ database, then to switch the Internet access to this server and
disconnect the not yet updated database. Then you perform the same updates on
this database.
Making your
Web-site have a good performance is a delicate matter. There are many aspects
and components to consider when you want to improve the performance. First of
all, there is the experienced performance by the people on the Web. But this is
influenced by the performance of the Internet itself.
Second, you
need to investigate what type of system you have. Is it a large database
through which the Web-surfers browse but do not insert or update data? Or is it
a system where people continuously perform database-updates. For example, in
the case where people place orders to buy your products. And of course, the
most important thing is to use the right hardware. Investing money in hardware
is always cheaper then investing it in people (hours). Sometimes installing
some extra memory, disk-space or buying a faster computer is the easiest and
smartest thing to do. Also you need to configure your operating system and
database management system the right way. All efforts have no use when this is
not done properly. And then you have of course the application and data
structure. With a good designed application and database, one can improve the
performance enormously. It is very important to analyze the properties of the
database management system and the programming language. Every SQL
implementation is different, so study the effect of your queries and indexes.
Also study how data manipulation (insert, update, delete) transactions with
respect to locking mechanism is organized. When exploiting large databases, one
can learn from theories in the field of Data Warehouses. In a Data Warehouse
you introduce data redundancy to improve the queries. Because the data in a
Data Warehouse is quite static, the extra costs during updates are accepted.
All this
analysis must be done with respect to the number of page requests per minute
that are performed on the server. Each page view can result into multiple
database queries or transactions. What is acceptable in terms of milliseconds
that are needed to generate the page, depends very much on the number of
requests that are expected on your server. Is your goal to serve the whole
Internet community like the popular search-engines such as Alta Vista and
Yahoo, then executing a page request must be served more quickly then in the
case when you expect about 10 visitors per hour. In the last case, you must
place the page generation speed in perspective of the overall Internet
performance. In the first case, you must make sure that each page request is dealt
with within a fraction of a second, to be able to serve as much requests
parallel as possible.
When your
server gets bombed with requests, you can duplicate the system and distribute
the requests equally over the servers. For really complex Web-sites that server
different applications and tasks, it is wise to use dedicated server for each
tasks. An advanced implementation of these kinds of Web-sites are called Web
farms. A Web lets a corporation use all of its Web-sites as though they were
part of a single system while at the same time letting each individual Web-site
stay under the control of its local owner (from “Harvest profits from Web
farms”, in: Datamation, march 1997).
Besides new
technology, you might be faced with new working hours and aspects in the field
of human resources. Due to the fact that the Web-applications have to be up and
running outside office hours, you need your system manager and web-master to do
work on any time of the day. This means changing labor agreements.
All the new
aspects on technology and organization together, make companies feel that they
should outsource the technical part of the Internet activities. There are two
aspects to this. First, is the outsourcing of the hardware, software and the
Internet connectivity. Internet providers seem to be the most appropriate parties
for this. This might be very well the best choice if you have a simple
Web-application and easily to standardize services like email and newsgroups.
When the focus lies on a complex Web-application you must be aware of the fact
that the system management of the hardware and software are very intertwined
with application development. And application development is again intertwined
with business activities. These things have always been the case, but for
Web-applications this is even more true. You cannot deny that business on the
Internet is very much IT-driven.
An
alternative is, to outsource the Internet activities to consultancy companies.
They have traditionally focused on application development and business
analysis. They can more easily support your specific business needs.
Outsourcing
might be a very good option. However, if you do this in an early stage, your
organization misses the experience of all that is involved. You get too easily
in the situation that your partner needs to spend more money and time to serve
you, than you think is necessary. And when you want to change your requirements
constantly the partner will protest because it does not fit in the agreement.
As long as you do not have your Internet activities well defined, you should be
careful with outsourcing it. Insourcing, to get the right expertise in house
for the moment, is a better option for the initial phase. If you do want to
outsource, make sure that you do not lose your flexibility.
The Web is
becoming the new medium for
exploiting your business for the near future. Although the Internet is still
slow, the necessary bandwidth will become available soon. This means that
people and organizations will be surfing the Net daily. You have to prepare
yourself now and get the right experience for electronic commerce on the Web.
The challenge is to learn to manage the new technology and in the same time to
find out what business activities you need to exploit on the Web and how to do
this. Because there are too many variable and uncertain factors, you have to
accept the growing pains that follow Internet projects. You have to learn while
doing. This means that you have to go live with a pilot application on the Web
of which you know that is far from optimal and that will raise expectations
that you cannot meet. Most often, you can achieve in the beginning great
results in a very short time. But the trick is to manage the phases that follow
the initial pilot project. You will be faced with more and more investments in
hardware, software and people, while the profits of your Web-application are
almost zero. A clear view on a strategic level for the companies business on
the Web is necessary, but is very hard to achieve because the Web is only there
for a short while and nobody exactly knows what it will be and what it will do
in a few years time. You need to organize your projects in a flexible and
adaptable way. The technical and marketing people have to learn to work
together, even though they might feel that they are frustrated in their work
every now and then. A good understanding of the technical concepts of
Web-applications is essential to be successful. Keep in mind that your business
depends more and more on information technology. The (Informix) slogan “If you
can imagine IT, you can manage IT” has basically been valid already for the
last decades.
The author
Drs.
H.D.Bouman RI is a consultant of Antares Informatisering B.V. in the
Netherlands. He was and is involved as a project manager in different
Web-projects, where amongst the Dutch Yellow Pages (www.goudengids.nl).
References
1.
Bouman,
H.D., OTIF - Een ontwerptechniek voor informatiefuncties in een 4GL-omgeving.
Antares Informatisering B.V., Nieuwegein, 1996.
2.
Tolido,
R.J.H., IAD - Het evolutionair ontwikkelen van informatiesystemen. Academic
Service, Schoonhoven, 1996.
3.
Eskrow,
D.,Harvest profits from Web farms. Datamation, martch 1997.
4.
Bordeware
Firewall -
http://www.kaon.co.nz/border/border.html
