Summary

As successful web sites gain popularity and their companies, started by creative individuals without software development experience, undergo rapid growth, the original mentality of ad-hock development and production testing causes major disruptions and downtime for the organization. At some point, there comes a time when such companies need to step back and revisit their entire way of developing and releasing software to production. A step backwards, while delaying the hot item of the day, can enable numerous fast steps forward on an ongoing basis, and eliminate the disruptions that are natural without a well defined process.

Separate Environments

As a first step, it necessary to isolate several environments for the following purposes:

Development

• This is where you actually create and modify software and perform unit testing and initial integration testing.

Testing

• This environment should closely mirror production and there are whole volumes that exist and can be written about testing strategies and methodologies. The bottom line is that here your software should undergo functional testing and integration testing with production like systems and interfaces. Furthermore, you may want to perform load, stress or performance testing within this setting.
• Keep in mind that you should create deployment scripts that are also testing within this environment.
• Make sure to test your back-out plan in the event of some major problems during production release.
• For large sites, the performance testing environment may be a small scale version of production and performance results need to be scaled in reference to the actual differences. Note that having different environments for performance testing and production always presents some risk, but most companies cannot justify the added expense of a complete production replica.
• An approach taken by some companies involved two mirrored production environments, one as the primary and one as the backup. The backup environment is then used for performance testing as well as deploying new software releases. For example, you would deploy new software on the backup site, test it for performance and functionality and then switch the backup to be the primary production site when deploying. The old production site is then synchronized with the new software changes and is ready to serve as the backup.

Production

• Your tested deployment scripts are run against the production environment. The more automation the better as chances for human error are decreased. If major problems occur, execute your tested back-out plan.

Testing Processes

As your software grows in functionality, size and complexity it becomes more and more critical to document the software functional requirements as well as the matching test plans. You want to focus on covering the following items:

Manual test scripts

• Create step by step instructions to test your entire software suite covering as much functionality as possible.

Automated regression testing

• If you have the budget, consider tools such as Mercury WinRunner to automate your tests and enable fast regression tests. Keep in mind that such tools take full-time staff to create and maintain scripts.
• Open source tools are available that entail some coding. For example, Junit combined with HttpUnit if you have Java expertise.

Performance testing

• Again, if budgets allow, consider tools such as Mercury LoadRunner to automate performance testing. The tools will enable to test how changes to software effect performance and to document all the performance characteristics of your software.
• Again, open source or free tools can be utilized to perform this function. Consider Jmeter (http://jakarta.apache.org/jmeter/), httperf (http://www.hpl.hp.com/personal/David_Mosberger/httperf.html) and the "Web Application Stress Tool" from Microsoft.

Bug Tracking

• For a large project consider using a bug tracking tool such as Bugzilla or one of its variants.

Do not have software developers test their own code

• Consider that it does not make sense to have a software developer test his or her own code. Obviously, they tested it during development, but its unlikely that they will think and test the same way as others who are unfamiliar with the code.

System Documentation

As mentioned partially above, here is a list of documents you should create and maintain for your system:

Functional requirements and specifications

• Define the goals of the system
• Describe business processes
• Document inputs and outputs for each process as well as results
• Do not include technical details in this documentation

Detailed technical design

• Diagram and describe all the physical and software components
• Include flow diagrams and process descriptions
• Include scenario walkthroughs for complex business process paths
• Show how data changes at various steps in each process

Hardware, software, network configurations

• Document each environment that you maintain, including:
• server lists
• ip addresses
• software that is running on each server
• hardware configurations

Operating procedures

• At some point as you hire multiple people you will find it advantageous to have operating procedures for the new hires to read. Take some time and write down the critical daily, monthly, yearly tasks that you perform in business operations. This will save you time getting new people up to speed as well as help to recall details of infrequently performed activities.

Configuration Management

This means that your organization versions all software, documentation and deployment artifacts in such as way that any version or snapshot of your configuration can easily be recovered. Here are some factors to consider regarding configuration management:

Version control

• Tools such as CVS or SourceSafe provide excellent repositories for code, documents and other artifacts

Build scripts

• Version your software build scripts that are used to compile your code into components that can be deployed.

Install scripts

• Create and version your installation scripts.

Hardware/Software/Network configurations

• Document and version your hardware configurations as well as any third party software and interfaces that are used.

Functional Requirements, Technical Designs and Test Plans

• Make sure you retain these critical documents under version control.

Production Monitoring

Expand the effort to setup automated monitoring for network and server performance. Email notifications will instantly alert you when there are problems so that you can react before customer calls start coming in. The topic is a separate discussion in and of itself, but here are some ideas to keep in mind:

• Monitor network traffic, watch for unusual spikes in either the up or down direction
• Tools such as MRTG and Cacti are worth considering
• Monitor server health and site response times
• OpenNMS http://opennms.org/, Nagios http://www.nagios.org/, Big Brother http://bb4.com/, Big Sister http://bigsister.graeff.com/, plus many others. (also see http://www.slac.stanford.edu/xorg/nmtf/nmtf-tools.html) provide real-time monitoring of most services such as http, database, disk usage, cpu utilization, while providing alerts when thresholds are exceeded.