So the basics of a proper software development environment are going to consist of a Dev Environment, QA Environment, Staging Environment and Production Environment, with a version control server supporting these environments that contains your code. The only exception to this is your Production environment – that environment may or may not have direct access to the version control server for security reasons. If it does not have access to the version control server then you will not be able to export your Trunk to the Production servers so you will need to have a build process that enables you to build install scripts that install the new version of code into Production (more on that later). Let’s look at an overview of each environment:

Dev Environment – this environment is owned by the software developers (programmers). This is an environment that enables them to work on new code and do anything necessary to the environment to support their development efforts. This can include re-configuration of the environment, dumping new data, changing file access and even crashing their environment. Normally this environment is configured “similar” to the Production environment, but scaled down significantly (like they may have 1 web server and 1 database server versus 5 web servers and 8 database servers like in the Production environment). Likewise the server(s) the programmers are developing on are actually a shared hardware environment with each developer having their own “area” to work in by virtualizing the server and providing each develop their own private slice to develop in (this is a great configuration as they can work independent of the other developers and not worry about bringing them down if they want to start and stop processes and likewise reduce server cost in reducing the number of servers needed to support the developers). This environment would be for developers only to work in and even though it may be virtualized, there still may be many development servers based on the size of the development team. The environment is considered dynamic and unstable due to the constant changes that occur in the process of developing code to support new functionality for an application. This environment needs access to the version control system to check in & out code.

QA Environment – this environment is owned by the software testers. This is an environment that enables the software testers to work in a “controlled” environment that is as close to a Production configuration as possible. Since a Production environment normally consists of numerous servers, databases and network gear it normally isn’t possible to duplicate Production due to cost reasons. Therefore the QA environment will be a scaled down instance of Production. So if the Production environment consists of 5 web servers that are load balanced and 8 databases servers that consist of replication and offline database instances used for reporting – you might want to create a minimal hardware environment without the load balancing or full replication in place. Therefore you might just have 1 web server and 1 database server that contains all the database and table instances with replication contained on that one server instead of across numerous servers. Although this reduces the similarities between QA and Production, it is a more cost effective approach when several QA environments are required. Normally the QA Department will identify that issue as a business risk and offset that risk with having a Staging environment that will be closer in configuration to the Production environment that QA can log into and perform Production load tests to address the performance of the application & hardware when under load testing and using load balancers.

Now the QA environment will normally consist of several duplicate environments. So if you have scaled down your Production environment to 1 web server and 1 database server for QA, then you will most likely want to have several of these environments so that you can be testing various projects concurrently that have different delivery dates without impacting the testing of another project. So if you look at this image we show a basic diagram of how you may have a QA1 and QA2 environment (there could be more). So why can’t I test multiple projects on one environment and reduce the hardware environments? You can, however you run into a coordination effort within the test team when it comes to shared processes and data. Maybe you have Project 1, 2, 3 and 4 that are going to be deployed to Production next week but Project 5 is a large project that will not be delivered to Production for another two weeks. For projects that make significant changes to some processes and/or database schemas, and that project is not being delivered to Production the same time as the other projects currently being tested, then you really need a separate environment to host that project. Otherwise you invalidate the testing of the other projects due to changes in the environment. Normally you want to have several QA environments and target projects to these servers based on functional changes and delivery dates. It is also very important that your database server be robust enough to hold a very large amount of data if not a full copy of the Production database. Numerous issues will always revolve around data and if you don’t have a robust database you may completely miss defects in your system. This environment needs access to the version control system to check out code only (changes to code should not be made in the QA environment, only in development – follow proper standards and processes). The developers should not have access to this environment to avoid any changes being made (security/code integrity issue). Access should be restricted to guest access for developers, but full access to the testers.

QA Tools – in regards to tools I specialize in and use the Quality Center, LoadRunner and QuickTest Pro so I will address environments based on these tools. This will give you a basis on configurations that you can apply to other tools:

Quality Center – This is a Test Management tool and is normally a single server meeting the specs specified by the installation guide. I normally have the Quality Center software and the Quality Center database (I typically use Microsoft SQL Server) installed on the same server. If you have a robust server you can avoid having to purchase multiple servers. This will reduce cost. The Quality Center then connects to all the various PCs of the users accessing the tool. Normally this server is behind the firewall.

LoadRunner – normally I like to have 3 servers for this setup – one main server hosting the full LoadRunner installation and where the Controller is run from. Then I like to have a minimum of two servers used by VUGen to off load the scenarios that are going to be executed. These servers take care of connecting to the farm of servers that are going to be load, stress and/or performance tested. Likewise it helps to keep the data processing being collected during the load tests, off the server that is running the Controller for more accurate monitoring of the load test. The server hosting the Controller is normally behind the firewall; however the Virtual User Generators may or may not be behind the firewall depending on the type of load testing being performed.

QuickTest Pro is installed on individual PCs so there is no special setup or server needed. If a Concurrent License Server is being used, I would just install that on the same server as the Quality Center to save money, because this “server software” is minimal in resources and a very small footprint on this server. Likewise QuickTest Pro needs to be able to connect to the Quality Center to save, open and run test scripts so having the Concurrent License Server software running on this server helps to eliminate any access issues with the Quality Center.

Staging Environment – this is an isolated environment that is as close to the Production environment as possible. This environment is used primarily for two purposes:
1.) User Acceptance Testing (UAT) where the business user(s) log in to verify the functionality before it goes live in Production.
2.) Performance Testing.
If the Production environment is too large to duplicate here due to cost reasons, then create a scaled down version that maintains the same type of hardware specs, database and network gear (like load balancers, firewalls, routers, switches etc) but yet still provides for a very similar configuration so that you can extrapolate your results correctly. So instead of having 6 web servers with 2 load balancers, you might have 2 web servers with 1 load balancer. At least this way you can do performance testing with a load balancer with the test scenario. Likewise the database should ideally be a Production copy. Normally you would want to use a copy of the backup snapshot here. As long as the hardware specs and hardware configurations are the same as Production (just scaled down in quantity) then you should be able to do a decent extrapolation of your load/stress/performance testing results to predict how Production will perform. If this environment is not the same hardware specs and setup, then it is impossible to do any kind of extrapolation of data, and you’ll only be able to use this as a UAT environment. This environment will have access to the version control system to check out the code base being deployed to Production and is intended to have restricted access to everyone using it – it should be just like it would be in Production, however there should be NO connection to Production. I ran into this once where one environment was doing Production database replication to the Staging environment that everyone forgot about. So when I ran a performance test with LoadRunner I directly impacted Production performance since the backup database was maintaining a live connection to the Production database. By slowing down the Staging server significantly with a performance test, the backup instance could not update from Production as expected and affected the end-user’s experience on the web application. Not good. This old design had been overlooked and we then looked at moving this database replication to another Production database.

Production Environment – this is your robust Production environment that can handle high volumes of load, maintain security and have limited server and database access by appropriate personnel only (SysAdmins, DBA’s, Release Manager, and potentially architects – every environment is different on Production access; the fewer people with access maintains the best security). This environment may not have access to the version control system. If it does then the Release Manager can deploy code directly to the server, but if it does not then a build tool needs to be used to bundle the code into an install script/build file/tar file/etc. This is a great approach if you can do it because you take security to a higher level and you avoid mistakes made by individuals executing their various tasks in the Production environment. In an environment where a build tool is not used, then you typically have at least 3 individuals in Production doing tasks for every release – a release engineer, SysAdmin and a DBA. With a build tool, all these tasks are scripted and it becomes an executable script that a SysAdmin will move to the Production environment and run. Simple and reduces the risk of error made by numerous people working directly on the Production servers.

In software development it is critical to maintain a historical trace of the changes your software programs go through and what code is being moved to what environment.  Software developers (programmers) need the ability to go back in time whether it is recent or in the past, to a specific version of the code if necessary.  Even Test Engineers implementing test automation scripts have the same need.  This is only possible when using a version control tool wrapped in process.  This document is a simple overview and explanation of the key principles involved in implementing a simple yet effective Configuration Management process.

Configuration Management encompasses four key principles in software development:

1. Process Control
2. Version Control
3. Build Management
4. Release Management

Process Control
We begin with Process control because without Process and Procedures you will be working in a chaotic environment where things are done at random and you have no ability to revert to early changes in a code base.  Some people resist Process as they think it is too restrictive and doesn’t enable them to get the “job done”.  A solid process will enable people to work efficiently and in a “controlled environment” so that changes are not lost and easily accessed if needed.  Process does NOT have to be restrictive, annoying or time-consuming.  A properly structured process should be “no big deal” and work procedures should execute smoothly.

This Process should define the tools used and how code is moved across environments.  This process should be defined, documented and the tracking of code versions implemented across environments should be tracked (ideally in a tool).  Surprisingly it doesn’t take a lot of effort to implement a solid process and enable you to “mature” your software development process which helps you climb the Capability Maturity Model (CMM) levels.  I have implemented simple processes that have enabled me to pass sophisticated process audits conducted by Ernst & Young without any deficiencies on the first audit.

Processes can be as complex as you want but I discourage that as the more complex, the more effort it is to implement and easier for people to find excuses not to use it.  Keep it Simple so that it will be used!  It can be automated or manual – my current process is a mix with a lot of manual process but with some scripts to help with deployments.

Process begins with documentation!  Document your process and make the process known.  If you can’t easily explain the process then there will be a definite problem with implementing it.  With the documentation should be some simple workflow diagrams to provide a simple visual of process – some people are more visual and understand things quicker by looking at a diagram.  You can use a simple tool like Powerpoint or get more advanced with a tool like Visio.

Now you need some tools to finish the process…

Version Control
Version control is simply the ability to track specific versions of code that is being developed.  A developer will “commit” changes to the Version Control repository to save specific points in time with what the code was like.  By doing this a developer has the ability to go back in time and grab a specific version of the code in case it was decided a specific code change was not needed or all together wrong and the developer wants to start over again – this is the easiest and most sophisticated way of doing this.  The process of saving (“committing”) specific versions of code and then grabbing a specific version is known as “check-in” and “check-out”. Each time someone needs to edit a file, they check it out for exclusive editing and then check it back in to the repository when finished, thus creating a new version.

When using Version Control you can also do the following tasks:

• Roll back to a previous version of a specific file
• Compare two versions of a file
• Provides a locking mechanism so that no one else can modify the file while you are making your changes
• Create branches so that multiple developers can be developing different code at that same time
• Ability to move specific file versions from one environment to another (Dev to QA to Staging to Production)

Build Management
This part of the process can be either manual, automated or a hybrid of some automation with some manual processes – this is what I currently implement.

The term “Build Management” is defined by Wikipedia as:

     “Software build management is the defined and controlled process of delivering the right artifacts efficiently.”

What is important here is that you need to define and control what versions of software are being moved from environment to environment – and most importantly what is being tested by the software testers is the same code being pushed to Production.

Build Management Process:

• Trunk – A typical process is to begin with a base foundation of a code base.  This is known as the “Trunk” and is representative to what is actually in your Production environment.
• Branch – A developer will request a “branch” which is a copy of the “trunk” for the developer to work in.  Numerous “branches” can be created so that multiple developers can work on various projects at the same time without stepping on each other.  This is important when one project will be delivered to Production before another project; this enables the developers to work independently of each other.  A “branch” can then be provided to the testing team to be verified for its functionality (typically Functional Testing and Regression Testing are applied to a branch).  This “branch” can easily be moved to a secure QA environment for testing.  Multiple projects can be handled and tested independently this way.
• Staging Release – when a group of projects have been tested and ready to be tested at the “System/Integration Testing” level, then the verified “branches” will be merged into a transitional version of the code base that would ideally be deployed to Production.  This naming convention can be anything.  In our environment we first create a “Candidate Release” which is all the branches merged into a single code base (the “future trunk”) and tested in the QA environment (remember this process brings all the code modifications together in one code base so now the testers can conduct “System/Integration Testing”, “Performance Testing” and a final “Regression Test”).  Once this has been certified, we then create a “Stable Release” (which is a copy of this Candidate Release) for deployment to the Staging environment and ultimately the version that is moved to the Production environment.
• Trunk update – finally once the Production environment has been updated with the new code base, the “trunk” needs to be updated as the “trunk” is always representative of the code base in Production.  So this code needs to be “merged” to the trunk.  Now all existing branches being worked on that have not been deployed to Production yet need to be updated with the “trunk” in order to stay current.

“ALWAYS PROTECT THE TRUNK!”

Release Management
Release management is the execution process of the Build Management Process and the Process of moving code from one environment to another.  It is critical that security be implemented here as only a restricted few individuals should have the ability to merge code and have server access to move code from one environment to another.  Without this you lose all control as anyone would be able to move whatever version of code to another environment without anyone knowing.  This can not only create major issues in code, but can make debugging a Production issue a nightmare and then when someone follows the actual process and moves new “controlled code” to Production they will end up replacing the code that bypassed the process because it was never merged to the trunk properly and gets lost.

Software Configuration Management Tools
I currently use the Open Source Tool “Subversion (SVN)”.  We use this in a Linux environment and primarily version control the following types of files: PHP, Javascript, HTML, and jpg images (however there are many other files under our configuration management tool but this is the majority of files we manage.

Open Source Tool:

• Subversion (SVN)