ASI

This blog article might seem to many people as a very basic topic and the content obvious. However I’m surprised at how often I still come across technology departments that do not have proper hardware environments for software development that maintains code integrity, security, isolation and Production similar configurations. Let’s discuss the basic high-level configuration concept of a proper software development environment.  Note that this discussion is a high-level discussion and will not discuss details about actual hardware specs and layouts since every development shop has different requirements.

So the basics of a proper software development environment are going to consist of a Dev Environment, QA Environment, Staging Environment and Production Environment, with a version control server supporting these environments that contains your code. The only exception to this is your Production environment – that environment may or may not have direct access to the version control server for security reasons. If it does not have access to the version control server then you will not be able to export your Trunk to the Production servers so you will need to have a build process that enables you to build install scripts that install the new version of code into Production (more on that later). Let’s look at an overview of each environment:

Dev Environment – this environment is owned by the software developers (programmers). This is an environment that enables them to work on new code and do anything necessary to the environment to support their development efforts. This can include re-configuration of the environment, dumping new data, changing file access and even crashing their environment. Normally this environment is configured “similar” to the Production environment, but scaled down significantly (like they may have 1 web server and 1 database server versus 5 web servers and 8 database servers like in the Production environment). Likewise the server(s) the programmers are developing on are actually a shared hardware environment with each developer having their own “area” to work in by virtualizing the server and providing each develop their own private slice to develop in (this is a great configuration as they can work independent of the other developers and not worry about bringing them down if they want to start and stop processes and likewise reduce server cost in reducing the number of servers needed to support the developers). This environment would be for developers only to work in and even though it may be virtualized, there still may be many development servers based on the size of the development team. The environment is considered dynamic and unstable due to the constant changes that occur in the process of developing code to support new functionality for an application. This environment needs access to the version control system to check in & out code.

QA Environment – this environment is owned by the software testers. This is an environment that enables the software testers to work in a “controlled” environment that is as close to a Production configuration as possible. Since a Production environment normally consists of numerous servers, databases and network gear it normally isn’t possible to duplicate Production due to cost reasons. Therefore the QA environment will be a scaled down instance of Production. So if the Production environment consists of 5 web servers that are load balanced and 8 databases servers that consist of replication and offline database instances used for reporting – you might want to create a minimal hardware environment without the load balancing or full replication in place. Therefore you might just have 1 web server and 1 database server that contains all the database and table instances with replication contained on that one server instead of across numerous servers. Although this reduces the similarities between QA and Production, it is a more cost effective approach when several QA environments are required. Normally the QA Department will identify that issue as a business risk and offset that risk with having a Staging environment that will be closer in configuration to the Production environment that QA can log into and perform Production load tests to address the performance of the application & hardware when under load testing and using load balancers.

Now the QA environment will normally consist of several duplicate environments. So if you have scaled down your Production environment to 1 web server and 1 database server for QA, then you will most likely want to have several of these environments so that you can be testing various projects concurrently that have different delivery dates without impacting the testing of another project. So if you look at this image we show a basic diagram of how you may have a QA1 and QA2 environment (there could be more). So why can’t I test multiple projects on one environment and reduce the hardware environments? You can, however you run into a coordination effort within the test team when it comes to shared processes and data. Maybe you have Project 1, 2, 3 and 4 that are going to be deployed to Production next week but Project 5 is a large project that will not be delivered to Production for another two weeks. For projects that make significant changes to some processes and/or database schemas, and that project is not being delivered to Production the same time as the other projects currently being tested, then you really need a separate environment to host that project. Otherwise you invalidate the testing of the other projects due to changes in the environment. Normally you want to have several QA environments and target projects to these servers based on functional changes and delivery dates. It is also very important that your database server be robust enough to hold a very large amount of data if not a full copy of the Production database. Numerous issues will always revolve around data and if you don’t have a robust database you may completely miss defects in your system. This environment needs access to the version control system to check out code only (changes to code should not be made in the QA environment, only in development – follow proper standards and processes). The developers should not have access to this environment to avoid any changes being made (security/code integrity issue). Access should be restricted to guest access for developers, but full access to the testers.

QA Tools – in regards to tools I specialize in and use the Quality Center, LoadRunner and QuickTest Pro so I will address environments based on these tools. This will give you a basis on configurations that you can apply to other tools:

Quality Center – This is a Test Management tool and is normally a single server meeting the specs specified by the installation guide. I normally have the Quality Center software and the Quality Center database (I typically use Microsoft SQL Server) installed on the same server. If you have a robust server you can avoid having to purchase multiple servers. This will reduce cost. The Quality Center then connects to all the various PCs of the users accessing the tool. Normally this server is behind the firewall.

LoadRunner – normally I like to have 3 servers for this setup – one main server hosting the full LoadRunner installation and where the Controller is run from. Then I like to have a minimum of two servers used by VUGen to off load the scenarios that are going to be executed. These servers take care of connecting to the farm of servers that are going to be load, stress and/or performance tested. Likewise it helps to keep the data processing being collected during the load tests, off the server that is running the Controller for more accurate monitoring of the load test. The server hosting the Controller is normally behind the firewall; however the Virtual User Generators may or may not be behind the firewall depending on the type of load testing being performed.

QuickTest Pro is installed on individual PCs so there is no special setup or server needed. If a Concurrent License Server is being used, I would just install that on the same server as the Quality Center to save money, because this “server software” is minimal in resources and a very small footprint on this server. Likewise QuickTest Pro needs to be able to connect to the Quality Center to save, open and run test scripts so having the Concurrent License Server software running on this server helps to eliminate any access issues with the Quality Center.

Staging Environment – this is an isolated environment that is as close to the Production environment as possible. This environment is used primarily for two purposes:
1.) User Acceptance Testing (UAT) where the business user(s) log in to verify the functionality before it goes live in Production.
2.) Performance Testing.
If the Production environment is too large to duplicate here due to cost reasons, then create a scaled down version that maintains the same type of hardware specs, database and network gear (like load balancers, firewalls, routers, switches etc) but yet still provides for a very similar configuration so that you can extrapolate your results correctly. So instead of having 6 web servers with 2 load balancers, you might have 2 web servers with 1 load balancer. At least this way you can do performance testing with a load balancer with the test scenario. Likewise the database should ideally be a Production copy. Normally you would want to use a copy of the backup snapshot here. As long as the hardware specs and hardware configurations are the same as Production (just scaled down in quantity) then you should be able to do a decent extrapolation of your load/stress/performance testing results to predict how Production will perform. If this environment is not the same hardware specs and setup, then it is impossible to do any kind of extrapolation of data, and you’ll only be able to use this as a UAT environment. This environment will have access to the version control system to check out the code base being deployed to Production and is intended to have restricted access to everyone using it – it should be just like it would be in Production, however there should be NO connection to Production. I ran into this once where one environment was doing Production database replication to the Staging environment that everyone forgot about. So when I ran a performance test with LoadRunner I directly impacted Production performance since the backup database was maintaining a live connection to the Production database. By slowing down the Staging server significantly with a performance test, the backup instance could not update from Production as expected and affected the end-user’s experience on the web application. Not good. This old design had been overlooked and we then looked at moving this database replication to another Production database.

Production Environment – this is your robust Production environment that can handle high volumes of load, maintain security and have limited server and database access by appropriate personnel only (SysAdmins, DBA’s, Release Manager, and potentially architects – every environment is different on Production access; the fewer people with access maintains the best security). This environment may not have access to the version control system. If it does then the Release Manager can deploy code directly to the server, but if it does not then a build tool needs to be used to bundle the code into an install script/build file/tar file/etc. This is a great approach if you can do it because you take security to a higher level and you avoid mistakes made by individuals executing their various tasks in the Production environment. In an environment where a build tool is not used, then you typically have at least 3 individuals in Production doing tasks for every release – a release engineer, SysAdmin and a DBA. With a build tool, all these tasks are scripted and it becomes an executable script that a SysAdmin will move to the Production environment and run. Simple and reduces the risk of error made by numerous people working directly on the Production servers.