Best Practices of Continuous Integration and Testing

A good video summary of what CI is, can be found here

The Practices of Continuous Integration include the following 11 detailed in Martin Fowler's
Practices of Continuous Integration
This page of notes is still being compiled asat: 8 Apr 2016 

Practice 1 – Maintain a Single Source Repository

Using Version Control Systems (VCS') such as ...

Practice 2 – Automate - Version-Control / Build / Deploy / Test

Using CI tools such as

• Anthill: Now distributed by IBM and implemented in Java. It has a proprietary license.
• Bamboo: Atlassian’s CI server isn’t open source, but the Java code is provided to customers for free. It is also free for open source and community projects to use.
• Buildbot: An open source offering - Brian Warner’s Python based CI tool.
• Continuum: An Apache build tool developed in Java.
• Cruise Control: First shipped in 2001, CruiseControl can fairly claim to be the server that started it all. It was originally developed by employees at ThoughtWorks, but is now an independent project. It's open source, and has versions written in Java, C# and Ruby.
• easyCIS: Freeware CI server developed by Václav Zahradník on top of the .NET framework.
• FinalBuilder Server/Continua CI: CI server from VSoft written in C#.
• Go: ThoughtWorks' continuous delivery tool written in Java.
• Gump: The Apache Software Foundation's first CI tool. Written in Python it supports Apache Ant and Maven, amongst others.
• Hudson: Java CI server now managed by the Eclipse Foundation.
• Jenkins: An open source tool written in Java. The project was forked from Hudson after a dispute with Oracle and is now led by CloudBees.
• Mojo: OpenMake’s free Java-based CI server.
• Parabuild: Java-based server from Viewtier Systems.
• Pulse: Java-based CI server from Zutubi.
• QuickBuild: PMEase’s Java-based CI tool.
• Sin: CI tool written in C# by Casper Hornstrup. It supports Subversion and is licensed with GPL.
• TeamCity: Java-based CI server from JetBrains, makers of IntelliJ.
• Team Foundation Server: Microsoft’s CI sever, widely used by .NET developers.
• TravisCI: Hosted continuous integration service for the open source community.
• Zed: Java-based CI Server from Hericus Software.

Controlling tools to:

Build

• Maven - is a "build management tool" (orchestrator) for defining how your .java files get compiled to .class 'es, packaged into .jar (or .war or .ear) files, then (pre/post)processed with tools to manage the CLASSPATH, and all sorts of other tasks required to build a project.
• Ant:
• Ruby:
• Rake:
• Powershell:
• WinAutomation:

The following is an example hypothetical build process you might use with a simple software system

1. Get the source. You may need to download or fetch the source from a source code repository. For this, you might need to know the tag or version of the source code you want to build.
2. Prepare a build area. You will probably want to create a set of directories, perhaps according to some standardized directory layout.
3. Configure the build. In this step, you will determine what optional components can be built based on the current environment. You might want to set build numbers and version numbers to be included in the build.
4. Validate the source code. You may have a standard style guide and you wish to ensure all code conforms to this before you build a release.
5. Compile the source code
6. Build the compiled code into libraries potentially including non-code resources such as properties, images and sound files.
7. Run the system's tests to validate the build.
8. Build the documentation for the software. This may range from something as simple as collecting text files up to processing content through some form of publishing system to produce the documentation in its final form
9. Package up all of the components of the software – code, resources, images, documentation, etc. – into a deployable package. You might need to produce several packages in different formats for different target users
10. Deploy the software to some standard location for use or distribution

Deploy

• Powershell: 
• Octopus: normal
• Ansible: a radically simple IT automation engine
  
  • Used for cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs.
  • It uses no agents and no additional custom security infrastructure, so it's easy to deploy - and most importantly, it uses a very simple language (YAML, in the form of Ansible Playbooks) that allow description of automation jobs in a way that approaches plain English.
  • Ansible manages Linux/Unix machines using SSH by default.
  • From version 1.7, Ansible also supports Windows machines. Native PowerShell remoting is used, rather than SSH.
    Ansible will still be run from a Linux control machine, and uses the “winrm” Python module to talk to remote hosts.
• dssd

Test

• Ranorex: 
• BOLD: normal
• BOLD: normal

Jenkins Tutorial videos
Jeff Jantz - Continuous Integration with Jenkins on Amazon EC2 [1 / 5]
Jeff Jantz - Installing and configuring a Jenkins CI Server on an Amazon EC2 [2 / 5]
Jeff Jantz - Configuring Jenkins on Amazon EC2 [3 / 5]
Jeff Jantz - Continuous Integration with Jenkins on Amazon EC2 [4 / 5]
Jeff Jantz - Continuous Integration with Jenkins on Amazon EC2 [5 / 5]
Sriram Gunda - Installing Jenkins as a Windows Service

TeamCity Tutorial videos
TeamCity UserGuide - Introduction [1 / 9]
TeamCity UserGuide - Web Interface Overview [2 / 9]
TeamCity UserGuide - Build Configurations [3 / 9]
TeamCity UserGuide - Build Results [4 / 9]
TeamCity UserGuide - Working with Tests [5 / 9]
TeamCity UserGuide - Code Analysis [6 / 9]
TeamCity UserGuide - Working with Changes [7 / 9]
TeamCity UserGuide - Personal Builds [8 / 9]
TeamCity UserGuide - Notification and User Profiles [9 / 9]
TeamCity UserGuide - Getting started with TeamCity
 

Practice 3 – Make Your Build Self-Testing

Here is just a short excerpt of tools and frameworks:

• UI tests: Selenium, Robot Framework, Protractor, WinAutomation
• Service tests: JBehave, FitNesse, SoapUI, JMeter Performance Tests
• Unit tests: JUnit, TestNG, Mockito (Mock Framework)

Practice 4 – Everyone Commits To the Mainline Every Day

Practice 5 – Every Commit Should Build the Mainline on an Integration Machine

Since branching and merging has become easier with Git, there are various workflows available on how to develop software with Git. The most popular one is called GitFlow.
Here is a list of various Git workflows and comparisons that you can examine, in case you want to learn more on the topic:

• Git Flow, Vincent Driessen
• GitHub Flow
• GitLab Flow
• Atlassian Git workflow comparison

Practice 6 –  Fix Broken Builds Immediately

Practice 7 – Keep the Build Fast

Practice 8 – Test in a Clone of the Production Environment

Practice 9 – Make it Easy for Anyone to Get the Latest Executable

Practice 10 – Everyone can see what’s happening

Practice 11 – Automate Deployment

To deploy your software you should use a single deployment script.
This deployment script will be used for deploying to every environment.
As a result you automatically test the script before doing an actual production deployment.

Coming Up To Speed with The Open Source World and some of its players

CAVEAT: Comments in this post are just personal opinions gathered from various sources

Also see my pages on: Summary of Framework Terms all in one place
and: Testing Methodology Summary

• 
  

• .Net - think of it as Microsoft’s “me too” version of Java, in other words: machine-independent compiled code.

• Apache Ant - is a Java based software tool for automating software build processes
  (also see Make and Rake)
•  

• Ansible - a radically simple IT automation engine used for cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs.
•  
• Apache -  is an HTTP Server, serving HTTP.
  (Apache Tomcat is a Servlet and JSP Server serving Java technologies)
•  
• C# (pronounced C-sharp) - a relatively new programming language designed by Microsoft for a wide range of enterprise applications that run on the .NET Framework. An evolution of C and  C++, the C# language is simple, modern, type safe and object oriented.

• Chef - is a configuration management tool written in Ruby and Erlang. It uses a pure-Ruby, domain-specific language (DSL) for writing system configuration "recipes"
•  
  
• CI - Continuous Integration - an automated build/test system that continuously validates code changes as hey are submitted to a code repository.
  Examples of CI server solutions include:
• Jenkins - The basic functionality of Jenkins is to execute a predefined list of steps based on a certain trigger.
  The trigger might for example be an change in a version control system or a time based trigger, e.g., a build every 20 minutes.
  The list of steps can for example include:
• perform a software build with Apache Maven or Gradle
• Run a shell script
• Archive the build result
• Initiate integration tests
• Bamboo - Bamboo is the product of Atlassian. When compared to Jenkins, Bamboo is sexy. It is easier to use and it looks better. Usability is one of the first differences one would notice. Moreover, it is well-integrated with the rest of Atlassian’s products (JIRA, Confluence, BitBucket, etc).
• TeamCity - 
• 
  
• Docker is a Linux based open-source project that automates the deployment of applications inside software containers, by providing an additional layer of abstraction and automation of operating-system-level virtualization on Linux
•  
• .EAR (Enterprise archive) - can contain any J2EE object - typically contains all of the components that make up a particular J2EE application
•  EJB - Enterprise  Java Beans
• JMS - Java Message Service API
• Servlets and all other web objects of a WAR
•  
• EJB - Enterprise JavaBeans are J2EE software components that exist and operate in a container. The container provides all the services an EJB needs to interact with other EJBs
  see below:OC4J
•  
• Apache Felix Framework subproject is an implementation of the OSGi R6 core framework specification
•  
• Git - is a widely used source code management system for software development. It is a distributed revision control system with an emphasis on speed, data integrity, and support for distributed, non-linear workflows.
  It comes in two main forms:
  
  1. GitLab - self hosted
  2. GitHub - cloud hosted AaaS

• Glassfish - is the reference implementation of J2EE an open-source application server project started by Sun Microsystems for the Java EE platform and now sponsored by Oracle.
  GlassFish runs straight on top of the Apache Felix and uses a derivative of Apache Tomcat as the servlet container for serving Web content
•  
• GNOME - (GNU Network Object Model Environment, pronounced gah-NOHM) is a graphical user interface (GUI) and set of computer desktop applications for users of the Linux computer operating system. It's intended to make a Linux operating system easy to use for non-programmers and generally corresponds to the Windows desktop interface and its most common set of applications.
•  
•  IntelliJ IDEA - JAVA IDE
•  

• .JAR (Java Archive) - a group of .class files 
• 
  
• JAVA - one of the most widely adopted programming languages, used by some 9 million developers and running on 7 billion devices worldwide. It’s also the programming language used to develop all native Android apps. Java’s popularity with developers is due to the fact that the language is grounded in readability and simplicity. Java has staying power since it has long-term compatibility, which makes sure older applications continue to work now into the future.
•  
• JavaScript – not to be confused with Java – is another one of the world’s most popular and powerful programming languages, and is used to spice up web pages by making them interactive.
•  

• Jenkins - cross-platform, continuous integration and continuous delivery application. It is a server-based system running in a servlet container such as Apache Tomcat. It supports SCM tools including AccuRev, CVS, Subversion, Git, Mercurial, Perforce, Clearcase and RTC, and can execute Apache Ant and Apache Maven based projects as well as arbitrary shell scripts and Windows batch commands
  
  There are similar tools for other programming frameworks such as:

• Buildbot — a Python system to automate the compile/test cycle to validate code changes.
• Tox — an automation tool providing packaging, testing and deployment of Python software.
• Travis-CI — a distributed CI server which builds tests for open source projects for free.
• Django-Jenkins — Django (Python) Web Framework integration with Jenkins.

•  
• keytool - is a key and certificate management utility. It allows users to administer their own public/private key pairs and associated certificates
•  
• GNU Make is a tool which controls the generation of executables and other non-source files of a program from the program's source files.
  Make gets its knowledge of how to build your program from a file called the makefile, which lists each of the non-source files and how to compute it from other files.
  Make enables the end user to build and install your package without knowing the details of how that is done -- because these details are recorded in the makefile that you supply.
•   
  
• Maven - is a software project management and comprehension tool. Based on the concept of a project object model (POM), Maven can manage a project's build, reporting and documentation from a central piece of information. An Orchestration tool if you will ... bringing it all together.
•  
•  OC4J - stands for Oracle Containers for J2EE, Oracle's original lightweight app server.
  It is a slimmed down version of Oracle Application Server, written entirely in Java executing on a standard JDK VM.
  Oracle WebLogic Server (WLS) falls into the "Strategic" bucket while OC4J falls into the "Continue and Converge" bucket  see above:EJB  see Ref1:  see Ref2:
• 
  

• The OSGi Framework. OSGi defines a dynamic module system for Java™. The OSGi service platform has a layered architecture, and is designed to run on various standard Java profile

• Open AM - started as Sun Directory Server Access Management Edition (DSAME) in early 2000s. After many years and (at least) three rebrandings, the product was finally released as OpenSSO. That’s where Oracle struck and killed the product.
  ForgeRock has picked it up. ForgeRock Access Management is built from the OpenAM project. 
• OpenID Connect is a standard. OpenAM is a product which amongst many other things, implements the OpenID Connect standard 
• OpenAM is formally Java 6, which is a problem in itself as Java 6 does not have any public updates for almost two years. Worse is that bulk of the OpenAM code is still efficiently Java 1.4 or even older.
• OpenAM is huge. It consists of approx. 2 million lines of source code which is also quite complicated.
• OpenAM is in fact (at least) two somehow separate products. There is “AM” part and “FM” part. And these two were not integrated in the cleanest way.
• OpenAM debugging is a pain. It is almost uncontrollable, it floods log files with useless data and the little pieces of useful information are lost in it.
• OpenAM is still using obsolete technologies such as JAX-RPC. JAX-RPC is a really bad API

• Orbeon Forms - is an open source, standard-based web forms solution, which includes:  
• Form Builder: a browser-based WYSIWYG form authoring tool. 
• Form Runner: a runtime environment which facilitates the deployment and integration of large and complex formsIt implements the W3C XForms standard and is available in an open source Community Edition, as well as a commercially supported Professional Edition
•  
• Orchestrator - "Microsoft System Center 2012 Orchestrator provides a simplified way of building complex automation.
  Rather than writing multiple lines of PowerShell code to determine whether  specific events have occurred and taking actions based on those events, you use Orchestrator's canvas, on which you can drag and drop GUI objects relating to monitored events, configure those objects to issue alerts and connect to specific action objects in relation to resolving the work events. Instead of multiple lines of what sometimes seems to be arcane PowerShell code, you can accomplish the same tasks by using a drag-and-drop process that might take you all of 30 seconds.
  Orchestrator can also build upon already developed PowerShell Scripts by wrapping them in advanced pre and post processing events, extending their functionality and reducing human intervention in operation of those scripts.
  
  Orchestrator is a complete solution that goes well beyond basic automation through drag and Drop scrpting.
  Rather than waiting for a member of the support team to raise a job, you can automate the process entirely through the use of an Orchestrator runbook that is triggered by an event or condition (i.e an email is received or a file arrives in a folder), raises a job in a job-tracking system, runs a complex operation to resolve the work event, adds outcome information to the job in the job-tracking system, and then closes that job. The alert event is resolved, the job is logged and closed, and everything is completed without direct human intervention."
•  
• Powershell - is the most powerful automation tool that Microsoft has to offer, and its both a shell and a scripting language.
•  
•  Puppet - an open-source configuration management tool. It runs on many Unix-like systems as well as on Microsoft Windows, and includes its own declarative language to describe system configuration.
•  
• PuTTY - is a free and open-source terminal emulator, serial console and network file transfer application. It supports several network protocols, including SCP, SSH, Telnet, rlogin, and raw socket connection.
•  
• Python - is a general purpose programming language which is simple and incredibly readable since closely resembles the English language. It’s a great language for beginners, all the way up to seasoned professionals. Python bumped Java as the language of choice in introductory programming courses.
•  

• Rake - Rake is a Domain Specific build language, programmed in Ruby and similar in purpose to make and ant.
•  
• Samba - is a suite of Unix applications that speak the Server Message Block (SMB) protocol that provide seamless file and print services to SMB/CIFS clients." Samba allows for interoperability between Linux/Unix servers and Windows-based clients
•  
• SCOrch - stands for System Center Orchestrator. Orchestrator serves the role as the “automater” in Microsoft’s System Center suite, and it’s used to create “Runbooks” which are basically workflows that are used to complete a series of tasks.
  While there are many options to build a Runbook through the GUI, sometimes you might want to be able to do more with your Runbooks.
  This is where integrating your own PowerShell scripts can come in handy.
•  
• Spring is a framework that helps you to "wire" different components together.
  What Spring does is to wire the classes up by using a XML file, this way all the objects are instantiated and initialized by Spring and injected in the right places (Servlets, Web Frameworks, Business classes, DAOs, etc.)
  Spring is the "glue" used to integrate different frameworks and your own code.
•  
• Spring Security is a Java/Java EE framework that provides authentication, authorization and other security features for enterprise applications.
  There are related Spring projects for each of:

• Spring LDAP
• Spring Security OAuth
• Spring Security Grails
• Spring Security SAML
• Spring Security Kerberos

•  
• Subversion - Apache Subversion (often abbreviated SVN, after the command name svn) is a software versioning and revision control system distributed as free software. See also GiT
•  
•  Tarantula is a free tool for managing software testing in agile software projects. It is licensed as open source software.

• Tomcat - is an application Servlet and JSP Server serving Java technologies
  - see also Apache
•  

• Vagrant is computer software that creates and configures virtual development environments.
  It can be seen as a higher-level wrapper around virtualization software such as VirtualBox, VMware, KVM and Linux Containers (LXC), and around configuration management software such as Ansible, Chef, Salt, and Puppet.
  Since version 1.6, Vagrant natively supports Docker containers, which in some cases can serve as a substitute for a fully virtualized operating system.

• VirtualBox - supports the creation and management of guest virtual machines running versions and derivations of Windows, Linux, BSD, OS/2, Solaris, Haiku, OSx86 and others, and limited virtualization of OS X guests on Apple hardware.
  It may be installed on a number of host operating systems, including: Linux, OS X, Windows, Solaris, and OpenSolaris.
  There are also ports to FreeBSD and Genode
•  
• .WAR (Web Archive) - Group of web application components such as .
• JSP - Java Server Pages
• servlets - a Java class that is used to extend the capabilities of servers
• .HTML pages
• .CSS - Cascading Style Sheets
• .JS - JavaScript files

• WebLogic Server implements Java 2 Platform, Enterprise Edition (J2EE) which is the standard platform for developing multi-tier Enterprise applications based on the Java.
•  
• A WebLogic Server^TM J2EE application consists of one of the following modules or applications running on WebLogic Server:

• Web application modules—HTML pages, servlets, JavaServer Pages, and related files.
• Enterprise Java Beans (EJB) modules—entity beans, session beans, and message-driven beans. See Enterprise JavaBean Modules 
•  Connector modules—resource adapters. See Connector Modules.
•  Enterprise applications—Web application modules, EJB modules, and resource adapters packaged into an application.

• WinSCP (Windows Secure Copy) - is a free and open-source SFTP, FTP, WebDAV and SCP client for Microsoft Windows. Its main function is secure file transfer between a local and a remote computer.

•  WinAutomation - is the most advanced Windows Automation app, featuring a Macro Recorder and a Web Recorder. With WinAutomation you can teach your computer to perform your repetitive tasks automatically.

•  
• ZK is an open-source Ajax Web application framework, written in Java, that enables creation of graphical user interfaces for Web applications with little required programming knowledge.
  
  The core of ZK consists of an Ajax-based event-driven mechanism with over 123 XUL and 83 XHTML-based components and a mark-up language for designing user interfaces.
  Programmers design their application pages in feature-rich XUL/XHTML components, and manipulate them upon events triggered by end user's activity.
  It is similar to the programming model found in desktop GUI-based applications.
• It's important to note that ZK does not use the standard web request <=> response mechanism and DOES NOT send form fields to the server using GET + query parameters or one POST request. Instead, AJAX requests are sent to the server to update the internal state of each screen widget. At the browser, ZK only downloads a JSON description of the web page and uses a client renderer to turn that into a UI.
• In addition to component-based programming in a manner similar to Swing, ZK supports a mark-up language for rich user interface definition called ZUML