Automating deployment of a Java project to Azure using AppVeyor

The Automating deployment of a Java project to Azure using AppVeyor

Note on the title, after much debate with my colleague (looking at you Alastair) I would like to note that in this post I do not want to engage on the subtle differences between Continuous Deployment, Continuous Delivery, and Continuous Integration. This post is purely about automating the deployment of a Java Project housed in a Git repository to Azure using AppVeyor. If you would like engage on these subtle differences you can go here.

This is my first experience in setting up Continuous Deployment (CD) automatic deployment for a Java Project.
The project, in a nutshell, is a Vaadin web application, with a MS SQL Database as a persistence layer hosted on Azure.

Project environment

First off we need to define the Project Environment a little better:

What Tech
Front-end Development IDE NetBeans (Java Project)
Front-end UI framework Vaadin
Vaadin add-ons TouchKit & Charts
Front-end Build tool Maven
Source Control BitBucket (One Repository containing the Front-end project and the Database project)
Database project Development IDE Visual Studio (Database Project)
Database project Build tool MSBuild
CI Server/Service AppVeyor (i.e. Windows Environment)
Front-end deployment Azure App Service
Database project deployment Azure SQL Database

I will be focusing solely on the development branch build-and-deploy. For simplicity’s sake I will not discuss automated unit testing nor I am setting up any of the other builds. (i.e. Nightly-Metric-Build, Release-Candidate-Build etc.)

Where do we start?

Setting up CI the automatic deployment begins with source control.

Every time a push is made to our project’s development branch on BitBucket (only after peer review people!) the CI automatic build server (AppVeyor) must be informed.
It will then pull down the latest version of the development branch code for the repository and…

  • build it
  • package it
  • deploy it.

Simple! Right?!

Lets jump in.

Triggering the build


Before we can build the project we have to trigger the build on AppVeyor.

AppVeyor integrates easily with BitBucket with a basic point and click integration setup.
If you are a complete newbie to AppVeyor you can get some guidance in the AppVeyor documentation, but rest assured that this part is not the difficult.

AppVeyor creates an Integration Service entry in your BitBucket repository that will notify your AppVeyor Project when pushes are made to your Bitbucket Repository.
I prefer removing this Integration Service entry and replacing it with a BitBucket WebHook to notify my AppVeyor Project.
But as of this writing both methods work and it’s up to you which method you prefer.

The thing to note is the that you should make your AppVeyor Project aware of which branch it should build, package and deploy:

  1. Go to your AppVeyor Project’s Settings.
  2. On the General tab find the Default branch field, set it to ‘develop’, or the name of your development branch.
  3. The next field is the Branches to build dropdown box, select: ‘Only branches specified below’.
  4. Click the Add branch link-button, enter ‘develop’, or the name of your development branch, into the textbox that appears.
  5. Scroll to the bottom of the page and click the Save button.

The AppVeyor Project will now be triggered each time a push is made to development branch in BitBucket.

The Build


Remember our repository contains two projects: a Java Project and a Visual Studio Database Project.

Our build environment knows what to do with the Visual Studio Database Project as the AppVeyor Visual Studio VM is setup for Visual Studio Projects, but the build environment has no idea what to do with the Java Project so we need to perform some extra steps:

  1. Prep: i.e. Install everything required to build the Java Project build. (Chocolatey to the rescue!);
  2. Override: the Java Projects .properties files to contain the deployment environment properties and not the development environment properties. (I created a custom PowerShell script to do this but there may be better ways);
  3. Build: Tell our build environment to create the Java Web application package for our Java project. (Maven to the rescue!).

Prepping the build environment for Java

If you have not heard of Chocolatey, now you have! Chocolatey is a linux style package manager for Windows built on Nuget. If you ever needed to install anything on a Windows environment Chocolatey is your friend!

The Build Worker has Chocolatey installed so we can easily call on the power of choco via PowerShell.
And lucky for us the only thing we need to install via Chocolatey is the ‘Maven’ Build tool choco-package, and add the install folder locations of Maven and Java to the environment Path variable.

  1. Go to you AppVeyor project’s Settings.
  2. On the Environment tab, find the Build worker image field and select ‘Visual Studio 2015’.
  3. Find the Install script field, select the PS button on the right.
  4. Insert the following script
  ### Unattended Install of Maven using chocolatey
  choco install maven -y -f
  ### Add the Maven and Java directories to path variable.
  ### This is important! And note the version numbers you enter should match the version number installed.
  ### Otherwise the build will fail. i.e. if Java or Maven is updated you must update this script!
  $env:Path += ";C:\Tools\apache-maven-3.3.3\bin\;C:\Program Files\Java\jdk1.8.0\bin"
  1. Click the Save button at the bottom of the page.

TIP: To find the correct folders to add to your Path variable you may need to Remote Desktop into your Build Worker. You can do that by adding the following PowerShell script to the Init Script field:

$blockRdp = $true; iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/enable-rdp.ps1'))

When the build starts the Build Worker Remote Desktop credentials will be displayed in the build output.

NOTE: The build will pause until a file on the desktop is deleted, therefore the above script should be commented out when you want the build to be unattended (read automatic).

Override the Java .properties files

In our Java project we have two .properties files, config and email, which are located in the src/main/resources/ folder. Depending on your project setup both the files and their location may be different.

The following script, named OverWriteProperties.ps1, is located in the root of our repository, and overwrites the .properties files with new values sourced from environment variables. The last two lines, which are commented out, allow you to do a sanity check by displaying the generated file’s contents in the build output. If you run into problems, simply uncomment these lines and you will be able to confirm your properties.

# Override config.properties file
$configPropertiesPath = "src/main/resources/config.properties"
Set-Content $configPropertiesPath "db.host=$env:property_override_config_db_host"
Add-Content $configPropertiesPath "db.port=$env:property_override_config_db_port"
Add-Content $configPropertiesPath "db.name=$env:property_override_config_db_name"
Add-Content $configPropertiesPath "db.user=$env:property_override_config_db_user"
Add-Content $configPropertiesPath "db.password=$env:property_override_config_db_password"
Add-Content $configPropertiesPath "db.driver=$env:property_override_config_db_driver"

# Override email.properties file
$emailPropertiesPath = "src/main/resources/email.properties"
Set-Content $emailPropertiesPath "email.host=$env:property_override_email_host"
Add-Content $emailPropertiesPath "email.port=$env:property_override_email_port"
Add-Content $emailPropertiesPath "email.from=$env:property_override_email_from"
Add-Content $emailPropertiesPath "email.user=$env:property_override_email_user"
Add-Content $emailPropertiesPath "email.password=$env:property_override_email_password"
Add-Content $emailPropertiesPath "email.enabled=$env:property_override_email_enabled"

# Output contents of .config files
$config = Get-Content $configPropertiesPath
$email = Get-Content $emailPropertiesPath

Write-Output $configPropertiesPath
Write-Output $emailPropertiesPath

#Write-Output $config
#Write-Output $email

You can now overwrite the development environment properties during your build by setting the corresponding environment variable on the Environment tab in your AppVeyor settings.

NOTE: The script assumes that you will override all properties, i.e. you have to set a environment variable for each property in the .properties file.

To make life a little easier for our software developers I made two further changes to the project.

  1. I made a rule that .properties files do not get committed to the repository. (i.e. .properties was added to our .gitignore file)
  2. I created an environment property setup script for each of the devs. This script sets the environment variables for a specific developer’s environment and creates the appropriate .properties files for the project to run on their local machine. There is one script for each developer in the team, dev1_CreateProperies.ps1, dev2_CreateProperties.ps1 etc. These scripts are also located in the root of the repository and look like this: (The variable values in [] should be replaced with values appropriate for the specific dev)
# set environment variables for override script
   # Set config.property file values
   $env:property_override_config_db_host="localhost"
   $env:property_override_config_db_port="[DbPortNumberHere]"
   $env:property_override_config_db_name="[DbNameHere]"
   $env:property_override_config_db_user="[DbUserNameHere]"
   $env:property_override_config_db_password="[DbPasswordHere]"

   # Set email.property file values
   $env:property_override_email_host="[MailHostUrlOrIpHere]"
   $env:property_override_email_port="[MailHostPortHere]"
   $env:property_override_email_from="[FromEmailAddressHere]"
   $env:property_override_email_user="[MailSenderUserNameHere]"
   $env:property_override_email_password="[MailSenderPasswordHere]"
   $env:property_override_email_enabled="1"

   # Create full qualified script path.
   $scriptPath = Join-Path -Path "." -ChildPath "OverWriteProperties.ps1"

   # Invoke the override creator script to update the file content.
   Invoke-Expression "$scriptPath"

As a side note the script above will only run on windows, so if your team consists of developers working on a mix of environments you may want to create BASH scripts with a similar function, or even a Pash script you want to cover multiple platforms in one go.

TIP: On email settings for projects, we use Mailtrap.io, it simulates an SMTP server but puts all emails into one "mailbox", allowing you to test that your Application is composing and sending mail correctly without the risk of actually sending mail.

Build the Java Project

As mentioned earlier your Build Worker knows what to do with the Database Project and will build it automatically.

For the Java Project Maven comes to rescue. Basically, all we need to do is ask Maven to package our Java Project for us and because we are using Vaadin we must also pass our Vaadin License keys as command line variables to Maven.

On your AppVeyor Project’s Settings page, goto the Build tab and set the After build script field type to CMD and insert the following script.

REM You may use any of your Vaadin Developer Licenses in your build environment.

mvn -Dvaadin.charts.developer.license=cef377ec-79a6-425b-87bc-65bc8a5634ac -Dvaadin.touchkit.developer.license=5d71c10e-509c-4e25-b193-d4a18d990114 package

NOTE: The above Keys were randomly generated, you should replace them with your own keys. 😉

TIP: Maven is CHATTY, very CHATTY! To make your build output log manageable you may want to consider using the -q command line argument to make the maven output less verbose.

Packaging


By packaging I am referring to the creation of deployable artifacts.
Because we have two distinct deployments (the DACPAC file for the database project and the WAR file for the Java project) we are required to create two distinct deployment artefacts.

DACPAC file

The dacpac artefact is very simple to create.
All we need to do is tell the Appveyor Project where the dacpac file is and give it a name to allow us to easily reference it during deployment.

  1. Goto the Artifacts tab of your Appveyor project.
  2. Click on Add artifact.
  3. In the Path to artifact field type: [DatabaseProjectFolder]\bin\Release[DatabaseProjectName].dacpac

    NOTE: Even though we are working with a development deployment the build is in Release mode because pre- and post- deployment scripts in a Database Project are not executed in Debug mode.

    TIP: To get the path and filename you can Remote Desktop into the Build Worker instance and browse your project directories.

  4. In the Deployment name field type "DACPACFILE".
  5. In the Type field select "Auto".
  6. Click the Save button.

WAR file

The WAR file is a little more complex due to some Tomcat nuances and the way AppVeyor’s FTP deployment works.
Maven will place your packaged WAR file in the target folder within your Java Project’s folder structure, and AppVeyor’s FTP deployment will copy that folder structure to the deployment server and Tomcat will make that structure part of the URL (which we do not want.)

To solve this issue we have to copy the packaged WAR file to the build root folder before creating the deployment artifact.

  1. Goto the Build tab of your AppVeyor project.
  2. Append the following to the After build script:
    COPY C:\projects\[BuildProjectFolderName]\target\[JavaProjectName].war C:\projects\[BuildProjectFolderName]\[JavaProjectName].war
    

    You’re After build script should now look something like this:

    REM You may use any of your Vaadin Developer Licenses in your build environment.
    
    mvn -Dvaadin.charts.developer.license=cef377ec-79a6-425b-87bc-65bc8a5634ac -Dvaadin.touchkit.developer.license=5d71c10e-509c-4e25-b193-d4a18d990114 package
    
    COPY C:\projects\[BuildProjectFolderName]\target\[JavaProjectName].war C:\projects\[BuildProjectFolderName]\[JavaProjectName].war
    

    NOTE: Your [JavaProjectName] must match your pom.xml file’s artifactId element value and you should add a finalName element to your Java Project’s pom.xml file:

    [The following snippet is a bit confusing. Can we remove the tag? AC]

    <project>     
         ...
         </pluginManagement>
         <finalName>${artifactId}</finalName>
         </build>
    </project>
    

    Otherwise your WAR will not be loaded correctly by Tomcat.

    TIP: To get the correct path and filename you can Remote Desktop into the Build Worker instance and browser your project directories.

  3. Click the Save button.

Now we are ready to create the deployment artefact for the WAR file.

  1. Goto the Artifacts tab of your AppVeyor project.
  2. Click on Add artifact.
  3. In the Path to artifact field, type: [JavaProjectName].war
  4. In the Deployment name field, type: "WARFILE".
  5. In the Type field select "Auto".
  6. Click the Save button.

Deployment


Before we can setup the actual deployment we must first create the deployment environment on Azure and get the credentials we will need to do the actual deployment from our AppVeyor build worker.

For this project we require an Azure App Service with Tomcat enabled and an Azure SQL database.

I am not going to give you a blow-by-blow for setting up an Azure App Service or SQL database, the Azure documentation is great so you can find information there. I am going to focus on the extra steps for setting up Tomcat, getting your deployment credentials via the Azure Portal and setting up the deployments in AppVeyor.

Enable Tomcat on your Azure App Service

After you have setup your App Service you should enable Tomcat by setting your Java Version, Java Minor version and Web container attributes on the Application Settings tab of your App Service. I set mine to Java 8, 1.8.0_60 and Newest Tomcat 8.0 respectively. After you have saved the settings your App Service will restart and Tomcat will be up and running.

Get your App Service FTP credentials

To get the App Service’s FTP deployment credentials I always download the Publishing profile xml file from the App Service’s main dashboard window’s toolbar. i.e. Click the Get Publishing profile button.

Your publishing profile will include an element with an attribute publishMethod=FTP. This element contains the credential attributes and url you’ll need to upload your WAR artifact via FTPS. They are the publishUrl, userName and userPWD.

Take note of them as we will need them when setting up the deployment of the WAR artifact.

Get your Azure SQL Server credentials

The important thing to remember here is that to get a SQL Database you need a SQL Server first. And for simplicity’s sake let us assume that you are going to be using the SQL Server’s admin user’s credentials to do the DACPAC deployment. Thus when setting up the SQL Server you need to note these credentials.

To get your connection string (which will not include the User Id or Password) you have to navigate to the SQL Database’s main dashboard and click on Show database connection strings. The one you are after is the ADO.NET (SQL authentication) connection string. It will look something like this:

Server=tcp:someserversname.database.windows.net,1433;Data Source=someserversname.database.windows.net;Initial Catalog=somedatabasename;Persist Security Info=False;User ID={your_username};Password={your_password};Pooling=False;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;

Use the above connection string to build a connection string that looks like this:

Data Source=tcp:someserversname.database.windows.net,1433;database=somedatabasename;Initial Catalog=somedatabasename;User Id={your_username}@someserversname;Password={your_password};

Note that you need to replace someservername with your SQL Server’s name in both the Data Source attribute and the User Id attribute. Likewise you have to replace somedatabasename with your database’s name in both the database attribute and the Initial Catalog attribute. And of course you have to replace and with their corresponding values.

Now we are ready to setup the deployments in AppVeyor.

Deployment of the Database artifact

  1. Go to your AppVeyor Project’s Settings.
  2. On the Deployment tab click on Add deployment.
  3. For your Deployment Provider select SQL Server Database.
  4. In the .dacpac artifact to deploy field type the name of your dacpac artifact. i.e. "DACPACFILE".
  5. In the Target SQL Server connection string field provide the connection string we built up in Get your Azure SQL server credentials.
  6. You can keep the default values for the rest of the settings for now, but you may want to tweak them later.
  7. Click the Save button.

Deployment of WAR artifact

  1. Go to your AppVeyor Project’s Settings.
  2. On the Deployment tab click on Add deployment.
  3. For your Deployment Provider select FTP.
  4. For your Protocol select FTPS.
  5. In the Host field enter the publishUrl you identified in Get your App Service FTP credentials step.
  6. In the Username field enter the userName you identified in Get your App Service FTP credentials step. (Whole value as it is stated in the Publishing profile)
  7. In the Password field enter the userPWD you identified in the Get your App Service FTP credentials step.
  8. In the Remote folder field enter ‘site/wwwroot/webapps/’. Tomcat knows to load WAR files dropped into this folder.
  9. In the Artifact(s) field supply the name of your WAR artifact. i.e. "WARFILE".
  10. Click the Save button.

You are done. Go and make a commit to your development branch and your AppVeyor project should take care of the building and deployment for you. 😉

To view the result you hit the url for your App Server plus the pom.xml artifactId you specified in Packing -> WAR File, for example: ‘https://asiteofmine.azurewebsites.net/MyVaadinProject/&#8217;

Note: Tomcat urls are CaSe sensitive thus your url case must match your pom.xml artifactId exactly.

Also note: that after the ftp transfer has completed Tomcat still has to load the WAR file before you’ll be able to see your project running, the loading may take a minute or two.

Conclusion

All-and-all it took quite some time to get all of this working, but at least it is possible. 😉

There are two things I would like to note:

  1. Sometimes if the site is hit before Tomcat has finished loading the new WAR file the site falls over and only 404s are returned. To remedy this we just redeploy via AppVeyor. I don’t like the seemingly random nature of this problem and the fact that the build server still reports a successful build and deploy. (Thank you UptimeRobot for reporting on the site being down 😉 )
  2. The build time is very long. This project is neither complex nor large and it takes 7-8 minutes to run a build and deploy cycle + the time it takes Tomcat to load the new WAR file. A comparable MVC project takes 2 minutes on average. I don’t know if this is due to my limited knowledge concerning Java/Maven or if it is just the nature of building Java projects but our team finds this rather frustrating.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s