Posts in 'blog' – Page 3

Global Build Numbers in Jenkins Multibranch Pipeline Builds

We have a setup for CI that uses Jenkins to continually build our iOS and Android apps. We use the Jenkins multibranch pipeline plugin to allow us to have automatically created separate build jobs for each branch in our project. We use Git Flow and so there are feature branches created for each piece of development work carried out.

Jenkins 2.0 Multibranch jobs

This was great, and we had a system that would dynamically generate a new App Id or package id for each feature branch and upload them to HockeyApp for distribution. This meant that you could easily install an app built from a specific branch to test it out pre-merge. The problem being as we are using more and more external APIs a lot of them depend upon the App Id or package id. As these ids change dynamically then it causes problems with the 3rd party APIs.

So we decided to scale things back a bit and instead of having multiple feature packages e.g. com.enquos.totalhealth.feature.analytics and com.enquos.totalhealth.feature.exercise_ui we would just have a single package id of com.enquos.totalhealth.feature. The develop and release ids would stay the same.

We wanted to keep the separate jobs in Jenkins though, as it made it easier to see which specific branches were failing tests.

The problem then is that each branch build job started it's build number at 1. So if we got to the 10th build of com.enquos.totalhealth.feature.analytics then there would be an app to download with build number 10. If we then had the first build of another branch e.g. com.enquos.totalhealth.feature.exercise_ui then the build numbering would restart at 1.

It appears that a 'global' build number is not something Jenkins supports, or does any of the plugins I found. I did find some that looked like they might help, but ultimately didn't support Jenkins 2.0's pipeline DSL.

So, instead I wrote a very simple small HTTP server in Python that serves up an incremental integer every time you connect to it. The integer is stored in a file so can carry on after a restart. It is a simple single-threaded server based on the built-in Python BaseHTTPServer so that sorts out any concurrency issues.

from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
import socket

class NumberServer(BaseHTTPRequestHandler):
    def _set_headers(self):
        self.send_response(200)
        self.send_header('Content-type', 'text/plain')
        self.end_headers()

    def do_GET(self):
        self._set_headers()

        # Try and open number file and read number
        try:
            f = open("number.txt", "r")
            number = int(f.read())
            f.close()
        except IOError, ValueError:
            # if we fail then just assume 1
            number = 1

        # send the number to the client
        self.wfile.write(str(number))

        # write incremented number back to file
        f = open("number.txt", "w")
        number = str(number+1)
        f.write(number)
        f.close()

    def do_HEAD(self):
        self._set_headers()

class HTTPServerV6(HTTPServer):
  address_family = socket.AF_INET6        

def run(server_class=HTTPServerV6, handler_class=NumberServer, port=80):
    server_address = ('::', port)
    httpd = server_class(server_address, handler_class)
    print 'Starting number server...'
    httpd.serve_forever()

if __name__ == "__main__":
    from sys import argv

    if len(argv) == 2:
        run(port=int(argv[1]))
    else:
        run()

The latest version of the code is available on Github.

Matts-iMac:~ matt$ curl http://numberserver:9999/
28Matts-iMac:~ matt$ curl http://numberserver:9999/
29Matts-iMac:~ matt$ curl http://numberserver:9999/
30Matts-iMac:~ matt$ 

Testing it you can see it returns a new number each time: 28, 29, 30. There is no newline returned as the output is intended to be consumed directly by a script.

The Jenkinsfile has been modified to make a call to the URL and use the result in a variable for later use. Groovy makes this surprisingly easy and concise to do:

env.BUILD_ID = 'http://numberserver.quernus.co.uk:9999/'.toURL().text

From the Jenkinsfile we can also set the build number displayed in Jenkins so that it matches the one we use:

currentBuild.displayName = "#" + env.BUILD_ID

You can see then the global build numbers for a particular job as they show up non sequential as other jobs have run in-between these ones:

Jenkins 2.0 global build numbers

This puts the build number in an environment variable. This is then looked at later by both the Gradle build script for our Android builds:

int getVersionCodeFromEnv(String versionName) {
    System.getenv("BUILD_ID) as Integer ?: 0
}

…and the Fastlane config for our iOS builds:

      increment_build_number(
        xcodeproj: project,
        build_number: ENV['BUILD_ID']
      )

And the end result is that the feature branch app on Fabric has a combination of jobs built from different branches, but that the build number is both unique and increasing for each build:

Fabric Beta feature branch builds with unique build numbers across branches

I can tap to install any previous build of this app from any of the feature branches for testing.

Oh, and I suppose I can now cross ‘microservices’ off my buzzword bingo sheet now, right? ;)

Jenkins 2.0 and Multi-branch pipeline builds for iOS apps with Fastlane

Jenkins 2.0 beta is out and has included a multi-branch pipeline plugin that allows automatic build of feature branches from Github. Here is how I set it up to build feature branch builds of our iOS apps

Getting Proper UTF-8 Output From Fastlane on Jenkins 2.0 Pipeline builds

Jenkins 2.0 pipeline jobs get their locale from the master not slave, so you need to set the local on master to get UTF-8 output working correctly

Safer deleting of a directory

A safer alternative to rm -rf for use in scripts

Jenkins, Github, and IPv6

Github and some other sites don't yet support IPv6. But I want our build servers to be IPv6 only. Here is how I achieved it using OpenBSD's NAT64 and unbound's DNS64 functions

Smart Nation Singapore - Bristol Festival of Ideas - Festival of the Future City

Singapore aims to be the world’s first Smart Nation, with fuller use of technology to live, work and play resulting in improved quality of life for individuals; business opportunities for enterprises; and a government that uses technology to better serve and anticipate citizens’ needs.

NSScotland 2015 Conference

A write up of some of the talks I found interesting at the NSScotland conference in Edinburgh. Dealing with maps; Building a mental health app; Working with distributed teams; and a 30-year old calculator codebase

Force git to use ssh instead of HTTPS

Sometimes you have tools that reference a github url with https and you want them to use ssh instead so your ssh key works

Plone Conference 2015 - Bucharest

The Plone Conference 2015 happened in Bucharest, Romania. Once again, an awesome event! Great keynotes on the state of Plone and accessibility.

Creating Custom PDF Filter on OS X

Sometimes you want to reduce the size of a PDF, say with images embedded. OS X has a built in filter for this, but it is a bit harsh. You can create you own custom ones though.