One Jenkins Job to Rule them All

“Much that once was is lost. For none now live who remember it.”Lord of The Rings

The quot above describes really well the current state in my old team, and oddly enough the “Ring” fits very well with the solution we had for our build requirements.

As a Tech. Lead for a professional services team I found myself looking at countless projects and git repositories of the same type as we provide more and more solutions to our customers. It was always another WCF service or a REST net core service or a DLL.

Regardless of the type we always package them the same. We publish the WCF\ REST or build on release the DLL zip the entire content and ship it out to our deliver server for other reams to grab and deploy

The process above was done Manually (Barbaric!!!) before I joined the team, so I decided to change this.

The goal was to create a reusable job template that will help us to create new Jenkins jobs for every solution we had, so I immediately dove into Jenkins Pipelines while they are awesome when working on small amount of project after a while we hit a snag with them because of the following couple of reasons:

  • Every solution had its own pipeline in his git repository, this caused dozens of Jenkins jobs been created and it was very hard to manage them all
  • When a change was required during the build process or publication (like server change) we had to god and update all pipelines (I know we could have use a variable but we didn’t)

So one day I was playing around with bash and figured that I could write a publish scrip to rule them all. and here we are, the magic script (s) that make our build process work with a single parametrize job in Jenkins with the following input params

  • GIT_URL – the git repo url to build
  • BRANCH – the git branch to build
  • SOLUTION_TYPE – .net core \ wcf \ dll (closed list)
  • SOLUTION_FILE_NAME – (OPTIONAL!) the sln file that should be used for build \ publish only if its different than the git repository name

The first script Jenkins will run is BuildVars.sh its goal is to establish some global variables the will be used in other scripts.

# Remove echo
set +x
echo .
echo .
echo === Parse Solution ID from Git URL ===
basename=$(basename ${GIT_URL})
echo $basename
RepoName=${basename%.*}
echo $RepoName

echo .
echo .
echo === Cloning Repository ${GIT_URL} ===
git clone ${GIT_URL} -b ${BRANCH}
cd ${RepoName}


echo .
echo .
echo === Setting Default Build Variables ===
# This is the way we identify the version of the releas
VERSION_TAG=$(git describe --always --dirty --long --tags)
VERSION_TAG=${VERSION_TAG/-dirty/}
VERSION_TAG=${VERSION_TAG//-/.}


# Check if SOLUTION_FILE_NAME is provided and overwrite SOLUTION_ID
if test -z "$SOLUTION_FILE_NAME" 
then
	echo SOLUTION_FILE_NAME param was not provided, assuming that solution file is set to ${RepoName}
    export SOLUTION_ID=${RepoName}
else
	echo Setting SOLUTION_ID to provided ${SOLUTION_FILE_NAME}
    export SOLUTION_ID=${SOLUTION_FILE_NAME}
fi

export SOLUTION_FILENAME="${SOLUTION_ID}.sln"
export PUBLISH_PROJECT_TARGET_DIR=${SOLUTION_ID}
export PUBLISH_PROJECT_TARGET_FILENAME="${SOLUTION_ID}.csproj"

export PUBLISH_DIR_LOCAL="/d/DEV/Releases/${SOLUTION_ID}"
export NUGET_REPO_GLOBAL="https://nuget.org/api/v2/"
export NUGET_REPO_PS="http://localhost:8090/nuget"

echo SOLUTION_ID = ${SOLUTION_ID}
echo VERSION_TAG = ${VERSION_TAG}
echo SOLUTION_FILENAME = ${SOLUTION_FILENAME}
echo PUBLISH_PROJECT_TARGET_DIR = ${PUBLISH_PROJECT_TARGET_DIR}
echo PUBLISH_PROJECT_TARGET_FILENAME = ${PUBLISH_PROJECT_TARGET_FILENAME}
echo PUBLISH_DIR_LOCAL = ${PUBLISH_DIR_LOCAL}
echo NUGET_REPO_GLOBAL = ${NUGET_REPO_GLOBAL}
echo NUGET_REPO_PS = ${NUGET_REPO_PS}

echo
echo
echo === Overwriting Build Variables from build.props.ini ===
if [ ! -f build.props.ini ]; then
    echo "build.props.ini file not found skipping overwrites!"
else
	config=$(cat build.props.ini)
	echo Loaded following overwrites
    echo $config
    echo $pwd
	source ./build.props.ini
fi


echo
echo
echo === Final Build Variables ===
echo SOLUTION_ID = ${SOLUTION_ID}
echo VERSION_TAG = ${VERSION_TAG}
echo SOLUTION_FILENAME = ${SOLUTION_FILENAME}
echo PUBLISH_PROJECT_TARGET_DIR = ${PUBLISH_PROJECT_TARGET_DIR}
echo PUBLISH_PROJECT_TARGET_FILENAME = ${PUBLISH_PROJECT_TARGET_FILENAME}
echo PUBLISH_DIR_LOCAL = ${PUBLISH_DIR_LOCAL}
echo NUGET_REPO_GLOBAL = ${NUGET_REPO_GLOBAL}
echo NUGET_REPO_PS = ${NUGET_REPO_PS}
cat &gt; ${WORKSPACE}/generated.jenkins.vars <<EOL
RepoName=${RepoName}
SOLUTION_ID=${SOLUTION_ID}
VERSION_TAG=${VERSION_TAG}
SOLUTION_FILENAME=${SOLUTION_FILENAME}
PUBLISH_PROJECT_TARGET_DIR=${PUBLISH_PROJECT_TARGET_DIR}
PUBLISH_PROJECT_TARGET_FILENAME=${PUBLISH_PROJECT_TARGET_FILENAME}
PUBLISH_DIR_LOCAL=${PUBLISH_DIR_LOCAL}
NUGET_REPO_GLOBAL=${NUGET_REPO_GLOBAL}
NUGET_REPO_PS=${NUGET_REPO_PS}
... 
EOL
echo
echo

So there is a lot going on there let me talk about the highlights.
Versioning
We use git describe --always --dirty --long --tags to get the latest tag with the commit count since, this proved out to be a good strategy specially when you have bunch of different version running around the then a bug comes in for one of them, go figure which code base you need to check without it.
There is a different process we do to patch versions just before the build is done but this is a whole different story to tell.
Solution ID
Mostly all our solutions have the same name for both .sln file and the git repository. so we use basename to get the repository name form the git URL
Variable Overrides
I wanted to allow any project to overwrite any variable to give the flexibility to change anything if required per project so we use source ./build.props.ini to load a build.props.ini file that will contain any overrides required for the variables.

From this point we just use the input variable SOLUTION_TYPE to identify how we should publish and run the appropriate script

This is how it looks in the Jenkins UI

But this was not enough! I don’t want to copy paste the git URL select the project type or event navigate to Jenkins when i start a build, so I needed some way to trigger it automatically from multiple different git repositories.
Introducing Url based Job trigger:

Now we are getting there! but still I need a way to allow our developers to trigger this request in a simple manner. Presenting: ps-release.sh script this script will search the current directory and will try to find out what is the solution type, what is the remote git repository url and issue the POST request to the jenkins server

#!/bin/bash
TOKEN='YourPasswordHere'

GIT_URL='Unknown'
bitbucket_remote_host='git@bitbucket.org:your_company/'
github_remote_host='git@github.com:your_company/'
fullOriginUrl=$(git config --get remote.origin.url)
echo fullOriginUrl $fullOriginUrl
fullOriginUrl=${fullOriginUrl/\/src/}

echo fullOriginUrl2 $fullOriginUrl

gitRepoName=$(basename $fullOriginUrl .git)
echo gitRepoName $fullOriginUrl


echo "Trying to determine git or bitbucket remote:[$fullOriginUrl]"
echo
shopt -s nocasematch
if [[ $fullOriginUrl =~ "github" ]]; then
    echo 'github'

    GIT_URL="$github_remote_host$gitRepoName.git"
elif [[ $fullOriginUrl =~ "bitbucket" ]]; then
    echo 'bitbucket'
    GIT_URL="$bitbucket_remote_host$gitRepoName.git"
else
    echo 'Cannot determine git remote source its not git nor bitbucket.'
    exit 125 
fi


echo "Trying to determine release project type"
projectFile=$(find -name "*.csproj" | head -n 1)
svcFile=$(find -name "*.svc" | head -n 1)
outputType=$(grep -oPm1 "(?<=<OutputType&gt;)[^<]+" $projectFile)
TargetFramework=$(grep -oPm1 "(?<=<TargetFramework&gt;)[^<]+" $projectFile)


echo "Project File:     ${projectFile}"
echo "SVC File:         ${hasSvcFile}"
echo "Output Type:      ${outputType}"
echo "TargetFramework:  ${TargetFramework}"

if [[ $svcFile != "" ]]; then
    echo "Found svc file in solution, assuming WCF project."
    SOLUTION_TYPE='WCF'
elif [[ $outputType =~ "Library" ]]; then
    echo "Found OutputType=Library in .csproj file in solution and not svc files, assuming DLL project."
    SOLUTION_TYPE='DLL'
else
echo "Could not identify WCF or DLL project, assuming .Net Core"
    SOLUTION_TYPE='NET_CORE'
fi

SOLUTION_FILE_NAME=$(basename $(find -name "*.sln") .sln)
BRANCH=${1:-master}

echo
echo
echo "Variables are set to:"
echo "GIT_URL:            [$GIT_URL]"
echo "SOLUTION_FILE_NAME: [$SOLUTION_FILE_NAME]"
echo "BRANCH:             [$BRANCH]"
echo "SOLUTION_TYPE:      [$SOLUTION_TYPE]"

requestUrl="http://jenkins:8080/view/PS/job/PS.Release/buildWithParameters?token=${TOKEN}&amp;GIT_URL=${GIT_URL}&amp;BRANCH=${BRANCH}&amp;SOLUTION_TYPE=${SOLUTION_TYPE}&amp;SOLUTION_FILE_NAME=${SOLUTION_FILE_NAME}"

echo "calling url:[$requestUrl]"
result=$(curl --request GET \
    --max-time 5 \
    --write-out %{http_code} \
    --silent \
    --output /dev/null \
    --url $requestUrl_external_ip)

echo result is: $result

So this will work well.. whats left for the DEV is install under /usr/bin/ run “sh ps-release.sh” and presto the build starts!
But wait there is more… I love command line and this is very nice for me, most of our team are windows guys and gals so I wanted something more familiar to them, I wanted this to become a button in the context menu of the folder I’m in or the folder i have clicked. Presenting: Windows Registry hack for adding right-click context menu Items
There is nothing like a good Windows Registry hack in the morning! here is the .reg script and here is the ps-release-context-menu.reg

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Directory\shell\ps_release]
@="PS-Release"
"Icon"="C:\\WINDOWS\\system32\\Defrag.exe"

[HKEY_CLASSES_ROOT\Directory\shell\ps_release]\command]
@="cmd.exe /s /k pushd \"%V\" && sh ps-release.sh"

[HKEY_CLASSES_ROOT\Directory\Background\shell\ps_release]
@="PS-Release"
"Icon"="C:\\WINDOWS\\system32\\Defrag.exe"

[HKEY_CLASSES_ROOT\Directory\Background\shell\ps_release\command]
@="cmd.exe /s /k pushd \"%V\" && sh ps-release.sh"

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.