sudolabs logo
Let's build

  • Tech

How we automatized our release process into just 3 clicks

In the previous article, we talked about the release process with automated changelog generation with a tool named lerna-changelog. As I mentioned at the end of that article, there were a lot of possibilities to improve. Let's take a look at them!

5. 10. 2021

Filip Jenik

Motivation

In the previous article, we talked about the release process with automated changelog generation with a tool named lerna-changelog. As I mentioned at the end of that article, there were a lot of possibilities to improve. Our release process has a lot of steps, which we are doing manually, so the first thing that comes to mind is automatization. This wasn't the only improvement in our minds. We were thinking about improving the whole releasing process because with the current workflow we were losing some important changes (mostly to hotfixes) from our changelog.


Goals

We set a couple of goals. There is a common saying about programmers - they are lazy. So our first goal was automatization. In the current implementation of the release process, we had to do more than 20 manual steps to release. Every release, they were the same. The other goals were to improve our whole release process - to synchronize develop branch with the master branch (after the release was done), changelog generation after merging hotfixes into the master branch, and a lot more.


Design

The first solution was fine. It was working and we were able to release new features. As time passed we started noticing some shortcomings. There were always conflicts between master and develop after releasing, missing hotfixes in the changelog, generation of tags was done manually, manually parsed changelog, etc.

Because of these reasons, we decided that these improvements should be part of our priorities. We scratched a few ideas and possible solutions and we were iterating and modifying them. After a few meetings on this topic where we discussed our plans and ideas, we found a possible solution.

From the planning point of view, this task looked problematic, huge, and very complex. Because of that, we split it into smaller tasks that can be done separately and independently from each other. We've grouped these tasks under an epic named "Automate release process". With this solution, we were able to deliver this feature by small tasks during more sprints so we could test our solution in the production environment in small parts, whilst delivering some other features to the product.

We decided to create 4 workflows:

  1. Open PR from develop to staging

    • when develop is ahead of staging create a PR that will accumulate over time and when we decide to create a new release candidate (RC), we will merge it.

  2. Open PR from staging to master and create RC

    • when staging is ahead of master create release candidate PR with the TODO changelog parsed into the PR description to inspect, test changes and mark tested tasks.

  3. Push to master flow

    • Append CHANGELOG.md with the newest changes (Create commit)

    • Create a new tag

  4. Open PR from master to develop

    • When master is ahead of develop, create a PR that will update the develop branch to merge all the hotfixes.


Workflows

After we've written down a short description of design, we were ready to look deeper into each one of these workflows.

Open PR from develop to staging

This workflow is quite simple. Its only purpose is to create a pull request from develop to staging when there are new commits in the develop branch. This lets us release all the new features into the staging environment for our in-house and our customer testing.

Steps

  1. Detect if the develop branch is ahead of staging - if not, the rest of the steps are skipped

  2. Check if PR from develop to staging already exists - if yes, the rest of the steps are skipped

  3. Create a PR from develop to staging

Workflow

name: Create pull request from develop to staging
on:
push:
branches:
- develop
jobs:
create-develop-to-staging-pr:
runs-on: ubuntu-latest
name: Create pull request from develop to staging
steps:
- name: Detect changes between develop and staging
id: "detect-changes-between-develop-and-staging"
uses: fjenik/detect-changes-between-two-branches@0.0.3
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
target-branch: staging
source-branch: develop
- name: Detect changes output
run: echo "Output of detect changes ${{ steps.detect-changes-between-develop-and-staging.outputs.is-source-branch-ahead }}"
- name: Check if pr already exists
id: "check-if-pr-already-exists"
if: ${{ steps.detect-changes-between-develop-and-staging.outputs.is-source-branch-ahead == 'true' }}
uses: fjenik/check-if-pr-exists@0.0.3
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
target-branch: staging
source-branch: develop
- name: Print output
run: echo "Output of pull request already exists ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created }}"
- name: Create pull request
if: ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created == 'false' }}
uses: fjenik/create-pull-request@0.0.12
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
target-branch: staging
source-branch: develop
pr-title: "👾 Merge develop to staging"
pr-body: |
_This PR was generated via github actions workflow_
**Don't squash merge this PR**

Open PR from staging to master and create RC

This workflow is triggered on a push into the staging branch. The job of this workflow is to create a PR from staging into master with a correctly formatted description generated from the changelog. The PR will be created only when the staging branch is ahead of the master branch. The changelog is parsed by the merged PR and formatted into a TODO list where the author of the PR is mentioned.

After this, a PR is created and a new message is sent to our development Slack channel, where we ask everyone to test our new RC and mark their features or bug fixes as done (in the TODO list). We are not releasing until this list is fully checked. That's the rule.

Steps

  1. Checkout repository into our GitHub action (if you want to work with anything else than your current repository version - for example tags, you have to add [input fetch-depth: 0](https://github.com/actions/checkout#checkout-v2))

  2. Setup NodeJS

  3. Setup GitHub action cache

  4. Install the packages

  5. Check if the staging is ahead of the master branch

  6. Check if this PR already exists

  7. Get new changes and a new version of the future release

    1. Get new changes to changelog with [lerna-changelog](https://github.com/lerna/lerna-changelog) and their modification into the correct format - parsing each merged PR into the TODO list

    2. Get a current version of the repository - [git describe --abbrev=0](https://git-scm.com/docs/git-describe)

    3. Get a new version type from the input parameter - in this workflow, it's every minor increment

    4. Set NEW_VERSION and NEW_CHANGES (TODO list) outputs of the action

  8. Create PR with TODO list and correct version in the PR title

  9. Send a notification to all team members about the new release candidate via Slack

Workflow

name: Create pull request from staging to master
on:
push:
branches:
- staging
jobs:
create-staging-to-master-pr:
runs-on: ubuntu-latest
name: Create pull request from staging to master
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- uses: actions/setup-node@v1
with:
node-version: "12.14.1"
- name: Cache node modules
uses: actions/cache@v2
env:
cache-name: cache-node-modules
with:
# npm cache files are stored in `~/.npm` on Linux/macOS
path: |
~/.npm
~/.cache
key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-build-${{ env.cache-name }}-
${{ runner.os }}-build-
${{ runner.os }}-
- run: npm ci
- name: Detect changes between staging and master
id: "detect-changes-between-staging-and-master"
uses: fjenik/detect-changes-between-two-branches@0.0.3
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
target-branch: master
source-branch: staging
- name: Detect changes output
run: echo "Output of detect changes ${{ steps.detect-changes-between-staging-and-master.outputs.is-source-branch-ahead }}"
- name: Check if pr already exists
id: "check-if-pr-already-exists"
if: ${{ steps.detect-changes-between-staging-and-master.outputs.is-source-branch-ahead == 'true' }}
uses: fjenik/check-if-pr-exists@0.0.3
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
target-branch: master
source-branch: staging
- name: Print output
run: echo "Output of pul request already exists ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created }}"
- name: Get new version and description
id: get-new-version-and-description
if: ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created == 'false' }}
env:
GITHUB_AUTH: ${{secrets.GITHUB_TOKEN}}
run: |
node ${GITHUB_WORKSPACE}/.github/workflows/getRcDescriptionAndNewTag
- name: Check outputs new version
if: ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created == 'false' }}
run: echo "Output of detect changes ${{ steps.get-new-version-and-description.outputs.NEW_VERSION }}"
- name: Check outputs new changes
if: ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created == 'false' }}
run: echo "Output of detect changes ${{ steps.get-new-version-and-description.outputs.NEW_CHANGES }}"
- name: Create pull request
id: "create-pr"
if: ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created == 'false' }}
uses: fjenik/create-pull-request@0.0.13
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
target-branch: master
source-branch: staging
pr-title: "👾 Release candidate ${{ steps.get-new-version-and-description.outputs.NEW_VERSION }}"
pr-body: |
_This PR was generated via github actions workflow to create RC pull request_
If you are tagged in some of the issues this RC is made of, please mark these TODO checkboxes when you've successfully tested the feature on the staging environment
[staging environment](link to our staging environment)
${{ steps.get-new-version-and-description.outputs.NEW_CHANGES }}
**Don't squash merge this PR**
- name: Slack Notification
if: ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created == 'false' }}
uses: tokorom/action-slack-incoming-webhook@main
env:
INCOMING_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}
with:
text: "👾 Hello! New RC (${{ steps.get-new-version-and-description.outputs.NEW_VERSION }}) was created! You know what to do! https://github.com/<our-org>/<our-project>/pull/${{ steps.create-pr.outputs.pull-id }} :fire:"

Push to master flow

In this workflow, we are focusing on the automatization of creating a new version (tag) and appending new changes into the changelog after a merge into the master branch. This workflow is the main improvement over our old release process because back then, we were not creating a new version after each merge into master, and also, we were not updating our changelog after a direct merge into the master branch (hotfix).

With this workflow, we automate the longest and the most annoying part of our release process, where we had to manually parse the changelog and create a new git tag. We also improved our versioning - if the pull request is a part of our release process (when we are releasing from the staging branch), it's a minor increment but when we're merging into the master from a different branch it's a hotfix, so we're just changing the patch version. Major versions will only be incremented after a big change - like migration to mono-repository or architecture migration into serverless.

Steps

  1. Checkout repository into our GitHub action (if you want to work with anything else than your current repository version - for example tags, you have to add [input fetch-depth: 0](https://github.com/actions/checkout#checkout-v2))

  2. Setup NodeJS

  3. Setup GitHub action cache

  4. Install the packages

  5. Setup git config

  6. Run core script

    1. Validate inputs

    2. Get new changes for changelog with lerna-changelog and its modification to correct format

    3. Get a current version of the repository - [git describe --abbrev=0](https://git-scm.com/docs/git-describe)

    4. Get new version by input parameter (patch or minor) - with semverInc package

    5. Format data to changelog

    6. Modify the changelog

    7. Add change(s) to git - git add --all

    8. Commit all new changes - git commit -am 'Update changelog ${newVersion} [skip ci] - we are using [skip ci] to protect ourselves from recursive workflow triggering

    9. Generate a new tag and push into origin - npx lerna version ${newVersion} --force-publish --yes -m "publish new tag - %s [skip-ci]

Workflow

name: Add new tag and generate changelog
on:
push:
branches:
- master
paths-ignore:
- "./CHANGELOG.md"
- "./package.json"
- "./api/package.json"
- "./frontend/package.json"
jobs:
generate-new-tag:
runs-on: ubuntu-latest
name: Add new tag and generate changelog
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- uses: actions/setup-node@v1
with:
node-version: "12.14.1"
- name: Cache node modules
uses: actions/cache@v2
env:
cache-name: cache-node-modules
with:
# npm cache files are stored in `~/.npm` on Linux/macOS
path: |
~/.npm
~/.cache
key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-build-${{ env.cache-name }}-
${{ runner.os }}-build-
${{ runner.os }}-
- run: npm ci
- name: Get version type to change
env:
VERSION: ${{ contains(github.event.head_commit.message, 'staging') }}
ACTIONS_ALLOW_UNSECURE_COMMANDS: true
run: |
if ${VERSION} == true; then
echo "::set-env name=CHANGE_VERSION::minor"
else
echo "::set-env name=CHANGE_VERSION::patch"
fi
- name: "Print version change"
run: "echo ${{env.CHANGE_VERSION}}"
- name: Setup git config
run: |
# setup the username and email. I tend to use 'GitHub Actions Bot' with no email by default
git config user.name "GitHub Actions Bot"
git config user.email "<>"
- name: Run core script
env:
GITHUB_AUTH: ${{secrets.GITHUB_TOKEN}}
run: |
node ${GITHUB_WORKSPACE}/.github/workflows/generateChangelogAndCreateTag -v ${{env.CHANGE_VERSION}}

Open PR from master to develop

This workflow is used to synchronize develop branch with master branch. We often had a problem with outdated code in our develop branch (hotfix merges) when we wanted to release, so we needed to merge master into develop, which means we had a lot of conflicts to handle. With this workflow, we simplified our lives, and with every push to master we are automatically creating a PR (if it does not exist) which we're immediately reviewing and then merging into develop.

Steps

On every push into the master branch, this is what we do:

  1. Check if master branch is ahead of develop - if not, the rest of the steps are skipped

  2. Check if this PR already exists - if not, the rest of the steps are skipped

  3. Create PR from master to develop

Workflow

name: Create pull request from master to develop
on:
push:
branches:
- master
jobs:
create-master-to-develop-pr:
runs-on: ubuntu-latest
name: Create pull request from master to develop
steps:
- name: Detect changes between master and develop
id: "detect-changes-between-master-and-develop"
uses: fjenik/detect-changes-between-two-branches@0.0.3
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
target-branch: develop
source-branch: master
- name: Detect changes output
run: echo "Output of detect changes ${{ steps.detect-changes-between-master-and-develop.outputs.is-source-branch-ahead }}"
- name: Check if pr already exists
id: "check-if-pr-already-exists"
if: ${{ steps.detect-changes-between-master-and-develop.outputs.is-source-branch-ahead == 'true' }}
uses: fjenik/check-if-pr-exists@0.0.3
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
target-branch: develop
source-branch: master
- name: Print output
run: echo "Output of pull request already exists ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created }}"
- name: Create pull request
if: ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created == 'false' }}
uses: fjenik/create-pull-request@0.0.12
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
target-branch: develop
source-branch: master
pr-body: |
_This PR was generated via github actions workflow to keep `develop` always in sync with `master` branch_
Please review and merge.
**Don't squash merge this PR**
pr-title: "👾 Merge `master` to `develop`"

Results

After we developed this, we were surprised by how easy it was to implement this improvement and how it worked on the first try. We didn't run into any big logical problems or any bugs during the development process. Why didn't we do it sooner? I don't know. But I'm glad that we have done it.

Conclusion

In this article, we've talked about the automatization of the release process in just 3 clicks on GitHub. We walked through the design of our solution, how we planned this improvement, and we talked about each workflow separately. With our new release process, we are able to deliver new features anytime and by anyone. A person who will be responsible for the next release will not have to know anything about our application or even current architecture. It's simple. You just have to follow simple instructions (merge, merge and MERGE).

As always, there is a lot of things to improve. There are new ideas every time on how to simplify or improve our product delivery. In the nearest future we are going to implement notifications on environment variables changes, because, in larger releases, these changes can be forgotten.

Share:

Don't miss any news

By clicking the button above I consent to the
Privacy Policy.

hello@sudolabs.ioPrivacy policy & Terms
Navigateworkservicescareersblogcontact
Office

DUETT Business Residence

Námestie osloboditeľov 3/A

040 01 Košice, Slovakia