# How we automatized our release process into just 3 clicks In the previous article, we talked about the release process with automated changelog generation with a tool named lerna-changelog. As I mentioned at the end of that article, there were a lot of possibilities to improve. Let's take a look at them! - URL: https://sudolabs.com/insights/how-we-automatized-our-release-process-into-just-3-clicks - Published: 2021-10-05 - Updated: 2024-01-16 - Author: Filip Jenik - Categories: Tech - Language: English ## Motivation In the [previous article](https://sudolabs.io/blog/automated-release-process-for-lerna-monorepo), we talked about the release process with automated changelog generation with a tool named **lerna-changelog**. As I mentioned at the end of that article, there were a lot of possibilities to improve. Our release process has a lot of steps, which we are doing manually, so the first thing that comes to mind is automatization. This wasn't the only improvement in our minds. We were thinking about improving the whole releasing process because with the current workflow, we were losing some important changes (mostly to hotfixes) from our changelog. ## Goals We set a couple of goals. There is a common saying about programmers - they are lazy. So our first goal was automatization. In the current implementation of the release process, we had to do more than 20 manual steps to release. Every release, they were the same. The other goals were to improve our whole release process - to synchronize the **develop branch** with the **master branch** (after the release was done), **changelog generation after merging hotfixes** into the master branch, and a lot more. ## Design The first solution was fine. It was working and we were able to release new features. As time passed we started noticing some shortcomings. There were always **conflicts between master and develop after releasing**, missing hotfixes in the changelog, generation of tags was done manually, manually parsed changelog, etc. Because of these reasons, we decided that these improvements should be part of our priorities. We scratched a few ideas and possible solutions and we were iterating and modifying them. After a few meetings on this topic where we discussed our plans and ideas, we found a possible solution. ![](https://a.storyblok.com/f/117609/214x777/0a0bd096a8/diagram.png) From the planning point of view, this task looked _problematic_, _huge_, and _very complex_. Because of that, we **split it into smaller tasks that can be done separately and independently from each other.** We've grouped these tasks under an epic named "**Automate release process"**. With this solution, we were able to deliver this feature by small tasks during more sprints so we could test our solution in the production environment in small parts, whilst delivering some other features to the product. We decided to create 4 workflows: 1. Open PR from `develop` to `staging` - when **develop** is ahead of **staging** create a PR that will accumulate **over time** and when we decide to create a new release candidate (RC), we will merge it. 2. Open PR from `staging` to `master` and create RC - when **staging** is ahead of **master** create **release candidate PR** with the TODO changelog parsed into the PR description to inspect, **test changes** and mark tested tasks. 3. Push to master flow - Append **CHANGELOG.md with the newest changes** (Create commit) - Create a **new tag** 4. Open PR from `master` to `develop` - When the **master is ahead of develop**, create a PR that will **update** the **develop branch** to merge all the hotfixes. ## Workflows After we've written down a short description of the design, we were ready to look deeper into each one of these workflows. ### Open PR from `develop` to `staging` This workflow is quite simple. Its only purpose is to create a pull request from `develop` to `staging` when there are new commits in the `develop` branch. This lets us release all the new features into the staging environment for our in-house and our customer testing. #### Steps 1. Detect if the `develop` branch is ahead of `staging` - if not, the rest of the steps are skipped 2. Check if PR from `develop` to `staging` already exists - if yes, the rest of the steps are skipped 3. Create a PR from `develop` to `staging` #### Workflow ``` name: Create pull request from develop to staging on: push: branches: - develop jobs: create-develop-to-staging-pr: runs-on: ubuntu-latest name: Create pull request from develop to staging steps: - name: Detect changes between develop and staging id: "detect-changes-between-develop-and-staging" uses: fjenik/detect-changes-between-two-branches@0.0.3 with: repo-token: ${{ secrets.GITHUB_TOKEN }} target-branch: staging source-branch: develop - name: Detect changes output run: echo "Output of detect changes ${{ steps.detect-changes-between-develop-and-staging.outputs.is-source-branch-ahead }}" - name: Check if pr already exists id: "check-if-pr-already-exists" if: ${{ steps.detect-changes-between-develop-and-staging.outputs.is-source-branch-ahead == 'true' }} uses: fjenik/check-if-pr-exists@0.0.3 with: repo-token: ${{ secrets.GITHUB_TOKEN }} target-branch: staging source-branch: develop - name: Print output run: echo "Output of pull request already exists ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created }}" - name: Create pull request if: ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created == 'false' }} uses: fjenik/create-pull-request@0.0.12 with: repo-token: ${{ secrets.GITHUB_TOKEN }} target-branch: staging source-branch: develop pr-title: "👾 Merge develop to staging" pr-body: | _This PR was generated via github actions workflow_ **Don't squash merge this PR** ``` ### Open PR from `staging` to `master` and create RC This workflow is triggered on a **push into the staging branch**. The job of this workflow is to create a PR from `staging` into `master` with a correctly formatted description generated from the changelog. The PR will be created **only when the staging branch is ahead of the master branch**. The changelog is parsed by the merged PR and formatted into a TODO list where the author of the PR is mentioned. ![](https://a.storyblok.com/f/117609/1520x1482/56099af375/release_pr_example.png) After this, a PR is created and a new message is sent to our development Slack channel, where we ask everyone to test our new RC and mark their features or bug fixes as done (in the TODO list). We are not releasing until this list is fully checked. That's the rule. #### Steps 1. Checkout repository into our GitHub action (if you want to work with anything else than your current repository version - for example tags, you have to add `[input fetch-depth: 0](https://github.com/actions/checkout#checkout-v2)`) 2. Setup NodeJS 3. Setup GitHub action cache 4. [Install the packages](https://docs.npmjs.com/cli/v7/commands/npm-ci) 5. Check if **the staging is ahead of the master** branch 6. Check if this PR already exists 7. Get new changes and a new version of the future release 1. Get new changes to changelog with `[lerna-changelog](https://github.com/lerna/lerna-changelog)` and their modification into the correct format - parsing each merged PR into the TODO list 2. Get a current version of the repository - `[git describe --abbrev=0](https://git-scm.com/docs/git-describe)` 3. Get a new version type from the input parameter - in this workflow, it's every [minor](https://en.wikipedia.org/wiki/Software_versioning) increment 4. Set **NEW\_VERSION** and **NEW\_CHANGES** (TODO list) outputs of the action 8. Create PR with TODO list and correct version in the PR title 9. **Send a notification to all team members** about the new release candidate via Slack #### Workflow ``` name: Create pull request from staging to master on: push: branches: - staging jobs: create-staging-to-master-pr: runs-on: ubuntu-latest name: Create pull request from staging to master steps: - uses: actions/checkout@v2 with: fetch-depth: 0 - uses: actions/setup-node@v1 with: node-version: "12.14.1" - name: Cache node modules uses: actions/cache@v2 env: cache-name: cache-node-modules with: # npm cache files are stored in `~/.npm` on Linux/macOS path: | ~/.npm ~/.cache key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }} restore-keys: | ${{ runner.os }}-build-${{ env.cache-name }}- ${{ runner.os }}-build- ${{ runner.os }}- - run: npm ci - name: Detect changes between staging and master id: "detect-changes-between-staging-and-master" uses: fjenik/detect-changes-between-two-branches@0.0.3 with: repo-token: ${{ secrets.GITHUB_TOKEN }} target-branch: master source-branch: staging - name: Detect changes output run: echo "Output of detect changes ${{ steps.detect-changes-between-staging-and-master.outputs.is-source-branch-ahead }}" - name: Check if pr already exists id: "check-if-pr-already-exists" if: ${{ steps.detect-changes-between-staging-and-master.outputs.is-source-branch-ahead == 'true' }} uses: fjenik/check-if-pr-exists@0.0.3 with: repo-token: ${{ secrets.GITHUB_TOKEN }} target-branch: master source-branch: staging - name: Print output run: echo "Output of pul request already exists ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created }}" - name: Get new version and description id: get-new-version-and-description if: ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created == 'false' }} env: GITHUB_AUTH: ${{secrets.GITHUB_TOKEN}} run: | node ${GITHUB_WORKSPACE}/.github/workflows/getRcDescriptionAndNewTag - name: Check outputs new version if: ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created == 'false' }} run: echo "Output of detect changes ${{ steps.get-new-version-and-description.outputs.NEW_VERSION }}" - name: Check outputs new changes if: ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created == 'false' }} run: echo "Output of detect changes ${{ steps.get-new-version-and-description.outputs.NEW_CHANGES }}" - name: Create pull request id: "create-pr" if: ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created == 'false' }} uses: fjenik/create-pull-request@0.0.13 with: repo-token: ${{ secrets.GITHUB_TOKEN }} target-branch: master source-branch: staging pr-title: "👾 Release candidate ${{ steps.get-new-version-and-description.outputs.NEW_VERSION }}" pr-body: | _This PR was generated via github actions workflow to create RC pull request_ If you are tagged in some of the issues this RC is made of, please mark these TODO checkboxes when you've successfully tested the feature on the staging environment [staging environment](link to our staging environment) ${{ steps.get-new-version-and-description.outputs.NEW_CHANGES }} **Don't squash merge this PR** - name: Slack Notification if: ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created == 'false' }} uses: tokorom/action-slack-incoming-webhook@main env: INCOMING_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }} with: text: "👾 Hello! New RC (${{ steps.get-new-version-and-description.outputs.NEW_VERSION }}) was created! You know what to do! https://github.com///pull/${{ steps.create-pr.outputs.pull-id }} :fire:" ``` ### Push to master flow In this workflow, we are focusing on the automatization of creating a new version (tag) and appending new changes into the changelog after a merge into the `master` branch. This workflow is the main improvement over our old release process because back then, we were not creating a new version after each merge into master, and also, we were not updating our changelog after a direct merge into the `master` branch (hotfix). With this workflow, we automate the longest and the most annoying part of our release process, where we had to manually parse the changelog and create a new git tag. We also improved our versioning - if the pull request is a part of our release process (when we are releasing from the staging branch), it's a **minor increment** but when we're merging into the master from a different branch it's a hotfix, so we're just **changing the patch version**. Major versions will only be incremented after a big change - like migration to mono-repository or architecture migration into serverless. #### Steps 1. Checkout repository into our GitHub action (if you want to work with anything else than your current repository version - for example tags, you have to add `[input fetch-depth: 0](https://github.com/actions/checkout#checkout-v2)`) 2. Setup NodeJS 3. Setup GitHub action cache 4. [Install the packages](https://docs.npmjs.com/cli/v7/commands/npm-ci) 5. [Setup git config](https://git-scm.com/docs/git-config) 6. Run core script 1. Validate inputs 2. Get new changes for changelog with `lerna-changelog` and its modification to correct format 3. Get a current version of the repository - `[git describe --abbrev=0](https://git-scm.com/docs/git-describe)` 4. Get new version by input parameter (patch or minor) - with [semverInc package](https://www.npmjs.com/package/semver) 5. Format data to changelog 6. Modify the changelog 7. Add change(s) to git - `git add --all` 8. Commit all new changes - `git commit -am 'Update changelog ${newVersion} [skip ci]` - we are using \[skip ci\] to protect ourselves from recursive workflow triggering 9. Generate a new tag and push into origin - `npx lerna version ${newVersion} --force-publish --yes -m "publish new tag - %s [skip-ci]` #### Workflow ``` name: Add new tag and generate changelog on: push: branches: - master paths-ignore: - "./CHANGELOG.md" - "./package.json" - "./api/package.json" - "./frontend/package.json" jobs: generate-new-tag: runs-on: ubuntu-latest name: Add new tag and generate changelog steps: - uses: actions/checkout@v2 with: fetch-depth: 0 - uses: actions/setup-node@v1 with: node-version: "12.14.1" - name: Cache node modules uses: actions/cache@v2 env: cache-name: cache-node-modules with: # npm cache files are stored in `~/.npm` on Linux/macOS path: | ~/.npm ~/.cache key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }} restore-keys: | ${{ runner.os }}-build-${{ env.cache-name }}- ${{ runner.os }}-build- ${{ runner.os }}- - run: npm ci - name: Get version type to change env: VERSION: ${{ contains(github.event.head_commit.message, 'staging') }} ACTIONS_ALLOW_UNSECURE_COMMANDS: true run: | if ${VERSION} == true; then echo "::set-env name=CHANGE_VERSION::minor" else echo "::set-env name=CHANGE_VERSION::patch" fi - name: "Print version change" run: "echo ${{env.CHANGE_VERSION}}" - name: Setup git config run: | # setup the username and email. I tend to use 'GitHub Actions Bot' with no email by default git config user.name "GitHub Actions Bot" git config user.email "<>" - name: Run core script env: GITHUB_AUTH: ${{secrets.GITHUB_TOKEN}} run: | node ${GITHUB_WORKSPACE}/.github/workflows/generateChangelogAndCreateTag -v ${{env.CHANGE_VERSION}} ``` ### Open PR from `master` to `develop` This workflow is used to **synchronize the develop branch with the master branch**. We often had a problem with **outdated code in our develop branch** (hotfix merges) when we wanted to release, so we needed to merge master into develop, which meant **we had a lot of conflicts to handle**. With this workflow, _we simplified our lives_, and with every push to master we **automatically create a PR** (if it does not exist) which we _immediately review and then merge into develop._ #### Steps On every push into the `master` branch, this is what we do: 1. Check if `master` branch is **ahead** of `develop` - if not, the rest of the steps are skipped 2. Check if this PR already exists - if not, the rest of the steps are skipped 3. Create PR from `master` to `develop` #### Workflow ``` name: Create pull request from master to develop on: push: branches: - master jobs: create-master-to-develop-pr: runs-on: ubuntu-latest name: Create pull request from master to develop steps: - name: Detect changes between master and develop id: "detect-changes-between-master-and-develop" uses: fjenik/detect-changes-between-two-branches@0.0.3 with: repo-token: ${{ secrets.GITHUB_TOKEN }} target-branch: develop source-branch: master - name: Detect changes output run: echo "Output of detect changes ${{ steps.detect-changes-between-master-and-develop.outputs.is-source-branch-ahead }}" - name: Check if pr already exists id: "check-if-pr-already-exists" if: ${{ steps.detect-changes-between-master-and-develop.outputs.is-source-branch-ahead == 'true' }} uses: fjenik/check-if-pr-exists@0.0.3 with: repo-token: ${{ secrets.GITHUB_TOKEN }} target-branch: develop source-branch: master - name: Print output run: echo "Output of pull request already exists ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created }}" - name: Create pull request if: ${{ steps.check-if-pr-already-exists.outputs.is-pr-already-created == 'false' }} uses: fjenik/create-pull-request@0.0.12 with: repo-token: ${{ secrets.GITHUB_TOKEN }} target-branch: develop source-branch: master pr-body: | _This PR was generated via github actions workflow to keep `develop` always in sync with `master` branch_ Please review and merge. **Don't squash merge this PR** pr-title: "👾 Merge `master` to `develop`" ``` ## Results After we developed this, we were surprised by how easy it was to implement this improvement and how it worked on the first try. We didn't run into any big logical problems or any bugs during the development process. Why didn't we do it sooner? I don't know. But I'm glad that we have done it. ## Conclusion In this article, we've talked about the automatization of the release process in just 3 clicks on GitHub. We walked through the design of our solution, and how we planned this improvement, and we talked about each workflow separately. With our new release process, we are able to deliver new features anytime and by anyone. A person who will be responsible for the next release will not have to know anything about our application or even current architecture. It's simple. You just have to follow simple instructions (merge, merge, and **MERGE**). As always, there are a lot of things to improve. There are new ideas every time on how to simplify or improve our product delivery. In the near future we are going to implement notifications on environment variables changes, because, in larger releases, these changes can be forgotten.