Change-log generator - GitHub + CircleCI

A changelog is a file which contains a curated, chronologically ordered list of notable changes for each version of a project.

Pavol Madar

A changelog is a file which contains a curated, chronologically ordered list of notable changes for each version of a project. Keep a Changelog

It’s the best way to document changes in an application. If you have a bigger application and, let's say you want to fix bugs and deliver new features on a weekly basis, it becomes really hard to follow all the changes without a well structured documentation and versioning of changes. Change-log has benefits for all the members of the team. Testers see what to test in the new version, developers can easily get a sense of what has been fixed or changed and see the current version of the app and product manager can track the progress and see the pace of the development.

Why do we need another change-log generator?

If you search for change-log generators you can find a few, but there is one drawback. Every change-log library I’ve been able to find generates the change-log according to the commits and not closed issues. As a result of this, it contains no issue descriptions. But in our case, we need to have a list of issues with bug/feature description that everybody can easily understand in our change-log.

Dev workflow:

Tester creates a bug report
Bug report is checked and Github issues with additional technical description of the problem are created
The bug is eventually fixed and PR linked to the issue is opened
We then wait for the reviews and merge the PR - the issue is automatically closed

After some time, we are going to deploy new version of the codebase. We are using git-flow for managing the branches.

Deployment workflow:

Create a new release branch from the development branch (e.g. release/1.0.2) and push it to origin
Our CI service (CircleCI) runs the tests and deployment which also contains change-log generator scripts
All the issues that have been closed until the last deployment are published under the new release version in our change-log
At the same time, they are marked as deployed by adding a comment that contains the current version number
The script sends a slack notification about the new deploy

Change-log generator implementation

Let’s walk through the implementation now. We need to add these scripts in the package.json file:

"scripts": {
	...,
	"get-closed-issues": "node ./scripts/get-closed-issues.js",
	"create-changelog": "node ./scripts/create-changelog.js",
	"send-changelog-notification": "node ./scripts/send-changelog-notification.js"
}

We will call each one of these in the next sections from our CI service. First we need to call the get-closed-issues script which fetches all the issues. This script will be called as the first thing in the CI workflow in the getClosedIssues job. After successful test, build and deploy jobs, we are going to run create-changelog script which generates and deploys HTML file with the change-log. send-changelog-notification will send a slack notification about successful deploy. The first script will run in the generateAndDeployChangelog job and the second one in the sendChangelogNotification job.

Get closed issues

getClosedIssues job fetches the latest issues from GitHub, saves it to the closed-issues.json and persists it to the CircleCI workspace.

# .circle/config.yml
  getClosedIssues:
    <<: *defaults
    steps:
      - checkout
      - run: npm install
      - run: mkdir tmp
      - run: npm run get-closed-issues -- --output="closed-issues.json"
      - persist_to_workspace:
          root: ./
          paths:
            - closed-issues.json

getClosedIssues job runs as the first thing together with the test and build jobs in our CI workflow:

# .circle/config.yml
workflows:
  build-deploy:
    jobs:
      - getClosedIssues:
          filters:
            branches:
              only:
                - /release\/.*/
      - build:
          filters:
            branches:
              only:
                - develop
                - /release\/.*/
                - beta
                - staging
                - master

Now, let’s take a look at the get-closed-issues script.

// get-closed-issues.js
const fs = require('fs')
const util = require('util')
const exec = util.promisify(require('child_process').exec)
const args = require('yargs').argv
const fetch = require('node-fetch')
const issuesFragment = require('./issuesFragment')
const getIssueMeta = require('./getIssueMeta')

const owner = process.env.GITHUB_OWNER
const repo = process.env.GITHUB_REPO
const authToken = process.env.GITHUB_AUTH_TOKEN
const branch = process.env.CIRCLE_BRANCH
const developmentBranch = process.env.DEVELOPMENT_BRANCH
const changelogLabel = process.env.CHANGELOG_LABEL

In the first part of the script we assign all of the environmental variables, that will be needed in the following parts of the script. As you can see, we need to get owner, repo and authToken from our repository. Next up we assign the branch we are deploying right now, developmentBranch from which we’ve created current branch and the label (changelogLabel) which marks the issues to appear in the change-log.

// get-closed-issues.js
const getClosedIssues = async () => {
  const { stdout } = await exec(`git rev-list --right-only --count origin/${branch}...origin/${developmentBranch}`)
  const commitsBehindDevelopmentBranch = Number.parseInt(stdout)

In the beginning of getClosedIssues function, we checked if the current branch is behind the development branch and if not, we continue generating change-log. If the current branch is behind the development branch it means that there are new commits (closed issues) in development branch which we are not able to recognise from the issues closed in the current branch. If we want to be 100% confident about closed issues in the current branch, the development and current branch have to be in sync. I was not able to find any way around this, but it should be fine because you usually create change-log for test stage directly from the development branch.

// get-closed-issues.js
	let newIssues = []
  if (commitsBehindDevelopmentBranch === 0) {
    const issueQueryResponse = await fetch('https://api.github.com/graphql', {
      method: 'POST',
      headers: { Authorization: `bearer ${authToken}` },
      body: JSON.stringify({
        query: `query issues {
          repository(owner: "${owner}", name: "${repo}") {
            issues(first: 100, states: CLOSED, orderBy: {field: UPDATED_AT, direction: DESC}) {
              nodes {
                number
                title
                body
                bodyHTML
                createdAt
                closedAt
                comments(last: 5) {
                  nodes {
                    body
                  }
                }
                assignees(first: 10) {
                  nodes {
                    name
                  }
                }
                labels(first: 15) {
                  nodes {
                    name
                  }
                }
                url
              }
            }
          }
        }`
      })
    }).then(res => res.json())

Next thing to do is to fetch the latest issues.

// get-closed-issues.js
    const issues = issueQueryResponse.data.repository.issues.nodes
    newIssues = issues.filter(({ comments, labels }) => {
      const isInvalid = labels.nodes.some(({ name }) => name.includes('Rejected') || name.includes('Duplicate') || !name.includes(changelogLabel))
      const issueMeta = getIssueMeta(comments.nodes)
      return !isInvalid && !issueMeta
    })
  }

After that, we filter out the issues that contain Rejected or Duplicate label or do not contain changelogLabel.

// get-closed-issues.js
  fs.writeFileSync(args.output, JSON.stringify({ issues: newIssues }))
}

getClosedIssues()

Lastly, we save the file to the disk at the location provided in the script argument.

Generate and deploy change-log

generateAndDeployChangelog will run as the last thing after successful test, build and deploy jobs.

# .circle/config.yml
  generateAndDeployChangelog:
    <<: *defaults
    steps:
      - checkout
      - attach_workspace:
          at: .
      - run: npm install
      - run: mkdir changelog
      - run: npm run create-changelog -- --input="./closed-issues.json" --output="./changelog/index.html"
      - run: sudo npm install -g surge
      - run: surge --project ./changelog --domain http://changelog.yourdomain.surge.sh

# .circle/config.yml
      - generateAndDeployChangelog:
          requires:
            - deploy

In this job, we are going to attach a workspace where we’ve persisted closed-issues.json and generate the HTML change-log file from it. In the next steps of the job, we install and use surge.sh for deployment. Surge.sh is service for hosting static files and I’ve chosen this service because it’s simple and works great, but you can choose whatever hosting provider you like. But the most interesting thing in this job is create-changelog script, so let’s explore that, piece by piece. We first create fetchIssuesGroupedByVersion function which will be used in the main function of this script. It fetches the last 100 closed issues, then filters only those, which have already been deployed and groups them by the release version.

// create-changelog.js
const fetchIssuesGroupedByVersion = async () => {
  const issueQueryResponse = await fetch('https://api.github.com/graphql', {
    method: 'POST',
    headers: { Authorization: `bearer ${authToken}` },
    body: JSON.stringify({
      query: `query issues {
        repository(owner: "${owner}", name: "${repo}") {
          issues(first: 100, states: CLOSED, orderBy: {field: UPDATED_AT, direction: DESC}) {
            nodes {
              number
              title
              body
              bodyHTML
              createdAt
              closedAt
              comments(last: 5) {
                nodes {
                  body
                }
              }
              assignees(first: 10) {
                nodes {
                  name
                }
              }
              labels(first: 15) {
                nodes {
                  name
                }
              }
              url
            }
          }
        }
      }`
    })
  }).then(res => res.json())

In this part of the function, we are fetching the last 100 closed issues. In our case it’s enough to have the last 100 closed issues in the change-log but the number is up to you.

// create-changelog.js
  const releases = new Map()
  issueQueryResponse.data.repository.issues.nodes.forEach((issue) => {
    const isInvalid = issue.labels.nodes.some(({ name }) => name.includes('Rejected') || name.includes('Duplicate') || !name.includes(changelogLabel))
    const issueMeta = getIssueMeta(issue.comments.nodes)
    if (!isInvalid && issueMeta) {
      releases.set(issueMeta, [...(releases.get(issueMeta) || []), issue])
    }
  }, {})
  const arrayOfVersions = []
  releases.forEach((issues, issueMeta) => {
    const { version, deployedAt } = JSON.parse(issueMeta)
    arrayOfVersions.push({ version, deployedAt, issues })
  })
  return orderBy(arrayOfVersions, 'version', 'desc')
}

In the next part, we are filtering out the invalid issues (same as last time) and the issues which do not have any meta information. Function getIssueMeta checks the body of the issue and the meta information. This part is a little bit tricky, because there is no official way to store metadata in GitHub issues. After some research, I’ve found that you can store metadata in the issue comments. You just need to wrap it in an HTML comment, so that it’s not visible in the GitHub UI. I use this trick to mark the issue as deployed by adding a comment like this:

Issue has been deployed in **${branch}**.<!--${JSON.stringify({ version: branch, deployedAt })}-->

Then we can use the function getIssueMeta to check whether the issue has been already deployed or not and get the metadata, including the version containing the deployed issues and the date of deployment.

As you can see, we regenerate the whole change-log on every deploy. This way, you can change/fix your change-log before every release.

Let's move to the the main function of the script in the create-changelog script.

// create-changelog.js
const createChangelog = async () => {
  const closedIssuesFileContent = fs.readFileSync(args.input)
  const closedIssues = closedIssuesFileContent && JSON.parse(closedIssuesFileContent)
  let newIssues = []
  const oldIssuesGroupedByLabel = await fetchIssuesGroupedByVersion()
  if (closedIssues && closedIssues.issues && closedIssues.issues.length !== 0) {
    const deployedAt = new Date().toISOString()
    newIssues = [{ version: branch, deployedAt, issues: closedIssues.issues }]
    await Promise.all(closedIssues.issues.map(({ number }) =>
      fetch(`https://api.github.com/repos/${owner}/${repo}/issues/${number}/comments`, {
        method: 'POST',
        headers: {
          Accept: 'application/vnd.github.v3+json',
          Authorization: `bearer ${authToken}`
        },
        body: JSON.stringify({
          body: `Issue has been deployed in **${branch}**.<!--${JSON.stringify({ version: branch, deployedAt })}-->`
        })
      })
    ))
  }
  const HTML = ChangelogTemplate({
    versions: [
      ...newIssues,
      ...oldIssuesGroupedByLabel.map(({ version, deployedAt, issues }) => ({
        version,
        deployedAt,
        issues
      }))
    ]
  })
  fs.writeFileSync(args.output, HTML)
}

createChangelog()

In createChangelog, we are going to read the closed issues up to the latest release and then call fetchIssuesGroupedByVersion to fetch the issues from already deployed releases. Now we have all the data we need to generate the change-log, but before we do that, we call the mutation which marks all the issues after the latest release as closed and adds the metadata with the version and date of the release. Finally, we call ChangelogTemplate which creates an HTML file with the change-log. writeFileSync writes the HTML to a file, at the path provided in the script argument.

Send change-log notification to the Slack

The last thing that we want to do after we’ve generated and deployed the change-log, is to send a slack notification with the information about the deployment.

# .circle/config.yml
  sendChangelogNotification:
    <<: *defaults
    steps:
      - checkout
      - attach_workspace:
          at: .
      - run: npm install
      - run: npm run send-changelog-notification -- --input="./closed-issues.json"

# .circle/config.yml
      - sendChangelogNotification:
          requires:
            - generateAndDeployChangelog

// send-changelog-notification.js
const fs = require('fs')
const fetch = require('node-fetch')
const args = require('yargs').argv

const branch = process.env.CIRCLE_BRANCH
const slackUrl = process.env.SLACK_WEBHOOK_URL

const sendChangelogNotification = async () => {
  const closedIssuesFileContent = fs.readFileSync(args.input)
  const closedIssues = closedIssuesFileContent && JSON.parse(closedIssuesFileContent)
  if (closedIssues && closedIssues.issues && closedIssues.issues.length !== 0) {
    await fetch(slackUrl, {
      method: 'POST',
      body: JSON.stringify({
        username: `${branch}`,
        text: `_New version deployed_ \n${closedIssues.issues.reduce((acc, { title, url, number }) => `${acc}- ${title} <${url}|#${number}> \n`, '')}`,
        mrkdwn: true
      })
    })
  }
}

sendChangelogNotification()

In this section, we’ve created and delivered a markdown message to Slack. The message contains the stage name, version of the release and the list of closed issues.

Conclusion

We’ve walked through the process of creating and publishing a change-log. You can check the whole implementation in the example repository. As always, there is a lot of space to improve, e.g. it would be nice to transform the scripts into a library, or create a GitHub App / use GitHub Actions, so stay tuned for the next posts.