OpenStack is a platform to build private and public cloud, it includes so many projects to do this, each project has its function. For example: Neutron is to provide network as a service, Nova is to provide compute as a service, etc. There is a large OpenStack community including developers and users to support it like developing new features. So one a big question is that how to verify source code from developers to make sure the source code is already in production. Although there is no any QA (Quality Assurance) and Tester. In order to answer this question, OpenStack community has launched CI (Continuous Integration). Each source code pushed will be verified carefully before merging.

When start investigating CI in OpenStack, you will see that OpenStack uses its OpenStack to build CI for OpenStack. That will make you confuse. Don’t worry, I will explain to you clearly.

Before contributing, each developer needs to have and understand DevOps culture in OpenStack and understand the workflow.

The workflow of contributing as following:

The workflow contributing

OpenStack uses GIT model. A developer will clone source code on master branch from GitHub (like Nova, Neutron, Keystone), then they have to check out a new branch with a name related to bugs or their feature. After coding the new feature then they must have unittest or functional tests, even api test which depends on the scope of feature. After passing all old and new tests on local, they can push new source code to Gerrit (OpenStack use Gerrit to store source code for reviewing). At that time, OpenStack CI will run all test cases in the project and mask +1 if passing and -1 if not. Like this picture.

CI OpenStack result

So the requirement of each developer when uploading new feature or bugs is to write full unit tests and know how to set up a CI test for the feature if necessary. It will be reviewed by other developers and core-reviewer, they can discuss on the patch set to get the best solution. After getting an agreement then two core-reviewers will mask +2 and workflow +1, then CI server will run gate tests. If passing all gate tests then the source will be merged to master branch.

For now, maybe you ask your self how can OpenStack build such good CI like that?

Architecture OpenStack

The above picture is the CI architecture. CI at OpenStack scale:

  • > 1500 repositories: There are more 1500 repos in OpenStack GitHub, it means the repos will be used CI system.
  • 2000 jobs per hour: CI server will create and delete 2000 VMs (Virtual Machine) per hour, that is an impressive number.
  • 13 regions of 5 public and 2 private OpenStack cloud: CI server backend is supported by the private and public clouds on 13 regions.

OpenStack uses Zuul as CI server and ansible playbook to control virtual machine.

  • Zuul: This component is the heart of Zuul, it listens events from Gerrit and requests to other component run necessary jobs when having a patch set pushed.

  • Zuul merger: to merge patch set to GitHub. When passing all test cases then Zuul merger will start merging the patch set to GitHub.

  • Zuul launcher: to run ansible playbook and report result to Zuul. After having VM information then Zuul will send it to Zuul launcher to run ansible playbook and send result to Zull after finishing.

  • Nodepool: to manager Cloud backend. It is a middle layer to get requirements from Zuul and interact with OpenStack API to create VM for testing.

  • Cloud 1, 2, 3: This is private and public cloud, which are sponsored by Cloud providers. Especially, the private and public cloud is built from OpenStack with last release. For example: the clouds are built on OpenStack Pike or Ocata.

When developers submit a patch set to Gerrit, then a notification will be sent to Zuul, then Zuul will request Cloud to provide a VM via Nodepool. Nodepool will send the information of the VM to Zuul, Zuul will request to Zuul Launcher with the information to run ansible playbook. After that, all test cases will be run on the VM to make sure source code is good. When core-reviewers accept this patch set then Zuul merger will start to merge this patch to GitHub repo. That’s all steps of a patch set before merging.

Conclusion: OpenStack is a good solution to build private and public cloud. More over, it is also a very good solution for backend in CICD. CI/CD server can interact with OpenStack system via API to provide VM for running test cases and deploy source code. After finishing, the VM will be deleted automatically to saving resources.