In the early days of GitLab.com, most of the application, including Rails worker processes, Sidekiq background processes, and Git storage, all ran on a single server. A single server is easy to deploy to and maintain. The same structure is what most smaller GitLab instances still use for their self-managed Omnibus installation. Scaling is done vertically, meaning; adding more RAM, CPU, and disk space.
Moving from vertical to horizontal scaling
Soon we ran out of options to continue scaling the system vertically, and we had to move to scaling horizontally by adding new servers. To have the repositories available on all the nodes, NFS (Network File System) was used to mount these to each application server and background workers. NFS is a well-known technology for sharing file systems across a network. For each server, each storage node needed to be mounted. The advantage: GitLab.com could keep adding more servers and scale. However NFS had multiple disadvantages too: the visibility is decreased to what type of file system operation is performed. Even worse, one NFS storage node's outage impacted the whole site, and took the whole site down. On the other hand, Git operations can be quite CPU/IOPS intensive too, so we began a balancing act between adding more nodes, and thus reducing reliability, versus scaling nodes vertically.
Considering NFS alternatives
Over two years ago, we started to look for alternatives. One of the first ideas was to remove the dependency on NFS with Ceph. Ceph is a distributed file system that was meant to replace NFS in an architecture like ours. Like NFS, this would solve our scaling problem on the system level, meaning that little to no changes would be required to GitLab as an application. However, running a Ceph cluster in the cloud didn't have the performance characteristics that were required. Briefly we flirted with the idea of moving away from the cloud, but this would have had major implications for our own infrastructure team, and given that many of our customers do run in the cloud, we decided to stay in the cloud.
Introducing Gitaly
So it was clear that the application needed to be redesigned, and a new service would be introduced to handle all Git requests. We named it Gitaly.
The planned architecture at the project start
As the diagram shows, the new Git server would have a number of distinct clients. To make sure the protocol for the server and its clients is well defined, Protocol Buffers was used. The client calls are handled by leveraging gRPC. Combined, they allowed us to iteratively add RPCs and move away from NFS, in favor of an HTTP boundary. With the technologies chosen, the migration started. The ultimate goal: v1.0, meaning no disk access was required to the Git storage nodes for GitLab.com.
Shipping such an architectural change should not influence the performance, nor the stability of the self-managed installations of GitLab, so for each RPC a feature flag gated the use of it. When the RPC had gone through a series of tests on both correctness and performance impact, the gate was removed. To determine stability we used Prometheus for monitoring and the ELK stack for sifting through massive numbers of structured log messages.
The server was written in Go, while the application is a large Rails monolith.
Rails had a great amount of code that was still very valuable. This code got
extracted to the lib/gitlab/git directory, allowing easier vendoring. The idea
was to start a sidecar next to the Go server, reusing the old code. About once a week the
code would be re-vendored. This allowed Ruby developers on other teams to
write code once, and ship it. Bonus points could be earned if the boilerplate code
was written to call the same function in Ruby!
The new service wasn't all sunshine and rainbows though, at times it felt like the improved visibility was hurting our ability to ship. For example, it became clear that the illusion of an attached disk created N + 1 problems. And even though this is a well-known problem in Ruby on Rails, the tools to combat it are all tailored toward using it with ActiveRecord, Rails' ORM.
Nearing v1.0
With each RPC introduced, v1.0 was getting closer and closer. But how could we be sure everything was migrated before unmounting all NFS mount points? A trip switch got introduced, guarding the details required to get to the full path of each repository. Without this data there was no way to execute any Git operation through NFS. Luckily, the trip switch never went off, so now it was clear NFS wasn't being used. The next step was unmounting on our staging environment! Again, this was very uneventful. Leaving the volumes unmounted for a full week, and not seeing any indication of unexpected errors, the logical next step was our production instance.
Days later we started rolling out these changes to production: first the background workers were unmounted, than we moved onto higher impact services. At the end of the day, all drives were unmounted without customer impact.
What's next?
So, where is this v1.0 tag? We didn't tag it, and I don't think we will. v1.0 is a state for our Git infrastructure, and a goal for the team, rather than the code base. That being said, the next mental goal is allowing all customers to run without NFS. At the time of writing, some features like administrative tasks, aren't using Gitaly just yet. These are slated for v1.1, and our next objective.
Want to know more about our Gitaly journey? Read about how we're making your Git data highly available with Praefect and how a fix in Go 1.9 sped up our Gitaly service by 30x.
Photo by Jason Hafso on Unsplash
