Du Wei is a programmer who has been involved in Internet R&D and O&M for nearly a decade. He is one of the co-authors of "Linux System Case Study". He is currently working at Brightwind, focusing on DevOps, cloud computing, big data and other related fields.
Application Background
HiAR is a new-generation Augmented Reality (AR) development platform built by Brightwind, providing easy-to-use, powerful, cross-platform AR services. It allows developers to easily use the most cutting-edge computer vision technology and computer graphics technology to quickly build personalized AR applications.
Cloud service is an important infrastructure in HiAR platform. Whether it's highly available or scalable, service discovery plays an integral role. Before using service discovery, we encountered several pain points:
◆ To add a service node to the system, we need to manually modify the configuration file of Nginx/LVS, and modify DNS records.
◆ When the application service releases a new version, we still need to manually modify the Nginx configuration file to take the node offline, wait for the release to be successful, and then modify the Nginx configuration file again to bring the service online.
◆ Although we have since improved the operations of the above two scenarios by writing scripts to improve the process to a semi-automatic, semi-manual way, it is still not very convenient, and combining it with service registration can be fully automated.
◆ The DNS on the intranet is down and we need to maintain the DNS service.
◆ There is no service registration, which limits Docker to being used as a lightweight virtual machine.
Now, with service discovery, everything is easy and fun. Adding or subtracting service nodes can automatically update the Nginx/LVS configuration file; DNS throw it aside! Just use IP; access Mesos+Docker to play with elastic scaling.
Why Consul
There have been many articles comparing Zookeeper, etcd, and Consul, so I won't repeat the analogy here. Consul has low O&M costs, is simple to deploy, easy to use, and has everything you need, which should be cost-effective for small and medium-sized teams.
Before we get into the real world, let's take a look at what Consul has to offer.
◆ Service registration. Tells the service registry that a new service has been added via the HTTP API or DNS.
◆ Service discovery. Through the HTTP API or DNS, the address and port of the target service is known.
◆ Health check. Support multiple ways, HTTP, TCP, Docker, Shell script customized monitoring.
◆ Configuration Template. Consul Template is responsible for periodically getting information from the service registry and automatically updating the configuration file and reloading it if there are changes.
The above four points already meet the needs of many organizations. Of course, this is not all Consul, Consul also has a lot of icing on the cake, such as: visual Web interface, support for multiple data centers.
Practical experience
Our use of Consul can be summarized in four areas: deployment, application, management, and upgrade.
Deployment
Consul Cluster has two roles, Server and Client. server is usually 3~5, which is the official recommendation. consul Client is the node that needs to do service registration or service discovery.
Consul deployment is simple, out-of-the-box, a consul executable file, there is no messy dependencies. Download the compiled Consul agent executable from the official website and upload it to all the nodes in the Server and Client roles, and you're ready to start the consul agent.
Let's take a look at how to start a Consul cluster (3 Servers, 1 Client).
Experimental environment:
server01 192.168.1.11
server02 192.168.1.12
server03 192.168.1.13
client01 192.168.1.21
Separately. Log in to Server01, Server02, and Server03 and start the agent.
[worker@server01 ~]$ consul agent -server -bootstrap-expect 2 -data-dir /tmp/consul -bind= 192.168.1.11 -node=server01
[worker@server02 ~]$ consul agent -server -bootstrap-expect 2 -data-dir /tmp/consul -bind=192.168.1.12 - node=server02
[worker@server03 ~]$ consul agent -server -bootstrap-expect 2 -data-dir /tmp/consul -bind=192.168.1.13 -node=server03
Log in to Server03 in a new window and join the cluster of Server01 and Server02.
[worker@server03 ~]$ consul join 192.168.1.11 192.168.1.12
The above steps are done to initialize the Server node, and you can rejoin the cluster later on by starting it with the -rejoin parameter.
[worker@server01 ~]$ consul agent -server -bootstrap-expect 2 -data-dir /tmp/consul -bind=192.168.1.11 -node=server01 -rejoin
[ worker@server02 ~]$ consul agent -server -bootstrap-expect 2 -data-dir /tmp/consul -bind=192.168.1.12 -node=server02 -rejoin
[worker@ server03 ~]$ consul agent -server -bootstrap-expect 2 -data-dir /tmp/consul -bind=192.168.1.13 -node=server03 -rejoin
And that's it for the three Server nodes deployed! Next, deploy the Client node. Next, the Client nodes are deployed, and like the Server nodes, there are three steps: initial startup, manual join, and rejoin to the cluster.
[worker@client01 ~]$ consul agent -data-dir /tmp/consul -bind=192.168.1.21 -node=client01
On Client01, you can open a new login window and join Server01's cluster.
[worker@client01 ~]$ consul join 192.168.1.11
Client01 node can be rejoined to the cluster for future maintenance by starting it with the -rejoin parameter.
[worker@client01 ~]$ consul agent -data-dir /tmp/consul -bind=192.168.1.21 -node=client01 -rejoin
Up to this point, we have built a Consul cluster. However, how to do service registration and service discovery? This has to be closely aligned with the actual requirements and is further explained in the next subsections.
Application
Consul does not stand alone. In order to take full advantage of Consul, it can be applied in combination with tools such as Nginx, LVS, Docker, and so on.
Nginx and LVS are the basic components of the system, and RecoService, FeatureService, and SearchService are SOA-based internal services. The former discovers services to the Consul cluster and the latter registers services with the Consul cluster.Consul is the glue and the switch that makes the whole system work and realizes elastic scaling at low cost.
The access layer, Nginx, is responsible for reverse proxying and load balancing, and the Nginx nodes run two Consul-related services. One is Consul Agent, which is a Consul Client; the other is Consul Template, which is a service discovery and configuration update service.Consul Template is responsible for periodically querying the local Consul Agent, and if there is a change in the registration information of the relevant service, then it will update Nginx's configuration file and reload the Nginx services.
Running Consul Template is a key step in enabling elastic scaling:
$ consul-template -consul 127.0.0.1:8500 -template "/etc/nginx/conf/vhosts/test.ctmpl:/etc/ nginx/conf/vhosts/test.conf:nginx -s reload"
In the above command, test.conf is the virtual host configuration file for Nginx, and test.ctmpl is the template that corresponds to that configuration file. Here's the code snippet for the template on load balancing:
upstream test-cluster {
ip_hash;{{range service "test"}}
server {{.Address}}:{{.Port}};{{end}}
< p>}Logical layer, SOA-based internal service clusters. Communication between different internal service clusters requires service discovery, and LVS is introduced here for service discovery. The advantage is that you don't have to implement service discovery in the code of the internal service, and you have to do load balancing when the scale is large. Similar to Nginx in the access layer, LVS also uses Consul Template to periodically query the local Consul Agent, update the relevant configuration files, and then reload the service.
How do internal services register with the service center? There are two ways, one is through Consul's service registration HTTP API, by the service itself after the startup call API to register itself, and the other is by defining the service in the configuration file for registration. It is recommended to use the latter way to do service registration. How does it work? Read on :)
Add a configuration file consul.json to the project, specify the service name and service port, and add a health check as follows:
{
"service":
{
"name" : "test",
"port" : 9999,
"check":
{
"tcp": "127.0.0.1:9999",
"interval": "10s"
}
}
}
The final step to register the service is to specify a configuration file in the Consul agent starts by specifying the configuration file as follows:
$ consul agent -data-dir /tmp/consul -node=test -bind=192.168.1.21 -config-dir=/tmp/consul.json
Managing
One is node management, which is the management of the Consul process. Since Consul Agent itself is not highly available, it is necessary for us to take over the Consul process, we use Systemd, you can also choose Supervisord or Upstart which are process management tools.
The second is cluster management, Consul provides a visual management interface. You can view all services and nodes, as well as their health tests and current status.
Upgrading
Since Consul is related to the normal operation of the whole system, you still have to be very careful when upgrading. It is best to test the system in a test environment several times before upgrading it to a production environment. The upgrade scenarios can be categorized into the following three, which need to be addressed before upgrading.
◆ Specialized upgrades. Check the upgrade-specific page to see if there are any special instructions for the current version being upgraded. For example, if you are upgrading from a version before 0.5.1 to 0.6, you need to migrate the data with the help of the tool consul-migrate.
◆ Incompatible upgrades. Use consul -v to check the backward compatible protocol version number of the new version. When there is incompatibility with the current version, you need to upgrade in two steps. First, upgrade the entire cluster once with the parameter -protocal=old protocol version number, and then restart all nodes by removing the parameter -protocal from the startup command.
◆ Standard upgrade. If neither of the above is true, then congratulations, all you need to do is a simple standard upgrade. I.e., stop the old version of the agent and start the new version of the agent. ps: Actually, most cases are standard upgrades.
The recommended order for upgrading nodes is to upgrade the Server's Follower node first, then the Server's Leader node, and finally all Client's nodes.
Conclusion
The introduction of service registration and discovery in a system is a hair-trigger transformation, but the entire system architecture will benefit from it, especially the modern microservice architecture. I believe that many systems are equipped with load balancing, health checks, heartbeat detection, etc. Take advantage of service discovery, then elastic scaling, high availability of services, and gray scale releases, are naturally a matter of course.