Tuesday, August 1, 2017
Setting up Nomad and Consul in EC2 instead of Mesophere
Setting up Nomad and Consul in EC2 instead of Mesophere
Setting up Nomad instead of Mesophere. Notes on me setting up a Nomad set of servers for development in EC2.
Server nodes for Nomad and Consul
Three EC2 Medium (triad) machines.
Each server runs
- consul server agent,
- nomad server agent
- No Docker here
Worker Nodes (aka client nodes)
Three to X client Agent Nodes (EC2 LARGE or better)
Each client agent node runs
- consul client agent
- nomad server agent
- Docker daemon
These connect to home server triads (consul and nomad).
Prod cluster
Server nodes for Nomad and Consul
Five EC2 Large (triad) machines.
Each server runs
- consul server agent,
- nomad server agent
- No Docker here
Worker Nodes (aka client nodes)
Three to X client Agent Nodes (as large as we need, at least one machine per AZ)
Each client agent node runs
- consul client agent
- nomad server agent
- Docker daemon
These connect to home server triads (consul and nomad).
Implementation details
You have four roles
server1
server2
server3
worker-node
All worker-nodes are ephemeral. They can get blown away.
The servers:
server1
, server2
, server3
form a triad cluster. Any triad member can die and be replaced. They should be started back up with the same basic ip address info.Since we are running
consul
, nomad
, and zookeeper
on the triad, there is no advantage to using consul
for nomad
server node discovery because consul
is installed on the same triad of servers.The server
server1
is a bit special because it has the script to connect the servers into a cluster. For the most part, server1
is identical to the others.Server 1
Server1 Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :
$script = <<SCRIPT
# Update apt and get dependencies
sudo apt-get update
sudo apt-get install -y unzip curl wget vim
# Download Nomad
echo Fetching Nomad...
cd /tmp/
curl -sSL https://releases.hashicorp.com/nomad/0.2.3/nomad_0.2.3_linux_amd64.zip -o nomad.zip
echo Installing Nomad...
unzip nomad.zip
sudo chmod +x nomad
sudo mv nomad /usr/bin/nomad
sudo mkdir -p /etc/nomad.d
sudo chmod a+w /etc/nomad.d
sudo mkdir -p /opt/nomad/data
sudo mkdir -p /var/log/nomad
sudo chmod a+w /var/log/nomad
sudo cp /vagrant/server.hcl /etc/nomad.d/
echo Fetching Consul...
curl -sSL https://releases.hashicorp.com/consul/0.6.3/consul_0.6.3_linux_amd64.zip -o consul.zip
echo Installing Consul...
unzip consul.zip
sudo chmod +x consul
sudo mv consul /usr/bin/consul
sudo mkdir -p /etc/consul.d
sudo chmod a+w /etc/consul.d
sudo mkdir -p /opt/consul/data
sudo mkdir -p /var/log/consul
sudo chmod a+w /var/log/consul
sudo cp /vagrant/consul.json /etc/consul.d/
echo Starting nomad
cd ~
sudo nohup nomad agent -config /etc/nomad.d/server.hcl &>nomad.log &
echo Starting Consul
sudo nohup consul agent -config-file /etc/consul.d/consul.json &>consul.log &
SCRIPT
Vagrant.configure(2) do |config|
config.vm.box = "base-box"
config.vm.hostname = "nomad"
config.vm.provision "shell", inline: $script, privileged: false
config.vm.network "private_network", ip: "10.21.0.10"
# Increase memory for Parallels Desktop
config.vm.provider "parallels" do |p, o|
p.memory = "1024"
end
# Increase memory for Virtualbox
config.vm.provider "virtualbox" do |vb|
vb.memory = "1024"
end
# Increase memory for VMware
["vmware_fusion", "vmware_workstation"].each do |p|
config.vm.provider p do |v|
v.vmx["memsize"] = "1024"
end
end
config.vm.provider :aws do |aws, override|
aws.keypair_name = "my-app-key"
aws.region = "us-west-2"
# Ubuntu public Amazon EC2 image for Ubuntu 64 bit
aws.ami = "ami-9abea4fb"
override.ssh.username = "ubuntu"
override.ssh.private_key_path = "/opt/aws/my-app-key.pem"
aws.tags = {
Name => my-app-cluster-server-1
}
# vpc-d14dacb5
aws.subnet_id = "subnet-abc123ab"
aws.security_groups = "sg-abc123ab"
aws.private_ip_address="10.21.0.10"
override.vm.hostname = "ip-21-10-0-10"
# override.ssh.host = "10.20.0.10" //NOT EXPOSED TO VPN traffic yet
# We have to use public IP address because we dont have the VPC tied to vpn traffic
aws.associate_public_ip = true
end
end
server.hcl
bind_addr = "10.21.0.10"
advertise {
# We need to specify our hosts IP because we cant
# advertise 0.0.0.0 to other nodes in our cluster.
rpc = "10.21.0.10:4647"
}
# Increase log verbosity
log_level = "DEBUG"
# Setup data dir
data_dir = "/opt/nomad/data"
# Enable the server
server {
enabled = true
start_join = ["10.21.0.11", "10.21.0.10", "10.21.0.12"]
retry_join = ["10.21.0.11", "10.21.0.10", "10.21.0.12"]
retry_interval = "15s"
}
server-bootstrap.hcl
bind_addr = "10.21.0.10"
advertise {
# We need to specify our hosts IP because we cant
# advertise 0.0.0.0 to other nodes in our cluster.
rpc = "10.21.0.10:4647"
}
# Increase log verbosity
log_level = "DEBUG"
# Setup data dir
data_dir = "/opt/nomad/data"
# Enable the server
server {
enabled = true
# Self-elect, should be 3 or 5 for production
bootstrap_expect = 3
}
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.