Tuesday, August 1, 2017

Setting up Nomad and Consul in EC2 instead of Mesophere

Setting up Nomad and Consul in EC2 instead of Mesophere


Setting up Nomad instead of Mesophere. Notes on me setting up a Nomad set of servers for development in EC2.

Server nodes for Nomad and Consul

Three EC2 Medium (triad) machines.
Each server runs
  • consul server agent,
  • nomad server agent
  • No Docker here

Worker Nodes (aka client nodes)

Three to X client Agent Nodes (EC2 LARGE or better)
Each client agent node runs
  • consul client agent
  • nomad server agent
  • Docker daemon
These connect to home server triads (consul and nomad).

Prod cluster

Server nodes for Nomad and Consul

Five EC2 Large (triad) machines.
Each server runs
  • consul server agent,
  • nomad server agent
  • No Docker here

Worker Nodes (aka client nodes)

Three to X client Agent Nodes (as large as we need, at least one machine per AZ)
Each client agent node runs
  • consul client agent
  • nomad server agent
  • Docker daemon
These connect to home server triads (consul and nomad).

Implementation details

You have four roles
  • server1
  • server2
  • server3
  • worker-node
All worker-nodes are ephemeral. They can get blown away.
The servers: server1server2server3 form a triad cluster. Any triad member can die and be replaced. They should be started back up with the same basic ip address info.
Since we are running consulnomad, and zookeeper on the triad, there is no advantage to using consul for nomad server node discovery because consul is installed on the same triad of servers.
The server server1 is a bit special because it has the script to connect the servers into a cluster. For the most part, server1 is identical to the others.

Server 1

Server1 Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

$script = <<SCRIPT
# Update apt and get dependencies
sudo apt-get update
sudo apt-get install -y unzip curl wget vim

# Download Nomad
echo Fetching Nomad...
cd /tmp/
curl -sSL https://releases.hashicorp.com/nomad/0.2.3/nomad_0.2.3_linux_amd64.zip -o nomad.zip

echo Installing Nomad...
unzip nomad.zip
sudo chmod +x nomad
sudo mv nomad /usr/bin/nomad
sudo mkdir -p /etc/nomad.d
sudo chmod a+w /etc/nomad.d
sudo mkdir -p /opt/nomad/data
sudo mkdir -p /var/log/nomad
sudo chmod a+w /var/log/nomad
sudo cp /vagrant/server.hcl /etc/nomad.d/

echo Fetching Consul...
curl -sSL https://releases.hashicorp.com/consul/0.6.3/consul_0.6.3_linux_amd64.zip -o consul.zip
echo Installing Consul...
unzip consul.zip
sudo chmod +x consul
sudo mv consul /usr/bin/consul
sudo mkdir -p /etc/consul.d
sudo chmod a+w /etc/consul.d
sudo mkdir -p /opt/consul/data
sudo mkdir -p /var/log/consul
sudo chmod a+w /var/log/consul
sudo cp /vagrant/consul.json /etc/consul.d/


echo Starting nomad
cd ~
sudo nohup nomad agent -config /etc/nomad.d/server.hcl &>nomad.log &


echo Starting Consul
sudo nohup consul agent -config-file /etc/consul.d/consul.json &>consul.log &

SCRIPT

Vagrant.configure(2) do |config|
config.vm.box = "base-box"
config.vm.hostname = "nomad"
config.vm.provision "shell", inline: $script, privileged: false
config.vm.network "private_network", ip: "10.21.0.10"



# Increase memory for Parallels Desktop
config.vm.provider "parallels" do |p, o|
p.memory = "1024"
end

# Increase memory for Virtualbox
config.vm.provider "virtualbox" do |vb|
vb.memory = "1024"
end

# Increase memory for VMware
["vmware_fusion", "vmware_workstation"].each do |p|
config.vm.provider p do |v|
v.vmx["memsize"] = "1024"
end
end

config.vm.provider :aws do |aws, override|
aws.keypair_name = "my-app-key"
aws.region = "us-west-2"
# Ubuntu public Amazon EC2 image for Ubuntu 64 bit
aws.ami = "ami-9abea4fb"
override.ssh.username = "ubuntu"
override.ssh.private_key_path = "/opt/aws/my-app-key.pem"

aws.tags = {
Name => my-app-cluster-server-1
}


# vpc-d14dacb5
aws.subnet_id = "subnet-abc123ab"
aws.security_groups = "sg-abc123ab"
aws.private_ip_address="10.21.0.10"
override.vm.hostname = "ip-21-10-0-10"
# override.ssh.host = "10.20.0.10" //NOT EXPOSED TO VPN traffic yet
# We have to use public IP address because we dont have the VPC tied to vpn traffic
aws.associate_public_ip = true

end
end

server.hcl


bind_addr = "10.21.0.10"

advertise {
# We need to specify our hosts IP because we cant
# advertise 0.0.0.0 to other nodes in our cluster.
rpc = "10.21.0.10:4647"
}

# Increase log verbosity
log_level = "DEBUG"

# Setup data dir
data_dir = "/opt/nomad/data"

# Enable the server
server {
enabled = true
start_join = ["10.21.0.11", "10.21.0.10", "10.21.0.12"]
retry_join = ["10.21.0.11", "10.21.0.10", "10.21.0.12"]
retry_interval = "15s"
}

server-bootstrap.hcl

bind_addr = "10.21.0.10"

advertise {
# We need to specify our hosts IP because we cant
# advertise 0.0.0.0 to other nodes in our cluster.
rpc = "10.21.0.10:4647"
}


# Increase log verbosity
log_level = "DEBUG"

# Setup data dir
data_dir = "/opt/nomad/data"

# Enable the server
server {
enabled = true

# Self-elect, should be 3 or 5 for production
bootstrap_expect = 3
}

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.