Orchestration Using Terraform#
HashiCorp Terraform https://www.terraform.io/ is an infrastructure as code tool (IaC). Users can deploy a data center infrastructure using a declarative configuration language known as HashiCorp Configuration Language (HCL).. Terraform is currently one of the most popular infrastructure automation tools available. VSC Cloud also provides some template examples that could be used to deploy virtual infrastructures within VSC Tier-1 Cloud in an automated way (hpcugent/openstack-templates).
Terraform The client is available for different Operating Systems like Windows, Linux or macOS (https://www.terraform.io/downloads) but it is also available from UGent login node login.hpc.ugent.be.
Create application credentials for Terraform#
Terraform uses OpenStack application credentials to authenticate to VSC Cloud Tier-1 public API. It is a good practice to generate a new application credential just to be used with Terraformframework. The process is the same described in section application credentials.
Important
Make sure you download the new application credential as yaml file instead of openRC.
At this point you should have a clouds.yaml text file with these lines:
# This is a clouds.yaml file, which can be used by OpenStack tools as a source
# of configuration on how to connect to a cloud. If this is your only cloud,
# just put this file in ~/.config/openstack/clouds.yaml and tools like
# python-openstackclient will just work with no further config. (You will need
# to add your password to the auth section)
# If you have more than one cloud account, add the cloud entry to the clouds
# section of your existing file and you can refer to them by name with
# OS_CLOUD=openstack or --os-cloud=openstack
clouds:
openstack:
auth:
auth_url: https://cloud.vscentrum.be:13000
application_credential_id: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
application_credential_secret: "xxxxxxxxxxxxxxxxxxxxxxxxxxx"
region_name: "regionOne"
interface: "public"
identity_api_version: 3
auth_type: "v3applicationcredential"
As the file comments state, you should copy the current clouds.yaml to your VSC login node $HOME login.hpc.ugent.be: ~/.config/openstack/clouds.yaml, or locally if you have installed Terraform in your own laptop or computer. Terraform will use this file to authenticate to OpenStack API automatically.
Getting Terraform examples#
You can connect to UGent login node login.hpc.ugent.be
to use
terraform. Login to the login node with your VSC account first:
ssh -A [email protected]
Important
It is important to forward your ssh agent with -A
when SSH-ing to the login node.
If this is the first time using the templates, download the VSC Terraform examples from github from hpcugent/openstack-templates:
git clone https://github.com/hpcugent/openstack-templates
Make sure you have ~/.config/openstack/clouds.yaml
available from
the login node (see previous section).
Danger
Do not share your application’s credential file clouds.yaml
or put this
file in a public place.
chmod 600 ~/.config/openstack/clouds.yaml
Basic VM configuration#
Tip
If you are not using the VSC login node, you need to make sure to:
Install openstack client:
Ubuntu:
sudo apt install python3-openstackclient
We’ve provided examples of how to use the terraform modules in the examples/
directory.
Navigate to the environment directory first:
cd ~/openstack-templates/terraform/environment
The single.tf.example
deploys one VM with a public IP address and can be configured with extra options
Copy the single.tf.example
code to the main.tf
file like so:
cat examples/single.tf.example >> main.tf
The file now contains the definition for one VM with several options you can customize:
variable |
explanation |
Possible values |
---|---|---|
vm_name |
Sets the name of the virtual machine. |
(string) |
image_name |
Sets the operating system image for the machine. |
See Image list |
flavor_name |
Sets the machine flavor. |
see Flavors list. |
nginx_enabled |
Installs nginx and exposes ports 80 and 443 (See Automated variables) |
true, false |
nfs_network |
Connects the vm to the NFS network (Does not create a share). (See NFS_Share)Only set true if you requested access |
true, false |
vsc_enabled |
Connects the vm to the VSC network. Only set true if you requested access. |
true, false |
is_windows |
Configures windows-specific behavior if |
true/false |
More advanced options are described further on.
Automated variables#
The terraform templates include the ability to automatically adapt your VM based on certain variables. This is accomplished by installing scripts on the VM when it is first created. It requires ssh access to the VM.
These features can be disabled by setting scripts_enabled=false
.
Nginx#
nginx_enabled
will automatically install nginx and start the service.
It will also expose ports 80
and 443
, or two random ports if alt_http = true
.
Volumes#
If you set automount = true
on a volume (see Volumes), volumes will automatically be mounted at /mnt/volname
and will have a filesystem created. Size changes will result in the filesystem being automatically resized.
Volumes#
if you need large amounts of storage and don’t want to use an NFS share, you can instead attach an additional block-storage volume with the volumes
variable (see examples/vm_with_volumes.tf.example
).
The size
represents the volume size in gigabytes.
Note
This variable will create and attach the volume as a regular disk, but it will not create a filesystem or mount it unless you set automount = true
.
volumes = {
vol1 = {
size = 100
}
}
Automount#
The module supports automatically creating a filesystem, mounting it and resizing it as necessary.
It adds these arguments to volumes
:
Variable |
Explanation |
Values |
---|---|---|
automount |
Automatically create a filesystem and mount it |
true/false |
filesystem |
Sets the filesystem to be created |
see here (default: “ext4”) |
Deploy Terraform templates#
If you have followed the previous steps now you can init and deploy your infrastucture to Tier-1 VSC cloud.
If you haven’t deployed any template yet, you need to initiate terraform:
Move to environment directory first:
cd ~/openstack-templates/terraform/environment
This command performs several different initialization steps in order to prepare the current working directory for use with Terraform:
terraform init
Now you can check and review your Terraform plan, from the same directory:
terraform plan
You will see a list of the resources required to deploy your infrastructure, Terraform also checks if there is any systax error in your templates. Your infrastructure is not deployed yet, review the plan and then just deploy it to VSC Tier-1 Cloud running:
terraform apply
Terraform will show your plan again and you will see this message:
..
..
Do you want to perform these actions?
Terraform will perform the actions described above.
Only ’yes’ will be accepted to approve.
Enter a value:
Type yes and press enter and wait a few minutes. If everything is correct and if you have enough quota Terraform will show you a message after creating all the required resources.
..
..
Apply complete! Resources: 6 added, 0 changed, 0 destroyed.
Outputs:
MyVM = "SSH: ssh -A -p 56315 [email protected]"
Tip
Usually the username (rocky/ubuntu/…) will be provided by terraform output
. If you forget or are not sure what username to use, just connect with root
and a message will appear telling you the correct user.
Depending on the options you’ve added/enabled, the output may contain additional info
..
..
Outputs:
MyVM = <<EOT
SSH: ssh -A -p 53594 rocky@193.190.80.3
HTTP: http://193.190.80.3:52159 (HTTPS :57773)
VSC: 172.24.49.6
EOT
Your cloud infrastructure is ready to be used.
Tip
If you forgot your VM’s details, just run terraform output
If you make any changes to the template, just run terraform apply
again.
Important
It is important to keep a backup of your terraform directory, specially
all the files within the environment directory:
~/openstack-templates/terraform/environment
Terraform generates several files in this directory to keep track of any change in your infrastructure. If for some reason you lost or remove these files you will not able to modify or change the current Terraform plan (only directly from OpenStack).
Warning
Do not remove or modify the port_something_ssh.json
files. These keep track of the external ports used. Deleting them will break Terraform.
Private VMs#
If you don’t need/want a VM to be exposed to the internet, you can set the public
variable to false
.
You can then reach those VMs through your public VM(s) with ssh-agent forwarding (ssh -A
). See examples/frontend_backend.tf.example
for a complete example.
Note
Private VMs do not support Automated variables
Advanced variables#
There’s some extra variables you can configure:
Variable |
Explanation |
Values |
---|---|---|
access_key |
the name of the ssh key you want to associate with the vm/cluster |
(string) |
userscript |
A shell script that is executed when the VM is first created. |
(string) |
project_name |
The name VSC of the project you want to create the resurce in |
VSC_XXXX |
alt_http |
Use randomly generated ports for http instead of port 80/443 |
true/false (default false) |
public |
Add a public IP if true |
true/false (default true) |
custom_secgroup_rules |
A list of security group rules |
map of objects (see Firewall ) |
volumes |
A list of extra volumes |
map of objects (see Volumes) |
cloud_init |
Cloud-init “part” to execute when the VM is first created |
|
nfs_size |
DEPRECATED, use nfs_share module |
(Gigabytes) |
scripts_enabled |
Enables/disables optional ansible scripts |
true/false (default true) See Automated variables |
vsc_ip |
Manually set a VSC floating ip |
ip address (default null) |
rootdisk_size |
Manually sets the size of the rootdisk, overriding the flavor settings |
(Gigabytes) |
Firewall#
On a public VM, these ports will be open and exposed to the internet by default:
A random ssh port
If
nginx_enabled=true
:Port 80/443
2 random ports if
http_enabled=true
You may need to open more ports, either within the vm network or exposed to the internet.
You can add extra port forwarding rules through the custom_secgroup_rules
variable.
See the /examples/custom_secgroup.tf.example
file for an example.
Opening ports
Note
Ports will not be exposed to the internet by default, but it will allow other VMs to connect to that port.
custom_secgroup_rules = {
node_exporter = {
port = 9100
protocol = "tcp"
remote_ip_prefix = "0.0.0.0/0"
}
}
Important
Be sure to add the subnet to the remote_ip_prefix
, and note that this rule does not support individual IPs.
Exposing ports to the internet
You can set expose = true
for a particular port and terraform will select a random external port to forward to your chosen local port.
custom_secgroup_rules = {
node_exporter = {
port = 9100
protocol = "tcp"
remote_ip_prefix = "0.0.0.0/0"
expose = true
}
}
Will output something like:
MyVM = <<EOT
SSH: ssh -A -p 51274 [email protected]
node_exporter 9100 -> 193.190.80.3:51273
EOT
Important
Be sure to add the subnet to the remote_ip_prefix
, and note that this rule does not support individual IPs.
Warning
Exposing ports to the internet is potentially dangerous. Make sure that the application you’re exporting is properly secured.
Further customization#
You can also modify and add more resources for the current templates. This task is out of the scope of this document, please refer to official Terraform documentation to add you own changes https://www.terraform.io/docs or ask to VSC Cloud admins via email at cloud@vscentrum.be.
Troubleshooting#
This section describes some common errors you may encounter. If this section does not help you, don’t hesitate to contact cloud@vscentrum.be for help.
A duplicate port forwarding entry with same attributes already exists
If you get the error:
│ Error: Error creating openstack_networking_portforwarding_v2: Bad request with: [POST https://cloud.vscentrum.be:13696/v2.0/floatingips/64f2705c-43ec-4bdf-864e-d18fee013e3f/port_forwardings], error message: {"NeutronError": {"type": "BadRequest", "message": "Bad port_forwarding request: A duplicate port forwarding entry with same attributes already exists, conflicting values are {'floatingip_id': '64f2705c-43ec-4bdf-864e-d18fee013e3f', 'external_port': 80, 'protocol': 'tcp'}.", "detail": ""}}
This means that a different VM in your project is already using port 80 and/or 443.
You can have multiple VMs with nginx_enabled=true
by setting alt_http=true
for that particular VM.
remote-exec provisioner error
If you get the error:
│ Error: remote-exec provisioner error
│
│ with module.MyVMExample.null_resource.testconnection[0],
│ on ../modules/single_instance/ansible.tf line 17, in resource "null_resource" "testconnection":
│ 17: provisioner "remote-exec" {
│
│ interrupted - last error: SSH authentication failed ([email protected]:51307): ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no
│ supported methods remain
Then terraform is unable to SSH to your instance. There are a couple of possible reasons:
You are on a login node but have not forwarded your ssh-agent with
ssh -A
If you have, check if your ssh-agent is configured correctly
You do not have ssh access to this VM/it was created with the wrong keypair
Ask whoever does have access to add your ssh public key to the VM
If this is a new VM and it is using the wrong keypair, use the Advanced Variable
access_key
to set the correct one.
Tip
You can also set scripts_enabled=false
if you do not want any convenience scripts or if there is no better solution
Feel free to contact us cloud@vscentrum.be for help.