05/31/23 | EverOps
As we tread further into 2023 large learning models (LLMs) also known as generative Artificial Intelligence (AI) have become increasingly accessible to the general public. Chatbots and other tools created from these models are able to assist us with a broad range of subject matters, augmenting our ability to quickly access the knowledge of the public internet. The process of effectively interacting with these models is commonly referred to as prompt engineering, and as generative AI technology weaves itself into the fabric of our society it will become a desired skill set for all kinds of workers. As DevOps engineers, we can utilize prompt engineering to query these AI models to consume information more efficiently and increase our productivity. In this blog article, we will jump into the basics of AI prompt writing, learn how you can utilize generative AI to speed up your DevOps workflows, and how to ensure that AI responses provide value for you.
In the world of DevOps, technical documentation and troubleshooting answers are widely available on the internet but are widely spread out and don’t always provide a full answer to help you solve a problem. Generative AI can provide faster and personalized answers to these problems in most cases but it requires creating a descriptive prompt for the model to consume in order to receive a useful response.
Learning how to prompt an AI model to provide answers that benefit you can help you make the most of this new technology. Providing the AI the right context when asking a question can make a major difference in the response you get from the AI. While simple tasks can be answered without much context, more complicated tasks require giving the AI adequate information to assist you with providing a correct response. Good Prompt engineering can help you get the most out of a Large Learning Model like ChatGPT or Google Bard.
Understanding how to design prompts is an essential skill because the quality of response you will get from generative AI is heavily influenced by your input. You should design prompts that are clear and concise while also providing enough context to guide the AI towards a solution that is helpful.
First, you should make sure your prompt is clearly defined for the problem you are trying to solve. A loosely defined prompt can lead to a wide variety of answers that may not be relevant to your problem. For example, If you are trying to figure out an issue with a programming error, you should tell the model what programming language and packages you are using, and what the error message is and when it occurs.
It’s also important to be concise with your prompts, providing unnecessary details can lead the model to focus on irrelevant information in your query. When asking an AI to optimize code, you should just provide the snippet of code that needs to be optimized and specify what it needs to be optimized for as opposed to providing the entire codebase or providing unnecessary details about the project.
Finally, providing context for complex queries is crucial. The more context you can provide when troubleshooting a complex problem can ensure that the AI model can understand the intent of your request. For example, when troubleshooting a software issue you should detail the expected functionality, explain what’s currently going wrong, and specify any constraints or requirements. When I notice the AI is taking the response in a different direction than I intended, I refine my question and provide additional context. Doing this will help the AI in delivering a practical solution every time. Let’s look at an example where we start with a request for a basic python script that utilizes the AWS boto3 package to get information about cloud instances and help guide the chatbot to a more useful answer.
ChatGPT-4 Prompt: Write a python script that uses boto3 to gather a list of EC2 instances and their details including tags and exports them to a CSV file.
ChatGPT-4 Response:
Sure, below is a simple Python script that uses Boto3 to get details of EC2 instances and their tags and then writes the data to a CSV file.
import boto3
import csv
session = boto3.Session(
aws_access_key_id='YOUR_ACCESS_KEY',
aws_secret_access_key='YOUR_SECRET_KEY',
region_name='YOUR_REGION_NAME'
)
ec2_resource = session.resource('ec2')with open('ec2_instances.csv', 'w', newline='') as csvfile:
fieldnames = ['instance_id', 'instance_type', 'availability_zone', 'state', 'tags']
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader()
# Iterate through all instances
for instance in ec2_resource.instances.all():
tags = {}
if instance.tags is not None:
# Extract the tag key-value pairs
for tag in instance.tags:
tags[tag['Key']] = tag['Value']
writer.writerow({
'instance_id': instance.id,
'instance_type': instance.instance_type,
'availability_zone': instance.placement['AvailabilityZone'],
'state': instance.state['Name'],
'tags': tags
})In this script, please replace ‘YOUR_ACCESS_KEY’, ‘YOUR_SECRET_KEY’, and ‘YOUR_REGION_NAME’ with your AWS credentials.
This script gathers details about every instance in the specified region, including the instance ID, instance type, availability zone, state, and tags, and writes these details to a CSV file named ‘ec2_instances.csv’. The tags are written as a dictionary of key-value pairs.
Note: Running this script requires the Boto3 library and the appropriate AWS credentials. Make sure to install Boto3 with pip install boto3 if it is not installed, and ensure that your AWS credentials have the necessary permissions to list EC2 instances. Also, AWS access and secret keys should not be hard-coded in scripts for security reasons. Instead, use environment variables or AWS CLI to manage them securely.
To make this script more useful let’s ask the AI to make some changes and make additional considerations when generating the code.
Additional ChatGPT-4 Prompt: Let’s update the script so it iterates over a list of regions and includes the region of each instance in the export. Let’s also make sure any API calls are using exponential backoff for failed requests and have exception checks with logging where appropriate.
ChatGPT-4 Response:
Sure, here’s how you can modify the script to iterate over a list of regions, include the region for each instance, implement exponential backoff for failed requests using the botocore library’s built-in mechanism, and add exception handling with logging:
import boto3
import botocore
import csv
import time
import logging
# Setup logging
logging.basicConfig(filename='ec2_instances.log', level=logging.ERROR)
# Create a session using your AWS credentials
session = boto3.Session(
aws_access_key_id='YOUR_ACCESS_KEY',
aws_secret_access_key='YOUR_SECRET_KEY'
)
# List of regions to check
regions = ['us-east-1', 'us-west-2', 'eu-west-1']
# Open the CSV file for writing
with open('ec2_instances.csv', 'w', newline='') as csvfile:
fieldnames = ['region', 'instance_id', 'instance_type', 'availability_zone', 'state', 'tags']
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader()
# Iterate through regions
for region in regions:
ec2_resource = session.resource('ec2', region_name=region)
retry_count = 0
while retry_count <= 5:
try:
# Iterate through all instances
for instance in ec2_resource.instances.all():
tags = {}
if instance.tags is not None:
# Extract the tag key-value pairs
for tag in instance.tags:
tags[tag['Key']] = tag['Value']
writer.writerow({
'region': region,
'instance_id': instance.id,
'instance_type': instance.instance_type,
'availability_zone': instance.placement['AvailabilityZone'],
'state': instance.state['Name'],
'tags': tags
})
# If successful, break the retry loop
break
except botocore.exceptions.EndpointConnectionError as e:
# If failed due to connection error, log the error and apply exponential backoff
logging.error(f'EndpointConnectionError in region {region}: {str(e)}')
retry_count += 1
time.sleep(2 ** retry_count)
except Exception as e:
# For any other exceptions, log the error and break the loop
logging.error(f'Unexpected error in region {region}: {str(e)}')
breakThis script now iterates through the regions specified in the regions list. For each region, it uses Boto3 to gather instance details and write them to the CSV file.
It also implements exponential backoff for handling EndpointConnectionError exceptions, which occur when there are connectivity issues. If such an error is encountered, the script waits for a period of time (determined by 2 ** retry_count, where retry_count is the number of retry attempts so far) before attempting to connect again. It will retry up to five times before moving on to the next region.
All errors are logged to a file named ec2_instances.log. You’ll need to make sure that your program has the necessary permissions to create and write to this file.
By providing additional context to the model for our original question we can transform a basic script to something that provides more utility, fault tolerance, and useful logging for debugging. We can see how prompt design can influence whether you will get a basic response or a detailed solution that will meet your needs.
Writing code and configuration files by hand can become tedious and time consuming.
Generative AI can help you write boilerplate for Infrastructure as Code using its vast knowledge of API documentations and configuration examples. In my experience I’ve found it helpful at quickly creating terraform blueprints for AWS resources that I can then fine tune for my needs. This cuts down a lot of unnecessary time writing terraform boilerplate and searching through documentation to find examples for the resources you need to create. For example, let’s say I’m trying to set up a service like Hashicorp Vault on an EC2 instance and want to utilize the KMS auto-unseal function that Vault offers and provide a secure key policy for the KMS key it creates.
ChatGPT-4 Prompt: Create the terraform for the AWS IAM resources and KMS resources to utilize the KMS auto-unseal process in Hashicorp Vault, ensure the KMS key policy only allows access to the admin role for administration and the vault role.
ChatGPT-4 Response:
Here’s an example of the Terraform configuration you asked for:
provider "aws" {
region = "us-west-2"
}
resource "aws_iam_role" "vault" {
name = "vault"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
resource "aws_iam_role" "admin" {
name = "admin"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"AWS": "arn:aws:iam::account-id:root"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
resource "aws_kms_key" "vault" {
description = "KMS key for Vault auto-unseal"
deletion_window_in_days = 10
key_usage = "ENCRYPT_DECRYPT"
is_enabled = true
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Allow use of the key",
"Effect": "Allow",
"Principal": {
"AWS": "${aws_iam_role.vault.arn}"
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "*"
},
{
"Sid": "Allow attachment of persistent resources",
"Effect": "Allow",
"Principal": {
"AWS": "${aws_iam_role.admin.arn}"
},
"Action": [
"kms:CreateGrant",
"kms:ListGrants",
"kms:RevokeGrant"
],
"Resource": "*",
"Condition": {
"Bool": {
"kms:GrantIsForAWSResource": "true"
}
}
}
]
}
EOF
}
output "aws_kms_key_id" {
description = "The globally unique identifier for the KMS key"
value = aws_kms_key.vault.key_id
}
output "aws_iam_role_arn" {
description = "The ARN assigned by AWS for this role"
value = aws_iam_role.vault.arn
}While I may need to make some adjustments to this terraform code for my unique environment, this example generated by ChatGPT-4 would be a great start to creating the terraform resources to utilize the AWS integrated functionality of Vault.
I’ve also found it very useful when dealing with all kinds of Kubernetes configuration and troubleshooting. Generative AI can help design kubernetes manifests and helm charts, as well as convert examples that use one deployment method to another. I find this very helpful when there isn’t a specific example for the deployment tool I’m using.
While generative AI can be a helpful tool to support your DevOps efforts, it’s important to ensure you are following your companies guidelines on AI use and not putting proprietary information into a model that is using your conversations to train the model for other interactions. We touch more on this subject in our State of AI blog article (link here). Companies have also started to build their own private LLMs for employees to use to avoid the potential data exposure that public generative AIs pose.
While AI can provide excellent solutions to problems, it can also have erratic responses to queries and sometimes provide answers that might not be the best solution. It’s always important to double-check what an AI has written and make sure it’s useful to solving your problem. While AI is great for rapidly learning new subjects and techniques, it’s important to ensure you develop a solid understanding of the subjects you are using AI for assistance with so you are confident in delivering a solution that follows the best practices. I’ve found that I’ve always needed to research a code package that ChatGPT recommends for a coding solution to make sure it’s a well-known and supported package, and to make sure that I’m using the latest version in my code.
It’s important to stay aware of new features and offerings as AI services advance in capability to ensure you continue to get the most value from this technology. For example, many of the generative chat solutions have started to add features to browse the web for data past its current knowledge base cutoff. This means you can now ask the chatbot to search the web and make sure it’s using the latest knowledge available in its responses when generating code or configurations.
For example, we can compare ChatGPT-4 versus ChatGPT-4 with Web Browsing enabled when creating a go programming example for creating a JSON Web Token using a well-known package.
Prompt: Show me an example of creating a JWT in Go using the latest version of an available package
ChatGPT-4 Response: You can use the jwt-go package to create JWT (JSON Web Tokens) in Go. This package allows you to create, sign, and verify tokens. Here’s a basic example of how to create a JWT.
package main
import (
"fmt"
"time"
"github.com/dgrijalva/jwt-go"
)
var jwtKey = []byte("my_secret_key")
type Claims struct {
Username string `json:"username"`
jwt.StandardClaims
}
func main() {
expirationTime := time.Now().Add(5 * time.Minute)
claims := &Claims{
Username: "testUser",
StandardClaims: jwt.StandardClaims{
ExpiresAt: expirationTime.Unix(),
},
}
token := jwt.NewWithClaims(jwt.SigningMethodHS256, claims)
tokenString, err := token.SignedString(jwtKey)
if err != nil {
fmt.Println("Error: ", err)
return
}
fmt.Println("Generated token: ", tokenString)
}While this answer may work, if you research this package on Github you can see that the maintainer has ceased development on the package and it has moved to a new repository where it is being maintained by a new team. This shows the importance of double checking a programming example when using a model that is limited by a knowledge cutoff date. Now let’s see the answer we get when using the web search.
ChatGPT-4 with Web Browsing Response: Here is a simple example of creating, signing, and encoding a JWT (JSON Web Token) using the HMAC signing method with the github.com/golang-jwt/jwt package:
package main
import (
"fmt"
"time"
"github.com/golang-jwt/jwt"
)
// For HMAC signing method, the key can be any []byte. It is recommended to generate
// a key using crypto/rand or something equivalent. You need the same key for signing
// and validating.
var hmacSampleSecret []byte
func main() {
// Create a new token object, specifying signing method and the claims
// you would like it to contain.
token := jwt.NewWithClaims(jwt.SigningMethodHS256, jwt.MapClaims{
"foo": "bar",
"nbf": time.Date(2015, 10, 10, 12, 0, 0, 0, time.UTC).Unix(),
})
// Sign and get the complete encoded token as a string using the secret
tokenString, err := token.SignedString(hmacSampleSecret)
fmt.Println(tokenString, err)
}By utilizing the web browsing plugin for ChatGPT-4 we are now able to see an example of creating a JSON web token using a well-supported package and the latest techniques mentioned in the documentation.
As more companies create plugins for these services there will be more interoperability between these models and the tools and services we use. Staying informed of these new plugins and knowing when to utilize them can help you get the most value from your prompts.
Leveraging generative AI in your DevOps workflows can improve your efficiency by enhancing your workflows and enabling faster problem solving. Using the process of prompt engineering, we can help ensure we get the best possible answers from our questions to the model.
At the end of the day, as devops engineers we still need to have a deep understanding of the infrastructure, services, and coding languages we use to be able to develop and maintain complex systems. This standard of knowledge is necessary when dealing with AI to ensure you are providing the right context for your prompts and also to ensure the information you are receiving is valid. It’s important to remember that you are an engineer and that generative AI is a tool. Solutions provided by AI should always be tested and cross-checked to ensure they follow best practices for reliability, performance, security and use the latest technologies available.