This lab shows how to create monitoring with Prometheus+Grafana for better monitoring for webapp server on EC2 using docker.
This is the diagram for this lab
1. Create web-app EC2 Server
Create EC2 for web-app server on EC2 using new or created KeyPair. I used ubuntu image with t2.micro in this lab for minimum cost.
Create new security group allowing SSH, HTTP and Port 5000
In here for port 5000 if you use Elastic IP for Webapp server you can use Source type as that Prometheus Server EIP. If you don’t use Elastic IP for webapp server the public IP will change every time the node restarts.
Under Advanced Details, use the user data, so then required packages are installed when server is up. Then Launch Instance
#!/bin/bash
apt update -y
apt install -y docker.io docker-compose
usermod -aG docker ubuntu
systemctl enable docker
systemctl start docker
After logging in to the web-app server using the public IP and keypair. Create webapp directory and go inside that directory
mkdir webapp && cd webapp
Create app.py file for webapp test using below code.
from flask import Flask, Response
app = Flask(__name__)
@app.route("/")
def index():
return "Web server is running"
@app.route("/metrics")
def metrics():
return Response('webapp_up 1\n', mimetype='text/plain')
if __name__ == "__main__":
app.run(host="0.0.0.0", port=5000)
Create Dockerfile from below. You can take reference from https://docs.docker.com/reference/dockerfile/ for Dockerfile.
FROM python:3.9-slim
WORKDIR /app
COPY app.py .
RUN pip install flask
CMD ["python", "app.py"]
Create docker-compose.yml. You can take reference from https://docs.docker.com/reference/compose-file/ for docker-compose.yml
version: '3'
services:
web:
build: .
ports:
- "5000:5000"
Run
docker-compose up -d --build
-d means detach mode and --build means build the image new instead of reusing the old image. After that use docker ps to check the running docker.
2. Create Prometheus server On EC2 with docker
Create another EC2 for Prometheus server using the above steps. But this time instead of opening port 5000 open port 9090 for Prometheus.
After connecting to Prometheus server,
mkdir prometheus && cd prometheus
Create prometheus.yml file for config of target server. You can take reference from here https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'web-app'
static_configs:
- targets: ['<WEB_EC2_PUBLIC_IP>:5000']
Create docker-compose.yml
version: '3'
services:
prometheus:
image: prom/prometheus
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
After that check http://<PROMETHEUS_PUBLIC_IP>:9090/targets
you should see the webserver as targe
3. Create Grafana server On EC2 with docker
Create EC2 as above steps for Grafana server. But this time open port 3000 for Grafana Serve
Login to Grafana Server and create grafana directory
mkdir grafana && cd grafana
Create docker-compose.yml . I want to avoid hardcoding the grafana password in docker-compose file so I will use another method by creating .env and add the line GRAFANA_ADMIN_PASSWORD=admin123 use your desired password. Here I used admin123 for grafana admin password. Use .gitignore to ignore uploading to git for .env file so then your password is not uploaded to git.
version: '3'
services:
grafana:
image: grafana/grafana
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}
Then run the following
docker-compose up -d
You can check http://<GRAFANA_PUBLIC_IP>:3000
if Grafana is up, use
username: admin
password: <your input password>
Go to Connections>Data Sources>Prometheus to show data send from Prometheus in Grafana
Go to Dashboard>New Dashboard>Add Visualization>(Select Data Source as Prometheus). You can use metrics for example up for server up in visualization. Then save dashboard
You can see as below as server uptime.
You can also integrate alerting with PagerDuty, Slack etc. I use PagerDuty to test alert
Go to Alert>New Alert Rule. Use metrics as up or webapp_up and when query is Equal to 0 the alert will trigger. Create the required Alert Folder, Evaluation Grp and most importantly create contact points.
Go to Create contact points and choose integration as PagerDuty. And you have to copy the Integration key from PagerDuty and put it here.
In Pager Duty Go to Services>New Service and create a service with desired name.
Choose the Service created and go to Integrations and Add integration. You can choose Events API V2 or Prometheus. Any of them working. Copy the Integration key from Pager Duty and paste in Grafana Contact Point.
Try stopping the webapp container using docker stop <container ID>. After some time you will receive an email from Pager Duty as below.
After all the testing has been finished don’t forget to delete all three EC2 that you used for this lab to avoid additional cost.
