Prometheus and Grafana Interview Questions and Answers

Introduction

Prometheus and Grafana are the most popular open-source monitoring tools used in production today.
Prometheus is used for collecting and storing metrics, while Grafana is used for visualization and dashboards.
Here are some important interview questions and answers that a Sys Engineer should know.
The answers are kept simple and clear for easy understanding.

Q1: What is Prometheus and why is it used?

Answer: Prometheus is a monitoring system that collects time-series metrics (CPU, memory, disk, etc.).
It is used because it is simple, powerful, and works well with cloud and container environments.
It can also trigger alerts when metrics cross thresholds.

Q2: What is Grafana and how does it work with Prometheus?

Answer: Grafana is a visualization tool that creates dashboards and graphs.
Prometheus stores the data, and Grafana reads this data using queries (PromQL).
Together, they provide complete monitoring with data + visualization.

Q3: What is PromQL?

Answer: PromQL is the query language used by Prometheus.
It allows you to filter, aggregate, and analyze metrics.
Example: rate(node_cpu_seconds_total[5m]) shows CPU usage over the last 5 minutes.

Q4: How does Prometheus collect metrics?

Answer: Prometheus pulls metrics from exporters.
An exporter is a small program that exposes metrics in Prometheus format (HTTP endpoint).
Example: Node Exporter for server metrics, Blackbox Exporter for network checks.

Q5: What is the difference between Node Exporter and Pushgateway?

Answer:
– Node Exporter → Exposes server metrics like CPU, memory, disk.
– Pushgateway → Used when jobs cannot be scraped directly (batch jobs), they push metrics to Prometheus.

Q6: What is the retention policy in Prometheus?

Answer: Retention policy defines how long metrics are stored.
By default, Prometheus keeps data for 15 days.
You can change it with: --storage.tsdb.retention.time=30d.

Q7: What is an Alertmanager in Prometheus?

Answer: Alertmanager is used to send notifications when an alert is triggered.
Prometheus generates alerts based on rules, and Alertmanager sends them via email, Slack, PagerDuty, etc.

Q8: What is the architecture of Prometheus?

Answer:
– Prometheus Server → Collects and stores metrics.
– Exporters → Provide metrics.
– Alertmanager → Sends alerts.
– Grafana → Visualizes metrics.
This simple architecture makes it reliable and scalable.

Q9: How do you scale Prometheus?

Answer: Prometheus is single-node, so for large scale:
– Use federation (multiple Prometheus servers pulling data from each other).
– Use remote storage integrations (Thanos, Cortex, VictoriaMetrics) for long-term and scalable storage.

Q10: What are some common Grafana panels you use?

Answer: Common panels are Graph, Gauge, Table, Heatmap.
Grafana also supports custom plugins and JSON queries for more advanced dashboards.

Q11: How do you secure Grafana?

Answer:
– Enable authentication (LDAP, SSO, OAuth).
– Use HTTPS.
– Limit admin access.
– Set up role-based access for dashboards.

Q12: What is the difference between Prometheus and ELK?

Answer: Prometheus is for metrics monitoring (CPU, memory, performance),
while ELK (Elasticsearch, Logstash, Kibana) is for log monitoring.
Both are used together in many companies.

Q13: What are best practices for Prometheus setup?

Answer:

Use Node Exporter on every server.
Set proper retention policies.
Use Alertmanager with grouping and silencing to avoid alert spam.
Use service discovery in Kubernetes/Cloud instead of static configs.
Integrate with Grafana for visualization and reporting.

Q14: Scenario – If Prometheus server is slow, what do you check?