Security for Developers #12: Security Logging and Monitoring

In the previous tutorial, you learned how to scan dependencies for vulnerabilities. But what happens when an attack is already in progress? Without proper logging and monitoring, you will not know until it is too late. In this article, you will learn what to log, what never to log, how to detect attacks, and how to set up meaningful alerts.

The OWASP Top 10 lists “Security Logging and Monitoring Failures” as A09 because most breaches go undetected for months.

Why Security Logging Matters

The average time to detect a data breach is 204 days (IBM Cost of a Data Breach Report 2024). That means attackers have over 6 months to steal data before anyone notices.

Proper logging helps you:

Detect attacks in progress — brute force, injection attempts, privilege escalation
Investigate incidents — understand what happened, when, and how
Meet compliance requirements — GDPR, SOC 2, PCI DSS all require audit logs
Improve security over time — learn from past incidents

What to Log (Security Events)

Authentication Events

{
  "timestamp": "2026-06-03T14:30:00Z",
  "event": "login_failed",
  "user_id": "user_123",
  "ip": "203.0.113.42",
  "user_agent": "Mozilla/5.0...",
  "reason": "invalid_password",
  "attempt_count": 3
}

Log these authentication events:

Event	Why
Successful login	Track normal access patterns
Failed login	Detect brute force attacks
Password change	Detect account takeover
MFA enrollment/removal	Detect MFA bypass attempts
Token refresh	Track active sessions
Logout	Complete session lifecycle
Account lockout	Detect brute force threshold

Authorization Events

{
  "timestamp": "2026-06-03T14:35:00Z",
  "event": "authorization_denied",
  "user_id": "user_456",
  "resource": "/api/admin/users",
  "method": "GET",
  "reason": "insufficient_role",
  "required_role": "admin",
  "user_role": "viewer"
}

Log these:

Access denied to protected resources
Role changes (promotion/demotion)
Admin actions (user creation, deletion, config changes)
API key creation and revocation

Suspicious Activity

{
  "timestamp": "2026-06-03T14:40:00Z",
  "event": "rate_limit_exceeded",
  "ip": "203.0.113.42",
  "endpoint": "/api/login",
  "requests_per_minute": 150,
  "threshold": 10
}

Log these:

Rate limit violations
Input validation failures (possible injection attempts)
Unusual request patterns (unusual hours, geographic anomalies)
File upload of suspicious types
Large data exports

What NEVER to Log

This is just as important as what you log. Logging sensitive data creates a new attack target.

Never log:

Passwords (plaintext or hashed)
Credit card numbers
Social Security numbers or national IDs
JWT tokens or session tokens
API keys or secrets
Personal health information
Full request bodies with sensitive data

Masking Sensitive Data

Go:

package main

import (
    "fmt"
    "regexp"
    "strings"
)

// Mask email: "user@example.com" -> "u***@example.com"
func maskEmail(email string) string {
    parts := strings.SplitN(email, "@", 2)
    if len(parts) != 2 || len(parts[0]) == 0 {
        return "***"
    }
    return string(parts[0][0]) + "***@" + parts[1]
}

// Mask credit card: "4111111111111111" -> "****1111"
func maskCreditCard(cc string) string {
    if len(cc) < 4 {
        return "****"
    }
    return "****" + cc[len(cc)-4:]
}

// Mask JWT: "eyJhbGciOiJI..." -> "eyJ***"
func maskToken(token string) string {
    if len(token) < 6 {
        return "***"
    }
    return token[:3] + "***"
}

Python:

import re

def mask_email(email):
    """Mask email: user@example.com -> u***@example.com"""
    parts = email.split("@")
    if len(parts) != 2 or not parts[0]:
        return "***"
    return parts[0][0] + "***@" + parts[1]

def mask_credit_card(cc):
    """Mask credit card: 4111111111111111 -> ****1111"""
    if len(cc) < 4:
        return "****"
    return "****" + cc[-4:]

def mask_token(token):
    """Mask token: eyJhbGciOiJI... -> eyJ***"""
    if len(token) < 6:
        return "***"
    return token[:3] + "***"

Structured Logging

Use structured logging (JSON) instead of plain text. Structured logs are easier to search, filter, and analyze.

Go (with slog)

package main

import (
    "log/slog"
    "os"
)

func main() {
    logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))

    // Security event log
    logger.Info("authentication_success",
        slog.String("event", "login"),
        slog.String("user_id", "user_123"),
        slog.String("ip", "203.0.113.42"),
        slog.String("method", "password"),
    )

    // Failed authentication
    logger.Warn("authentication_failed",
        slog.String("event", "login_failed"),
        slog.String("user_id", "user_123"),
        slog.String("ip", "203.0.113.42"),
        slog.String("reason", "invalid_password"),
        slog.Int("attempt", 3),
    )
}

Python (with structlog)

import structlog

logger = structlog.get_logger()

# Security event log
logger.info("authentication_success",
    event="login",
    user_id="user_123",
    ip="203.0.113.42",
    method="password",
)

# Authorization denied
logger.warning("authorization_denied",
    event="access_denied",
    user_id="user_456",
    resource="/api/admin/users",
    reason="insufficient_role",
)

JavaScript (with pino)

const pino = require("pino");
const logger = pino({ level: "info" });

// Security event
logger.info({
  event: "login_failed",
  userId: "user_123",
  ip: "203.0.113.42",
  reason: "invalid_password",
  attemptCount: 3,
}, "Authentication failed");

Log Levels for Security Events

Level	When to Use	Examples
INFO	Normal security events	Successful login, password change
WARN	Suspicious but not confirmed	Failed login, rate limit hit, authorization denied
ERROR	Security violations	Invalid token, injection attempt detected
CRITICAL	Active attack or breach	Multiple accounts compromised, data exfiltration detected

Detecting Attacks with Log Analysis

Pattern: Brute Force Detection

Python:

import structlog
from collections import defaultdict
from datetime import datetime, timedelta

logger = structlog.get_logger()

# NOTE: In production, use a thread-safe store (e.g., Redis) instead of a
# plain dict. This in-memory dict is not safe for concurrent requests.
failed_logins = defaultdict(list)

def check_brute_force(user_id, ip, timestamp):
    """Detect brute force: 5+ failed logins in 15 minutes"""
    key = f"{user_id}:{ip}"
    cutoff = timestamp - timedelta(minutes=15)

    # Remove old attempts
    failed_logins[key] = [
        t for t in failed_logins[key] if t > cutoff
    ]

    failed_logins[key].append(timestamp)

    if len(failed_logins[key]) >= 5:
        logger.critical("brute_force_detected",
            user_id=user_id,
            ip=ip,
            attempts=len(failed_logins[key]),
            window_minutes=15,
        )
        return True
    return False

Pattern: Anomalous Access

def check_anomalous_access(user_id, ip, country, hour):
    """Flag access from unusual locations or times"""
    usual_countries = get_user_usual_countries(user_id)
    usual_hours = get_user_usual_hours(user_id)

    if country not in usual_countries:
        logger.warning("anomalous_access",
            user_id=user_id,
            ip=ip,
            country=country,
            reason="unusual_country",
        )

    if hour not in usual_hours:
        logger.warning("anomalous_access",
            user_id=user_id,
            hour=hour,
            reason="unusual_hour",
        )

Centralized Logging

Do not store logs on individual servers. Centralize them for analysis.

Architecture

Application -> Log Shipper -> Central Log Store -> Dashboard/Alerts
(stdout)      (Fluent Bit)    (Elasticsearch/     (Grafana/
                               Loki/CloudWatch)    Kibana)

Popular Stacks

Stack	Components	Cost
ELK	Elasticsearch + Logstash + Kibana	Self-hosted (free) or Elastic Cloud
PLG	Promtail + Loki + Grafana	Self-hosted (free) or Grafana Cloud
Cloud-native	CloudWatch (AWS) / Cloud Logging (GCP)	Pay per use
Datadog	Agent + Datadog platform	~$0.10/GB ingested

Docker Compose with Loki + Grafana

# docker-compose.yml
services:
  app:
    image: myapp:latest
    logging:
      driver: "json-file"
      options:
        max-size: "10m"

  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log
      - ./promtail-config.yml:/etc/promtail/config.yml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"

Alerting

Logs are useless if nobody reads them. Set up alerts for critical security events.

Alert Rules

Alert	Condition	Severity
Brute force attempt	10+ failed logins from same IP in 5 min	High
Account takeover	Login from new country after password change	Critical
Privilege escalation	Non-admin accessing admin endpoints	High
Rate limit abuse	100+ rate limit hits from same IP in 1 min	Medium
Injection attempt	SQL/XSS patterns in input validation logs	High
Mass data export	Single user downloading 10x normal data volume	Critical

Avoid Alert Fatigue

Too many alerts means people ignore them. Keep these rules:

Only alert on actionable events. If nobody needs to do anything, it is a log, not an alert.
Tune thresholds. Start high, lower gradually.
Group related alerts. 50 failed logins from one IP should be one alert, not 50.
Use severity levels. Critical: wake someone up. High: respond today. Medium: respond this week.

Audit Trails

An audit trail is a chronological record of all actions in a system. It answers: who did what, when, and from where.

Requirements for a good audit trail:

Immutable — logs cannot be modified or deleted
Timestamped — precise timestamps in UTC
Complete — every state change is recorded
Searchable — easy to find events by user, time, or action

Go example: Audit log middleware

func auditMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        userID := getUserID(r)
        logger.Info("api_access",
            slog.String("user_id", userID),
            slog.String("method", r.Method),
            slog.String("path", r.URL.Path),
            slog.String("ip", r.RemoteAddr),
            slog.String("user_agent", r.UserAgent()),
        )
        next.ServeHTTP(w, r)
    })
}

Log Retention

How long to keep logs depends on regulations and business needs:

Regulation	Minimum Retention
GDPR	As long as necessary (minimize)
SOC 2	1 year
PCI DSS	1 year (3 months immediately available)
HIPAA	6 years
General best practice	90 days hot, 1 year cold storage

Prevention Checklist

Defense	Priority	Notes
Log all authentication events	High	Successes and failures
Log authorization denials	High	Detect privilege escalation
Never log passwords, tokens, or PII	High	Mask sensitive data
Use structured logging (JSON)	High	Easier to search and analyze
Centralize logs	High	Do not keep logs on app servers only
Set up alerts for brute force	High	10+ failed logins in 5 min
Set up alerts for anomalous access	Medium	Unusual country or time
Maintain audit trails for admin actions	Medium	Required for compliance
Define log retention policy	Medium	Based on regulation requirements
Test alerting regularly	Low	Verify alerts actually fire

What is Next?

In the next tutorial, you will learn about Container and Docker Security — running containers as non-root, scanning images, managing secrets in Docker, and hardening your deployment.

Why Security Logging Matters#

What to Log (Security Events)#

Authentication Events#

Authorization Events#

Suspicious Activity#

What NEVER to Log#

Masking Sensitive Data#

Structured Logging#

Go (with slog)#

Python (with structlog)#

JavaScript (with pino)#

Log Levels for Security Events#

Detecting Attacks with Log Analysis#

Pattern: Brute Force Detection#

Pattern: Anomalous Access#

Centralized Logging#

Architecture#

Popular Stacks#

Docker Compose with Loki + Grafana#

Alerting#

Alert Rules#

Avoid Alert Fatigue#

Audit Trails#

Log Retention#

Prevention Checklist#

What is Next?#

Related Articles#