Logging and Observability

Key features

• Level order, low to high: DEBUG (10) < INFO (20) < WARNING (30) < ERROR (40) < CRITICAL (50)
• A logger AND each of its handlers has its own level — a record must clear both to be emitted
• logging.getLogger(__name__) gives each module its own named logger in one shared hierarchy
• A formatter controls the TEXT shape of a record (timestamp, level, message) — it does not filter anything
• logging.config.dictConfig() configures loggers/handlers/formatters from one data structure, not scattered setup calls

Command

import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

handler = logging.StreamHandler()
handler.setLevel(logging.WARNING)          # handler filters independently
handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s: %(message)s'))
logger.addHandler(handler)

Key features

• A custom logging.Formatter subclass can return json.dumps(...) from format() instead of a text template
• The extra= keyword on any log call attaches custom fields to that one record, read back via record.<name>
• Structured logs make grep-free querying possible: "show every ERROR where service=payments" instead of text search
• python-json-logger and structlog are the common third-party choices; the stdlib supports the pattern with no extra install
• Log aggregators (Elasticsearch, Datadog, CloudWatch Logs Insights) index structured fields directly — free text they can only full-text search

Command

import logging, json

class JsonFormatter(logging.Formatter):
    def format(self, record):
        payload = {
            'level': record.levelname,
            'logger': record.name,
            'message': record.getMessage(),
            'request_id': getattr(record, 'request_id', None),
        }
        return json.dumps(payload)

Key features

• logging.LoggerAdapter wraps a logger and injects extra context (like a request ID) into every call automatically
• A request ID is usually generated once per incoming request (uuid4, or read from an incoming X-Request-ID header)
• Forwarding the SAME correlation ID to downstream services turns a single-service ID into a distributed trace anchor
• logger.exception(msg) — called only inside an except block — automatically attaches the full traceback to the log record
• logger.error(msg, exc_info=True) does the same as logger.exception() but can be called with any level, not just ERROR

Command

import logging, uuid

class RequestIdAdapter(logging.LoggerAdapter):
    def process(self, msg, kwargs):
        return f"[{self.extra['request_id']}] {msg}", kwargs

logger = logging.getLogger(__name__)
request_logger = RequestIdAdapter(logger, {'request_id': str(uuid.uuid4())[:8]})
request_logger.info('handling request')   # every call carries the same ID

Key features

• A Counter metric only goes up (requests served, errors); a Histogram/Gauge captures a distribution or a point-in-time value (latency, queue depth)
• A trace is one request's full path; a span is one unit of work within it — spans nest, forming a tree
• A child span's parent_id equals its parent span's span_id — that shared ID is what stitches spans from DIFFERENT services into one trace
• OpenTelemetry is the vendor-neutral standard API/SDK for both metrics and tracing — instrument once, export to any compatible backend
• A SpanProcessor + Exporter pair decides where spans go (console, Jaeger, an OTLP collector) — the instrumentation code itself never changes

Command

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span('handle-request') as span:
    span.set_attribute('http.method', 'GET')
    with tracer.start_as_current_span('query-database'):
        ...   # this span is automatically nested under handle-request

Key features

• Liveness check: is the process itself deadlocked/broken → orchestrator restarts the container on failure
• Readiness check: can this instance handle a request right now → orchestrator stops routing traffic, does NOT restart
• A readiness check should verify real dependencies (database, cache) — a liveness check should stay cheap and dependency-free
• A liveness check that pings the database can cause a restart LOOP if the database itself is briefly down — the process was fine
• Kubernetes calls these livenessProbe and readinessProbe; both usually map to a lightweight HTTP endpoint (GET /healthz, /readyz)

Command

def liveness() -> dict:
    return {'status': 'ok'}          # cheap, no external dependency checks

def readiness(db_ok: bool, cache_ok: bool) -> tuple[dict, int]:
    ready = db_ok and cache_ok
    body = {'status': 'ok' if ready else 'unavailable', 'checks': {'db': db_ok, 'cache': cache_ok}}
    return body, 200 if ready else 503