Hello

Don't Make Your Developers Think About Observability

Source of Inspiration

Don't Make Me Think

• Users shouldn't have to think, the interface should make it obvious.
• Satisficing — users will choose the first reasonable option.

The Design of Everyday Things

• Look at how ordinary objects communicates the intended use.
• Slips vs Mistakes — lower cognition errors vs higher cognition errors

Let's design our tooling around these ideas

Observability in a nutshell

• Ability to know how your system is performing at any given moment.
• Inputs
• Outputs

Why should you care?

• It's 3am and you get a PagerDuty alert, where do you go first?
• Marketing asks you if the new campaign has affected the funnel
• You're shipping a feature, how do you know it's working?

3 Pillars of Observability

• Metrics
• Logs
• Traces

Metrics

Input:

increment('purchase', by: 1)

Output:

| metric   | count |
|----------|-------|
| purchase |     1 |

Logs

Input:

Rails.logger.info("Purchase made.")

Output:

2026-10-03 02:34:56 INFO Purchase made.

Traces

Input:

tracer.span("purchase") do |span|
  tracer.span("charge_card")  { ... }
  tracer.span("send_receipt") { ... }
end

Output:

purchase                120ms  trace=7c3a-1f9d
├─ charge_card           95ms
└─ send_receipt          18ms

Wide Events

• Shed light on the "unknown unknowns"
• Structured logs that emphasize:
  • High cardinality fields
  • Multiple dimensions

def create_order(order_params)
  Rails.logger.info("Order Started")
  
  if (order = Order.create(order_params))
    Rails.logger.info("Order Created: #{order.id}")
  else
    Rails.logger.info("Order Not Created")
  end
end

def create_order(order_params)
  order = Order.create(order_params)
  Rails.logger.info("create_order", created: order.persisted?, order_id: order.id)
end

More!

def create_order(order_params)
  started_at = Time.current
  order = Order.create(order_params)

  Rails.logger.info("create_order",
    # what happened
    created: order.persisted?, errors: order.errors.full_messages,
    duration_ms: ((Time.current - started_at) * 1000).round,
    # who did it
    user_id: Current.user.id, account_id: Current.account.id,
    plan: Current.account.plan, signed_up_at: Current.user.created_at,
    # what they bought
    order_id: order.id, total_cents: order.total_cents, currency: order.currency,
    item_count: order.line_items.size, coupon: order.coupon&.code,
    payment_method: order.payment_method, shipping_country: order.ship_to.country,
    # where it ran
    request_id: Current.request_id, app_version: ENV["GIT_SHA"],
    region: ENV["REGION"], host: Socket.gethostname,
    feature_flags: FeatureFlag.for_user(Current.user)
  )
end

MOAR!

def create_order(order_params)
  started_at = Time.current
  order = Order.create(order_params)

  Rails.logger.info("create_order",
    # what happened
    created: order.persisted?, errors: order.errors.full_messages, error_count: order.errors.count,
    duration_ms: ((Time.current - started_at) * 1000).round, db_time_ms: Current.db_time_ms,
    sql_count: Current.sql_count, cache_hits: Current.cache_hits, cache_misses: Current.cache_misses,
    allocations: GC.stat(:total_allocated_objects), gc_count: GC.count, retries: Current.retries,
    # who did it
    user_id: Current.user.id, account_id: Current.account.id, plan: Current.account.plan,
    signed_up_at: Current.user.created_at, user_role: Current.user.role, locale: Current.user.locale,
    time_zone: Current.user.time_zone, lifetime_orders: Current.user.orders.count,
    lifetime_value_cents: Current.user.lifetime_value_cents, email_verified: Current.user.verified?,
    two_factor: Current.user.otp_enabled?, impersonated_by: Current.impersonator&.id,
    referral_source: Current.user.referral_source, cohort: Current.user.cohort, vip: Current.user.vip?,
    # what they bought
    order_id: order.id, total_cents: order.total_cents, subtotal_cents: order.subtotal_cents,
    tax_cents: order.tax_cents, shipping_cents: order.shipping_cents, discount_cents: order.discount_cents,
    currency: order.currency, item_count: order.line_items.size, sku_count: order.skus.uniq.size,
    skus: order.skus, categories: order.categories, coupon: order.coupon&.code,
    gift_card: order.gift_card?, gift_wrap: order.gift_wrap?, subscription: order.subscription?,
    payment_method: order.payment_method, card_brand: order.card&.brand, card_country: order.card&.country,
    installments: order.installments, shipping_method: order.shipping_method,
    shipping_country: order.ship_to.country, shipping_region: order.ship_to.region,
    shipping_postal: order.ship_to.postal_code, billing_matches_shipping: order.billing_matches_shipping?,
    fulfillment_center: order.fulfillment_center, estimated_delivery: order.estimated_delivery_on,
    backordered_items: order.backordered_items.size, fraud_score: order.fraud_score,
    # how they got here
    utm_source: Current.utm[:source], utm_medium: Current.utm[:medium], utm_campaign: Current.utm[:campaign],
    referrer: Current.referrer, landing_page: Current.landing_page, session_id: Current.session_id,
    session_page_views: Current.session_page_views, cart_age_minutes: Current.cart_age_minutes,
    abandoned_carts: Current.user.abandoned_carts.count, ab_tests: Current.ab_test_variants,
    # what they used
    user_agent: Current.user_agent, browser: Current.browser.name, browser_version: Current.browser.version,
    os: Current.browser.platform, device: Current.browser.device_type, mobile_app_version: Current.app_build,
    screen_size: Current.screen_size, ip_country: Current.geo.country, ip_city: Current.geo.city,
    # where it ran
    request_id: Current.request_id, trace_id: Current.trace_id, app_version: ENV["GIT_SHA"],
    deployed_at: ENV["DEPLOYED_AT"], region: ENV["REGION"], availability_zone: ENV["AZ"],
    host: Socket.gethostname, pid: Process.pid, thread_id: Thread.current.object_id,
    puma_workers: Puma.stats_hash[:workers], db_pool_busy: ActiveRecord::Base.connection_pool.stat[:busy],
    queue_depth: SolidQueue::Job.count, ruby_version: RUBY_VERSION, rails_version: Rails.version,
    feature_flags: FeatureFlag.for_user(Current.user), maintenance_mode: Maintenance.on?,
    moon_phase: Moon.phase(Date.current)
  )
end

That can be very overwhelming.

But we can make it easy.

Don't make me think.

• about inputs

def add_to_cart(product, cart)
  Rails.logger.debug('starting add to cart')

  Rails.logger.info('adding product to cart')
  cart.add_product(product)

  Rails.logger.debug('finished add to cart')
end

The ask

• "Check the inventory before adding to a cart."

def add_to_cart(product, cart)
  Rails.logger.debug('starting add to cart')

  Rails.logger.info('adding product to cart')
+ if Inventory.in_stock?(product)
+  # Rails.logger.info('product in stock')
+  # or?
+  # Rails.logger.debug('product in stock')
    cart.add_product(product)
+ else
+   Rails.logger.warn('product not in stock')
+ end

  Rails.logger.debug('finished add to cart')
end

The fix

• Encapsulate your observability

class Command::Base
  include ActiveModel::Model
  include ActiveModel::Attributes

  def run
    event("command", **context) { perform }
  end

  def event(name, **payload)
    started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
    yield
  ensure
    elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started
    Honeybadger.event(name, payload.merge(duration_ms: (elapsed * 1000).round))
  end

  def context
    attributes.to_h do |name, value|
      value.is_a?(ActiveRecord::Base) ? [:"#{name}_id", value.id] : [name.to_sym, value]
    end.merge(class: self.class.name, **extra_context)
  end
end

def add_to_cart(product, cart)
  Command::AddToCart.new(product, cart).run
end

class Command::AddToCart < Command::Base
  attribute :product
  attribute :cart

  def perform
    if Command::InventoryInStock.new(product).run
      cart.add_product(product)
    end
  end
end

class Command::InventoryInStock < Command::Base
  attribute :product

  def perform = in_stock?

  def in_stock?
    return @in_stock if defined? @in_stock
    @in_stock = Inventory.in_stock?(product)
  end

  def extra_context
    { in_stock: in_stock? }
  end
end

{ event: "command", class: "Command::InventoryInStock", product_id: 1, cart_id: 1, duration_ms: 12, in_stock: true }
{ event: "command", class: "Command::AddToCart", product_id: 1, cart_id: 1, duration_ms: 123 }

class Command::SendThankYouEmail < Command::Base
  attribute :user
  attribute :order

  def perform
    CustomerMailer.thank_you(user, order).deliver_later
  end
end

New feature

• "Override the recipient email and send a personal note."

class Command::SendThankYouEmail < Command::Base
  attribute :user
  attribute :order
+ attribute :override_email
+ attribute :note

  def perform
-   CustomerMailer.thank_you(user, order).deliver_later
+   CustomerMailer.thank_you(user, order, override_email, note).deliver_later
  end
end

{ event: "command", class: "Command::SendThankYouEmail", user_id: 1, order_id: 1,
  override_email: "jsmith@gmail.com", note: "Thanks Jane!", duration_ms: 123 }

class Command::Base
  def run
-   event("command", **context) { perform }
+   event("command", **filtered_context) { perform }
  end

+ def filtered_context
+   ActiveSupport::ParameterFilter
+     .new(Rails.application.config.filter_parameters)
+     .filter(context)
+ end
end

{ event: "command", class: "Command::SendThankYouEmail", user_id: 1, order_id: 1,
  override_email: "[FILTERED]", note: "[FILTERED]", duration_ms: 123 }

class Command::ChargeCard < Command::Base
  attribute :order

  def perform
    Stripe::Charge.create(amount: order.total_cents, customer: order.user.stripe_id)
  rescue Stripe::CardError => e
    Rails.logger.error("charge failed: #{e.message}")
    Honeybadger.notify(e)
    raise
  end
end

class Command::Base
  def event(name, **payload)
    started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-   yield
+   yield.tap { payload[:status] = "ok" }
+ rescue => e
+   payload.merge!(status: "error", error_class: e.class.name, error_message: e.message)
+   raise
  ensure
    elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started
    Honeybadger.event(name, payload.merge(duration_ms: (elapsed * 1000).round))
  end
end

class Command::ChargeCard < Command::Base
  attribute :order

  def perform
    Stripe::Charge.create(amount: order.total_cents, customer: order.user.stripe_id)
  end
end

class Command::Base
  include ActiveModel::Model
  include ActiveModel::Attributes

  def run
    event("command", **filtered_context) { perform }
  end

  def event(name, **payload)
    started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
    yield.tap { payload[:status] = "ok" }
  rescue => e
    payload.merge!(status: "error", error_class: e.class.name, error_message: e.message)
    raise
  ensure
    elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started
    Honeybadger.event(name, payload.merge(duration_ms: (elapsed * 1000).round))
  end

  def context
    attributes.to_h do |name, value|
      value.is_a?(ActiveRecord::Base) ? [:"#{name}_id", value.id] : [name.to_sym, value]
    end.merge(class: self.class.name, **extra_context)
  end

  def filtered_context
    ActiveSupport::ParameterFilter
      .new(Rails.application.config.filter_parameters)
      .filter(context)
  end

  def extra_context = {}
end

Let's talk about context

More context = More dimensionality

class EventContextMiddleware
  ECS_TASK = JSON.parse(Net::HTTP.get(URI("#{ENV["ECS_CONTAINER_METADATA_URI_V4"]}/task"))) rescue {}

  def call(env)
    request = Rack::Request.new(env)
    brand = detect_brand(request)
    namespace = detect_namespace(request)

    Honeybadger.event_context(
      namespace: namespace,
      brand: brand,
      mobile_app_platform: request.get_header("HTTP_X_MOBILE_APP_PLATFORM"),
      mobile_app_version: request.get_header("HTTP_X_MOBILE_APP_VERSION"),
      host: Socket.gethostname,
      app_version: ENV["GIT_SHA"],
      ecs_task: ECS_TASK["TaskARN"]&.split("/")&.last,
      ecs_family: ECS_TASK["Family"],
      availability_zone: ECS_TASK["AvailabilityZone"],
    )

    @app.call(env)
  end
end

class ApplicationJob < ActiveJob::Base
  ECS_TASK = JSON.parse(Net::HTTP.get(URI("#{ENV["ECS_CONTAINER_METADATA_URI_V4"]}/task"))) rescue {}

  around_perform do |job, block|
    context = {
      job_class: job.class.name,
      queue: job.queue_name,
      attempt: job.executions,
      queue_latency_ms: ((Time.current - job.enqueued_at) * 1000).round,
      scheduled: job.scheduled_at.present?,
      host: Socket.gethostname,
      app_version: ENV["GIT_SHA"],
      ecs_task: ECS_TASK["TaskARN"]&.split("/")&.last,
      ecs_family: ECS_TASK["Family"],
      availability_zone: ECS_TASK["AvailabilityZone"],
    }.compact

    Honeybadger.event_context(context) do
      block.call
    end
  end
end

Getting data in is only half the job

Let's make it easy

Don't make me think.

• about outputs

"It's 3am and you get a PagerDuty alert, where do you go first?"

Command::Base.descendants.each do |command|
  ErrorTracker.upsert(
    name: "#{command.name} error rate",
    query: { event_type: "command", class: command.name },
    alert_when: "error_rate > 5%",
    message: <<~MSG
      {{distinct_count user_id}} users affected
      {{distinct_count order_id}} orders affected
    MSG
  )
end

Command::SendThankYouEmail error rate 12% (normally < 0.5%)

since 02:47 · 318 failures
291 users affected
304 orders affected

"Marketing asks you if the new campaign has affected the funnel"

SELECT
  class,
  COUNT(DISTINCT user_id) FILTER (WHERE time <  '2026-09-15') AS before,
  COUNT(DISTINCT user_id) FILTER (WHERE time >= '2026-09-15') AS after
FROM events
WHERE event_type = 'command'
  AND class IN ('Command::AddToCart', 'Command::Checkout', 'Command::SendThankYouEmail')
  AND time >= '2026-09-01' AND time < '2026-09-29'
GROUP BY class

                              Before          After campaign
Command::AddToCart            10,000          14,500
Command::Checkout              4,200 (42%)     5,100 (35%)
Command::SendThankYouEmail     3,900 (39%)     4,800 (33%)

"You're shipping a feature, how do you know it's working?"

jobs:
  telemetry-links:
    steps:
      - name: Link changed commands to telemetry
        run: |
          classes=$(git diff --name-only --diff-filter=AM "origin/$BASE...HEAD" -- '*.rb' \
            | xargs -r grep -hoP 'class \K\S+(?=\s*<\s*Command::Base)' \
            | sort -u)

          [ -z "$classes" ] && exit 0

          {
            echo "### Telemetry for changed commands"
            echo
            for c in $classes; do
              echo "- [\`$c\`](https://example.com/events/q?event_type=command&class=$c)"
            done
          } > comment.md

          gh pr comment "$PR" --body-file comment.md

### Telemetry for changed commands

- [`Command::AddToCart`](https://example.com/events/q?event_type=command&class=Command::AddToCart)
- [`Command::InventoryInStock`](https://example.com/events/q?event_type=command&class=Command::InventoryInStock)
- [`Command::SendThankYouEmail`](https://example.com/events/q?event_type=command&class=Command::SendThankYouEmail)

Where to go from here

• Work to reduce the cognitive load on your devs
• Inputs: make it easy to emit wide events with lots of context
• Outputs: eliminate any friction for your devs to get your data back out

Further reading

• Don't Make Me Think, Steve Krug
• The Design of Everyday Things, Don Norman
• Observability Engineering, Charity Majors, Liz Fong-Jones & George Miranda
• Rails instrumentation: ActiveSupport::Notifications
• Rails events: Rails.event.notify

We are hiring!

Multiple roles across Engineering, Marketing, and more!

https://www.phoenix.ca/careers

Come Find Me!

rsbondoc@gmail.com
https://roelbondoc.com