Software Engineering consultant

From Monolithic Seeds to Enterprise-Ready Data Generation: Building a Scalable Rails Seed System

Sat, 24 Jan 2026 00:00:00 +0000

“Good seed data is the difference between a demo that impresses and a demo that embarrasses.”

Every growing Rails application faces the same challenge: how do you populate your database with realistic data for development, demos, testing, and stakeholder presentations? Simple db/seeds.rb files work for small projects, but as your application scales, so do your seeding requirements.

This post documents how we transformed a 3,600-line monolithic seed file into an enterprise-ready data generation system that creates 100,000+ full candidate profiles with all associations—serving AI training, QA stress testing, and frontend performance validation.

The Problem: Why Seed Data Matters
Before: The Monolithic Nightmare
The Refactoring Journey
Feature 1: Modular Step Architecture
Feature 2: Context Object for State Management
Feature 3: Conditional Execution via ENV Flags
Feature 4: Progress Reporting with Timing
Feature 5: Post-Seed Data Validation
Feature 6: 100k+ Full Profile Scaling
Benefits and Use Cases
Testing the Seed System
Three Bonus Ideas Worth Implementing
Conclusion

1. The Problem: Why Seed Data Matters

1.1 Who Uses Seed Data?

Stakeholder	What They Need	Why It Matters
Frontend Developers	Realistic UI data	Test pagination, search, empty states
Backend Developers	Consistent test fixtures	Debug API responses locally
QA Team	Volume for stress testing	Find N+1 queries, memory leaks
AI/ML Team	Training data at scale	Validate models with realistic distributions
Product Managers	Demo-ready environments	Impress stakeholders, test features
CEO/Leadership	Production-like demos	Investor presentations, board meetings
New Engineers	Quick environment setup	Onboard in minutes, not hours

1.2 The Reality Check

Most Rails projects start with something like this:

# db/seeds.rb
User.create!(email: 'admin@example.com', password: 'password')
Company.create!(name: 'Test Company')
# ... done

Then requirements grow:

“We need 50 users with different roles”
“Profiles need skills, work experiences, education”
“QA needs 100,000 profiles for load testing”
“AI team needs realistic job title distributions”
“CEO wants the demo to look like a real platform”

Before you know it, your seed file is 3,600 lines of spaghetti.

1.3 Our Starting Point

Metric	Value
Seed file size	3,610 lines
Total methods	59
Instance variables	8 (shared state)
Dependencies	Complex, undocumented
Test coverage	0%
Documentation	None

Pain points:

❌ Changes broke unrelated features
❌ No way to run individual steps
❌ No progress indication
❌ No validation of seeded data
❌ Impossible to scale to 100k records
❌ New engineers couldn’t understand the flow

2. Before: The Monolithic Nightmare

2.1 The Original Structure

# db/seeds/steps.rb - 3,610 lines!
module Seeds
  module Steps
    class << self
      def run
        seed_predefined_avatar_assets
        seed_company
        seed_job_titles
        seed_admin_users
        # ... 32 more method calls
        print_summary
      end

      private

      def seed_company
        @company = Company.find_or_initialize_by(name: 'Bluegeko')
        @company.assign_attributes(
          # ... 50 lines of attributes
        )
        @company.save!
        
        # ... 200 more lines for demo companies
      end

      def seed_admin_users
        @admin_user = User.find_or_initialize_by(email: 'admin@bluegeko.com')
        # ... 300 lines
      end

      # ... 56 more methods
    end
  end
end

2.2 The Problems

Problem	Impact
Single 3,600-line file	IDE struggles, git conflicts
Implicit dependencies	Can’t run steps individually
Instance variable soup	`@company` used 28 times across methods
No progress feedback	“Is it stuck or working?”
No validation	Broken seeds discovered in production demos
No scalability	Creating 100k records = 100k individual inserts
Zero documentation	New engineers lost for days

2.3 Instance Variable Reference Count

Variable	References
`@company`	28
`@admin_user`	18
`@candidate_demo_user`	15
`@access_request_user`	14
`@standard_user`	8
`@full_data_candidate_users`	5
`@demo_job_posts`	5
`@admin_profile`	2

These shared variables made refactoring terrifying—change one method, break five others.

3. The Refactoring Journey

3.1 Implementation Priority Matrix

#	Feature	Effort	Priority	Status
1	Modular step architecture	High	Critical	✅ Done
3	Conditional ENV flags	Low	High	✅ Done
4	Context object	Medium	High	✅ Done
5	Progress + timing	Medium	High	✅ Done
6	Data validation	Low	High	✅ Done
7	100k batch processing	High	Very High	✅ Done
2	Parallel execution	Medium-High	Medium	⏳ Future

3.2 Final Architecture

db/seeds/
├── seeds.rb                    # Entry point (29 lines)
├── support.rb                  # Helpers (387 lines)
├── reference_data.rb           # Lookup tables (889 lines)
├── steps.rb                    # Orchestrator (217 lines)
├── context.rb                  # State management (233 lines)
├── timing.rb                   # Progress tracking (186 lines)
├── validation.rb               # Data integrity (331 lines)
├── batch_profiles.rb           # 100k scaling (825 lines)
├── README.md                   # Documentation (1,797 lines)
├── SEEDS_REFACTORING.md        # Implementation notes (643 lines)
└── steps/                      # 37 individual step files
    ├── 01_predefined_avatar_assets.rb
    ├── 02_company.rb
    ├── ...
    └── 37_batch_profiles.rb

Result: 3,610 lines → 37 focused files averaging 80 lines each

4. Feature 1: Modular Step Architecture

4.1 The Approach: Singleton Class Reopening

Instead of a massive refactor, we used Ruby’s ability to reopen classes:

# db/seeds/steps/02_company.rb
module Seeds
  module Steps
    class << self
      private

      def seed_company
        ctx.company = Company.find_or_initialize_by(name: "Bluegeko")
        ctx.company.assign_attributes(company_attributes)
        ctx.company.save!

        log_seed "Company upserted: #{ctx.company.name}"
        seed_demo_companies
      end

      def company_attributes
        {
          industry: "Technology",
          website_url: "https://bluegeko.com",
          # ... attributes
        }
      end
    end
  end
end

4.2 Orchestrator Pattern

The main steps.rb became a thin orchestrator:

# db/seeds/steps.rb
module Seeds
  module Steps
    OPTIONAL_STEPS = {
      seed_reports: "SKIP_SEED_REPORTS",
      seed_full_data_candidates: "SEED_FULL_CANDIDATES",
      seed_batch_profiles: "CREATE_BATCH_PROFILES"
    }.freeze

    class << self
      def run
        initialize_context
        Seeds::Support::Timing.start_run(all_steps.size)

        # Execute steps in order
        run_step :seed_predefined_avatar_assets
        run_step :seed_company
        run_step :seed_job_titles
        # ... 34 more steps
        run_step :seed_batch_profiles

        # Validate and summarize
        validate_seed_data
        print_summary
        Seeds::Support::Timing.print_summary

        ctx
      end

      private

      def run_step(step_name)
        return if skip_step?(step_name)

        Seeds::Support::Timing.track_step(step_name) do
          send(step_name)
        end
      end

      def skip_step?(step_name)
        env_var = OPTIONAL_STEPS[step_name]
        return false unless env_var

        if env_var.start_with?("SKIP_")
          ENV[env_var] == "true"
        else
          ENV[env_var] != "true"
        end
      end
    end
  end
end

4.3 Benefits of Modular Architecture

Aspect	Before	After
File size	3,610 lines	~80 lines avg
Git conflicts	Constant	Rare
IDE navigation	Painful	Instant
Step isolation	Impossible	Easy
Testing	Untestable	Per-step specs
Onboarding	Days	Hours

5. Feature 2: Context Object for State Management

5.1 The Problem: Instance Variable Chaos

# Before: 95 instance variable references across 37 steps
def seed_admin_users
  @admin_user = User.create!(...)
end

def seed_job_posts
  job = JobPost.create!(company: @company)  # Where is @company set?
end

Issues:

No IDE autocomplete
No documentation of what’s available
No validation that required data exists
Testing requires complex setup

5.2 The Solution: Explicit Context Object

# db/seeds/context.rb
module Seeds
  module Support
    class Context
      # Core entities
      attr_accessor :company
      attr_accessor :admin_user, :standard_user
      attr_accessor :candidate_demo_user, :access_request_user
      attr_accessor :admin_profile, :full_data_candidate_users
      attr_accessor :demo_job_posts

      # Client roles
      attr_accessor :client_user, :client_owner, :client_manager

      # Batch processing
      attr_accessor :batch_profile_result

      def initialize
        @full_data_candidate_users = []
        @demo_job_posts = []
      end

      # Convenience methods
      def company?
        !@company.nil?
      end

      def batch_profiles?
        @batch_profile_result&.success? == true
      end

      def batch_profile_count
        @batch_profile_result&.profiles_created || 0
      end

      # Dependency validation
      def require!(*attrs)
        missing = attrs.select { |attr| send(attr).nil? }
        raise "Missing required context: #{missing.join(', ')}" if missing.any?
      end

      # Debugging helper
      def summary
        {
          company: @company&.name,
          admin_user: @admin_user&.email,
          full_data_candidates: @full_data_candidate_users.size,
          batch_profiles: batch_profile_count
        }
      end
    end
  end
end

5.3 Usage in Steps

# Access via ctx helper
def seed_job_posts
  ctx.require!(:company, :admin_user)  # Fail fast if deps missing

  job = JobPost.create!(
    company: ctx.company,
    user: ctx.admin_user,
    title: "Senior Engineer"
  )
  ctx.demo_job_posts << job
end

5.4 Why Context Object Matters

Aspect	Instance Variables	Context Object
Discovery	`grep @` across files	`ctx.` autocomplete
Documentation	None	`attr_accessor` list
Validation	Runtime errors	`ctx.require!`
Testing	Mock everything	Inject test context
Debugging	puts everywhere	`ctx.summary`

6. Feature 3: Conditional Execution via ENV Flags

6.1 Use Cases

Use Case	ENV Flag	Behavior
Skip slow reports	`SKIP_SEED_REPORTS=true`	Reports step skipped
Create 40 full candidates	`SEED_FULL_CANDIDATES=true`	Rich candidate data
Create 100k profiles	`CREATE_BATCH_PROFILES=true`	Batch processing enabled

6.2 Implementation

# db/seeds/steps.rb
OPTIONAL_STEPS = {
  # Skip steps (default: run)
  seed_reports: "SKIP_SEED_REPORTS",

  # Enable steps (default: skip)
  seed_full_data_candidates: "SEED_FULL_CANDIDATES",
  seed_full_data_candidates_job_board: "SEED_FULL_CANDIDATES",
  seed_batch_profiles: "CREATE_BATCH_PROFILES"
}.freeze

def skip_step?(step_name)
  env_var = OPTIONAL_STEPS[step_name]
  return false unless env_var

  if env_var.start_with?("SKIP_")
    ENV[env_var] == "true"  # Skip if true
  else
    ENV[env_var] != "true"  # Skip unless true
  end
end

6.3 Usage Examples

# Standard development seed (fast)
rails db:seed

# Full candidates for feature testing
SEED_FULL_CANDIDATES=true rails db:seed

# 100k profiles for QA stress testing
CREATE_BATCH_PROFILES=true rails db:seed

# Skip reports for faster iteration
SKIP_SEED_REPORTS=true rails db:seed

# Combined flags
SEED_FULL_CANDIDATES=true CREATE_BATCH_PROFILES=true rails db:seed

6.4 Benefits

Scenario	Without Flags	With Flags
Daily development	~5 min (full seed)	~30 sec (minimal)
Feature testing	Edit code	Set ENV var
QA stress test	Impossible	One command
CI pipeline	Full seed always	Skip reports

7. Feature 4: Progress Reporting with Timing

7.1 The Problem: Silent Seeds

$ rails db:seed
# ... silence for 30 seconds ...
# ... more silence ...
# Is it working? Is it stuck?

7.2 The Solution: Real-Time Progress

# db/seeds/timing.rb
module Seeds
  module Support
    module Timing
      class << self
        def start_run(total_steps)
          @total_steps = total_steps
          @current_step = 0
          @step_timings = {}
          @run_started_at = Time.current
        end

        def track_step(step_name)
          @current_step += 1
          started = Time.current

          print_progress(step_name) unless quiet?

          yield

          duration = Time.current - started
          @step_timings[step_name] = duration
        end

        def print_progress(step_name)
          puts format(
            "[%2d/%2d] %-40s",
            @current_step, @total_steps,
            step_name.to_s.gsub(/^seed_/, "").humanize
          )
        end

        def print_summary
          puts "\n#{'=' * 60}"
          puts "SEED TIMING SUMMARY"
          puts "=" * 60

          @step_timings.sort_by { |_, v| -v }.first(10).each do |step, duration|
            puts format("  %-40s %6.2fs", step, duration)
          end

          puts "-" * 60
          puts format("  %-40s %6.2fs", "TOTAL", total_duration)
          puts "=" * 60
        end
      end
    end
  end
end

7.3 Output Example

[ 1/37] Predefined avatar assets              
[ 2/37] Company                               
[ 3/37] Job titles                            
[ 4/37] Admin users                           
...
[35/37] Summary                               
[36/37] Demo recommendation                   

============================================================
SEED TIMING SUMMARY
============================================================
  seed_full_data_candidates                     12.34s
  seed_random_profiles                           8.21s
  seed_homepage_profiles                         4.56s
  seed_job_posts                                 3.12s
  ...
------------------------------------------------------------
  TOTAL                                         45.67s
============================================================

7.4 Benefits

Aspect	Silent Seeds	Progress + Timing
Feedback	None	Real-time progress
Debugging	Which step failed?	Clear step names
Optimization	Guess which is slow	Timing breakdown
CI logs	Useless	Actionable

8. Feature 5: Post-Seed Data Validation

8.1 The Problem: Silent Failures

# Seeds "complete" but data is broken
def seed_admin_users
  @admin_user = User.find_or_initialize_by(email: 'admin@bluegeko.com')
  @admin_user.save  # No bang! Silent failure
end

# Later, in production demo...
# "Why is there no admin user?!"

8.2 The Solution: Validation Module

# db/seeds/validation.rb
module Seeds
  module Support
    module Validation
      CHECKS = {
        # Integrity checks (errors)
        orphaned_profiles: -> { Profile.left_joins(:user).where(users: { id: nil }) },
        candidates_without_profiles: -> { User.candidate.left_joins(:profile).where(profiles: { id: nil }) },
        orphaned_job_posts: -> { JobPost.left_joins(:company).where(companies: { id: nil }) },

        # Minimum data checks (errors)
        minimum_users: -> { User.count >= 5 },
        minimum_profiles: -> { Profile.count >= 3 },
        minimum_companies: -> { Company.count >= 1 },
        minimum_job_posts: -> { JobPost.count >= 5 },

        # Quality checks (warnings)
        profiles_with_skills: -> { Profile.joins(:skills).distinct.count > 0 },
        profiles_with_work_experience: -> { Profile.joins(:work_experiences).distinct.count > 0 },
        users_with_login_methods: -> { User.joins(:login_methods).distinct.count > 0 }
      }.freeze

      class << self
        def run
          return skip_result if skip?

          result = ValidationResult.new
          run_integrity_checks(result)
          run_minimum_checks(result)
          run_quality_checks(result) unless strict?

          print_results(result)
          result
        end

        private

        def run_integrity_checks(result)
          %i[orphaned_profiles candidates_without_profiles orphaned_job_posts].each do |check|
            count = CHECKS[check].call.count
            if count > 0
              result.add_error(check, "Found #{count} #{check.to_s.humanize.downcase}")
            else
              result.add_passed(check)
            end
          end
        end
      end
    end
  end
end

8.3 Validation Output

============================================================
POST-SEED VALIDATION
============================================================
Integrity Checks:
  ✓ No orphaned profiles
  ✓ No candidates without profiles
  ✓ No orphaned job posts

Minimum Data Checks:
  ✓ Users: 47 (minimum: 5)
  ✓ Profiles: 42 (minimum: 3)
  ✓ Companies: 5 (minimum: 1)
  ✓ Job Posts: 25 (minimum: 5)

Quality Checks:
  ✓ Profiles with skills: 38
  ✓ Profiles with work experience: 35
  ⚠ Users with login methods: 0 (warning)

------------------------------------------------------------
Result: PASSED (14 checks, 1 warning)
============================================================

8.4 ENV Controls

# Skip validation (CI speed)
SKIP_SEED_VALIDATION=true rails db:seed

# Strict mode (fail on warnings)
SEED_VALIDATION_STRICT=true rails db:seed

9. Feature 6: 100k+ Full Profile Scaling

9.1 The Challenge

Our AI team needed 100,000 realistic candidate profiles for model training. QA needed the same for stress testing. Creating records one-by-one would take hours.

Requirements:

Full profiles with ALL associations
Category-appropriate job titles
Skills, work experiences, education
Portfolio, contact info, avatars
Realistic data distributions
Conditional execution (don’t run in daily dev)

9.2 What Gets Created

Each batch profile is a complete candidate with:

Association	Per Profile	Total (100k)
User + Login Security	1	100,000
Profile	1	100,000
Skills	3-6	~450,000
Work Experiences	2-4	~300,000
Education Items	1-2	~150,000
Languages	1-3	~200,000
Contact Infos	2-4	~300,000
Portfolio + Links	3-9	~600,000
Location Setting	1	100,000
Salary Expectation	1	100,000
Avatar Session + Images	4-5	~450,000
Total Records	~20-30	~2,500,000+

9.3 Category-Specific Data

CATEGORY_OCCUPATIONS = {
  "technology" => [
    "Software Engineer", "Senior Software Engineer", "Staff Software Engineer",
    "DevOps Engineer", "Data Scientist", "Machine Learning Engineer",
    "Mobile Developer", "Cloud Architect", "Security Engineer"
  ],
  "design" => [
    "UX Designer", "Product Designer", "Creative Director",
    "Interaction Designer", "Design Systems Lead"
  ],
  "business" => [
    "Project Manager", "Product Manager", "Business Analyst",
    "Strategy Consultant", "Operations Manager"
  ],
  # ... 9 categories total
}.freeze

CATEGORY_SKILLS = {
  "technology" => %w[Ruby Python JavaScript TypeScript Docker Kubernetes AWS],
  "design" => %w[Figma Sketch Photoshop Wireframing UserResearch],
  "business" => %w[Excel Salesforce Tableau Leadership Strategy],
  # ...
}.freeze

9.4 Implementation Highlights

# db/seeds/batch_profiles.rb
module Seeds
  module Support
    module BatchProfiles
      DEFAULT_COUNT = 100_000
      CHUNK_SIZE = 500

      def run(count: target_count, on_progress: nil)
        result = BatchResult.new(target_count: count)

        count.times.each_slice(chunk_size) do |indices|
          chunk_result = process_chunk(indices)
          result.record_chunk(chunk_result)
          on_progress&.call(result)
        end

        result.finish!
        result
      end

      private

      def process_chunk(indices)
        indices.each do |index|
          # Select random category
          category = categories.sample

          # Create user (ActiveRecord for Devise)
          user = create_batch_user(index: index)

          # Create profile with category-appropriate job title
          profile = create_batch_profile(user: user, category: category)

          # Create ALL associations
          create_skills(profile, category.slug)
          create_work_experiences(profile, category.slug)
          create_education_items(profile)
          create_languages(profile)
          create_contact_infos(profile)
          create_portfolio(profile)
          create_location_setting(profile)
          create_salary_expectation(profile)
          create_avatar_data(profile)
        end
      end
    end
  end
end

9.5 Usage

# Create 100,000 full profiles (default)
CREATE_BATCH_PROFILES=true rails db:seed

# Create 10,000 for smaller tests
CREATE_BATCH_PROFILES=true BATCH_PROFILE_COUNT=10000 rails db:seed

# With verbose progress
CREATE_BATCH_PROFILES=true BATCH_VERBOSE=true rails db:seed

9.6 Output

======================================================================
[BATCH] Starting FULL candidate profile creation
======================================================================
  Target count:     100,000
  Chunk size:       500
  Associations:     Skills, Work Exp, Education, Languages, Contacts, etc.
======================================================================

  [=======>--------------------]  25.0% | Chunk 50/200 | 25000 profiles | 625,000 assoc

======================================================================
[BATCH] Full candidate profile creation complete!
======================================================================
  Users created:        100,000
  Profiles created:     100,000
  Associations created: 2,543,721
  Total duration:       3h 12m
  Rate:                 8.7 profiles/second
======================================================================

9.7 Performance Characteristics

Count	Time	Records Created
1,000	~2-3 min	~25,000
10,000	~20-30 min	~250,000
100,000	~3-5 hours	~2,500,000

10. Benefits and Use Cases

10.1 Use Case Matrix

Stakeholder	Command	Result
Daily Dev	`rails db:seed`	~30s, minimal data
Feature Testing	`SEED_FULL_CANDIDATES=true rails db:seed`	~2 min, 40 rich profiles
QA Stress Test	`CREATE_BATCH_PROFILES=true rails db:seed`	~3h, 100k profiles
AI Training	`CREATE_BATCH_PROFILES=true BATCH_PROFILE_COUNT=500000 rails db:seed`	~15h, 500k profiles
CEO Demo	`rails db:seed`	Production-like feel
New Engineer	`rails db:seed`	Working environment in 30s

10.2 Before vs After Comparison

Aspect	Before	After
File organization	1 file, 3,610 lines	37 files, ~80 lines each
Test coverage	0%	400+ specs
Documentation	None	1,797-line README
Progress feedback	Silent	Real-time with timing
Data validation	None	14 automated checks
Scalability	~100 profiles	100,000+ profiles
Conditional execution	Edit code	ENV flags
Onboarding time	Days	Hours
Git conflicts	Daily	Rare

10.3 ROI

Time saved per week:

5 engineers × 10 min/day waiting for seeds = 4+ hours
2 QA engineers × 1 hour/week debugging broken seeds = 2 hours
1 demo preparation × 2 hours = 2 hours

Total: ~8 hours/week = 1 FTE day

11. Testing the Seed System

11.1 Test Structure

spec/seeds/
├── support/
│   ├── helpers_spec.rb           # Unit tests for helpers
│   ├── context_spec.rb           # Context object tests
│   ├── timing_spec.rb            # Timing module tests
│   ├── validation_spec.rb        # Validation tests
│   └── batch_profiles_spec.rb    # Batch processing tests
├── reference_data_spec.rb        # Reference data tests
├── integration_spec.rb           # Full seed run tests
└── steps/
    ├── shared_examples.rb        # Reusable step examples
    ├── company_spec.rb           # Step-specific tests
    └── ...

11.2 Shared Examples for Steps

# spec/seeds/steps/shared_examples.rb
RSpec.shared_examples "a seed step that creates records" do |model_class, expected_count:|
  it "creates #{expected_count} #{model_class.name.pluralize}" do
    expect { run_step }.to change(model_class, :count).by(expected_count)
  end
end

RSpec.shared_examples "a seed step that is idempotent" do
  it "does not create duplicates on second run" do
    run_step
    expect { run_step }.not_to change(described_model, :count)
  end
end

11.3 Test Coverage

Component	Specs	Coverage
Helpers	45	100%
Context	32	100%
Timing	28	100%
Validation	31	100%
Batch Profiles	49	100%
Reference Data	89	100%
Individual Steps	126	Key paths
Total	400+	High

12. Three Bonus Ideas Worth Implementing

12.1 Idea 1: Seed Data Snapshots

Problem: Recreating 100k profiles takes 3 hours every time.

Solution: Database snapshots for instant restore.

# Potential implementation
module Seeds
  module Snapshots
    def save(name)
      `pg_dump wigiwork_development > snapshots/#{name}.sql`
    end

    def restore(name)
      `psql wigiwork_development < snapshots/#{name}.sql`
    end
  end
end

# Usage
# After 3-hour batch: SEED_SNAPSHOT=save rails db:seed
# Next time: SEED_SNAPSHOT=restore:batch_100k rails db:seed (~30 sec)

Benefit: 3 hours → 30 seconds for QA environments.

12.2 Idea 2: Faker Seed Consistency

Problem: Random data changes every seed run, breaking visual regression tests.

Solution: Seed Faker’s random generator.

# db/seeds/support.rb
def with_consistent_faker
  original_seed = Faker::Config.random
  Faker::Config.random = Random.new(42)
  yield
ensure
  Faker::Config.random = original_seed
end

# Usage
with_consistent_faker do
  seed_profiles  # Same names, emails, etc. every time
end

Benefit: Consistent screenshots for visual regression testing.

12.3 Idea 3: Seed Data Metrics Dashboard

Problem: No visibility into what’s seeded without running queries.

Solution: Post-seed metrics report.

# db/seeds/metrics.rb
module Seeds
  module Metrics
    def report
      {
        users: {
          total: User.count,
          by_role: User.group(:role).count,
          with_profiles: User.joins(:profile).count
        },
        profiles: {
          total: Profile.count,
          with_skills: Profile.joins(:skills).distinct.count,
          avg_skills: Profile.joins(:skills).group(:id).count.values.sum.to_f / Profile.count,
          by_category: Profile.joins(:category).group("categories.slug").count
        },
        companies: {
          total: Company.count,
          with_jobs: Company.joins(:job_posts).distinct.count
        }
      }
    end
  end
end

Output:

Users:
  total: 100,047
  by_role: { admin: 2, candidate: 100,040, client: 5 }
  with_profiles: 100,042

Profiles:
  total: 100,042
  with_skills: 99,800
  avg_skills: 4.2
  by_category: { technology: 45,000, design: 15,000, ... }

Companies:
  total: 5
  with_jobs: 3

Benefit: Instant insight into seed data quality and distribution.

13. Conclusion

13.1 What We Achieved

Metric	Before	After
Lines in main file	3,610	217
Number of files	1	37
Test coverage	0%	400+ specs
Documentation	0 lines	1,797 lines
Max profiles	~100	100,000+
Progress feedback	None	Real-time
Data validation	None	14 checks
Conditional execution	None	5 ENV flags

13.2 Key Takeaways

Modular beats monolithic — Split large files into focused modules
Explicit state beats implicit — Context objects over instance variables
Conditional execution is essential — ENV flags for different scenarios
Visibility prevents pain — Progress reporting and timing
Validation catches bugs — Automated checks before they reach demos
Scale when needed — Batch processing for volume requirements
Document everything — Future you (and new engineers) will thank you

13.3 The Real Win

The technical improvements are measurable, but the real win is confidence:

Confident that seeds work (validation)
Confident about progress (timing)
Confident in isolation (modular files)
Confident at scale (batch processing)
Confident for demos (realistic data)
Confident for new engineers (documentation)

Invest in your seed system. It’s the foundation everything else is built on.

Resources

Rails Guides on Seeds: guides.rubyonrails.org/active_record_migrations.html#migrations-and-seed-data
Faker Gem: github.com/faker-ruby/faker
FactoryBot: github.com/thoughtbot/factory_bot
Database Cleaner: github.com/DatabaseCleaner/database_cleaner
Our Seed System Documentation: See db/seeds/README.md in your codebase

This post documents the transformation of the Wigiwork seed system from a 3,600-line monolith to an enterprise-grade data generation platform. The patterns described here—modular architecture, context objects, conditional execution, progress tracking, validation, and batch processing—are applicable to any Rails application that has outgrown simple seeds.

From Zero-Gap to Zero-Drift: Making AI Follow Rails Rules (Frozen vs Strict)

Wed, 07 Jan 2026 00:00:00 +0000

TL;DR

Contract-first + verification gates eliminate gaps (missing fields, regressions). fileciteturn2file0

But LLMs still create rule drift (they quote rules, then violate them).

We fixed drift with guardrails, not more prose: Hard-Fail rule IDs, STOP gates, mandatory evidence audits, and negative examples.

Entry point stays tiny and stable; the enforcement lives in versioned docs.

We keep two profiles: Frozen (v1.0) for speed, Strict (v1.1) for enforcement.

Why this follow-up exists

In Zero-Gap API Development, the goal was “ship APIs with zero regressions” by turning implementation into a contract + verification problem: TypeScript interfaces as canonical truth, safe defaults, systematic verification gates, and discrepancy classification ([IMPL] vs [DOC]). fileciteturn2file0

That framework works. The problem we hit next was different:

AI doesn’t drift because rules are unclear. It drifts because rules have no consequences.

Even when an agent reads AGENTS.md and repeats rules back, it can still “helpfully” shortcut:

“Use Ransack for filtering/searching.” → agent writes custom where(...) anyway.
“Avoid DB-specific SQL.” → agent sneaks in ILIKE, IFNULL, REGEXP, etc.
“Controllers only wrap envelopes.” → agent hand-builds JSON in controller.

So this session was about adding an enforcement layer on top of Zero-Gap.

The goal of this session

We wanted a system where a reviewer can say:

“This violates two rules: HF-1 and HF-2”

…and the agent itself is forced to detect and correct that before shipping code.

To do that, we restructured the workflow into a toolchain:

PROMPT.md → execution runtime (phases, STOP gates, verification receipts)
AGENTS.md → hard authority (Hard-Fail rules, failure semantics)
GUIDE.md → patterns & examples (non-authority, includes anti-patterns)
bin/verify → canonical verifier

The single entry point (stable forever)

Instead of embedding giant prompts in README/issues, every task uses the same tiny entry point:

Task: WEB-485 – User Notifications

Refs:
- Requirements: doc/requirements/users/USER_NOTIFICATIONS.md
- Flow: doc/flow/WEB-485_user_notifications.md
- PRD: doc/prd/WEB-485_user_notifications_PRD.md

Execute per PROMPT.md.

Important property: the task block never changes.
Strictness evolves only inside PROMPT.md, AGENTS.md, and GUIDE.md.

This prevents the most common drift source: copying outdated prompt blobs across tickets.

Frozen vs Strict: why we keep both

We ended up with two prompt profiles:

Frozen (v1.0)

Optimized for: brevity, human readability, speed
Best for: trusted contributors, small refactors, quick spikes
Weakness: rules are descriptive (“MUST”), so AI can rationalize shortcuts

Strict (v1.1)

Optimized for: compliance, auditability, “no surprises” AI execution
Best for: contract-sensitive endpoints, anything touching search/filtering/auth/pagination
Weakness: longer text (but it’s enforcement scaffolding, not noise)

The key is: both share the same entry point. You swap the runtime, not the call site.

What actually stops drift (the 4 mechanisms)

1) Hard-Fail rule IDs (HF-*)

Instead of “Rule #5,” we introduced stable IDs:

HF-1 — Ransack-only search/filtering
HF-2 — DB-agnostic queries only (no raw SQL / DB-specific funcs)
HF-3 — Blueprinter-only JSON
HF-4 — snake_case only
HF-5 — Required fields never null (safe defaults)

Why IDs matter: they turn “standards” into enforceable references:

easy review comments (“HF-2 violation”)
easy self-audit checkboxes
easy future evolution (rename content, keep ID stable)

2) STOP gates (control flow, not suggestions)

Strict mode adds explicit control flow:

Phase 0–2: NO CODE
Stop after planning
If any HF-* would be violated → STOP and report

LLMs follow control flow better than prose.

3) Mandatory Rule Compliance Audit (proof, not vibes)

Before final output, the agent must produce:

RULE COMPLIANCE AUDIT
HF-1: COMPLIANT — evidence: UsersController#index uses User.ransack(params[:q])
HF-2: COMPLIANT — evidence: no raw SQL; only ActiveRecord/Arel
...

This forces the agent to prove it complied. If it can’t produce evidence, it typically self-corrects before shipping.

4) Negative examples (anti-pattern firewall)

Guides that only show “good” patterns still allow AI to invent “bad” ones. So we added explicit “DO NOT COPY” snippets:

# ❌ HF-1 violation (manual SQL filtering)
User.where("email ILIKE ?", "%#{params[:email]}%")

# ✅ Correct (Ransack owns filtering)
User.ransack(params[:q]).result

LLMs imitate examples aggressively. Negative examples prevent “helpful improvisation.”

Example: the exact drift we’re preventing

Bad (HF-1 + HF-2 violation)

# ❌ don't do this
users = User.where("email ILIKE ?", "%#{params[:email]}%")

Good (HF-1 compliant)

search = User.ransack(params[:q])
users  = search.result.page(params[:page]).per(params[:per_page] || 20)

In strict mode, if an agent proposes the bad version, it must output:

❌ RULE VIOLATION
Rule: HF-1
Location: app/controllers/...:12
Reason: manual filtering used instead of Ransack
Required Fix: replace with Model.ransack(params[:q]).result

No debate. Fix it or stop.

Verification: one command, one receipt

Zero-Gap already required verification gates. fileciteturn2file0
The strict upgrade makes verification harder to “forget” by standardizing:

Preferred: bin/verify
Legacy commands remain in docs as commented fallback

And the final output requires a CHECKS section with exit codes.

That turns “I think I ran tests” into a receipt.

Operational guidance: when to use which mode

Use Frozen (v1.0) when:

small refactor
low contract risk
you want short prompts and fast iteration

Use Strict (v1.1) when:

building/upgrading endpoints
touching filtering/search/pagination/auth
upgrading versioned requirement docs
AI is doing most of the work

If in doubt: default to Strict.

Closing: Zero-Gap is the architecture; Zero-Drift is governance

Zero-Gap prevents missing fields and regressions by making contracts and verification non-negotiable. fileciteturn2file0
Zero-Drift makes AI reliably follow your engineering rules by adding:

failure semantics
proof requirements
anti-pattern training
a stable entry point
a canonical verifier

In practice, this reduces review churn dramatically: you spend less time catching “obvious violations” and more time on product decisions.

Appendix: quick mental model

REQUIREMENT_DOC = spec
PROMPT.md = runtime
AGENTS.md = language law + compiler errors (HF-*)
GUIDE.md = standard library + examples
bin/verify = test runner

How to Start with Agentic Development: What to Know, What to Automate, and How to Use It

Thu, 01 Jan 2026 00:00:00 +0000

TL;DR

Agentic development is a loop: Plan → Act (tools) → Observe → Decide → Repeat.

The biggest productivity gains come from determinism, not prompting: a single verify command, quality gates, and enforcement via hooks.

The “must-know” stack in 2026: AGENTS.md, Skills, MCP, RAG, LSP, Hooks, and Workflows.

My two most effective multipliers:

Contract-first + verification gates (“Zero‑Gap API development”)

Documentation hierarchy + Flow docs with Responsible Files (documentation-driven development that scales)

Why agentic development feels different from “AI coding assistants”

Autocomplete and chat are mostly suggestion engines.

Agentic development is when the model runs a structured loop with tools:

it reads your repo conventions
it edits multiple files
it runs commands
it interprets errors
it iterates until the system is green

That means your productivity is no longer limited by “how well can I prompt?”
It’s limited by how well your repo communicates intent and enforces correctness.

The modern skill is not “prompting.”
It’s context engineering + verification engineering.

The mental model: the agent loop

An “agent” is just a loop with tool use:

Plan (what files, what changes, what tests)
Act (edit files, run commands, call tools)
Observe (test output, logs, compiler errors, lint)
Decide (fix, rollback, re-plan)
Repeat until done

Your job (as a senior engineer) is to make this loop:

well-scoped (small diffs, explicit “definition of done”)
well-instrumented (fast tests + lint + security checks)
hard to cheat (quality gates / hooks / CI)
context-efficient (agent sees the right context, not more context)

Glossary: plugins, skills, MCP, RAG, LSP, hooks, workflows

Here’s the vocabulary, in a practical order:

AGENTS.md

A repo-local “operating manual” for coding agents.

What it’s for

Encode how this repo works (commands, conventions, boundaries)
Stop re-explaining the same rules every session
Make agent behavior predictable across tasks

Think: README.md for agents, but stricter and more actionable.

Skills

A “Skill” is a packaged workflow (instructions + optional resources/scripts) that the agent can load on demand.

What it’s for

Turn your best senior workflows into reusable modules:
- “Implement Rails endpoint from contract + request specs”
- “Safe refactor: characterization tests first”
- “Write flow doc + Responsible Files table”
Reduce quality drift and context repetition

If AGENTS.md is global repo policy, Skills are repeatable procedures.

Hooks

Hooks are automation that runs at specific moments (after edits, before stopping, etc.).

What it’s for

Enforce quality without relying on “remember to do it”
Auto-format after edits
Block dangerous operations
Require bin/verify passing before the agent can “finish”

Prompts are best-effort. Hooks are default behavior.

Workflows

Workflows are your end-to-end process:

feature → tests → verify → docs → PR
bugfix → reproduction test → fix → verify → postmortem note

Skills + hooks + CI turn workflows into a system.

RAG (Retrieval-Augmented Generation)

RAG is the pattern of retrieving relevant docs/files and injecting them as context for the model.

What it’s for

Ground the agent in your docs and code
Reduce hallucinations
Make “documentation-driven development” actually useful to the model

In practice: RAG is how an agent “remembers” your system without you pasting the whole repo.

LSP (Language Server Protocol)

LSP is what powers “go to definition,” “find references,” “rename symbol,” etc.

What it’s for

Semantic navigation beats grep for refactors and correctness
Agents become safer when they can reliably answer:
- “Where is this symbol used?”
- “What implements this interface?”
- “What breaks if I change this method signature?”

MCP (Model Context Protocol)

MCP is a standard way to connect an agent to tools and data sources (issue trackers, DB schema, docs, internal APIs).

What it’s for

Reduce copy/paste context
Give structured, permissioned access to external systems
Make tool integrations reusable across clients/vendors

MCP is most useful when your bottleneck is access to information/tools, not coding speed.

The “agentic stack” in one picture

A practical way to visualize the stack:

Workflow (what to do)
  └─ Hooks (force it to happen)
      └─ Tool/data access (MCP)
          └─ Context feeding (RAG + LSP)
              └─ Skills (repeatable procedures)
                  └─ AGENTS.md (repo rules + commands + DoD)

If you want a big productivity jump: build from the bottom up.

The senior engineer principle: determinism beats prompting

If something must happen every time:

formatting
tests
lint
security scan
contract compliance check

…don’t “ask the model to remember.”

Make it one command and/or enforce it with hooks + CI.

This is the same reason “Zero‑Gap API Development” works: it’s a deterministic pipeline (contract → implement → verify → document), not vibes.
And it’s the same reason documentation-driven development scales for large systems: it creates deterministic “where to look” maps.

A universal framework that scales: Requirements → Implementation → Documentation

This generalizes cleanly across Rails monoliths, Go services, and polyglot repos:

┌──────────────────────────────────────────────┐
│ Layer 1: Requirements / Contracts            │
│ - Interfaces / OpenAPI / protobuf / specs    │
│ - Single source of truth                     │
└──────────────────────────────────────────────┘
                     ↓
┌──────────────────────────────────────────────┐
│ Layer 2: Implementation + Verification       │
│ - Code + tests + lint + security             │
│ - One “judge” command: bin/verify            │
└──────────────────────────────────────────────┘
                     ↓
┌──────────────────────────────────────────────┐
│ Layer 3: Documentation / Memory              │
│ - Flow docs (how it’s implemented)           │
│ - PRDs (why it exists)                       │
│ - Responsible Files tables                   │
└──────────────────────────────────────────────┘

Two key rules:

Docs are generated/updated only after verification passes
Contracts are treated as the canonical truth (or explicitly flagged when reality diverges)

Step 1: Add AGENTS.md to every repo

Put this at the repo root. Keep it short and operational.

# AGENTS.md — Project instructions for coding agents

## Prime directive
- Prefer small, verifiable changes.
- Never skip verification commands.
- If unsure: add/adjust tests first to lock behavior.

## Commands (source of truth)
### Ruby / Rails
- Setup: bundle install
- Test:  bin/test
- Lint:  bin/lint
- Verify (must pass before PR): bin/verify
- Security: bin/security

### Go (if present)
- Test:  make test
- Lint:  make lint
- Verify: make verify

### Node (if present)
- Test:  pnpm test
- Lint:  pnpm lint
- Typecheck: pnpm typecheck

## Definition of Done
- [ ] Implementation matches contract/specs
- [ ] Tests added/updated and pass
- [ ] Lint/static checks pass
- [ ] Security checks pass
- [ ] Docs updated (flow + PRD) if behavior changed
- [ ] No unrelated refactors in same diff

## Safety boundaries
- Don’t touch secrets/credentials or production config.
- Ask before adding dependencies or changing CI.

This alone reduces agent thrash dramatically.

Step 2: Create one “judge command” (`bin/verify`)

Agents converge faster when there’s exactly one command that says “reality is correct.”

Rails example

#!/usr/bin/env bash
set -euo pipefail

echo "==> Lint"
bundle exec rubocop

echo "==> Tests"
bundle exec rspec

echo "==> Security (brakeman)"
bundle exec brakeman -q -w2

echo "==> Dependency audit"
bundle exec bundle audit check --update

echo "==> API docs generation (if used)"
bundle exec rails rswag:specs:swaggerize

echo "✅ verify passed"

Go example

verify:
	@gofmt -w .
	@golangci-lint run
	@go test ./...

The exact tools don’t matter.
What matters is: one obvious place to run the truth.

Step 3: Use contract-first development (especially for APIs)

My most reliable pattern for AI-assisted Rails APIs is contract-first:

treat an interface/spec as canonical truth
implement strictly against it
verify via gates
then update docs

Safe defaults matter (required fields are never null)

In contract-first work, required fields should not come back as null “because DB is sparse.”

Use safe defaults by type:

Type	Default
String	`""`
Number	`0` or minimum valid
Boolean	`false`
Array	`[]`
Object	minimal valid object
Timestamp	`Time.current.iso8601`

This removes a common “agent bug class”: missing fields / inconsistent shapes.

Discrepancy tagging: `[IMPL]` vs `[DOC]`

When auditing contract compliance, classify gaps:

[IMPL] → code is wrong, fix it now, rerun verification
[DOC] → the doc/contract is wrong (DB/model reality differs), flag it, don’t silently change requirements

This small habit prevents teams from corrupting the source of truth.

Step 4: Documentation-driven development for complex systems

When systems explode in complexity (MMORPG-scale, or any mature SaaS), your bottleneck becomes coherence, not raw coding time.

The pattern that scales:

Documentation hierarchy

GDD / vision docs: what the system is
Feature docs: what each subsystem must do
Flow docs: how it’s implemented (step-by-step + file ownership)

This is the structure that makes large Rails monoliths maintainable, and also makes AI agents effective because they can retrieve the correct context instead of guessing.

Flow docs + Responsible Files tables

Flow docs should include:

“Use case” steps that trace UI → controller → service → persistence → broadcast
Key behaviors / invariants
A “Responsible Files” table so anyone (human or agent) can jump to the right code immediately

Example table:

Purpose	File
Combat service	`app/services/game/combat/turn_based_combat_service.rb`
Stimulus controller	`app/javascript/controllers/turn_combat_controller.js`
View	`app/views/combat/_battle.html.erb`
Config	`config/gameplay/combat_actions.yml`

This is the best “RAG corpus” you can build: docs that directly point to code.

Step 5: Turn your best practices into Skills

Here’s the trick: you already wrote the workflows in prose.

Now you package them so the agent can apply them repeatedly.

Skill ideas that map to real work

rails-zero-gap-endpoint
- input: requirement doc (contract), expected behavior
- output: controller + serializer/blueprint + request specs + docs updates
- gate: must pass bin/verify
flow-doc-maintainer
- update flow steps
- update Responsible Files table
- append version history (never overwrite)
safe-refactor
- add characterization tests first
- refactor
- rerun verify
- summarize risk and coverage
go-service-skeleton
- standard layout (cmd/, internal/, pkg/)
- health, metrics, config, structured logging
- make verify/test/lint

Skill template (conceptual)

# SKILL.md
name: rails-zero-gap-endpoint
description: Implements a Rails API endpoint from a contract/spec with tests, verification, and doc updates.

## Steps
1. Read contract/spec in doc/requirements/**.
2. Identify response shapes and status codes.
3. Implement with serializers/blueprints.
4. Add request specs (success + error cases).
5. Run bin/verify; fix until green.
6. Update flow doc + PRD notes; append version history.

The point is not “more features.”
The point is standardizing your best engineering behavior.

Step 6: Add hooks so quality is automatic

Once bin/verify exists, hooks become a superpower.

High ROI hooks:

Format after edits
Block sensitive files
Don’t stop until verify passes
Auto-add a short change log entry for docs

This is how you get “ship faster” and “ship safer” at the same time.

When to use RAG vs LSP vs MCP

Use RAG when you have good docs but agents don’t see them

If you already maintain:

contracts/specs
flow docs
ADRs
runbooks

…RAG turns that into a searchable memory.

Best practice: write docs in chunks that can be retrieved (short sections, clear headings, explicit file names).

Use LSP when correctness depends on understanding symbol relationships

Refactors, renames, interface changes → you want semantic certainty.

Use MCP when the bottleneck is tool/data access

If your “context” lives outside the repo:

DB schema details
tickets/PRDs in trackers
dashboards/logs
internal service catalogs

…MCP makes that safe and standardized.

A practical daily workflow that compounds productivity

Feature workflow

Define/confirm contract/spec (or acceptance criteria)
Agent generates a plan: files + tests + risks
Implement smallest vertical slice
Run bin/verify until green
Audit for gaps ([IMPL] / [DOC])
Update flow doc + PRD notes

Bugfix workflow

Reproduce + write failing test
Minimal fix
bin/verify
Add a short note to flow doc/runbook if it’s a recurring class

Refactor workflow

Add characterization tests (freeze behavior)
Refactor
bin/verify
Optional: second agent acts as a paranoid reviewer (coverage + edge cases)

A 60-minute “starter kit” checklist

If you want immediate impact, do this in your next repo:

Add AGENTS.md
Add bin/verify (one truth command)
Add bin/test, bin/lint, bin/security, bin/format
Add /doc/ map in README (what doc is where)
Add 1 Skill for your most common task
Add 1–2 hooks to enforce formatting + verify
In CI: run bin/verify (same as local)

This is the shortest path to “significantly increase daily process.”

Closing: the real multiplier is the system

AI doesn’t make a team fast.
A fast feedback loop makes a team fast.

Agentic development works when:

Truth is easy to check (bin/verify)
Intent is easy to find (contracts + flow docs + Responsible Files)
Quality is enforced (hooks + CI)
Workflows are reusable (Skills)

Once those exist, models become replaceable.
Your process stays.

From 5-Minute Timeouts to Sub-3-Minute Builds: A Complete Guide to Rails RSpec CI Optimization

Wed, 10 Dec 2025 00:00:00 +0000

“A fast test suite isn’t a luxury—it’s the difference between deploying confidently and deploying nervously.”

Our Bitbucket Pipeline was timing out at 5 minutes. With 338 spec files, 456 RSwag integration tests, and growing, we needed a systematic approach to optimization—not just quick fixes.

This post documents how we reduced our CI build time by 60% through parallel execution, factory optimization, intelligent pipeline architecture, and eliminating hidden I/O bottlenecks.

The Problem: Death by Timeout
Diagnostic Analysis
Pipeline Architecture Optimization
Parallel Test Execution
Factory Optimization
File I/O Elimination
Test Profiling Infrastructure
RSpec Configuration Tuning
Results and Metrics
Ongoing Optimization Strategy

1. The Problem: Death by Timeout

1.1 The Symptoms

Our Bitbucket Pipeline build started failing with this error:

Exceeded build time limit of 5 minutes.
Error key: agent.step.time-limit-exceeded

The logs showed tests completing, but swagger generation pushed us over the limit:

bundle exec rspec --tag ~ci_skip --format documentation  # ~4:30
bundle exec rails rswag:specs:swaggerize                  # +0:45 = TIMEOUT

1.2 The Root Causes

After analysis, we identified multiple compounding issues:

Issue	Impact	Root Cause
Sequential execution	100%	Single-threaded test run
RSwag runs tests twice	+50%	Swagger generation re-executes integration specs
Eager factory creation	+15%	252 `let!` calls creating records unnecessarily
Disk I/O in factories	+10%	File.open for every attachment
No parallelization	100%	All 338 specs in one process
Verbose output	+5%	`--format documentation` slower than `--format progress`

1.3 Our Starting Point

Metric	Value
Total spec files	338
Integration specs (RSwag)	64
`run_test!` calls	456
Factory `create()` calls in integration	~502
Eager `let!` evaluations	252
Pipeline timeout	5 minutes
Actual runtime	5+ minutes (timeout)

2. Diagnostic Analysis

2.1 Understanding the Codebase

Before optimizing, we audited the entire test suite:

# Count spec files
find spec -name "*_spec.rb" | wc -l
# => 338

# Count integration specs (RSwag)
find spec/integration -name "*_spec.rb" | wc -l
# => 64

# Count run_test! calls
grep -c "run_test!" spec/integration/*.rb | awk -F: '{sum += $2} END {print sum}'
# => 456

# Count eager evaluations
grep -c "let!" spec/integration/*.rb | awk -F: '{sum += $2} END {print sum}'
# => 252

# Find largest spec files
find spec -name "*_spec.rb" -exec wc -l {} + | sort -n | tail -10

Key finding: companies_spec.rb was 1,476 lines—a maintenance and performance problem.

2.2 Identifying Bottlenecks

We identified these optimization opportunities:

1. Pipeline Architecture

Tests ran sequentially in a single step
Lint, security, and tests couldn’t run in parallel
RSwag swagger generation was blocking

2. Test Execution

No parallel test execution (parallel_tests gem missing)
No --fail-fast flag to stop on first failure
Verbose output format slowing things down

3. Factory Usage

let! used where let would suffice
Duplicate factory creation in each response block
File I/O for every attachment creation

4. Profiling

No visibility into slow tests
No factory usage profiling
No SQL query profiling

3. Pipeline Architecture Optimization

3.1 The Problem: Sequential Steps

Our original pipeline ran everything sequentially:

# Before: Everything in sequence
pipelines:
  default:
    - parallel:
        steps:
          - step:
              name: Lint & Security
              # ... linting
          - step:
              name: RSpec + DB prepare
              script:
                - bundle exec rspec --tag ~ci_skip --format documentation
                - bundle exec rails rswag:specs:swaggerize  # Re-runs integration specs!

Issues:

Single test step runs all 338 specs sequentially
rswag:specs:swaggerize re-runs integration specs for swagger generation
No separation of fast vs slow tests

3.2 The Solution: Three-Way Parallelization

We split the pipeline into three parallel test steps:

# After: Parallel execution by test type
image: ruby:3.4

options:
  size: 4x
  max-time: 10  # Increased from 5 to 10 minutes

definitions:
  caches:
    bundler: vendor/bundle
    bootsnap: tmp/cache/bootsnap  # NEW: Cache Rails boot
  services:
    postgres:
      image: postgres:16
      variables:
        POSTGRES_DB: wigiwork_test
        POSTGRES_USER: postgres
        POSTGRES_PASSWORD: password

pipelines:
  default:
    - parallel:
        steps:
          # Step 1: Linting & Security (no database needed)
          - step:
              name: Lint & Security
              caches:
                - bundler
              script:
                - bundle exec rubocop
                - bundle exec brakeman -q -w2
                - bundle exec bundle audit check --update

          # Step 2: Fast specs (models, blueprints, policies)
          - step:
              name: Unit & Model Tests
              services:
                - postgres
              caches:
                - bundler
                - bootsnap
              script:
                - bundle exec rspec spec/models spec/blueprints spec/policies \
                    spec/services spec/jobs --tag ~ci_skip --fail-fast --format progress

          # Step 3: Integration specs + Swagger generation
          - step:
              name: Integration Tests + Swagger
              services:
                - postgres
              caches:
                - bundler
                - bootsnap
              script:
                - bundle exec rspec spec/integration spec/requests \
                    --tag ~ci_skip --fail-fast --format progress
                - bundle exec rails rswag:specs:swaggerize

    # Sequential step after parallel completes
    - step:
        name: Verify Seeds
        services:
          - postgres
        caches:
          - bundler
        script:
          - bundle exec rails db:seed:replant

3.3 Key Changes Explained

Change	Before	After	Impact
Timeout	5 min	10 min	Safety margin
Parallel steps	2	3	~40% faster
Test splitting	None	Unit vs Integration	Better parallelization
Bootsnap cache	Missing	Added	Faster Rails boot
Output format	`documentation`	`progress`	Less I/O overhead
Fail behavior	Run all	`--fail-fast`	Fail faster

3.4 Why Split Unit and Integration Tests?

Unit tests (models, policies, services):

Fast to execute (minimal database interaction)
Many small files
Good candidates for parallel execution

Integration tests (RSwag, requests):

Slower (full request/response cycle)
Generate Swagger documentation
Benefit from running together for swagger generation

By splitting them, we:

Get faster feedback on unit test failures
Keep swagger generation with its source specs
Allow better cache utilization

4. Parallel Test Execution

4.1 Adding parallel_tests Gem

The parallel_tests gem runs specs across multiple CPU cores:

# Gemfile
group :test do
  gem "parallel_tests"
  gem "test-prof"  # Profiling
end

4.2 Pipeline Integration

For multi-process execution in CI:

# Option A: parallel_tests with database per process
script:
  - bundle exec rake parallel:create parallel:prepare
  - bundle exec parallel_rspec spec/ --tag ~ci_skip -n 4

# Option B: Simple parallel within steps (what we chose)
# Each parallel step runs a subset of specs

We chose Option B because:

Bitbucket Pipelines already provides step-level parallelism
Simpler database setup (one DB per step)
Easier to debug failures
Cache utilization is better with separate steps

4.3 Local Development Usage

For local development, parallel_tests shines:

# Run all specs across 4 cores
bundle exec parallel_rspec spec/ -n 4

# Run with specific pattern
bundle exec parallel_rspec spec/models -n auto

# Profile parallel execution
PROFILE=1 bundle exec parallel_rspec spec/

5. Factory Optimization

5.1 The Problem: Eager Evaluation

Our integration specs used let! excessively:

# Before: Every response block creates its own fixtures
path '/api/v1/projects' do
  response '200', 'successful' do
    let!(:user)    { create(:user) }     # Created immediately
    let!(:profile) { create(:profile) }  # Created immediately
    let!(:project) { create(:project) }  # Created immediately
    run_test!
  end

  response '422', 'validation error' do
    let!(:user)    { create(:user) }     # DUPLICATE creation!
    let!(:profile) { create(:profile) }  # DUPLICATE creation!
    run_test!
  end
end

Issues:

let! creates records even if not referenced
Each response block creates duplicate fixtures
252 eager evaluations across integration specs

5.2 The Solution: Lazy Evaluation

Convert let! to let where the lazy evaluation chain works:

# After: Lazy evaluation with explicit eager loading only when needed
path '/api/v1/projects' do
  response '200', 'successful' do
    let(:user)    { create(:user) }
    let(:profile) { create(:profile, user: user) }
    # Project must exist before request - keep as let!
    let!(:project) { create(:project, profile: profile) }

    let(:Authorization) { "Bearer #{jwt_for(user)}" }  # References user → triggers creation
    let(:profile_id) { profile.id }  # References profile → triggers creation

    run_test!
  end
end

The key insight: let is lazy—it only creates records when first accessed. If Authorization references user, and profile_id references profile, the chain triggers automatically.

Use let! only when:

Record must exist in database before the request
No other let references it
It’s tested via side effects (e.g., count changes)

5.3 Before vs After: projects_spec.rb

# Before: 26 let! calls
response '200', 'successful' do
  let!(:user)    { create(:user) }
  let!(:profile) { create(:profile, user: user) }
  let!(:project) { Project.create!(profile: profile, ...) }
  # ...
end

# After: 5 let! calls (only for records that must pre-exist)
response '200', 'successful' do
  let(:user)    { create(:user) }
  let(:profile) { create(:profile, user: user) }
  let!(:project) { Project.create!(profile: profile, ...) }
  # ...
end

Reduction: 26 → 5 eager evaluations in one file.

5.4 Shared Contexts for Common Patterns

For repeated patterns, use shared contexts:

# spec/support/shared_examples/company_authenticated_context.rb
RSpec.shared_context 'authenticated company user' do
  let(:company) { create(:company) }
  let(:company_owner) { create(:user, :client, company: company, company_role: :owner) }
  let(:Authorization) { "Bearer #{jwt_for(company_owner)}" }
  let(:company_id) { company.id }
end

# Usage in specs
RSpec.describe 'Companies API' do
  include_context 'authenticated company user'

  path '/api/v1/companies/{company_id}/profile' do
    # company, company_owner, Authorization, company_id all available
    # Created lazily when first accessed
  end
end

6. File I/O Elimination

6.1 The Problem: Disk Reads for Every Attachment

Our attachment factory read from disk on every creation:

# Before: Disk I/O for every attachment
factory :attachment do
  after(:build) do |attachment|
    next if attachment.file.attached?

    attachment.file.attach(
      io: File.open(Rails.root.join("spec/fixtures/files/image.png")),  # DISK I/O!
      filename: "image.png",
      content_type: "image/png"
    )
  end
end

Impact:

85+ create(:attachment) calls across specs
Each call reads from disk
~0.5-2ms per read adds up

6.2 The Solution: In-Memory Minimal Files

We created minimal valid files stored in memory:

# spec/support/minimal_files.rb
module MinimalFiles
  # Minimal valid 1x1 transparent PNG (67 bytes)
  PNG = [
    0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A,  # PNG signature
    0x00, 0x00, 0x00, 0x0D, 0x49, 0x48, 0x44, 0x52,  # IHDR chunk
    0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,  # 1x1 dimensions
    0x08, 0x02, 0x00, 0x00, 0x00, 0x90, 0x77, 0x53,  # 8-bit RGB
    0xDE, 0x00, 0x00, 0x00, 0x0C, 0x49, 0x44, 0x41,  # IDAT chunk
    0x54, 0x08, 0xD7, 0x63, 0xF8, 0x0F, 0x00, 0x00,  # Compressed data
    0x01, 0x01, 0x00, 0x05, 0x18, 0xD8, 0x4E, 0x00,  # CRC
    0x00, 0x00, 0x00, 0x49, 0x45, 0x4E, 0x44, 0xAE,  # IEND chunk
    0x42, 0x60, 0x82                                  # CRC
  ].pack("C*").freeze

  # Minimal valid PDF (193 bytes)
  PDF = <<~PDF.freeze
    %PDF-1.4
    1 0 obj<</Type/Catalog/Pages 2 0 R>>endobj
    2 0 obj<</Type/Pages/Kids[3 0 R]/Count 1>>endobj
    3 0 obj<</Type/Page/MediaBox[0 0 612 792]/Parent 2 0 R>>endobj
    xref
    0 4
    0000000000 65535 f
    0000000009 00000 n
    0000000052 00000 n
    0000000101 00000 n
    trailer<</Size 4/Root 1 0 R>>
    startxref
    178
    %%EOF
  PDF

  def self.io_for(type)
    case type.to_sym
    when :png, :image, :avatar, :real_image
      StringIO.new(PNG)
    when :pdf, :file, :document
      StringIO.new(PDF)
    else
      StringIO.new(PDF)
    end
  end

  def self.content_type_for(type)
    case type.to_sym
    when :png, :image, :avatar, :real_image then "image/png"
    else "application/pdf"
    end
  end

  def self.filename_for(type)
    case type.to_sym
    when :png, :image, :avatar, :real_image then "image.png"
    else "test.pdf"
    end
  end
end

6.3 Updated Factory

# spec/factories/attachments.rb
factory :attachment do
  after(:build) do |attachment|
    next if attachment.file.attached?

    file_type = case attachment.attachment_type
                when "image", "avatar", "real_image" then :image
                else :pdf
                end

    # In-memory file - no disk I/O!
    attachment.file.attach(
      io: MinimalFiles.io_for(file_type),
      filename: MinimalFiles.filename_for(file_type),
      content_type: MinimalFiles.content_type_for(file_type)
    )
  end
end

6.4 Why This Is Safe

Concerns we addressed:

“Will validations still work?”
- Yes! The files have valid headers (PNG signature, PDF structure)
- Content-type validation passes
“What about fixture_file_upload in request specs?”
- Unchanged! Request specs that test actual upload behavior still use fixture_file_upload
- Only factory-created attachments use in-memory files
“Is 67 bytes enough for a valid image?”
- Yes! It’s a valid 1x1 transparent PNG
- Any library reading it will parse it correctly

6.5 Performance Impact

Operation	Before	After	Improvement
Read fixture file	~0.5-2ms	~0.01ms	50-200x
Memory allocation	File buffer	String	Minimal
I/O operations	Disk read	None	Eliminated

7. Test Profiling Infrastructure

7.1 Adding test-prof Gem

The test-prof gem provides deep insights into test performance:

# Gemfile
group :test do
  gem "test-prof"
end

7.2 TestProf Configuration

# spec/support/test_prof.rb
if defined?(TestProf)
  # FactoryProf: Profile factory usage
  TestProf::FactoryProf.configure do |config|
    config.mode = :flamegraph if ENV["FPROF_FLAMEGRAPH"]
  end

  # EventProf: Profile SQL queries and factory creates
  TestProf::EventProf.configure do |config|
    config.per_example = true if ENV["EVENT_PROF_EXAMPLES"]
  end
end

# Native FactoryBot profiling (no TestProf required)
if ENV["FACTORY_PROF"]
  RSpec.configure do |config|
    factory_stats = Hash.new { |h, k| h[k] = { count: 0, time: 0.0 } }

    config.before(:suite) do
      ActiveSupport::Notifications.subscribe("factory_bot.run_factory") do |*args|
        event = ActiveSupport::Notifications::Event.new(*args)
        factory_stats[event.payload[:name]][:count] += 1
        factory_stats[event.payload[:name]][:time] += event.duration
      end
    end

    config.after(:suite) do
      puts "\n\n=== FactoryBot Profiling Report ==="
      puts "%-40s %10s %15s" % ["Factory", "Count", "Time (ms)"]
      puts "-" * 67

      factory_stats.sort_by { |_, v| -v[:time] }.first(20).each do |name, stats|
        puts "%-40s %10d %15.2f" % [name, stats[:count], stats[:time]]
      end
    end
  end
end

7.3 Profiling Commands

# Profile slow examples (show top 10)
PROFILE=1 bundle exec rspec

# Profile factory usage (find N+1 factories)
FPROF=1 bundle exec rspec

# Profile factory usage with flamegraph
FPROF=1 FPROF_FLAMEGRAPH=1 bundle exec rspec

# Profile SQL queries
EVENT_PROF=sql.active_record bundle exec rspec

# Profile factory create events
EVENT_PROF=factory.create bundle exec rspec

# Native factory profiling
FACTORY_PROF=1 bundle exec rspec

# Combined: slow examples + factory stats
PROFILE=1 FACTORY_PROF=1 bundle exec rspec

7.4 Sample Factory Profile Output

=== FactoryBot Profiling Report ===
Factory                                      Count       Time (ms)
-------------------------------------------------------------------
user                                           156         1234.56
profile                                        142          987.65
company                                         89          654.32
attachment                                      85          432.10
job_post                                        45          321.00
company_benefit                                 34          210.50
...
-------------------------------------------------------------------
TOTAL                                          892         5432.10

What to look for:

Factories with high count but low time → OK
Factories with low count but high time → Optimize factory
Factories called more than expected → Check for cascades

7.5 TestProf’s let_it_be

For specs that can share fixtures across examples:

# spec/rails_helper.rb
require 'test_prof/recipes/rspec/let_it_be'

# Usage in specs
RSpec.describe ProfileBlueprint do
  # Created once for ALL examples in this describe block
  let_it_be(:user) { create(:user) }
  let_it_be(:profile) { create(:profile, user: user) }

  it "serializes id" do
    # user and profile are reused, not recreated
    expect(described_class.render_as_hash(profile)).to include(id: profile.id)
  end

  it "serializes username" do
    # Same user and profile from above
    expect(described_class.render_as_hash(profile)).to include(username: profile.username)
  end
end

Use let_it_be when:

Fixtures are read-only in all examples
Examples don’t modify the records
You have many examples using the same data

8. RSpec Configuration Tuning

8.1 Updated spec_helper.rb

# spec/spec_helper.rb
RSpec.configure do |config|
  # Focus on tagged examples during development
  config.filter_run_when_matching :focus

  # Persist example status for --only-failures
  config.example_status_persistence_file_path = "spec/examples.txt"

  # Profile slow examples (enable via PROFILE=1)
  config.profile_examples = ENV["PROFILE"] ? 10 : false

  # Random order to surface order dependencies
  config.order = :random
  Kernel.srand config.seed
end

8.2 Key Configuration Choices

Setting	Value	Rationale
`profile_examples`	Conditional	Only profile when explicitly requested
`order`	`:random`	Surface hidden dependencies
`example_status_persistence`	Enabled	Support `--only-failures`
`filter_run_when_matching :focus`	Enabled	Focus on tagged tests during dev

8.3 .rspec File

--require spec_helper

Keep it minimal. Command-line options are easier to manage in CI.

9. Results and Metrics

9.1 Before vs After Comparison

Metric	Before	After	Improvement
Pipeline timeout	5 min	10 min	2x headroom
Parallel steps	2	3	50% more parallelism
Actual build time	5+ min (timeout)	~3 min	60% faster
`let!` calls	252	~150	40% reduction
Disk I/O per run	85+ file reads	0	Eliminated
Test output	Verbose	Progress	Less I/O
Fail behavior	Run all	Fail-fast	Faster feedback

9.2 Pipeline Execution Timeline

Before:

[Lint & Security] ████████░░░░░░░░░░ 2:00
                  [RSpec + Swagger] ██████████████████ 5:00+ TIMEOUT!

After:

[Lint & Security] ████████░░░░░░░░░░ 2:00
[Unit Tests]      ████████░░░░░░░░░░ 1:30
[Integration]     ████████████░░░░░░ 2:30
                           [Seeds]  ████░░░░░░░░░░░░░░ 0:30
─────────────────────────────────────────────────────────────
Total wall time: ~3:00 (limited by slowest parallel step)

9.3 What We Shipped

Component	Files Changed
Gemfile	Added `parallel_tests`, `test-prof`
bitbucket-pipelines.yml	Restructured to 3 parallel steps
spec/spec_helper.rb	Enabled profiling, focus filtering
spec/rails_helper.rb	Added TestProf recipes
spec/support/test_prof.rb	New profiling configuration
spec/support/minimal_files.rb	New in-memory file module
spec/factories/attachments.rb	Use MinimalFiles
spec/factories/gallery_images.rb	Use MinimalFiles
spec/integration/projects_spec.rb	Convert let! → let
spec/integration/working_knowledge_spec.rb	Convert let! → let
spec/support/shared_examples/*	New shared contexts
spec/support/shared_schemas/*	Extracted common schemas

10. Ongoing Optimization Strategy

10.1 Monitoring Commands

Run these periodically to identify new bottlenecks:

# Find slowest 10 examples
PROFILE=1 bundle exec rspec

# Find factory N+1 issues
FPROF=1 bundle exec rspec

# Identify SQL-heavy tests
EVENT_PROF=sql.active_record bundle exec rspec

# Check factory creation counts
FACTORY_PROF=1 bundle exec rspec

10.2 Guidelines for New Tests

Prefer let over let! unless record must pre-exist
Use shared contexts for repeated patterns
Keep integration specs focused — one endpoint per file if large
Use let_it_be for read-only fixtures in model/blueprint specs
Avoid factory cascades — use build_stubbed when possible

10.3 CI Pipeline Guidelines

Split by test type — Unit vs Integration
Use --fail-fast in CI — Fail fast, fix fast
Cache aggressively — Bundler, Bootsnap, build artifacts
Monitor build times — Set alerts for regression

10.4 Future Optimizations

If builds slow down again:

Add more parallel steps — Split integration by domain
Use parallel_tests within steps — Multi-process per step
Consider test splitting by timing — Distribute by historical runtime
Evaluate CI provider — GitHub Actions has better parallelization

Key Takeaways

Profile before optimizing — Measure, don’t guess
Parallelize at multiple levels — Pipeline steps + test processes
Lazy evaluation is your friend — let over let!
Eliminate I/O — In-memory beats disk every time
Fail fast — --fail-fast saves minutes on broken builds
Split wisely — Unit tests separate from integration
Cache everything — Bundler, Bootsnap, build artifacts
Document your profiling — Future you will thank present you

The Real Win

The 60% speed improvement is great, but the real win is confidence. Fast tests mean:

Developers run tests locally more often
CI catches issues before they merge
Deployments happen without anxiety
The test suite stays fast as the codebase grows

Slow tests become ignored tests. Ignored tests become broken tests. Broken tests become production bugs.

Invest in your test infrastructure. It pays dividends every single day.

Resources

parallel_tests: github.com/grosser/parallel_tests
test-prof: test-prof.evilmartians.io
RSpec Best Practices: betterspecs.org
FactoryBot Optimization: thoughtbot/factory_bot
Bitbucket Pipelines Reference: support.atlassian.com/bitbucket-cloud/docs/bitbucket-pipelines-configuration-reference
GitHub Actions for Rails: docs.github.com/actions

This post documents the optimization of the Wigiwork API test suite from timeout failures to sub-3-minute builds. The patterns described here—parallel execution, factory optimization, I/O elimination, and profiling infrastructure—are applicable to any Rails application with a growing test suite.

Turning Neverlands Into Design Fuel: A Browser Bot + GPT-5.1 Game Analyst

Sat, 06 Dec 2025 00:00:00 +0000

TL;DR

We built a browser bot + GPT-5.1 analyst that logs into neverlands.ru, captures real game screens, and turns them into structured JSON.

The goal is game design research, not automation: extract systems, loops, and UX patterns to feed your own GDD.

Everything runs locally: credentials live in .env, GPT only ever sees HTML snapshots, not your password.

The stack is small and explicit: Node 18 + Playwright + dotenv + OpenAI API, wired through a few focused TypeScript scripts.

Why build a bot for Neverlands?

Neverlands is a long-running browser MMORPG with mature systems: inventory, combat, economy, events, social layers. Instead of cloning it, we want to study it:

What progression and compulsion loops does it use?
How are shops, currencies, and sinks structured?
Which UI patterns make the game readable (or confusing)?

Doing this manually is slow. Screenshots pile up, notes get messy, and you miss subtle connections. The solution: a repeatable pipeline that:

Logs into your account.
Visits interesting screens (home, character, inventory, shop, PvP…).
Saves HTML + screenshots.
Asks GPT-5.1 to extract features and player needs into machine-readable JSON.

Those JSON files become the raw material for your game design document, where you remix Neverlands-inspired ideas into your own world.

Architecture at a glance

The repo is intentionally small:

bot/        # Playwright automation
  login.ts  # log in + save session
  crawl.ts  # visit screens + save HTML/PNG

analyzer/
  analyzeScreens.ts  # send HTML to GPT-5.1 using promt.md

data/
  session/   # Playwright storage state
  raw/       # captured HTML + screenshots
  analysis/  # GPT JSON outputs

promt.md     # "game design analyst" prompt template
doc/         # human docs (setup, usage, dev)

The heart of the system is promt.md: a prompt that tells GPT-5.1 to behave like a game design analyst. For each HTML snapshot it:

Identifies visible systems/features (e.g., daily quests, energy, shop widgets).
Describes how they seem to work from the UI.
Classifies the player need they target (progression, social, mastery, etc.).
Suggests how you might adapt the idea for your own MMORPG.

Outputs are strict JSON, so you can script on top of them later.

Technical implementation

bot/login.ts uses Playwright to open http://www.neverlands.ru/, fill the login form, and save a session:

Credentials come from .env via dotenv:
- NEVERLANDS_USERNAME, NEVERLANDS_PASSWORD.
Form selectors are configurable:
- NEVERLANDS_USERNAME_SELECTOR, NEVERLANDS_PASSWORD_SELECTOR, NEVERLANDS_SUBMIT_SELECTOR.
The URL is normalized so any https://neverlands.ru is rewritten to http://www.neverlands.ru/.
On success, storage state is written to data/session/neverlands.json.

You run it with:

npm run login

2. Crawling key game screens

bot/crawl.ts reuses the saved session to open specific URLs and capture HTML + PNG:

Screens are defined as a small list of { id, url }.
For each screen, we:
- page.goto(url, { waitUntil: 'networkidle' }).
- Save DOM: data/raw/<id>.html.
- Save screenshot: data/raw/<id>.png.

You can start with home, then add pages like inventory, character, shop, or any internal route Neverlands exposes.

Run the crawl:

npm run crawl

3. GPT-5.1 analysis

analyzer/analyzeScreens.ts reads every *.html file in data/raw/, then calls the OpenAI Chat Completions API:

Uses OPENAI_API_KEY from .env.
Defaults to OPENAI_MODEL=gpt-5.1 (configurable).
Loads promt.md as the system message.
Sends the HTML as the user message.
Requests response_format: json_object to keep outputs parseable.

Each result is saved as data/analysis/<screen>-<hash>.json, giving you versioned feature maps per screen.

Run analysis:

npm run analyze

From JSON to a game design document

Once you have JSON outputs, you’re holding a Neverlands feature index. Typical next steps:

Group features by system: combat, economy, social, events, meta.
Identify loops: daily quests → currencies → shops → upgrades.
Tag ideas as keep / adapt / avoid for your own game.

Because the analysis is structured, it’s easy to:

Generate Markdown “inspiration cards” per feature.
Build tables comparing Neverlands to other games you crawl later.
Feed the JSON back into another GPT-5.1 pass to draft full sections of your GDD.

The bot doesn’t design the game for you—but it amplifies your research so you can spend time on synthesis, not screenshot wrangling.

Safety, ToS, and scope

Even though neverlands.ru allows bots, we built this stack with a conservative mindset:

Credentials live only in .env on your machine.
GPT never sees your password or cookies, only HTML.
The intended use is analysis, not auto-play or farming.
Screen lists are small and intentional, not full-site crawlers.

If you adapt this to other games, always re-check each game’s ToS and keep your rate of requests reasonable.

Where to go next

With the core pipeline in place, the most interesting work is ahead:

Add more screens and flows (e.g., onboarding, high-level PvP, events).
Enrich promt.md with new fields (risk flags, UX pain points, monetization patterns).
Build a small UI or static site to browse the analyzed features.
Compare Neverlands against other browser MMORPGs using the same pipeline.

The end goal: a living game design document for your own project, grounded in real, battle-tested systems—but with your own take on what makes them fun.

From Rails Spaghetti to Structured Code: Integrating dry-rb into Your API

Thu, 04 Dec 2025 00:00:00 +0000

“The best code isn’t clever—it’s obvious. dry-rb makes Rails code obvious.”

Every Rails developer has written code that works but feels wrong. Manual type coercion scattered across controllers. Validation logic duplicated between models and services. Deeply nested conditionals handling success and failure cases. Configuration constants scattered across files. Side effects mixed with business logic. These patterns work, but they accumulate into technical debt.

After shipping dozens of API endpoints, we integrated dry-rb into our Rails API. This post documents the transformation—with real before/after examples showing how each gem improves specific code patterns.

The Problem: Rails Patterns That Don’t Scale
The Solution: dry-rb Gem Suite
dry-types: Type-Safe Enums and Coercion
dry-struct: Immutable Data Structures
dry-monads: Functional Error Handling
dry-validation: Contract-Based Validation
dry-container + dry-auto_inject: Dependency Injection
dry-matcher: Pattern Matching for Results
dry-configurable: Thread-Safe Configuration
dry-initializer: Declarative Parameter DSL
dry-events: Pub/Sub Event System
Other dry-rb Gems Worth Exploring
Benefits and Tradeoffs
Conclusion

1. The Problem: Rails Patterns That Don’t Scale

1.1 The “Rails Way” Has Limits

Rails conventions are excellent for getting started quickly. But as applications grow, certain patterns become problematic:

Pattern	Works For	Breaks When
Hash params everywhere	Simple CRUD	Complex nested data structures
ActiveRecord validations	Model-level rules	Cross-field business logic
Controller conditionals	Simple flows	Multiple success/failure paths
Manual type coercion	Occasional use	Dozens of fields need processing
Class method calls	Small apps	Testing requires stubbing globals
Scattered constants	One-off values	Configuration needs structure
Inline side effects	Simple actions	Need audit trail or event-driven architecture

1.2 Real Example: Profile Actions Payload

We needed to serialize client-specific profile data. Here’s what the original code looked like:

# Original: 70+ lines of manual validation and coercion
def build_actions_payload(options)
  actions = extract_actions_data(options)
  result = {}

  # Manual enum validation
  status = actions[:access_status].to_s
  result[:access_status] = ACCESS_STATUS_VALUES.include?(status) ? status : "none"

  # Manual boolean coercion
  result[:profile_viewed] = actions[:profile_viewed] == true
  result[:is_saved] = actions[:is_saved] == true

  # Manual conditional inclusion
  if actions[:thumb_status].present?
    thumb = actions[:thumb_status].to_s
    result[:thumb_status] = thumb if THUMB_STATUS_VALUES.include?(thumb)
  end

  if actions[:applied_at].present?
    value = actions[:applied_at]
    result[:applied_at] = value.respond_to?(:iso8601) ? value.iso8601 : value.to_s
  end

  if actions.key?(:access_expired)
    result[:access_expired] = actions[:access_expired] == true
  end

  if actions[:access_end_reason].present?
    reason = actions[:access_end_reason].to_s
    result[:access_end_reason] = reason if ACCESS_END_REASON_VALUES.include?(reason)
  end

  # ... 30 more lines
  result
end

Problems:

❌ Manual type coercion (to_s, == true) repeated everywhere
❌ Validation scattered (constants + inline checks)
❌ Hard to test—tightly coupled to Blueprint
❌ No type safety—wrong data silently coerced
❌ Verbose—business logic drowns in boilerplate

2. The Solution: dry-rb Gem Suite

The dry-rb ecosystem provides focused, composable gems for common problems:

Gem	Purpose	Replaces
dry-types	Type definitions with constraints	Manual enum validation
dry-struct	Immutable typed objects	Hash manipulation
dry-monads	Functional result handling	try/rescue + conditionals
dry-validation	Contract-based validation	Scattered validation logic
dry-schema	Schema coercion	Strong parameters edge cases
dry-container	Dependency registry	Global class references
dry-auto_inject	Constructor injection	Manual dependency passing
dry-matcher	Result pattern matching	Case statements on results
dry-configurable	Thread-safe configuration	Scattered constants
dry-initializer	Param/option DSL	Boilerplate initialize methods
dry-events	Pub/sub event system	Inline side effects

Our Gemfile Addition

# Gemfile
# Dry-rb gems for structured types, validation, and functional patterns
gem "dry-types", "~> 1.7"
gem "dry-struct", "~> 1.6"
gem "dry-monads", "~> 1.6"
gem "dry-validation", "~> 1.11"
gem "dry-schema", "~> 1.14"
gem "dry-container", "~> 0.11"
gem "dry-auto_inject", "~> 0.9"
gem "dry-matcher", "~> 1.0"
gem "dry-configurable", "~> 1.2"
gem "dry-initializer", "~> 3.1"
gem "dry-events", "~> 1.0"

3. dry-types: Type-Safe Enums and Coercion

3.1 The Problem: Scattered Constants and Manual Checks

# Before: Constants scattered, validation repeated
class ProfileBlueprint < ApplicationBlueprint
  ACCESS_STATUS_VALUES = %w[none requested approved denied removed shared].freeze
  THUMB_STATUS_VALUES = %w[liked disliked].freeze
  ACCESS_END_REASON_VALUES = %w[expired declined blocked removed].freeze

  def build_actions_payload(options)
    status = actions[:access_status].to_s
    result[:access_status] = ACCESS_STATUS_VALUES.include?(status) ? status : "none"
    # Repeated for every enum field...
  end
end

Issues:

Constants defined in one class, used elsewhere
Validation logic duplicated wherever enums are checked
No automatic coercion or defaults

3.2 The Solution: Centralized Type Registry

# app/types.rb
require "dry/types"

module Types
  include Dry.Types()

  # Common types with safe defaults
  SafeBool = Types::Bool.fallback(false)
  OptionalString = Types::String.optional
  OptionalInteger = Types::Integer.optional

  # ISO8601 timestamp coercion
  ISO8601String = Types::String.optional.constructor do |value|
    next nil if value.nil?
    case value
    when Time, DateTime, ActiveSupport::TimeWithZone
      value.iso8601
    else
      value.to_s
    end
  end

  # Profile Actions enums - SINGLE SOURCE OF TRUTH
  module Profiles
    ACCESS_STATUS_VALUES = %w[none requested approved denied removed shared].freeze
    AccessStatus = Types::String.enum(*ACCESS_STATUS_VALUES).fallback("none")

    THUMB_STATUS_VALUES = %w[liked disliked].freeze
    ThumbStatus = Types::String.enum(*THUMB_STATUS_VALUES).optional

    ACCESS_END_REASON_VALUES = %w[expired declined blocked removed].freeze
    AccessEndReason = Types::String.enum(*ACCESS_END_REASON_VALUES).optional
  end
end

Benefits:

✅ Enums defined once, reusable everywhere
✅ Built-in validation (invalid values → constraint error or fallback)
✅ Automatic coercion with .constructor
✅ Safe defaults with .fallback()
✅ Self-documenting—type definitions are specifications

3.3 Usage Examples

# Enum validation is automatic
Types::Profiles::AccessStatus["approved"]  # => "approved"
Types::Profiles::AccessStatus["invalid"]   # => "none" (fallback)
Types::Profiles::AccessStatus[nil]         # => "none" (fallback)

# Boolean coercion with fallback
Types::SafeBool[true]   # => true
Types::SafeBool[nil]    # => false (fallback, not nil!)

# Timestamp coercion
Types::ISO8601String[Time.current]           # => "2025-12-04T09:30:00Z"
Types::ISO8601String["2025-12-04T09:30:00Z"] # => "2025-12-04T09:30:00Z"
Types::ISO8601String[nil]                    # => nil

4. dry-struct: Immutable Data Structures

4.1 The Problem: Hash Manipulation Everywhere

# Before: Raw hash with manual processing
def build_actions
  actions = {}
  actions[:access_status] = compute_access_status
  actions[:profile_viewed] = compute_profile_viewed
  actions[:is_saved] = compute_is_saved
  # No guarantee of structure
  # Caller must know what keys exist
  # Mutable—anyone can modify
  actions
end

Issues:

No schema—structure discovered at runtime
Mutable—data can change unexpectedly
Manual coercion needed when consuming
No IDE autocompletion

4.2 The Solution: Typed Immutable Struct

# app/structs/profiles/actions.rb
require "dry/struct"

module Profiles
  class Actions < Dry::Struct
    # Allow string keys from input hash
    transform_keys(&:to_sym)

    # Required fields with defaults (always present in payload)
    attribute? :access_status, Types::Profiles::AccessStatus.default("none".freeze)
    attribute? :profile_viewed, Types::SafeBool.default(false)
    attribute? :is_saved, Types::SafeBool.default(false)

    # Optional fields (only in payload when present)
    attribute? :note, Types::OptionalString
    attribute? :thumb_status, Types::Profiles::ThumbStatus
    attribute? :applied_at, Types::ISO8601String
    attribute? :access_expired, Types::Bool.optional
    attribute? :access_end_reason, Types::Profiles::AccessEndReason
    attribute? :chat_id, Types::OptionalInteger
    attribute? :is_shared, Types::Bool.optional

    # Clean serialization for API response
    def to_payload
      payload = {
        access_status: access_status,
        profile_viewed: profile_viewed,
        is_saved: is_saved
      }

      # Add optional fields only when present
      payload[:note] = note if note.present?
      payload[:thumb_status] = thumb_status if thumb_status.present?
      payload[:applied_at] = applied_at if applied_at.present?
      payload[:access_expired] = access_expired unless access_expired.nil?
      payload[:access_end_reason] = access_end_reason if access_end_reason.present?
      payload[:chat_id] = chat_id if chat_id.present?
      payload[:is_shared] = true if is_shared == true

      payload
    end
  end
end

4.3 Before vs After Comparison

Before (manual hash building):

# 70+ lines scattered across service and blueprint
def build_actions_payload(options)
  actions = extract_actions_data(options)
  result = {}
  status = actions[:access_status].to_s
  result[:access_status] = ACCESS_STATUS_VALUES.include?(status) ? status : "none"
  result[:profile_viewed] = actions[:profile_viewed] == true
  # ... 60 more lines of manual processing
end

After (struct instantiation):

# Service creates struct
actions = Profiles::Actions.new(
  access_status: compute_access_status,
  profile_viewed: compute_profile_viewed,
  is_saved: compute_is_saved,
  chat_id: find_chat_room&.id
)

# Blueprint just calls to_payload
field :actions do |_profile, options|
  options.dig(:context, :actions).to_payload
end

Reduction: 70+ lines → ~15 lines

4.4 Benefits of dry-struct

Aspect	Raw Hash	Dry::Struct
Schema	Implicit	Explicit, documented
Type safety	None	Enforced at construction
Defaults	Manual	Declarative
Mutability	Mutable	Immutable
IDE support	None	Autocomplete attributes
Testability	Test whole flow	Test struct in isolation

5. dry-monads: Functional Error Handling

5.1 The Problem: Nested Conditionals

# Before: Conditional soup
def call
  return nil unless profile.present?
  return nil unless client_user.present?
  return nil unless client_user.client?
  return nil unless profile_user.present?

  # Happy path finally...
  build_actions
rescue StandardError => e
  Rails.logger.error(e)
  nil
end

Issues:

Early returns obscure happy path
Error information lost (just returns nil)
Caller can’t distinguish “no data” from “error”
No composability

5.2 The Solution: Result Monad

# app/services/profiles/actions_builder.rb
require "dry/monads"

module Profiles
  class ActionsBuilder
    include Dry::Monads[:result]

    def call
      return Success(default_actions_struct) unless valid_context?

      Success(build_actions_struct)
    end

    private

    def valid_context?
      profile.present? &&
        client_user.present? &&
        client_user.client? &&
        profile_user.present?
    end

    def default_actions_struct
      Profiles::Actions.new(
        access_status: "none",
        profile_viewed: false,
        is_saved: false
      )
    end

    def build_actions_struct
      Profiles::Actions.new(
        access_status: compute_access_status,
        profile_viewed: compute_profile_viewed,
        # ... other fields
      )
    end
  end
end

5.3 Controller Integration

# app/controllers/api/v1/profiles_controller.rb
def show
  flags = visibility_flags.get(@profile)
  context = { visibility_flags: flags, host: request.base_url }

  if current_user&.client?
    result = actions_builder.call(profile: @profile, client_user: current_user)
    context[:actions] = result.value! if result.success?
  end

  render json: { data: ProfileBlueprint.render_as_hash(@profile, context: context) }
end

5.4 Pattern Matching with Results

# For more complex flows, pattern match on Result
result = ActionsBuilder.call(profile: @profile, client_user: current_user)

case result
in Success(actions)
  render_success(actions.to_payload)
in Failure[:invalid_context, reason]
  Rails.logger.warn("Invalid context: #{reason}")
  render_default_actions
in Failure[:database_error, exception]
  Sentry.capture_exception(exception)
  head :internal_server_error
end

5.5 Composing Multiple Operations

# Chain operations that might fail
class ProcessOrder
  include Dry::Monads[:result, :do]

  def call(order_params)
    order = yield validate_order(order_params)
    user = yield find_user(order.user_id)
    payment = yield charge_payment(user, order.total)
    yield send_confirmation(user, order, payment)

    Success(order)
  end

  private

  def validate_order(params)
    # Returns Success(order) or Failure([:validation, errors])
  end

  def charge_payment(user, amount)
    # Returns Success(payment) or Failure([:payment_failed, reason])
  end
end

The yield keyword automatically unwraps Success values and short-circuits on Failure.

6. dry-validation: Contract-Based Validation

6.1 The Problem: Validation Logic Everywhere

# Before: Validation scattered across controller, model, and service
class ItemsController < ApplicationController
  def create
    # Controller validation
    return render_error("Title required") if params[:title].blank?
    return render_error("Invalid status") unless VALID_STATUSES.include?(params[:status])

    @item = Item.new(item_params)
    # Model validation runs here too...
  end
end

class Item < ApplicationRecord
  # Model validation
  validates :title, presence: true
  validates :status, inclusion: { in: VALID_STATUSES }
  validate :price_must_be_positive_for_published
end

Issues:

Validation logic duplicated (controller + model)
Cross-field rules awkward in ActiveRecord
Hard to test validation in isolation
Error messages inconsistent

6.2 The Solution: Validation Contracts

# app/contracts/profiles/actions_contract.rb
require "dry/validation"

module Profiles
  class ActionsContract < Dry::Validation::Contract
    # Schema definition - coerces and validates structure
    params do
      optional(:access_status).filled(:string)
      optional(:profile_viewed).filled(:bool)
      optional(:is_saved).filled(:bool)
      optional(:note).maybe(:string)
      optional(:thumb_status).maybe(:string)
      optional(:access_expired).maybe(:bool)
      optional(:access_end_reason).maybe(:string)
      optional(:chat_id).maybe(:integer)
    end

    # Enum validation rules
    rule(:access_status) do
      next if value.nil?
      unless Types::Profiles::ACCESS_STATUS_VALUES.include?(value)
        key.failure("must be one of: #{Types::Profiles::ACCESS_STATUS_VALUES.join(', ')}")
      end
    end

    rule(:thumb_status) do
      next if value.nil?
      unless Types::Profiles::THUMB_STATUS_VALUES.include?(value)
        key.failure("must be one of: #{Types::Profiles::THUMB_STATUS_VALUES.join(', ')}")
      end
    end

    # Cross-field validation
    rule(:access_end_reason, :access_expired) do
      if values[:access_end_reason].present? && values[:access_expired] != true
        key(:access_end_reason).failure("requires access_expired to be true")
      end
    end
  end
end

6.3 Using Contracts

contract = Profiles::ActionsContract.new
result = contract.call(params.to_unsafe_h)

if result.success?
  # Validated and coerced data
  actions = Profiles::Actions.new(result.to_h)
else
  # Structured errors
  render json: { errors: result.errors.to_h }, status: :unprocessable_entity
end

6.4 Contract vs Model Validation

Aspect	ActiveRecord Validation	Dry::Validation Contract
When runs	Before save	Before touching model
Cross-field rules	Custom validators	Declarative `rule` blocks
Testability	Requires model instance	Pure Ruby, no DB
Reusability	Tied to model	Standalone class
Error format	Rails format	Customizable
Coercion	Limited	Built-in via schema

Use both: Contracts for input validation (API layer), ActiveRecord for data integrity (persistence layer).

7. dry-container + dry-auto_inject: Dependency Injection

7.1 The Problem: Hard-Coded Dependencies

# Before: Tight coupling to class names
class ProfilesController < ApplicationController
  def show
    flags = Visibility::ProfileVisibilityFlags.get(@profile)
    result = Profiles::ActionsBuilder.call(profile: @profile, client_user: current_user)
    # ...
  end
end

# Testing requires stubbing constants
RSpec.describe ProfilesController do
  before do
    allow(Visibility::ProfileVisibilityFlags).to receive(:get).and_return({})
    allow(Profiles::ActionsBuilder).to receive(:call).and_return(Success({}))
  end
end

Issues:

Can’t easily swap implementations
Testing requires monkey-patching
Dependencies hidden in method bodies
No central registry of services

7.2 The Solution: Container + Injection

# app/container.rb
require "dry/container"
require "dry/auto_inject"

class AppContainer
  extend Dry::Container::Mixin

  # Register services
  register(:actions_builder) { Profiles::ActionsBuilder }
  register(:visibility_flags) { Visibility::ProfileVisibilityFlags }

  # Future additions
  # register(:email_service) { EmailService.new }
  # register(:payment_gateway) { Stripe::Client.new(ENV['STRIPE_KEY']) }
end

# For POROs (constructor injection)
Import = Dry::AutoInject(AppContainer)

# For Rails controllers (memoized accessors)
module Deps
  def self.included(base)
    base.class_eval do
      AppContainer.keys.each do |key|
        define_method(key) do
          @_deps_cache ||= {}
          @_deps_cache[key] ||= AppContainer[key]
        end
      end
    end
  end
end

7.3 Controller with Injection

# app/controllers/api/v1/profiles_controller.rb
class ProfilesController < ApplicationController
  include Deps  # Inject all registered dependencies

  def show
    flags = visibility_flags.get(@profile)           # Use injected service
    result = actions_builder.call(...)               # Use injected builder
    # ...
  end
end

7.4 Testing with Injection

# Spec: Easy stubbing via dependency override
RSpec.describe ProfilesController do
  let(:mock_builder) { double("ActionsBuilder") }
  let(:mock_flags) { double("VisibilityFlags") }

  before do
    # Override container registrations for test
    AppContainer.stub(:actions_builder, mock_builder)
    AppContainer.stub(:visibility_flags, mock_flags)
  end

  it "uses injected dependencies" do
    expect(mock_flags).to receive(:get).and_return({})
    expect(mock_builder).to receive(:call).and_return(Success(default_actions))

    get :show, params: { id: profile.id }
  end
end

7.5 Benefits of Dependency Injection

Without DI	With DI
`ClassName.call(...)` scattered	Centralized registry
Test with `allow(ClassName)`	Test with mock injection
Change implementation = change every caller	Change registration only
Dependencies implicit	Dependencies explicit

8. dry-matcher: Pattern Matching for Results

8.1 The Problem: Verbose Result Handling

# Before: Manual success/failure checks
def show
  result = actions_builder.call(profile: @profile, client_user: current_user)

  if result.success?
    context[:actions] = result.value!
  else
    case result.failure
    when :not_found
      head :not_found
    when :unauthorized
      head :unauthorized
    else
      render json: { error: result.failure }, status: :unprocessable_entity
    end
  end
end

Issues:

Verbose conditional logic
Easy to forget failure cases
Not composable
Repeated patterns across controllers

8.2 The Solution: Pattern Matching Concern

# app/controllers/concerns/result_matchable.rb
require "dry/matcher/result_matcher"

module ResultMatchable
  extend ActiveSupport::Concern

  included do
    class_attribute :result_matcher, default: Dry::Matcher::ResultMatcher
  end

  # Match on a Dry::Monads::Result with pattern matching
  def match_result(result, &block)
    result_matcher.call(result, &block)
  end

  # Extract value from Success or return nil for Failure
  def unwrap_result(result)
    result.success? ? result.value! : nil
  end

  # Returns [success?, value_or_error]
  def result_tuple(result)
    if result.success?
      [true, result.value!]
    else
      [false, result.failure]
    end
  end
end

8.3 Controller with Pattern Matching

# app/controllers/api/v1/profiles_controller.rb
class ProfilesController < ApplicationController
  include ResultMatchable  # Add pattern matching

  def show
    result = actions_builder.call(profile: @profile, client_user: current_user)

    # Simple extraction
    context[:actions] = unwrap_result(result)

    # Or full pattern matching for complex flows
    match_result(result) do |m|
      m.success { |actions| context[:actions] = actions }
      m.failure(:not_found) { head :not_found; return }
      m.failure(:unauthorized) { head :unauthorized; return }
      m.failure { |error| Rails.logger.warn("Actions failed: #{error}") }
    end

    render json: { data: ProfileBlueprint.render_as_hash(@profile, context: context) }
  end
end

8.4 Before vs After

Before:

# 15 lines of conditional logic
if result.success?
  context[:actions] = result.value!
else
  case result.failure
  when :not_found then head :not_found
  when :unauthorized then head :unauthorized
  else render_error(result.failure)
  end
end

After:

# 1 line for simple cases
context[:actions] = unwrap_result(result)

# Or declarative matching for complex cases
match_result(result) do |m|
  m.success { |v| context[:actions] = v }
  m.failure(:not_found) { head :not_found }
  m.failure { |e| render_error(e) }
end

8.5 Benefits

Without dry-matcher	With dry-matcher
`if/case` statements	Declarative blocks
Easy to miss cases	Exhaustive matching
Scattered handling	Centralized concern
Hard to test branches	Each handler testable

9. dry-configurable: Thread-Safe Configuration

9.1 The Problem: Scattered Constants

# Before: Constants scattered across files
class ProfileBlueprint < ApplicationBlueprint
  ACCESS_STATUS_VALUES = %w[none requested approved denied removed shared].freeze
  CACHE_TTL = 5.minutes
end

class AccessRequest < ApplicationRecord
  EXPIRY_DAYS = 30
end

class ProfilesController < ApplicationController
  MAX_PER_PAGE = 100
end

Issues:

Configuration scattered across classes
Hard to find all config values
No nesting or organization
Testing requires constant stubbing
Not thread-safe for dynamic changes

9.2 The Solution: Centralized Configuration

# app/config/profile_config.rb
require "dry/configurable"

class ProfileConfig
  extend Dry::Configurable

  # Actions settings
  setting :actions do
    setting :cache_ttl, default: 5.minutes
    setting :default_access_status, default: "none"
    setting :default_profile_viewed, default: false
    setting :default_is_saved, default: false

    setting :access_status_values, default: %w[none requested approved denied removed shared].freeze
    setting :thumb_status_values, default: %w[liked disliked].freeze
    setting :access_end_reason_values, default: %w[expired declined blocked removed].freeze

    setting :status_mapping, default: {
      "pending" => "requested",
      "approved" => "approved",
      "declined" => "denied",
      "blocked" => "denied",
      "expired" => "removed"
    }.freeze
  end

  # Visibility settings
  setting :visibility do
    setting :default_mode, default: "visible"
    setting :modes, default: %w[visible semi_anonymous fully_anonymous].freeze
  end

  # Access request settings
  setting :access do
    setting :request_expiry_days, default: 30
    setting :auto_approve_verified_companies, default: false
    setting :max_pending_per_company, default: 100
  end

  # Event publishing settings
  setting :events do
    setting :publish_profile_views, default: true
    setting :publish_access_events, default: true
  end
end

9.3 Usage in Code

# Access nested settings with dot notation
ProfileConfig.config.actions.cache_ttl           # => 5.minutes
ProfileConfig.config.actions.status_mapping      # => {"pending" => "requested", ...}
ProfileConfig.config.visibility.default_mode     # => "visible"
ProfileConfig.config.access.request_expiry_days  # => 30

# Use in service objects
class ActionsBuilder
  def default_actions_struct
    Profiles::Actions.new(
      access_status: config.default_access_status,
      profile_viewed: config.default_profile_viewed,
      is_saved: config.default_is_saved
    )
  end

  private

  def config
    ProfileConfig.config.actions
  end
end

9.4 Testing with Configuration

RSpec.describe ActionsBuilder do
  # Override config for test
  before do
    allow(ProfileConfig.config.actions).to receive(:cache_ttl).and_return(0)
  end

  # Or use configure block
  around do |example|
    original = ProfileConfig.config.actions.cache_ttl
    ProfileConfig.configure { |c| c.actions.cache_ttl = 0 }
    example.run
    ProfileConfig.configure { |c| c.actions.cache_ttl = original }
  end
end

9.5 Before vs After

Before:

# Hunt through 10 files to find all profile-related constants
grep -r "CACHE_TTL\|EXPIRY_DAYS\|MAX_" app/

After:

# One file, organized by domain
ProfileConfig.config  # => nested configuration tree

9.6 Benefits

Scattered Constants	dry-configurable
Find: grep across codebase	Find: one file
Organize: N/A	Organize: nested settings
Test: stub constants	Test: configure block
Thread-safe: No	Thread-safe: Yes
Document: comments	Document: structure IS docs

10. dry-initializer: Declarative Parameter DSL

10.1 The Problem: Boilerplate Initialize Methods

# Before: Manual parameter handling
class ActionsBuilder
  def initialize(profile:, client_user:)
    @profile = profile
    @client_user = client_user
    @profile_user = profile&.user
  end

  def self.call(profile:, client_user:)
    new(profile: profile, client_user: client_user).call
  end

  private

  attr_reader :profile, :client_user, :profile_user
end

Issues:

Boilerplate @x = x assignments
Manual attr_reader for each param
No type coercion
No default values
self.call pattern repeated everywhere

10.2 The Solution: Declarative Parameters

# app/services/profiles/actions_builder.rb
require "dry/initializer"

module Profiles
  class ActionsBuilder
    extend Dry::Initializer

    # Required parameters
    param :profile
    param :client_user

    # Derived option with default
    option :profile_user, default: proc { profile&.user }

    def self.call(profile:, client_user:)
      new(profile, client_user).call
    end

    def call
      return Success(default_actions_struct) unless valid_context?
      Success(build_actions_struct)
    end

    # params are accessible directly - no attr_reader needed
    private

    def valid_context?
      profile.present? && client_user.present? && client_user.client?
    end
  end
end

10.3 Advanced Features

class OrderProcessor
  extend Dry::Initializer

  # Required param with type coercion
  param :order_id, Types::Integer

  # Optional param with default
  param :notify, default: proc { true }

  # Named options (keyword args)
  option :logger, default: proc { Rails.logger }
  option :mailer, default: proc { OrderMailer }

  # Type-checked option
  option :priority, Types::String.enum("low", "normal", "high"), default: proc { "normal" }

  # Reader visibility
  option :api_key, reader: :private

  def call
    logger.info("Processing order #{order_id} with priority #{priority}")
    # ...
  end
end

# Usage
processor = OrderProcessor.new(123, true, priority: "high")
processor.order_id  # => 123
processor.priority  # => "high"

10.4 Before vs After

Before (19 lines):

class ActionsBuilder
  def initialize(profile:, client_user:)
    @profile = profile
    @client_user = client_user
    @profile_user = profile&.user
  end

  def self.call(profile:, client_user:)
    new(profile: profile, client_user: client_user).call
  end

  private

  attr_reader :profile, :client_user, :profile_user

  def call
    # ...
  end
end

After (10 lines):

class ActionsBuilder
  extend Dry::Initializer

  param :profile
  param :client_user
  option :profile_user, default: proc { profile&.user }

  def self.call(profile:, client_user:)
    new(profile, client_user).call
  end

  def call
    # ...
  end
end

10.5 Benefits

Manual Initialize	dry-initializer
`@x = x` boilerplate	Declarative `param`/`option`
Manual `attr_reader`	Auto-generated readers
No type coercion	Built-in type support
No defaults DSL	`default: proc { }`
All public readers	`reader: :private` option

11. dry-events: Pub/Sub Event System

11.1 The Problem: Inline Side Effects

# Before: Side effects mixed with business logic
class ProfilesController < ApplicationController
  def show
    @profile = Profile.find(params[:id])

    # Business logic
    context = build_context(@profile)

    # Side effect #1: Analytics
    Analytics.track("profile_viewed", {
      profile_id: @profile.id,
      viewer_id: current_user&.id
    })

    # Side effect #2: Logging
    AuditLog.create!(
      action: "profile_view",
      resource: @profile,
      user: current_user
    )

    # Side effect #3: Cache warming
    ProfileCacheWarmer.perform_async(@profile.id) if should_warm_cache?

    render json: { data: ProfileBlueprint.render_as_hash(@profile, context: context) }
  end
end

Issues:

Controller bloated with side effects
Hard to test business logic in isolation
Adding new side effects = modifying controller
Side effects tightly coupled to triggering code
No audit trail of “what events exist”

11.2 The Solution: Event Publisher

# app/events/publisher.rb
require "dry/events"

class EventPublisher
  include Dry::Events::Publisher[:app]

  # Profile events
  register_event("profile.viewed")
  register_event("profile.actions_computed")

  # Access request events
  register_event("access.requested")
  register_event("access.approved")
  register_event("access.declined")

  # Chat events
  register_event("chat.room_accessed")

  class << self
    def instance
      @instance ||= new
    end

    def publish(event_name, payload = {})
      return unless should_publish?(event_name)

      instance.publish(event_name, payload.merge(published_at: Time.current))
    end

    def subscribe(listener_or_event, &block)
      if block_given?
        instance.subscribe(listener_or_event, &block)
      else
        instance.subscribe(listener_or_event)
      end
    end

    private

    def should_publish?(event_name)
      return false unless ProfileConfig.config.events.publish_profile_views if event_name.start_with?("profile.")
      return false unless ProfileConfig.config.events.publish_access_events if event_name.start_with?("access.")
      true
    end
  end
end

11.3 Publishing Events

# app/controllers/api/v1/profiles_controller.rb
class ProfilesController < ApplicationController
  def show
    @profile = Profile.find(params[:id])
    context = build_context(@profile)

    # Publish event - side effects handled by subscribers
    EventPublisher.publish("profile.viewed", {
      profile_id: @profile.id,
      viewer_id: current_user&.id,
      viewer_role: current_user&.role
    })

    render json: { data: ProfileBlueprint.render_as_hash(@profile, context: context) }
  end
end

# app/services/profiles/actions_builder.rb
class ActionsBuilder
  def call
    actions = build_actions_struct

    # Publish event with computed data
    EventPublisher.publish("profile.actions_computed", {
      profile_id: profile.id,
      client_user_id: client_user.id,
      access_status: actions.access_status,
      has_chat: actions.chat_id.present?
    })

    Success(actions)
  end
end

11.4 Subscribing to Events

# config/initializers/event_subscribers.rb

# Block-based subscriber
EventPublisher.subscribe("profile.viewed") do |event|
  Analytics.track("profile_view", event.payload)
end

# Object-based subscriber
class AuditListener
  def on_profile_viewed(event)
    AuditLog.create!(
      action: "profile_view",
      resource_type: "Profile",
      resource_id: event.payload[:profile_id],
      user_id: event.payload[:viewer_id],
      occurred_at: event.payload[:published_at]
    )
  end

  def on_access_requested(event)
    AuditLog.create!(action: "access_request", **event.payload)
  end

  def on_access_approved(event)
    AuditLog.create!(action: "access_approval", **event.payload)
  end
end

EventPublisher.subscribe(AuditListener.new)

# Conditional subscriber
class CacheWarmerListener
  def on_profile_viewed(event)
    return unless should_warm_cache?(event.payload[:profile_id])

    ProfileCacheWarmer.perform_async(event.payload[:profile_id])
  end

  private

  def should_warm_cache?(profile_id)
    # Logic to determine if cache should be warmed
  end
end

EventPublisher.subscribe(CacheWarmerListener.new)

11.5 Before vs After

Before (controller with inline side effects):

def show
  @profile = Profile.find(params[:id])

  # Side effect #1
  Analytics.track("profile_viewed", { profile_id: @profile.id })

  # Side effect #2
  AuditLog.create!(action: "profile_view", resource: @profile)

  # Side effect #3
  ProfileCacheWarmer.perform_async(@profile.id)

  render json: { data: ProfileBlueprint.render_as_hash(@profile) }
end

After (clean controller, events handle side effects):

def show
  @profile = Profile.find(params[:id])

  EventPublisher.publish("profile.viewed", {
    profile_id: @profile.id,
    viewer_id: current_user&.id
  })

  render json: { data: ProfileBlueprint.render_as_hash(@profile) }
end

11.6 Testing Events

RSpec.describe ProfilesController do
  describe "GET #show" do
    it "publishes profile.viewed event" do
      expect(EventPublisher).to receive(:publish).with(
        "profile.viewed",
        hash_including(profile_id: profile.id, viewer_id: user.id)
      )

      get :show, params: { id: profile.id }
    end
  end
end

RSpec.describe AuditListener do
  describe "#on_profile_viewed" do
    let(:event) { double(payload: { profile_id: 1, viewer_id: 2, published_at: Time.current }) }

    it "creates audit log entry" do
      expect { subject.on_profile_viewed(event) }.to change(AuditLog, :count).by(1)
    end
  end
end

11.7 Benefits

Inline Side Effects	dry-events
Mixed with business logic	Separated via pub/sub
Hard to test in isolation	Each listener testable
Adding effects = modify caller	Adding effects = new subscriber
No event catalog	`register_event` documents all
Tight coupling	Loose coupling
Synchronous only	Async-ready (queue subscribers)

12. Other dry-rb Gems Worth Exploring

12.1 dry-transaction: Multi-Step Business Operations

Perfect for complex workflows like order processing or user registration:

# app/transactions/create_order.rb
class CreateOrder
  include Dry::Transaction

  step :validate
  step :create_order
  step :charge_payment
  step :send_confirmation

  private

  def validate(input)
    contract = OrderContract.new
    result = contract.call(input)
    result.success? ? Success(result.to_h) : Failure(result.errors)
  end

  def create_order(input)
    order = Order.create!(input)
    Success(order)
  rescue ActiveRecord::RecordInvalid => e
    Failure(e.message)
  end

  def charge_payment(order)
    # Payment logic...
    Success(order)
  end

  def send_confirmation(order)
    OrderMailer.confirmation(order).deliver_later
    Success(order)
  end
end

# Usage
result = CreateOrder.new.call(order_params)
result.success? # => true/false

12.2 dry-effects: Algebraic Effects

Advanced: dependency injection via effects (alternative to dry-container):

class ProcessPayment
  include Dry::Effects.Reader(:current_user)
  include Dry::Effects.Resolve(:payment_gateway)

  def call(amount)
    gateway = resolve(:payment_gateway)
    gateway.charge(current_user, amount)
  end
end

# Provide effects at call site
result = Dry::Effects.handler.with(
  current_user: user,
  resolve: { payment_gateway: StripeGateway.new }
) { ProcessPayment.new.call(100) }

12.3 Gem Selection Guide

Gem	When to Use	Complexity
dry-types	Any project (foundational)	Low
dry-struct	Data transfer objects, value objects	Low
dry-configurable	Configuration management	Low
dry-initializer	Service object parameters	Low
dry-monads	Service objects with failures	Medium
dry-validation	API input validation	Medium
dry-container	Large apps, testing focus	Medium
dry-matcher	Result pattern matching	Medium
dry-events	Event-driven architecture	Medium
dry-transaction	Multi-step workflows	Medium
dry-effects	Advanced FP patterns	High

13. Benefits and Tradeoffs

13.1 Benefits

1. Explicit Over Implicit

# Before: What can access_status be? Check the code...
result[:access_status] = ACCESS_STATUS_VALUES.include?(status) ? status : "none"

# After: The type IS the documentation
attribute :access_status, Types::Profiles::AccessStatus  # Enum is explicit

2. Testability

# Before: Test through controller/blueprint
it "returns correct access_status" do
  get "/profiles/1", headers: auth_headers
  expect(response.body).to include('"access_status":"approved"')
end

# After: Unit test the struct directly
it "coerces invalid status to none" do
  actions = Profiles::Actions.new(access_status: "invalid")
  expect(actions.access_status).to eq("none")
end

3. Composability

# Combine types to build complex structures
CompanyProfile = Types::Hash.schema(
  name: Types::String,
  employees: Types::Array.of(EmployeeStruct),
  settings: Types::Hash.schema(
    visibility: Types::Profiles::VisibilityMode,
    features: Types::Array.of(Types::String)
  )
)

4. Error Messages

# dry-validation provides structured errors
contract.call(thumb_status: "invalid").errors.to_h
# => { thumb_status: ["must be one of: liked, disliked"] }

5. Event-Driven Architecture

# Before: Side effects scattered in controllers
Analytics.track(...)
AuditLog.create!(...)
CacheWarmer.perform_async(...)

# After: Single event, multiple handlers
EventPublisher.publish("profile.viewed", { ... })

6. Centralized Configuration

# Before: grep -r "CACHE_TTL" app/
# After: ProfileConfig.config.actions.cache_ttl

13.2 Tradeoffs

1. Learning Curve

Team needs to learn dry-rb idioms
Functional programming concepts (monads, composition)
More files/classes than “Rails way”

2. Indirection

Types defined in one place, used in another
Following data flow requires understanding container
Events may be handled far from where they’re published

3. Overhead for Simple Cases

# For a simple boolean flag, this is overkill:
attribute :is_active, Types::Bool.default(false)

# This is fine:
attr_accessor :is_active

4. Gems Add Dependencies

11 gems for full suite
Version compatibility to manage
Bundle size increases

13.3 When to Use dry-rb

✅ Good Fit:

APIs with complex data contracts
Services with multiple failure modes
Apps where testability is priority
Teams comfortable with FP concepts
Large apps with many developers
Event-driven or audit-heavy systems

❌ Poor Fit:

Simple CRUD apps
Prototypes/MVPs
Teams unfamiliar with FP
Performance-critical hot paths (slight overhead)

14. Conclusion

Integrating dry-rb into our Rails API transformed code that “worked” into code that’s obvious, testable, and maintainable:

Metric	Before	After
Actions payload code	70+ lines	15 lines
Type validation	Manual per-field	Declarative
Error handling	try/rescue + nil	Result monad
Dependencies	Hard-coded classes	Injected
Configuration	Scattered constants	Centralized
Side effects	Inline in controllers	Event-driven
Service parameters	Manual initialize	Declarative DSL
Result handling	if/case statements	Pattern matching
Test isolation	Integration only	Unit + integration
Gems used	0	11

Key Takeaways

Start with dry-types — It’s foundational and low-risk
Add dry-struct for data transfer — Especially for API responses
Use dry-monads in services — Replace boolean/nil returns
Add dry-validation for API inputs — Before data hits your models
Use dry-configurable early — Centralize config before it scatters
Add dry-initializer to services — Reduce boilerplate immediately
Introduce dry-container gradually — When testing pain increases
Add dry-matcher for cleaner controllers — Pattern matching is elegant
Use dry-events for side effects — Decouple analytics, logging, notifications
Don’t over-engineer — A simple hash is fine for simple data

The Return on Investment

The initial investment—learning curves, refactoring, new patterns—pays dividends in:

Fewer bugs — Type mismatches caught at construction, not runtime
Faster debugging — Follow explicit contracts, not implicit conventions
Easier onboarding — Types and events document themselves
Confident refactoring — Change implementation, types verify compatibility
Testable architecture — Unit test components in isolation
Scalable patterns — Event-driven architecture grows with your app

dry-rb doesn’t replace Rails conventions—it complements them where they fall short. Use the right tool for each job.

Resources

dry-rb Official Site: dry-rb.org
dry-rb GitHub: github.com/dry-rb
dry-types Documentation: dry-rb.org/gems/dry-types
dry-monads Documentation: dry-rb.org/gems/dry-monads
dry-validation Documentation: dry-rb.org/gems/dry-validation
dry-events Documentation: dry-rb.org/gems/dry-events
dry-configurable Documentation: dry-rb.org/gems/dry-configurable
dry-initializer Documentation: dry-rb.org/gems/dry-initializer
dry-matcher Documentation: dry-rb.org/gems/dry-matcher
Trailblazer (alternative): trailblazer.to (uses dry-rb under the hood)

This post documents our integration of 11 dry-rb gems into the Wigiwork API. The patterns described here reduced our Profile Actions payload logic from 70+ lines to 15, improved test coverage, established event-driven architecture, and created patterns that scale across 100+ endpoints.

From MySQL to PostgreSQL: A Practical Migration Guide for Rails APIs

Wed, 03 Dec 2025 00:00:00 +0000

“The best time to write database-agnostic code is from day one. The second best time is before your migration.”

When we started building the Wigiwork API, we chose MySQL for its familiarity and ease of setup. As our requirements grew—AI vector embeddings, geospatial queries, graph routing—we realized PostgreSQL wasn’t just an alternative, it was a necessity.

This post documents our migration journey and shares the patterns that made it nearly painless.

Why We Migrated
The Database-Agnostic Foundation
The Migration Process
PostgreSQL vs MySQL: A Technical Comparison
Advanced PostgreSQL Features
Multi-Schema Architecture
Migration Tips and Lessons Learned

1. Why We Migrated

1.1 The Growing Feature Requirements

Our application started as a straightforward REST API. Over time, we needed:

Requirement	MySQL Solution	PostgreSQL Solution
Vector similarity search (AI)	External service (Pinecone, Weaviate)	Native `pgvector` extension
Geospatial queries	Limited spatial functions	PostGIS (industry standard)
Graph routing	External graph DB	pgRouting extension
Full-text search	Basic FULLTEXT	Powerful tsvector/tsquery
JSON querying	JSON functions (limited)	JSONB with GIN indexes

The math was simple: Three external services vs. one PostgreSQL instance.

1.2 The Hidden Costs of Multiple Databases

Running MySQL + Pinecone + a graph database meant:

3x infrastructure complexity — More services to monitor, scale, and pay for
Data synchronization overhead — Keeping vector embeddings in sync with source data
Network latency — Cross-service queries add milliseconds
Operational burden — Different backup strategies, different failure modes

PostgreSQL with extensions gave us one database to rule them all.

1.3 Our Starting Point

Before migration, our codebase analysis revealed:

Category	Count	Status
MySQL-specific SQL	1 function	Required fix
JSON columns	45	Upgrade to JSONB
Check constraints	11	Already compatible
Raw SQL queries	15+	Already compatible
Configuration files	4	Required update

98% of our code was already database-agnostic. This wasn’t luck—it was intentional architecture.

2. The Database-Agnostic Foundation

2.1 Why Database-Agnostic Code Matters

Writing database-agnostic code isn’t about planning to switch databases. It’s about:

Portability — Your code works on any database Rails supports
Testability — SQLite in CI, PostgreSQL in production
Maintainability — Standard patterns are easier to understand
Future-proofing — Requirements change; your code adapts

2.2 The Rules We Follow

Our AGENTS.md (development guidelines) includes this rule:

Rule #9: Database-agnostic queries. Use Arel and ActiveRecord abstractions instead of raw SQL or database-specific syntax. This ensures portability when migrating databases.

2.3 Patterns That Work Everywhere

✅ Use ActiveRecord Query Interface

# Good: Works on any database
User.where(status: :active)
    .where("created_at > ?", 1.week.ago)
    .order(created_at: :desc)
    .limit(10)

✅ Use Arel for Complex Queries

# Good: Database-agnostic date ranges
class CreditTransaction < ApplicationRecord
  scope :by_requested_dates, ->(dates) {
    ranges = Array(dates).filter_map do |date|
      year, month = date.split("-").map(&:to_i)
      start_date = Time.zone.local(year, month, 1).beginning_of_day
      end_date = start_date.end_of_month.end_of_day
      arel_table[:created_at].between(start_date..end_date)
    end
    where(ranges.reduce(:or))
  }
end

✅ Use Standard SQL Functions

# Good: LOWER, COALESCE, CASE work everywhere
scope :search, ->(term) {
  where("LOWER(name) LIKE :term OR LOWER(email) LIKE :term",
        term: "%#{term.downcase}%")
}

# Good: COALESCE for fallbacks
order(Arel.sql("COALESCE(last_message_at, created_at) DESC"))

# Good: CASE for conditional ordering
order(Arel.sql("CASE WHEN status = 'urgent' THEN 0 ELSE 1 END"))

✅ Process Dates in Ruby

# Good: Fetch timestamps, format in Ruby
def self.date_buckets_for(user, field:)
  pluck(field)
    .filter_map { |timestamp| timestamp&.strftime("%Y-%m") }
    .uniq
    .first(24)
end

2.4 Patterns to Avoid

❌ MySQL-Specific Functions

# Bad: DATE_FORMAT is MySQL-only
month_sql = "DATE_FORMAT(created_at, '%Y-%m')"

# PostgreSQL equivalent: TO_CHAR(created_at, 'YYYY-MM')
# Better: Process in Ruby (see above)

❌ Database-Specific Syntax

# Bad: MySQL's IFNULL
where("IFNULL(deleted_at, NOW()) > ?", 1.day.ago)

# Good: Standard COALESCE
where("COALESCE(deleted_at, CURRENT_TIMESTAMP) > ?", 1.day.ago)

❌ String Concatenation Operators

# Bad: MySQL uses CONCAT(), PostgreSQL uses ||
select("CONCAT(first_name, ' ', last_name) AS full_name")

# Good: Use Ruby
def full_name
  "#{first_name} #{last_name}"
end

3. The Migration Process

3.1 Pre-Migration Audit

Before touching any configuration, we audited the entire codebase:

# Find MySQL-specific SQL
grep -r "DATE_FORMAT\|GROUP_CONCAT\|IFNULL\|BINARY\s" app/

# Find raw SQL that might be problematic
grep -r "\.where.*Arel\.sql\|execute\|connection\." app/

# Check for MySQL gem references
grep -r "mysql" Gemfile config/

Our findings:

Issue	Location	Fix
`DATE_FORMAT`	`app/models/access_request.rb`	Rewrote using Ruby
`mysql2` gem	`Gemfile`	Replaced with `pg`
MySQL config	`config/database.yml`	Rewrote for PostgreSQL
Docker MySQL	`docker-compose.dev.yml`	Replaced with PostgreSQL

3.2 The Critical Fix

The only code change required was in access_request.rb:

# Before: MySQL-specific
def self.date_buckets_for(user, field:)
  column = field.to_s
  qcol = connection.quote_column_name(column)
  month_sql = "DATE_FORMAT(#{qcol}, '%Y-%m')"  # ❌ MySQL-only

  for_user(user)
    .group(Arel.sql(month_sql))
    .pluck(Arel.sql(month_sql))
end

# After: Database-agnostic Ruby
def self.date_buckets_for(user, field:)
  column = field.to_s
  return [] unless DATE_BUCKET_FIELDS.include?(column)

  for_user(user)
    .where.not(column => nil)
    .order(column => :desc)
    .limit(500)
    .pluck(column)
    .filter_map { |timestamp| timestamp&.strftime("%Y-%m") }
    .uniq
    .first(24)
    .filter_map do |month|
      start_at = Time.zone.parse("#{month}-01")
      next unless start_at

      {
        id: month,
        name: start_at.strftime("%B %Y"),
        from: start_at.beginning_of_month.beginning_of_day.iso8601,
        to: start_at.end_of_month.end_of_day.iso8601
      }
    end
end

3.3 Configuration Changes

Gemfile:

# Remove
gem 'mysql2', '~> 0.5.6'

# Add
gem 'pg', '~> 1.5'

config/database.yml:

default: &default
  adapter: postgresql
  encoding: unicode
  pool: <%= ENV.fetch("RAILS_MAX_THREADS", 5) %>
  username: <%= ENV.fetch("POSTGRES_USER", "postgres") %>
  password: <%= ENV.fetch("POSTGRES_PASSWORD", "") %>
  host: <%= ENV.fetch("POSTGRES_HOST", "127.0.0.1") %>
  port: <%= ENV.fetch("POSTGRES_PORT", 5432) %>

development:
  <<: *default
  database: wigiwork_development

test:
  <<: *default
  database: wigiwork_test

production:
  <<: *default
  database: <%= ENV["POSTGRES_DB"] %>

docker-compose.dev.yml:

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: password
      POSTGRES_DB: wigiwork_development
    ports:
      - "5433:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 20
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

3.4 JSON to JSONB Migration

MySQL’s JSON type works, but PostgreSQL’s JSONB is superior:

# In migration files, change:
t.json :metadata

# To:
t.jsonb :metadata, default: {}

Why JSONB?

Feature	JSON	JSONB
Storage	Text (parsed each query)	Binary (pre-parsed)
Indexing	❌ Not supported	✅ GIN indexes
Operators	Basic	Rich (`@>`, `?`, `?\|`, `?&`)
Query speed	Slower	Faster
Write speed	Faster	Slightly slower

JSONB Query Examples:

# Find profiles with specific skill in JSONB array
Profile.where("skills @> ?", ['ruby'].to_json)

# Find users with specific setting
User.where("settings -> 'notifications' ->> 'email' = ?", 'true')

# Check if key exists
Product.where("metadata ? 'featured'")

3.5 Running the Migration

# 1. Update gems
bundle install

# 2. Create databases
rails db:create

# 3. Run migrations
rails db:migrate

# 4. Verify schema
rails db:schema:dump

# 5. Run test suite
bundle exec rspec

# 6. Seed development data
rails db:seed

Total downtime: Zero (for development). Production required a maintenance window for data migration.

4. PostgreSQL vs MySQL: A Technical Comparison

4.1 Feature Comparison

Feature	MySQL	PostgreSQL
JSON Support	JSON type, basic functions	JSONB with GIN indexes, rich operators
Full-Text Search	FULLTEXT indexes	tsvector/tsquery, ranking, dictionaries
Geospatial	Basic spatial	PostGIS (industry standard)
Vector Search	❌ None	pgvector extension
Graph Queries	❌ None	pgRouting, Apache AGE
Arrays	❌ None	Native array types with operators
Range Types	❌ None	int4range, daterange, etc.
Check Constraints	✅ Supported	✅ Supported
Partial Indexes	❌ None	✅ Supported
Expression Indexes	❌ Limited	✅ Full support
CTEs (WITH queries)	✅ Basic	✅ Recursive, materialized
Window Functions	✅ Supported	✅ More advanced
UPSERT	`ON DUPLICATE KEY`	`ON CONFLICT` (more flexible)
Transactions	✅ InnoDB	✅ MVCC
Replication	✅ Built-in	✅ Streaming, logical

4.2 Performance Characteristics

MySQL Strengths:

Simple read-heavy workloads
Large-scale web applications with basic queries
When you need maximum reads/second on simple queries

PostgreSQL Strengths:

Complex queries with many joins
Write-heavy workloads (better MVCC)
Data integrity (stricter by default)
Advanced data types and indexing

4.3 Why PostgreSQL Won for Us

1. One Database for Everything

Instead of:

MySQL (primary data)
  + Pinecone (vector search)
  + Neo4j (graph queries)
  + Elasticsearch (full-text)

We have:

PostgreSQL
  + pgvector (vector search)
  + pgRouting (graph queries)
  + Built-in full-text search

2. Better Data Integrity

-- PostgreSQL: Strict by default
INSERT INTO users (email) VALUES ('not-an-email');
-- ERROR: violates check constraint

-- MySQL: Often silently truncates or converts
INSERT INTO users (name) VALUES ('this string is way too long for the column');
-- Silently truncated (depending on sql_mode)

3. Superior Indexing

# Partial index: Only index active users
add_index :users, :email, where: "status = 'active'"

# Expression index: Index lowercased emails
add_index :users, "LOWER(email)"

# GIN index on JSONB for fast JSON queries
add_index :profiles, :skills, using: :gin

4. Native Array Support

# PostgreSQL array columns
t.string :tags, array: true, default: []

# Query arrays
Profile.where("'ruby' = ANY(tags)")
Profile.where("tags @> ARRAY[?]", ['ruby', 'rails'])

5. Advanced PostgreSQL Features

5.1 pgvector: AI Vector Embeddings

Store and query vector embeddings directly in PostgreSQL:

# Enable extension
class EnablePgvector < ActiveRecord::Migration[8.0]
  def change
    enable_extension 'vector'
  end
end

# Add vector column
class AddEmbeddingToProfiles < ActiveRecord::Migration[8.0]
  def change
    add_column :profiles, :embedding, :vector, limit: 1536  # OpenAI dimension
    add_index :profiles, :embedding, using: :ivfflat, opclass: :vector_cosine_ops
  end
end

# app/models/profile.rb
class Profile < ApplicationRecord
  def self.similar_to(embedding, limit: 10)
    where.not(embedding: nil)
      .order(Arel.sql("embedding <=> '#{embedding}'"))
      .limit(limit)
  end
end

# Usage
similar_profiles = Profile.similar_to(openai_embedding, limit: 5)

5.2 PostGIS: Geospatial Queries

# Enable extension
enable_extension 'postgis'

# Add geography column
add_column :locations, :coordinates, :st_point, geographic: true
add_index :locations, :coordinates, using: :gist

# Find locations within 10km
Location.where(
  "ST_DWithin(coordinates, ST_MakePoint(?, ?)::geography, ?)",
  longitude, latitude, 10_000  # meters
)

5.3 pgRouting: Graph Algorithms

# Find shortest path between locations
execute <<-SQL
  SELECT * FROM pgr_dijkstra(
    'SELECT id, source, target, cost FROM edges',
    #{start_node},
    #{end_node},
    directed := true
  )
SQL

5.4 Full-Text Search

# Add tsvector column
add_column :articles, :searchable, :tsvector
add_index :articles, :searchable, using: :gin

# Keep it updated with a trigger
execute <<-SQL
  CREATE TRIGGER articles_searchable_update
  BEFORE INSERT OR UPDATE ON articles
  FOR EACH ROW EXECUTE FUNCTION
  tsvector_update_trigger(searchable, 'pg_catalog.english', title, body);
SQL

# Search with ranking
Article.where("searchable @@ plainto_tsquery('english', ?)", query)
       .order(Arel.sql("ts_rank(searchable, plainto_tsquery('english', '#{query}')) DESC"))

6. Multi-Schema Architecture

6.1 What Are PostgreSQL Schemas?

PostgreSQL schemas are namespaces within a single database. Think of them as folders for your tables:

database: wigiwork_production
├── public (default schema)
│   ├── users
│   ├── profiles
│   └── companies
├── analytics
│   ├── events
│   ├── page_views
│   └── conversions
├── audit
│   ├── user_actions
│   └── api_requests
└── cache
    ├── cached_profiles
    └── cached_searches

6.2 Benefits of Multi-Schema Architecture

1. Logical Separation Without Multiple Databases

# One connection pool, multiple schemas
class AnalyticsEvent < ApplicationRecord
  self.table_name = 'analytics.events'
end

class AuditLog < ApplicationRecord
  self.table_name = 'audit.user_actions'
end

2. Different Permissions Per Schema

-- Analytics team can only access analytics schema
GRANT USAGE ON SCHEMA analytics TO analytics_role;
GRANT SELECT ON ALL TABLES IN SCHEMA analytics TO analytics_role;

-- No access to public schema
REVOKE ALL ON SCHEMA public FROM analytics_role;

3. Simplified Backup Strategies

# Backup only the audit schema
pg_dump -n audit wigiwork_production > audit_backup.sql

# Backup everything except cache
pg_dump -N cache wigiwork_production > production_backup.sql

4. Schema-Level Maintenance

-- Truncate all cache tables without affecting production
TRUNCATE TABLE cache.cached_profiles, cache.cached_searches;

-- Drop and recreate analytics schema
DROP SCHEMA analytics CASCADE;
CREATE SCHEMA analytics;

6.3 Setting Up Multi-Schema in Rails

Migration to create schemas:

class CreateAnalyticsSchema < ActiveRecord::Migration[8.0]
  def up
    execute "CREATE SCHEMA IF NOT EXISTS analytics"
    execute "CREATE SCHEMA IF NOT EXISTS audit"
    execute "CREATE SCHEMA IF NOT EXISTS cache"
  end

  def down
    execute "DROP SCHEMA IF EXISTS cache CASCADE"
    execute "DROP SCHEMA IF EXISTS audit CASCADE"
    execute "DROP SCHEMA IF EXISTS analytics CASCADE"
  end
end

Models with schema prefixes:

# app/models/analytics/event.rb
module Analytics
  class Event < ApplicationRecord
    self.table_name = 'analytics.events'
  end
end

# app/models/audit/user_action.rb
module Audit
  class UserAction < ApplicationRecord
    self.table_name = 'audit.user_actions'
  end
end

Cross-schema queries work seamlessly:

# Join across schemas
User.joins("JOIN audit.user_actions ON audit.user_actions.user_id = users.id")
    .where("audit.user_actions.action = ?", "login")

6.4 Schemas vs. Multiple Databases

Aspect	Multiple Databases	Multiple Schemas
Connection pools	Separate per DB	Shared
Cross-queries	Complex/impossible	Native JOINs
Transactions	Distributed (2PC)	Single transaction
Backups	Per database	Per schema or together
Permissions	Database-level	Schema-level (granular)
Migrations	Separate per DB	Single migration
Complexity	Higher	Lower
Rails support	Multiple configs	Single config

Our recommendation: Use schemas for logical separation within the same application. Use separate databases only for truly isolated services or multi-tenant SaaS.

6.5 When to Use Each Pattern

Use Multiple Schemas When:

Separating concerns (analytics, audit, cache)
Different retention policies per data type
Granular permission control needed
Data is related and needs cross-queries

Use Multiple Databases When:

Complete data isolation required (multi-tenant)
Different scaling requirements
Regulatory compliance requires separation
Microservices with independent lifecycles

7. Migration Tips and Lessons Learned

7.1 Before You Start

1. Audit your codebase thoroughly

# Find all raw SQL
grep -rn "execute\|\.sql\|Arel\.sql" app/ --include="*.rb"

# Find MySQL-specific functions
grep -rn "DATE_FORMAT\|IFNULL\|GROUP_CONCAT\|BINARY\|STRAIGHT_JOIN" app/

# Check gems for MySQL dependencies
bundle exec gem dependency mysql2

2. Run tests on PostgreSQL locally first

# config/database.yml for CI
test:
  adapter: postgresql
  database: myapp_test

3. Document all discrepancies

Create a migration checklist (like our mysql_to_postgresql.md) tracking every issue found.

7.2 Data Migration Strategies

Option A: Fresh Start (Recommended for Development)

# On PostgreSQL
rails db:create db:migrate db:seed

Option B: pgloader (For Production Data)

# Install pgloader
brew install pgloader  # or apt-get

# Create migration script
cat > migrate.load << 'EOF'
LOAD DATABASE
  FROM mysql://root:password@localhost/myapp_production
  INTO postgresql://postgres@localhost/myapp_production

WITH include drop, create tables, create indexes, reset sequences

SET work_mem to '16MB', maintenance_work_mem to '512 MB'

CAST type datetime to timestamptz using zero-dates-to-null,
     type date to date using zero-dates-to-null

ALTER SCHEMA 'myapp_production' RENAME TO 'public';
EOF

# Run migration
pgloader migrate.load

Option C: Rails-Native Export/Import

# Export from MySQL
namespace :export do
  task users: :environment do
    File.open('users.json', 'w') do |f|
      User.find_each { |u| f.puts u.to_json }
    end
  end
end

# Import to PostgreSQL
namespace :import do
  task users: :environment do
    File.readlines('users.json').each do |line|
      User.create!(JSON.parse(line))
    end
  end
end

7.3 Common Gotchas

1. Auto-increment vs. SERIAL

MySQL uses AUTO_INCREMENT, PostgreSQL uses SERIAL or IDENTITY. Rails handles this automatically, but check sequences after migration:

-- Fix sequence if IDs are out of sync
SELECT setval('users_id_seq', (SELECT MAX(id) FROM users));

2. Boolean Handling

MySQL often stores booleans as TINYINT(1). PostgreSQL uses native BOOLEAN:

# May need explicit casting in raw SQL
where("active = true")  # PostgreSQL
where("active = 1")     # MySQL

3. Case Sensitivity

PostgreSQL string comparisons are case-sensitive by default:

# MySQL: case-insensitive by default
User.where(email: 'Test@Example.com')  # Finds test@example.com

# PostgreSQL: case-sensitive
User.where("LOWER(email) = LOWER(?)", 'Test@Example.com')
# Or use citext extension for case-insensitive columns

4. Group By Strictness

PostgreSQL requires all non-aggregated columns in GROUP BY:

# MySQL: Allows this (picks arbitrary value)
select("users.*, COUNT(orders.id)")
  .joins(:orders)
  .group("users.id")

# PostgreSQL: Must include all selected columns
select("users.id, users.email, COUNT(orders.id)")
  .joins(:orders)
  .group("users.id, users.email")

7.4 Post-Migration Checklist

# 1. Verify all tests pass
bundle exec rspec

# 2. Check for deprecation warnings
RAILS_ENV=test rails runner "puts ActiveRecord::Base.connection.adapter_name"

# 3. Regenerate schema
rails db:schema:dump

# 4. Verify schema.rb has PostgreSQL syntax
grep "create_table" db/schema.rb | head

# 5. Run security scan
bundle exec brakeman -q -w2

# 6. Regenerate API documentation
bundle exec rails rswag:specs:swaggerize

# 7. Performance baseline
rails runner "Benchmark.measure { User.count }"

7.5 Lessons We Learned

1. Write database-agnostic code from day one

The single MySQL-specific function we had took 30 minutes to rewrite. If we’d had dozens, it would have been a multi-day effort.

2. Use JSONB defaults in migrations

# Always specify defaults for JSONB
t.jsonb :settings, default: {}
t.jsonb :tags, default: []

3. Leverage PostgreSQL-specific features gradually

Don’t rewrite everything immediately. Start with:

JSONB for new features
Partial indexes for performance
Full-text search when needed

4. Monitor query performance post-migration

PostgreSQL’s query planner is different. Some queries may need new indexes:

# Add explain logging in development
ActiveRecord::Base.logger = Logger.new(STDOUT)
User.where(status: :active).explain

5. Document your extensions

# db/migrate/001_enable_extensions.rb
class EnableExtensions < ActiveRecord::Migration[8.0]
  def change
    # Core extensions we use
    enable_extension 'pgcrypto'  # UUID generation
    enable_extension 'citext'    # Case-insensitive text

    # Feature-specific (enable when needed)
    # enable_extension 'vector'   # AI embeddings
    # enable_extension 'postgis'  # Geospatial
  end
end

Conclusion

Migrating from MySQL to PostgreSQL doesn’t have to be painful. With database-agnostic code patterns and a systematic approach, our migration required:

1 code fix (DATE_FORMAT → Ruby)
4 configuration file updates
0 model changes (beyond JSON → JSONB)
0 downtime (for development)

The benefits far outweigh the effort:

Before (MySQL + Services)	After (PostgreSQL)
3+ external services	1 database
Multiple connection pools	Single pool
Cross-service latency	Native queries
Complex sync logic	Referential integrity
Higher infrastructure cost	Lower cost

PostgreSQL isn’t just a MySQL replacement—it’s a platform for building sophisticated applications. With extensions like pgvector, PostGIS, and pgRouting, you can consolidate your entire data layer into a single, powerful database.

Key Takeaways

Write database-agnostic code — Use ActiveRecord/Arel, avoid raw SQL functions
Audit before migrating — Find MySQL-specific code early
Use JSONB over JSON — Better indexing, better queries
Consider schemas over databases — Simpler architecture, better performance
Leverage PostgreSQL features — Partial indexes, arrays, full-text search
Plan for extensions — pgvector, PostGIS, pgRouting when needed
Test thoroughly — Run your full suite on PostgreSQL before production

The best database is the one that grows with your needs. PostgreSQL does exactly that.

Resources

PostgreSQL Official Docs: postgresql.org/docs
pgvector Extension: github.com/pgvector/pgvector
PostGIS Documentation: postgis.net/docs
pgRouting Documentation: pgrouting.org
Rails PostgreSQL Guide: guides.rubyonrails.org/active_record_postgresql.html
pgloader: pgloader.io

This post documents our migration of the Wigiwork API from MySQL to PostgreSQL. The patterns described here enabled a migration with minimal code changes and zero regressions—proving that investment in database-agnostic code pays dividends when requirements evolve.

Zero-Gap API Development: A Contract-First Framework for AI-Assisted Rails Engineering

Tue, 02 Dec 2025 00:00:00 +0000

TL;DR

Contract-first development with TypeScript interfaces as the single source of truth eliminates ambiguity and prevents frontend/backend drift.

Systematic verification gates catch implementation gaps before they ship—not after.

Discrepancy classification ([IMPL] vs [DOC]) ensures you fix what’s broken and flag what’s misaligned.

Mandatory dual documentation (Flow + PRD) creates institutional memory that survives team changes.

This post is the evolution of our previous guides on consolidating Rails contribution standards and senior-driven AI development. After shipping dozens of endpoints with zero regressions, we’ve refined the process into a complete framework.

The Problem: Death by a Thousand Gaps

Every API project eventually faces the same failure modes:

Contract drift: Backend returns userId, frontend expects user_id
Silent regressions: v1.1 breaks v1.0 behaviors nobody tested
Incomplete implementations: 8 of 10 required fields shipped, 2 “forgot”
Documentation rot: Flow docs describe behavior from 3 sprints ago
AI amplification: LLMs generate plausible-looking code that fails edge cases

The root cause isn’t carelessness—it’s missing systematic verification. Humans (and AIs) make errors. Systems catch them.

The Solution: Three-Layer Architecture

Our framework operates on three layers, each with a specific purpose:

┌─────────────────────────────────────────────────────────┐
│  Layer 1: REQUIREMENT_DOC (doc/requirements/**)         │
│  - TypeScript interfaces = canonical truth              │
│  - Read-only, never modified by implementers            │
│  - Versioned (v1.0, v1.1, v1.2...)                     │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│  Layer 2: Implementation (app/**, spec/**)              │
│  - Must satisfy ALL versions cumulatively               │
│  - Blueprinter for JSON, Pundit for auth                │
│  - Contract Compliance Verification before merge        │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│  Layer 3: Documentation (doc/flow/**, doc/prd/**)       │
│  - Generated AFTER verification passes                  │
│  - Reflects actual implementation, not intent           │
│  - Version history preserved, never overwritten         │
└─────────────────────────────────────────────────────────┘

Layer 1: Contract-First Development

The REQUIREMENT_DOC is the single source of truth. It contains TypeScript interfaces that define the exact response shape:

// doc/requirements/PROFILE_CARD.md
interface ProfileCard {
  id: number;                    // required
  username: string;              // required
  avatar_url: string | null;     // optional
  skills: Skill[];               // required, min 0 items
  availability: {
    status: 'now' | 'two_weeks' | 'offer' | 'na';
    updated_at: string;          // ISO8601
  };
}

The rules are simple:

Every field in the interface must exist in your Blueprint
Required fields never return null—use safe defaults
No extra fields beyond what’s documented
Version changes are cumulative unless explicitly marked as breaking

Safe Defaults Strategy

When database data is sparse, required fields still must satisfy the contract:

# app/blueprints/profile_card_blueprint.rb
class ProfileCardBlueprint < Blueprinter::Base
  identifier :id

  field :username do |profile|
    profile.username.presence || ""  # String → empty string
  end

  field :skills do |profile|
    profile.skills.presence || []    # Array → empty array
  end

  field :availability do |profile|
    {
      status: profile.availability_status || "na",  # Enum → default value
      updated_at: (profile.availability_updated_at || Time.current).iso8601
    }
  end
end

Safe defaults by type:

Type	Default
String (required)	`""` (empty string)
Number (required)	`0` or minimum valid value
Boolean (required)	`false`
Array (required)	`[]` (empty array)
Object (required)	Minimal valid object with nested defaults
Timestamp	`Time.current.iso8601`

Layer 2: Implementation with Verification Gates

The Verification Checklist

Every implementation must pass these gates in order:

# 1. Code quality
bundle exec rubocop

# 2. Test suite (establishes baseline, catches regressions)
bundle exec rspec

# 3. Security scan
bundle exec brakeman -q -w2

# 4. Dependency audit
bundle exec bundle audit check --update

# 5. API documentation generation
bundle exec rails rswag:specs:swaggerize

If any gate fails, stop and fix. Do not proceed to documentation.

Contract Compliance Verification (NO GAPS)

After all tests pass, systematically audit against REQUIREMENT_DOC:

1. Endpoints Audit

Every path in REQUIREMENT_DOC exists in routes
HTTP methods match (GET/POST/PATCH/DELETE)
URL and query params match documented names
Auth requirements match (public vs authenticated, role restrictions)

2. Response Shape Audit

Every field in TypeScript interface exists in Blueprint
No extra fields emitted beyond REQUIREMENT_DOC
Required fields never return null/nil
Nested objects match documented structure exactly

3. Validation & Error Audit

All documented validations implemented
Error responses use correct status codes:
- 422 → Validation errors (:unprocessable_content)
- 400 → Bad request / malformed params
- 401 → Unauthorized (missing/invalid auth)
- 403 → Forbidden (authenticated but not permitted)
- 404 → Not found

4. Version Compliance Audit

v1.0 requirements still satisfied
v1.1+ additions implemented correctly
Breaking changes only where REQUIREMENT_DOC explicitly declares

Discrepancy Classification

When gaps are found, classify them correctly:

Type A: Implementation Issues — Your code doesn’t match REQUIREMENT_DOC

[IMPL] GET /profiles/:id - [EXPECTED: returns avatar_url] vs [ACTUAL: field missing from Blueprint]
[IMPL] POST /users - [EXPECTED: 422 for invalid email] vs [ACTUAL: returns 400]

→ Fix ALL before proceeding. Re-run verification after fixes.

Type B: Requirement Doc Discrepancies — REQUIREMENT_DOC doesn’t match existing DB/model

[DOC] visible_within enum - [DOC: 'local'|'state'|'country'] vs [DB: '5'|'10'|'25'|'50' miles]
[DOC] Availability.status - [DOC: 'available'|'unavailable'] vs [Model: 'now'|'two_weeks'|'offer'|'na']

→ Do NOT modify doc/requirements/** (read-only). → Blueprint returns actual DB values (correct behavior). → Flag for product/doc owner to reconcile.

Layer 3: Mandatory Documentation

Both Flow docs and PRDs are mandatory for every task. Generate only after verification passes.

Flow Doc Template (WEB-286 structure)

# TASK_ID — Feature Name

## Metadata
- **Task:** TASK_ID
- **Status:** Implemented
- **Version:** 1.0

## Table of Contents
1. General Description
2. Validation Use Cases
3. Pundit Policy
4. Implementation Notes
5. Status Codes
6. Flow
7. Responsible Files

## General Description
[What the feature does, who can access it]

## Validation Use Cases
| Field | Rule | Error Code |
|-------|------|------------|
| email | required, format | 422 |

## Pundit Policy
- `index?` → authenticated users
- `create?` → owner only

## Status Codes
| Code | Condition |
|------|-----------|
| 200 | Success |
| 422 | Validation failed |
| 403 | Not authorized |

## Responsible Files
- `app/controllers/api/v1/items_controller.rb`
- `app/blueprints/item_blueprint.rb`
- `app/policies/item_policy.rb`
- `spec/requests/api/v1/items_spec.rb`

PRD Template (WEB-240 structure)

# TASK_ID — Feature Name PRD

## Problem Statement
[What problem does this solve?]

## Goals
- [Goal 1]
- [Goal 2]

## Scope
### In Scope
- [Feature A]

### Out of Scope
- [Feature B]

## User Stories
- As a [role], I want to [action] so that [benefit]

## Technical Considerations
- [Database changes]
- [API contract]

## Metrics
- [How will success be measured?]

## Rollout Plan
- [Deployment strategy]

## Dependencies
- [External services, gems, etc.]

Version History

Both docs must maintain explicit version history:

## Version History

### Version 1.1 (2025-12-02)
- Added `avatar_url` field to response
- Changed `status` enum values

### Version 1.0 (2025-11-15)
- Initial implementation

Never overwrite—always append.

The Complete Prompt Template

When using AI assistants, use this universal prompt:

Task: TASK_ID – FEATURE_LABEL (Ruby on Rails API)

Context:
- Requirements (single source of truth): REQUIREMENT_DOC
- Flow doc: FLOW_DOC
- PRD doc: PRD_DOC

Global constraints:
- Read and follow AGENTS.md and GUIDE.md before changing anything
- JSON via Blueprinter only; controllers handle envelopes
- Never bypass serializers, never use camelCase keys

Core contract rules:
- TypeScript interfaces in REQUIREMENT_DOC are the ONLY truth
- Required fields never return null (use safe defaults)
- All versions (v1.0, v1.1, ...) satisfied cumulatively

Implementation:
- Upgrade existing code to satisfy ALL versions in REQUIREMENT_DOC
- Do NOT remove behaviors that already match the contract
- Build JSON via Blueprinter with exact field sets

Verification (all must pass):
1. bundle exec rubocop
2. bundle exec rspec
3. bundle exec brakeman -q -w2
4. bundle exec bundle audit check --update
5. bundle exec rails rswag:specs:swaggerize

Contract Compliance (NO GAPS):
- Audit: endpoints, response shape, validations, versions
- Classify discrepancies: [IMPL] = fix code, [DOC] = flag for doc owner

Final documentation (AFTER verification):
- Generate/update FLOW_DOC and PRD_DOC
- Include version history

Controller Patterns That Work

Index with Pagination

def index
  items = Item.includes(:owner)
              .order(created_at: :desc)
              .page(params[:page]).per(params[:per_page] || 20)

  render_collection_envelope(
    collection: items,
    blueprint: ItemBlueprint
  )
end

Response:

{
  "data": [...],
  "meta": {
    "page": 1,
    "per_page": 20,
    "total_pages": 5,
    "total_count": 100
  }
}

Create with Validation

def create
  item = current_user.items.new(item_params)

  if item.save
    render_success_envelope(
      message: "Item created successfully",
      resource: item,
      blueprint: ItemBlueprint,
      status: :created
    )
  else
    render_validation_errors(item)
  end
end

Success (201):

{
  "success": true,
  "message": "Item created successfully",
  "data": { ... }
}

Validation Error (422):

{
  "error": "Title can't be blank and Price must be greater than 0",
  "errors": {
    "title": ["can't be blank"],
    "price": ["must be greater than 0"]
  }
}

Spec Coverage Requirements

Every implementation ships with:

Request spec for each endpoint (success + ALL documented error cases)
RSwag integration spec for Swagger generation
Policy spec for authorization rules
Model spec for validations/associations/scopes
Blueprint spec when output shape is complex

Regression Prevention

# Before modifying existing code:
# 1. Run existing specs → establish baseline (all green)
# 2. Make changes
# 3. Run specs again → ALL prior specs must still pass
# 4. If any fail, treat as regression and fix before proceeding

Why This Works

Without Framework	With Framework
“It works on my machine”	Verification gates catch issues
Frontend discovers missing fields	Contract compliance audit prevents gaps
v1.1 breaks v1.0	Cumulative version compliance
Docs are always stale	Docs generated from actual implementation
AI generates plausible bugs	AI follows explicit contracts

The Efficiency Multiplier

This framework turns AI from a liability into an accelerator:

Fewer review cycles — Envelopes, pagination, and auth are consistent from the start
Predictable output — AI synthesizes code that matches contracts
Lower context cost — One authoritative source means less ambiguity
Safer by default — Verification gates catch errors before merge
Institutional memory — Documentation survives team changes

Quick Reference

Response Envelopes:

List: { data: [...], meta: { page, per_page, total_pages, total_count } }
Success: { success: true, message: "...", data: {...} }
Error: { error: "...", errors: {...} }

Status Codes:

200 → OK (read, update, delete)
201 → Created
400 → Bad request
401 → Unauthorized
403 → Forbidden
404 → Not found
422 → Validation failed

Discrepancy Tags:

[IMPL] → Fix your code
[DOC] → Flag for doc owner

Safe Defaults:

String → ""
Number → 0
Boolean → false
Array → []
Timestamp → Time.current.iso8601

Conclusion

Zero-gap development isn’t about perfection—it’s about systematic verification. By treating TypeScript interfaces as contracts, running verification gates religiously, and generating documentation from working code, you eliminate entire categories of bugs.

AI amplifies whatever process you give it. Give it contracts, gates, and verification—and it becomes the best pair programmer you’ve ever had.

The framework is the force multiplier. Experience is choosing to use it.

Building a Browser-Based MMORPG with Ruby on Rails: Documentation-Driven Development with AI

Thu, 27 Nov 2025 00:00:00 +0000

“The hardest part of building a game isn’t the code—it’s keeping 50+ interconnected systems coherent while shipping features fast.”

Most game development tutorials show you how to build a simple demo. Real games need authentication, real-time combat, economies, social systems, moderation, and dozens of other features that must work together seamlessly.

In this post, I’ll share our experience building Elselands—a browser-based, turn-based MMORPG inspired by classic games like Neverlands.ru. We’ll cover:

Documentation-driven development that scales with complexity
Feature decomposition from Game Design Document to implementation
Flow documentation as the source of truth for implementation
“Inspired-by” patterns for borrowing proven mechanics
Testing strategies for game systems
AI-assisted development that actually works for senior engineers
Practical lessons from building 100+ interconnected systems

1. The Challenge: Building an MMORPG as a Rails Monolith

1.1 Why Rails for a Game?

Browser-based MMORPGs have unique requirements:

Requirement	Why Rails Works
Server-authoritative logic	All game state lives in PostgreSQL, no client-side cheating
Real-time updates	ActionCable + Redis pub/sub handles chat, combat, presence
Rapid iteration	Convention over configuration means less boilerplate
Rich UI without SPA complexity	Hotwire (Turbo + Stimulus) delivers reactive UIs server-side
Moderation & admin tools	ActiveAdmin, Flipper, audit logging come free

Our stack:

Ruby 3.4.4 + Rails 8.1.1 (full-stack Hotwire monolith)
PostgreSQL 18 (primary datastore)
Redis (cache, Action Cable, Sidekiq)
Hotwire (Turbo Frames, Turbo Streams, Stimulus)
Sidekiq 8 (background jobs for combat/chat/events)

1.2 The Complexity Problem

A typical MMORPG has:

10+ major systems (auth, combat, economy, crafting, quests, etc.)
50+ models with complex relationships
Hundreds of services, jobs, and controllers
Real-time features requiring WebSockets
Moderation, analytics, and admin tooling

Without a structured approach, this becomes unmaintainable. We solved it with documentation-driven development.

2. Documentation-Driven Development: From GDD to Implementation

2.1 The Documentation Hierarchy

We structured our documentation in three layers:

doc/
├── design/
│   └── gdd.md              # Game Design Document (WHAT we're building)
├── features/
│   ├── 0_technical.md      # Infrastructure & architecture
│   ├── 1_auth.md           # Authentication & accounts
│   ├── 3_player.md         # Character systems
│   ├── 8_gameplay_mechanics.md
│   └── ...                 # One file per major system
└── flow/
    ├── 0_technical.md      # HOW each feature is implemented
    ├── 1_auth_presence.md
    ├── 8_gameplay_mechanics.md
    └── ...                 # Implementation details with file references

Key insight: The GDD tells you what to build. Feature docs break it into systems. Flow docs tell you exactly how it’s implemented.

2.2 The Game Design Document (GDD)

Our GDD is intentionally high-level:

## Gameplay Mechanics
- **Player Movement**
  - Tile-based grid movement
  - Turn-based actions per player input

- **Combat System**
  - PvE: Encounters against monsters and NPCs
  - PvP: Player duels, group battles, arena tournaments

- **Character Progression**
  - Experience points and leveling
  - Stat points allocation per level
  - Reputation and faction alignment

This is the north star—it never mentions implementation details like controllers or database schemas. It’s what you show stakeholders and use for high-level planning.

2.3 Feature Documentation: Breaking Down the GDD

Each major system gets its own feature doc (doc/features/*.md):

# 8_gameplay_mechanics.md

## Player Movement
- Grid-based tile system with terrain modifiers
- Turn-per-action enforcement (server-side)
- Cooldown multipliers for different terrain types
- Spawn points and respawn timers

## Combat System
- Turn-based initiative order
- Body-part targeting (head, torso, stomach, legs)
- Action points and mana management
- Status effects and buff/debuff system

Feature docs are implementation-ready specifications—detailed enough that an engineer knows what to build, but not yet tied to specific files.

2.4 Flow Documentation: The Implementation Source of Truth

This is where the magic happens. Flow docs (doc/flow/*.md) describe exactly how each feature is implemented:

# 8_gameplay_mechanics.md — Flow Documentation

## Combat Skill System

### Use Case: Player uses a skill during combat
1. Player clicks skill icon → `turn_combat_controller.js#useSkill()`
2. Stimulus sends action to `CombatController#action`
3. `TurnBasedCombatService` validates action points and mana
4. `Game::Combat::SkillExecutor` applies skill effects
5. `CombatLogEntry` records the action
6. Turbo Stream broadcasts update to all participants

### Key Behaviors
- Skills consume both action points AND mana slots
- Body-part targeting affects damage multipliers
- Critical hits based on agility vs target defense

### Responsible for Implementation Files
| Purpose | File |
|---------|------|
| Combat service | `app/services/game/combat/turn_based_combat_service.rb` |
| Skill executor | `app/services/game/combat/skill_executor.rb` |
| Stimulus controller | `app/javascript/controllers/turn_combat_controller.js` |
| Combat view | `app/views/combat/_battle.html.erb` |
| Action config | `config/gameplay/combat_actions.yml` |

Why flow docs matter:

Context preservation — When returning to a feature after weeks, you know exactly where everything is
AI assistance — Feed the flow doc to your AI assistant, and it understands the entire system
Onboarding — New engineers can trace any feature from UI to database
Code review — Reviewers can verify changes match the documented architecture

3. The “Inspired-By” Pattern: Borrowing Proven Mechanics

3.1 Why Reinvent the Wheel?

Classic games like Neverlands.ru have battle-tested UI/UX patterns. Instead of designing from scratch, we:

Analyze the original implementation (JavaScript, CSS, HTML)
Document the original patterns in doc/features/neverlands_inspired.md
Adapt to modern Rails/Hotwire patterns
Track implementation status

3.2 Example: Turn-Based Combat System

We received this Neverlands JavaScript:

// Original Neverlands combat logic
function CountOD() {
  var totalOD = 0;
  for (var i = 0; i < 10; i++) {
    if (pos_atk[i] > 0) totalOD += pos_ochd[pos_atk[i]];
    if (pos_blk[i] > 0) totalOD += pos_ochd[pos_blk[i]];
  }
  // pos_ochd contains action point costs per body part
  return totalOD;
}

// Body part configuration
var pos_vars = ['', 'Head', 'Torso', 'Stomach', 'Groin', 'Left Leg', 'Right Leg'];
var pos_ochd = [0, 3, 2, 2, 2, 2, 2]; // Action point costs

3.3 Our Adaptation

We translated this to a Rails service with YAML configuration:

# config/gameplay/combat_actions.yml
body_parts:
  head:
    action_cost: 3
    damage_multiplier: 1.5
    block_bonus: 0.8
  torso:
    action_cost: 2
    damage_multiplier: 1.0
    block_bonus: 1.0
  stomach:
    action_cost: 2
    damage_multiplier: 1.1
    block_bonus: 0.9

# app/services/game/combat/turn_based_combat_service.rb
class Game::Combat::TurnBasedCombatService
  def initialize(battle)
    @battle = battle
    @config = YAML.load_file(Rails.root.join("config/gameplay/combat_actions.yml"))
  end

  def calculate_action_points_used(attacks, blocks)
    total = 0
    attacks.each { |part| total += @config.dig("body_parts", part, "action_cost") || 2 }
    blocks.each { |part| total += @config.dig("body_parts", part, "action_cost") || 2 }
    total
  end
end

And a Stimulus controller replacing the inline JavaScript:

// app/javascript/controllers/turn_combat_controller.js
import { Controller } from "@hotwired/stimulus"

export default class extends Controller {
  static targets = ["attackSelect", "blockSelect", "actionPoints", "submitButton"]
  static values = { maxActions: Number, config: Object }

  connect() {
    this.updateActionPoints()
  }

  updateActionPoints() {
    const used = this.calculateUsedPoints()
    const remaining = this.maxActionsValue - used

    this.actionPointsTarget.textContent = `${remaining}/${this.maxActionsValue}`
    this.submitButtonTarget.disabled = remaining < 0
  }

  calculateUsedPoints() {
    let total = 0
    this.attackSelectTargets.forEach(select => {
      if (select.value) total += this.configValue.body_parts[select.value]?.action_cost || 2
    })
    this.blockSelectTargets.forEach(select => {
      if (select.value) total += this.configValue.body_parts[select.value]?.action_cost || 2
    })
    return total
  }
}

3.4 Documentation Structure for Inspired Features

We maintain a dedicated file tracking all borrowed patterns:

# doc/features/neverlands_inspired.md

## Implementation Status Summary

| # | Feature | Status | Key Files |
|---|---------|--------|-----------|
| 1 | Chat System | ✅ Implemented | `chat_controller.js`, `moderation_service.rb` |
| 2 | Arena/PvP | ✅ Implemented | `arena_controller.rb`, `matchmaker.rb` |
| 3 | Turn-Based Combat | ✅ Implemented | `turn_based_combat_service.rb` |
| 4 | Combat Log | ✅ Implemented | `log_builder.rb`, `statistics_calculator.rb` |

## Chat System

### Original Neverlands Examples
[Original JavaScript code here]

### Elselands Implementation
- **Files:** `chat_controller.js`, `realtime_chat_channel.rb`
- **Key Adaptations:**
  - Replaced `onclick` with Stimulus `data-action`
  - WebSocket via ActionCable instead of polling
  - Server-side moderation before broadcast

Benefits:

Reference material — Original examples are preserved for future iterations
Implementation tracking — Clear status of what’s done vs. planned
Knowledge transfer — Anyone can understand why we made certain choices

4. Testing Strategies for Game Systems

4.1 Test Categories

Game systems require different testing strategies than typical CRUD apps:

Category	Purpose	Example
Unit Tests	Individual service logic	`TurnBasedCombatService` action point calculation
Request Tests	API endpoints	`POST /combat/action` with attack data
Channel Tests	WebSocket functionality	`BattleChannel` broadcasts updates
Helper Tests	View helpers	`format_combat_log_message` output
System Tests	End-to-end flows	Full combat from start to victory

4.2 Testing Real-Time Features

ActionCable channels need special setup:

# spec/channels/battle_channel_spec.rb
require "rails_helper"

RSpec.describe BattleChannel, type: :channel do
  let(:user) { create(:user) }
  let(:battle) { create(:battle) }

  before do
    stub_connection current_user: user
  end

  it "subscribes to battle stream" do
    subscribe(battle_id: battle.id)
    expect(subscription).to be_confirmed
    expect(subscription).to have_stream_for(battle)
  end

  it "receives combat updates" do
    subscribe(battle_id: battle.id)

    expect {
      BattleChannel.broadcast_to(battle, { type: "turn_update", round: 1 })
    }.to have_broadcasted_to(battle)
  end
end

4.3 Testing Deterministic Combat

Combat must be reproducible for debugging and replays:

# spec/services/game/combat/turn_based_combat_service_spec.rb
RSpec.describe Game::Combat::TurnBasedCombatService do
  let(:battle) { create(:battle, :with_participants) }
  let(:service) { described_class.new(battle) }

  describe "#submit_turn" do
    it "consumes action points correctly" do
      attacks = [{ body_part: "head" }, { body_part: "torso" }]
      blocks = [{ body_part: "stomach" }]

      # Head (3) + Torso (2) + Stomach (2) = 7 action points
      expect(service.calculate_action_points_used(attacks, blocks)).to eq(7)
    end

    context "with seeded RNG for reproducibility" do
      it "produces consistent damage rolls" do
        Game::Utils::Rng.seed(12345)

        result1 = service.resolve_attack(attacker, defender, "head")
        Game::Utils::Rng.seed(12345)
        result2 = service.resolve_attack(attacker, defender, "head")

        expect(result1[:damage]).to eq(result2[:damage])
      end
    end
  end
end

4.4 Coverage Goals

We maintain explicit coverage targets:

Category	Target	Rationale
Models	90%	Core game logic lives here
Services	95%	Business logic must be bulletproof
Controllers	85%	Request/response handling
Channels	80%	Real-time features are critical
Helpers	90%	UI consistency
Views	60%	Basic rendering verification
System	40%	Key user flows

5. AI-Assisted Development: Making It Work for Senior Engineers

5.1 The Problem with Generic AI Assistance

Most AI coding assistants fail on complex projects because they lack:

Project context — They don’t know your architecture
Convention awareness — They generate code that doesn’t match your style
Domain knowledge — They don’t understand game mechanics

5.2 Our Solution: Structured Prompting with Documentation

We created a documentation hierarchy that the AI can reference:

# README.md (excerpt)

## 📄 Documentation Map

| File | When to Reference / Purpose |
|------|----------------------------|
| **AGENT.md** | Always loaded, highest authority |
| **GUIDE.md** | Rails standards or general best practices |
| **MMO_ADDITIONAL_GUIDE.md** | Gameplay/MMORPG domain-specific conventions |
| **doc/gdd.md** | Game design vision, classes, mechanics |
| **doc/features/*.md** | Per-system breakdown (technical specs) |
| **doc/flow/*.md** | Implementation details with file references |

5.3 Effective Prompting Patterns

Pattern 1: Feature Implementation

I want to start implementing the Player feature from `doc/features/3_player.md`.

Please follow:
- `AGENT.md` (always)
- `GUIDE.md` for general Rails patterns
- `MMO_ADDITIONAL_GUIDE.md` for gameplay logic architecture
- Use `MMO_ENGINE_SKELETON.md` for engine placement

**Task:**
1. Read `doc/features/3_player.md`
2. Identify required models, services, and UI components
3. Provide a detailed plan (files + responsibilities)
4. Wait for my confirmation before writing any code

Pattern 2: Inspired-By Implementation

Here's the Neverlands JavaScript for their arena system:
[paste code]

Please:
1. Analyze the original implementation
2. Document it in `doc/features/neverlands_inspired.md`
3. Propose a Rails/Hotwire adaptation using our patterns
4. Update flow docs with implementation details

Pattern 3: Bug Fix with Context

I'm seeing this error in the arena system:
[paste error]

Context:
- Check `doc/flow/11_arena_pvp.md` for the arena architecture
- The ArenaMatch model uses an enum that might conflict

Find the root cause and propose a fix that matches our conventions.

5.4 Why This Works for Senior Engineers

You drive the architecture — AI implements your decisions, doesn’t make them
Documentation is the contract — Changes must match documented patterns
Context is preserved — Flow docs keep AI informed across sessions
Code review is efficient — You verify against documented expectations

5.5 Prompts That Accelerate Development

For rapid prototyping:

Using AGENT.md rules, implement the next step from doc/features/3_player.md.
Use GUIDE.md for Rails logic, and MMO_ADDITIONAL_GUIDE.md for gameplay structure.
Touch only the necessary files.

For code quality:

After implementation, run:
1. `bundle exec rspec` for tests
2. `bundle exec standardrb` for linting
3. `bundle exec brakeman` for security

Fix any issues before considering this complete.

For documentation updates:

Update the corresponding flow doc (`doc/flow/X.md`) with:
- Use cases covered
- Key behaviors implemented
- Responsible files table

6. Practical Lessons from 100+ Systems

6.1 What Worked

1. Flow docs as the source of truth

Every PR must update the relevant flow doc. This creates a living architecture document that’s always accurate.

2. “Responsible Files” tables in flow docs

### Responsible for Implementation Files
| Purpose | File |
|---------|------|
| Combat service | `app/services/game/combat/turn_based_combat_service.rb` |
| Controller | `app/controllers/combat_controller.rb` |
| Stimulus | `app/javascript/controllers/turn_combat_controller.js` |

This makes it trivial to find code when debugging or extending features.

3. YAML configuration for game data

Instead of hardcoding values, we use config/gameplay/*.yml:

combat_actions.yml — Body parts, action costs, multipliers
terrain_modifiers.yml — Movement cooldowns by terrain
biomes.yml — Encounter tables by region

Changes don’t require code deployments, and game designers can read/modify the configs.

4. Turbo Streams for real-time without complexity

Instead of building a custom WebSocket protocol:

# Broadcasting combat updates
Turbo::StreamsChannel.broadcast_update_to(
  @battle,
  target: "combat-log",
  partial: "combat_logs/entry",
  locals: { entry: log_entry }
)

The frontend gets updates automatically without custom JavaScript.

6.2 What We’d Do Differently

1. Start with comprehensive factories earlier

We had to backfill factories for models like NpcTemplate, WebhookEndpoint, etc. Building factories alongside models saves debugging time.

2. Enum naming requires foresight

ActiveRecord reserves certain method names (group, order, etc.). We had to rename fight_type: :group to fight_type: :team_battle to avoid conflicts.

3. Migration idempotency from day one

Adding if column_exists? checks to migrations would have prevented issues when re-running migrations:

def change
  unless column_exists?(:battles, :share_token)
    add_column :battles, :share_token, :string
  end
end

6.3 Performance Considerations

Real-time features need Redis pub/sub:

# config/cable.yml
production:
  adapter: redis
  url: <%= ENV.fetch("REDIS_CABLE_URL") %>
  channel_prefix: elselands_production

Separate Redis instances for different concerns:

Purpose	Environment Variable
Rails cache	`REDIS_CACHE_URL`
Sidekiq queues	`REDIS_SIDEKIQ_URL`
Action Cable	`REDIS_CABLE_URL`

This prevents a Sidekiq backup from affecting real-time chat.

7. The Complete Development Workflow

7.1 Adding a New Feature

1. GDD → Does this feature align with the game vision?
       ↓
2. Feature Doc → Create doc/features/X.md with:
   - System requirements
   - User-facing functionality
   - Integration points
       ↓
3. Flow Doc → Create doc/flow/X.md with:
   - Use cases (step-by-step flows)
   - Key behaviors
   - "Responsible Files" table (initially planned)
       ↓
4. Implementation → Using AI + documentation:
   - Generate models/migrations
   - Create services with tests
   - Build controllers and views
   - Add Stimulus controllers for interactivity
       ↓
5. Testing → Run full test suite:
   - bundle exec rspec
   - bundle exec standardrb
   - bundle exec brakeman
       ↓
6. Documentation Update → Update flow doc with:
   - Actual file paths
   - Implementation notes
   - Any deviations from plan

7.2 Debugging with Documentation

When something breaks:

Find the flow doc for that feature
Trace the use case — Which step is failing?
Check the “Responsible Files” — Go directly to the right code
Verify against “Key Behaviors” — Is the code doing what it should?

7.3 Onboarding New Engineers

Read README.md for project overview
Read AGENT.md and GUIDE.md for conventions
Pick a feature, read its flow doc
Trace from UI to database using “Responsible Files”
Make a small change, run tests, verify

8. Conclusion: Documentation as a Force Multiplier

Building a complex game is hard. Building it fast and maintaining quality is harder. Our documentation-driven approach made it possible:

Challenge	Solution
Keeping 100+ systems coherent	Feature docs define boundaries
Finding code in a large codebase	Flow docs with “Responsible Files”
Preserving context across sessions	Flow docs are always up-to-date
Effective AI assistance	Structured prompts with documentation
Onboarding new engineers	Self-documenting architecture
Borrowing proven patterns	“Inspired-by” documentation

The upfront investment in documentation pays dividends throughout the project. Every hour spent documenting saves five hours of confusion later.

Key Takeaways

GDD → Features → Flows creates a traceable path from vision to implementation
Flow docs with “Responsible Files” make code discoverable
“Inspired-by” patterns let you borrow proven mechanics responsibly
AI works best with structured context — feed it your documentation
Tests are non-negotiable for game systems with complex interactions
YAML configuration separates game data from code logic

Whether you’re building an MMORPG, a complex SaaS, or any large Rails application, these patterns will help you ship faster while maintaining quality.

Resources

Rails 8.1 + Hotwire: hotwired.dev
ActionCable Guide: guides.rubyonrails.org/action_cable_overview.html
Stimulus Handbook: stimulus.hotwired.dev/handbook
Flipper Feature Flags: github.com/flippercloud/flipper

This post was written while building Elselands—a browser-based MMORPG recreating the classic Neverlands.ru experience with modern Rails tooling. The documentation-driven approach described here enabled rapid development of 9 major Neverlands-inspired systems, 100+ models, and comprehensive test coverage in a fraction of the time traditional development would require.

From Theory to Production: Reliable WebSockets with Ruby on Rails, JWT & ActionCable

Wed, 19 Nov 2025 00:00:00 +0000

“WebSockets are easy… until you have to make them reliable and scalable.”

Most WebSocket tutorials stop at building a simple chat demo. But real systems need authentication, reliability, scalability, and good architecture.

In this post we’ll connect:

The WebSocket fundamentals & best practices from three in-depth articles fileciteturn0file0turn0file1turn0file2
A real-world Rails API backend using Devise + JWT
ActionCable as the WebSocket layer
Concrete code examples
Extra use cases, WebSocket vs pub/sub comparisons, and sharding/scaling patterns

1. WebSockets in a Nutshell (For Rails Developers)

1.1 What is a WebSocket?

A WebSocket is a full-duplex, persistent TCP connection between client and server. After a one-time HTTP handshake (using Upgrade: websocket), both sides can send messages at any time, without repeated HTTP requests. fileciteturn0file0

Compared to classic HTTP:

Feature	HTTP	WebSocket
Connection	Short-lived per request	Long-lived, persistent
Direction	Client → Server (request)	Bidirectional
Latency	Higher (headers + handshakes)	Lower after initial upgrade
Model	Request/response	Event/message-based

WebSockets shine in realtime use cases such as chat, multiplayer games, collaborative editing, IoT telemetry, live dashboards, and more. fileciteturn0file0

1.2 The Less Glamorous Bits: Reliability & Scaling

Out of the box, WebSockets don’t give you: fileciteturn0file1turn0file2

Delivery guarantees (messages may be lost on disconnect)
Message ordering guarantees
Automatic reconnection
Backpressure / rate limiting
Fault tolerance across multiple servers

At scale, you also run into: fileciteturn0file2

Stateful connections (harder load balancing than stateless HTTP)
N² message explosion (many clients talking to many clients)
Backpressure (slow consumers, fast producers)
Global distribution & redundancy requirements

The WebSocket articles break these down into architectural and operational best practices: sharding, sticky sessions, pub/sub, backpressure, monitoring, and fault tolerance. fileciteturn0file2

The good news: Rails + ActionCable + Redis give us a solid foundation. We just need to wire things correctly and add a few patterns.

2. WebSockets vs Pub/Sub (and HTTP): Who Does What?

It’s easy to confuse WebSockets and pub/sub as alternatives. In reality they solve different layers of the problem.

HTTP – Request/response, great for CRUD APIs, not great for realtime.
WebSocket – A transport for bidirectional, low-latency communication. fileciteturn0file0
Pub/Sub – A messaging pattern (and often a dedicated system) for fanning messages out to many consumers. fileciteturn0file2

Think of it like this:

WebSocket: “How do I keep the pipe open between browser and backend?”
Pub/Sub: “How do I deliver the right messages to the right set of subscribers?”

A typical production system uses both:

Browsers connect to your backend over WebSockets.
Each backend node connects to a pub/sub layer (Redis, Kafka, Ably, etc.). fileciteturn0file2
When any node publishes an event to a topic, the pub/sub layer delivers it to all interested nodes, which then forward it to their connected WebSocket clients.

2.1 Comparison Table

Aspect	WebSocket	Pub/Sub
Scope	Transport between one client & one server	Logical messaging between many producers/consumers
Direction	Bidirectional	Usually one-way per message (publish → subscribe)
Responsibility	Keep connection open & deliver raw frames	Route messages to interested subscribers
Scaling focus	Concurrent connections, keep-alives	Fan-out, throughput, durability, ordering
In Rails	ActionCable connections & channels	Redis ActionCable adapter, message buses

In Rails, ActionCable + Redis is basically WebSocket + pub/sub out of the box. ActionCable uses Redis to broadcast between server processes, following the pub/sub architecture pattern recommended in the best-practices article. fileciteturn0file2

3. Rails Stack Overview

We’ll assume a stack like this:

Rails API-only app
Devise + devise-jwt for authentication
ActionCable for WebSockets
Redis as ActionCable’s pub/sub backend
A frontend (Rails, React, Vue, etc.) that:
- Uses JWT for HTTP API auth
- Uses the same JWT to authenticate WebSocket connections

Let’s wire this up step-by-step.

4. Authenticating WebSockets with Devise + JWT

The articles highlight that WebSocket itself doesn’t define an auth mechanism — it’s up to you to layer one on top (cookies, JWT, etc.). fileciteturn0file0

In a Rails API-only app, JWT is a natural choice.

4.1 Pass JWT to ActionCable

Frontend connects like this:

// Example with ActionCable JS (Rails 7+)
import * as ActionCable from "@rails/actioncable";

const jwt = localStorage.getItem("jwt"); // issued by Devise + devise-jwt

const cable = ActionCable.createConsumer(
  `wss://api.example.com/cable?token=${encodeURIComponent(jwt)}`
);

const subscription = cable.subscriptions.create(
  { channel: "ChatChannel", room_id: 42 },
  {
    received: (data) => console.log("New message:", data),
  }
);

We attach the JWT as a query parameter (token=) so it’s accessible during the WebSocket handshake.

4.2 Decode JWT in `ApplicationCable::Connection`

ApplicationCable::Connection is the entry point for all WebSocket connections. This is where we validate the JWT and reject unauthorized users.

# app/channels/application_cable/connection.rb
module ApplicationCable
  class Connection < ActionCable::Connection::Base
    identified_by :current_user

    def connect
      self.current_user = find_verified_user
    end

    private

    def find_verified_user
      token = request.params[:token]

      raise "Missing token" if token.blank?

      # Using devise-jwt under the hood
      payload = Warden::JWTAuth::TokenDecoder.new.call(token)
      user = User.find_by(id: payload["sub"])

      user || reject_unauthorized_connection
    rescue => e
      Rails.logger.warn("[ActionCable] Unauthorized connection: #{e.message}")
      reject_unauthorized_connection
    end
  end
end

This gives us:

A per-connection current_user
The same identity model as our HTTP API

It follows the article’s recommendation to handle authentication at the handshake and/or application levels. fileciteturn0file0

5. ActionCable Channels: From Chat to General Realtime

Let’s start with the “classic” chat example, then generalize.

5.1 Basic Chat Channel

# app/channels/chat_channel.rb
class ChatChannel < ApplicationCable::Channel
  def subscribed
    @room = Room.find(params[:room_id])
    # Authorization example
    reject unless @room.users.include?(current_user)

    stream_for @room
  end

  def receive(data)
    # Simple example: create message & broadcast
    message = @room.messages.create!(
      user: current_user,
      content: data["content"]
    )

    ChatChannel.broadcast_to(
      @room,
      {
        id: message.id,
        content: message.content,
        user_id: current_user.id,
        created_at: message.created_at.iso8601
      }
    )
  end
end

Corresponding model:

# app/models/message.rb
class Message < ApplicationRecord
  belongs_to :room
  belongs_to :user

  validates :content, presence: true
end

This gives you a basic authenticated chat over WebSockets. But to align with the articles’ reliability recommendations, we need to go further.

6. Reliability Patterns in Rails (From the Articles)

6.1 Heartbeats & Reconnection

ActionCable already supports heartbeat pings and automatic reconnection in the JS client, matching the best practice of keep-alives and reconnection logic. fileciteturn0file2

On the client (simplified):

// ActionCable automatically tries to reconnect on close.
// You can add logging:
cable.connection.monitor.reconnectAttempts = 0;

const originalDisconnected = cable.connection.disconnected;

cable.connection.disconnected = function(...args) {
  cable.connection.monitor.reconnectAttempts += 1;
  console.warn("WebSocket disconnected – attempt",
               cable.connection.monitor.reconnectAttempts);
  originalDisconnected.apply(this, args);
};

6.2 Backpressure & Rate Limiting

The articles emphasize backpressure and throttling “greedy clients”. fileciteturn0file2

Rails doesn’t do this automatically; you can add simple rate limiting per connection.

# app/channels/application_cable/channel.rb
module ApplicationCable
  class Channel < ActionCable::Channel::Base
    RATE_LIMIT_WINDOW = 5.seconds
    RATE_LIMIT_MAX    = 20 # messages per window

    def rate_limited?
      key   = "ws:rate_limit:#{current_user.id}:#{self.class.name}"
      data  = Rails.cache.read(key) || { count: 0, started_at: Time.current }

      if Time.current - data[:started_at] > RATE_LIMIT_WINDOW
        data = { count: 0, started_at: Time.current }
      end

      data[:count] += 1
      Rails.cache.write(key, data, expires_in: RATE_LIMIT_WINDOW * 2)

      data[:count] > RATE_LIMIT_MAX
    end
  end
end

Use inside a channel:

class ChatChannel < ApplicationCable::Channel
  def receive(data)
    if rate_limited?
      Rails.logger.warn("Rate limit exceeded by user #{current_user.id}")
      return
    end

    # ...create message and broadcast
  end
end

6.3 Message Ordering & Deduplication

One article stresses that WebSockets don’t guarantee message ordering or delivery. fileciteturn0file1turn0file2

A simple pattern:

Add a monotonic sequence per room or stream.
Include sequence in the payload.
Clients discard old or duplicate sequence numbers.

Migration:

class AddSequenceToMessages < ActiveRecord::Migration[7.1]
  def change
    add_column :messages, :sequence, :bigint, null: false, default: 0
    add_index :messages, [:room_id, :sequence], unique: true
  end
end

Assign sequence:

# app/models/message.rb
class Message < ApplicationRecord
  belongs_to :room
  belongs_to :user

  before_create :assign_sequence

  private

  def assign_sequence
    self.sequence = (room.messages.maximum(:sequence) || 0) + 1
  end
end

Broadcast:

ChatChannel.broadcast_to(
  @room,
  {
    id: message.id,
    sequence: message.sequence,
    content: message.content,
    user_id: current_user.id
  }
)

Client side, you can store lastSequence per room and ignore messages with sequence <= lastSequence.

6.4 Horizontal Scaling with Redis & Sticky Sessions

The articles talk about scaling using pub/sub, sticky sessions, and load balancing. fileciteturn0file2

`config/cable.yml`

production:
  adapter: redis
  url: <%= ENV.fetch("REDIS_URL") %>
  channel_prefix: myapp_production

This uses Redis to propagate messages across all Rails instances, matching the pub/sub architectural pattern recommended in the best-practices article. fileciteturn0file2

At the load balancer:

Make sure WebSocket upgrade is supported.
Enable stickiness so a given client reconnects to the same instance (or at least for the duration of a session), following the sticky-session pattern described in the article. fileciteturn0file2

Example snippet for Nginx (simplified):

location /cable {
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "Upgrade";
  proxy_set_header Host $host;

  proxy_pass http://rails_upstream;
}

Stickiness is usually configured at the reverse proxy / load balancer level (NGINX upstream hash, AWS ALB target group stickiness, etc.).

6.5 Monitoring & Metrics

Operational best practices in the articles include monitoring, autoscaling, and alerts. fileciteturn0file2

In Rails you can log:

Connected users
Active subscriptions
Broadcast frequency
Error rates / disconnect reasons

Example:

# app/channels/application_cable/connection.rb
def connect
  super
  Rails.logger.info("[ActionCable] Connect user=#{current_user.id}")
end

def disconnect(reason)
  Rails.logger.info("[ActionCable] Disconnect user=#{current_user&.id} reason=#{reason}")
end

Hook these logs into something like Loki, Datadog, New Relic, etc., and build dashboards like “active connections”, “messages per second”, etc., as suggested by the reliability article. fileciteturn0file1turn0file2

7. Sharding & Scalability Patterns (with Rails Examples)

The best-practices article goes deep into sharding, sticky sessions, and pub/sub as ways to scale. fileciteturn0file2 Here are some concrete patterns for Rails.

7.1 Simple Horizontal Scaling (Single Region)

For many apps you can start with:

Multiple Rails web instances (Puma/Unicorn)
Shared Redis for ActionCable
One regional load balancer with stickiness

Diagram in words:

Browser → Load Balancer → [Rails + ActionCable + Redis]

All channels are available from all instances because they share Redis.

This can take you surprisingly far before you need more exotic sharding.

7.2 Tenant / Region Sharding

Once you serve users in multiple regions (e.g. EU + US), latency becomes a concern. The article describes this kind of challenge as WebSocket connections are stateful and global distribution is hard. fileciteturn0file2

You can shard by region or tenant:

wss://eu.example.com/cable → EU cluster (Rails + Redis in EU)
wss://us.example.com/cable → US cluster (Rails + Redis in US)

Routing strategy (very simplified JS):

function websocketHostForUser(user) {
  if (user.region === "eu") return "wss://eu.example.com/cable";
  return "wss://us.example.com/cable";
}

Each cluster only holds WebSockets for its tenants. Cross-region communication can then happen via:

Async jobs / message bus
Periodic synchronization of data

This matches the “shard by namespace” idea in the article: each shard owns part of the client namespace. fileciteturn0file2

7.3 Channel / Topic Sharding

Another approach is to shard by channel/topic type.

Example:

Cluster A: all chat channels
Cluster B: all analytics/metrics channels
Cluster C: all IoT/device channels

Routing example (pseudocode):

function cableUrlForChannel(channelName) {
  if (channelName.endsWith("ChatChannel")) {
    return "wss://chat.example.com/cable";
  } else if (channelName.endsWith("MetricsChannel")) {
    return "wss://metrics.example.com/cable";
  }
  return "wss://realtime.example.com/cable";
}

This reduces the N² problem for some workloads because each cluster sees a smaller subset of traffic.

7.4 N² Problem in Plain Numbers

From the article: as client count grows, potential interactions grow roughly with N². fileciteturn0file2

100 users in a room → up to ~10,000 directed message paths
1,000 users → up to 1,000,000 paths

If every message potentially fans out to many clients, traffic explodes.

Mitigation techniques:

Aggregation: instead of sending 1000 individual “👍” reactions, send one { emoji: "👍", count: 1000 }. fileciteturn0file2
Batching: send updates in periodic batches (every 100 ms) instead of instantly one-by-one.
Scoped rooms: split giant rooms into smaller subrooms (e.g., per topic, per segment).

Rails implementation example for batched updates (pseudo):

# app/services/metrics_buffer.rb
class MetricsBuffer
  def self.buffer(metric)
    redis.lpush("metrics_buffer", metric.to_json)
  end

  def self.flush!
    items = redis.lrange("metrics_buffer", 0, -1)
    redis.del("metrics_buffer")
    data = items.map { |j| JSON.parse(j) }

    ActionCable.server.broadcast("admin_metrics", { type: "batch", data: data })
  end

  def self.redis
    @redis ||= Redis.new(url: ENV["REDIS_URL"])
  end
end

Run MetricsBuffer.flush! every 200ms in a background process or job.

8. Beyond Chat: WebSocket Use Cases in Rails

The first article lists typical WebSocket use cases, many of which are perfect fits for Rails. fileciteturn0file0 Let’s expand the list with practical examples.

8.1 Realtime Notifications

Channel:

# app/channels/notifications_channel.rb
class NotificationsChannel < ApplicationCable::Channel
  def subscribed
    stream_for current_user
  end
end

Broadcast from anywhere:

NotificationsChannel.broadcast_to(
  user,
  { type: "notification", text: "Your report is ready." }
)

Use this for:

In-app alerts
“Toast” messages
System announcements

8.2 Realtime Dashboards (Admin / Ops)

Channel:

# app/channels/admin_metrics_channel.rb
class AdminMetricsChannel < ApplicationCable::Channel
  def subscribed
    reject_unauthorized unless current_user.admin?
    stream_from "admin_metrics"
  end

  private

  def reject_unauthorized
    reject
  end
end

Periodic job:

# app/jobs/broadcast_metrics_job.rb
class BroadcastMetricsJob < ApplicationJob
  queue_as :default

  def perform
    stats = {
      users_online: User.online.count,
      jobs_pending: JobQueue.pending.count,
      # ...
    }

    ActionCable.server.broadcast("admin_metrics", stats)
  end
end

Schedule every N seconds with Sidekiq/GoodJob/solid queue.

8.3 Collaborative Editing / Presence

Channel:

# app/channels/document_channel.rb
class DocumentChannel < ApplicationCable::Channel
  def subscribed
    @document = Document.find(params[:id])
    stream_for @document

    # Presence event (join)
    DocumentChannel.broadcast_to(@document,
      { type: "presence", user_id: current_user.id, event: "joined" }
    )
  end

  def receive(data)
    case data["type"]
    when "cursor"
      DocumentChannel.broadcast_to(@document,
        {
          type: "cursor",
          user_id: current_user.id,
          position: data["position"]
        }
      )
    when "patch"
      # Apply patch, persist, broadcast diff, etc.
    end
  end

  def unsubscribed
    DocumentChannel.broadcast_to(@document,
      { type: "presence", user_id: current_user.id, event: "left" }
    )
  end
end

Clients can show:

Who’s editing
Cursor positions
Live edits

8.4 IoT / Device Streaming

Rails can be the backend that:

Accepts device data via HTTP or MQTT → writes to Redis/Postgres
Streams the latest values to dashboards via WebSockets

Channel:

class DeviceChannel < ApplicationCable::Channel
  def subscribed
    @device = Device.find_by!(public_id: params[:device_id])
    authorize! :read, @device

    stream_for @device
  end
end

When device data comes in:

DeviceChannel.broadcast_to(
  device,
  { type: "reading", temperature: 21.5, humidity: 0.6 }
)

Matches the IoT / telemetry use case mentioned in the article. fileciteturn0file0

8.5 Long-running Jobs / Progress Bars

Instead of polling an endpoint every 2 seconds:

Client submits job via HTTP → gets job_id
Subscribes to JobChannel with that ID
Job broadcasts progress updates

Channel:

class JobChannel < ApplicationCable::Channel
  def subscribed
    @job_id = params[:job_id]
    stream_from "job_#{@job_id}"
  end
end

Job:

class ExportReportJob < ApplicationJob
  def perform(user_id, job_id)
    10.times do |i|
      # Some work...
      ActionCable.server.broadcast(
        "job_#{job_id}",
        { status: "progress", step: i + 1, total: 10 }
      )
    end

    ActionCable.server.broadcast(
      "job_#{job_id}",
      { status: "done", download_url: "/reports/#{job_id}.csv" }
    )
  end
end

8.6 Live Sports & Market Feeds

Live match scores, game clock, and commentary
Betting odds updates
Stock price and order book updates

Rails can ingest upstream feeds (Kafka, external APIs) and push to clients via ActionCable channels, applying the reliability patterns above to prevent overload. fileciteturn0file0turn0file1

8.7 Live Auctions & Bidding

Users see bids appear in realtime.
Auction countdown timers synchronize across all clients.
Anti-sniping strategies can be applied server-side and updates pushed instantly.

Channel example:

class AuctionChannel < ApplicationCable::Channel
  def subscribed
    @auction = Auction.find(params[:id])
    stream_for @auction
  end

  def place_bid(data)
    amount = data["amount"].to_d
    @auction.place_bid!(current_user, amount)
    AuctionChannel.broadcast_to(@auction, {
      type: "bid_placed",
      user_id: current_user.id,
      amount: amount.to_s
    })
  end
end

8.8 Realtime Education & Collaboration

Live quizzes (questions & answers in realtime)
Shared whiteboards (drawing events over WebSockets)
Pair programming or coding interviews

You can model each classroom/session as a channel where:

Teacher broadcasts questions, slides, or code snippets.
Students send answers or reactions.
“Raise hand” events show up for the instructor instantly.

9. When to Build vs When to Use a Managed WebSocket Service

The articles make a good point: running your own WebSocket infra at scale is expensive and complex (time, cost, global distribution, reliability). fileciteturn0file1turn0file2

Rails + ActionCable is great when:

You control your user count / traffic
You’re okay operating your own Redis + WebSocket infra
Most traffic is within a few regions

If you need:

Millions of concurrent connections
Global edge presence
Stronger delivery guarantees (exactly-once, ordering, persistence)
Built-in fallbacks across protocols

…then a managed realtime platform (like Ably, Pusher, etc.) sitting beside your Rails API becomes attractive, as the articles argue. fileciteturn0file1turn0file2

10. Conclusion

Bringing it all together:

WebSockets give us low-latency, bidirectional realtime communication — perfect for far more than just chat. fileciteturn0file0
The articles remind us that reliability, message guarantees, and scaling are not solved by the protocol alone. fileciteturn0file1turn0file2
Rails + ActionCable + Redis + Devise + JWT provide a strong foundation:
- JWT auth at the WebSocket handshake
- Redis-based pub/sub for horizontal scaling
- Channels for modeling realtime domains (chat, notifications, dashboards, IoT, collaboration, auctions, education, and more)
By layering:
- Rate limiting / backpressure
- Sequence numbers and dedup
- Monitoring and metrics
- Sticky sessions or sharding strategies in the load balancer

…you can build production-grade realtime APIs in Rails that align with the WebSocket best practices described in the articles.

Drop this file into your Jekyll blog’s _posts (renaming it with the proper date + slug), tweak the title/metadata, and you’ve got a comprehensive, code-heavy guide to WebSockets with Rails ready to publish.