Problem Statement
At Uber, we process millions of ride requests daily. Each request involves geocoding, pricing calculations, and driver matching. Implementing these as sequential operations would be disastrously slow. Design and implement a Worker Pool that processes jobs concurrently with a fixed number of workers.
Requirements
Implement a worker pool with the following specifications:
- A fixed number of worker goroutines (configurable)
- A jobs channel to receive work
- A results channel to send completed work
- Graceful shutdown when all jobs are processed
Example Implementation
package main
import (
"fmt"
"sync"
)
type Job struct {
ID int
Payload string
}
type Result struct {
JobID int
Output string
}
func worker(id int, jobs <-chan Job, results chan<- Result, wg *sync.WaitGroup) {
defer wg.Done()
for job := range jobs {
// Process the job
output := fmt.Sprintf("Worker %d processed: %s", id, job.Payload)
results <- Result{JobID: job.ID, Output: output}
}
}
func main() {
numWorkers := 5
numJobs := 100
jobs := make(chan Job, numJobs)
results := make(chan Result, numJobs)
var wg sync.WaitGroup
// Start workers
for w := 1; w <= numWorkers; w++ {
wg.Add(1)
go worker(w, jobs, results, &wg)
}
// Send jobs
for j := 1; j <= numJobs; j++ {
jobs <- Job{ID: j, Payload: fmt.Sprintf("job-%d", j)}
}
close(jobs)
// Wait and close results
go func() {
wg.Wait()
close(results)
}()
// Collect results
for result := range results {
fmt.Println(result.Output)
}
}Real-World Scenarios
- Image Processing Pipeline: Resize/compress thousands of user uploads
- API Aggregation: Fetch data from multiple microservices concurrently
- Log Processing: Parse and index high-volume log streams
Follow-up Questions
- How would you handle job failures and retries?
- How do you limit memory usage with very large job queues?
- How would you implement priority queues for jobs?