songtianlun/today_ai_weather
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: optimize batch weather art generation

Browse Source 
- Introduce BATCH_SIZE constant to limit processed cities
- Shuffle and limit eligible cities processing to enhance worker efficiency

This update improves the performance of the BatchGenerateWeatherArtsWorker
by ensuring that only a set number of cities are processed within the
allotted time, reducing the risk of timeouts and making the overall
system more responsive.
This commit is contained in:
songtianlun 2025-02-01 15:01:36 +08:00
parent 9417358625
commit f2951e2741
1 changed files with 2 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
app/workers/batch_generate_weather_arts_worker.rb
View File

@ -1,19 +1,16 @@
class BatchGenerateWeatherArtsWorker class BatchGenerateWeatherArtsWorker
include Sidekiq::Worker include Sidekiq::Worker
GENERATION_INTERVAL = 24.hours GENERATION_INTERVAL = 24.hours
MAX_DURATION = 50.minutes MAX_DURATION = 50.minutes
SLEEP_DURATION = 120.seconds SLEEP_DURATION = 120.seconds
BATCH_SIZE = 20
def perform(*args) def perform(*args)
start_time = Time.current start_time = Time.current
cities_to_process = get_eligible_cities.shuffle.take(BATCH_SIZE)
cities_to_process = get_eligible_cities
cities_to_process.each do |city| cities_to_process.each do |city|
break if Time.current - start_time > MAX_DURATION break if Time.current - start_time > MAX_DURATION
Rails.logger.info "Generating weather art for #{city.name}" Rails.logger.info "Generating weather art for #{city.name}"
GenerateWeatherArtWorker.perform_async(city.id) GenerateWeatherArtWorker.perform_async(city.id)
sleep SLEEP_DURATION sleep SLEEP_DURATION
end end
@ -23,7 +20,6 @@ class BatchGenerateWeatherArtsWorker
def get_eligible_cities def get_eligible_cities
cutoff_time = Time.current - GENERATION_INTERVAL cutoff_time = Time.current - GENERATION_INTERVAL
City.active City.active
.joins("LEFT JOIN ( .joins("LEFT JOIN (
SELECT city_id, MAX(created_at) as last_generation_time SELECT city_id, MAX(created_at) as last_generation_time