- Introduce a Redis-based lock to prevent concurrent execution of
batch generation tasks.
- Set a TTL of 300 seconds for the lock to ensure it is released
after a timeout.
- Add logging for situations where a task is already in progress.
This enhancement ensures that batch tasks do not overlap, which can
lead to data inconsistencies and resource contention. The locking
mechanism improves the reliability of the batch processing system.
- Refactor generate_sitemap method to support locale
- Add setup_sitemap_config method for configuration
- Implement sitemap generation for default and localized paths
This update enhances the sitemap generation process by supporting
multiple languages. Users can now access a sitemap with language
prefixes, improving SEO and usability for different locales.
Additionally, error handling has been improved to log specific
errors related to locale generation.
- Update SitemapsController to use MinIO credentials
- Modify RefreshSitemapWorker to adapt to MinIO settings
- Change AWS configuration in initializers and storage files
- Add MinIO credentials to credentials.yml.enc
This commit transitions the application from using AWS S3 to MinIO for
storage. It updates all relevant configurations and ensures that the
application can now interact with MinIO seamlessly, including
support for both production and development environments.
- Modify the `PER_RUN_ION` constant of `BatchGenerateWeatherArtsWorker` from
3 to 2, reducing the number of images generated on each run.
- This change reduces the limit and will not affect other parts
of the application.
- It is about reducing the number of images generated per run and
will impact the performance of the application.
- It will not affect the other parts.
</commit_message>
- Include Redis gem in the Gemfile and update Gemfile.lock
- Implement locking mechanism in the RefreshSitemapWorker to prevent concurrent executions
- Use Redis to manage distributed locks for the sitemap generation process
This change introduces Redis for managing access controls in the sitemap
refresh workflow. It prevents multiple simultaneous jobs from running,
improving stability and performance when generating sitemaps.
- Change default_host assignment to use ENV.fetch for
RAILS_SITEMAP_DEFAULT_HOST
- This allows for easier configuration based on the environment,
particularly useful for production and development setups.
- Update query to select weather arts with last generation time
greater than the cutoff time.
- Adjust logging for clarity, including brackets around limits
and counts for better readability.
- Change from appending to selected cities to concatenating selected
cities, ensuring the full list is preserved.
This commit resolves an issue where the logic for filtering
recent weather arts was incorrect, potentially leading to missing
relevant entries. Additionally, it improves log output formatting
for better debugging.
- Change default host for SitemapGenerator from local server
to the new public URL. This allows the sitemap to be
generated correctly in the production environment.
- Ensures that the sitemap points to the correct domain, improving
search engine visibility and indexing.
- Added logging for skipped recent cities.
- Enhanced logging for selected cities after filtering.
This refactor improves the observability of the city selection
process in the BatchGenerateWeatherArtsWorker. The added logs
provide better insight during debugging and monitoring, allowing
developers to track recent selections and identify any issues
related to city filtering more effectively.
- Introduce `get_recent_cities` to filter out cities that have been recently generated weather arts.
- Update `select_cities` method to exclude recent cities from the selection process, ensuring diversity in the generated arts.
- Modify the `select_countries` method to maintain its functionality while integrating recent city filtering.
This update enhances the system's ability to select active cities efficiently, reducing redundancy and improving the variety of generated weather arts by avoiding recent candidates.
- Refactor city selection to better optimize the process
- Implement debugging logs for clearer tracking of city selection
- Modify logic to fairly distribute remaining city slots across countries
These changes enhance the efficiency of the city selection process
while providing useful debug information to track operations. The new
logic ensures that cities are selected based on activity status and
spreads the selection across provided countries if additional slots
are available. This improves the probability of including diverse
cities in the generation process, reducing potential bias in
selection.
- Display the latest weather art on the cities index page.
- Update the weather art preview image size to be big.
- Improve image display by using webp format.
- Adjust the daily and per-run generation limits.
These changes enhance the user experience by providing more
up-to-date and visually appealing content, while also controlling
the resource usage during image generation.
- Update city selection to consider countries.
- Use daytime instead of sunrise for time checks.
- Add per-run generation limit
This commit refactors the weather art generation process to
increase the efficiency. The city selection has been updated to
consider country. Daytime check has been updated to determine
if the city is within the correct local time. A limit is added
to restrict the number of cities processed in a single run.
Add logging for the daily generated counts
The change log adds a log statement to display the
count of slots generated.
The code change includes two parts:
* a debug log of
number of slots generated and
the number of
generation slots
* a change that generates the
number of
art per day
to the
user's log
informing
the art generated.
The logs are added for monitoring.
The log
statement outputs the
number of generated art and the limit.
</commit_message>
- Updated timezone parsing to handle JSON format.
- Modified how timezones are fetched in multiple models.
- Added a migration to convert existing timezones data.
- Modified the country model to format timezones.
This commit fixes an issue where the timezone data was not
being correctly parsed, leading to potential errors in time
calculations. The changes ensure the application correctly
handles and displays timezone information, improving the
accuracy of time-related features.
- Changed `timezones` attribute from text to JSONB in `Country`.
- Updated related model methods to handle JSONB data.
- Added a migration to convert existing timezone data.
- Used safe navigation operators to prevent errors.
This change improves the storage and management of timezone
information by using the JSONB data type. It includes data
migration to handle existing timezone data.
The logging level of the batch job worker has been changed from info to debug. The change affects how the batch job worker outputs information about the cities it is processing.
- Original behavior: The batch job worker logs cities' information at the info level.
- New behavior: The batch job worker logs cities' information at the debug level.
The motivation behind this change is to reduce the noise in the logs. The new behavior will only log cities' information that is relevant for debugging purposes.
- Add formatted_current_time method to City model.
- Modify BatchGenerateWeatherArtsWorker to check local time.
- Update Sidekiq scheduler to run every hour.
This commit improves the weather art generation by considering
local time for image generation and increasing the frequency
of batch processing for testing purposes. The new method
provides formatted time information for each city. This change
ensures that images are generated during daylight hours.
- Change the order of displayed city attributes in logs
- Adjust tabulation for clearer output
This commit modifies the logging format used in the
BatchGenerateWeatherArtsWorker class. The attribute order for
city information in the log output has been changed to ensure
a more intuitive display, with regions now appearing before
countries and states. This will enhance readability during
logging without altering any functionality.
- Implement daily generation limit for weather arts
- Improve city selection by prioritizing active cities
- Introduce logging for city generation start time and list
This update enhances the efficiency of the `BatchGenerateWeatherArtsWorker`. It ensures that the system respects a daily limit on the number of generated images and improves how cities are selected for processing by prioritizing active cities while also conditionally selecting others if slots remain. Additionally, the added logging provides better insights into the generation process.
- Change `by_popularity` method in `City` model to use `start_time` for filtering views.
- Update `by_popularity` in `WeatherArt` model to limit results and incorporate the correct popularity calculation.
- Adjust `HomeController` to define `@popular_shuffle_arts` and `@latest_arts` with appropriate scopes.
These changes improve the accuracy of popularity retrieval for both cities and weather arts, simplifying the logic in the controller and leveraging updated model methods for performance.
- Change generation interval from 24 hours to 36 hours
This change provides more flexibility in scheduling weather arts generation, which could lead to better resource management or accommodate changes in data availability.
- Change return condition in add_watermark method from
'return' to 'nil'.
- Ensure that the method correctly acknowledges the
presence of an existing watermark image, allowing for
further processing if necessary.
This change improves the clarity and functionality of the
add_watermark method by explicitly indicating when no
further action should be taken.
- Remove unused `ruby-vips` gem from Gemfile
- Update Dockerfile to remove `libvips42` package, which is no longer needed
- Comment out legacy watermarking code in `AddWatermarkToWeatherArtWorker`
These changes streamline the dependencies, making the
project lighter and reduce potential security issues with
unused gems and packages. The watermarking functionality is now
commented out for potential future use but is not currently
needed.
- Introduce BATCH_SIZE constant to limit processed cities
- Shuffle and limit eligible cities processing to enhance worker efficiency
This update improves the performance of the BatchGenerateWeatherArtsWorker
by ensuring that only a set number of cities are processed within the
allotted time, reducing the risk of timeouts and making the overall
system more responsive.
- Include actual update time in weather report
- Rename watermark worker file for clarity
This commit enhances the weather data report by
updating the format to include the actual update time
retrieved from the weather service API. Additionally,
the watermark worker file has been renamed to improve
readability and consistency in the naming convention.
- Include the image_processing gem to handle image manipulation
- Create a background worker for adding watermarks to weather art images
- Update WeatherArt model to attach watermark images
- Add a new watermark image asset
This commit enhances the WeatherArt feature by allowing images to
have watermarks added asynchronously, improving the visual
presentation of the art. It ensures sufficient image dimensions
before processing and includes error handling for the worker.
- Removed unnecessary leading and trailing blank lines in several
ActiveAdmin register files.
- Reformatted string delimiters for consistency, changing single quotes
to double quotes in tracking events in the Cities and WeatherArts
controllers.
- Ensured proper spacing in array definitions across several
models, including Ahoy::Event and Ahoy::Visit.
These changes improve code readability and maintain consistency
throughout the codebase by ensuring uniform use of quotes and
removing excess whitespace.
- Integrate Ahoy gem for tracking user events and visits
- Create models for Ahoy events and visits
- Implement admin interfaces for managing events and visits
- Add background job for cleaning up old analytics data
- Update application controller and other relevant controllers to track specific actions
This commit implements a comprehensive event tracking system that logs user interactions
within the application. Additionally, it includes mechanisms for managing and
cleaning historical visit and event data, ensuring efficient data handling.
- Create SitemapsController to serve sitemaps
- Configure AWS S3 storage for sitemaps
- Update routes to include sitemap paths
- Add SitemapsHelper module
- Configure SitemapGenerator with AWS adapter
- Update storage configurations for AWS
This feature adds sitemap functionality to the application, enabling search engines to discover and index its content more efficiently. It includes configuration for AWS S3 storage to host the sitemaps and updates the application's routes to serve them.
- Removed conditional check for production environment in
the initializer to ensure the sitemap refresh worker
always runs at startup.
- Renamed Sidekiq configuration file from sidekiq.yml to
sidekiq_scheduler.yml to better reflect its purpose.
These changes streamline the initialization process
for scheduled tasks and enhance the clarity of the
configuration files used in the project.
- Change SLEEP_DURATION from 60 seconds to 120 seconds
This change is made to improve resource management
and allow for a more efficient operation of the batch
processing tasks performed by the worker. It helps in
avoiding potential overload on system resources.
- Change generation interval from 6 hours to 24 hours
- Increase sleep duration from 3 seconds to 60 seconds
These adjustments aim to optimize the worker's performance and reduce resource consumption. This change reflects a shift towards longer generation cycles, potentially improving throughput.
- Added a space in the array passed to `add_index` in the migration
- Removed unused columns `last_weather_fetch` and `last_image_generation` from the `cities` table
- Ensured consistent code style in the `GenerateWeatherArtWorker` and added required newline at the end of files
These changes improve code readability and maintainability while ensuring that
no unnecessary fields exist in the database schema.
- Add logging to track the generation of weather art for each city.
- Refactor image attachment process to streamline the code by removing the
separate method for attaching images.
- Ensure proper handling of the temporary file used for image processing.
These changes improve observability during the weather art generation
process and encapsulate the image attachment logic within the primary
method, reducing the overhead of a method call. The adjustments also
ensure that temporary files are managed correctly to prevent resource
leaks.
- Introduced constants for configuration settings such as generation interval, maximum duration, and sleep duration.
- Updated the `perform` method to utilize these constants for better readability and maintainability.
- Refactored the `perform` method in `GenerateWeatherArtWorker` to improve flow and error handling by creating separate methods for fetching weather data, generating prompts, images, and handling database transactions.
- Cleaned up city seeding data by removing unnecessary fields while maintaining required functionality.
These changes improve the overall readability of the code and make it easier to adjust the behavior of the workers in the future without digging through the logic.
- Implement caching methods for last weather fetch and image generation
- Adjust sleep duration in BatchGenerateWeatherArtsWorker from 10 seconds to 3 seconds
- Remove unused fields `last_weather_fetch` and `last_image_generation` from the cities table
- Add index on the weather_arts table for optimized querying
This refactor improves data retrieval performance for weather data
associated with cities. Caching reduces database load while the
worker modification allows for faster iterations in generating
weather arts without significantly impacting performance.
- update default host in sitemap generator
- refresh sitemap on worker with new host
These changes allow the sitemap to be correctly generated for different environments and to handle the new host correctly, which was the motivation behind this update. No other side effects are expected.
- Ensure consistent use of double quotes for strings in the
Gemfile and sitemap configuration files.
- Add spaces for better readability in array declarations
within the RefreshSitemapWorker.
These changes improve the readability of the code without
changing any functionality. Adhering to a consistent coding
style helps maintainability and team collaboration.
- Introduced `SeoConcern` module to handle SEO meta tags
- Integrated `meta-tags` gem for customizable meta tags
- Created `RefreshSitemapWorker` to automate sitemap updates
- Added relevant meta tags in controllers for weather art and cities
- Configured sitemap generation settings
These changes improve the SEO of the application by ensuring that
pages have appropriate meta tags. Additionally, a sitemap is now
generated and refreshed daily, enhancing site visibility to search
engines.
- Change sleep duration from 1 minute to 10 seconds in
BatchGenerateWeatherArtsWorker.
This change addresses API limitations by reducing the wait time
between job submissions, thereby increasing the efficiency of
the batch processing for generating weather arts.
- Change BatchGenerateWeatherArtsWorker to use city ID instead of city object
- Update API endpoint to accept city ID
- Improve performance by reducing database queries
This change improves the performance of the BatchGenerateWeatherArtsWorker by reducing database queries and increasing efficiency. It is a necessary modification to ensure the worker functions correctly and efficiently.
- Include Sidekiq::Worker for asynchronous task execution
- Implement condition to skip execution if weather data is up to date
These changes enable efficient weather art generation by leveraging
background processing, thereby improving overall application
performance and responsiveness.
- Change from using GenerateWeatherArtJob to GenerateWeatherArtWorker
- Maintain asynchronous job processing
This refactoring improves the clarity of the job class being used
for generating weather art and adheres to project conventions. The
change does not affect the API rate limits as the sleep duration
remains the same.
- Replace synchronous job processing with asynchronous
- Ensure processing does not exceed API rate limits
This change allows for better performance by offloading the
queue management to Sidekiq, enabling more efficient
parallel processing of weather art generation. It also
prevents potential API rate limit violations by maintaining
a delay between requests.
- Renamed `BatchGenerateWeatherArtsJob` to `BatchGenerateWeatherArtsWorker`
- Renamed `GenerateWeatherArtJob` to `GenerateWeatherArtWorker`
- Updated Sidekiq configuration to set Redis URL and logger level
- Modified `sidekiq.yml` to use the new worker class names and added queue configurations
These changes help in aligning the class names with their purpose as
workers in Sidekiq, while also ensuring better configuration for
Redis and logging.
- Standardized quotation marks in Gemfile and Ruby files
- Improved consistency for Sidekiq and Sidekiq Scheduler
- Removed unnecessary blank line in GenerateWeatherArtWorker class
These changes ensure a consistent coding style across the project, making
it easier to read and maintain. Adjustments to the Gemfile update the
formatting without altering the functionality.
- Implement BatchGenerateWeatherArtsWorker to handle batch
processing of weather art generation.
- Create GenerateWeatherArtWorker for individual weather art
generation tasks.
- Update Dockerfile to include redis-tools for Sidekiq support.
- Modify Gemfile to add sidekiq and sidekiq-scheduler gems.
- Configure Sidekiq in initializers and set up routes for
Sidekiq dashboard.
- Include a sidekiq.yml configuration for scheduling jobs.
- Create compose.yaml for Docker services including web,
database, Redis, and Sidekiq workers.
These changes introduce background processing capabilities
using Sidekiq, allowing for efficient generation of weather
art through scheduled and managed job queues, optimizing
performance and scalability.
