opensourcepolitics / decidim-app

Open Source Politics' Decidim reference implementation

Decidim app by OSP

Decidim

Decidim is a digital platform for citizen participation. Related documentation can be found here

Open Source Politics

This repository contains the code of the decidim-app implemented for our customers.

It consists of the main application with modules developed by the community that we often use.

It includes official modules supported by the community and community-based modules developed by us our our partners

List of our modules

You can find below an exhaustive list of modules with their repository links and latest version available :

Decidim Module	Version	Brief description
decidim-core	v0.27.4
decidim-conferences	v0.27.4
decidim-initiatives	v0.27.4
decidim-templates	v0.27.4
decidim-cache_cleaner	✅	Allow admins to clear cache of the application in the back-office
decidim-decidim_awesome	✅	An awesome module that allows (among others) : adding extra-fields for proposals creation, fullscreen iframe component, image in rich text editor, customs redirections etc.
decidim-friendly_signup	✅	Module that drastically simplify the registration process of users by deleting some registration fields
decidim-homepage_interactive_map	✅	Module that allow the adding of a content-block on the homepage diplaying a map of assemblies in order to allow geo-located participation
decidim-ludens	✅	Gamified tutorial in the admin back-office to help admins understand how Decidim works
decidim-phone_authorization_handler	✅	Module allowing gathering phone number on a particular participant action
decidim-spam_detection	✅	Module adding a spam detection algorithm that runs periodically detecting spam accounts
decidim-term_customizer	✅	Module allowing the change of translated strings
decidim-gallery	✅	Module allowing the creation of galleries
decidim-extra_user_fields	✅	Module allowing the creation of new fields in users form

Some non-official customizations can be found see OVERLOADS.MD.

🚀 Getting started

• See our installation guide to run a decidim-app by OSP locally
• See our Docker installation guide to run a decidim-app by OSP locally with Docker
• See our homepage interactive map module to configure module (OSX/Ubuntu)

👋 Contributing

• See our contributing guide

🔒 Security

Security is very important to us. If you have any issue regarding security, please disclose the information responsibly by sending an email to security[at]opensourcepolitics[dot]eu and not by creating a Github issue.

License

The decidim-app is licensed under the AGPLv3, same license as Decidim.

decidim-app's Issues

Change the time of the daily digest and the day of the weekly digest for sending notifications

Describe the problem
Currently, the daily digest is set to 12PM and the weekly digest is set to Saturday.

To do

• Set the SendNotificationMailDaily task to 7AM
• Set the SendNotificationMailWeeklytask to Tuesday at 8AM

Thank you!

Assets should not be included in version control

Assets generated by webpacker should not be kept in version control
Dockerfile should generate assets at built

Add issue template

Description

We should add an issue template for new contributions

Investigate on inconsistency results on CI, sometimes the hash generated is diff...

decidim-app/lib/decidim/assets_hash.rb

Line 26 in 2fd92f7

# TODO: Investigate on inconsistency results on CI, sometimes the hash generated is different for the slice 1-2 without apparent reason.

module Decidim
  class AssetsHash
    def initialize
      @app_dependencies_patterns = %w(Gemfile Gemfile.lock package.json yarn.lock)
      @assets_patterns = %w(app/assets/**/* app/packs/**/* vendor/**/* packages/**/* lib/assets/**/*)
    end

    def self.run
      new.run
    end

    def run
      app_assets_hashes = files_digest(@assets_patterns)
      app_dependencies_hashes = files_digest(@app_dependencies_patterns)

      File.write("tmp/assets_manifest.json", JSON.pretty_generate(app_assets_hashes))

      digest("#{app_assets_hashes.values.join("\n")}#{app_dependencies_hashes.values.join("\n")}")
    end

    def files_digest(patterns)
      # TODO: Investigate on inconsistency results on CI, sometimes the hash generated is different for the slice 1-2 without apparent reason.
      Array.wrap(patterns).map { |pattern| Dir.glob(pattern) }
           .flatten
           .sort
           .each_with_object({}) do |file, result|
        next unless File.file?(file)

        result[file] = digest(File.read(file))
      end
    end

    private

    def digest(value)
      Digest::SHA256.hexdigest(value)
    end
  end
end

Simplify Rack Attack configuration

Describe the bug
A clear and concise description of what the bug is.

Method self.rack_enabled? is complicated, we should simplify it
cf:

decidim-app/lib/decidim_app/rack_attack.rb

Line 5 in d74b075

def self.rack_enabled?

[v0.26] - Move Sentry implementation from HTML head to footer

Description

At the moment we're loading Sentry JS scripts in app/views/layouts/decidim/_head_extra.html.erb.
Since this script is not critical for the application, I think we should load it at the end of the page to increase page loading.

Tasks

• Move Sentry scripts at the end of HTML

What do you think ?

Remove "exit 0" code from rake tasks

Description

Some rake tasks returns exit code 0 when done, it may occur inconsistencies on CI since Rspec returns a 0 status code which means valid status code

TODO

• Remove exit statements from rake tasks
• Avoid exit status code in rake task and whole application

Add disclaimer on /admin/initiatives page in back-office

Describe the bug
We need to add a disclaimer to the /admin/initiatives page to warn administrators that when a initiative type is created, it is automatically displayed in the front office. Unlike the Processes and Assemblies spaces, the Initiative space does not contain a logic for publishing/depublishing a type of initiative.

We have already developed a similar disclaimer on the /admin/initiatives_types/new page, which we should also add to the global initiatives page in the back office.

Related to #480

Expected behavior
As an admin, I want to be notified on the /admin/initiatives page that when I create a new type of initiative, it will be displayed in the front office.

Screenshots
Here is the disclaimer that should be added to the top of the page

Add tests

decidim-app/lib/tasks/db.rake

Line 90 in b571eaf

# TODO: Add tests

      end

      desc "Delete surveys related to deleted component"
      # TODO: Add tests
      task clean: :environment do
        Rails.logger = Logger.new($stdout)
        # ActiveRecord::Base.logger = Logger.new($stdout)

Add tests

decidim-app/lib/tasks/db.rake

Line 42 in b571eaf

# TODO: Add tests

    namespace :admin_log do
      desc "List admin log related to orphans data"
      # TODO: Add tests
      task orphans: :environment do
        Rails.logger = Logger.new($stdout)
        # ActiveRecord::Base.logger = Logger.new($stdout)

ci: Configure codecov/project only on push on master branch

Describe the bug
A clear and concise description of what the bug is.

Codecov/project is always failing on every PR, we should configure GH workflow to run only on new releases (master branch)

Missing translation on private user import uploader

Describe the bug
The translation fr.activemodel.attributes.participatory_space_private_user_csv_import.file is missing when I import private users form a file in a participatory space as an admin.

To Reproduce

1. Go to /participatory_space_private_users/csv_import/ in a participatory process
2. Click on "Add a file"
3. See the missing translation

Screenshots

Add tests

decidim-app/lib/tasks/import.rake

Line 7 in b571eaf

# TODO: Add tests

namespace :import do
  desc "Usage: rake import:user FILE='<filename.csv>' ORG=<organization_id> ADMIN=<admin_id> PROCESS=<process_id> [VERBOSE=true]'"
  # TODO: Add tests
  task user: :environment do
    def validate_input
      validate_file

Bump to 0.27.0 when released

Decidim::Gallery.configure do |config|

config.enable_animation = Rails.application.secrets.dig(:modules, :gallery, :enable_animation)

end

https://api.github.com/OpenSourcePolitics/decidim-app/blob/d111822d5695b1ae2ff71579b151211fb640b814/config/initializers/decidim_gallery.rb#L3

# frozen_string_literal: true

# TODO: Bump to 0.27.0 when released
# Decidim::Gallery.configure do |config|
#   config.enable_animation = Rails.application.secrets.dig(:modules, :gallery, :enable_animation)
# end

Cannot change initiative type as an admin

Describe the bug
As an admin, I cannot change the type of an initiative sent in technical validation. Indeed, when I update the initiative after changing its type, the new type is not taking in account and the previous remains.

To Reproduce

1. Go to /admin/initiatives
2. Click on the initiative configuration button of an initiative sent in technical validation
3. Modify the "Type" field (at least 2 initiatives types must be created)
4. Update the changes
5. See that the new initiative type is not applied

Expected behavior
As an admin, I should be able to change the type of an initiative sent in technical validation.

Screenshots
Original type of the initiative

Changing the type and updating it

Changing the type and updating it

Additional context
Unable to reproduce on nightly.decidim.org (0.28v) !

Switch to master

https://api.github.com/OpenSourcePolitics/decidim-app/blob/90cd014721b9498f9463822b4ce1bef99bbddf47/.github/workflows/ci_cd.yml#L190

      DATABASE_PASSWORD: postgres
      DATABASE_HOST: host.docker.internal
    steps:
      # TODO: Switch to master
      - uses: OpenSourcePolitics/build-and-test-images-action@develop
        with:
          registry: ${{ vars.REGISTRY_ENDPOINT }}
          namespace: ${{ vars.REGISTRY_NAMESPACE }}

Extract to a lib

decidim-app/lib/tasks/repair_data.rake

Line 6 in b571eaf

# TODO: Extract to a lib

namespace :decidim do
  namespace :repair do
    desc "Check for nicknames that doesn't respect valid format and update them, if needed force update with REPAIR_NICKNAME_FORCE=1"
    # TODO: Extract to a lib
    task nickname: :environment do
      logger = Logger.new($stdout)
      logger.info("[decidim:repair:nickname] :: Checking all nicknames...")

Disable Deface being applied twice

Deface is applied twice in production, this is not useful and result in a minor performance deterioration

remove this job

https://api.github.com/OpenSourcePolitics/decidim-app/blob/dcb0b1c63f452ea473dd2b80b8897457c774615f/.github/workflows/ci_cd.yml#L242

          key: ${{ secrets.ANSIBLE_KEY }}
          port: ${{ secrets.SSH_PORT }}
          script: ansible-playbook -u ${{ secrets.ANSIBLE_USERNAME }} --private-key="~/.ssh/ansible-deploy/ansible-deploy" -i /home/${{ secrets.ANSIBLE_USERNAME }}/ansible/decidim/inventories/staging.yml /home/${{ secrets.ANSIBLE_USERNAME }}/ansible/decidim/playbooks/update_decidim_app.yml
  # TODO: remove this job
  build_and_push_image_wip:
    name: Build and push image to Registry
    needs: [lint, tests, system_tests, test_build]
    runs-on: ubuntu-latest
    steps:
      - uses: OpenSourcePolitics/build-and-push-images-action@master
        with:
          registry: ${{ vars.REGISTRY_ENDPOINT }}
          namespace: ${{ vars.REGISTRY_NAMESPACE }}
          password: ${{ secrets.TOKEN }}
          image_name: ${{ vars.IMAGE_NAME }}
          tag: "wip"
  build_and_push_image_dev:
    name: Build and push image to Registry
    if: "github.ref == 'refs/heads/develop'"

Remove this task in the next release

decidim-app/lib/tasks/restore_dump.rake

Line 3 in b571eaf

# TODO: Remove this task in the next release

# frozen_string_literal: true

# TODO: Remove this task in the next release
task :restore_dump do
  # :nocov:
  ActiveSupport::Deprecation.warn("rake restore_dump is deprecated and will be remove shortly")
  $stdout.puts "Path to dump file (/path/to/dump): "
  local_path = $stdin.gets.to_s.strip
  dump = local_path[-1].split("/")[-1]
  sh "docker cp '#{local_path}' decidim-app_database_1:'/tmp/#{dump}'"
  sh "docker exec -it decidim-app_database_1 su postgres -c 'pg_restore -c -O -v -d osp_app /tmp/#{dump}'"
  # :nocov:
end

Investigate why we need to set this

https://api.github.com/OpenSourcePolitics/decidim-app/blob/c543097dcc2736214f43ed6c8930ab68aa514166/config/application.rb#L30

    # DON'T FORGET to ALSO set this in `config/initializers/carrierwave.rb`
    config.action_mailer.asset_host = "https://#{Rails.application.secrets[:asset_host]}/" if Rails.application.secrets[:asset_host].present?

    # TODO: Investigate why we need to set this
    Rails.application.config.action_mailer.delivery_job = "ActionMailer::MailDeliveryJob"

    config.backup = config_for(:backup).deep_symbolize_keys

    config.action_dispatch.default_headers = {

Cache on filters

Implement cache on filters

must work with image src

@param [String] deprecated_endpoint

https://api.github.com/OpenSourcePolitics/decidim-app/blob/dd254bd654e0302552aa13df0ef5e7c3baf83464/app/services/decidim/repair_url_in_content_service.rb#L17

# frozen_string_literal: true

module Decidim
  # Looks for any occurence of "@deprecated_endpoint" in every database columns of type COLUMN_TYPES
  # For each field containing @deprecated_endpoint:
  #   - Looks for the current ActiveStorage::Blob with the given filename
  #   - Find the blob's service_url
  #   - Replace the @deprecated_endpoint with the blob's service_url in text
  #   - Update the column
  # Context:
  # After S3 assets migration with rake task "bundle exec rake scaleway:storage:migrate_from_local", every linked documents URL were well updated.
  # However every links added to text fields redirecting to an uploaded file were outdated and still redirects to the old S3 bucket
  class RepairUrlInContentService
    COLUMN_TYPES = [:string, :jsonb, :text].freeze
    DEFAULT_LOGGER = Rails.logger

    # TODO: must work with image src
    # @param [String] deprecated_endpoint
    def self.run(deprecated_endpoint)
      new(deprecated_endpoint).run
    end

    # @param [String] deprecated_endpoint
    def initialize(deprecated_endpoint, logger = nil)
      @logger = logger || DEFAULT_LOGGER
      @deprecated_endpoint = deprecated_endpoint
    end

    def run

../scaleway.config is hardcoded here, should be a parameter

https://api.github.com/OpenSourcePolitics/decidim-app/blob/c64ef2723bcb3f554223665b413ddff0883d2981/lib/k8s_organization_exporter.rb#L40

# frozen_string_literal: true

class K8sOrganizationExporter
  FORBIDDEN_ENVIRONMENT_KEYS = %w(BACKUP_ENABLED
                                  BACKUP_S3SYNC_ENABLED
                                  BACKUP_S3SYNC_ACCESS_KEY
                                  BACKUP_S3SYNC_SECRET_KEY
                                  BACKUP_S3SYNC_BUCKET
                                  BACKUP_S3RETENTION_ENABLED).freeze
  ORGANIZATION_COLUMNS = %w(id
                            default_locale
                            available_locales
                            users_registration_mode
                            force_users_to_authenticate_before_access_organization
                            available_authorizations
                            file_upload_settings).freeze

  def initialize(organization, logger, export_path, hostname, image)
    @organization = organization
    @logger = logger
    @export_path = export_path
    @hostname = hostname
    @image = image
    @database_name = Rails.configuration.database_configuration[Rails.env]["database"]
  end

  def self.export!(organization, logger, export_path, hostname, image)
    new(organization, logger, export_path, hostname, image).export!
  end

  def export!
    creating_directories
    exporting_env_vars
    exporting_configuration
    dumping_database
    retrieve_active_storage_files
  end

  def retrieve_active_storage_files
    # TODO: ../scaleway.config is hardcoded here, should be a parameter
    @logger.info("retrieving active storage files from bucket #{bucket_name} into #{organization_export_path}/buckets/#{resource_name}--de")
    system("rclone copy scw-storage:#{bucket_name} #{organization_export_path}/buckets/#{resource_name}--de --config ../scaleway.config --progress --copy-links")
  end

  def dumping_database
    @logger.info("dumping database #{@database_name} to #{organization_export_path}/postgres/#{resource_name}--de.dump")
    system("pg_dump -Fc #{@database_name} > #{organization_export_path}/postgres/#{resource_name}--de.dump")
  end

  def exporting_configuration
    @logger.info("exporting application configuration to #{organization_export_path}/application.yml")
    File.write("#{organization_export_path}/application.yml", YAML.dump(organization_settings))
  end

  def exporting_env_vars
    # TODO: Shouldn't we export this to a k8s secret?
    @logger.info("exporting env variables to #{organization_export_path}/manifests/#{resource_name}--de.yml")
    File.write("#{organization_export_path}/manifests/#{resource_name}-config.yml",
               YAML.dump(all_env_vars))
  end

  def creating_directories
    @logger.info("creating organization directories")
    @logger.info("#{organization_export_path}/buckets/#{resource_name}--de")
    FileUtils.mkdir_p("#{organization_export_path}/buckets/#{resource_name}--de")
    @logger.info("#{organization_export_path}/manifests")
    FileUtils.mkdir_p("#{organization_export_path}/manifests")
    @logger.info("#{organization_export_path}/postgres")
    FileUtils.mkdir_p("#{organization_export_path}/postgres")
  end

  def all_env_vars
    env_vars.merge!(smtp_settings).merge!(omniauth_settings)
  end

  def env_vars
    @env_vars ||= Dotenv.parse(".env")
                        .reject { |key, _value| FORBIDDEN_ENVIRONMENT_KEYS.include?(key) }
  end

  def omniauth_settings
    return {} unless @organization.omniauth_settings

    @organization.omniauth_settings.each_with_object({}) do |(key, value), hash|
      hash[key.upcase] = if Decidim::OmniauthProvider.value_defined?(value)
                           decrypt(value)
                         else
                           value
                         end
    end
  end

  def smtp_settings
    settings = @organization.smtp_settings.deep_dup
    settings["password"] = Decidim::AttributeEncryptor.decrypt(settings["encrypted_password"])
    settings.delete("encrypted_password")

    settings.transform_keys do |key|
      "SMTP_#{key.upcase}"
    end
  end

  def organization_columns
    org_columns_sql = "SELECT row_to_json(o,true) FROM (SELECT #{ORGANIZATION_COLUMNS.join(", ")} FROM decidim_organizations WHERE id=#{@organization.id}) AS o;"
    org_columns_record = ActiveRecord::Base.connection.execute(org_columns_sql)
    JSON.parse(org_columns_record.first["row_to_json"])
  end

  def organization_settings
    {
      apiVersion: "apps.libre.sh/v1alpha1",
      kind: "Decidim",
      metadata: {
        name: resource_name
      },
      spec: {
        image: @image,
        host: @organization.host,
        additionalHosts: @organization.secondary_hosts,
        organization: organization_columns,
        timeZone: @organization.time_zone,
        envFrom: [
          {
            secretRef: {
              name: "#{resource_name}-config"
            }
          }
        ]
      }
    }.deep_stringify_keys
  end

  def organization_export_path
    @organization_export_path ||= "#{@export_path}/#{resource_name}"
  end

  def resource_name
    @resource_name ||= "#{@hostname}--#{organization_slug}"
  end

  def bucket_name
    @bucket_name ||= env_vars["SCALEWAY_BUCKET_NAME"]
  end

  def organization_slug
    @organization_slug ||= @organization.host.parameterize(separator: "_", preserve_case: true)
  end

  private

  def decrypt(value)
    Decidim::AttributeEncryptor.decrypt(value)
  rescue ActiveSupport::MessageEncryptor::InvalidMessage
    value
  end
end

Investigate on inconsistency results on CI, sometimes the hash generated is diff...

decidim-app/lib/decidim/assets_hash.rb

Line 24 in b85c46e

# TODO: Investigate on inconsistency results on CI, sometimes the hash generated is different for the slice 1-2 without apparent reason.

module Decidim
  class AssetsHash
    def initialize
      @app_dependencies_patterns = %w(Gemfile Gemfile.lock package.json yarn.lock)
      @assets_patterns = %w(app/assets/**/* app/packs/**/* vendor/**/* packages/**/* lib/assets/**/*)
    end

    def self.run
      new.run
    end

    def run
      app_assets_hashes = files_digest(@assets_patterns)
      app_dependencies_hashes = files_digest(@app_dependencies_patterns)

      digest("#{app_assets_hashes}#{app_dependencies_hashes}")
    end

    def files_digest(patterns)
      # TODO: Investigate on inconsistency results on CI, sometimes the hash generated is different for the slice 1-2 without apparent reason.
      Array.wrap(patterns).map { |pattern| Dir.glob(pattern) }
           .flatten
           .sort
           .select { |file| File.file?(file) }
           .map { |file| digest(File.read(file)) }
           .join("\n")
    end

    private

    def digest(value)
      Digest::SHA256.hexdigest(value)
    end
  end
end

remove this step when we have a slim image

https://api.github.com/OpenSourcePolitics/decidim-app/blob/7b88db0a8db5745a361596c7924ef4a2a47bc826/.github/workflows/ci_cd.yml#L255

          password: ${{ secrets.TOKEN }}
          image_name: ${{ vars.IMAGE_NAME }}
          tag: "develop"
  # TODO: remove this step when we have a slim image
  build_and_push_image_slim:
    name: Build and push image to Registry
    if: "github.ref == 'refs/heads/develop'"
    runs-on: ubuntu-latest
    steps:
      - uses: OpenSourcePolitics/build-and-push-images-action@master
        with:
          registry: ${{ vars.REGISTRY_ENDPOINT }}
          namespace: ${{ vars.REGISTRY_NAMESPACE }}
          password: ${{ secrets.TOKEN }}
          image_name: ${{ vars.IMAGE_NAME }}
          tag: "slim"
  generate_release:
    name: Generate release
    needs: [lint, tests, system_tests, test_build]

Use latest version

https://api.github.com/OpenSourcePolitics/decidim-app/blob/dbe637d3594371f236cc504782f167e7242f95b1/.github/workflows/ci_cd.yml#L91

          key: asset-cache-${{ runner.os }}-${{ steps.cache-hash.outputs.hash }}
      - run: mkdir -p ./spec/tmp/screenshots
        name: Create the screenshots folder
      # TODO: Use latest version
      - uses: nanasess/setup-chromedriver@v2
        with:
          chromedriver-version: "114.0.5735.90"
      - run:  bundle exec rake "test:run[exclude, spec/system/**/*_spec.rb, ${{ matrix.slice }}]"
        name: RSpec
      - run: ./.github/upload_coverage.sh decidim-app $GITHUB_EVENT_PATH

What is storage in this context?

https://api.github.com/OpenSourcePolitics/decidim-app/blob/8e64f4d7958b459bc0719ee39f09f0ffb0dadfb7/app/services/decidim/backup_service.rb#L99

        cmd += " -U '#{@options[:db_conf][:username]}'" if @options[:db_conf][:username].present?
        cmd = "PGPASSWORD=#{@options[:db_conf][:password]} #{cmd}" if @options[:db_conf][:password].present?
        cmd += " -d '#{@options[:db_conf][:database]}'" if @options[:db_conf][:database].present?
        cmd += " -f '#{file_path}'"

        Rails.logger.info("Started backup_database with #{cmd}")

        execute_backup_command(file_path, cmd)
      else
        Rails.logger.error "Cannot connect to DB with configuration"
        Rails.logger.error @options[:db_conf].except(:password)
        Rails.logger.error "DB password was #{@options[:db_conf][:password].present? ? "present" : "missing"}"
        # do not exit here because we can still try to do the other backups
        false
      end
    end

    def backup_uploads
      # TODO: What is storage in this context?
      if file_exists?("storage")
        file = generate_backup_file_path("storage", "tar.bz2")

        cmd = "tar -jcf #{file} storage"

Add tests

decidim-app/lib/tasks/db.rake

Line 76 in b571eaf

# TODO: Add tests

    namespace :surveys do
      desc "List surveys related to deleted component"
      # TODO: Add tests
      task orphans: :environment do
        Rails.logger = Logger.new($stdout)
        # ActiveRecord::Base.logger = Logger.new($stdout)

Investigate on flaky causing a double call on rake task when tested

https://api.github.com/OpenSourcePolitics/decidim-app/blob/a9ed482b6e94bdcfec7df5140b76d33df02b859b/spec/lib/tasks/create_system_admin_task_spec.rb#L17

require "spec_helper"

describe "rake decidim_app:create_system_admin", type: :task do
  before do
    allow(Decidim::SystemAdminCreator).to receive(:create!).with(ENV).and_return(true)
  end

  it "preloads the Rails environment" do
    expect(task.prerequisites).to include "environment"
  end

  it "invokes the admin creator" do
    task.execute

    # TODO: Investigate on flaky causing a double call on rake task when tested
    expect($stdout.string).to include("System admin created successfully\n")
  end
end

Enable push notifications for PWA

Describe the bug
0.27 version add the possibility to install the platform as a PWA.
We are missing some configuration

Fixing advice

1. Add 2 environment variables :

• VAPID_PUBLIC_KEY
• VAPID_PRIVATE_KEY
  (use bin/rails decidim:pwa:generate_vapid_keys to generate the values)

2. Add these variables to the secrets.yml

  vapid:
    enabled: <%%= Decidim::Env.new("VAPID_PUBLIC_KEY").to_boolean_string %>
    public_key: <%%= ENV["VAPID_PUBLIC_KEY"] %>
    private_key: <%%= ENV["VAPID_PRIVATE_KEY"] %>

see : https://github.com/decidim/decidim/blob/release/0.27-stable/decidim-generators/lib/decidim/generators/app_templates/secrets.yml.erb#L154C1-L157C50

Expected behavior
If all is well configure we should see a setting for Push notifications in the notifications settings page

Screenshots

Additional context
See discussions :

• https://github.com/decidim/decidim/blob/release/0.27-stable/CHANGELOG.md#331-generate-the-vapid-keys-by-running-the-command
• decidim/decidim#8774

Empty personal data columns on PDF signatures exports of initiatives

Describe the bug
As an admin, when I export click on the button to export PDF of signatures of an initiative, empty columns that correspond to personal data of users are displayed in the export file. It is normal that the column are empty, because the " Collect participant personal data on signature " setting is disabled but it doesn't make sense that these columns are displayed.

To Reproduce

1. Go to /admin/initiatives
2. Click on the configuration button of a published initiative
3. Click on the "Export PDF signatures"button
4. See that empty columns are displayed in the export file.

Expected behavior
As an admin, if I do not enable the "Collectif participant personal data on signature" setting, I expect to not have personal data column in the export file of the signatures.

Screenshots

Additional context
A PR has been merged on https://github.com/decidim/decidim in 0.28 version: decidim/decidim#11998

Add tests

decidim-app/lib/tasks/db.rake

Line 7 in b571eaf

# TODO: Add tests

  namespace :db do
    namespace :notification do
      desc "List notifications related to orphans data"
      # TODO: Add tests
      task orphans: :environment do
        Rails.logger = Logger.new($stdout)
        # ActiveRecord::Base.logger = Logger.new($stdout)

[v0.26] - undefined method `setup` for Decidim::Devise:Module

Description

On version 0.26, an error occurs when application is initialised, raised at decidim-core/lib/decidim/core/engine.rb:570.
It seems there is a context issue where the class Devise is interpreted as in part of module Decidim.

Step to reproduce

• bundle exec rake db:seed --trace
• An error should occur

Stacktrace

** Invoke db:schema:load (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
rake aborted!
NoMethodError: undefined method `setup' for Decidim::Devise:Module
/Users/quentinchampenois/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/bundler/gems/decidim-8ad4ad8f4104/decidim-core/lib/decidim/core/engine.rb:570:in `block in <class:Engine>'
/Users/quentinchampenois/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/railties-6.0.4.8/lib/rails/initializable.rb:32:in `instance_exec'
/Users/quentinchampenois/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/railties-6.0.4.8/lib/rails/initializable.rb:32:in `run'
/Users/quentinchampenois/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/railties-6.0.4.8/lib/rails/initializable.rb:61:in `block in run_initializers'

More informations

The development_app is successfully generated with the generator

Cache assets:precompile

In order to reduce our energy footprint I propose to precompile the assets before the unit and system tests. This will not speed up the overall testing time but it will avoid compiling them for each test.

Shouldn't we export this to a k8s secret?

https://api.github.com/OpenSourcePolitics/decidim-app/blob/c64ef2723bcb3f554223665b413ddff0883d2981/lib/k8s_organization_exporter.rb#L56

# frozen_string_literal: true

class K8sOrganizationExporter
  FORBIDDEN_ENVIRONMENT_KEYS = %w(BACKUP_ENABLED
                                  BACKUP_S3SYNC_ENABLED
                                  BACKUP_S3SYNC_ACCESS_KEY
                                  BACKUP_S3SYNC_SECRET_KEY
                                  BACKUP_S3SYNC_BUCKET
                                  BACKUP_S3RETENTION_ENABLED).freeze
  ORGANIZATION_COLUMNS = %w(id
                            default_locale
                            available_locales
                            users_registration_mode
                            force_users_to_authenticate_before_access_organization
                            available_authorizations
                            file_upload_settings).freeze

  def initialize(organization, logger, export_path, hostname, image)
    @organization = organization
    @logger = logger
    @export_path = export_path
    @hostname = hostname
    @image = image
    @database_name = Rails.configuration.database_configuration[Rails.env]["database"]
  end

  def self.export!(organization, logger, export_path, hostname, image)
    new(organization, logger, export_path, hostname, image).export!
  end

  def export!
    creating_directories
    exporting_env_vars
    exporting_configuration
    dumping_database
    retrieve_active_storage_files
  end

  def retrieve_active_storage_files
    # TODO: ../scaleway.config is hardcoded here, should be a parameter
    @logger.info("retrieving active storage files from bucket #{bucket_name} into #{organization_export_path}/buckets/#{resource_name}--de")
    system("rclone copy scw-storage:#{bucket_name} #{organization_export_path}/buckets/#{resource_name}--de --config ../scaleway.config --progress --copy-links")
  end

  def dumping_database
    @logger.info("dumping database #{@database_name} to #{organization_export_path}/postgres/#{resource_name}--de.dump")
    system("pg_dump -Fc #{@database_name} > #{organization_export_path}/postgres/#{resource_name}--de.dump")
  end

  def exporting_configuration
    @logger.info("exporting application configuration to #{organization_export_path}/application.yml")
    File.write("#{organization_export_path}/application.yml", YAML.dump(organization_settings))
  end

  def exporting_env_vars
    # TODO: Shouldn't we export this to a k8s secret?
    @logger.info("exporting env variables to #{organization_export_path}/manifests/#{resource_name}--de.yml")
    File.write("#{organization_export_path}/manifests/#{resource_name}-config.yml",
               YAML.dump(all_env_vars))
  end

  def creating_directories
    @logger.info("creating organization directories")
    @logger.info("#{organization_export_path}/buckets/#{resource_name}--de")
    FileUtils.mkdir_p("#{organization_export_path}/buckets/#{resource_name}--de")
    @logger.info("#{organization_export_path}/manifests")
    FileUtils.mkdir_p("#{organization_export_path}/manifests")
    @logger.info("#{organization_export_path}/postgres")
    FileUtils.mkdir_p("#{organization_export_path}/postgres")
  end

  def all_env_vars
    env_vars.merge!(smtp_settings).merge!(omniauth_settings)
  end

  def env_vars
    @env_vars ||= Dotenv.parse(".env")
                        .reject { |key, _value| FORBIDDEN_ENVIRONMENT_KEYS.include?(key) }
  end

  def omniauth_settings
    return {} unless @organization.omniauth_settings

    @organization.omniauth_settings.each_with_object({}) do |(key, value), hash|
      hash[key.upcase] = if Decidim::OmniauthProvider.value_defined?(value)
                           decrypt(value)
                         else
                           value
                         end
    end
  end

  def smtp_settings
    settings = @organization.smtp_settings.deep_dup
    settings["password"] = Decidim::AttributeEncryptor.decrypt(settings["encrypted_password"])
    settings.delete("encrypted_password")

    settings.transform_keys do |key|
      "SMTP_#{key.upcase}"
    end
  end

  def organization_columns
    org_columns_sql = "SELECT row_to_json(o,true) FROM (SELECT #{ORGANIZATION_COLUMNS.join(", ")} FROM decidim_organizations WHERE id=#{@organization.id}) AS o;"
    org_columns_record = ActiveRecord::Base.connection.execute(org_columns_sql)
    JSON.parse(org_columns_record.first["row_to_json"])
  end

  def organization_settings
    {
      apiVersion: "apps.libre.sh/v1alpha1",
      kind: "Decidim",
      metadata: {
        name: resource_name
      },
      spec: {
        image: @image,
        host: @organization.host,
        additionalHosts: @organization.secondary_hosts,
        organization: organization_columns,
        timeZone: @organization.time_zone,
        envFrom: [
          {
            secretRef: {
              name: "#{resource_name}-config"
            }
          }
        ]
      }
    }.deep_stringify_keys
  end

  def organization_export_path
    @organization_export_path ||= "#{@export_path}/#{resource_name}"
  end

  def resource_name
    @resource_name ||= "#{@hostname}--#{organization_slug}"
  end

  def bucket_name
    @bucket_name ||= env_vars["SCALEWAY_BUCKET_NAME"]
  end

  def organization_slug
    @organization_slug ||= @organization.host.parameterize(separator: "_", preserve_case: true)
  end

  private

  def decrypt(value)
    Decidim::AttributeEncryptor.decrypt(value)
  rescue ActiveSupport::MessageEncryptor::InvalidMessage
    value
  end
end

Remove if not needed?

https://api.github.com/OpenSourcePolitics/decidim-app/blob/30760c1b86109344038ebab9bf36c3e6e5265623/lib/decidim_app/config.rb#L3

# frozen_string_literal: true

# TODO: Remove if not needed?
module DecidimApp
  module Config
  end

Change log level

Describe the bug
A clear and concise description of what the bug is.

Add environment variable to change log level

Default: :warning

Investigate on flaky causing a double call on rake task when tested

https://api.github.com/OpenSourcePolitics/decidim-app/blob/a9ed482b6e94bdcfec7df5140b76d33df02b859b/spec/lib/tasks/create_admin_task_spec.rb#L17

require "spec_helper"

describe "rake decidim_app:create_admin", type: :task do
  before do
    allow(Decidim::AdminCreator).to receive(:create!).with(ENV).and_return(true)
  end

  it "preloads the Rails environment" do
    expect(task.prerequisites).to include "environment"
  end

  it "invokes the admin creator" do
    task.execute

    # TODO: Investigate on flaky causing a double call on rake task when tested
    expect($stdout.string).to include("Admin created successfully\n")
  end
end

Add tests

decidim-app/lib/tasks/db.rake

Line 59 in b571eaf

# TODO: Add tests

      end

      desc "Delete admin log related to orphans data"
      # TODO: Add tests
      task clean: :environment do
        Rails.logger = Logger.new($stdout)
        # ActiveRecord::Base.logger = Logger.new($stdout)

Image standard Decidim

Fix ambiguous id column on projects query

Describe the bug
A clear and concise description of what the bug is.

When I go to a component of budgets and order the projects by "most voted" after filtering by category an error raises.

Duplicate of decidim/decidim#11469

Stacktrace
If applicable, add the error stacktrace to help explain your problem.

[94507f2e-2872-40a0-b772-6a32f901fe75] ActionView::Template::Error (PG::AmbiguousColumn: ERREUR:  la référence à la colonne « id » est ambigüe
[94507f2e-2872-40a0-b772-6a32f901fe75]      5: </div>
[94507f2e-2872-40a0-b772-6a32f901fe75]      6: 
[94507f2e-2872-40a0-b772-6a32f901fe75]      7: <div class="budget-list">
[94507f2e-2872-40a0-b772-6a32f901fe75]      8:   <% projects.each do |project| %>
[94507f2e-2872-40a0-b772-6a32f901fe75]      9:     <%= render partial: "project", locals: { project: project } %>
[94507f2e-2872-40a0-b772-6a32f901fe75]     10:   <% end %>
[94507f2e-2872-40a0-b772-6a32f901fe75]     11: </div>
[94507f2e-2872-40a0-b772-6a32f901fe75]   

Understand why JSON is used in this case

https://api.github.com/OpenSourcePolitics/decidim-app/blob/2eac3a853585ba710733c63275db6f2b42edef07/lib/k8s_organization_exporter.rb#L90

# frozen_string_literal: true

class K8SOrganizationExporter
  FORBIDDEN_ENVIRONMENT_KEYS = %w(BACKUP_ENABLED
                                  BACKUP_S3SYNC_ENABLED
                                  BACKUP_S3SYNC_ACCESS_KEY
                                  BACKUP_S3SYNC_SECRET_KEY
                                  BACKUP_S3SYNC_BUCKET
                                  BACKUP_S3RETENTION_ENABLED).freeze
  ORGANIZATION_COLUMNS = %w(id
                            default_locale
                            available_locales
                            users_registration_mode
                            force_users_to_authenticate_before_access_organization
                            available_authorizations
                            file_upload_settings).join(", ").freeze

  def initialize(organization, logger, export_path, hostname)
    @organization = organization
    @logger = logger
    @export_path = export_path
    @hostname = hostname
  end

  def self.export!(organization, logger, export_path, hostname)
    new(organization, logger, export_path, hostname).export!
  end

  def export!
    creating_directories
    exporting_env_vars
    exporting_configuration
    dumping_database
    retrieve_active_storage_files
  end

  def retrieve_active_storage_files
    @logger.info("retrieving active storage files from bucket #{bucket_name} into #{organization_export_path}/buckets/#{resource_name}--de")
    system("rclone copy scw-storage:#{bucket_name} #{organization_export_path}/buckets/#{resource_name}--de --config ../scaleway.config --progress --copy-links")
  end

  def dumping_database
    @logger.info("dumping database #{@database_name} to #{organization_export_path}/postgres/#{resource_name}--de.dump")
    system("pg_dump -Fc #{@database_name} > #{organization_export_path}/postgres/#{resource_name}--de.dump")
  end

  def exporting_configuration
    @logger.info("exporting application configuration to #{organization_export_path}/application.yml")
    File.write("#{organization_export_path}/application.yml", YAML.dump(organization_settings))
  end

  def exporting_env_vars
    @logger.info("exporting env variables to #{organization_export_path}/manifests/#{resource_name}--de.yml")
    File.write("#{organization_export_path}/manifests/#{resource_name}-config.yml",
               YAML.dump(env_vars.merge!(smtp_settings).merge!(omniauth_settings)))
  end

  def creating_directories
    @logger.info("creating organization directories")
    @logger.info("#{organization_export_path}/buckets/#{resource_name}--de")
    FileUtils.mkdir_p("#{organization_export_path}/buckets/#{resource_name}--de")
    @logger.info("#{organization_export_path}/manifests")
    FileUtils.mkdir_p("#{organization_export_path}/manifests")
    @logger.info("#{organization_export_path}/postgres")
    FileUtils.mkdir_p("#{organization_export_path}/postgres")
  end

  def env_vars
    @env_vars ||= Dotenv.parse(".env")
                        .reject { |key, _value| FORBIDDEN_ENVIRONMENT_KEYS.include?(key) }
  end

  def omniauth_settings
    @organization.omniauth_settings.each_with_object({}) do |(key, value), hash|
      hash[key.upcase] = Decidim::OmniauthProvider.value_defined?(value) ? Decidim::AttributeEncryptor.decrypt(value) : nil
    end
  end

  def smtp_settings
    settings = @organization.smtp_settings.deep_dup
    settings["password"] = Decidim::AttributeEncryptor.decrypt(settings["encrypted_password"])
    settings.delete("encrypted_password")

    settings.transform_keys do |key|
      "SMTP_#{key.upcase}"
    end
  end

  def organization_columns
    # TODO: Understand why JSON is used in this case
    org_columns_sql = "SELECT row_to_json(o,true) FROM (SELECT #{ORGANIZATION_COLUMNS} FROM decidim_organizations WHERE id=#{@organization.id}) AS o;"
    org_columns_record = ActiveRecord::Base.connection.execute(org_columns_sql)
    JSON.parse(org_columns_record.first["row_to_json"])
  end

  def organization_settings
    {
      apiVersion: "apps.libre.sh/v1alpha1",
      kind: "Decidim",
      metadata: {
        name: resource_name
      },
      spec: {
        image: @image,
        host: @organization.host,
        additionalHosts: @organization.secondary_hosts,
        organization: organization_columns,
        timeZone: @organization.time_zone,
        envFrom: [
          {
            secretRef: {
              name: "#{resource_name}-config"
            }
          }
        ]
      }
    }.deep_stringify_keys
  end

  def organization_export_path
    @organization_export_path ||= "#{@export_path}/#{resource_name}"
  end

  def resource_name
    @resource_name ||= "#{@hostname}--#{@organization_slug}"
  end

  def bucket_name
    @bucket_name ||= env_vars["SCALEWAY_BUCKET_NAME"]
  end

  def organization_slug
    @organization_slug ||= @organization.host.parameterize(separator: "_", preserve_case: true)
  end
end

Investigate on inconstitent results on CI, sometimes the hash generated is diffe...

decidim-app/lib/decidim/assets_hash.rb

Line 42 in 8c2b1f3

# TODO: Investigate on inconstitent results on CI, sometimes the hash generated is different for the slice 1-2 without apparent reason.

    end

    def files_cat(*files)
      # TODO: Investigate on inconstitent results on CI, sometimes the hash generated is different for the slice 1-2 without apparent reason.
      files.map { |pattern| Dir.glob(pattern) }
           .flatten
           .sort

Add tests

decidim-app/lib/tasks/db.rake

Line 25 in b571eaf

# TODO: Add tests

      end

      desc "Delete notifications related to orphans data"
      # TODO: Add tests
      task clean: :environment do
        Rails.logger = Logger.new($stdout)
        # ActiveRecord::Base.logger = Logger.new($stdout)

Customization for France Connect SSO

Description

France Connect usage conditions requires to implement SSO button with an image

Expected behaviour

We must implement FC SSO links with an image

Disable the language selector in the account settings if only one locale is configured

Describe the issue
Currently, when only one locale is configured on a Decidim platform, the language sector is still activated in the account settings, which is unnecessary as only one language can be selected (see screenshot below).

Expected behaviour
When only one locale is configured on Decidim, the language selector is disabled in the account settings.

👉 Related to the Metaproposal https://meta.decidim.org/processes/roadmap/f/122/proposals/17647

Use latest version

https://api.github.com/OpenSourcePolitics/decidim-app/blob/dbe637d3594371f236cc504782f167e7242f95b1/.github/workflows/ci_cd.yml#L159

          key: asset-cache-${{ runner.os }}-${{ steps.cache-hash.outputs.hash }}
      - run: mkdir -p ./spec/tmp/screenshots
        name: Create the screenshots folder
      # TODO: Use latest version
      - uses: nanasess/setup-chromedriver@v2
        with:
          chromedriver-version: "114.0.5735.90"
      - run:  bundle exec rake "test:run[include, spec/system/**/*_spec.rb, ${{ matrix.slice }}]"
        name: RSpec
      - run: ./.github/upload_coverage.sh decidim-app $GITHUB_EVENT_PATH

Missing translation in the image uploader when creating an initiative

Describe the bug
The translation fr.activemodel.attributes.attachment.documents is missing in the image uploader when a user create a new initiative in FO.

To Reproduce

1. Go to /initiatives as a user
2. Create a new initiative
3. Add an image to your initiative
4. See that a translation is missing in the image uploader

Screenshots

Investigate on inconsistency results on CI, sometimes the hash generated is diff...

decidim-app/lib/decidim/assets_hash.rb

Line 26 in dee355d

# TODO: Investigate on inconsistency results on CI, sometimes the hash generated is different for the slice 1-2 without apparent reason.

module Decidim
  class AssetsHash
    def initialize
      @app_dependencies_patterns = %w(Gemfile Gemfile.lock package.json yarn.lock)
      @assets_patterns = %w(app/assets/**/* app/packs/**/* vendor/**/* packages/**/* lib/assets/**/*)
    end

    def self.run
      new.run
    end

    def run
      app_assets_hashes = files_digest(@assets_patterns)
      app_dependencies_hashes = files_digest(@app_dependencies_patterns)

      File.write("tmp/assets_manifest.json", JSON.pretty_generate(app_assets_hashes))

      digest("#{app_assets_hashes.values.join("\n")}#{app_dependencies_hashes.values.join("\n")}")
    end

    def files_digest(patterns)
      # TODO: Investigate on inconsistency results on CI, sometimes the hash generated is different for the slice 1-2 without apparent reason.
      Array.wrap(patterns).map { |pattern| Dir.glob(pattern) }
           .flatten
           .sort
           .each_with_object({}) do |file, result|
        next unless File.file?(file)

        result[file] = digest(File.read(file))
      end
    end

    private

    def digest(value)
      Digest::SHA256.hexdigest(value)
    end
  end
end