openedx-unsupported / edx-certificates

The code edX uses to generate certificates

Archive Notice

This repository is in the process of being Archived. Please see the DEPR ticket for more details: https://openedx.atlassian.net/browse/DEPR-160

edx-certificates

This is the code we use the generate certificates at edX.

Generate edX certificates

This script will continuously monitor an xqueue queue for the purpose of generating a course certificate for a user.

Generating sample certificates

1. Create a new python virtualenv

   mkvirtualenv certificates

2. Clone the certificate repo

   git clone https://github.com/edx/edx-certificates.git

3. Clone the internal certificate repo for templates and private data (optional)

   git clone [email protected]:edx/edx-certificates-internal

4. Install the python requirements into the virtualenv

   pip install -r edx-certificates/requirements/base.txt

5. In order to generate sample certificates that are uploaded to S3 you will need access to the verify-test bucket, create a ~/.boto file in your home directory

   [Credentials]
   aws_access_key_id = *****
   aws_secret_access_key = ****

   • Or for edX use the boto.example in the ${CERT_PRIVATE_DIR} repo:

     cp edx-certificates-internal/boto.example ~/.boto

6. Set an environment variable to point to the internal repo for certificate templates

   export CERT_PRIVATE_DIR=/path/to/edx-certificates-internal

7. In the edx-certificates directory generate a sample certificate:

   cd edx-certificates
   python create_pdfs.py \
       --course-id 'course-v1:edX+DemoX+Demo_Course' \
       --name 'Guido' \
       --no-upload \
       ;

   • course-v1:edX+DemoX+Demo_Course should be a valid course id found in ${CERT_PRIVATE_DIR}

   • View all options with:

     python create_pdfs.py --help

Behavioral Overview

The certificate_agent.py script will continuously monitor a queue for certificate generation, it does the following:

• Connect to the xqueue server
• Poll for a single certificate request
• If it finds one, it:
  • Processes the request
  • Post a results json back to the xqueue server

A global exception handler will catch any error during the certificate generation process and post a result back to the LMS via the xqueue server indicating there was a problem.

optional arguments:
  -h, --help         show this help message and exit
  --aws-id AWS_ID    AWS ID for write access to the S3 bucket
  --aws-key AWS_KEY  AWS KEY for write access to the S3 bucket

Generation overview

TODO

Internationalization and Localization

There is none. Every renderer should have complete Unicode support throughout, but every certificate template render method is built around English-grammar sentence interpolation. Consequently, if the target language of the certificate has very different sentence structure, you may find the easiest way to add support for your language is to fork an existing renderer. We suggest _generate_v3_dynamic_certificate.

If you think that your new renderer would be useful to the international Open edX community, please issue a pull request! We'd love to have 3_dynamic renderers for new languages.

Some members of the Open edX community maintain multiple private modifications to the open source edX projects, and use a branching scheme similar to the one described by Giulio Gratta in this edX Eng blog post. If you have many changes to the code base, a system like this might help you stay organized. If, on the other hand you are only forking one template renderer, this may be too complex. Please do whatever works best for your site.

Logging

Logging is setup similar to Django logging, logsettings.py will generate a configuration dict for logging where in a production environment all log messages are sent through rsyslog

Tests

To run the test suite:

1. Configure your credential information in settings.py. You will need to specify:

    CERT_KEY_ID = # The id for the key which will be used by gpg to sign certificates
    CERT_AWS_ID = # Amazon Web Services ID
    CERT_AWS_KEY = # Amazon Web Services Key
    CERT_BUCKET = # Amazon Web Services S3 bucket name
   

   It is also acceptable to leave the AWS KEY and ID values as none and instead use .boto file or run this code from a server that has an IAM role that gives it write access to the bucket in the configuration.

2. To run all of the tests from the certificates directory, run:

    nosetests
   

   These are more integration tests than unit tests, and will be exercising your certificate configuration, your file pathing, and your S3 credentials. Some tests may fail, but the code may still be working properly; you'll have to investigate to discover what the failed test is diagnostic of. To run just the tests for local on-disk publishing run:

    nosetests tests.gen_cert_test:test_cert_gen
   

Troubleshooting:

• If tests fail with errors, try running:

  pip install -r requirements/base.txt

  to install necessary requirements.

• If your verification pages are wrong or gnupg starts throwing errors, you may not have gpg installed. See gnupg for help installing.

• If you run create_pdf.py and get unicode errors, you may need to set the LC_ALL environment variable. To set it permanently, like from your ~/.bashrc, try:

  export LC_ALL=en_US.UTF-8

  But if you only want to set it for one test run, you could say:

  LC_ALL=en_US.UTF-8 python ./create_pdf.py

• If you are running on a Linux virtual machine being hosted by MacOS, and your git checkout is being NFS mounted, you may have library import errors because of Mac case-folding semantic preservation by Python's pathlib. Try doing a git checkout inside the vm that isn't exported from MacOS.

Roadmap/TODO/Future Features

• Paralellism - Certification should be embarassingly parallel, except we deal with xqueue, which lacks atomic pop(). If we ever refactor queue.py to use raw celery queues or similar, we should also actualize parallel certification.

• Dynamic scaling and placement of signatures from scanned bitmaps, making cert rendering completely dynamic and freeing us from the tyranny of template preparation. (Freeing us from the tyranny of configuration preparation would come in a future PR.)

• Kill XQueue - nobody really likes xqueue. Several ops people have expressed a desire to see it replaced by smarter intermediate layers that use celery task queues or similar. This repo should be sufficiently modular that only queue.py should need to be changed to work this way.

edx-certificates's Issues

make template repo migration tool

Today I migrated our old cert configuration to the new format in I python in about 5 minutes. It just occurred to me that it would help edx.org run the new repo if I automated that. It's a small job but I'm on the train without a seat so this is a reminder.

to do make a template readme

Write a readme in templates making clear that the external template repo has precisely that structure but is under separate source control. @jrbl

Could I pull _generate_eduX_certificate in Chinese?

@jrbl
As title ,Could I pull _generate_eduX_certificate in Chinese like stanford/stanford_cme

edx-certificate-internal

Wiki says anyone needs to clone this repo [email protected]:edx/edx-certificates-internal bu5t this is not public.

Need a release tag

Appreciate if you can please add a release tag.

Fix UnicodeEncodeError for long course names

I'm coming from a fresh Aspen release (OPENEDX_RELEASE=aspen.1) installation of the edX Full Stack on a single Ubuntu 12.04 64-bit at AWS E2 and after restore some courses I decided to generate some certificates. As the courses are in Portuguese most of the students and course names have non-ascii characters that should be decode using utf-8.

So I can see the decode method has been called (.decode('utf-8')) but for long course names it seems to called twice raising an UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128) error.

My solution (after pull the last commits) was remove the .decode('utf-8') method ESOedX@48aa56e#diff-dca6aff9cafd628c46cf3346375d02a3R1170 (please check my commit) but I'm not sure if it's the best solution. If yes I can make a pull request.

After pull the last commits from master I got this error below which I solved removing .decode('utf-8') from COURSE_LONG=self.long_course line 1170.

/edx/app/edxapp/edx-platform$ sudo less /edx/var/log/supervisor/certstderr.log

Traceback (most recent call last):
  File "/edx/app/certs/certificates/certificate_agent.py", line 198, in <module>
    main()
  File "/edx/app/certs/certificates/certificate_agent.py", line 130, in main
    download_url) = cert.create_and_upload(name.encode('utf-8'), grade=grade, designation=designation)
  File "/edx/app/certs/certificates/gen_cert.py", line 313, in create_and_upload
    designation=designation,)
  File "/edx/app/certs/certificates/gen_cert.py", line 385, in _generate_certificate
    designation,
  File "/edx/app/certs/certificates/gen_cert.py", line 576, in _generate_v1_certificate
    self.course, self.long_course)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)

Position of student name, course name in generated certificate

Position of student name, course name has to be re-arranged in generated certificate according to the number of text it contains rather than fixing in a static position.

New template

Hi ! i am investing about how this works, so i want to know if i need a new model i need to program a new method or just fork and modified generate_vX_certificate method ?

Missing robots.txt

Search engines seem to be indexing potentially sensitive user data on this site (search "site:verify.edx.org"). I believe this can be fixed by simply adding an appropriate robots.txt.