cordery / django-countries-plus

Django model & fixture representing all top level country data from Geonames.org

Django Countries Plus

django-countries-plus provides a model and fixture containing all top level country data from Geonames.org (http://download.geonames.org/export/dump/countryInfo.txt)

This package also provides a convenience middleware that will look up a country in the database using a defined meta header, ex: the Cloudflare provided geoip META header HTTP_CF_IPCOUNTRY. This country object will be attached to the request object at request.country.

Country Model

The model provides the following fields (original geonames.org column name in parentheses).

• iso (ISO)
• iso3 (ISO3)
• iso_numeric (ISO-Numeric)
• fips (fips)
• name (Country)
• capital
• area (Area(in sq km))
• population (population)
• continent (continent)
• tld (tld)
• currency_code (CurrencyCode)
• currency_name (CurrencyName)
• currency_symbol (Not part of the original table)
• phone (Phone)
• postal_code_format (Postal Code Format)
• postal_code_regex (Postal Code Regex)
• languages (Languages)
• geonameid (geonameid)
• neighbors (neighbours)
• equivalent_fips_code (EquivalentFipsCode)

Installation

Step 1: Install From PyPi

pip install django-countries-plus

Step 2: Add countries_plus to your settings INSTALLED_APPS

Step 3: Run python manage.py migrate

Step 4: Load the Countries Data

1. Load the countries data into your database with the update_countries_plus management command.

   python manage.py update_countries_plus

2. (alternative) Load the provided fixture from the fixtures directory.

   python manage.py loaddata PATH_TO_COUNTRIES_PLUS/countries_plus/countries_data.json.gz

Usage

Retrieve a Country:

from countries_plus.models import Country
usa = Country.objects.get(iso3='USA')

Update the countries data with the latest geonames.org data:

python manage.py update_countries_plus

This management command will download the latest geonames.org countries data and convert it into Country objects. Existing Country objects will be updated if necessary. No Country objects will be deleted, even if that country has ceased to exist.

Add the Request Country to each Request

1. Add countries_plus.middleware.AddRequestCountryMiddleware to your MIDDLEWARE setting.

2. add the following two settings to your settings.py:

   COUNTRIES_PLUS_COUNTRY_HEADER - A string defining the name of the meta header that provides the country code. Ex: 'HTTP_CF_COUNTRY' (from https://support.cloudflare.com/hc/en-us/articles/200168236-What-does-CloudFlare-IP-Geolocation-do-)

   COUNTRIES_PLUS_DEFAULT_ISO - A string containing an iso code for the country you want to use as a fallback in the case of a missing or malformed geoip header. Ex: 'US' or 'DE' or 'BR'

   Example:

   COUNTRIES_PLUS_COUNTRY_HEADER = 'HTTP_CF_COUNTRY'
   COUNTRIES_PLUS_DEFAULT_ISO = 'US'
   

Add the Request Country to the Request Context

1. Enable the optional middleware as described above
2. Add countries_plus.context_processors.add_request_country to your 'context_processors' option in the OPTIONS of a DjangoTemplates backend instead (Django 1.8)

Requirements

Python: 3.7+ Django: Tested against the LTS or latest versions of 2.2, 3, and 4.

Integrating with django-languages-plus

If you also have django-languages-plus(https://pypi.python.org/pypi/django-languages-plus) installed then you can run the following command once to associate the two datasets and generate a list of culture codes (pt_BR for example):

from languages_plus.utils import associate_countries_and_languages
associate_countries_and_languages()

Notes on 1.0.1

• Two countries (Dominican Republic and Puerto Rico) have two phone number prefixes instead of 1. These prefixes are now comma separated.
• The Country model has had all fields with undefined lengths (ex: name) expanded to max_length=255. Defined length fields (ex: Iso, Iso3) are unchanged.
• The Country model will no validate on save and reject values of the wrong length. The test suite has been expanded to test this.

Notes on 1.0.0

• The data migration has been removed in favour of the new management command and manually loading the fixture.
• The fixture is no longer named initial_data and so must be loaded manually, if desired.
• In order to provide better compatibility with the way Django loads apps the Country model is no longer importable directly from countries_plus.
• The get_country_by_request utility function has been moved into the Country model, and is available as Country.get_by_request(request)
• Test coverage has been substantially improved.
• If you have been running an earlier version you should run python manage.py update_countries_plus to update your data tables as they may contain incorrect data.

Troubleshooting

I get the following error when trying to run a migration adding a new ForeignKey to the Country model:

django.db.utils.OperationalError: (3780, "Referencing column 'new_country_id' and referenced column 'iso' in foreign key constraint 'companies_company_new_country_id_1a75fd29_fk_countries' are incompatible.")

Due to a decision made many years ago that cannot be easily changed now, countries_plus uses a CharField(max_length=2) for its primary key (the iso column). This means that in MySQL and probably other databases the charset and collation of the two fields (your ForeignKey and the Country.iso) field must be identical. Default collations may change over time, for example MySQL changed its default charset to utf8mb4 and collation to utf8mb4_0900_ai_ci in 8.0. This can cause new tables (such as countries_plus_country) to be created with a different collation than older tables that were migrated to a newer version of MySQL.

To solve the problem, either:

• convert the countries_plus_country table to use the older collation than your related table is using.
• or perhaps preferably, convert your related table to use the new utf8mb4_0900_ai_ci collation that the countries_plus table is using (and any other new tables in your database)

For example, running the following would fix the issue by converting your related table:

ALTER TABLE <YOUR_TABLE> CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;

Running Tests

Does the code actually work?

$ poetry install
$ poetry run pytest

Or for the full tox suite:

$ poetry install
$ pip install tox
$ tox

django-countries-plus's Issues

table <database_name>.countries_plus_country does not exist

After an error I had to delete my database and recreate it from scratch.
I have now an empty database.
If I run python manage.py makemigrations I get a django.db.utils.Programming Error :
table <database_name>.countries_plus_country does not exist

same problem if I run python manage.py migrate.

How can I create the countries_plus_country in my database after installing pycountry and django_countries_plus

DB OperationalError during migration when adding a Country foreignkey to an existing model

Hi back mates !
Thanks for the (very) quick fix to my latest issue, I am unfortunately coming to you with a new one ...

I am trying to migrate my models from django-countries to django-countries-plus.
I have a Company model with a "country" CountryField from django-countries. I am now adding a Country foreignkey "new_country".

from countries_plus.models import Country
from django_countries.fields import CountryField  # pragma: no cover

class Company(models.Model):  # pragma: no cover
   ....
    country = CountryField(blank=True)
    new_country = models.ForeignKey(Country, blank=True, null=True, on_delete=models.SET_NULL) # added during new migration

The migration file appears to show the right instructions:

    dependencies = [
        ('countries_plus', '0006_alter_country_options'),
        ('companies', '0014_individualactor_keyword_company_is_relevant_and_more'),
    ]

    operations = [
        migrations.AddField(
            model_name='company',
            name='new_country',
            field=models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL,
                                    to='countries_plus.country'),
        ),
    ]

But the migration cannot go through, with the following logs:

  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (3780, "Referencing column 'new_country_id' and referenced column 'iso' in foreign key constraint 'companies_company_new_country_id_1a75fd29_fk_countries' are incompatible.")

I have run django-countries-plus successfully in the past and processed 0006_alter_country_options on a past migration. Countries have been installed with manage.py update_countries_plus.

I am running django 4.0.3, django-countries 7.3.2, django-countries-plus 2.1.0.
Currently the mysql DB is running utf8mb4_0900_ai_ci

Nationalities from Countries.

Am trying to access the Nationalities from this library, forexample :

Country : Uganda, Nationality : Ugandan.
Country : America, Nationality: American.

etc.

How can this be achieved

Geonames file format change?

Tried running python manage.py update_countries_plus today in a new project and ran into the following error:

countries_plus.utils.GeonamesParseError: I couldn't parse the Geonames file (http://download.geonames.org/export/dump/countryInfo.txt). The format may have changed. An updated version of this software may be required, please check for updates and/or raise an issue on github. Specific error: Unexpected field length: {'area': [u'Ensure that there are no more than 1 decimal place.']}

This error also exists in other projects where update_countries_plus previously worked without issue.

All of the projects (the new one + the old ones) are using v1.0.1 from pip.

Missing "search_fields" in CountryAdmin

Hey ! I just found this module and it's really useful !

I am trying to enable auto-complete on a Country foreignkey field in the CompanyAdmin class of my Company model.

autocomplete_fields = ['country']

However I get the following error:

CountryAdmin must define "search_fields", because it\'s referenced by CompanyAdmin.autocomplete_fields

I wonder if there is a workaround to have auto-complete enabled on a ForeignKeyField for easier selection of country.
Otherwise, is it planned to eventually have a "search_fields" attribute in the CountryAdmin class ?

Thanks in advance,

Provide factories for tests

Since data migrations are problematic in tests, and it may be faster to not rely on them (e.g. with pytest-django's --reuse-db), it would be helpful to have factories for this.

I am using factory-boy myself. Do you think it makes sense to provide factories through django-countries-plus itself?

(the same applies to django-languages-plus)

For example:

class CountryFactory(factory.django.DjangoModelFactory):
    """Factory for a Country"""
    class Meta:
        model = 'countries_plus.models.Country'
        django_get_or_create = ('name', 'iso')

    name = factory.Sequence(lambda n: "Country %d" % n)
    iso = factory.Sequence(lambda n: str(n))
    iso3 = factory.Sequence(lambda n: str(n))
    iso_numeric = factory.Sequence(lambda n: n)

Change gettext to gettext_lazy in Country model

In general it is better to use gettext_lazy:
https://github.com/cordery/django-countries-plus/blob/1eb0a0b61c83669224c33952b3fb4096c83cb587/countries_plus/models.py#L6C38-L6C45

phone field is too short: psycopg2.DataError: value too long for type character varying(5)

I am seeing the following error when running the update_countries_plus command:

Traceback (most recent call last):
  File "…/pyenv/project/lib/python3.4/site-packages/django/db/backends/utils.py", line 67, in execute
    return self.cursor.execute(sql, params)
psycopg2.DataError: value too long for type character varying(5)

sql:
UPDATE "countries_plus_country" SET "iso3" = %s, "iso_numeric" = %s, "fips" = %s, "name" = %s, "capital" = %s, "area" = %s, "population" = %s, "continent" = %s, "tld" = %s, "currency_code" = %s, "currency_symbol" = %s, "currency_name" = %s, "phone" = %s, "postal_code_format" = NULL, "postal_code_regex" = NULL, "languages" = %s, "geoname id" = %s, "neighbours" = %s, "equivalent_fips_code" = NULL WHERE "countries_plus_country"."iso" = %s

params:
('ATG', 28, 'AC', 'Antigua and Barbuda', "St. John's", '443.0', 86754, 'NA', '.ag', 'XCD', '$', 'Dollar', '+1-268', 'en-AG', 3576396, '', 'AG')

The "varying(5)" field seems to be phone, and the value here is '+1-268'.

Add UN as a country

We are developing an application that requires the United Nations listed as a country. Is it possible to add "United Nations (country)" with "UN" as its alpha-2 code to the list of countries? This country and code is reserved directly by ISO 3166/MA for the United Nations according to https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2

Fixtures mistakes

There are mistakes in your fixtures. The codes of several countries are incorrect. For example, for Trinidad, the phone code should be 1-868. In your fixtures, it is -686!

update_countries_plus - broken

I have tested this issue with various python versions and installations (dokku, bare metal, dev and production), and can confirm that python manage.py update_countries_plus fails with following traceback:

python manage.py update_countries_plus
Traceback (most recent call last):
File "manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/home/indigo/.pyenv/versions/3.6.5/lib/python3.6/site-packages/django/core/management/init.py", line 381, in execute_from_command_line
utility.execute()
File "/home/indigo/.pyenv/versions/3.6.5/lib/python3.6/site-packages/django/core/management/init.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/indigo/.pyenv/versions/3.6.5/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/indigo/.pyenv/versions/3.6.5/lib/python3.6/site-packages/django/core/management/base.py", line 364, in execute
output = self.handle(*args, **options)
File "/home/indigo/.pyenv/versions/3.6.5/lib/python3.6/site-packages/countries_plus/management/commands/update_countries_plus.py", line 10, in handle
num_updated, num_created = update_geonames_data()
File "/home/indigo/.pyenv/versions/3.6.5/lib/python3.6/site-packages/countries_plus/utils.py", line 218, in update_geonames_data
return parse_geonames_data(r.iter_lines())
File "/home/indigo/.pyenv/versions/3.6.5/lib/python3.6/site-packages/countries_plus/utils.py", line 235, in parse_geonames_data
if line[0] == "#":
IndexError: string index out of range

I suspect a change to geonames.org dataset causes this to fail.