This application parses the Epidoc XML files that store transcriptions for the Inscriptions in Ancient Israel and Palestine project.
It consists of three main parts:
- Create a virtual environment:
python -m venv my_venv
- Activate the virtual environment:
source ./my_venv/bin/activate
Note that you might need to source
a different activate
script depending on your shell. If you're using fish
, for example, you'll run
source ./my_venv/bin/activate.fish
- Install dependencies within the activated virtualenv:
pip install -r requirements.txt
- Run the ingestion script:
python ./ingest_inscriptions.py
The ingestion script uses upserts on data that can change, so you can rerun the script whenever you need.
Source: epidoc_parser.py
The Epidoc Parser, as its name suggests, handles the Epidoc inscription files. If it seems like some data isn't being pulled in, it's a good idea to start by looking here.
You'll see a function defined for each unit of data --- e.g., rather than trying to glom everything onto a large inscription
object, we start by pulling out things like the description
:
def get_description(self):
commentary = self.tree.xpath(
"//tei:div[@type = 'commentary']/tei:p/text()", namespaces=NAMESPACES
)
if len(commentary) > 0:
return whitespace_regex.sub(" ", commentary[0])
return None
If you add a field, make sure to add it in ingest_inscriptions.py as well --- otherwise it won't be persisted to the database.
Source: db.py
Models: models.py
Inscriptions are stored in a fairly compact relational database. The main Inscription
model contains most of the action:
@dataclass
class Inscription(Base):
__tablename__ = "inscriptions"
id: Mapped[int] = mapped_column(primary_key=True)
...
An Inscription
belongs_to
a number of other objects (City
, Provenance
, etc. --- occasionally, as with Language
, we need a join table), and it has_many
Edition
objects.
Every Edition
must have an edition_type
, defined as
class EditionType(enum.Enum):
DIPLOMATIC = "diplomatic"
TRANSCRIPTION = "transcription"
TRANSCRIPTION_SEGMENTED = "transcription_segmented"
TRANSLATION = "translation"
We're using PostgreSQL's trigram index support to handle search.
Source: app.py
The web server defines a RESTful interface for interacting with the data. For the most part, they'll want to make GET
requests to /inscriptions
, with or without the query parameters to define their search criteria.
As a convenience (and breaking with REST conventions), we provide a /map/inscriptions
endpoint that returns all matching inscriptions, but without performing many of the JOIN
s that take place for the paginated requests at /inscriptions
.
You can get a general idea of how deployment works locally by running docker compose up
from the root directory of this repository.
On Reclaim, we only use the Dockerfile
, rather than docker compose
, as Reclaim provides the Postgres instance for us.