joao-conde / feup-lapd

Repository to host the Markup Languages and Document Processing project, a fourth year course @FEUP

Hi, I'm João 👋

I am:

• a software engineer

• a master in computer science

• from the amazing city of Porto, Portugal

📧 You can reach me at [email protected]

feup-lapd's Issues

Pitch Rehearsel

I suggest Wednesday afternoon for a rehearsal of the Pitch and live demo.

Public

Make this beauty public

Code cleanup for the "structure_the_unstructured" function + parallelism

This function is too long and should be split into more readable ones, in doing so, one should also go for the parallelization of the operations, if such is deemed relevant by the group. The function is in the stuns.py file

Avoid duplicate datasets

in the structure_the_unstructured a new dataset is always created regardless of being a duplicate, this should be handle by, for instance, performing a hash of all the files in the dataset folder and checking if it already exists, it should also be included as an optional operation by adding it as a boolean value to the argparse options (see the --verbose case)

Redo the "produce_report" function after the refactor

This will include looking at how it was working before and how the data it received changed, some html knowledge is required

Implement Metrics, according to the script

This involves updating the extract_metrics method in the sensor class, if necessary implement it in child classes that call this one. The metrics are the ones in the report and the ones mentioned in the FRAUNHOFER/FH meeting

Correct ID on inserted documents

Sample code:

from bson.binary import JAVA_LEGACY
from bson.codec_options import CodecOptions
from pymongo import MongoClient
from uuid import uuid4

client = MongoClient(...)
db = client.get_database(<DB_NAME>, CodecOptions(uuid_representation=JAVA_LEGACY)) #DB_NAME should be demdata_db in our case
id = db.cenas.insert_one({'_id': uuid4(), 'other': '...'}).inserted_id #when inserting new documents into the DB, set the _id property as done here
print(id) #printed as UUID4 (converted by driver), in mongo shell appears as BinData(3, ...)

LaTeX Report

After the code

Manual Documentation Cleanup

A cleanup of each "subproduct" (api, app) 's README.md file is required and also a review for the packages used as only those that are strictly necessary, so:

• review README instructions for instalation/deploy
• ensure only required packages/modules are specified in the requirements/packages

Setup with pymongo

• Improve the readme instructions for deplying the db and on which port to access it
• make a PoC of a document inserted in a test-collection in the database
• tell the others

Give context of the user in dispatcher

Because we are processing users separatly, the message of "No dispatcher found for File.txt" or "Dispatcher found" appears N times, being N the number of users. Giving context or showing only once maybe?

Implement new sensors

This task includes a lot of copy paste and should happen after #5

Descomentar demdataui no docker-compose.yml

Submeter

• Paper
• Code (without .git, .ARCHIVE, data folders and report folders)
• updated docs
• presentation

Slides

Sugarcoat with API

• Extra docker image
• Small Flask server with some GET routes focusing on metrics/DB expansion