Integrating SDK and Worker Functionality into py-clean-arch? about py-clean-arch

[cdddg]

Following the title, what would be the integrated project structure?

Project Structure Adjustment

• Objective: Enhance maintainability and scalability.
• Approach: Modularize components into sdk and workers directories.

SDK Integration

• Purpose: Synchronize SDK updates with API changes, ensuring consistency.
• Structure:
  • sdk/ directory under src/ for client logic, data models, and exceptions.
• Reason: Facilitates SDK's close collaboration with the main project and simplifies data model sharing.

Workers Integration

• Purpose: To seamlessly incorporate background tasks, such as Celery tasks and message queue (MQ) consumers, utilizing the logic from the use case layer.
• Structure:
  • workers/ directory under src/ is designated for configurations, task definitions, and scripts for MQ consumers.
• Advantages:
  • Flexibility: This structure allows for easy addition, modification, and removal of background tasks as the project evolves.
  • Scalability: It supports scaling the processing of background tasks independently from the main application, facilitating efficient handling of varying workloads.
• Reason: Centralizing the management of background tasks not only streamlines the project organization but also enhances the ability to handle complex workflows and asynchronous operations, improving overall project performance and maintainability.

Example

./
├── src/
│   ├── di/
│   ├── entrypoints/
│   ├── usecases/
│   ├── models/
│   ├── repositories/
│   ├── common/
│   ├── settings/
│   │
│   ├── workers/                  - Configuration and scripts for background tasks and consumers.
│   │   ├── celery_engine/        - Contains Celery beat and worker configurations. 
│   │   │                         - (Other suggested names: celery_hub, celery_workers, task_scheduler)
│   │   └── mq_consumer/          - Message queue consumer scripts.
│   │
│   ├── sdk/                      - Toolkit for interfacing with the API via HTTPX, returning function calls as project entities.
│   │   └── aio/client.py         - Asynchronous client implementation.
│   │
│   └── main.py                   - Main file to start the application.
└── tests/