0x48piraj / rqmts

rqmts - Generate pip requirements.txt file for any project

Rqmts

Rqmts - Generate requirements.txt file for any project by analysing package imports


Click here to see the demo.
Click here for documentation.

About the project

Rqmts is a fantastic stand-alone tool which generates requirements.txt file for any project by analysing package imports.

It does not requires any dependency (works out-of-the-box), not needs internet to work (is completely offline, upto this moment), nor uses regular expressions in such a violent way as existing projects do. Instead, it uses simple heuristic techniques and parse conditional trees, which is a better method for extracting imported names from statements, functions, etc.

Why this project

Questions

• Why not just use pip's freeze command to generate a requirements.txt file for my project?
• Why to re-invent the wheel when there are modules such as pipreqs, pigar, poetry already present?
• Why not manually?

Answers

• Why not just pip freeze?
  • pip freeze only saves the packages that are installed with pip install in your environment.
  • pip freeze saves all packages in the environment including those that you don't use in your current project. (if you don't have virtualenv)
• Why re-invent the wheel (?)
  • pipreqs fails on many occasions (see - pipreqs/issues)
  • pigar queries pypi servers, big no-no. Ideally, it should be local, on fallback? then maybe. Other than that, pigar recommends using Pipenv (pipenv has serious issues)
  • poetry quotes "Be aware, however, that it will also install poetry's dependencies which might cause conflicts."
  • Sheer curiousity. "can I create a project that has potential of collecting thosands of stars and most importantly, hundreds of contributors?"
• Manually?
  • Are you serious right now?

Installation

Use pip to install rqmts. This is the recommended way of running Rqmts.

Rqmts also provides a script that will run a rqmts instance isolated from the rest of your system by using file-less/memory-based execution.

curl -sSL https://raw.githubusercontent.com/0x48piraj/rqmts/master/Rqmts.py | python

Alternatively, you can download Rqmts.py from the root directory and execute it separately.

Usage

Command-line Interface and Interactive mode

Challenges

The major challenge of this project is to extract the required metadata from modules which are first extracted from the input script.

• Version numbers in python can be in very different places depending on the case
• Package name in the package index is independent of the module name we import

and these quirks make this project interesting. There's a funny comment in the source which reflects the diversity between us and it goes like :

# module_name.__version__ sucks, because we suck (PEP 0396)

This project aims to combine the best existing strategies to cover the broadest possible set of cases (if not all). The project was built keeping in mind the modular programming paradigms and so other than being readable it's easily extensible making it possible to add new strategies/algorithms quickly.

Changelog

Playing with different versions? Read the Changelog.

Contribute

All patches are Welcome! Please see CONTRIBUTING.md for further details. For discussions, create a new issue or ping me over Twitter

Again, if you have any issues or suggestions/patches, please do not hesitate to open an issue or a pull request!

Running the tests

This project uses unittest.

The Python unit testing framework, sometimes referred to as "PyUnit," is a Python language version of JUnit, by Kent Beck and Erich Gamma. JUnit is, in turn, a Java version of Kent’s Smalltalk testing framework. Each is the de facto standard unit testing framework for its respective language.

You may need to install the package (rqmts) for setup beforehand, using

pip install --user rqmts

All tests are encapsulated in one single script named testsuite.py and all the respective test-cases are under tests/testcases/

For getting started,

python tests/testsuite.py

License

This software is licensed under BSD 3-Clause "New" or "Revised" License. To view a copy of this license, visit BSD 3-Clause.

rqmts's Issues

Trailing newline on final requirements.txt

Problematic LoC

            g.write(req + '\n')

Code reference

rqmts/rqmts/__main__.py

Line 145 in 1326f02

with open(REQS_PATH, 'w') as g:

shallow and deep scan modes

Modes :

• --shallow : fast but prone to errors
• --deep : slow but accurate

Combining both the demos

Screen-casting new demos of rrmts in action.

Problematic selenium module parsing

While parsing tests/testcases/test.py, it incorrectly parses and records the following line,

from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

as selenium.webdriver.common.desired_capabilities which then corrupts the version fetching sub-routine.

Problematic code reference

rqmts/rqmts/parsing.py

Line 20 in 1326f02

def parse(code):

Terminal string styling

Making the script clean, and colorful.

Update the Changelog [Documentation]

Update the Changelog by seeing rqmts/releases

Better ASCII Logo

Designing a better ASCII logo. Current logo can be found in here

Fails to detect on some modules

Current behavior :

rqmts.py fails to detect packages which have different names.

Expected behavior :

Successfully detects and parses out the dependency (package version)

Logcat

C:\Users\rqmts>py -3 rqmts.py
Enter path of the python script : C:\tests\run.py
[!] System package: `re`
[!] System package: `time`
[!] System package: `string`
[!] System package: `strgen`  <---- NOT A SYSTEM PACKAGE
[!] System package: `random`
[!] System package: `os`
[!] System package: `time`
[+] Success: Parsed the dependencies correctly
[*] Saving generated requirements.txt
[+] Success: requirements.txt saved
[+] Path where it can be found: C:\tests\requirements.txt

References :

1. pypi-strgen
2. pip install StringGenerator

Doesn't work on non-win32 os

Required change:
class style():
BLACK = ""
RED = ""
GREEN = ""
YELLOW = ""
BLUE = ""
MAGENTA = ""
CYAN = ""
WHITE = ""
UNDERLINE = ""
RESET = ""
to
class style():
BLACK = lambda x: str(x)
RED = lambda x: str(x)
GREEN = lambda x: str(x)
YELLOW = lambda x: str(x)
BLUE = lambda x: str(x)
MAGENTA = lambda x: str(x)
CYAN = lambda x: str(x)
WHITE = lambda x: str(x)
UNDERLINE = lambda x: str(x)
RESET = lambda x: str(x)

Writing unit tests

[test.py]

import lxml
import requests
import re
import time
import string
import strgen
from random import randint
import os
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import time
from faker import Faker
from bs4 import BeautifulSoup
import pymouse

Saving the generated requirements.txt

Trailing whitespaces corrupt file_path

LOGCAT

[*] Path not provided, invoking interactive mode ...
[*] Enter the path of Python script
    ---->  C:\Users\XXX\Desktop\rqmts\tests\test.py
[-] Invalid path. Exiting ...

Handling custom module imports

rqmts stops upon spotting a custom module import.

LOGCAT :
input :
"import numpy as np
import cv2
from mss import mss
import time
from getkeys import key_check
import os"

Output

Some modules were not successfully imported, and it means either of the two things :
1. No such module(s) (just ignore this warning then)
2. Module(s) not found in your system. (pip install <module-name>)
There seems to be discrepancies in following module(s) :
1 getkeys
Quitting.

Optimizing the codebase

Areas that can be improved/modified :

• Scans some modules more than once.
• several pip show calls.

Discrepancies in modules spotted but no modules listed

Output

Some modules were not successfully imported, and it means either of the two things :
1. No such module(s) (just ignore this warning then)
2. Module(s) not found in your system. (pip install <module-name>)
There seems to be discrepancies in following module(s) :
Quitting.

imports -
"import numpy as np
from PIL import Image
import cv2
import matplotlib.pyplot as plt
import pickle

from matplotlib import style
import time
"

Implementing modular software design

Making the project readable, reliable and maintainable without too much effort.

Implement Command-line Interface

rqmts.py --path path/to/file
rqmts.py -p path/to/file

rqmts.py --interactive
rqmts.py -i

rqmts.py --help
rqmts.py -h

Unreliable module parsing

LOGCAT :

--REDACTED--
[*] Path not provided, invoking interactive mode ...
[*] Enter the path of Python script
    ----> C:\CryptoDecrypt.py
Error importing n that each byte's msb is a parity bit
Error importing and the input for decryption.
Error importing y be "hex"
Error importing "base64"
Error importing Total number of users compromised : "
Error importing str(c))
Error importing the strength is really 112 bits).
Error importing ":" ,crypt.decryptStringENC(json['IONLogin'][i]['data']))
Error importing "url"
Error importing or "quoted-printable".
Error importing use a key length of 128
Error importing i
Error importing -Key Triple-DES
Error importing ption
[!] System package found : json
[!] System package found : sys
[!] System package found : i
[!] System package found : ption
[!] System package found : chilkat
[+] Success: Parsed the dependencies correctly
[*] Saving generated requirements.txt
[+] Success: requirements.txt saved
[+] Path where it can be found: C:\requirements.txt

C:\>

Flag for verbose output

Adding a --verbose/-v flag to argparse

Enhance `usage` group in cmdline iface

Current behavior :

usage: __main__.py [-h] [-p FILE]

Expected behavior :

usage: rqmts [-h] [-p FILE]

[Question] no working on ubuntu 22.04 bash or zsh

Hi @0x48piraj

I have been trying to run this tool, but I got: