eddings / get-python-code Goto Github PK

Code for getting Python repositories from Github and saving to a SQL database (specificially IBM DB2).   

Code generated with ChatGPT.  Not yet tested.

Description by ChatGPT for the Github API code:

"Here's a combined version of the code that retrieves all Python repositories on GitHub, retrieves the code for each repository along with its metadata, and stores each file in the repository separately."

"This code retrieves the list of all Python repositories on GitHub sorted by stars, saves each repository's metadata to a repositories table, retrieves all the Python files in each repository, and saves each file's metadata and contents to a files table. Note that this code assumes that the repositories table has a column called name of type VARCHAR or TEXT to store the name of each repository, a column called url of type VARCHAR or TEXT to store the URL of each repository, and a column called stars of type INTEGER to store the number of stars each repository has. If your table has a different schema, you will need to modify the SQL statements in the ibm_db.prepare functions accordingly. Similarly, this code assumes that the files table has a column called name of type VARCHAR or TEXT to store the name of each file, a column called content of type VARCHAR or TEXT to store the contents of each file, and a column called repository_id of type INTEGER to store the ID of the repository that the file belongs to. If your table has a different schema, you will need to modify the SQL statements in the ibm_db.prepare functions accordingly."


Description by ChatGPT for the Stackoverflow API code (stackoverflow.py):

"This code retrieves the list of all Python questions on Stack Overflow sorted by votes, saves each question's metadata to a questions table, retrieves all the code snippets in each question, and saves each snippet's metadata and content to a snippets table. Note that this code assumes that the questions table has a column called title of type VARCHAR or TEXT to store the title of each question, a column called url of type VARCHAR or TEXT to store the URL of each question, and a column called votes of type INTEGER to store the number of votes each question has. If your table has a different schema, you will need to modify the SQL statements in the ibm_db.prepare functions accordingly. Similarly, this code assumes that the snippets table has a column called code of type VARCHAR or TEXT to store the code of each snippet, and a column called question_id of type INTEGER to store the ID of the question that the snippet belongs to. If your table has a different schema, you will need to modify the SQL statements in the ibm_db.prepare functions accordingly."

Description by ChatGPT for the LeetCode API code (leetcode.py):

"This code retrieves the list of all Python solutions on LeetCode sorted by acceptance rate, saves each solution's metadata to a solutions table, retrieves all the code snippets in each solution, and saves each snippet's metadata and content to a snippets table. Note that this code assumes that the solutions table has a column called question_id of type INTEGER to store the ID of the LeetCode question that the solution belongs to, a column called title of type VARCHAR or TEXT to store the title of each question, a column called url of type VARCHAR or TEXT to store the URL of each question, and a column called acceptance_rate of type DECIMAL or FLOAT to store the acceptance rate of each solution. If your table has a different schema, you will need to modify the SQL statements in the ibm_db.prepare functions accordingly. Similarly, this code assumes that the snippets table has a column called code of type VARCHAR or TEXT to store the code of each snippet, and a column called solution_id of type INTEGER to store the ID of the solution that the snippet belongs to. If your table has a different schema, you will need to modify the SQL statements in the ibm_db.prepare functions accordingly."

For Gitlab (gitlab.py):

"This code retrieves the list of all Python repositories on GitLab sorted by stars, saves each repository's metadata to a repositories table, retrieves all the Python files in each repository, and saves each file's metadata and content to a files table. Note that this code assumes that the repositories table has a column called name of type VARCHAR or TEXT to store the name of each repository, a column called url of type VARCHAR or TEXT to store the URL of each repository, and a column called stars of type INTEGER to store the number of stars of each repository. If your table has a different schema, you will need to modify the SQL statements in the ibm_db.prepare functions accordingly. Similarly, this code assumes that the files table has a column called name of type VARCHAR or TEXT to store the name of each file, a column called path of type VARCHAR or TEXT to store the path of each file relative to the repository root, a column called content of type VARCHAR or TEXT to store the content of each file, and a column called repository_id of type INTEGER to store the ID of the repository that the file belongs to. If your table has a different schema, you will need to modify the SQL statements in the ibm_db.prepare functions accordingly. "

For Github Gist (gists.py): 

"This code retrieves the list of all public Gists on GitHub, saves each Gist's metadata to a gists table, retrieves all the Python files in each Gist, and saves each file's metadata and content to a files table. Note that this code assumes that the gists table has a column called url of type VARCHAR or TEXT to store the URL of each Gist, a column called description of type VARCHAR or TEXT to store the description of each Gist, and a column called stars of type INTEGER to store the number of stars of each Gist. If your table has a different schema, you will need to modify the SQL statements in the ibm_db.prepare functions accordingly. Similarly, this code assumes that the files table has a column called name of type VARCHAR or TEXT to store the name of each file, a column called content of type VARCHAR or TEXT to store the content of each file, and a column called gist_id of type INTEGER to store the ID of the Gist that the file belongs to. If your table has a different schema, you will need to modify the SQL statements in the ibm_db.prepare functions accordingly. Note also that this code assumes that you have a personal access token for the GitHub API and that you have replaced <your_access_token> with your token."

Papers with Code (paperswithcode.py):

"Retrieves all papers tagged with "Python" on Papers with Code, saves their metadata to an IBM DB2 database, and also downloads and saves their associated code files in a separate table called papers_code_files."

"Retrieves the associated code files for each paper, saves them to a papers_code_files table in the same database, and links them to the corresponding paper using the paper_id foreign key. Note that you will need to create the papers_code_files table in your IBM DB2 database with columns paper_id (foreign key to papers table), filename, and content before running this code."

Google Scholar (googlescholar.py):

"Unfortunately, Google Scholar does not provide an official API for programmatic access to their search results. As a result, any code that scrapes their website could potentially violate their terms of service. However, there are some third-party libraries that provide a Python interface to Google Scholar, such as scholarly (https://github.com/scholarly-python-package/scholarly). You can use this library to search for papers and retrieve their metadata. Here's an example Python code that uses scholarly to search for papers with the keyword "Python" and saves their metadata to an IBM DB2 database."

PDF files (pdf.py):

"To extract Python code from each PDF file in a directory and save it to an IBM DB2 database, you can use the pdfminer library to extract text from PDF files and the ibm_db library to connect to a DB2 database and insert data."

"This code should extract text from each PDF file in a directory, extract Python code using a regular expression, and save the code to an IBM DB2 database. Note that this code assumes that there is a table named PYTHON_CODE in the database with a column named CODE. You will need to modify the dsn_ variables to match your database connection details, and adjust the regular expression pattern to match the format of the Python code in your PDF files."

Website (website.py):

"This code should start by defining the domain to scrape, and then recursively follow links within the domain starting from the homepage. For each URL, it extracts Python code and metadata using regular expressions, and saves the details to an IBM DB2 database. Note that this code assumes that there is a table named WEBPAGE_DATA in the database with columns named URL, TITLE, DESCRIPTION, and CODE. You will need to modify the dsn_ variables to match your database connection details, and adjust the regular expression patterns to match the format of the Python code and metadata on the website."

Kaggle / Jupyter (ipynb.py):

"Use the Kaggle API to download the Kaggle notebooks. For example, you can download all the notebooks in the "python" category by running !kaggle datasets download -d stack-exchange/python -p /path/to/download/folder in your terminal or command prompt. This will download the notebooks as .zip files to the specified folder."

"This code will extract all of the Python code cells from the Kaggle notebooks in the specified download folder and save them to an IBM DB2 database. You can modify the code to customize the SQL query, change the database connection details, or process the notebooks in a different way."