GithubHelp home page GithubHelp logo

aws / porting-advisor-for-graviton Goto Github PK

View Code? Open in Web Editor NEW
147.0 17.0 24.0 182 KB

License: Apache License 2.0

PowerShell 2.23% Shell 3.07% Python 81.37% Go 0.06% Java 7.32% HTML 5.62% Ruby 0.13% Dockerfile 0.21%

porting-advisor-for-graviton's Introduction

Porting Advisor for Graviton

This is a fork of Porting advisor, an open source project by the ARM High Performance Computing group. Originally, it was coded as a Python module that analyzed some known incompatibilities for C and Fortran code.

It is a command line tool that analyzes source code for known code patterns and dependency libraries. It then generates a report with any incompatibilities with our Graviton processors. This tool provides suggestions of minimal required and/or recommended versions to run on Graviton instances for both language runtime and dependency libraries. It can run on non-ARM based machines (no Graviton processor needed). This tool does not work on binaries, just source code. It does not make any code modifications, it doesn’t make API level recommendations, nor does it send data back to AWS.

PLEASE NOTE: Even though we do our best to find known incompatibilities, we still recommend to perform the appropriate tests to your application on a Graviton instance before going to Production.

This tool scans all files in a source tree, regardless of whether they are included by the build system or not. As such it may erroneously report issues in files that appear in the source tree but are excluded by the build system. Currently, the tool supports the following languages/dependencies:

  • Python 3+
    • Python version
    • PIP version
    • Dependency versions in requirements.txt file
  • Java 8+
    • Java version
    • Dependency versions in pom.xml file
    • JAR scanning for native method calls (requires JAVA to be installed)
  • Go 1.11+
    • Go version
    • Dependency versions on go.mod file
  • C, C++, Fortran
    • Inline assembly with no corresponding aarch64 inline assembly.
    • Assembly source files with no corresponding aarch64 assembly source files.
    • Missing aarch64 architecture detection in autoconf config.guess scripts.
    • Linking against libraries that are not available on the aarch64 architecture.
    • Use of architecture specific intrinsic.
    • Preprocessor errors that trigger when compiling on aarch64.
    • Use of old Visual C++ runtime (Windows specific).
    • The following types of issues are detected, but not reported by default:
      • Compiler specific code guarded by compiler specific pre-defined macros.
    • The following types of cross-compile specific issues are detected, but not reported by default.
      • Architecture detection that depends on the host rather than the target.
      • Use of build artifacts in the build process.

For more information on how to modify issues reported, use the tool’s built-in help:

./porting-advisor-linux-x86_64 -–help

If you run into any issues, see our CONTRIBUTING file.

How to run:

As a container

By using this option, you don't need to worry about Python or Java versions, or any other dependency that the tool needs. This is the quickest way to get started.

Pre-requisites

Build container image

NOTE: if using containerd, you can substitute docker with nerdctl

docker build -t porting-advisor .

NOTE: on Windows you might need to run these commands to avoid bash scripts having their line ends changed to CRLF:

git config core.autocrlf false
git reset --hard

Run container image

After building the image, we can run the tool as a container. We use -v to mount a volume from our host machine to the container.

We can run it directly to console:

docker run --rm -v my/repo/path:/repo porting-advisor /repo

Or generate a report:

docker run --rm -v my/repo/path:/repo -v my/output:/output porting-advisor /repo --output /output/report.html

Windows example:

docker run --rm -v /c/Users/myuser/repo:/repo -v /c/Users/myuser/output:/output porting-advisor /repo --output /output/report.html

As a Python script

Pre-requisites

  • Python 3.10 or above (with PIP3 and venv module installed).
  • (Optionally) Open JDK 17 (or above) and Maven 3.5 (or above) if you want to scan JAR files for native methods.

Enable Python Environment

Linux/Mac:

python3 -m venv .venv
source .venv/bin/activate

Powershell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1

Install requirements

pip3 install -r requirements.txt

Run tool (console output)

python3 src/porting-advisor.py ~/my/path/to/my/repo

Run tool (HTML report)

python3 src/porting-advisor.py ~/my/path/to/my/repo --output report.html

As a binary

Generating the binary

Pre-requisites

  • Python 3.10 or above (with PIP3 and venv module installed).
  • (Optionally) Open JDK 17 (or above) and Maven 3.5 (or above) if you want the binary to be able to scan JAR files for native methods.

The build.sh script will generate a self-contained binary (for Linux/MacOS). It will be output to a folder called dist.

By default, it will generate a binary named like porting-advisor-linux-x86_64. You can customize generated filename by setting environment variable FILE_NAME.

./build.sh

For Windows, the Build.ps1 will generate a folder with an EXE and all the files it requires to run.

.\Build.ps1

Running the binary

Pre-requisites

Once you have the binary generated, it will only require Java 11 Runtime (or above) if you want to scan JAR files for native methods. Otherwise, the file is self-contained and doesn't need Python to run.

Default behaviour, console output:

$ ./porting-advisor-linux-x86_64 ~/my/path/to/my/repo

Generating HTML report:

$ ./porting-advisor-linux-x86_64 ~/my/path/to/my/repo --output report.html

Generating a report of just dependencies (this creates an Excel file with just the dependencies we found on the repo, no suggestions provided):

$ ./porting-advisor-linux-x86_64 ~/my/path/to/my/repo --output dependencies.xlsx --output-format dependencies

Sample console report output:

./dist/porting-advisor-linux-x86_64 ./sample-projects/
| Elapsed Time: 0:00:03

Porting Advisor for Graviton v1.0.0
Report date: 2023-01-06 23:48:20

13 files scanned.
detected java code. we recommend using Corretto. see https://aws.amazon.com/corretto/ for more details.
detected python code. if you need pip, version 19.3 or above is recommended. we detected that you have version 22.2.1.
detected python code. min version 3.7.5 is required. we detected that you have version 3.10.6. see https://github.com/aws/aws-graviton-getting-started/blob/main/python.md for more details.
./sample-projects/java-samples/pom.xml: dependency library: leveldbjni-all is not supported on Graviton
./sample-projects/java-samples/pom.xml: using dependency library snappy-java version 1.1.3. upgrade to at least version 1.1.4
./sample-projects/java-samples/pom.xml: using dependency library zstd-jni version 1.1.0. upgrade to at least version 1.2.0
./sample-projects/python-samples/incompatible/requirements.txt:3: using dependency library OpenBLAS version 0.3.16. upgrade to at least version 0.3.17
detected go code. min version 1.16 is required. version 1.18 or above is recommended. we detected that you have version 1.15. see https://github.com/aws/aws-graviton-getting-started/blob/main/golang.md for more details.
./sample-projects/java-samples/pom.xml: using dependency library hadoop-lzo. this library requires a manual build  more info at: https://github.com/aws/aws-graviton-getting-started/blob/main/java.md#building-multi-arch-jars
./sample-projects/python-samples/incompatible/requirements.txt:5: dependency library NumPy is present. min version 1.19.0 is required.
detected java code. min version 8 is required. version 11 or above is recommended. see https://github.com/aws/aws-graviton-getting-started/blob/main/java.md for more details.

Use --output FILENAME.html to generate an HTML report.

porting-advisor-for-graviton's People

Contributors

amazon-auto avatar as14692 avatar beau-gosse-dev avatar danielkleinstein avatar dependabot[bot] avatar jamolina avatar jasonrandrews avatar mrkdeng avatar natict avatar santosh-at-github avatar yubingjiaocn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

porting-advisor-for-graviton's Issues

Testing bug report

Describe the bug

A clear and concise description of what the bug is.

Expected behavior

A clear and concise description of what you expected to happen.

Steps to Reproduce

Steps to reproduce the behavior:

  1. Run with parameters '...'
  2. Open report...
  3. Scroll down to '...'
  4. See error

Screenshots

If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information)

  • OS: [e.g. Windows, macOS, Linux, etc]
  • Version/Distro: [e.g. 11, Big Sur, Ubuntu 18.04]
  • Processor Architecture: [e.g. x86/x64, ARM]
  • Python Version:
  • Java Version (if applicable):

Additional context

Add any other context about the problem here.

Add Binary

I see that a containerized version of this tool is available now. In order to integrate this tool into our ecosystem, it would help if there was a pre-built binary available in this repo. This would save in building times and manual administration.

Testing, please ignore

Describe the bug

A clear and concise description of what the bug is.

Expected behavior

A clear and concise description of what you expected to happen.

Steps to Reproduce

Steps to reproduce the behavior:

  1. Run with parameters '...'
  2. Open report...
  3. Scroll down to '...'
  4. See error

Screenshots

If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information)

  • OS: [e.g. Windows, macOS, Linux, etc]
  • Version/Distro: [e.g. 11, Big Sur, Ubuntu 18.04]
  • Processor Architecture: [e.g. x86/x64, ARM]
  • Python Version:
  • Java Version (if applicable):

Additional context

Add any other context about the problem here.

Canary alarm triggers false postive

Describe the bug

Since the alarm triggers if a canary isn't run over the period of 1 day, if day 1's canary runs at 12:00 PM and day 2 runs at 12:01 PM, the alarm will trigger. And alarms can't be set to scan more than 1 day.

Expected behavior

No alarm if run a few minutes off

Steps to Reproduce

Wait for canaries to run

On Windows PowerShell not scanning the jar files

Describe the bug

Tried jar scanning for org.eclipse.swt.win32.win32.x86-4.3.jar. It is working as expected in Linux and MacOS but on Windows (Windows Server 2022 Datacenter 64bit) it is not reporting the expected incompatibility.

Expected behavior

It is expected to report the incompatibility in the Windows PowerShell

Steps to Reproduce

Steps to reproduce the behavior:

OpenJDK 18 (I tried in 17 as well), Python 3.11.3, Maven 3.8.8

  1. Download jar org.eclipse.swt.win32.win32.x86-4.3.jar \testrepo
  2. python src/porting-advisor.py .\testrepo

This should report incompatibilities like this

testrepo/lib/org.eclipse.swt.win32.win32.x86-4.3.jar: JAR has native methods but no libraries found for aarch64/Linux. Native methods: static final native int org.eclipse.swt.awt.SWT_AWT.getAWTHandle(java.awt.Canvas),..........continued listing all native methods...

But if you run this in Windows, the above incompatibility is not getting reported instead it says below

| Elapsed Time: 0:00:08

Porting Advisor for Graviton v1.0.0
Report date: 2023-04-27 05:57:29

1 file scanned.
detected java code. we recommend using Corretto. see https://aws.amazon.com/corretto/ for more details.
detected java code. min version 8 is required. version 11 or above is recommended. see https://github.com/aws/aws-graviton-getting-started/blob/main/java.md for more details.

Report generated successfully. Hint: you can use --output FILENAME.html to generate an HTML report.

Screenshots

If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information)

  • OS: [e.g. Windows, macOS, Linux, etc] Windows
  • Version/Distro: [e.g. 11, Big Sur, Ubuntu 18.04] Windows 2022
  • Processor Architecture: [e.g. x86/x64, ARM] x64
  • Python Version: 3.11
  • Java Version (if applicable): 17,18

Additional context

Add any other context about the problem here.

Print a descriptive message when running as a script on Python < 3.10

Description

When running as a script, even on current AL2/AL2023 porting advisor fails with an exception due to old Python version installed by default.

Solution idea(s)

Print a descriptive message to guide users who missed the prerequisite for Python 3.10+

Additional context

None.

build.sh fails on Amazon Linux 2023 with Python 3.11

Describe the bug

build.sh fails on AL 2023 with Python 3.11 on Graviton instances (c6g)

Expected behavior

build.sh will complete and generate the executable in the dist/ directory

Steps to Reproduce

Steps to reproduce the behavior:

  1. Create c6g instance with AL 2023
  2. Install software
sudo dnf install git -y
sudo dnf install python3.11 -y
  1. Clone repo
git clone https://github.com/aws/porting-advisor-for-graviton.git
cd porting-advisor-for-graviton
  1. Create Python virtual environment
python3.11 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
  1. Build
./build.sh
  1. Error
ERROR: No matching distribution found for pyinstaller==5.0.1

Desktop

  • OS: Amazon Linux 2023
  • Processor Architecture: aarch64
  • Python Version: 3.11

JavaScript Support

Description

Please add support for language: JavaScript & Typescript

Solution idea(s)

It should be have exactly as it is currently working but with support for Javascript and Typescript

[Other Issue]

Description of issue

I try to run the tool to scan java application but I have the following output

Porting Advisor for Graviton v1.0.0
Report date: 2023-02-14 17:01:13

1 files scanned.
detected java code. we recommend using Corretto. see https://aws.amazon.com/corretto/ for more details.
detected java code. min version 8 is required. version 11 or above is recommended. see https://github.com/aws/aws-graviton-getting-started/blob/main/java.md for more details.

image

I have Corretto 17 installed with Maven 3.9, one thing to note that the project is gradle project.
Am I using it wrong?
Thanks

Ability to execute on demand using a GH Bot

As a customer with multiple repos, it would be useful to run porting-advisor-for-graviton on demand on my codebase, either through mentions within github, or custom actions.

Deliver binary or container image format

Description

Current release only has source code. Since porting advisor requires Python 3.10 or above, this pre-req makes porting advisor difficult to run on Amazon Linux 2, which only ships with Python 3.7 and no further version available.

Solution idea(s)

Deliver pre-built binary or self-contained container image on each release.

C# / .Net support

Description

According to the Readme .Net / C# (dotnet) is not supported by I got my hopes up after I found sample-projects/dotnet-samples/ with a sample .csproj. However I get "0 files scanned".

Solution idea(s)

  • C# / .Net support. Parse .csproj and .sln at least for .Net "Core"
  • Add .sln to sample project
  • Update Readme to mention C# / .Net support

Additional context

We're using .Net 6 (LTS). Though I read somewhere that also 4.8 supports ARM nowadays.

Output of python3 src/porting-advisor.py ./sample-projects/dotnet-samples (on macOS 13.2, Darwin Kernel Version 22.3.0, Python 3.10.9):

| Elapsed Time: 0:00:00

Porting Advisor for Graviton v1.0.0
Report date: 2023-02-09 12:16:29

0 files scanned.
No issues found.

Ability to determine whether a python module listed in requirements.txt has a aarch64 version

Description

When Porting advisor for Graviton evaluates a python code base, it checks the python version and verify the content of the requirements.txt file against a list of know packages for which it has a recommended version.
However, if a package that is not part of the list of know packages is available in PyPI (or any other repository) and missing a aarch64 wheel (in the case of a binary package), porting advisor doesn't issue a warning.

Solution idea(s)

For each package listed in requirements.txt, Porting advisor for Graviton should check on PyPI (or any other repository listed in requirements.txt) whether the package is binary or pure python, check whether there's a aarch64 binary wheel for the package and indicate the minimum version.
If there's no aarch64 package available, it should show a warning.

Graviton Test issue

Describe the bug

A clear and concise description of what the bug is.

Expected behavior

A clear and concise description of what you expected to happen.

Steps to Reproduce

Steps to reproduce the behavior:

  1. Run with parameters '...'
  2. Open report...
  3. Scroll down to '...'
  4. See error

Screenshots

If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information)

  • OS: [e.g. Windows, macOS, Linux, etc]
  • Version/Distro: [e.g. 11, Big Sur, Ubuntu 18.04]
  • Processor Architecture: [e.g. x86/x64, ARM]
  • Python Version:
  • Java Version (if applicable):

Additional context

Add any other context about the problem here.

setup-environment.sh isn't using python3.10 or higher even if it is available in PATH while creating PAG binary

Describe the bug

While trying to create binary for the PAG, I get into errors. When I run "setup-environment.sh" in debug mode (bash -x), I see that it usages default Python version on the instance (python 3.9 in case of Amazon Linux 2023) even if a newer python version (3.11) is available in PATH.

Here is the error I am getting when I run the PAG binary:

[ec2-user@ip-172-31-30-166 porting-advisor-for-graviton]$ ./dist/porting-advisor-linux-x86_64 sample-projects/java-samples
Traceback (most recent call last):
  File "porting-advisor.py", line 23, in <module>
  File "advisor/__init__.py", line 26, in main
  File "PyInstaller/loader/pyimod02_importers.py", line 499, in exec_module
  File "advisor/main.py", line 28, in <module>
ModuleNotFoundError: No module named 'advisor.reports.report_factory'
[28175] Failed to execute script 'porting-advisor' due to unhandled exception!
[ec2-user@ip-172-31-30-166 porting-advisor-for-graviton]$

Expected behavior

setup-environment.sh should automatically identify python executable with version 3.10 or higher available in PATH or throw error if not found.

Steps to Reproduce

Steps to reproduce the behavior:

  1. Launch AL2023 instance using AMI ami-06ca3ca175f37dd66 in us-east-1 region.
  2. SSH to the instance and execute below commands
sudo yum install python3.11 python3-pip java-11-amazon-corretto maven git -y
pip3 install pip --upgrade
git clone https://github.com/aws/porting-advisor-for-graviton.git
cd porting-advisor-for-graviton
./build.sh
./dist/porting-advisor-linux-x86_64 
  1. The built binary throws error related to advisor while trying to execute it.
  2. And when I try to install advisor, it also throws error:
AssertionError: Multiple .egg-info directories found

Screenshots

If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information)

  • OS: amazon Linux 2023
  • latest
  • Processor Architecture: x86/x64 and ARM64
  • Python Version: NA
  • Java Version (if applicable): NA

Additional context

Add any other context about the problem here.

porting-advisor is not able to detect some issues

Describe the bug

I used porting-advisor to scan a repo, it says No issues found.

image

When I am trying to build that application on graviton, it gives me above error.
In one of the cmake file, it has following code snippets, but porting-advisor didn't detect it

# Determine CPU architecture
if (   CMAKE_C_COMPILER_ARCHITECTURE_ID   MATCHES "(x86|X86|x64|X64|amd64|AMD64|i386|i686)"
    OR CMAKE_CXX_COMPILER_ARCHITECTURE_ID MATCHES "(x86|X86|x64|X64|amd64|AMD64|i386|i686)"
    OR CMAKE_SYSTEM_PROCESSOR             MATCHES "(x86|X86|x64|X64|amd64|AMD64|i386|i686)"
    )
    set(onesdk_arch "x86")
elseif (   CMAKE_C_COMPILER_ARCHITECTURE_ID   MATCHES "sparc"
        OR CMAKE_CXX_COMPILER_ARCHITECTURE_ID MATCHES "sparc"
        OR CMAKE_SYSTEM_PROCESSOR MATCHES "sparc")
    set(onesdk_arch "sparc")
else ()
    message(SEND_ERROR "CMAKE_C_COMPILER_ARCHITECTURE_ID = ${CMAKE_C_COMPILER_ARCHITECTURE_ID}")
    message(SEND_ERROR "CMAKE_CXX_COMPILER_ARCHITECTURE_ID = ${CMAKE_CXX_COMPILER_ARCHITECTURE_ID}")
    message(SEND_ERROR "CMAKE_SYSTEM_PROCESSOR = ${CMAKE_SYSTEM_PROCESSOR}")
    message(FATAL_ERROR "ONESDK: Unsupported architecture (see above).")
endif ()

Expected behavior

I expect porting advisor can detect above issues, because if I know it has issues, then I will not directly built it on graviton because I know it has issues, following is my expected behavior:
Something like: Issues detected: don't support aarch64

Steps to Reproduce

Steps to reproduce the behavior:

  1. Scan https://github.com/Dynatrace/OneAgent-SDK-for-C with porting-advisor
  2. you will see it says no issues found

Screenshots

image

Desktop (please complete the following information)

I guess it doesn't matter in this case?

Second test

Describe the bug

A clear and concise description of what the bug is.

Expected behavior

A clear and concise description of what you expected to happen.

Steps to Reproduce

Steps to reproduce the behavior:

  1. Run with parameters '...'
  2. Open report...
  3. Scroll down to '...'
  4. See error

Screenshots

If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information)

  • OS: [e.g. Windows, macOS, Linux, etc]
  • Version/Distro: [e.g. 11, Big Sur, Ubuntu 18.04]
  • Processor Architecture: [e.g. x86/x64, ARM]
  • Python Version:
  • Java Version (if applicable):

Additional context

Add any other context about the problem here.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.