vitasdk / vita-toolchain

vita-toolchain

These are a set of tools that, along with an ARM compiler and linker, provides a build system for PlayStation Vita(TM). Look at buildscripts for more information on building the complete toolchain with binutils/gcc/newlib. As such, this repo only contains Vita specific host tools. Check out the specifications for more details on the Vita executable format, these tools, and their inputs/outputs (including the YAML configuration format).

vita-toolchain file formats

name	processor	description
elf	-	A normal .elf format
velf	vita-elf-create.c	The vita's elf format (elf + some module infos)
self	-	The signed velf file + self headers
fself	vita-make-fself.c	The self file. but not signed. (fake self)
nid_db_json	-	really old nid db format
nid_db_classic	vita-import-parse.c vita-nid-db-yml.c	Base of the most used formats currently
nid_db_classic_v2	vita-import-parse.c vita-nid-db-yml.c	Added library version key to nid_db_classic
nid_db_classic_v3	vita-nid-db-yml.c	Added library stubname key to nid_db_classic_v2
nid_db_bypass	vita-nid-bypass.c	The yml format to bypass duplicate entry names in vita-libs-gen-2
module_config	vita-export-parse.c	The yml format for more detailed module configuration

vita-elf-create

usage: vita-elf-create [-v|vv|vvv] [-n] [-e config.yml] input.elf output.velf
    -v,-vv,-vvv:    logging verbosity (more v is more verbose)
    -s         :    strip the output ELF
    -n         :    allow empty imports
    -e yml     :    optional config options
    -g yml     :    generate an export config from ELF symbols
    -m list    :    specify the list of module entrypoints
    -p         :    skip stub privilege check
    input.elf  :    input ARM ET_EXEC type ELF
    output.velf:    output ET_SCE_RELEXEC type ELF

Converts a standard ET_EXEC ELF (outputted by arm-vita-eabi-gcc for example) to the Sony ELF format.

vita-elf-create also adds special symbols defined programmatically to module info.

type	mode	name	prototype	used by
variable	C	module_sdk_version	const SceUInt32 module_sdk_version;	Kernel
variable	C	sceUserMainThreadName	const char sceUserMainThreadName[];	Kernel
variable	C	sceUserMainThreadPriority	const SceUInt32 sceUserMainThreadPriority;	Kernel
variable	C	sceUserMainThreadStackSize	const SceSize sceUserMainThreadStackSize;	Kernel
variable	C	sceUserMainThreadCpuAffinityMask	const SceUInt32 sceUserMainThreadCpuAffinityMask;	Kernel
variable	C	sceUserMainThreadAttribute	const SceUInt32 sceUserMainThreadAttribute;	Kernel
variable	C	sceKernelPreloadModuleInhibit	const SceUInt32 sceKernelPreloadModuleInhibit;	Kernel
variable	C	sceLibcHeapUnitSize1MiB	unknown	SceLibc
variable	C	sceLibcHeapSize	SceSize sceLibcHeapSize;	SceLibc
variable	C	sceLibcHeapInitialSize	unknown	SceLibc
variable	C	sceLibcHeapExtendedAlloc	unknown	SceLibc
variable	C	sceLibcHeapDelayedAlloc	unknown	SceLibc
variable	C	sceLibcHeapDetectOverrun	unknown	SceLibc
function	C	user_malloc_init	unknown	SceLibc
function	C	user_malloc_finalize	unknown	SceLibc
function	C	user_malloc	unknown	SceLibc
function	C	user_free	unknown	SceLibc
function	C	user_calloc	unknown	SceLibc
function	C	user_realloc	unknown	SceLibc
function	C	user_memalign	unknown	SceLibc
function	C	user_reallocalign	unknown	SceLibc
function	C	user_malloc_stats	unknown	SceLibc
function	C	user_malloc_stats_fast	unknown	SceLibc
function	C	user_malloc_usable_size	unknown	SceLibc
function	C	user_malloc_for_tls_init	unknown	SceLibc
function	C	user_malloc_for_tls_finalize	unknown	SceLibc
function	C	user_malloc_for_tls	unknown	SceLibc
function	C	user_free_for_tls	unknown	SceLibc
function	C++	_Z8user_newj	user_new(std::size_t) throw(std::badalloc)	SceLibc
function	C++	_Z8user_newjRKSt9nothrow_t	user_new(std::size_t, std::nothrow_t const&)	SceLibc
function	C++	_Z14user_new_arrayj	user_new_array(std::size_t) throw(std::badalloc)	SceLibc
function	C++	_Z14user_new_arrayjRKSt9nothrow_t	user_new_array(std::size_t, std::nothrow_t const&)	SceLibc
function	C++	_Z11user_deletePv	user_delete(void*)	SceLibc
function	C++	_Z11user_deletePvRKSt9nothrow_t	user_delete(void*, std::nothrow_t const&)	SceLibc
function	C++	_Z17user_delete_arrayPv	user_delete_array(void*)	SceLibc
function	C++	_Z17user_delete_arrayPvRKSt9nothrow_t	user_delete_array(void*, std::nothrow_t const&)	SceLibc

vita-elf-export

usage: vita-elf-export mod-type elf exports imports
    mod-type: valid values: 'u'/'user' for user mode, else 'k'/'kernel' for kernel mode
    elf: path to the elf produced by the toolchain to be used by vita-elf-create
    exports: path to the config yaml file specifying the module information and exports
    imports: path to write the import yaml generated by this tool

Creates an import database YAML from a Sony ELF and config YAML. This import database is a YAML file (not to be confused with the config YAML file) which defines the NID mappings for a given module. This can be used by other tools for debugging and reversing purposes.

vita-libs-gen

usage: vita-libs-gen nids.yml [extra.yml ...] output-dir
    -c: Generate CMakeLists.txt instead of a Makefile

Given a list of import database YAML files (either from vita-elf-create or manually written by hackers for Sony owned modules), this will generate stub libraries that can be linked to such that vita-elf-create can properly generate a Sony ELF. After calling vita-libs-gen you need to run make or cmake in the output directory to build the stubs.

vita-libs-gen supports nid_db_classic_v2 yml file

Warning: You cannot add more than 1024 entries to a single library in cmake mode.

vita-libs-gen-2

usage: vita-libs-gen-2 -yml=<nids_db.yml|nids_db_yml_dir> -output=<output_dir> [-cmake=<true|false>] [-ignore-stubname=<true|false>]

Enhanced version of vita-libs-gen.

• better cleeanup for make version
• no 1024 limite for entries number in cmake mode
• support multi yml in once
• support stubname mapping

vita-libs-gen-2 supports nid_db_classic_v3 yml file

vita-nid-check

usage: vita-nid-check -dbdirver=<./path/to/db_dir> [-bypass=<./path/to/bypass.yml>] [-dbg=<debug|trace>] 

Checks all files in the selected directory.

• Is yml file
• Is entry sorted
• Is no duplicated name

Also cannot contain multiple firmware versions within the selected directory.

vita-nid-check supports nid_db_classic_v3 yml file

vita-make-fself

usage: vita-make-fself [-s|-ss] [-c] input.velf output-eboot.bin
    -s : Generate a safe eboot.bin. A safe eboot.bin does not have access
    to restricted APIs and important parts of the filesystem.
    -ss: Generate a secret-safe eboot.bin. Do not use this option if you don't know what it does.
    -c : Enable compression.

Generates a FSELF (the format expected of eboot.bin loaded by Vita) which wraps around the Sony ELF file. Optionally supports compression of the input ELF. Also allows marking a homebrew as "safe", which prevents it from harming the system.

vita-mksfoex

usage: mksfoex [options] TITLE output.sfo
    -d NAME=VALUE   Add a new DWORD value
    -s NAME=STR     Add a new string value

Generates a param.sfo file for a given title. param.sfo is required to launch a homebrew from LiveArea. The TITLE id can be anything but it is recommended that you use XXXXYYYYY where XXXX is an author specific identifier and YYYYY is a unique number identifying your homebrew. For example, molecularShell uses MLCL00001.

vita-pack-vpk

Usage:
    vita-pack-vpk [OPTIONS] output.vpk

  -s, --sfo=param.sfo     sets the param.sfo file
  -b, --eboot=eboot.bin   sets the eboot.bin file
  -a, --add src=dst       adds the file src to the vpk as dst
  -h, --help              displays this help and exit

Generates a VPK homebrew package. eboot.bin and param.sfo are required.

Development

Required libraries are libelf, zlib, libzip, and libyaml. Please note that there are some compatibility problems with built-in libelf so it is recommended that you download it from the provided link.

After getting the dependencies, build with

mkdir build && cd build
cmake ..
make

Note on Naming

Early in the development, there was a confusion on the meaning of "module" and "library" in context of the Vita. After the tools were written initially, we decided to reverse the meaning of those two words. All user-facing usage of those words have been changed (console outputs, messages, etc) but it is too much word to change all internal usage of those words (function/variable names, etc). Therefore you may be confused when reading the source since the meaning of "module" and "library" is used inconsistently. It would be great if someone could take the time to correct all the usages ("module" exports one or more "libraries" and imports zero or more "libraries"; eboot.bin is a module).

vita-toolchain's Issues

cmake toolchain does not support vita-make-fself secretsafe option

vita-elf-create crashes by ENOMEM under MinGW

I found sometimes calloc in sce-elf.c failed with errno ENOMEM under MinGW
Revert 0f5e16b fixed it.
I tried to look into the commit but found nothing for the issue.
@173210 any idea of this?

Implement compression for fself

This should be implemented in vita-make-fself where each segment is called on deflate and the segment info header is set to 2 (update offset/size accordingly).

Exports

Having exports will benefit homebrews and by the time plugins are supported, it will be very useful.

Unit Tests

Unit tests and regressions would be nice.

Fix C++ exceptions / threads

Throwing an exception on the Vita and catching it crashes the program.

Temporary workaround by @DaveeFTW:

Add this before main():

extern "C"
{
    unsigned int sleep(unsigned int seconds)
    {
        sceKernelDelayThread(seconds*1000*1000);
        return 0;
    }

    int usleep(useconds_t usec)
    {
        sceKernelDelayThread(usec);
        return 0;
    }

    void __sinit(struct _reent *);
}

__attribute__((constructor(101)))
void pthread_setup(void) 
{
    pthread_init();
    __sinit(_REENT);
}

Add the following to target_link_libraries():

-Wl,--whole-archive pthread -Wl,--no-whole-archive

Module Exports' size field is 1 byte, not 2

Probably for imports as well. See: SceKernelThreadMgr.

Ensure that generated sections are 0x4 aligned

I'm seeing this

.sceLib.stubs:0002575B ; ===========================================================================
.sceLib.stubs:0002575B
.sceLib.stubs:0002575B ; Segment type: Pure code

From this sample
taihen.velf.zip

This crashes the kernel on load because of the unaligned access.

Can not find command "-fbracket-depth=512"

This is the exact error I am having:

cd /Users/Ingwie/Work/SDKs/vita-toolchain/buildscripts/out/gcc-base-prefix/src/gcc-base-build && CFLAGS=-DNO_IMPLICIT_EXTERN_C -fbracket-depth=512 CXXFLAGS=-DNO_IMPLICIT_EXTERN_C -fbracket-depth=512 _CFLAGS= _CXXFLAGS= _CPPFLAGS= _LDFLAGS= /Users/Ingwie/Work/SDKs/vita-toolchain/buildscripts/command_wrapper.sh /Users/Ingwie/Work/SDKs/vita-toolchain/buildscripts/out/gcc-base-prefix/src/gcc-base/configure --build=x86_64-apple-darwin --host=x86_64-apple-darwin --target=arm-vita-eabi --prefix=/Users/Ingwie/Work/SDKs/vita-toolchain/buildscripts/out/toolchain_build --libdir=/Users/Ingwie/Work/SDKs/vita-toolchain/buildscripts/out/toolchain_build/lib --libexecdir=/Users/Ingwie/Work/SDKs/vita-toolchain/buildscripts/out/toolchain_build/lib --with-sysroot=/Users/Ingwie/Work/SDKs/vita-toolchain/buildscripts/out/toolchain_build/arm-vita-eabi --with-gmp=/Users/Ingwie/Work/SDKs/vita-toolchain/buildscripts/out/deps_build --with-mpfr=/Users/Ingwie/Work/SDKs/vita-toolchain/buildscripts/out/deps_build --with-mpc=/Users/Ingwie/Work/SDKs/vita-toolchain/buildscripts/out/deps_build --with-isl=/Users/Ingwie/Work/SDKs/vita-toolchain/buildscripts/out/deps_build --with-libelf=/Users/Ingwie/Work/SDKs/vita-toolchain/buildscripts/out/deps_build --with-python-dir=share/gcc-arm-vita-eabi --enable-languages=c,c++ --disable-decimal-float --disable-libffi --disable-libgomp --disable-libmudflap --disable-libquadmath --disable-libssp --disable-libstdcxx-pch --disable-nls --disable-shared --disable-tls --with-gnu-as --with-gnu-ld --with-newlib --disable-multilib --with-arch=armv7-a --with-tune=cortex-a9 --with-fpu=neon --with-float=hard --with-mode=thumb "--with-pkgversion=GNU Tools for ARM Embedded Processors" --disable-threads --without-headers
/bin/sh: -fbracket-depth=512: command not found
make[2]: *** [gcc-base-prefix/src/gcc-base-stamp/gcc-base-configure] Error 127
make[1]: *** [CMakeFiles/gcc-base.dir/all] Error 2
make: *** [all] Error 2

I am on macOS 10.14 and CMake 3.13.3, have obtained libyaml through Homebrew and have used this CMake command: cmake -DCMAKE_INSTALL_PREFIX=/usr/local/vitasdk/ ...

Remove need to "preprocess" ELF

Any valid ELF that linked with the stubs should work with the tools. We shouldn't assume things like debug info stripped.

build.sh fails when patching libzip

I think the version of libzip is wrong. IMHO, Having a build script other than buildscripts is considered harmful since it could cause such problems.

$ ./build.sh
[Step 0.0] Clone buildscripts...
Cloning into 'buildscripts'...
remote: Counting objects: 455, done.
remote: Total 455 (delta 0), reused 0 (delta 0), pack-reused 455
Receiving objects: 100% (455/455), 347.52 KiB | 312.00 KiB/s, done.
Resolving deltas: 100% (271/271), done.
Checking connectivity... done.
[Step 1.0] Create dirs...
[Step 1.1] Build zlib...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  440k  100  440k    0     0   146k      0  0:00:02  0:00:02 --:--:--  188k
Checking for gcc...
Building static library libz.a version 1.2.8 with gcc.
Checking for off64_t... No.
Checking for fseeko... Yes.
Checking for strerror... Yes.
Checking for unistd.h... Yes.
Checking for stdarg.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
Checking for attribute(visibility) support... No.
gcc -O3  -I. -c -o example.o test/example.c
gcc -O3    -c -o adler32.o adler32.c
gcc -O3    -c -o crc32.o crc32.c
gcc -O3    -c -o deflate.o deflate.c
gcc -O3    -c -o infback.o infback.c
gcc -O3    -c -o inffast.o inffast.c
gcc -O3    -c -o inflate.o inflate.c
gcc -O3    -c -o inftrees.o inftrees.c
gcc -O3    -c -o trees.o trees.c
gcc -O3    -c -o zutil.o zutil.c
gcc -O3    -c -o compress.o compress.c
gcc -O3    -c -o uncompr.o uncompr.c
gcc -O3    -c -o gzclose.o gzclose.c
gcc -O3    -c -o gzlib.o gzlib.c
gcc -O3    -c -o gzread.o gzread.c
gcc -O3    -c -o gzwrite.o gzwrite.c
gcc -O3  -I. -c -o minigzip.o test/minigzip.c
ar rc libz.a adler32.o crc32.o deflate.o infback.o inffast.o inflate.o inftrees.o trees.o zutil.o compress.o uncompr.o gzclose.o gzlib.o gzread.o gzwrite.o
gcc -O3  -o example example.o -L. libz.a
gcc -O3  -o minigzip minigzip.o -L. libz.a
cp libz.a /home/root3/vita-toolchain/install/lib
chmod 644 /home/root3/vita-toolchain/install/lib/libz.a
cp zlib.3 /home/root3/vita-toolchain/install/share/man/man3
chmod 644 /home/root3/vita-toolchain/install/share/man/man3/zlib.3
cp zlib.pc /home/root3/vita-toolchain/install/lib/pkgconfig
chmod 644 /home/root3/vita-toolchain/install/lib/pkgconfig/zlib.pc
cp zlib.h zconf.h /home/root3/vita-toolchain/install/include
chmod 644 /home/root3/vita-toolchain/install/include/zlib.h /home/root3/vita-toolchain/install/include/zconf.h
[Step 1.2] Build libzip...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  449k  100  449k    0     0   113k      0  0:00:03  0:00:03 --:--:--  142k
patching file CMakeLists.txt
Hunk #1 FAILED at 183.
1 out of 1 hunk FAILED -- saving rejects to file CMakeLists.txt.rej

Broken section names in output .velf

Somehow section names end up being cut, e.g. in the final .velf these sections appear: data.vstubs, ubs. Reproducible on Vita Doom.

Sign extend branch target in decode_rel_target

Two patches: http://pastie.org/10252178 http://pastie.org/10252200

We should probably rewrite it in a way that's easier to understand.

vita-libs-gen should produce 1 object file per function/variable export

Currently it produces 1 object file per module. When linked to a homebrew executable it will add all exports defined in this module, even if they aren't used by the homebrew. It's probably a better idea to just generate 1 object file per function or variable export so that unneeded function imports aren't linked into the final homebrew executable.

Reloc to the first function in a segment produces invalid relocation

For functions we have vrela->addend either -4 (thumb) or -8 (arm), then when it gets to symvaddr = vrela->symbol->value + vrela->addend; in sce_elf_write_rela_sections the resulting address will be before the segment start vaddr. As a result vita_elf_vaddr_to_segndx returns -1 and we get a broken converted relocation.

Remove need to look at db.json in vita-elf-create

Right now we need db.json as an input to get the names of modules/libraries. We can instead encode this information elsewhere. For example, in the section name as .sceLibs.SceSysmemForKernel or something. This would also remove the requirement of having to distribute a .json for any shared library.

Support weak imports

Proposed change: in stubs, change the first 4 bytes to be a "flags" field. Set to 0x8 for weak import. Then vita-libs-gen will create both a _stub and a _weak_stub for each non-kernel stub library. vita-elf-create will generate "return -1" for weak imports.

Create a reference toolchain

We should have a reference/binary distribution of the vita-toolchain tools along with an ARM toolchain (clang anyone?), headers, and a complete JSON database.

In the comments, suggest what we should bundle.

Fix alignment issue

We need to align segments to 0x1000 but somehow that results in problems.

vita-create-elf error

I am getting this error when compiling VitaShell on windows with the latest prebuilt toolchain.

vita-elf-create VitaShell.elf VitaShell.velf libpromoter/promoterutil.json
/var/atlassian/application-data/bamboo/xml-data/build-dir/PT-WM-BM/buildscripts/
build/vita-toolchain_host-prefix/src/vita-toolchain_host/src/vita-elf.c: Unable
to find module with NID 1586302498 for function symbol sceKernelExitThread/var/a
tlassian/application-data/bamboo/xml-data/build-dir/PT-WM-BM/buildscripts/build/
vita-toolchain_host-prefix/src/vita-toolchain_host/src/vita-elf.c: Unable to fin
d module with NID 2241471665 for function symbol sceKernelGetProcessId/var/atlas
sian/application-data/bamboo/xml-data/build-dir/PT-WM-BM/buildscripts/build/vita
-toolchain_host-prefix/src/vita-toolchain_host/src/vita-elf.c: Unable to find mo
dule with NID 2241471665 for function symbol sceKernelDeleteMutex/var/atlassian/
application-data/bamboo/xml-data/build-dir/PT-WM-BM/buildscripts/build/vita-tool
chain_host-prefix/src/vita-toolchain_host/src/vita-elf.c: Unable to find module
with NID 2241471665 for function symbol sceKernelDeleteThread/var/atlassian/appl
ication-data/bamboo/xml-data/build-dir/PT-WM-BM/buildscripts/build/vita-toolchai
n_host-prefix/src/vita-toolchain_host/src/vita-elf.c: Unable to find module with
 NID 2241471665 for function symbol sceKernelDelayThread/var/atlassian/applicati
on-data/bamboo/xml-data/build-dir/PT-WM-BM/buildscripts/build/vita-toolchain_hos
t-prefix/src/vita-toolchain_host/src/vita-elf.c: Unable to find module with NID
2241471665 for function symbol sceKernelExitDeleteThread/var/atlassian/applicati
on-data/bamboo/xml-data/build-dir/PT-WM-BM/buildscripts/build/vita-toolchain_hos
t-prefix/src/vita-toolchain_host/src/vita-elf.c: Unable to find module with NID
2241471665 for function symbol sceKernelGetThreadTLSAddr/var/atlassian/applicati
on-data/bamboo/xml-data/build-dir/PT-WM-BM/buildscripts/build/vita-toolchain_hos
t-prefix/src/vita-toolchain_host/src/vita-elf.c: Unable to find module with NID
2241471665 for function symbol sceKernelUnlockMutex/var/atlassian/application-da
ta/bamboo/xml-data/build-dir/PT-WM-BM/buildscripts/build/vita-toolchain_host-pre
fix/src/vita-toolchain_host/src/vita-elf.c: Unable to find module with NID 93942
2298 for function symbol sceKernelAllocMemBlock/var/atlassian/application-data/b
amboo/xml-data/build-dir/PT-WM-BM/buildscripts/build/vita-toolchain_host-prefix/
src/vita-toolchain_host/src/vita-elf.c: Unable to find module with NID 939422298
 for function symbol sceKernelFreeMemBlock/var/atlassian/application-data/bamboo
/xml-data/build-dir/PT-WM-BM/buildscripts/build/vita-toolchain_host-prefix/src/v
ita-toolchain_host/src/vita-elf.c: Unable to find module with NID 939422298 for
function symbol sceKernelGetMemBlockBase/var/atlassian/application-data/bamboo/x
ml-data/build-dir/PT-WM-BM/buildscripts/build/vita-toolchain_host-prefix/src/vit
a-toolchain_host/src/vita-elf.c: Unable to find module with NID 939422298 for fu
nction symbol sceKernelGetModel/var/atlassian/application-data/bamboo/xml-data/b
uild-dir/PT-WM-BM/buildscripts/build/vita-toolchain_host-prefix/src/vita-toolcha
in_host/src/vita-elf.c: Unable to find module with NID 769202194 for function sy
mbol sceKernelLibcGettimeofday/var/atlassian/application-data/bamboo/xml-data/bu
ild-dir/PT-WM-BM/buildscripts/build/vita-toolchain_host-prefix/src/vita-toolchai
n_host/src/vita-elf.c: Unable to find module with NID 769202194 for function sym
bol sceKernelPowerTick/var/atlassian/application-data/bamboo/xml-data/build-dir/
PT-WM-BM/buildscripts/build/vita-toolchain_host-prefix/src/vita-toolchain_host/s
rc/vita-elf.c: Unable to find module with NID 4076808046 for function symbol sce
IoLseek32/var/atlassian/application-data/bamboo/xml-data/build-dir/PT-WM-BM/buil
dscripts/build/vita-toolchain_host-prefix/src/vita-toolchain_host/src/vita-elf.c
: Unable to find module with NID 4076808046 for function symbol sceIoClose/var/a
tlassian/application-data/bamboo/xml-data/build-dir/PT-WM-BM/buildscripts/build/
vita-toolchain_host-prefix/src/vita-toolchain_host/src/vita-elf.c: Unable to fin
d module with NID 4076808046 for function symbol sceIoRead/var/atlassian/applica
tion-data/bamboo/xml-data/build-dir/PT-WM-BM/buildscripts/build/vita-toolchain_h
ost-prefix/src/vita-toolchain_host/src/vita-elf.c: Unable to find module with NI
D 4076808046 for function symbol sceIoDclose/var/atlassian/application-data/bamb
oo/xml-data/build-dir/PT-WM-BM/buildscripts/build/vita-toolchain_host-prefix/src
/vita-toolchain_host/src/vita-elf.c: Unable to find module with NID 4076808046 f
or function symbol sceIoWrite/var/atlassian/application-data/bamboo/xml-data/bui
ld-dir/PT-WM-BM/buildscripts/build/vita-toolchain_host-prefix/src/vita-toolchain
_host/src/vita-elf.c: Unable to find module with NID 3941406230 for function sym
bol sceKernelGetSystemSwVersionmake: *** [VitaShell.velf] Error 1

C:\vitasdk\VitaShell>

Vita-libs-gen seems to fail on windows.

So I tried re-generating the stubs by using the command on vita-libs-gen db.yml lib/
I grabbed the db.yml from vita-headers, and as soon as I press enter on CMD, it fails and windows reports "vita-libs-gen.exe has stopped working".

Large input .bss section results in a huge output .velf

Basically it seems like vita-toolchain allocates memory for the .bss section.

Relocations against a section not mapped to any segment results in an invalid converted relocation

Some debug sections, for example, .rel.debug_ranges, might exist while not being mapped to any segment. When we convert the relocation and use vita_elf_vaddr_to_segndx it will return -1. After that disastrous things happen, including passing the resulting value to vita_elf_vaddr_to_segoffset which will reference invalid memory. I think the proper behavior would be to ignore such relocations. #4 is also somewhat related.

Prebuilt toolchain repo down?

https://bintray.com/package/files/vitasdk/vitasdk/toolchain?order=desc&sort=fileLastModified&basePath=&tab=files

is currently returning a forbidden page on request

Require exact version of libelf, 0.8.13

Other libelfs have different versioning and are incompatible, if built with them, resulting vita-elf-create does not work at all. This issue gets reported about once per week, so we should force exact libelf version to stop people from building with their distro libelf.

Add travis ci

We should add travis to this repo. Ideally we add it to vita-header, newlib, etc. And we can use a webhook so success => autobuild trigger.

Parse new JSON db format

Coinciding with vitasdk/vita-headers#83

vita-elf-create crashes on latest nightly (Win x64)

I was using an old nightly version of vitasdk (the latest one using gcc 4.9) so i decided to update my environment to latest nightly version available on bintray.

After updating the toolchain i am consantly experiencing crashes with vita-elf-create (i tried to compile some samples, TrackPlug and VitaPad and got the same crashes with all of them).

Some screenshots related to this:
http://rinnegatamante.it/crash.png
http://rinnegatamante.it/crash2.png

CMake template

Not sure if this is the right project to put this, but since we decide that cmake is the way to go (do we?), we should have a toolchain file that does a lot of the work.

Regarding Windows, we should also decide what the best way of cmake support there is. Bundling it? Asking people to download it?

build.sh fails to get zlib

Produced when #99 was merged.

The page of zlib died.
http://zlib.net/zlib-1.2.8.tar.xz

This looks like a fix for buildscripts.
vitasdk/buildscripts@1085bca

As I noted in #98, I suggest to use buildscripts directly instead of having build.sh anyway.

Module NID not being written in module_info

Skip library NID matching in vita-elf-create

Right now, if the library NID mismatches, you get

vita-elf-create: Unable to find library with NID 910080232 for function symbol taiHookFunctionExportForKernel

But the kernel doesn't care about library NIDs, so we should ignore it too.

Memory leak of duplicated sections.

Using 8ebda9a with a few mods in main

$ vita-elf-create pkg.elf pkg.velf

=================================================================
==6436==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 212 byte(s) in 1 object(s) allocated from:
    #0 0x7f79d582fe60 in __interceptor_malloc /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cc:62
    #1 0x4247c1 in elf_utils_duplicate_scn_contents /home/root3/PS Vita/vita-toolchain/src/elf-utils.c:98
    #2 0x424b67 in elf_utils_duplicate_shstrtab /home/root3/PS Vita/vita-toolchain/src/elf-utils.c:131
    #3 0x407326 in main /home/root3/PS Vita/vita-toolchain/src/vita-elf-create.c:298
    #4 0x7f79d42ae290 in __libc_start_main (/usr/lib/libc.so.6+0x20290)

SUMMARY: AddressSanitizer: 212 byte(s) leaked in 1 allocation(s).

Heap buffer overflow while relocating

==1471==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000efd8 at pc 0x000000409322 bp 0x7fffd81455b0 sp 0x7fffd81455a0
READ of size 4 at 0x60200000efd8 thread T0
    #0 0x409321 in load_rel_table /home/root3/PS Vita/vita-toolchain/src/vita-elf.c:250
    #1 0x40b6d7 in vita_elf_load /home/root3/PS Vita/vita-toolchain/src/vita-elf.c:441
    #2 0x405d91 in main /home/root3/PS Vita/vita-toolchain/src/vita-elf-create.c:235
    #3 0x7fcf9b688290 in __libc_start_main (/usr/lib/libc.so.6+0x20290)
    #4 0x4035a9 in _start (/home/root3/vitasdk/bin/vita-elf-create+0x4035a9)

0x60200000efd8 is located 0 bytes to the right of 8-byte region [0x60200000efd0,0x60200000efd8)
allocated by thread T0 here:
    #0 0x7fcf9cc09e60 in __interceptor_malloc /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cc:62
    #1 0x7fcf9c728062  (/usr/lib/libelf.so.1+0xa062)

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/root3/PS Vita/vita-toolchain/src/vita-elf.c:250 in load_rel_table
Shadow bytes around the buggy address:
  0x0c047fff9da0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9db0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9dc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9dd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9de0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c047fff9df0: fa fa fa fa fa fa 00 00 fa fa 00[fa]fa fa 00 fa
  0x0c047fff9e00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9e10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9e20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9e30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9e40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==1471==ABORTING

The cause is the following line (#43 applied):

        memcpy(&insn, text_data->d_buf+(rel.r_offset - text_shdr.sh_addr), sizeof(insn));

The cause is relocation at 0x00010c94, which is out-of-bound of the section.

$ readelf -Sr
There are 28 section headers, starting at offset 0x1a1fc:

Section Headers:
(snip)
  [ 9] .ARM.exidx        ARM_EXIDX       00010c8c 010c8c 000008 00  AL  2   0  4
  [10] .rel.ARM.exidx    REL             00000000 01c7e4 000010 08     26   9  4
(snip)
Relocation section '.rel.ARM.exidx' at offset 0x1c7e4 contains 2 entries:
 Offset     Info    Type            Sym.Value  Sym. Name
00010c8c  0000022a R_ARM_PREL31      00008040   .text
00010c94  0000022a R_ARM_PREL31      00008040   .text
(snip)

The executable and ELF are available in #44.

vita-libs-gen crash

Using it on any yml file gives me: warning: line: zd, column: zd, unknow tag '(null)' and then crashes

Default .json files

Currently when executing vita-elf-create you have to pass it a list of JSON files so it can read names of modules and libraries and insert them into the final .velf. This leads to people using constructions like vita-elf-create input.elf output.velf $(VITASDK)/somewhere/db.json or just shipping their own copy of db.json with homebrew sources. Instead we should ship both db.json and extra.json with the SDK and always include both files even when no input .json is specified.

I suggest we put both db.json and extra.json to $TOOLCHAIN/share. We also add a compile time option to vita-toolchain, e.g. -DDEFAULT_JSON, which specifies a binary-relative path to the .json file.
For example, if we have the following structure:

./bin
...
./bin/vita-elf-create.exe
./bin/vita-libs-gen.exe
...
./share
./share/db.json
./share/extra.json

we'll pass something like -DDEFAULT_JSON=../share/db.json:../share/extra.json

vita-elf-create SIGSEGV

$ coredumpctl info 3927
           PID: 3927 (vita-elf-create)
           UID: 1000 (root3)
           GID: 1000 (root3)
        Signal: 11 (SEGV)
     Timestamp: 金 2016-09-09 15:10:33 JST (8min ago)
  Command Line: vita-elf-create pkg.elf pkg.velf
    Executable: /home/root3/vitasdk/bin/vita-elf-create
 Control Group: /user.slice/user-1000.slice/[email protected]/gnome-terminal-server.service
          Unit: [email protected]
     User Unit: [email protected]
         Slice: user-1000.slice
     Owner UID: 1000 (root3)
       Boot ID: 528875cd6e6c44b89d841510fa28478a
    Machine ID: 35f8f0dc980741838edbac02de1cec00
      Hostname: root3-X220t
      Coredump: /var/lib/systemd/coredump/core.vita-elf-create.1000.528875cd6e6c44b89d841510fa28478a.3927.1473401433000000000000.lz4
       Message: Process 3927 (vita-elf-create) of user 1000 dumped core.

                Stack trace of thread 3927:
                #0  0x000003b1a8c20a8c _dl_fini (ld-linux-x86-64.so.2)
                #1  0x000003b1a8483990 __run_exit_handlers (libc.so.6)
                #2  0x000003b1a84839ea exit (libc.so.6)
                #3  0x000003b1a846e298 __libc_start_main (libc.so.6)
                #4  0x0000000000402a7a _start (vita-elf-create)

Used vita-elf-create with fix #43.

Attached core, executable, and the ELF.
crash.zip

The source code of the ELF is available here.
173210/pkg: PKG Decryptor for PS Vita

Parameter has the same name with the global variable

Hi all,
There is a parameter which has the same name with a global variable found by Qihoo360 CodeSafe Team.
Details as bellow:

vita-toolchain/src/vita-pack-vpk.c

Line 81 in cc9473d

static void parse_add_subopt(char *optarg)

the name of the parameter is 'optarg'
in file 'getopt_core.h' line 36, the same named global variable 'optarg' declared here.

Cheers
Qihoo360 CodeSafe Team

Argument list too long

I get with vita-libs-gen
make: execvp: /usr/local/vitasdk/bin/arm-vita-eabi-ar: Argument list too long
make: *** [Makefile:417: libSceLibc_stub.a] Error 127

Package for ArchLinux

I wrote a quick AUR package for arch and had a few question:

• What is the current install path you have in mind ? I would suggest something like /usr/vita-toolchain, because that's how clean SDKs do their install under Linux, with a post-install script asking the user (or doing it) to update the $PATH with /usr/vita-toolchain/bin to access host-tools.
• A got a few compiler warnings related to libelf, I figured i'd just notify you, but you plan on shipping libelf with the repo in the future right ?

[ 84%] Building C object src/CMakeFiles/vita-elf-create.dir/varray.c.o
In file included from /tmp/makepkg/vita-toolchain-git/src/vita-toolchain/src/elf-defs.c:2:0:
/tmp/makepkg/vita-toolchain-git/src/vita-toolchain/src/elf-defs.h:25:0: warning: "SHT_ARM_EXIDX" redefined
 #define SHT_ARM_EXIDX 0x70000001
 ^
In file included from /usr/include/libelf.h:35:0,
                 from /tmp/makepkg/vita-toolchain-git/src/vita-toolchain/src/elf-defs.c:1:
/usr/include/elf.h:2490:0: note: this is the location of the previous definition
 #define SHT_ARM_EXIDX  (SHT_LOPROC + 1) /* ARM unwind section.  */
 ^
In file included from /tmp/makepkg/vita-toolchain-git/src/vita-toolchain/src/vita-elf-create.c:11:0:
/tmp/makepkg/vita-toolchain-git/src/vita-toolchain/src/elf-defs.h:25:0: warning: "SHT_ARM_EXIDX" redefined
 #define SHT_ARM_EXIDX 0x70000001
 ^
In file included from /usr/include/libelf.h:35:0,
                 from /tmp/makepkg/vita-toolchain-git/src/vita-toolchain/src/vita-elf-create.c:6:
/usr/include/elf.h:2490:0: note: this is the location of the previous definition
 #define SHT_ARM_EXIDX  (SHT_LOPROC + 1) /* ARM unwind section.  */
 ^
In file included from /tmp/makepkg/vita-toolchain-git/src/vita-toolchain/src/sce-elf.c:9:0:
/tmp/makepkg/vita-toolchain-git/src/vita-toolchain/src/elf-defs.h:25:0: warning: "SHT_ARM_EXIDX" redefined
 #define SHT_ARM_EXIDX 0x70000001
 ^
In file included from /usr/include/libelf.h:35:0,
                 from /tmp/makepkg/vita-toolchain-git/src/vita-toolchain/src/sce-elf.c:5:
/usr/include/elf.h:2490:0: note: this is the location of the previous definition
 #define SHT_ARM_EXIDX  (SHT_LOPROC + 1) /* ARM unwind section.  */
 ^
In file included from /tmp/makepkg/vita-toolchain-git/src/vita-toolchain/src/vita-elf.c:28:0:
/tmp/makepkg/vita-toolchain-git/src/vita-toolchain/src/elf-defs.h:25:0: warning: "SHT_ARM_EXIDX" redefined
 #define SHT_ARM_EXIDX 0x70000001
 ^
In file included from /usr/include/libelf.h:35:0,
                 from /tmp/makepkg/vita-toolchain-git/src/vita-toolchain/src/vita-elf.c:4:
/usr/include/elf.h:2490:0: note: this is the location of the previous definition
 #define SHT_ARM_EXIDX  (SHT_LOPROC + 1) /* ARM unwind section.  */
 ^
[ 92%] Linking C executable vita-libs-gen
[ 92%] Built target vita-libs-gen
[100%] Linking C executable vita-elf-create
[100%] Built target vita-elf-create

Any suggestions for the package ?

Add FOLDER option to vita_create_vpk()

It would be more convenient than manually adding each file.

Several issues with latest toolchain on Windows

I'm getting several different errors when trying to compile libraries/homebrews that used to compile fine with vitasdk using gcc 4.9.

Here are some of the errors i got:

vitaQuake: http://pastebin.com/Kq1KB1Dn
libpng (using vita-portlibs): http://pastebin.com/5X4e9Zdm

How can i solve them?

sceIoChdir linking issue

When i try to use sceIoChdir i get an error during linking phase. It seems to be the only broken func of io/stat.h funcs.

luaSystem.cpp:(.text+0x5e): undefined reference to `sceIoChdir'

Simplify relocations?

So up until now, I just went with the assumption that MOV/MOVT pairs needs to be decoded together. I woke up this morning and realized: wait. I actually implemented relocations in UVL from linux's code and never had to deal with it. So I looked at it again:
[http://lxr.free-electrons.com/source/arch/arm/kernel/module.c]

Yup, they don't do anything special. They use the addend in the instruction. That makes sense. Consider relocation of 0x8100607C into

MOV R0, #0x507C
MOVT R0, #0

Let's say they point to the same symbol and the symbol value + base is 0x81001000. Then the first relocation would become (0x81001000 + 0x507C) & 0xFFFF and the second would become (0x81001000 + 0x0) >> 16. Both would be correct.

Of course, this assumes that for these instructions, the relocation won't go beyond the range of the constant. But linux assumes this too, so I'm willing to bet that toolchains does not generate anything that breaks this assumption. Otherwise, we found a bug in Linux.

Fix travis CI IRC notifications

The IRC notifications should be exclusive to this repository, #vitasdk does not care about the status of forks using the code.

Change naming from library <=> module

As per specification 1.2 changes, references to the term "module" and the term "library" should be reversed.

Module refers to the entire SELF/suprx/skprx. Library refers to a collection of exports. Spec 1.0 had these terms reversed.

• Make changes in config (exports) file
• Make changes in import db file
• Make changes in console outputs
• Make changes in source code (function names, variable names, etc)

Some commands add a target that is always outdated

When one of the affected macros is used, it creates a target that is always outdated. It causes it to be rebuilt every time make (or ninja, etc.) is run. In turn, this increases incremental build times.

List of affected macros:

• vita_create_stubs
• vita_create_vpk
• vita_create_self

---

Let's take vita_create_stubs for example.
It does the following:

  add_custom_target(${target-dir} ALL
    COMMAND ${VITA_LIBS_GEN} ${VITA_LIBS_GEN_FLAGS} ${target-dir}.yml ${target-dir}
    COMMAND make -C ${target-dir}
    DEPENDS ${target-dir}.yml
    COMMENT "Building stubs ${target-dir}"

According to CMake's documentation,

The target has no output file and is ALWAYS CONSIDERED OUT OF DATE even if the commands try to create a file with the name of the target.

This is the reason it is rebuilt every time a build is triggered. add_custom_command should be used instead. The issue is that the documentation indicates that it requires an OUTPUT, and in this case the name of this output depends on the name specified in {target-dir}.yml, so it should be fetched somehow on generation time and configure_file() be used so that the generation is automatically retriggered when {target-dir}.yml is modified.

Output less debug information

Right now vita-elf-create will output a lot of debug information (all relocations, info about modules and modinfo). It should only output this info if it's explicitly requested, e.g. with -v.

Suggestion to add min version check to FindJansson.cmake

vita-elf-create not working properly if an imcompatible libelf is installed in system

I tried to compile vita-toolchain under ArchLinux and the samples are not compiled properly, the error shows:

vita-elf-create: dest = elf_begin(fileno(file), ELF_C_WRITE, NULL) failed: Request error: invalid ELF_C_ argument
vita-elf-create: Assertion failed: (dest = elf_utils_copy_to_file(argv[2], ve->elf, &outfile))

After investigating libelf calls, I found that vita-elf-create linked to /usr/lib/libelf.a(version 0.166) by default, which is not working properly as 0.8.13. since libelf is the dependency of systemd in ArchLinux so that I cannot uninstall it.
I found a simple way to resolve this problem: in cmake/Modules/Findlibelf.cmake, change 'libelf>=0.8' to 'libelf=0.8.13', so that libelf other than 0.8.13(just like the one in ArchLinux) won't be accepted.