defold / extender Goto Github PK

We want to hook up our metrics to a server again.

Multi-arch frameworks (ie xcframework) can contain both platform specific frameworks (ie .framework folders) and static libraries (.a files). We are currently expecting all xcframeworks to only include .frameworks.

In order to make it easier to compare builds between different servers, the command lines should be same

To make it easier to use 3rd party libraries its good to be able to specify a custom include path.

Some frameworks (resolved by Cocoapods) contains resources that should be copied into the final application. Make sure we detect these when resolving cocoapod dependencies and add them to the build output that we send back to the client.

The extender tests are failing in buildbot.

The tests are sending requests to an extender service inside a docker container. The service downloads Defold SDKs as part of the tests, and these files are written to the host's file system, namely in the build directory. When buildbot tries to clear the build directory, it's not allowed to delete the SDK directories put there by a docker user.

I had to clear the build directory manually:

$ awssh ci-slave-linux-64-2

ubuntu@ip-10-0-1-191:~$ sudo ls -al /home/builder/ci_slave/builds/extender/build/server/test-data/sdk/
total 44
drwxrwxrwx 11 builder builder 4096 Apr  4 13:38 .
drwxrwxrwx  3 builder builder 4096 Apr  4 13:38 ..
drwxr-xr-x  3    2222 docker  4096 Nov  7 10:47 0d7f8b51658bee90cb38f3d651b3ba072394afed
drwxr-xr-x  3    2222 docker  4096 Nov  7 10:47 1afccdb2cd42ca3bc7612a0496dfa6d434a8ebf9
drwxr-xr-x  3    2222 docker  4096 Nov  7 10:47 1e53d81a6306962b64381195f081d442d033ead1
drwxr-xr-x  3    2222 docker  4096 Jan 15 16:19 2406775912d235d2579cfe723ab4dbcea2ca77ca
drwxr-xr-x  3    2222 docker  4096 Nov  7 10:46 735ff76c8b1f93b3126ff223cd234d7ceb5b886d
drwxr-xr-x  3    2222 docker  4096 Nov  7 10:46 8e1d5f8a8a0e1734c9e873ec72b56bea53f25d87
drwxr-xr-x  3    2222 docker  4096 Nov  7 10:48 d126b0348d27c684d020e0bd43fde0a2771746f0
drwxr-xr-x  3    2222 docker  4096 Nov  7 10:48 d530758af74c2800d0898c591cc7188cc4515476
drwxr-xr-x  3    2222 docker  4096 Mar  9 17:55 fce7921da858a71876773c75920b74310ca7ac1f

ubuntu@ip-10-0-1-191:~$ sudo rm -rf /home/builder/ci_slave/builds/extender/build/server/test-data/sdk

Solution

The tests should cleanup the file system after themselves.

After a change on the server 610ee50, the linking towards FBAudienceNetwork stopped working, producing "unresolved symbol".

Currently using version 26.0.0. We need to upgrade as some newer dependencies use the element which isn't handled by older versions of the tool:

https://android-developers.googleblog.com/2020/07/preparing-your-build-for-package-visibility-in-android-11.html

https://forum.defold.com/t/ironsource-sdk-native-extension/64551/13

Is your feature request related to a problem? Please describe (REQUIRED):
I keep making the mistake of forgetting to change the build server URL between staging and production. The problem is that extender doesn't point out this mistake to me and instead tries to compile anyway and fails on some random missing files, resulting in an unhelpful error and a lot of headscratching.

Describe the solution you'd like (REQUIRED):
Extender should error out early if trying to compile with a version of bob for which it doesn't have a corresponding engine build on file.

Describe alternatives you've considered (REQUIRED):
Paying more attention, but that's not always achievable 😄. I usually remember the fix if I have the same problem twice, but this time I seem to have been fooled 3 times (that i can remember) already, since the errors tend to be different each time.

Additional context (OPTIONAL):
One instance in which I was fooled: defold/defold#4903

We're a bit behind the latest release

We can change the launch argument to the server to increase the heap size, but we'd like to investigate if we can stream the zip file to disc before unzipping. This should keep the memory usage small and also make for a more robust concurrency situation.

Build a huge extension: https://forum.defold.com/t/native-extensions/4946/178
with a heap size of 500mb

Callstack:

java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236) ~[na:1.8.0_161]
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[na:1.8.0_161]
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[na:1.8.0_161]
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:135) ~[na:1.8.0_161]
at org.eclipse.jetty.util.MultiPartInputStreamParser$MultiPart.write(MultiPartInputStreamParser.java:135) ~[jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.util.MultiPartInputStreamParser.parse(MultiPartInputStreamParser.java:728) ~[jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.util.MultiPartInputStreamParser.getParts(MultiPartInputStreamParser.java:422) ~[jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Request.getParts(Request.java:2317) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Request.extractMultipartParameters(Request.java:519) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Request.extractContentParameters(Request.java:441) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Request.getParameters(Request.java:365) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Request.getParameter(Request.java:996) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:70) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) ~[jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) ~[jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107) ~[spring-boot-actuator-1.4.2.RELEASE.jar!/:1.4.2.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) ~[jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) ~[jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[jetty-security-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) ~[jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Server.handle(Server.java:534) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]

Try setting the parameters low to reproduce this error:
spring.http.multipart.max-file-size: 1mb
spring.http.multipart.max-request-size: 1mb

Then build a native extension (e.g. this huge one: https://forum.defold.com/t/native-extensions/4946/178)

The callstack (before entering the ExtenderController):
java.lang.IllegalStateException: Request exceeds maxRequestSize (209715200)
at org.eclipse.jetty.util.MultiPartInputStreamParser.parse(MultiPartInputStreamParser.java:689) ~[jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.util.MultiPartInputStreamParser.getParts(MultiPartInputStreamParser.java:422) ~[jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Request.getParts(Request.java:2317) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Request.extractMultipartParameters(Request.java:519) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Request.extractContentParameters(Request.java:441) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Request.getParameters(Request.java:365) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Request.getParameter(Request.java:996) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:70) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) ~[jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) ~[jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107) ~[spring-boot-actuator-1.4.2.RELEASE.jar!/:1.4.2.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) ~[jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) [jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Server.handle(Server.java:534) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [jetty-io-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [jetty-io-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [jetty-io-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]

Sometimes the build data can trigger an Internal Server Error. In such a scenario no information except the response code and error title is sent back to the client. It would help greatly with extension development if we could send some additional information from the extender log back to the client.

Successful builds result in a .internal/cache/%platform%/build.zip. This archive contains a log.txt with the full extender log.

Failed builds do not result in a build.zip (and thus no log).

Thought: We recently added support for asynchronous extender builds using our command line tools (PR). Brief explanation: A build request is sent from the client to start a build. The response returns a job id. The job id is used to poll for build status and then later to retrieve the build result (ie the build.zip). We could return all log output from when polling and getting the final status. This output can be printed to the terminal (and then later to the editor console when async builds are supported also from the editor).

It is not possible to build a project on iOS with this extension if the project uses Admob or Ironsource or AppsFlyer extension.

Here is the log file when there is a conflict with Ironsource:
log.txt

Defold version: 1.4.7 and 1.4.8

Describe the problem
Some of ios framework links with target application in "dynamic" manner. It means that application should have "Frameworks/SomeLibrary.framework" inside application bundle.

Example

1. Use podfile like

platform :ios, '11.0'

pod 'HelpshiftX', '10.2.3'

2. Add to ext.manifest following lines

platforms:
    arm64-ios:
        context:
            linkFlags:  ['-Wl,-rpath,@executable_path/Frameworks']

3. Build and run application. Get following error during luanch

Exception Type:  EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note:  EXC_CORPSE_NOTIFY
Termination Description: DYLD, dyld: Using shared cache: 9816A151-4D39-3378-B552-7BCA27A9DCB5 | dependent dylib '@rpath/HelpshiftX.framework/HelpshiftX' not found for '/private/var/containers/Bundle/Application/413FDAD0-9FA7-4D03-A419-2B9A9A5A2C19/sample-app.app/defoldhelpshift', tried but didn't find: '/private/var/containers/Bundle/Application/413FDAD0-9FA7-4D03-A419-2B9A9A5A2C19/sample-app.app/Frameworks/HelpshiftX.framework/HelpshiftX' '@rpath/HelpshiftX.framework/HelpshiftX' '/System/Library/Frameworks/HelpshiftX.framework/HelpshiftX'
Highlighted by Thread:  0

From extender's cloudwatch logs:

2018-03-05 14:36:46.076 INFO 17 --- [qtp631659383-17] c.d.extender.services.DefoldSdkService : Downloading Defold SDK version d58cffb10ce9caf41aa38ebb1f90d2a520af46cc ...
2018-03-05 14:36:46.312 INFO 17 --- [qtp631659383-17] c.d.extender.services.DefoldSdkService : Using Defold SDK version d58cffb10ce9caf41aa38ebb1f90d2a520af46cc
2018-03-05 14:36:46.482 INFO 17 --- [qtp631659383-17] com.defold.extender.ExtenderController : top -b -n 1 -o %MEM
top - 14:36:46 up 5 days, 20:14, 0 users, load average: 0.17, 0.18, 0.10
Tasks: 12 total, 1 running, 11 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.0 us, 0.8 sy, 0.0 ni, 98.0 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16434008 total, 10409636 free, 1837072 used, 4187300 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 14232176 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17 extender 20 0 4012732 509520 17704 S 0.0 3.1 16:30.35 java
1 root 20 0 33700 11932 5660 S 0.0 0.1 0:00.84 my_init
15 root 20 0 72504 7716 5920 S 0.0 0.0 0:00.18 syslog-ng
10533 extender 20 0 39732 3548 3164 R 0.0 0.0 0:00.00 top
18 root 20 0 29268 2764 2524 S 0.0 0.0 0:00.31 cron
9 root 20 0 4388 1140 1056 S 0.0 0.0 0:02.43 runsvdir
10 root 20 0 4236 708 640 S 0.0 0.0 0:00.01 runsv
25175 root 20 0 7612 696 628 S 0.0 0.0 0:00.00 tail
11 root 20 0 4236 688 620 S 0.0 0.0 0:00.01 runsv
12 root 20 0 4236 656 588 S 0.0 0.0 0:00.01 runsv
14 root 20 0 4236 648 580 S 0.0 0.0 0:00.00 runsv
13 root 20 0 4236 632 564 S 0.0 0.0 0:00.01 runsv

2018-03-05 14:36:46.484 ERROR 17 --- [qtp631659383-17] com.defold.extender.ExtenderController : Internal Server Error

java.io.FileNotFoundException: File '/var/extender/sdk/d58cffb10ce9caf41aa38ebb1f90d2a520af46cc/defoldsdk/extender/build.yml' does not exist
at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:299) ~[commons-io-2.4.jar!/:2.4]
at org.apache.commons.io.FileUtils.readFileToString(FileUtils.java:1711) ~[commons-io-2.4.jar!/:2.4]
at org.apache.commons.io.FileUtils.readFileToString(FileUtils.java:1748) ~[commons-io-2.4.jar!/:2.4]
at com.defold.extender.Extender.loadYaml(Extender.java:181) ~[classes!/:na]
at com.defold.extender.Extender.(Extender.java:56) ~[classes!/:na]
at com.defold.extender.ExtenderController.buildEngine(ExtenderController.java:111) ~[classes!/:na]

AAR files may contain an assets folder but this is currently ignored by extender after resolving dependencies. The asets should be sent back to the client for inclusion in the apk/aab.

Also reported here: defold/defold#5710

The Cocoapod spec states that a resource can be either a .bundle "file" or individual files. When specifying as individual files the format also supports wildcards. We should use ExtenderUtil.listFilesGlob() when we copy these resources to the bundle output

Sub specs should inherit attributes from their parent.

https://guides.cocoapods.org/syntax/podspec.html#subspec

Subset of ios frameworks contains swift code (like Facebook, Sentry). Need to add support for compiling swift sources.

The max build job size is parsed to an int which is causing problems on really large build jobs:

Injected here: https://github.com/defold/extender/blob/dev/server/src/main/java/com/defold/extender/ExtenderController.java#L92

Parsed here: https://github.com/defold/extender/blob/dev/server/src/main/java/com/defold/extender/ExtenderController.java#L65

For configurations of spring.http.multipart.max-request-size larger than 2048mb we hit the max integer size. We should convert to a long instead and we should use getContentLengthLong() from HttpServletRequest (Java EE 7):

https://docs.oracle.com/javaee/7/api/javax/servlet/ServletRequest.html#getContentLengthLong--

Presumably, the user config (ext.manifest) is somehow broken, and will cause an unhandled exception.

2018-04-01 11:57:51.357 INFO 16 --- [tp556529265-490] com.defold.extender.Extender : Building engine for platform armv7-android with extension source /tmp/job5295972638235995063/upload
2018-04-01 11:57:51.366 ERROR 16 --- [tp556529265-490] com.defold.extender.ExtenderController : Internal Server Error

java.lang.NullPointerException: null
at com.defold.extender.Extender.isListOfStrings(Extender.java:205) ~[classes!/:na]
at com.defold.extender.Extender.mergeContexts(Extender.java:221) ~[classes!/:na]
at com.defold.extender.Extender.buildEngine(Extender.java:713) ~[classes!/:na]
at com.defold.extender.Extender.build(Extender.java:773) ~[classes!/:na]
at com.defold.extender.ExtenderController.buildEngine(ExtenderController.java:114) ~[classes!/:na]
at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_161]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:220) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:134) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116) ~[spring-webmvc-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) ~[spring-webmvc-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) ~[spring-webmvc-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) ~[spring-webmvc-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) ~[spring-webmvc-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) ~[spring-webmvc-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) ~[spring-webmvc-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [javax.servlet-api-3.1.0.jar!/:3.1.0]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[spring-webmvc-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar!/:3.1.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) [jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1712) [jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:247) [websocket-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) [jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) [spring-boot-1.4.2.RELEASE.jar!/:1.4.2.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) [jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105) [spring-boot-actuator-1.4.2.RELEASE.jar!/:1.4.2.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) [jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) [jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89) [spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) [jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) [jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) [jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107) [spring-boot-actuator-1.4.2.RELEASE.jar!/:1.4.2.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) [jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) [jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [jetty-servlet-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.Server.handle(Server.java:534) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [jetty-io-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [jetty-io-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [jetty-io-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]

When the Extender build is aborted by the client an EofException is thrown by the server and returned as a 500 Internal Server Error. This should instead be logged as a client error.

Stacktrace from cloudwatch logs:

org.eclipse.jetty.io.EofException: null
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:199) ~[jetty-io-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:420) ~[jetty-io-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:313) ~[jetty-io-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:140) ~[jetty-io-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:738) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241) ~[jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) ~[jetty-util-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:518) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:724) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:775) [jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:235) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:219) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:470) ~[jetty-server-9.3.14.v20161028.jar!/:9.3.14.v20161028]
at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:253) ~[na:1.8.0_161]
at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:211) ~[na:1.8.0_161]
at java.util.zip.ZipOutputStream.write(ZipOutputStream.java:331) ~[na:1.8.0_161]
at java.nio.file.Files.copy(Files.java:2909) ~[na:1.8.0_161]
at java.nio.file.Files.copy(Files.java:3069) ~[na:1.8.0_161]
at com.defold.extender.ZipUtils.zip(ZipUtils.java:44) ~[classes!/:na]
at com.defold.extender.ExtenderController.buildEngine(ExtenderController.java:117) ~[classes!/:na]

First, an investigation of what is needed.
We do have some older dependencies that we still need (e.g.Python 2.7) for our builds.

It may not be easy to do an incremental update, as we are making quite a leap.

We need collect the required tasks here...

It would be nice to have a way to exclude specific platforms' setups in the Dockerfile via environment variables so that users don't need to mess with the build process if they don't have SDKs for every platform. I propose adding EXCLUDE_<PLATFORM> build arguments for each supported platform and checking for its existence in every RUN command that requires that SDK. It would look something like:

# build.sh
...
if [ "${EXCLUDE_WINDOWS}" != "" ]; then
	ENV="${ENV} --build-arg EXCLUDE_WINDOWS"
fi
...

# docker-base/Dockerfile
...

#
# Windows
#

ARG EXCLUDE_WINDOWS

...

RUN if [ -z "$EXCLUDE_WINDOWS" ] ; then \
  echo "WIN32 2019 SDK" && \
  mkdir -p ${PLATFORMSDK_WIN32}/MicrosoftVisualStudio2019 && \
  wget -q -O - ${DM_PACKAGES_URL}/Microsoft-Visual-Studio-2019-${WINDOWS_MSVC_2019_VERSION}.tar.gz | tar xz -C ${PLATFORMSDK_WIN32}/MicrosoftVisualStudio2019 \
  ; fi

RUN if [ -z "$EXCLUDE_WINDOWS" ] ; then \
  echo "WIN32 10 SDK" && \
  mkdir -p ${PLATFORMSDK_WIN32}/WindowsKits && \
  wget -q -O - ${DM_PACKAGES_URL}/WindowsKits-${WINDOWS_SDK_10_VERSION}.tar.gz | tar xz -C ${PLATFORMSDK_WIN32}/WindowsKits \
  ; fi

# You get the idea
...

With that setup, the user doesn't need to provide Windows SDKs if they build with:

EXCLUDE_WINDOWS=1 ./server/scripts/build.sh

I did something very similar in my Rust fork, but that used a whitelist instead of blacklist which would break existing workflows.

Occasionally we get this error, regarding the "thin pool".
It turns out AWS/Docker doesn't reclaim unused space automatically.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/CannotCreateContainerError.html

sudo sh -c "docker ps -q | xargs docker inspect --format='{{ .State.Pid }}' | xargs -IZ fstrim /proc/Z/root/"

Instead of having the random job folder as part of the file paths, we can substitute it with either:

• /tmp/job123456 -> /extender/build/
  • Good for caching, and everyone gets the same paths
  • Still has to setup debug mappings locally
• /tmp/job123456 -> /path/to/user/project
  • Good for user
  • Our caching will not work across users

From gcc:

"
-fdebug-prefix-map=old=new

When compiling files residing in directory old, record debugging information describing them as if the files resided in directory new instead. This can be used to replace a build-time path with an install-time path in the debug info. It can also be used to change an absolute path to a relative path by using . for new. This can give more reproducible builds, which are location independent, but may require an extra command to tell GDB where to find the source files. See also -ffile-prefix-map.
"

We currently support Gradle 6.1.1 and Android Gradle plugin 4.0.1

https://github.com/defold/extender/blob/dev/server/docker-base/Dockerfile#L451-L452

Newer minimum versions here: https://developer.android.com/build/releases/gradle-plugin#updating-gradle

To speed up CI builds (which are usually run from a clean environment) where extensions rarely change it will help if extension outputs (library+resources) are cached on the extender server.

E.g. the user Tomires wants to build save dialogues for Linux, but the docker container doesn't currently contain such libraries.

The user tried something like this to work "g++ gtktest.cpp pkg-config --cflags --libs gtk+-3.0" in order to add the correct cflags and link flags. It seems like a good idea to support.

MVP: It should be possible to enable basic authentication for an extender instance.

Bonus: It should be configurable on a per user and per platform level

• user x has access to platform A but not B, C and D
• user y has access to platform B but not A, C and D
• user z has access to platforms A, B, C and D

For some reason jetifier doesn't apply for dependencies specified in gradle file. Example project:
Archive.zip

More info on forum https://forum.defold.com/t/add-helpshift-question/73562/17

I tried to build Sentry pod, but I got compiler's error regarding C sources

clang++ -c -arch arm64 -target arm-apple-darwin19 -m64 -O2 -g -stdlib=libc++ -miphoneos-version-min=11.0 -isysroot <redacted>/platformsdk/iPhoneOS16.2.sdk -nostdinc++ -isystem <redacted>/platformsdk/iPhoneOS16.2.sdk/usr/include/c++/v1  -DDLIB_LOG_DOMAIN="SENTRY" -DDDF_EXPOSE_DESCRIPTORS -DDM_PLATFORM_IOS -DDM_DEBUG  -fno-exceptions -fvisibility=hidden -fobjc-arc -Iupload/ -ICocoaPodsService/Pods/Sentry/  -F/var/folders/nq/zlqy380j2fb1xwwkry71j9jw0000gn/T/job11030230481155200560/CocoaPodsService/frameworks/lib/arm64-ios  -I<redacted>/sdk/b1f3977fbad9230bbf048e67e779d1e3c86f2cf6/defoldsdk//include -I<redacted>/sdk/b1f3977fbad9230bbf048e67e779d1e3c86f2cf6/defoldsdk//sdk/include -I<redacted>/sdk/b1f3977fbad9230bbf048e67e779d1e3c86f2cf6/defoldsdk//ext/include   CocoaPodsService/Pods/Sentry/Sources/SentryCrash/Recording/Tools/SentryCrashStackCursor_MachineContext.c -obuild/SentryCrashStackCursor_MachineContext.c_3.o
clang: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
In file included from CocoaPodsService/Pods/Sentry/Sources/SentryCrash/Recording/Tools/SentryCrashStackCursor_MachineContext.c:28:
CocoaPodsService/Pods/Sentry/Sources/SentryCrash/Recording/Tools/SentryCrashMemory.h:69:26: error: expected ')'
    const void *restrict const src, void *restrict const dst, int byteCount);

As I understood restrict keyword used only in C.
Podfile:

platform :ios, '11.0'

pod 'Sentry', :git => 'https://github.com/getsentry/sentry-cocoa.git', :tag => '7.31.5'

See defold/extension-facebook#55 (comment)

Steps to reproduce:

1. Add Info.plist in extension section

<key>NSPhotoLibraryUsageDescription</key>
<string>Your application needs permission to save the content to the gallery.</string>

2. In main project add Info.plist with section

<key>NSPhotoLibraryUsageDescription</key>
<string>The game needs access to your photo gallery so you can attach screenshots to tech support requests</string>

3. Build project for ios.

In build log got warning:

_WARNING: Plist overriding value for key 'NSPhotoLibraryUsageDescription': from 'The game needs access to your photo gallery so you can attach screenshots to tech support requests' to 'Your application needs permission to save the content to the gallery.'_

In result Info.plist we got section from extension Info.plist, not from main Info.plist.

Expected result:
Result Info.plist should contain section from main Info.plist, not from extension Info.plist.

Technically, for every platform we add separate Podfile. For example,

|-manifests
|--ios
|---Podfile
|--osx
|---Podfile

But for now Podfiles from different platforms are mixed during pod resolving step (look at https://github.com/defold/extender/blob/dev/server/src/main/java/com/defold/extender/services/CocoaPodsService.java#L625).

Steps to reproduce

1. Open firebase-analytics project from dev-podfile branch (https://github.com/defold/extension-firebase-analytics/tree/dev-podfile)
2. Try to build from editor for macos (simply hit Cmd+B)
3. Take a look at error

Actual result
Error

/firebase/ext.manifest
	Specs satisfying the `FirebaseAnalytics (= 8.13.0)` dependency were found, but they required a higher minimum deployment target.

But for osx target there are no Podfile and resolvePods step should be skipped.

Expected behavior:
If Podfile doens't exist for specific platform - resolvePods steps should be skipped.

When trying to use cocoapods of yandex mobile ads, defold fails to properly integrate its frameworks.

ld: warning: Could not find or use auto-linked framework 'YandexMobileMetrica'

Podfile

platform :ios, '12.0'
pod 'YandexMobileAds', '5.2.1'
pod 'YandexMobileAdsInstream', '0.12.0'

log.txt

The ManifestMergeTool.merge() method supports Platform.ANDROID, Platform.IOS, Platform.OSX and Platform.WEB:

https://github.com/defold/extender/blob/dev/server/manifestmergetool/src/main/java/com/defold/manifestmergetool/ManifestMergeTool.java#L127-L134

But when parsing the --platform argument "osx" and Platform.OSX is not checked for:

https://github.com/defold/extender/blob/dev/server/manifestmergetool/src/main/java/com/defold/manifestmergetool/ManifestMergeTool.java#L164-L170

Some builds take more time than they should.

Although the worst pain point is waiting for the unit tests, I feel that if it's easy to do, we should try building in parallel.

• Each extension by itself
• Android - The .java files by itself (probably solved by building each extension by itself)
• Android - The .dex file by itself, parallel to linking the engine

(LINK REMOVED)Today, we compile .c files with clang++, which enables other code checks.

EDIT: Also support .m files

Some of the pod specs contains section 'pod_target_xcconfig' (for example, https://github.com/getsentry/sentry-cocoa/blob/7.31.5/Sentry.podspec#L19).
Need to parse such sections and add flags for compiler because it can be crucial for building pod.
Some useful refs:

• https://developer.apple.com/documentation/xcode/build-settings-reference
• https://pewpewthespells.com/blog/buildsettings.html

Also note that definitions from main spec should be inherited by sub spec.

The ECS services (extender/magazine/scoop/codepad) have stopped working twice because the thin partition runs out of disk space. When this happens, all file systems are re-mounted as read-only. This is fatal for the Extender service since it cannot write the build result to /tmp.

From the Extender cloudwatch logs:
java.nio.file.FileSystemException: /tmp/job7405478965988413682: Read-only file system

Short-term solution:

• Extend the disk space on the logical volume "thin partition" on the ECS instances.

Long-term solution:

• Investigate and find a solution to why the thin partition fills up.

AWS documentation:
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-ami-storage-config.html

From the docs:

Amazon ECS-optimized AMIs from version 2015.09.d and later launch with an 8-GiB volume for the operating system that is attached at /dev/xvda and mounted as the root of the file system. There is an additional 22-GiB volume that is attached at /dev/xvdcz that Docker uses for image and metadata storage. Because the volume is not mounted, you cannot use standard storage information commands (such as df -h) to determine the available storage. However, you can use LVM commands ("sudo lvs") and "docker info" to find the available storage.

$ docker info
...
 Data Space Used: 42.42GB
 Data Space Total: 42.42GB
 Data Space Available: 0B

The docker containers were not even close to using up the total of 40 gb of disk:

• extender: 17 gb
• magazine: 1-2 gb
• scoop: 1-2 gb
• codepad: 1-2 gb

Failed attempt to clear up some disk space:

[ec2-user@ip-10-0-1-175 ~]$ docker system prune
WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all dangling images
        - all build cache
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B

When running run-standalone-local.sh the script tries to load server/scripts/shared/tools.sh. The functions in tools.sh are used when deploying to AWS and are not needed when running a local extender server.

https://github.com/defold/extender/blob/dev/server/scripts/run-standalone-local.sh#L12

Problem:
If mark some AndroidManifest.xml xml nodes as tools:node="remove" in application manifest and these nodes are presented inside libraries or other dependencies, in result apk marked nodes aren't removed.

Expected result
If we mark nodes with tools:node="remove" we don't expect these nodes in result AndroidManifest.xml.

Sample project:
extender_repro.zip

It turns out that the requires_arc option in a Podspec can be not only boolean but also a string or an array:

https://guides.cocoapods.org/syntax/podspec.html#requires_arc

In CocoapodsService we assume it is a boolean which throws an exception if the option is a string or array:

java.lang.ClassCastException: class java.lang.String cannot be cast to class java.lang.Boolean (java.lang.String and java.lang.Boolean are in module java.base of loader 'bootstrap')
	at com.defold.extender.services.CocoaPodsService.createPodSpec(CocoaPodsService.java:420)
	at com.defold.extender.services.CocoaPodsService.createPodSpec(CocoaPodsService.java:482)
	at com.defold.extender.services.CocoaPodsService.installPods(CocoaPodsService.java:548)
	at com.defold.extender.services.CocoaPodsService.resolveDependencies(CocoaPodsService.java:589)
	at com.defold.extender.Extender.resolve(Extender.java:2165)
	at com.defold.extender.ExtenderController.buildEngine(ExtenderController.java:224)
	at jdk.internal.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)

We've had incidents in the past when the disk space ran out. We need to add an alarm for this so we can be preemptive.

Newer versions of Firebase require module support. Adding -fmodules and -fcxx-modules results in error "cyclic dependency in module 'Darwin': Darwin -> std -> Darwin"

clang++ -c -arch arm64 -target arm-apple-darwin19 -m64 -O2 -g -stdlib=libc++ -miphoneos-version-min=11.0 -isysroot /Users/bjornritzl/extender/platformsdk/iPhoneOS16.2.sdk -nostdinc++ -isystem /Users/bjornritzl/extender/platformsdk/iPhoneOS16.2.sdk/usr/include/c++/v1  -DDLIB_LOG_DOMAIN="FIREBASECORE" -DDDF_EXPOSE_DESCRIPTORS -DDM_PLATFORM_IOS -DDM_DEBUG -DFirebase_VERSION=10.12.0 -DCOCOAPODS=1  -fno-autolink -fobjc-arc --language=objective-c -fmodules -fmodule-name=FirebaseCore  -Iupload/ -Iupload/firebase/include/ -ICocoaPodsService/Pods/FirebaseCore/FirebaseCore/Sources/Public/FirebaseCore/ -ICocoaPodsService/Pods/FirebaseCore/FirebaseCore/Sources/Public/ -ICocoaPodsService/Pods/FirebaseCore/FirebaseCore/Sources/ -ICocoaPodsService/Pods/FirebaseCore/FirebaseCore/ -ICocoaPodsService/Pods/FirebaseCore/FirebaseCore/Extension/ -ICocoaPodsService/Pods/FirebaseCore/ -ICocoaPodsService/Pods/FirebaseCoreInternal/ -ICocoaPodsService/Pods/FirebaseInstallations/FirebaseInstallations/Source/Library/InstallationsIDController/ -ICocoaPodsService/Pods/FirebaseInstallations/FirebaseInstallations/Source/Library/ -ICocoaPodsService/Pods/FirebaseInstallations/FirebaseInstallations/Source/Library/Public/FirebaseInstallations/ -ICocoaPodsService/Pods/FirebaseInstallations/FirebaseInstallations/Source/Library/Public/ -ICocoaPodsService/Pods/FirebaseInstallations/FirebaseInstallations/Source/Library/Private/ -ICocoaPodsService/Pods/FirebaseInstallations/FirebaseInstallations/Source/Library/InstallationsAPI/ -ICocoaPodsService/Pods/FirebaseInstallations/FirebaseInstallations/Source/Library/Errors/ -ICocoaPodsService/Pods/FirebaseInstallations/FirebaseInstallations/Source/Library/IIDMigration/ -ICocoaPodsService/Pods/FirebaseInstallations/FirebaseInstallations/Source/Library/InstallationsStore/ -ICocoaPodsService/Pods/FirebaseInstallations/FirebaseInstallations/Source/ -ICocoaPodsService/Pods/FirebaseInstallations/FirebaseCore/Extension/ -ICocoaPodsService/Pods/FirebaseInstallations/FirebaseCore/ -ICocoaPodsService/Pods/FirebaseInstallations/ -ICocoaPodsService/Pods/GoogleUtilities/GoogleUtilities/Environment/Public/GoogleUtilities/ -ICocoaPodsService/Pods/GoogleUtilities/GoogleUtilities/Environment/Public/ -ICocoaPodsService/Pods/GoogleUtilities/ -ICocoaPodsService/Pods/GoogleUtilities/GoogleUtilities/Logger/Public/GoogleUtilities/ -ICocoaPodsService/Pods/GoogleUtilities/GoogleUtilities/Logger/Public/ -ICocoaPodsService/Pods/GoogleUtilities/ -ICocoaPodsService/Pods/GoogleUtilities/GoogleUtilities/NSData+zlib/Public/GoogleUtilities/ -ICocoaPodsService/Pods/GoogleUtilities/GoogleUtilities/NSData+zlib/Public/ -ICocoaPodsService/Pods/GoogleUtilities/ -ICocoaPodsService/Pods/GoogleUtilities/GoogleUtilities/UserDefaults/Public/GoogleUtilities/ -ICocoaPodsService/Pods/GoogleUtilities/GoogleUtilities/UserDefaults/Public/ -ICocoaPodsService/Pods/GoogleUtilities/ -ICocoaPodsService/Pods/PromisesObjC/Sources/FBLPromises/include/ -ICocoaPodsService/Pods/PromisesObjC/Sources/FBLPromises/ -ICocoaPodsService/Pods/PromisesObjC/ -ICocoaPodsService/frameworks/headers/arm64-ios/  -F/Users/bjornritzl/Downloads/EXTENDER/job1/CocoaPodsService/frameworks/lib/arm64-ios  -I/Users/bjornritzl/projects/defold/tmp/dynamo_home//include -I/Users/bjornritzl/projects/defold/tmp/dynamo_home//sdk/include -I/Users/bjornritzl/projects/defold/tmp/dynamo_home//ext/include   CocoaPodsService/Pods/FirebaseCore/FirebaseCore/Sources/FIRDependency.m -obuild/FIRDependency.m_7.o
While building module 'Foundation' imported from CocoaPodsService/Pods/FirebaseCore/FirebaseCore/Extension/FIRComponentType.h:17:
While building module 'CoreFoundation' imported from /Users/bjornritzl/extender/platformsdk/iPhoneOS16.2.sdk/System/Library/Frameworks/Foundation.framework/Headers/Foundation.h:6:
While building module 'Darwin' imported from /Users/bjornritzl/extender/platformsdk/iPhoneOS16.2.sdk/System/Library/Frameworks/CoreFoundation.framework/Headers/CoreFoundation.h:16:
While building module 'std' imported from /Users/bjornritzl/extender/platformsdk/iPhoneOS16.2.sdk/usr/include/tgmath.h:31:
In file included from <module-includes>:1:
/Users/bjornritzl/extender/platformsdk/iPhoneOS16.2.sdk/usr/include/c++/v1/ctype.h:38:15: fatal error: cyclic dependency in module 'Darwin': Darwin -> std -> Darwin
#include_next <ctype.h>