kubernetes-retired / application-images

[EOL] Image build contents for Kubernetes applications.

application-images

This repository houses the Docker image build contents for (some) applications that run on Kubernetes. Specifically, it houses the build content and rules for any applications that require custom built Docker images.

Structure

Each directory should include:

• a top-level README describing the contents
• a Makefile with a make push rule that will push to gcr.io/google_containers (it has to be something that @k8s-oncall can push to).

Contributing

When in doubt, see the official Kubernetes contributing guidelines.

Merge guidelines

You or the reviewer can merge on a single LGTM. After you submit a PR, mention @k8s-oncall to get the image pushed (if you or the merger aren't able).

Joining the team

If you've submitted an image to this repo and will maintain it actively, consider asking to join the application-images-maintainers team.

application-images's Issues

Merge this repo with whichever repo the configs will live in

Let's not use a separate repo for images and configs.

cc @kubernetes/examples

Active usage of repository

Is this repository in active use? We're no longer using/supporting the images here for spark-native.
Some of the other contents also look stale at this point. If the bits are rotting, we might want to clean them up and repo possibly.

/cc @mattf @spiffxp

Standalone Spark Executors on Kubernetes throw "Connection Timeout"

Currently we're running Spark on K8s in AWS with Flannel.

Spark Version: 1.6.2
Spark Deployment Mode: Standalone
K8s Version: 1.3.7

The cluster is up, and all the components connect successfully. The issues appear when we submit a job to Spark.
In the Spark UI, when I go to the executors tab, I see wrong IP address for executors, which doesn't match the POD IP addresse.

POD IP Addresses from kubectl

$kubectl get po

Name	IP
spark-master-7v08x	10.70.20.5
spark-worker-1soa1	10.70.19.6
spark-worker-mg7q5	10.70.28.8
spark-worker-vvr4n	10.70.20.4

The executor IP Addresses from the Spark UI

Executor ID	Address
0	10.70.20.4:39955
1	10.70.20.4:48970
2	10.70.19.0:38067

If I check the executor logs, I can see exceptions because the executors are trying to connect to the wrong IP:

16/10/07 16:44:04 INFO RetryingBlockFetcher: Retrying fetch (2/3) for 1 outstanding blocks after 5000 ms
16/10/07 16:45:12 ERROR RetryingBlockFetcher: Exception while beginning fetch of 1 outstanding blocks (after 2 retries)
java.io.IOException: Failed to connect to /10.70.19.0:38067
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:167)
    at org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:90)
    at org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)
    at org.apache.spark.network.shuffle.RetryingBlockFetcher.access$200(RetryingBlockFetcher.java:43)
    at org.apache.spark.network.shuffle.RetryingBlockFetcher$1.run(RetryingBlockFetcher.java:170)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection timed out: /10.70.19.0:38067
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
    ... 1 more

I've found some related issues on the Spark Jira:

• Spark using Netty RPC gets wrong address in some setups
• shuffle broken when accessing standalone cluster through NAT

I've seen that @mattf and @zmerlynn did a lot of work around running spark in docker and on Kubernetes so I was hoping we can come up with some configuration workaround ( probably by using Kubernetes features) for launching the cluster which will prevent this from happening.

Update to 2.0.1

I have a branch on my fork where I update to spark 2.0.1 and Hadoop 2.7.3 and increase the ram sizes including ZEPPELIN_MEM so it can be used for a real workload. Curious if you are interested in getting these changes as a PR.

Docker files for native Spark

The k8s-spark-native docker files can live here till we find a better location for them.
This is single point that can be used, rather than having multiple custom images as we currently do.
I've committed here with some modifications, the files that were previously in https://github.com/erikerlandson/openshift-spark/tree/k8s-native.
Do let me know if I've missed something.

TODO:

• Clean up.
• Standardize and come up with a specification of shim scripts for people to enable building custom images.

/cc @mattf @erikerlandson @iyanuobidele @mccheah

Create a SECURITY_CONTACTS file.

As per the email sent to kubernetes-dev[1], please create a SECURITY_CONTACTS
file.

The template for the file can be found in the kubernetes-template repository[2].
A description for the file is in the steering-committee docs[3], you might need
to search that page for "Security Contacts".

Please feel free to ping me on the PR when you make it, otherwise I will see when
you close this issue. :)

Thanks so much, let me know if you have any questions.

(This issue was generated from a tool, apologies for any weirdness.)

[1] https://groups.google.com/forum/#!topic/kubernetes-dev/codeiIoQ6QE
[2] https://github.com/kubernetes/kubernetes-template-project/blob/master/SECURITY_CONTACTS
[3] https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-governance-template-short.md