Custom Build Outputs `--output` about bollard

[levinwinter]

Sorry for the delay, I was not working on this :)

For the moment, my idea is to copy-paste the bit of auto-generated code that I get when changing the protobuf file to DiffCopy(stream fsutil.types.Packet) into the repo and simply wire that up if the local exporter is selected.

As for the "interface" on how to integrate it into bollard, I was thinking of adding a field outputs to BuildImageOptions that takes a Option<ImageBuildOutput>. Currently, there is already a generic ImageBuildOutput in bollard, but perhaps a nicer interface would be something like this.

enum ImageBuildOutput
where
    T: Into<String> + Eq + Hash + Serialize,
{
    /// Exports a tarball to the specified path.
    Tar(T),
    /// Exports the filesystem to the specified path.
    Local(T),
}

Just sharing my ideas and keeping you updated. Will lyk once the PR is up :)

[fussybeaver]

There should be an integration test that demonstrates how to export an OCI image using the GRPC API: https://github.com/fussybeaver/bollard/blob/master/tests/export_test.rs

Note that, the link you provided uses the moby HTTP API (as opposed to buildkit's formal GRPC API), although this is also supported in Bollard through the build_image method, you cannot export OCI images using that API, as documented here.

[levinwinter]

Thank you for the pointers! I'm not super familiar with moby/buildkit, so I took some time to get a basic understanding.

Instead of exporting OCI images, I'd like to use the local or tar exporters (docs) that simply dump the file system of the last layer. I managed to get this up and running by adding a tar option to the ImageExporterEnum and using the docker container gRPC driver.

However, I would prefer to use the build_image method since it's easier and should, in theory (?), also support this. I tried extending that one, but I think it would also need to use the /grpc endpoint as opposed to /session (I keep getting issues with /moby.filesync.v1.FileSend/diffcopy not being recognized by the server).

Let me know if you'd be okay with adding the local/tar exporter, and if so, where you see best fit. I'd be happy to prepare a PR!

[fussybeaver]

Yes, adding the local and tar exporters to build_image would require a couple of changes in Bollard and a PR is very welcome. I'm not so sure you need to handle the /grpc endpoint - this is actually created by the moby docker server if you toggle the buildkit code path by providing a session as part of the BuildImageOptions.

The reason you get a /moby.filesync.v1.FileSend/diffcopy error is because buildkit initiates a GRPC request to save the exported payload to disk, but the filesend provider isn't registered as a routable part of the /session endpoint.

One option is to add the filesend plumbing to the /session endpoint and parameterise it somehow, presumably by adding the output field to BuildImageOptions, though how that field is parsed and interpreted is probably what needs some thought.

[levinwinter]

Awesome! I managed to get the /session endpoint to work with diffcopy. I already tried this before, but I didn't know I also need to register it with the X-Docker-Expose-Session-Grpc-Method header. Exporting a single file using tar is now working.

I'm having issues with the local exporter however. The read loop in FileSendImpl seems to "hang", probably because the protocol is more complex when sending multiple files? Is there a reference implementation for this somewhere? I tried to hunt around buildkit, but I'm not quite sure what exactly is expected.

The data that I receive in the loop is just empty packets (the number of empty packets being equal to the number of files that the last layer has).

[fussybeaver]

The reference implementation for the diffcopy / and filesend (curiously called filesync in buildkit) is here: https://github.com/moby/buildkit/blob/44ebf9071db49821538cd37c2687dd925c7c3661/session/filesync/filesync.go#L78

Although the whole end-to-end flow is somewhat spread across moby, buildkit and the buildx repositories (and quite difficult to follow)

It's possible that some information is stored in the GRPC header metadata, which is not handled in Bollard's implementation..

Be sure to rebase from master as the session headers should be registered uniformly with the grpc_handle method.

[levinwinter]

Thank you! I'm now using the grpc_handle method!

I think that the correct protocol is described here in the fsutil repository.

To receive meaningful data when selecting the local exporter (which sends multiple files), I needed to change the type of the streamed messages in the diffcopy method of the FileSend service. While before the messages were deseralized to empty BytesMessage (i.e. BytesMessage { data: [] }), I now receive meaningful fsutil.types.Packets that contain the filenames of the export. Do you have any idea why that could be?

// FileSync exposes local files from the client to the server.
service FileSync{
	rpc DiffCopy(stream fsutil.types.Packet) returns (stream fsutil.types.Packet);
	rpc TarStream(stream fsutil.types.Packet) returns (stream fsutil.types.Packet);
}

// FileSend allows sending files from the server back to the client.
service FileSend{
-	rpc DiffCopy(stream BytesMessage) returns (stream BytesMessage);
+	rpc DiffCopy(stream fsutil.types.Packet) returns (stream fsutil.types.Packet);
}

To add to this: When exporting using tar, the messages still need to be deserialized as BytesMessage and only when exporting using local one needs to use fsutil.types.Packet. I guess depending on which is passed as an argument, we could chose the correct implementation. Though I must say I'm not sure why this is the case in the first place.

[fussybeaver]

I've noticed that the buildkit protobuf has generated a separate FileSync implementation that takes a Packet:

bollard/codegen/proto/src/generated/moby.filesync.v1.rs

Lines 867 to 1128 in cf88562

	pub mod file_sync_server {
	#![allow(unused_variables, dead_code, missing_docs, clippy::let_unit_value)]
	use tonic::codegen::*;
	/// Generated trait containing gRPC methods that should be implemented for use with FileSyncServer.
	#[async_trait]
	pub trait FileSync: Send + Sync + 'static {
	/// Server streaming response type for the DiffCopy method.
	type DiffCopyStream: tonic::codegen::tokio_stream::Stream<
	Item = std::result::Result<
	super::super::super::super::fsutil::types::Packet,
	tonic::Status,
	>,
	>
	+ Send
	+ 'static;
	async fn diff_copy(
	&self,
	request: tonic::Request<
	tonic::Streaming<super::super::super::super::fsutil::types::Packet>,
	>,
	) -> std::result::Result<tonic::Response<Self::DiffCopyStream>, tonic::Status>;
	/// Server streaming response type for the TarStream method.
	type TarStreamStream: tonic::codegen::tokio_stream::Stream<
	Item = std::result::Result<
	super::super::super::super::fsutil::types::Packet,
	tonic::Status,
	>,
	>
	+ Send
	+ 'static;
	async fn tar_stream(
	&self,
	request: tonic::Request<
	tonic::Streaming<super::super::super::super::fsutil::types::Packet>,
	>,
	) -> std::result::Result<tonic::Response<Self::TarStreamStream>, tonic::Status>;
	}
	/// FileSync exposes local files from the client to the server.
	#[derive(Debug)]
	pub struct FileSyncServer<T: FileSync> {
	inner: _Inner<T>,
	accept_compression_encodings: EnabledCompressionEncodings,
	send_compression_encodings: EnabledCompressionEncodings,
	max_decoding_message_size: Option<usize>,
	max_encoding_message_size: Option<usize>,
	}
	struct _Inner<T>(Arc<T>);
	impl<T: FileSync> FileSyncServer<T> {
	pub fn new(inner: T) -> Self {
	Self::from_arc(Arc::new(inner))
	}
	pub fn from_arc(inner: Arc<T>) -> Self {
	let inner = _Inner(inner);
	Self {
	inner,
	accept_compression_encodings: Default::default(),
	send_compression_encodings: Default::default(),
	max_decoding_message_size: None,
	max_encoding_message_size: None,
	}
	}
	pub fn with_interceptor<F>(
	inner: T,
	interceptor: F,
	) -> InterceptedService<Self, F>
	where
	F: tonic::service::Interceptor,
	{
	InterceptedService::new(Self::new(inner), interceptor)
	}
	/// Enable decompressing requests with the given encoding.
	#[must_use]
	pub fn accept_compressed(mut self, encoding: CompressionEncoding) -> Self {
	self.accept_compression_encodings.enable(encoding);
	self
	}
	/// Compress responses with the given encoding, if the client supports it.
	#[must_use]
	pub fn send_compressed(mut self, encoding: CompressionEncoding) -> Self {
	self.send_compression_encodings.enable(encoding);
	self
	}
	/// Limits the maximum size of a decoded message.
	///
	/// Default: `4MB`
	#[must_use]
	pub fn max_decoding_message_size(mut self, limit: usize) -> Self {
	self.max_decoding_message_size = Some(limit);
	self
	}
	/// Limits the maximum size of an encoded message.
	///
	/// Default: `usize::MAX`
	#[must_use]
	pub fn max_encoding_message_size(mut self, limit: usize) -> Self {
	self.max_encoding_message_size = Some(limit);
	self
	}
	}
	impl<T, B> tonic::codegen::Service<http::Request<B>> for FileSyncServer<T>
	where
	T: FileSync,
	B: Body + Send + 'static,
	B::Error: Into<StdError> + Send + 'static,
	{
	type Response = http::Response<tonic::body::BoxBody>;
	type Error = std::convert::Infallible;
	type Future = BoxFuture<Self::Response, Self::Error>;
	fn poll_ready(
	&mut self,
	_cx: &mut Context<'_>,
	) -> Poll<std::result::Result<(), Self::Error>> {
	Poll::Ready(Ok(()))
	}
	fn call(&mut self, req: http::Request<B>) -> Self::Future {
	let inner = self.inner.clone();
	match req.uri().path() {
	"/moby.filesync.v1.FileSync/DiffCopy" => {
	#[allow(non_camel_case_types)]
	struct DiffCopySvc<T: FileSync>(pub Arc<T>);
	impl<
	T: FileSync,
	> tonic::server::StreamingService<
	super::super::super::super::fsutil::types::Packet,
	> for DiffCopySvc<T> {
	type Response = super::super::super::super::fsutil::types::Packet;
	type ResponseStream = T::DiffCopyStream;
	type Future = BoxFuture<
	tonic::Response<Self::ResponseStream>,
	tonic::Status,
	>;
	fn call(
	&mut self,
	request: tonic::Request<
	tonic::Streaming<
	super::super::super::super::fsutil::types::Packet,
	>,
	>,
	) -> Self::Future {
	let inner = Arc::clone(&self.0);
	let fut = async move {
	<T as FileSync>::diff_copy(&inner, request).await
	};
	Box::pin(fut)
	}
	}
	let accept_compression_encodings = self.accept_compression_encodings;
	let send_compression_encodings = self.send_compression_encodings;
	let max_decoding_message_size = self.max_decoding_message_size;
	let max_encoding_message_size = self.max_encoding_message_size;
	let inner = self.inner.clone();
	let fut = async move {
	let inner = inner.0;
	let method = DiffCopySvc(inner);
	let codec = tonic::codec::ProstCodec::default();
	let mut grpc = tonic::server::Grpc::new(codec)
	.apply_compression_config(
	accept_compression_encodings,
	send_compression_encodings,
	)
	.apply_max_message_size_config(
	max_decoding_message_size,
	max_encoding_message_size,
	);
	let res = grpc.streaming(method, req).await;
	Ok(res)
	};
	Box::pin(fut)
	}
	"/moby.filesync.v1.FileSync/TarStream" => {
	#[allow(non_camel_case_types)]
	struct TarStreamSvc<T: FileSync>(pub Arc<T>);
	impl<
	T: FileSync,
	> tonic::server::StreamingService<
	super::super::super::super::fsutil::types::Packet,
	> for TarStreamSvc<T> {
	type Response = super::super::super::super::fsutil::types::Packet;
	type ResponseStream = T::TarStreamStream;
	type Future = BoxFuture<
	tonic::Response<Self::ResponseStream>,
	tonic::Status,
	>;
	fn call(
	&mut self,
	request: tonic::Request<
	tonic::Streaming<
	super::super::super::super::fsutil::types::Packet,
	>,
	>,
	) -> Self::Future {
	let inner = Arc::clone(&self.0);
	let fut = async move {
	<T as FileSync>::tar_stream(&inner, request).await
	};
	Box::pin(fut)
	}
	}
	let accept_compression_encodings = self.accept_compression_encodings;
	let send_compression_encodings = self.send_compression_encodings;
	let max_decoding_message_size = self.max_decoding_message_size;
	let max_encoding_message_size = self.max_encoding_message_size;
	let inner = self.inner.clone();
	let fut = async move {
	let inner = inner.0;
	let method = TarStreamSvc(inner);
	let codec = tonic::codec::ProstCodec::default();
	let mut grpc = tonic::server::Grpc::new(codec)
	.apply_compression_config(
	accept_compression_encodings,
	send_compression_encodings,
	)
	.apply_max_message_size_config(
	max_decoding_message_size,
	max_encoding_message_size,
	);
	let res = grpc.streaming(method, req).await;
	Ok(res)
	};
	Box::pin(fut)
	}
	_ => {
	Box::pin(async move {
	Ok(
	http::Response::builder()
	.status(200)
	.header("grpc-status", "12")
	.header("content-type", "application/grpc")
	.body(empty_body())
	.unwrap(),
	)
	})
	}
	}
	}
	}
	impl<T: FileSync> Clone for FileSyncServer<T> {
	fn clone(&self) -> Self {
	let inner = self.inner.clone();
	Self {
	inner,
	accept_compression_encodings: self.accept_compression_encodings,
	send_compression_encodings: self.send_compression_encodings,
	max_decoding_message_size: self.max_decoding_message_size,
	max_encoding_message_size: self.max_encoding_message_size,
	}
	}
	}
	impl<T: FileSync> Clone for _Inner<T> {
	fn clone(&self) -> Self {
	Self(Arc::clone(&self.0))
	}
	}
	impl<T: std::fmt::Debug> std::fmt::Debug for _Inner<T> {
	fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
	write!(f, "{:?}", self.0)
	}
	}
	impl<T: FileSync> tonic::server::NamedService for FileSyncServer<T> {
	const NAME: &'static str = "moby.filesync.v1.FileSync";
	}
	}

So, maybe you just need to implement the trait with an appropriate Provider that implements the fileutil as you pointed out, and hook it up to the session endpoint.

[levinwinter]

I tried to implement FileSync, but that seems to be the wrong gRPC service/endpoint (/moby.filesync.v1.FileSync/DiffCopy vs. /moby.filesync.v1.FileSend/DiffCopy). From what I can understand when looking at the Go implementation, they have some sort of raw gRPC stream (of the FileSend service) and just serialize either the BytesMessage or Packet. To be honest, I'm a bit stuck since I don't know whether something equivalent is possible using tonic.

The only "idea" that comes to my mind is to copy the generated protobuf code and have a manual/alternative implementation at hand. But this feels super hack and I'm sure there must be a better way. If you have no idea I could also ask ob moby/buildkit.

[fussybeaver]

Sounds a little weird, one thing you could try is to enable the jaeger tracing interface, which will let you drill down into the payloads sent from buildkit. https://github.com/moby/buildkit?tab=readme-ov-file#opentelemetry-support

Regardless, do keep this thread up-to-date if you get a breakthrough somehow by hacking around the protobuf files..