Runtime problem when using Apple Silicon about fortran-curl

[certik]

I would not use variadic arguments from Fortran like this. I think it's only working by accident, and only on linux.

[perazz]

Just doublechecked the latest F2023 standard draft and it turns out that Ondrej @certik is right: at 18.3.7

A Fortran procedure interface is interoperable with a C function prototype if
[...]
(7) the prototype does not have variable arguments as denoted by the ellipsis (...)
[...]
Note 2 - The C language allows specification of a C function that can take a variable number of 
arguments (ISO/IEC 9899:2018, 7.16). This document does not provide a mechanism for Fortran 
procedures to interoperate with such C functions.

[interkosmos]

Unfortunately, I can’t reproduce this behaviour, neither on Linux nor FreeBSD. Hard to tell from the distance what the cause of this issue is, could be related to libcurl or GNU Fortran.

[perazz]

Froma quick test on my M1, the first error is actually on CURLOPT_TIMEOUT: from the main program,

    ! Set curl options.
    rc = curl_easy_setopt(curl_ptr, CURLOPT_DEFAULT_PROTOCOL, DEFAULT_PROTOCOL)
    if (rc/=CURLE_OK) stop 1
    rc = curl_easy_setopt(curl_ptr, CURLOPT_URL,              DEFAULT_URL)
    if (rc/=CURLE_OK) stop 2
    rc = curl_easy_setopt(curl_ptr, CURLOPT_FOLLOWLOCATION,   1)
    if (rc/=CURLE_OK) stop 3
    rc = curl_easy_setopt(curl_ptr, CURLOPT_TIMEOUT,          10) 
    if (rc/=CURLE_OK) stop 4 ! Stop here

[perazz]

On Apple M1 systems with homebrew there are two competing curl installs:

• system provided (currently 7.88.1)
• homebrew provided (8.2.1), which I link via

-L/opt/homebrew/Cellar/curl/8.2.1/lib/
-I/opt/homebrew/Cellar/curl/8.2.1/include/

I've tried manually linking against either, but the same behavior is seen

[interkosmos]

In the program examples/http/http.f90, could you please output the return code of curl_easy_perform(curl_ptr)?

rc = curl_easy_perform(curl_ptr)
print *, rc

Does the following example in C work on your system?

/* example.c */

#include <stdio.h>
#include <curl/curl.h>

int main(void)
{
    CURL *curl;
    CURLcode res;

    curl = curl_easy_init();

    if (curl) {
        curl_easy_setopt(curl, CURLOPT_URL, "https://www.netlib.org/slatec/readme");

        res = curl_easy_perform(curl);

        if (res != CURLE_OK)
            fprintf(stderr, "curl_easy_perform() failed: %s\n",
                    curl_easy_strerror(res));

        curl_easy_cleanup(curl);
    }

    curl_global_cleanup();

    return 0;
}

$ cc -I/usr/local/include -L/usr/local/lib -o example example.c -lcurl

[HenkPoley]

Yes, both work. The output is CURLE_URL_MALFORMAT (3).

% build/gfortran_E167FD2A985B468F/example/http
           3
STOP Error: curl_easy_perform() failed
% ./example
 ===== readme for slatec =====

SLATEC Common Mathematical Library, Version 4.1, July 1993
a comprehensive software library containing over
1400 general purpose mathematical and statistical routines
written in Fortran 77.

Together with ChatGPT (🙈) I managed to build something that successfully passes a string from Fortran to C on this system. Not sure if this helps any. Since it's not your library.

#include <stdio.h>
#include <string.h>
#include <curl/curl.h>

void http_get(char *url, int length){
  CURL *curl;
  CURLcode res;
  char c_url[length + 1];
  
  strncpy(c_url, url, length);
  c_url[length] = '\0';

  curl_global_init(CURL_GLOBAL_DEFAULT);
  curl = curl_easy_init();
  if(curl) {
    curl_easy_setopt(curl, CURLOPT_URL, c_url);
    res = curl_easy_perform(curl);
    if(res != CURLE_OK)
      fprintf(stderr, "curl_easy_perform() failed: %s\n",
              curl_easy_strerror(res));
    curl_easy_cleanup(curl);
  }
  curl_global_cleanup();
}

program http_get_fortran
  use, intrinsic :: iso_c_binding
  implicit none
  
  call perform_http_get("http://example.com")
  
contains

  subroutine perform_http_get(url_f)
    character(len=*) :: url_f
    character(len=18), target :: url_target
    type(C_PTR) :: url_c_ptr
    integer(kind=C_INT) :: length
    
    interface
      subroutine http_get(url, length) bind(C, name='http_get')
        import :: C_PTR, C_INT
        type(C_PTR), value :: url
        integer(kind=C_INT), value :: length
      end subroutine http_get
    end interface
    
    url_target = url_f
    length = len_trim(url_target)
    url_c_ptr = c_loc(url_target)
    
    call http_get(url_c_ptr, length)
  end subroutine perform_http_get

end program http_get_fortran

gcc -c http_get.c
gfortran app/main.f90 http_get.o -lcurl -o http_get_fortran_app

[HenkPoley]

For good measure a version that doesn't use the length (2nd) parameter, but successfully passes just a null terminated string.

#include <stdio.h>
#include <string.h>
#include <curl/curl.h>

void http_get(char *url){
  CURL *curl;
  CURLcode res;

  curl_global_init(CURL_GLOBAL_DEFAULT);
  curl = curl_easy_init();
  if(curl) {
    curl_easy_setopt(curl, CURLOPT_URL, url);
    res = curl_easy_perform(curl);
    if(res != CURLE_OK)
      fprintf(stderr, "curl_easy_perform() failed: %s\n",
              curl_easy_strerror(res));
    curl_easy_cleanup(curl);
  }
  curl_global_cleanup();
}

program http_get_fortran
  use, intrinsic :: iso_c_binding
  implicit none
  
  call perform_http_get("http://example.com")
  
contains

  subroutine perform_http_get(url_f)
    character(len=*) :: url_f
    character(len=18), target :: url_target
    type(C_PTR) :: url_c_ptr
    
    interface
      subroutine http_get(url) bind(C, name='http_get')
        import :: C_PTR
        type(C_PTR), value :: url
      end subroutine http_get
    end interface
    
    url_target = url_f // C_NULL_CHAR
    url_c_ptr = c_loc(url_target)
    
    call http_get(url_c_ptr)
  end subroutine perform_http_get

end program http_get_fortran

Which is sadly still different than the CURL * (or void *).

[interkosmos]

It seems as interface curl_easy_setopt() does not pass the whole character string to libcurl. Since libcurl 7.17.0, strings are copied by default, so, technically, it shouldn’t be a problem at all.

Does example http.f90 work when you add c_null_char to the URL?:

character(len=*), parameter :: DEFAULT_URL = 'https://www.netlib.org/slatec/readme' // c_null_char

[HenkPoley]

Sadly adding c_null_char wasn't enough.

program main
    use, intrinsic :: iso_c_binding
    use, intrinsic :: iso_fortran_env, only: i8 => int64
    use :: curl
    use :: http_callback
    implicit none
        
    character(len=*), parameter :: DEFAULT_PROTOCOL = 'http'
    character(len=*), parameter :: DEFAULT_URL      = 'https://www.netlib.org/slatec/readme' // c_null_char
        
    type(c_ptr)                 :: curl_ptr
    integer                     :: rc
    type(response_type), target :: response 

% build/gfortran_E167FD2A985B468F/example/http
           3
STOP Error: curl_easy_perform() failed

[interkosmos]

Could you try the latest commit that adds interfaces and calls to curl_global_init() and curl_global_cleanup()?

Which GNU Fortran compiler flags are set? If built with fpm, have you selected profile debug or release to make fortran-curl?

[HenkPoley]

Just fpm build (so that should be debug by default)

Sadly the latest commit still has issues.

% build/gfortran_E167FD2A985B468F/example/http                     
STOP Error: curl_easy_perform() failed

[certik]

On my Apple M1:

$ git clone https://github.com/interkosmos/fortran-curl
Cloning into 'fortran-curl'...
remote: Enumerating objects: 484, done.
remote: Counting objects: 100% (113/113), done.
remote: Compressing objects: 100% (72/72), done.
remote: Total 484 (delta 32), reused 54 (delta 24), pack-reused 371
Receiving objects: 100% (484/484), 98.20 KiB | 866.00 KiB/s, done.
Resolving deltas: 100% (228/228), done.
$ cd fortran-curl 
$ fpm build
 + mkdir -p build/dependencies
curl_macro.c                           done.
curl.f90                               done.
libfortran-curl.a                      done.
dict.f90                               done.
download.f90                           done.
getinfo.f90                            done.
gopher.f90                             done.
http.f90                               done.
imap.f90                               done.
smtp.f90                               done.
post.f90                               done.
version.f90                            done.
dict                                   done.
download                               done.
getinfo                                done.
gopher                                 done.
http                                   done.
imap                                   done.
smtp                                   done.
post                                   done.
version                                done.
[100%] Project compiled successfully.
$ fpm run --example http
Project is up to date
STOP Error: curl_easy_perform() failed

I tried to debug it a little bit, such as adding the null pointers at the end of the string, which I am quite sure you have to do anyway even on linux (that seems like a bug?), and adding "stop" into the callback (it doesn't get called). Nothing helps. It would require more serious debugging, but unfortunately I don't have time to do that.

The C example above #11 (comment) works:

$ ./example
 ===== readme for slatec =====

SLATEC Common Mathematical Library, Version 4.1, July 1993
a comprehensive software library containing over
1400 general purpose mathematical and statistical routines
written in Fortran 77.

[certik]

Yes, for me as well. But if you comment it out, it will still fail later as above.

[HenkPoley]

I've added some debugging prints to the actual libcurl. No need for mocks 🙈.

You can download curl from: https://curl.se/download.html

Build it with:

./configure --with-openssl  # or --with-secure-transport since this is macOS
make
make install

This gets installed in a different place than where macOS has libcurl. So we override the DYLD library path order.

export DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH

curl_easy_setopt() is located in lib/setopt.c at the bottom of the file.

You can add some debugging like this:

void print_string_hex(const char *str) {
    if (str == NULL) return;
    while (*str != '\0') {
        printf("%02x ", (unsigned char)*str);
        str++;
    }
    printf("\n");
}

/*
 * curl_easy_setopt() is the external interface for setting options on an
 * easy handle.
 *
 * NOTE: This is one of few API functions that are allowed to be called from
 * within a callback.
 */

#undef curl_easy_setopt
CURLcode curl_easy_setopt(struct Curl_easy *data, CURLoption tag, ...)
{
  va_list arg;
  CURLcode result;

  if(!data)
    return CURLE_BAD_FUNCTION_ARGUMENT;

  va_start(arg, tag);

  printf("Debug: Entering curl_easy_setopt, option: %d\n", tag);

  // Add your debugging prints here
  switch(tag) {
    case CURLOPT_DEFAULT_PROTOCOL:
    {
      char *proto = va_arg(arg, char *);
      printf("Debug: curl_easy_setopt, CURLOPT_DEFAULT_PROTOCOL: %s\n", proto);
      print_string_hex(proto);
      break;
    }
    case CURLOPT_URL:
    {
      char *url = va_arg(arg, char *);
      printf("Debug: curl_easy_setopt, CURLOPT_URL: %s\n", url);
      print_string_hex(url);
      break;
    }
    case CURLOPT_TIMEOUT:
    {
      long timeout = va_arg(arg, long);
      printf("Debug: curl_easy_setopt, CURLOPT_TIMEOUT: %ld\n", timeout);
      break;
    }
    default:
    {
      // Handle other options or do nothing
      break;
    }
  }

  result = Curl_vsetopt(data, tag, arg);

  va_end(arg);

  printf("Debug: Exiting curl_easy_setopt\n");

  return result;
}

I added some debugging prints like this to examples/http/http.f90:

    ! Set curl options.
    print *, 'CURLOPT_DEFAULT_PROTOCOL ', CURLOPT_DEFAULT_PROTOCOL, ' DEFAULT_PROTOCOL ', DEFAULT_PROTOCOL
    rc = curl_easy_setopt(curl_ptr, CURLOPT_DEFAULT_PROTOCOL, DEFAULT_PROTOCOL)
    print *, 'CURLOPT_URL ', CURLOPT_URL, ' DEFAULT_URL ', DEFAULT_URL
    rc = curl_easy_setopt(curl_ptr, CURLOPT_URL,              DEFAULT_URL)

And augmented curl_easy_setopt_char() inside src/curl.f90 with a print statement, so we can see it's actually been executed.

    ! CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...)
    function curl_easy_setopt_char(curl, option, parameter) result(rc)
        type(c_ptr),      intent(in) :: curl
        integer,          intent(in) :: option
        character(len=*), intent(in) :: parameter
        integer                      :: rc 
    
        print *, "Inside curl_easy_setopt_char"

        rc = curl_easy_setopt_c_char(curl, option, parameter // c_null_char)     
    end function curl_easy_setopt_char

Output of build/gfortran_E167FD2A985B468F/example/http is now:

% build/gfortran_E167FD2A985B468F/example/http
 CURLOPT_DEFAULT_PROTOCOL        10238  DEFAULT_PROTOCOL http
 Inside curl_easy_setopt_char
Debug: Entering curl_easy_setopt, option: 10238
Debug: curl_easy_setopt, CURLOPT_DEFAULT_PROTOCOL: ??m
b0 e1 a7 6d 01 
Debug: Exiting curl_easy_setopt
 CURLOPT_URL        10002  DEFAULT_URL https://www.netlib.org/slatec/readme
 Inside curl_easy_setopt_char
Debug: Entering curl_easy_setopt, option: 10002
Debug: curl_easy_setopt, CURLOPT_URL: ??m
b0 e1 a7 6d 01 
Debug: Exiting curl_easy_setopt
 Inside curl_easy_setopt_int
Debug: Entering curl_easy_setopt, option: 52
Debug: Exiting curl_easy_setopt
 Inside curl_easy_setopt_int
Debug: Entering curl_easy_setopt, option: 13
Debug: curl_easy_setopt, CURLOPT_TIMEOUT: 6134685552
Debug: Exiting curl_easy_setopt
 Inside curl_easy_setopt_int
Debug: Entering curl_easy_setopt, option: 99
Debug: Exiting curl_easy_setopt
 Inside curl_easy_setopt_int
Debug: Entering curl_easy_setopt, option: 78
Debug: Exiting curl_easy_setopt
 Inside curl_easy_setopt_funptr
Debug: Entering curl_easy_setopt, option: 20011
Debug: Exiting curl_easy_setopt
 Inside curl_easy_setopt_ptr
Debug: Entering curl_easy_setopt, option: 10001
Debug: Exiting curl_easy_setopt
STOP Error: curl_easy_perform() failed

It is good to see that the right fortran function gets called in the generic interface.

Something I notice is that two hexdumps of the strings on the C side are exactly the same. If there are two different strings, you would expect some difference, I think. But maybe I'm hexdumping the pointer, and fortran just re-uses the same memory area for passing the next string? The hex does vary between executions.

The CURLOPT_TIMEOUT value also doesn't quite make sense as being the number 10. Which could explain the CURLOPT_TIMEOUT return value @perazz is seeing: #11 (comment)

Maybe a compiler issue? But please take a look at my C code. I might have a made a mistake with pointers.

[HenkPoley]

Hmm, it looks like my call to va_arg() will modify the variadic argument list va_list arg (go to the next one for the next call).

So as a debugging step it's okay right now, but it does influence the code paths that follow.

It would be better to put the debug prints into Curl_vsetopt(), since they already interpret the CURLoption tag and load the variables with va_arg().

[perazz]

I've extracted a minimal test on variadic functions with C/Fortran interoperability and it does seem the problem lies there.

Have two files: cvar.c:

#include <stdio.h>
#include <stdarg.h>

// Simple wrapper for variable argument
int variable_arguments(int datatype, ...) {

   va_list valist;

   /* initialize valist for num number of arguments */
   va_start(valist, datatype);

   switch (datatype)
   {
   case 1:
       {
       int num = va_arg(valist, int);
       printf("INT  argument: %d \n",num);
       break;
       }
   case 2:
       {
       long numl = va_arg(valist, long);
       printf("LONG argument: %ld \n",numl);
       break;
       }
   default:
       {
       printf(" INVALID argument... \n");
       break;
       }
   }

   /* clean memory reserved for valist */
   va_end(valist);

   return 0;
}

and a main Fortran program:

program test_var
    use iso_c_binding
    use iso_fortran_env
    implicit none
    interface
        integer(c_int) function variable_arguments_int(datatype, valist) bind(c,name='variable_arguments')
           import
           integer(c_int), value :: datatype
           integer(c_int), value :: valist
        end function variable_arguments_int
        integer(c_int) function variable_arguments_long(datatype, var) bind(c,name='variable_arguments')
           import
           integer(c_int), value :: datatype
           integer(c_long), value :: var
        end function variable_arguments_long
    end interface

    integer(c_int) :: rc

    rc = variable_arguments_int (1,123_int32)
    rc = variable_arguments_long(2,123_int64)

end program test_var

The expected output is

INT  argument: 123 
LONG argument: 123

I build it with Homebrew gcc as gcc-13 cvar.c main.f90 -lgfortran -lquadmath but what I get is

INT  argument: 1795208544 
LONG argument: 6090175840 

[perazz]

If I instead write type-specific wrappers on the C side, then everything works well:

// Add type-specific interface
int variable_arguments_int(int datatype, int num)
{
    return variable_arguments(datatype, num);
}
int variable_arguments_long(int datatype, long num)
{
    return variable_arguments(datatype, num);
}

And then modifying the Fortran interface as (note different C names)

    interface
        integer(c_int) function variable_arguments_int(datatype, valist) bind(c,name='variable_arguments_int')
           import
           integer(c_int), value :: datatype
           integer(c_int), value :: valist
        end function variable_arguments_int
        integer(c_int) function variable_arguments_long(datatype, var) bind(c,name='variable_arguments_long')
           import
           integer(c_int), value :: datatype
           integer(c_long), value    :: var
        end function variable_arguments_long
    end interface

Now returns the correct output:

federico@Federicos-MBP test_var_fort % gcc-13 cvar.c main.f90 -lgfortran -lquadmath
federico@Federicos-MBP test_var_fort % ./a.out 
INT  argument: 123 
LONG argument: 123 

How can this work fine on Unix but not on macOS?