joachifm / cl-webkit Goto Github PK

Ref AeroNotix/lispkit#7

http://webkitgtk.org/reference/webkit2gtk/stable/WebKitWebContext.html#webkit-web-context-allow-tls-certificate-for-host

I believe it is.

If you think so too, I can put in the (small amount of) legwork to get it there.

http://webkitgtk.org/reference/webkit2gtk/stable/webkit2gtk-WebKitUserContent.html

http://webkitgtk.org/reference/webkit2gtk/stable/WebKitWebView.html#webkit-web-view-load-bytes

Currently, cc-flags is hard-coded in grovel.lisp.
We should use the output of pkg-config --cflags webkitgtk-3.0 instead.

While all the custom URL handling is happening on a separate thread asynchronously, any condition raised there segfaults the whole process. This should not happen, given that we use handler-case there. Should we also add a case for conditions? How do we ensure stability of erroring callbacks?

http://webkitgtk.org/reference/webkit2gtk/stable/WebKitWebView.html#WebKitWebView--user-content-manager

I'm working on a project where I need to communicate between the web page and CL code. I noticed that I can't hook up script message handlers with this library because webkit_user_content_manager_register_script_message_handle is misspelled in the API and should be webkit_user_content_manager_register_script_message_handler (note the R at the end).

http://webkitgtk.org/reference/webkit2gtk/stable/WebKitNavigationPolicyDecision.html#WebKitNavigationPolicyDecision--navigation-action

It freezes when trying to play a flash video (tested on SBCL and CCL), the equivalent C program works.

When running the test browser (test-browser-main) I get these warnings in terminal:

(process:53159): GLib-Net-WARNING **: 11:11:42.656: ../tls/gnutls/gtlscertificate-gnutls.c:145: invalid property id 3 for "private-key" of type 'GParamBoxed' in 'GTlsCertificateGnutls'

Versions of my software:

• gnutls: 3.6.16
• webkit: 2.34.0
• glib-networking: 2.66.0

Do you think this is a bug in Common Lisp wrapper or in my system?

We've got a bunch of poorly written, hard-coded callbacks in our codebase.
Today I discovered that cl-webkit has a generic way to handle callbacks. It's in the (unused!) callback.lisp file.

I was thus able to do this (atlas-engineer/nyxt@4e2ecdf#diff-9df864619800a42d41380418e1c5af907bdcc9c453e5ae1ce003951a86c0aed6R1824):

(let ((result-channel (make-channel 1)))
        (run-thread "WebKitGTK cookie-policy"
          (within-gtk-thread
            (let* ((context (webkit:webkit-web-view-web-context (gtk-object buffer)))
                   (cookie-manager (webkit:webkit-web-context-get-cookie-manager context)))
              ;; TODO: Update upstream to export and fix `with-g-async-ready-callback'.
              (webkit::with-g-async-ready-callback (callback
                                                     (declare (ignorable webkit::user-data webkit::source-object))
                                                     (calispel:! result-channel
                                                                 (webkit:webkit-cookie-manager-get-accept-policy-finish
                                                                  cookie-manager
                                                                  webkit::result)))
                (webkit:webkit-cookie-manager-get-accept-policy
                 cookie-manager
                 (cffi:null-pointer)
                 callback
                 (cffi:null-pointer))))))
        (calispel:? result-channel))

Benefits:

• No more need for a top-level callback definition.
• No more need to push / pop into a global *callbacks* list.

(By the way, did you know that this strategy is used by CFFI under the hood to implement defcallback? So here with cl-webkit we reinvent the wheel, but poorly.)

To do:

• Export with-g-async-ready-callback
• Fix define-g-async-ready-callback to declare all arguments as ignorable, otherwise we quickly get warnings.
• Find a way to pass the result symbol as parameter to with-g-async-ready-callback, otherwise the only way to get the result from the callback is to refer to the hard-coded webkit::result name, which ain't pretty.

Thoughts?

@aartaka

Define type-translators wherever necessary, so that you can pass
keyword syms instead.

• For all properties we care about, add a slot to the class definition & expose
  the symbol (ala f465dd0).
  Remove existing bindings to get/set methods for which there is a
  corresponding property.
• Remove remaining vestiges of the initial use of raw points
• Remove -class prefix from webkit classes

Commit 639c6cb introduced a dependency on alexandria but that change was not propagated to cl-webkit2.asd.

Since the only symbol used is alexandria:with-gensyms, I'd suggest adding this macro to the codebase instead of dragging the whole library.

CC @aartaka

WebKit 2.6 bumps the .so to 4.0.0 and removes WebKitWebGroup.
It seems not many distros ship 2.6 quite yet, so we might want to maintain
backwards compatibility with 2.4.5 for a while.
Before beginning to work on this we need to be able to do condtional compilation
based on the currently loaded webkit version.

In commit 203bada we've removed support for libwebkit2gtk-5.0 (webkitgtk-next in Guix) because cl-webkit fails to compile against it.

In fact, CFFI fails to load it at all. Any idea what's going on?

Current a Nyxt image cannot reload cl-webkit because it uses defpackage without the complete :export list.
This is particularly annoying when working on cl-webkit either from a Nyxt image, or simply because we can't reload the system.

The easy fix is to switch to uiop:define-package. OK to switch?

Hi @joachifm, when I modify your #'simple-browser-main function as follows and run it (after loading the cl-webkit2 and simple-browser.lisp libraries) the browser opens, gives the alert, and then freezes as soon as I click on the alert (which does not close).

If I then tab off the alert to the browser, the browser grays out and is unusable (even a window manager kill command will not close it).

The modified function:

(defun simple-browser-main ()
  "A single-window browser with no keyboard or mouse input.                                                                    
Loads and renders a single web page."
  (gtk:within-main-loop
    (let ((win (make-instance 'gtk:gtk-window))
          (view (make-instance 'webkit2:webkit-web-view)))
      (gobject:g-signal-connect win "destroy"
                                #'(lambda (widget)
                                    (declare (ignore widget))
                                    (gtk:leave-gtk-main)))
      (gtk:gtk-container-add win view)
      (webkit2:webkit-web-view-load-uri view "http://www.example.com")
      (webkit2:webkit-web-view-run-javascript
       view
       "alert(1);"
       (cffi:null-pointer)
       (cffi:null-pointer)
       (cffi:null-pointer))
      (gtk:gtk-widget-show-all win))))

Please also find my own repository I wrote in C to make sure this was an issue with the cl-webkit bindings and not the underlying webkit2 libraries.

https://github.com/ahungry/puny-browser/blob/master/main.c

I'm running Arch Linux and it uses the 4.0.so file for webkit2.

According to webkitgtk-2.34.1/Source/cmake/OptionsGTK.cmake,
when version of libsoup >= 2.99.9, WebKitGTK 2.34.1, compiles to libwebkit2gtk-4.1.so,
and there is no API changes in 4.1.

if (USE_SOUP2)
    set(SOUP_MINIMUM_VERSION 2.54.0)
    set(SOUP_API_VERSION 2.4)
else ()
    set(SOUP_MINIMUM_VERSION 2.99.9)
    set(SOUP_API_VERSION 3.0)
    set(ENABLE_SERVER_PRECONNECT ON)
endif ()
find_package(LibSoup ${SOUP_MINIMUM_VERSION})

if (NOT LibSoup_FOUND)
if (USE_SOUP2)
    message(FATAL_ERROR "libsoup is required.")
else ()
    message(FATAL_ERROR "libsoup 3 is required. Enable USE_SOUP2 to use libsoup 2 (disables HTTP/2)")
endif ()
endif ()

if (USE_GTK4)
    set(WEBKITGTK_API_VERSION 5.0)
    set(WEBKITGTK_API_DOC_VERSION 5.0)
elseif (USE_SOUP2)
    set(WEBKITGTK_API_VERSION 4.0)
    set(WEBKITGTK_API_DOC_VERSION 4.0)
else ()
    set(WEBKITGTK_API_VERSION 4.1)
    # No API changes in 4.1, so keep using the same API documentation.
    set(WEBKITGTK_API_DOC_VERSION 4.0)
endif ()

Building with SBCL 2.0.5 / ASDF 3.3.1 for quicklisp dist creation.

Trying to build commit id 094cbcb

cl-webkit2 fails to build with the following error:

Unhandled UNBOUND-VARIABLE in thread #<SB-THREAD:THREAD "main thread" RUNNING {1000A18083}>: The variable WEBKIT-WEB-PROCESS-TERMINATION-REASON is unbound.

Full log here

It compiles but the demo just... does nothing. No error messages or output. Any ideas?

Writing low-level bindings is error prone, boring, and generally a waste of time. Use gobject-introspection instead.

For example, to iterate over all the properties of WebKitSettings, do something like

#include <girepository.h>

int main(void) {
  GError* err;
  GITypelib* WebKit2 = g_irepository_require(NULL, "WebKit2", NULL, 0, &err);
  if (!WebKit2) g_error("fatal: %s\n", err->message);

  GBaseInfo* info = g_irepository_find_by_name(NULL, "WebKit2", "Settings");
  if (!info) g_error("fatal: Settings not found in the WebKit2 namespace!\n");
  GIPropertyInfo* prop;
  for (int i = 0; i < g_object_info_get_n_properties(info); ++i) {
    prop = g_object_info_get_property(info, i);
    g_print("%s  %s  %d\n", g_base_info_get_name(prop),
      g_type_tag_to_string(g_type_info_get_tag(g_property_info_get_type(prop))),
      g_property_info_get_flags(prop));
  }
}

From this one might generate a suitable lisp class, preferably via a generator program to avoid run-time overhead from introspection.

Use groveller for constants defined in https://github.com/joachifm/cl-webkit/blob/master/webkit2/webkit2.web-view.lisp

http://webkitgtk.org/reference/webkit2gtk/stable/WebKitURIResponse.html#WebKitURIResponse--http-headers

Previously the library supported Webkit1, I swtiched the library to support Webkit2. We should either support both (simple :or on the library lookup) or remove webkit1 bindings.

http://webkitgtk.org/reference/webkit2gtk/stable/WebKitDownload.html#WebKitDownload--allow-overwrite

I don't have a mac or use OSX but I had a bug report on IRC for Lispkit that mentioned that cl-webkit didn't seem to properly find the libraries for OSX.

Instead of having callers do certain conversions (pathname->string, for example) we could have functions take properly handled and semantic types using type translators.

http://common-lisp.net/project/cffi/manual/html_node/Foreign-Type-Translators.html#Foreign-Type-Translators

Just a heads-up as it may bite in the future.

https://webkitgtk.org/reference/webkit2gtk/stable/method.WebView.run_javascript_in_world.html

diff --git a/webkit2/webkit2.web-view.lisp b/webkit2/webkit2.web-view.lisp
index 6c1ab66..d17ec5c 100644
--- a/webkit2/webkit2.web-view.lisp
+++ b/webkit2/webkit2.web-view.lisp
@@ -233,6 +233,7 @@ defcfun "webkit_web_view_get_is_muted"
 #+webkit2-mute
 (export 'webkit-web-view-get-is-muted)
 
+;; Deprecated since 2.40, use webkit_web_view_evaluate_javascript instead.
 (defcfun "webkit_web_view_run_javascript" :void
   (web-view (g-object webkit-web-view))
   (script :string)
@@ -241,6 +242,7 @@ defcfun "webkit_web_view_run_javascript"
   (user-data :pointer))
 (export 'webkit-web-view-run-javascript)
 
+;; Deprecated since 2.40, use webkit_web_view_evaluate_javascript instead.
 (defcfun "webkit_web_view_run_javascript_in_world" :void
   (web-view (g-object webkit-web-view))
   (script :string)
@@ -303,6 +305,7 @@ defun webkit-web-view-evaluate-javascript
        (cffi:make-pointer callback-counter))))
 (export 'webkit-web-view-evaluate-javascript)
 
+;; Deprecated since 2.40, use webkit_web_view_evaluate_javascript_finish instead.
 (defcfun ("webkit_web_view_run_javascript_finish" %webkit-web-view-run-javascript-finish) webkit-javascript-result
   (web-view (g-object webkit-web-view))
   (result g-async-result)
@@ -313,6 +316,7 @@ defun webkit-web-view-run-javascript-finish
     (%webkit-web-view-run-javascript-finish web-view result err)))
 (export 'webkit-web-view-run-javascript-finish)
 
+;; Deprecated since 2.40, use webkit_web_view_evaluate_javascript instead.
 (defcfun "webkit_web_view_run_javascript_from_gresource" :void
   (web-view (g-object webkit-web-view))
   (resource :string)
@@ -321,6 +325,7 @@ defcfun "webkit_web_view_run_javascript_from_gresource"
   (user-data :pointer))
 (export 'webkit-web-view-run-javascript-from-gresource)
 
+;; Deprecated since 2.40, use webkit_web_view_evaluate_javascript_finish instead.
 (defcfun ("webkit_web_view_run_javascript_from_gresource_finish" %webkit-web-view-run-javascript-from-gresource-finish) webkit-javascript-result
   (web-view (g-object webkit-web-view))
   (result g-async-result)
@@ -531,6 +536,7 @@ defun webkit-web-view-can-execute-editing-command
    (cffi:make-pointer callback-counter)))
 (export 'webkit-web-view-can-execute-editing-command)
 
+;; Deprecated since 2.22, use jsc_value_get_context instead.
 (defcfun "webkit_web_view_get_javascript_global_context" js-global-context-ref
   (web-view (g-object webkit-web-view)))
 (export 'webkit-web-view-get-javascript-global-context)
@@ -543,11 +549,12 @@ defcfun "webkit_javascript_result_unref"
   (js-result webkit-javascript-result))
 (export 'webkit-javascript-result-unref)
 
+;; Deprecated since 2.22, use jsc_value_get_context instead.
 (defcfun "webkit_javascript_result_get_global_context" js-global-context-ref
   (js-result webkit-javascript-result))
 (export 'webkit-javascript-result-get-global-context)
 
-;; Deprecated, use webkit-javascript-result-get-js-value instead.
+;; Deprecated since 2.22, use webkit_javascript_result_get_js_value instead.
 (defcfun "webkit_javascript_result_get_value" js-value-ref
   (js-result webkit-javascript-result))
 (export 'webkit-javascript-result-get-value)

@joachifm, it seems that destroying the web-view object screws something up, requiring a complete reload. Not destroying the webview (removing it from the window container and abandoning it - or reusing it for next time) makes things work again.

I know it's an old project, but could you clarify how to manage the lifespan of a web-view properly?

http://webkitgtk.org/reference/webkit2gtk/stable/WebKitUserContentManager.html

Building with SBCL 2.2.11.160-65fe8bb77 / ASDF 3.3.5 for quicklisp dist creation.

Trying to build commit id d60d434

cl-webkit2 fails to build with the following error:

; caught ERROR:
;   READ error during COMPILE-FILE: Symbol "G-VARIANT-GET-TYPE" not found in the GLIB package. Line: 25, Column: 59, File-Position: 796 Stream: #<SB-INT:FORM-TRACKING-STREAM for "file /home/quicklisp/quicklisp-controller/dist/build-cache/cl-webkit/4167ad8cd968969185f1b3a05808b43a8fa0b9c5/cl-webkit-20221227-git/webkit2/util.lisp" {101F5AFBC3}>
...
Unhandled UIOP/LISP-BUILD:COMPILE-FILE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING {1001710003}>: COMPILE-FILE-ERROR while compiling #<CL-SOURCE-FILE "cl-webkit2" "util">

cl-webkit2/test fails to build with the following error:

Unhandled SIMPLE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING {1001710003}>: No package named "CFFI"

Full log here

http://webkitgtk.org/reference/webkit2gtk/stable/WebKitNavigationAction.html

Map header key to list of header values.

I've seen glib:with-g-error macro being used in webkit2/webkit2.web-view.lisp, but it's not present in recent commits, even though there is code referencing g-errors.

Should this macro be used everywhere g-error is used? It seems to be quite useful for debugging.

Adding to that, how do we represent g-errors? Is it plain :pointer or (:pointer (:struct glib:g-error)) constuction?

Demo refers to a webkit:webkit-website-data-manager that is not available anymore. Is there a fix?

Hi @joachifm, would it be possible to give commit access to @aartaka?
He's been working with @jmercouris and me on Nyxt and cl-webkit for a while now, we trust he is going to be a good maintainer!

Cheers!

Indeed, the only change that landed is rather small - bcf97b8.

The reason why I'm pushing for a new release is that I'm growing tired of asking Nyxt users which version of WebKitGTK they're using. They usually share the output of nyxt --system-information, which would render the question useless.

Thanks.

Arch Linux uses libwebkit2gtk-4.0.so, while this looks for a max of 3.0 - is it possible to support 4.0?

Calling webkit-web-view-evaluate-javascript over a snippet that returns a value that cannot be coerced to string raises an error.
Try evaluating document; to see what I mean.

The code in javascript-evaluation-complete is outdated, we can see it was taken from https://webkitgtk.org/reference/webkit2gtk/2.5.1/WebKitWebView.html#webkit-web-view-run-javascript-finish, while the latest doc suggests https://webkitgtk.org/reference/webkit2gtk/stable/WebKitWebView.html#webkit-web-view-run-javascript-finish.

I suggest we do the following:

• Test if the return value is a string. If so, convert to string as we currently do.
• If not, pass the pointer directly to the callback.

What about other atomic types like numbers?
@jmercouris @aartaka Thoughts?

See also #52.

The code example in the documentation for webkit_web_view_run_javascript_finish () has a useful idiom in it:

exception = jsc_context_get_exception (jsc_value_get_context (value));
if (exception)
    g_warning ("Error running javascript: %s", jsc_exception_get_message (exception));
else
    g_print ("Script result: %s\n", str_value);
g_free (str_value);

Exposing JS error messages can be useful in debugging bigger things, so we should probably use jsc_context_get_exception in javascript-evaluation-complete callback. To do so:

• Bind jsc_context_get_exception.
• Bind jsc_exception_get_message.
• Bind jsc_value_get_context.
• (optional) Bind other useful methods of JSCException for better error reporting.
• Maybe get rid of (webkit-javascript-result-get-global-context js-result) in javascript-evaluation-complete.

How does that sound?

EDIT: Unmatched parentheses O_o

C-style namespacing is unnecessary and redundant.