harris-chris / jot.jl Goto Github PK

View Code? Open in Web Editor NEW

41.0 41.0 6.0 3.42 MB

Streamlines the creation and management of AWS Lambda functions written in Julia

License: MIT License

Julia 99.59% Nix 0.41%

aws julia lambda

jot.jl's People

Stargazers

Watchers

Forkers

jennya-alexeyeva doorisajar potatoasad davibarreira banyan-team sklink

jot.jl's Issues

How does invocation work?

Lambda functions in the AWS documentation and examples have a signature (event, context) => result. The content of the event argument seems to depend on how the function is invoked, e.g. invocations from the AWS command line interface put just the payload in event, whereas if the function is invoked via a Lambda function URL, the event argument contains full information about the HTTP request. (All this subject to disclaimers of "based on what I'm seeing today but I am not an expert" :) )

So, say I use Jot to wrap up a Julia function fun--whether defined in a package or a script or anything else--which takes a single argument of type InputType.

When the resulting Lambda function is invoked, how is the Julia call fun(arg::InputType) constructed? Does this depend on how the function is invoked?
Correspondingly, if one wants to invoke the resulting Lambda, how does the payload need to be structured? Does this depend on whether the invocation is by AWS triggered event (e.g. API gateway), direct function URL, AWS CLI, ... ?

Thanks!

ERROR: Unsatisfiable requirements detected for package Jot [1043c0aa]:

Any ideas why I would run into the error below?
I am on Windows, but I don't think the error is related to the OS.

using GT7CarAlert
using Jot

responder = get_responder(GT7CarAlert, :response_func, Vector{String})
use_pc = false

julia> local_image = create_local_image(responder,julia_base_version="1.6.6",package_compile = use_pc)        
[+] Building 15.8s (14/19)
 => [internal] load build definition from Dockerfile                                                     0.0s 
 => => transferring dockerfile: 1.17kB                                                                   0.0s 
 => [internal] load .dockerignore                                                                        0.0s 
 => => transferring context: 2B                                                                          0.0s 
 => [internal] load metadata for docker.io/library/julia:1.6.6                                          11.8s 
 => [auth] library/julia:pull token for registry-1.docker.io                                             0.0s 
 => [ 1/14] FROM docker.io/library/julia:1.6.6@sha256:9ecd2bd191bc8b0a8e0f0b878f100917199a4654ea9121102  0.0s 
 => [internal] load build context                                                                        0.1s 
 => => transferring context: 144.08kB                                                                    0.1s 
 => CACHED [ 2/14] RUN apt-get update && apt-get install -y   build-essential                            0.0s 
 => CACHED [ 3/14] RUN mkdir -p /var/runtime/julia_depot                                                 0.0s 
 => CACHED [ 4/14] RUN mkdir -p /tmp                                                                     0.0s 
 => CACHED [ 5/14] RUN mkdir -p /var/runtime                                                             0.0s 
 => CACHED [ 6/14] WORKDIR /var/runtime                                                                  0.0s 
 => CACHED [ 7/14] RUN julia -e "using Pkg; Pkg.Registry.add("General"); "                               0.0s 
 => [ 8/14] COPY . .                                                                                     0.1s 
 => ERROR [ 9/14] RUN julia -e "using Pkg; Pkg.develop(path="/var/runtime/GT7CarAlert"); Pkg.instantiat  3.7s 
------
 > [ 9/14] RUN julia -e "using Pkg; Pkg.develop(path="/var/runtime/GT7CarAlert"); Pkg.instantiate(); ":       
#14 0.925    Resolving package versions...
#14 1.730 ERROR: Unsatisfiable requirements detected for package Jot [1043c0aa]:
#14 2.847  Jot [1043c0aa] log:
#14 2.847  ├─Jot [1043c0aa] has no known versions!
#14 2.847  └─restricted to versions * by GT7CarAlert [2a8c1adb] — no versions left
#14 2.847    └─GT7CarAlert [2a8c1adb] log:
#14 2.847      ├─possible versions are: 0.1.3 or uninstalled
#14 2.847      └─GT7CarAlert [2a8c1adb] is fixed to version 0.1.3
#14 2.847 Stacktrace:
#14 2.847   [1] check_constraints(graph::Pkg.Resolve.Graph)
#14 3.271     @ Pkg.Resolve /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Resolve/graphtype.jl:978
#14 3.338   [2] Pkg.Resolve.Graph(versions::Dict{Base.UUID, Set{VersionNumber}}, deps::Dict{Base.UUID, Dict{VersionNumber, Dict{String, Base.UUID}}}, compat::Dict{Base.UUID, Dict{VersionNumber, Dict{String, Pkg.Types.VersionSpec}}}, uuid_to_name::Dict{Base.UUID, String}, reqs::Dict{Base.UUID, Pkg.Types.VersionSpec}, fixed::Dict{Base.UUID, Pkg.Resolve.Fixed}, verbose::Bool, julia_version::VersionNumber)
#14 3.528     @ Pkg.Resolve /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Resolve/graphtype.jl:371
#14 3.529   [3] deps_graph(ctx::Pkg.Types.Context, uuid_to_name::Dict{Base.UUID, String}, reqs::Dict{Base.UUID, Pkg.Types.VersionSpec}, fixed::Dict{Base.UUID, Pkg.Resolve.Fixed})
#14 3.531     @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:541
#14 3.531   [4] resolve_versions!(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec})
#14 3.534     @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:411
#14 3.534   [5] targeted_resolve(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel)
#14 3.536     @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1229
#14 3.536   [6] tiered_resolve(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec})
#14 3.538     @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1215
#14 3.538   [7] _resolve
#14 3.538     @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1235 [inlined]
#14 3.539   [8] develop(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Vector{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
#14 3.590     @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1272
#14 3.591   [9] develop(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; shared::Bool, preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Iterators.Pairs{Symbol, Base.PipeEndpoint, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.PipeEndpoint}}})
#14 3.641     @ Pkg.API /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:149
#14 3.641  [10] develop(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.PipeEndpoint, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
#14 3.661     @ Pkg.API /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:80
#14 3.661  [11] develop(pkgs::Vector{Pkg.Types.PackageSpec})
#14 3.662     @ Pkg.API /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:78
#14 3.662  [12] #develop#14
#14 3.663     @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:75 [inlined]
#14 3.663  [13] develop
#14 3.663     @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:75 [inlined]
#14 3.664  [14] develop(; name::Nothing, uuid::Nothing, version::Nothing, url::Nothing, rev::Nothing, path::String, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
#14 3.667     @ Pkg.API /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:98
#14 3.667  [15] top-level scope
#14 3.668     @ none:1
------
executor failed running [/bin/sh -c julia -e "using Pkg; Pkg.develop(path=\"/var/runtime/GT7CarAlert\"); Pkg.instantiate(); "]: exit code: 1
ERROR: failed process: Process(setenv(`docker build --rm --iidfile id --tag 781406217097.dkr.ecr.eu-central-1.amazonaws.com/gt7caralert:latest .`; dir="C:\\Users\\BERNHA~1.KON\\AppData\\Local\\Temp\\jl_mWNP3Z"), ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error
   @ .\process.jl:525 [inlined]
 [2] run(::Cmd; wait::Bool)
   @ Base .\process.jl:440
 [3] run
   @ .\process.jl:438 [inlined]
 [4] create_local_image(responder::LocalPackageResponder{Vector{String}}; image_suffix::Nothing, aws_config::Nothing, image_tag::String, no_cache::Bool, julia_base_version::String, julia_cpu_target::String, package_compile::Bool, user_defined_labels::OrderedCollections.OrderedDict{String, String}, dockerfile_update::Jot.var"#173#177", build_args::OrderedCollections.OrderedDict{String, String})
   @ Jot C:\Users\bernhard.konig\.julia\packages\Jot\jzEEH\src\Jot.jl:284
 [5] top-level scope
   @ REPL[13]:1

Setting up Lambda with Jot; Error failed process

Thanks for the tip about the dependency argument to get_responder. I've tried so many approaches I forgot to cleanup unneeded dependencies. Now the error I get is:

ERROR: failed process: Process(setenv(`docker build --rm --iidfile id --tag 196490702512.dkr.ecr.us-east-1.amazonaws.com/collectFormLemmas:latest .`; dir="/var/folders/rr/4d86v6hs6273n1ts71_5209c0000gn/T/jl_SX5Pxa"), ProcessExited(125)) [125]

Here's a cleaned up version of my script:

#collectFormLemmas.jl

using Pkg; Pkg.activate(@__DIR__)
Pkg.instantiate(); Pkg.precompile()

using AWS, AWSS3, CodecZlib
using AWS: @service

const awsconfig = AWS.global_aws_config(; region="us-east-1") # pass keyword arguments to change defaults
const udepCCbucket = "commoncrawl"
#const udepCCkeyPrefix = "contrib/depcc/CC-MAIN-2016-07/conll/"
const formlemma_bucket = "commoncrawl-formlemmas"

function collectFormLemmas(fname)
    #@show fname
    m = match(r"(\d\d\d\d\d)\.gz$", fname)  #regex match for five digit file Id
    m === nothing && return 0
    id = m[1]
    parse(Int, id) > 1 && return 0 ##Comment this line when ready to process all files
    formStrings = Dict{String, Nothing}()       # hash table to make dictionary of seen words
    formLemmaStrings = Dict{String, Nothing}()  # hash table for words that have differing lemmas
    s3obj = s3_get(awsconfig, udepCCbucket, fname)
    for tokrow in eachline(GzipDecompressorStream(IOBuffer(s3obj)))  # read a line from a conlldat file
        m = match(r"^\d+\s+(\S+)\s+(\S+)", tokrow)  #regex match to capture 2nd and 3rd columns of tab-delimited data
        m === nothing && continue      #if line has a comment, than ignore it
        form = m[1]                    # first parenthesis group contains word form
        lemma = m[2]                   # second parenthesis group contains word lemma
        formStrings[form] = nothing    # add form to dictionary
        formStrings[lemma] = nothing   # add lemma to dictionary
        form != lemma && (formLemmaStrings[form*' '*lemma] = nothing)  #if form and lemma differ, then add the pair to the form-lemma dictionary
    end
    s3_put(awsconfig, formlemma_bucket, "forms-"*id, join(keys(formStrings), "\n"))
    s3_put(awsconfig, formlemma_bucket, "formLemmas-"*id, join(keys(formLemmaStrings), "\n"))
end

no method matching get_responder_package_name(::Nothing)

I've generated a package following the instructions on Julia and am running an error when attempting to call get_responder using a package. I think it may have to do with missing information in my Project.toml as the get_responder_from_package_spec method seems to be looking for pkg.repo.source

Here's the error:

ERROR: LoadError: MethodError: no method matching get_responder_package_name(::Nothing)
Closest candidates are:
  get_responder_package_name(::String) at ~/.julia/packages/Jot/jzEEH/src/Responder.jl:256
Stacktrace:
 [1] get_responder_from_package_spec(pkg::Pkg.Types.PackageSpec, response_function::Symbol, ::Type{Dict}; registry_urls::Vector{String})
   @ Jot ~/.julia/packages/Jot/jzEEH/src/Responder.jl:56
 [2] get_responder_from_local_package(path::String, response_function::Symbol, ::Type{Dict}; registry_urls::Vector{String})
   @ Jot ~/.julia/packages/Jot/jzEEH/src/Responder.jl:45
 [3] get_responder(path_url::String, response_function::Symbol, response_function_param_type::Type{Dict}; dependencies::Vector{String}, registry_urls::Vector{String})
   @ Jot ~/.julia/packages/Jot/jzEEH/src/Responder.jl:182
 [4] get_responder(path_url::String, response_function::Symbol, response_function_param_type::Type{Dict})
   @ Jot ~/.julia/packages/Jot/jzEEH/src/Responder.jl:166

This is my Project.toml:

name = "HelloWorld"
uuid = "e2d255a6-a9f0-4867-97a6-ac6706df2382"
authors = ["Matt <[email protected]>"]
version = "0.1.0"

[deps]
HiGHS = "87dc4568-4c63-4d18-b0c0-bb2238e4078b"
JuMP = "4076af6c-e467-56ae-b986-b466b2749572"

and my src/HelloWorld.jl:

module HelloWorld

using JuMP
using HiGHS #open source

increment_vector(v::Vector{Int}) = map(x -> x + 1, v)

end # module HelloWorld

Here's the script that's running the get_responder:

import Pkg;
Pkg.add(url="https://github.com/harris-chris/Jot.jl#main");

using Jot

Pkg.activate("./src/HelloWorld")

responder = get_responder("./src/HelloWorld", :increment_vector, Vector{Int})
local_image = create_local_image(responder; image_suffix="increment_vector")
remote_image = push_to_ecr!(local_image)
lambda_function = create_lambda_function(remote_image)

Compilation broken with PackageCompiler 2.x

Sample output from running create_local_image(some_responder; package_compile = true):

Step 20/20 : RUN julia init.jl
 ---> Running in e228ad12679e
    Updating registry at `/var/runtime/julia_depot/registries/General`
   Resolving package versions...
   Installed RelocatableFolders ─ v0.1.3
   Installed Scratch ──────────── v1.1.0
   Installed PackageCompiler ──── v2.0.1
    Updating `/var/runtime/julia_depot/environments/v1.6/Project.toml`
  [9b87118b] + PackageCompiler v2.0.1
    Updating `/var/runtime/julia_depot/environments/v1.6/Manifest.toml`
  [9b87118b] + PackageCompiler v2.0.1
  [05181044] + RelocatableFolders v0.1.3
  [6c6a2e73] + Scratch v1.1.0
Precompiling project...
  ✓ Scratch
  ✓ RelocatableFolders
  ✓ PackageCompiler
  3 dependencies successfully precompiled in 2 seconds (73 already precompiled)
ERROR: LoadError: UndefKeywordError: keyword argument sysimage_path not assigned

If I instead edit the line of init.jl that adds PackageCompiler as follows:

Pkg.add(name="PackageCompiler",version="1.7.7")

And then manually run docker build --rm --iidfile id --tag image path, it works.

Re-running create_local_image makes a new init.jl, though, so that fails.

PackageCompiler 2.x does have breaking changes: https://julialang.github.io/PackageCompiler.jl/dev/index.html#Upgrading-from-PackageCompiler-1.0.

I've used PackageCompiler 1.x before, but I'm not sure what sysimage_path should be set to in this context for compatibility with 2.x. Depending on 1.x by editing [get_init_script](https://github.com/harris-chris/Jot.jl/blob/c5c7eac8b99fb0e998757e5cbe7a5f59c2d40391/src/Scripts.jl#L85) would be a temporary workaround.

Running commands with Sudo?

Hey Harris, thanks a lot for the package. I'm trying to use it, but my docker command requires using sudo, hence I get an error when trying to run the code as a normal user in Julia. I don't want to have to run the whole script as a root user, as this seems unsafe. I wonder if you have any work around?

404 Not Found on runtime invocation after precompilation tests

We've move to use package_compile=true and our build is throwing after running the the Julia tests. Could be to do with the fact that we're running Jot within an Ubuntu Docker and are missing a package.

Error message

value.(c) = [1.0]
     Testing MyPackage tests passed
Starting runtime at http://127.0.0.1:9001/2018-06-01/runtime/invocation/
DEBUG: 2022-11-03T17:04:49.979 3f27 ➡️   "71698432475048495645485445484947114117110116105109101471051101181119997116105111110471101011201163272848480474946491310721111151165832495055464846484649585748484913106599991011121165832424742131085115101114456510310111011658327284848046106108474946564650131067111110116101110116457610111010311610458324813101310" (write)
DEBUG: 2022-11-03T17:04:50.920 3f27 ⬅️   "HTTP/1.1 404 Not Found\r\n" (readuntil)
DEBUG:                                  "Transfer-Encoding: chunked\r\n"
DEBUG:                                  "\r\n"
DEBUG: 2022-11-03T17:04:50.946 3f27 ⬅️   "0\r\n" (readuntil)
DEBUG: 2022-11-03T17:04:50.987 3f27 ⬅️   "\r\n" (readuntil)
ERROR: LoadError: HTTP.ExceptionRequest.StatusError(404, "GET", "/2018-06-01/runtime/invocation/next", HTTP.Messages.Response:
"""
HTTP/1.1 404 Not Found
Transfer-Encoding: chunked

Deploy Script

import Pkg;
Pkg.add(url="https://github.com/harris-chris/Jot.jl#main");
using Jot

cd("./src/MyPackage")

Pkg.activate("./")
Pkg.instantiate()

using MyPackage

# Build the Lambda responder and image --
responder = get_responder(MyPackage, :my_package, Dict)
local_image = create_local_image(responder; image_suffix="my_package", package_compile=true)
remote_image = push_to_ecr!(local_image)
lambda_function = create_lambda_function(remote_image)

Tests

println("Running tests...")

using Test;
import MyPackage;

result = MyPackage.say_hello()

Julia environment management

Say one has a package CoolPackage which contains a function to be wrapped up and package compiled via Jot. Is the intention to:

Add Jot to the CoolPackage environment and run Jot with CoolPackage activated?
Add Jot to the default Julia environment, activate CoolPackage, and because of environment stacking Jot will still be usable?
Create a new environment that contains both Jot and CoolPackage as dependencies and run Jot from there?

Thanks!

(PS - I really appreciate Jot, I was not super looking forward to simultaneously figuring out Docker, Windows Subsystem for Linux, ECR, Lambda container interfaces, etc. for the first time; Jot really seems to simplify greatly.)

ArgumentError: Package Jot does not have Pkg in its dependencies

Trying to follow along with the basic example in the readme/docs, an error is thrown. Here's a full terminal session:


(jotting) pkg> st
Status `~/jotting/Project.toml` (empty project)

julia> julia> import Pkg; Pkg.add(url="https://github.com/harris-chris/Jot.jl#main")
ERROR: UndefVarError: julia not defined
Stacktrace:
 [1] top-level scope
   @ REPL[3]:1

(jotting) pkg> st
Status `~/jotting/Project.toml` (empty project)

julia> import Pkg; Pkg.add(url="https://github.com/harris-chris/Jot.jl#main")
    Updating git-repo `https://github.com/harris-chris/Jot.jl#main`
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `~/jotting/Project.toml`
  [1043c0aa] + Jot v0.1.0 `https://github.com/harris-chris/Jot.jl#main#main`
    Updating `~/jotting/Manifest.toml`
  [a8cc5b0e] + Crayons v4.1.1
  [9a962f9c] + DataAPI v1.14.0
  [e2d170a0] + DataValueInterfaces v1.0.0
  [59287772] + Formatting v0.4.2
⌃ [cd3eb016] + HTTP v0.9.17
  [83e8ac13] + IniFile v0.5.1
  [ceb4388c] + IsURL v0.2.0
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [0f8b85d8] + JSON3 v1.12.0
  [1043c0aa] + Jot v0.1.0 `https://github.com/harris-chris/Jot.jl#main#main`
  [739be429] + MbedTLS v1.1.7
  [bac558e1] + OrderedCollections v1.6.0
  [9b87118b] + PackageCompiler v2.1.5
  [d96e819e] + Parameters v0.12.3
  [69de0a69] + Parsers v2.5.8
  [21216c6a] + Preferences v1.3.0
⌃ [08abe8d2] + PrettyTables v1.3.1
  [189a3867] + Reexport v1.2.2
  [05181044] + RelocatableFolders v1.0.0
  [6c6a2e73] + Scratch v1.2.0
  [66db9d55] + SnoopPrecompile v1.0.3
  [856f2bd8] + StructTypes v1.10.0
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.10.1
  [5c2747f8] + URIs v1.4.2
  [3a884ed6] + UnPack v1.0.2
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [4af54fe1] + LazyArtifacts
  [b27032c2] + LibCURL v0.6.3
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.8.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [fa267f1f] + TOML v1.0.0
  [a4e569a6] + Tar v1.10.1
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll v1.0.1+0
  [deac9b47] + LibCURL_jll v7.84.0+0
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.0+0
  [14a3606d] + MozillaCACerts_jll v2022.2.1
  [4536629a] + OpenBLAS_jll v0.3.20+0
  [83775a58] + Zlib_jll v1.2.12+3
  [8e850b90] + libblastrampoline_jll v5.1.1+0
  [8e850ede] + nghttp2_jll v1.48.0+0
  [3f19e933] + p7zip_jll v17.4.0+0
        Info Packages marked with ⌃ have new versions available and may be upgradable.

julia> open("increment_vector.jl", "w") do f
         write(f, "increment_vector(v::Vector{Int}) = map(x -> x + 1, v)")
       end
53

julia> using Jot

julia> increment_responder = get_responder("./increment_vector.jl", :increment_vector, Vector{Int})
build_dir = "/tmp/jl_lzzvM8"
  Generating  project increment_vector_package:
    increment_vector_package/Project.toml
    increment_vector_package/src/increment_vector_package.jl
  Activating project at `/tmp/jl_lzzvM8/increment_vector_package`
dependencies = String[]
  Activating project at `~/.julia/environments/v1.8`
LocalPackageResponder{Vector{Int64}}("/home/evan/jotting/increment_vector.jl", :increment_vector, Vector{Int64}, "/tmp/jl_lzzvM8", "increment_vector_package", String[])

julia> local_image = create_local_image(increment_responder; image_suffix="increment-vector")
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 5456k  100 5456k    0     0  6807k      0 --:--:-- --:--:-- --:--:-- 6807k
create_env_script = "using Pkg; cd(\"/tmp/jl_lzzvM8\") do;   Pkg.activate(\".\");   Pkg.add(url=\"https://github.com/harris-chris/Jot.jl\", rev=\"main\");   Pkg.develop(PackageSpec(path=\"./increment_vector_package\"));   Pkg.precompile(); end; "
Status `~/.julia/environments/v1.8/Project.toml` (empty project)
Pkg.status() = nothing
ERROR: ArgumentError: Package Jot does not have Pkg in its dependencies:
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have Jot checked out for development and have
  added Pkg as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with Jot
Stacktrace:
 [1] macro expansion
   @ ./loading.jl:1167 [inlined]
 [2] macro expansion
   @ ./lock.jl:223 [inlined]
 [3] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1144
 [4] eval
   @ ./boot.jl:368 [inlined]
 [5] eval
   @ ~/.julia/packages/Jot/p1f6v/src/Jot.jl:1 [inlined]
 [6] create_environment!(responder::LocalPackageResponder{Vector{Int64}})
   @ Jot ~/.julia/packages/Jot/p1f6v/src/Jot.jl:180
 [7] create_local_image(responder::LocalPackageResponder{Vector{Int64}}; image_suffix::String, aws_config::Nothing, image_tag::String, no_cache::Bool, julia_base_version::String, julia_cpu_target::String, function_test_data::Nothing, package_compile::Bool, user_defined_labels::OrderedCollections.OrderedDict{String, String}, dockerfile_update::Jot.var"#209#213", build_args::OrderedCollections.OrderedDict{String, String})
   @ Jot ~/.julia/packages/Jot/p1f6v/src/Jot.jl:334
 [8] top-level scope
   @ REPL[10]:1

julia> versioninfo()
Julia Version 1.8.5
Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 12 × AMD Ryzen 5 1600X Six-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, znver1)
  Threads: 1 on 12 virtual cores

(@v1.8) pkg> st
Status `~/.julia/environments/v1.8/Project.toml` (empty project)

Note that after the error is thrown, the environment has changed to @v1.8. The same error appears when using Julia 1.9.0-rc1. This is on Windows Subsystem for Linux (I got other errors running from straight Windows, put figure the readme just says tested against Linux so that's where I should run.)

Any suggestions? Am I missing something simple here? Thanks!

Using Project.toml while using PackageCompiler

Hi, is there a way to Project.toml to input dependencies instead of defining them in an array?

Dependency requires installing a bin file

We're looking to add CPLEX as a dependency to our package which requires having IBM® ILOG® CPLEX® Optimization Studio installed.

From their installation instructions:

ENV["CPLEX_STUDIO_BINARIES"] = "/opt/CPLEX_Studio1210/cplex/bin/x86-64_linux/"
import Pkg
Pkg.add("CPLEX")
Pkg.build("CPLEX")

My understanding is that we'll need to modify the generated Dockerfile to move the installation file into the Julia image that Jot.jl builds and run the installer in silent mode so that it's available here /opt/CPLEX_Studio1210/cplex/bin/x86-64_linux/ within the image.

Before I go ahead and fork your repo to give that a shot, do you have any suggestions that would allow us to avoid modifying Jot?

fails on `push_to_ecr!` (problem parsing output of `aws ecr describe-repositories`?)

I'm following the first example (here)[https://harris-chris.github.io/Jot.jl/stable/], and I get an issue at step #5, trying to run remote_image = push_to_ecr!(local_image):

julia> remote_image = push_to_ecr!(local_image)
Login Succeeded

...

ERROR: ArgumentError: invalid JSON at byte position 365 while parsing type String: ExpectedOpeningQuoteChar
            "createdAt": 1622815198.0,

I think it's trying to parse the output of aws ecr describe-repositories into a ECRRepo struct, but the types seem not quite compatible: the createdAt value should be some kind of number?

Option to add registries

It would be convenient if there was a place to specify additional Julia package registries that need to be set up as part of the Dockerfile. Something like a dockerfile_add_registries called here might work. An additional_registries argument, which could just be an array of String, could be used to insert lines that look like this for each additional registry:

"julia -e 'using Pkg;Pkg.Registry.add(RegistrySpec(url = \"https://path.to.some.registry.git\"))'"

I only need one additional registry for my use case, but I can imagine a case where someone might want to add more than one -- say, the NREL registry and a private registry.

At the moment, I generate the Dockerfile and modify it by hand, which does successfully build.

Long execution times

We're seeing initial execution times of 60s, while subsequent calls are 20ms. While we had thought this was a Lambda cold start issue, we've set up provisioned concurrency and confirmed with support that provisioned instances are being used:

"...you can see that the latency is equally long when it is provisioned concurrent invocation. Therefore ... the provisioned concurrency is correctly set.

They seem to suggest that there are two separate sections: Lambda handler and core logic.

"We recommend that you separate the code inside the lambda handler so that these long-time logic can be executed in the init phase."

Here's the document they referenced:
https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html#function-code

I still don't understand why we would be seeing subsequent requests coming in at 20ms. Seems like something is being cached and reused. I would assume that having provisioned instances would already have the initialization code ready, but maybe that switch to core logic is required for that step. Any thoughts?

Some more context:

Again, the initial phase of the function does not exist during provisioned concurrency because the initial phase is set in advance. However, if the logic that takes a long time does not run during init duration, it cannot be affected by provisioned concurrency setting.

To sum up, you need to save time by separating the time-consuming initialization logic outside the Lambda handler.

ERROR: Unable to locate created local image

Hello, I apologize if this is a trivial question, but I'm very new to both Julia and Docker, and I'm running into an issue when following the tutorial in the README. I'm importing the package, writing to the file, and creating the responder, but when I try to create the image from the responder, I get an error that says "fatal: not a git repository (or any of the parent directories): .git" Julia then goes through the process of making the file with no other issue until it reaches the end and shows an "ERROR: Unable to locate created local image" message. I've tracked this error down to the get_commit method in Jot.jl, but I'm not sure what's wrong with it, or if I'm doing something wrong. I've tried running it from inside a Github repo, but that didn't work.

Any advice/suggestions for how to resolve this would be great. Thank you very much in advance!

Function versioning

Apologies for continuing to bombard you here!

We're looking at a deploy process where we would publish versions of the Lambda function and point the API to the correct version when we're ready to make the switch.

This means new code would be built and uploaded by Jot, and we would manually publish the version in AWS.

When I ran the Jot deployment today, the previously deployed version on the function disappeared. I couldn't find the log indicating where the function is getting written within the Jot deploy process to check whether the current process is to purge.

We'll work around this by renaming the function in the code before deploying. Just wanted to check in to see whether you had a quick fix on hand.

Thanks!

Errors when following dependencies example

Good afternoon,

I'm rather new to Julia and am trying to follow along in the "To make a script with dependencies into a Lambda function..." example but I ran into issues when including the 1st and 3rd arguments below when calling the "create_local_image" method.

julia> ex1_local_image = create_local_image("ex1", ex1_responder; dependencies=["MySQL", "DBInterface", "DataFrames"])
ERROR: MethodError: no method matching create_local_image(::String, ::LocalPackageResponder{String}; dependencies=["MySQL", "DBInterface", "DataFrames"])

When I make the call only specifying the responder, I am able to move forward but run into a 2nd error

julia> ex1_local_image = create_local_image(ex1_responder)
ERROR: failed process: Process(aws configure get region, ProcessExited(1)) [1]

Where it seems I'm missing aws configuration details. I see where I can pass in details with the 3rd argument in the function definition but I'm unsure exactly what to pass.

Can you help me understand why the 1st snippet fails? It seems that it doesn't know what the dependencies argument is even though it is used in the example. Also, could you help me understand what I would need to pass for aws configuration to prevent the 2nd error?

Thanks!

Read-only file system error on Lambda

I've successfully deployed the function to Lambda but am getting this error when running a test in the AWS Lambda dashboard:

AWS_LAMBDA_RUNTIME_API = 127.0.0.1:9001, running Julia
OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k
ERROR: InitError: AWS_LAMBDA_RUNTIME_API = 127.0.0.1:9001, running Julia
OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k
ERROR: InitError: SystemError: opening file "/var/runtime/julia_depot/logs/scratch_usage.toml": Read-only file system
Stacktrace:
[1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
@ Base ./error.jl:176
[2] #systemerror#80
@ ./error.jl:175 [inlined]
[3] systemerror
@ ./error.jl:175 [inlined]
[4] open(fname::String; lock::Bool, read::Nothing, write::Nothing, create::Nothing, truncate::Nothing, append::Bool)
@ Base ./iostream.jl:293
[5] open(f::Scratch.var"#4#7"{String}, args::String; kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:append,), Tuple{Bool}}})
@ Base ./io.jl:382
[6] track_scratch_access(pkg_uuid::Base.UUID, scratch_path::String)
@ Scratch /var/runtime/julia_depot/packages/Scratch/u24Rl/src/Scratch.jl:174
[7] get_scratch!(parent_pkg::Base.UUID, key::String, calling_pkg::Base.UUID)
@ Scratch /var/runtime/julia_depot/packages/Scratch/u24Rl/src/Scratch.jl:236
[8] macro expansion
@ /var/runtime/julia_depot/packages/Scratch/u24Rl/src/Scratch.jl:302 [inlined]
[9] _scratch_dir
@ /var/runtime/julia_depot/packages/TimeZones/2oXvO/src/TimeZones.jl:34 [inlined]
[10] __init__()
@ TimeZones.TZData /var/runtime/julia_depot/packages/TimeZones/2oXvO/src/tzdata/TZData.jl:20
[11] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any})
@ Base ./loading.jl:831
[12] _tryrequire_from_serialized(modkey::Base.PkgId, path::String, sourcepath::String, depmods::Vector{Any})
@ Base ./loading.jl:938
[13] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt64)
@ Base ./loading.jl:1028
[14] _require(pkg::Base.PkgId)
@ Base ./loading.jl:1315
[15] _require_prelocked(uuidkey::Base.PkgId)
@ Base ./loading.jl:1200
[16] macro expansion
@ ./loading.jl:1180 [inlined]
[17] macro expansion
@ ./lock.jl:223 [inlined]
[18] require(into::Module, mod::Symbol)
@ Base ./loading.jl:1144
during initialization of module TZData
START RequestId: e94387be-016d-4451-a3c5-c1a16eb318b9 Version: $LATEST
RequestId: e94387be-016d-4451-a3c5-c1a16eb318b9 Error: Runtime exited with error: exit status 1
Runtime.ExitError
END RequestId: e94387be-016d-4451-a3c5-c1a16eb318b9
REPORT RequestId: e94387be-016d-4451-a3c5-c1a16eb318b9	Duration: 10262.32 ms	Billed Duration: 10263 ms	Memory Size: 2000 MB	Max Memory Used: 496 MB

So far I've tried to add the --privileged param to docker run. I also tried adding extra permissions to the IAM Role that was created by Jot.jl.

This build is coming from a GitHub runner.

Any suggestions?

Reusing Images

I can't quite tell from the documentation if there's already structures for doing this.

Let's say I have a package that has 5 different functions that I want to turn into Lambda functions. I'm aware that I could have one top level function that switches based on an input parameter, But it's nice to have separate functions with separate identities. However, I prefer not to pay for compiling separate images and storing separate images for each of these functions when all I really need is one image with multiple entry points.

Is that supported in Jot.jl?

harris-chris / jot.jl Goto Github PK

jot.jl's People

Stargazers

Watchers

Forkers

jot.jl's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs