logiccomp / lsl

Logical Student Language (LSL)

lsl's Issues

Make generator for -> be pure?

Right now, the generator for -> generates functions that generate an example of their output every time they are called. This means they are not pure functions, which feels odd given that we are introducing them before we introduce mutation. On the other hand, making them be pure (and not just constant functions) seems... harder? (hash the input, have some deterministic generator?).

start-gui window displaying outside the screen in 1080p resolution screens

The window that the start-gui function shows have a part displaying out the screen in Windows with 1080p monitors and 125% scale. The window cannot be resized smaller, making it impossible to click on the buttons bellow.

Improve error messages for existentials

The example given in the test suite mentions that it gets a sealed value, but doesn't mention the existential at all, which I imagine will be confusing. Can we store some metadata with the sealed value so when we print the error we can say a little bit about it (ideally highlighting where it was sealed).

More confusingly, if you generate two copies of the same existential (which I'm sure people will do!), and try to use one with the other, it goes wrong (good!) but I imagine the error will be quite hard to reason through.

e.g., like this:

(: mk-counter-pkg (-> (Exists (A) (Tuple (-> A) (-> A A) (-> A Integer)))))
(define (mk-counter-pkg)
  (list (λ () 0)
        (λ (x) (+ x 1))
        (λ (x) x)))

(define COUNTER0 (mk-counter-pkg))
(define COUNTER1 (mk-counter-pkg))
(define make-counter (first COUNTER0))
(define counter-incr (second COUNTER1))

(counter-incr (make-counter))

This produces, as an error:

. . mk-counter-pkg: contract violation
  expected: A
  given: #<seal>
  blaming: anonymous-module (as client)

(and highlights the A argument to the second field of the Tuple -- i.e., the argument to the counter-incr)

"size" argument for generators, shrinking?

I'm wondering if we want to have the ability to have check-contract go from small values to larger values, rather than using the same function to generate values.

Similarly, should we have a way to shrink values? Optional, defaulting to being absent?

ticks should increment all parent parameters

Currently, string=? counts only increment the closest parameter. Probably can misuse parameter guards to do this.

Figure out how to express "timing" for constant time assignment(s)

We had an assignment last year, that I'd like to do in some form:

Here in the current version: (not a particularly stable url)
https://courses.dbp.io/logics24/hw3a.html#(part._.Problem_3)

Essentially, they are given a not constant time password checking function, are asked to write a spec that verifies that, and then fix the code. Because timing really requires sampling, and I didn't have an easy way to do that, I simulated the time with a "clock tick" mutable ref. This meant, of course, they had to use specific functions that incremented that, but seemed to work okay.

Either we should figure out a better way of doing this (@camoy mentioned using allocations as a proxy for time, or really, change the problem to constant space; wonder if that still requires sampling), which maybe means having some ability to sample?

Or should just build in the same functionality as I had last year.

OneOf not working with Constant

> (contract-generate (Constant 0))
0
> (contract-generate (OneOf (Constant 0) (Constant 1)))
#<procedure:...untime/contract.rkt:137:0>

(I was trying to defineBit)

check-contract on functions that may error?

Some functions might have cases that are unrecoverable. e.g., eval on arithmetic, division by zero.

How should we handle this? erroring should be an acceptable possible result (at least, certain errors)...

Error messages (or something) going on with `verify-contract`

So this involves strings, so I expect it not to work (unless Rosette can do some magic and figure out they are constants, but...), but the error is not helpful:

(: letter-grade (-> Integer String))
(define (letter-grade n)
  (cond [(>= n 90) "A"]
        [(>= n 80) "B"]
        [(>= n 70) "C"]
        [(>= n 60) "D"]
        [else "F"]))
(check-contract letter-grade)

(: letter-grade-prop (-> Integer True))
(define (letter-grade-prop n)
  (let ([l (letter-grade n)])
    (member? l (list "A" "B" "C" "D" "F"))))
(check-contract letter-grade-prop)

(verify-contract letter-grade-prop)

Produces:

--------------------
unit tests > Unnamed test
FAILURE
name:       verify-contract
location:   /Users/dbp/code/lsl/lsl-lib/main.rkt:371:19
params:     '(#<procedure:...sl/lsl-lib/main.rkt:371:36>)

letter-grade-prop: contract violation
  expected: True
  given: #f
  blaming: /Users/dbp/teaching/logics24/website/lectures/6.rkt
--------------------
2 success(es) 1 failure(s) 0 error(s) 3 test(s) run

Rename "Flat" to "Immediate"

As we discussed, Flat is not a particularly intuitive name; Immediate might be better. This is in contrast with contracts, like Function and (mutable) Structs that all have to be delayed.

I think we should make this naming change and release it sometime this week; next week (middle of the week) we'll introduce them to creating contracts from scratch, so need to have done it by then if we are going to.

verify-contract not showing counter-examples

With the following example:

(: bad-mult (-> Real Real Real))
(define (bad-mult x y)
  (if (= x 10417)
      0
      (* x y)))
(check-contract bad-mult)

(: bad-mult-prop (-> Real Real True))
(define (bad-mult-prop x y)
  (= (bad-mult x y)
     (* x y)))
(check-contract bad-mult-prop)

(verify-contract bad-mult-prop)

Rosette correctly identifies that this is buggy, but doesn't give us the right counter-example (seems like it is printing the output?)

--------------------
unit tests > Unnamed test
FAILURE
name:       verify-contract
location:   /Users/dbp/code/lsl/lsl-lib/main.rkt:371:19
params:     '(#<procedure:...sl/lsl-lib/main.rkt:371:36>)

verification failure
  expected: True
  counterexample: #f
  blaming: /Users/dbp/teaching/logics24/website/lectures/6.rkt
--------------------
2 success(es) 1 failure(s) 0 error(s) 3 test(s) run

Special list contract don't seem to be generating correctly

Here's a minimal example:

(: f (-> (List Integer) True))
(define (f lon) #t)
(check-contract f)

This fails due to:

f: contract violation
  expected: (List Integer)
  given: '(#<procedure> ...

It seems like the List contract isn't generating its elements correctly.

Parametric signatures?

We have parametric contracts, though they have to be made concrete on use. Is it feasible to add contract variables? i.e., I can write:

(define-contract (Maybe T) (OneOf T (Constant #f))))

Which is great, but I can't (as far as I know!), write:

(: map (-> (-> X Y) (List X) (List Y)))
...

Recursive contract expanding infinitely?

Just defining this:

(define-contract SExp (OneOf Symbol
                             Integer
                             Boolean
                             (List SExp)))

Works fine (using, as I understand, the stuff you did to autodetect the recursive binding).

But trying to use it:

(: f (-> SExp True))
(define (f s) #t)

Seems to infinite loop (or at least, OOMs..)

Be able to change how many tests `check-contract` runs

Simplest might be an extra argument? i.e., (check-contract foo) does default, (check-contract N foo) generates N sets of inputs?

generated values should satisfy contract

If a generator fails its contract, we should give an informative contract error saying that the contract itself is to blame because it generated a bad value.

Check-contract on trees?

The following:

(define-struct leaf [value])
(define-struct node [left right])
(define-contract (Tree X) (OneOf (Leaf X) (Node (Tree X) (Tree X))))

(: height (-> (Tree Integer) Natural))
(define (height t)
  (cond [(leaf? t) 0]
        [(node? t) (add1 (max (height (node-left t)) (height (node-right t))))]))
(check-expect (height (make-leaf 0)) 0)
(check-expect (height (make-node (make-leaf 0) (make-node (make-leaf 0) (make-leaf 0)))) 2)

Works.

But this:

(check-contract height)

Errors.

top-interactions should reset vc

Ideally, #%top-interaction should be wrapped in with-vc to reset the VC in case of a Rosette error that sets the VC as a contradiction. However, this isn't so easy because we want the top-level to support definitions and with-vc creates a definition context. Sadly, there is no splicing form of with-vc right now. We will need to hack something up custom.

Allow `check-contract` to work at interactions

Right now, check-contract adds a test to be run with rackunit, which works great when you have it in the definitions and then run the file. It would be nice if it could also work interactively? I assume there is some context that the interactive prompt runs in that would allow the macro to detect that and immediately run-tests (it would be extra nice if this also worked with scribble/example...)

No more blame highlighting

I think with the switch to making check-contract report to rackunit, we lost error highlighting (or maybe it happened earlier?). With this example:

(: f (-> Integer Integer))
(define (f x)
  #f)
(check-contract f)

I get this in the interactions window:

But nothing highlighted in the definitions window.

(also, the location of the test seems off too: seems to be definition site for check-contract, rather than use site).

Especially since the blame only is at the level of the file, seems important to have the highlighting!

Fall-through predicate case for define-contract doesn't work for contracts with parameters

(define Foo (lambda (x) ...) works, but (define (Foo x) (lambda (y) ...)) errors with literal clause must occur within Flat; obviously wrapping the lambda with Flat works, but would be nice to have the shorthand.

check-satisfied error when arguments swapped links to location within rackunit

Here's a reproduction:

#lang lsl

(define (f x) #t)

(check-satisfied f #t)

The error message is... reasonable (seems like it could, in theory, be better, but it's not too bad), but the srcloc (the red X that is linked) is messed up for some reason....

Improve error messages for primitive functions from Rosette

Rosette has pretty bad messages for at least some functions; e.g., if i pass the wrong value to max:

(max #t #f)

I get the following error:

. . [assert] curried:extreme: expected real? arguments
  arguments: (#t)

Maybe we can wrap (reasonably named) contracts around them in the process of re-exporting them?

Have LSL have a regular release cadence (weekly, biweekly) and print a warning when running "out of date" version

If we hardcoded, in the code, a date after which the code should be updated, we should be able to have it print out a warning in the interactions window that the students should update. As long as we fix a release cadence, this seems like it might be easier to ensure students are regularly updating if they get actual feedback every time they run a program using an out of date version.

(This depends on their computer date being accurate within a day or two, but that seems like a fine assumption).

Contract violations on trace contracts a little confusing

I have a simple example:

(define-contract UniqueList (lambda (l) (equal? (length l) (length (remove-duplicates l)))))
(: ids UniqueList)
(define ids empty)

(: maybe-unique (-> (And Natural (Record ids))))
(define (maybe-unique)  (random 1000))

(check-contract maybe-unique 1000)

And it, of course, fails the contract, but the error doesn't say anything about UniqueList (which is actually the contract that was violated):

--------------------
--
local tests > Unnamed test
FAILURE
name:       check-contract
location:   eval:10:0
params:     '(#<procedure:.../lsl-lib/no-gui.rkt:416:38>)
 
maybe-unique: contract violation
expected: (And Natural (Record ids))
given: 947
counterexample: (#f)
blaming: top-level
--------------------
0 success(es) 1 failure(s) 0 error(s) 1 test(s) run

Have `check-contract` errors show inputs when outputs fail contract?

Say I have a function like:

(: my-prop (-> (List Integer) True))
(define (my-prop loi)
   (= (length loi) 1))

And I run (check-contract my-prop). If it finds a counter-example, the error just reports that the contract was violated because #f was received when True was expected.

e.g., an error like:

--------------------
module-level tests > Unnamed test
. FAILURE
name:       check-contract
location:   48-unsaved-editor:7:0
params:     '(#<procedure:.../lsl-lib/no-gui.rkt:409:38>)

my-prop: contract violation
  expected: True
  given: #f
  blaming: anonymous-module
--------------------
0 success(es) 1 failure(s) 0 error(s) 1 test(s) run

Like with verify-contract, we should see the inputs that were generated when we see the failure.

Allow structs to be mutable

I thought this was just a matter of adding #:mutable, but this is causing a test failure (good!), so obviously there is something going on. I get why this could cause problems (mutability certainly changes how equality can work!), but not sure how to fix it :)