Using FORMAT with a flexi-stream produces wrong format ~T directives about flexi-streams

[avodonosov]

Hi, several questions:

1. What lisp implementation do you use? It might be a bug in gray streams implementation provided by your lisp.
2. What do you mean "discards the formatting", how exactly the output looks?
3. Can you isolate it from hunchentoot? Create a self-contained test-case depending only on flexi-streams,
   where you open a file as binary stream, wrap it with a flexi stream and call format to the flexi stream.

[avodonosov]

I have reproduced it with SBCL and CCL on Windows, using the following test case:

;; Using flexi-streams
(with-open-file (s "/Users/anton/test.txt"
                   :element-type '(unsigned-byte 8)
                   :direction :output
                   :if-does-not-exist :create
                   :if-exists :supersede)
  (let ((stream (flex:make-flexi-stream s))
        (date 'some-date)
        (value 'some-value))
    (format stream "~&Last Update:~12T~A~%" date)
    (format stream "~&Value:~12T~A~%" value)
    (close stream)))

;; without flexi-streams
(with-open-file (s "/Users/anton/test.txt"
                   :element-type 'character
                   :direction :output
                   :if-does-not-exist :create
                   :if-exists :supersede)
  (let ((stream s)
        (date 'some-date)
        (value 'some-value))
    (format stream "~&Last Update:~12T~A~%" date)
    (format stream "~&Value:~12T~A~%" value)
    (close stream)))

[avodonosov]

See CLHS: http://www.lispworks.com/documentation/HyperSpec/Body/22_cfa.htm

I think flexi-streams does not allow FORMAT to determine current absolute column position, so FORMAT just defaults to emitting 2 spaces. Strictly speaking it is not a violation of CLHS, you should expect it as a possible behavior.

I don't know what methods should flexi-streams implement so that it can cooperate with FORMAT in calculation of the column position. If anyone knows, please tell.

[tuscland]

Hello Anton,

1. I use LispWorks 6.1.1 Enterprise 64 on OS X.
2. Instead of formatting items accordingly to the ~12T directive (i.e. varying the number of space character to adjust the column formatting) a constant number of space characters are produced. Exemple:

   LONGKEY2            VALUE2
   
   

Instead of:

KEY1 VALUE1
LONGKEY2 VALUE2

3. Something along that I guess:

````common-lisp
(let* ((binary-stream (flexi-streams:make-in-memory-output-stream))
       (stream (flexi-streams:make-flexi-stream binary-stream)))
  (format stream "~&KEY1~12TVALUE1~%")
  (format stream "~&LONGKEY2~12TVALUE2~%")
  (flexi-streams:octets-to-string
   (flexi-streams::vector-stream-vector binary-stream)))

Cam

[tuscland]

Thank you for reproducing this.
I will read the CLHS link you provided and search for a solution, if any.

[tuscland]

So it seems a FLEXI-OUTPUT-STREAM has all the required harness to track the current column number and standard CL methods to query for this value.

I have the impression the problem lies in STREAM-WRITE-SEQUENCE, in file output.lisp at line 151:
https://github.com/edicl/flexi-streams/blob/master/output.lisp#L151

This method is first called by FORMAT then called again when the characters are encoded into octets. I am not sure the rationale behind setting the column slot to nil in this situation. Commenting the line 151 fixes the problem, but I can't tell for sure that this is the right thing to do.

[tuscland]

Note that on Clozure CL the problem is also present but in a slightly different manner (and more close to what is said in the CLHS link you gave, "the column could not be determined because it is nil, therefore 2 spaces are used"):

Given this:

? (defun test-col ()
  (let* ((binary-stream (flexi-streams:make-in-memory-output-stream))
         (stream (flexi-streams:make-flexi-stream binary-stream)))
    (format stream "~&KEY1~12TVALUE1~%")
    (format stream "~&LONGKEY2~12TVALUE2~%")
    (flexi-streams:octets-to-string
     (flexi-streams::vector-stream-vector binary-stream))))
TEST-COL

;; Here line 151 (setq column nil) is not commented
? (test-col)
"KEY1  VALUE1
LONGKEY2  VALUE2
"

;; Redefine STREAM-WRITE-SEQUENCE, commenting line 151, and yielding the expected result
#<STANDARD-METHOD TRIVIAL-GRAY-STREAMS:STREAM-WRITE-SEQUENCE (FLEXI-STREAMS:FLEXI-OUTPUT-STREAM T T T)>

? (test-col)
"KEY1        VALUE1
LONGKEY2    VALUE2
"

[avodonosov]

Few lines above, at line 122 method stream-write-byte also sets 'olumn to nil.

The comment says "set column to NIL because we don't know how to handle binary output mixed with character output".

I suppose method stream-write-sequence uses the same reason.

And the bug is that doing so the stream-write-sequence code does not expect to be called from other flexi-streams which handles character output initiated by user. I.e. user performs character output, not binary output, but flexi-streams clears column as if user has output some bytes.

[tuscland]

Anton, I don't think this is the reason because the same problem occurs when using the stream returned by `hunchentoot:send-headers', which is a chunga binary stream.

[avodonosov]

Camille, I do not understand what you mean. You anyway wrap chunga stream into a flexi strream, because you can not format int a binary chunga stream, right?

[tuscland]

Yes this is correct.
Sorry, perhaps I misunderstood what you said:

And the bug is that doing so the stream-write-sequence code does not expect to be called from other flexi-streams which handles character output initiated by user. I.e. user performs character output, not binary output, but flexi-streams clears column as if user has output some bytes.

I understand from your description that the bug is the result of two flexi-streams writing into each other, like in the short test example I gave. Is that what the right meaning?

[avodonosov]

I think my comment above was not clear, I also made a typo there.

What I mean is stream-write-sequence code does not expect to be called from other code which handles character output on the same stream.

I.e. the (setq column nil) is intended for the case when user calls (flex:write-sequence flexi-stream <byte-array>) because there is no sensible way to adjust column after binary output operation.

But the code does not anticipate the case when user calls (cl:format flexi-stream <some-characters>) which calls (flex:stream-write-sequence flexi-stream <some-characters>) which calls (flex:stream-write-sequence flexi-stream <byte-array>).
I.e. the byte output is performed not by user, but as a part of character output implementation.

[avodonosov]

I think I know how to fix it, see the commit I done in my fork: avodonosov@958fa0e

But need to run tests before issuing a merge request.

Camille, anyway, in your application you probably will be safer to implement padding yourself, as CLHS does not guarantee FORMAT to be able to do padding on all streams.

[tuscland]

Nice solution! Indeed it is so much more logical to output raw binary to the underlying stream. I hope this is the good one.

With regards to padding and FORMAT, I am not looking for a generic solution, so as long as it works I'm happy with it. I was just surprised to see flexi-streams had support for column numbers, but was not using it.

Thank you!