gap-packages / io

GAP package IO to do input and output

README file for the IO GAP4 package

To get the newest version of this GAP 4 package download the archive file

io-x.x.tar.gz

and unpack it using

gunzip io-x.x.tar.gz; tar xvf io-x.x.tar

Do this in a directory called pkg, preferably (but not necessarily) in the pkg subdirectory of your GAP 4 installation. It creates a subdirectory called io.

To install this package do

cd io
./configure

If you installed io in another directory than the usual pkg subdirectory, do

./configure --with-gaproot=path

where path is a path to the main GAP root directory. See

./configure --help

for further options.

Afterwards call make to compile a binary file.

The package willnot work without this step.

If you installed the package in another pkg directory than the standard pkg directory in your GAP 4 installation, then you have to add the path to the directory containing your pkg directory to GAP's list of directories. This can be done by starting GAP with the -l command line option followed by the name of the directory and a semicolon. Then your directory is prepended to the list of directories searched. Otherwise the package is not found by GAP. Of course, you can add this option to your GAP startup script.

---

Recompiling the documentation is possible by the command gap makedoc.g in the IO directory. But this should not be necessary.

For bug reports, feature requests and suggestions, please refer to

https://github.com/gap-packages/io/issues

io's Issues

Checking IO's signal handler is installed

Various functions in IO, like IO_fork, require IO's signal handler is installed. At the moment, we don't check if the signal handler is installed and if it is not, then later function calls to IO_waitpid fail, as GAP has already swallowed the child.

I can think of 3(ish) reasonable solutions to this:

1. Make functions like IO_fork fail if IO's signal handler is not installed.
2. Install IO's signal handler whenever we run a function which requires.
3. Install IO's signal handler on package loading.
   3b) And make IO_InstallSIGCHLDHandler and IO_RestoreSIGCHLDHandler into no-op functions, and remove the ability to control the signal handler at all.

IO does not work in threads other than 0

IO does not work in any other thread than thread 0. To see this, type

!shell
IO_FindExecutable( "bash" );

Tests don't seem to pass in 32-bit cygwin

I've been tentatively looking into why Digraphs doesn't work properly in cygwin, at least in 32-bit (seems to be okay in 64-bit now). I'm using AppVeyor to experiment with a 32-bit cygwin setup for GAP and Digraphs. A lot of the problems with Digraphs seem to arise when we want to do things with reading from/writing to files, so I thought I would check how the io package behaves on its own. You can see a build of what happens here and the full log is here: LOG: io-tests-in-cygwin-32bit.txt (the build dies after 1 hour without completing).

I'll copy the most relevant parts of the test failures. In tst/bugfix.tst, it seems like IO_WriteLine is always off by one; and in tst/testgap.tst it looks like something is really wrong; I don't understand it. Perhaps I'm doing something wrong with my setup, but I thought it was worth pointing this out anyway. This uses the GAP master branch.

 ┌───────┐   GAP 4.dev of today
 │  GAP  │   https://www.gap-system.org
 └───────┘   Architecture: i686-pc-cygwin-default32-kv6
 Configuration:  gmp 6.1.2, GASMAN
 Loading the library and packages ...
 Packages:   GAPDoc 1.6.1, PrimGrp 3.3.1, SmallGrp 1.3, TransGrp 2.0.4
 Try '??help' for help. See also '?copyright', '?cite' and '?authors'
gap> ─────────────────────────────────────────────────────────────────────────────
Loading  IO 4.5.4 (Bindings for low level C library I/O routines)
by Max Neunhöffer (http://www-groups.mcs.st-and.ac.uk/~neunhoef).
maintained by:
   Max Horn (http://www.quendi.de/math).
Homepage: https://gap-packages.github.io/io
Report issues at https://github.com/gap-packages/io/issues
─────────────────────────────────────────────────────────────────────────────
true
Architecture: i686-pc-cygwin-default32-kv6

testing: /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:4
# Input is:
IO_WriteLine(f, s{[1..30]});
# Expected output:
31
# But found:
32
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:6
# Input is:
IO_WriteLine(f, s);
# Expected output:
262145
# But found:
262146
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:8
# Input is:
IO_WriteLine(f, s{[1..2^17]});
# Expected output:
131073
# But found:
131074
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:10
# Input is:
IO_WriteLine(f, s{[1..2^17+1]});
# Expected output:
131074
# But found:
131075
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:12
# Input is:
IO_WriteLine(f, s2{[1..30]});
# Expected output:
31
# But found:
32
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:14
# Input is:
IO_WriteLine(f, s2);
# Expected output:
262145
# But found:
262146
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:16
# Input is:
IO_WriteLine(f, s2{[1..2^17]});
# Expected output:
131073
# But found:
131074
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:18
# Input is:
IO_WriteLine(f, s2{[1..2^17+1]});
# Expected output:
131074
# But found:
131075
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:32
# Input is:
IO_WriteLine(f, s{[1..30]});
# Expected output:
31
# But found:
32
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:34
# Input is:
IO_WriteLine(f, s);
# Expected output:
262145
# But found:
262146
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:36
# Input is:
IO_WriteLine(f, s{[1..2^17]});
# Expected output:
131073
# But found:
131074
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/bugfix.tst:38
# Input is:
IO_WriteLine(f, s{[1..2^17+1]});
# Expected output:
131074
# But found:
131075
########
     281 ms (0 ms GC) and 13.1MB allocated for bugfix.tst
testing: /home/appveyor/gap/pkg/io-4.5.4/tst/children.tst
    1985 ms (0 ms GC) and 2.48MB allocated for children.tst
testing: /home/appveyor/gap/pkg/io-4.5.4/tst/sendstringbackground.tst
     703 ms (0 ms GC) and 41.3KB allocated for sendstringbackground.tst
testing: /home/appveyor/gap/pkg/io-4.5.4/tst/testgap.tst
      1 [main] gap 1492 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
  32646 [main] gap 1492 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
  37305 [main] gap 1492 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
      3 [main] gap 2400 fork: child -1 - forked process 1492 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 2912 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
   1145 [main] gap 2912 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1409 [main] gap 2912 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
 217235 [main] gap 2400 fork: child -1 - forked process 2912 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 684 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    417 [main] gap 684 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
    824 [main] gap 684 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
 433463 [main] gap 2400 fork: child -1 - forked process 684 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 3024 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
   6509 [main] gap 3024 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
  73122 [main] gap 3024 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
 717433 [main] gap 2400 fork: child -1 - forked process 3024 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 1908 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    628 [main] gap 1908 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1973 [main] gap 1908 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
 941757 [main] gap 2400 fork: child -1 - forked process 1908 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 2856 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    584 [main] gap 2856 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1622 [main] gap 2856 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
1201718 [main] gap 2400 fork: child -1 - forked process 2856 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 552 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
   2350 [main] gap 552 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   3357 [main] gap 552 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
1406641 [main] gap 2400 fork: child -1 - forked process 552 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 2192 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
   1050 [main] gap 2192 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   2687 [main] gap 2192 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
1617796 [main] gap 2400 fork: child -1 - forked process 2192 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 2296 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    520 [main] gap 2296 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1476 [main] gap 2296 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
1726817 [main] gap 2400 fork: child -1 - forked process 2296 died unexpectedly, retry 0, exit code 0x100, errno 11
      2 [main] gap 2316 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    692 [main] gap 2316 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1562 [main] gap 2316 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
1829935 [main] gap 2400 fork: child -1 - forked process 2316 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 2244 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    488 [main] gap 2244 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1415 [main] gap 2244 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
1949767 [main] gap 2400 fork: child -1 - forked process 2244 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 1760 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    593 [main] gap 1760 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1487 [main] gap 1760 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
2054953 [main] gap 2400 fork: child -1 - forked process 1760 died unexpectedly, retry 0, exit code 0x100, errno 11
      2 [main] gap 1412 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    669 [main] gap 1412 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1702 [main] gap 1412 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
2169496 [main] gap 2400 fork: child -1 - forked process 1412 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 2824 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    588 [main] gap 2824 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1510 [main] gap 2824 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
2270756 [main] gap 2400 fork: child -1 - forked process 2824 died unexpectedly, retry 0, exit code 0x100, errno 11
      1 [main] gap 420 fixup_mmaps_after_fork: VirtualAlloc failed for MAP_PRIVATE address 0x9FEA0000, Win32 error 1455
    569 [main] gap 420 C:\cygwin\home\appveyor\gap\.libs\gap.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
   1096 [main] gap 420 cygwin_exception::open_stackdumpfile: Dumping stack trace to gap.exe.stackdump
2384865 [main] gap 2400 fork: child -1 - forked process 420 died unexpectedly, retry 0, exit code 0x100, errno 11
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/testgap.tst:8
# Input is:
for f in files do Read(Filename(d[1], f)); od;
# Expected output:
# But found:
Error, AppendList: <list2> must be a small list (not the value 'fail')
Error, Unable to read compressed file correctly: 6
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/testgap.tst:9
# Input is:
for f in files do Read(Filename(d[1], f)); od;
# Expected output:
# But found:
Error, Usage: IO_HasData( f ) with IsFile(f)
Error, Unable to read compressed file correctly: 6
########
########> Diff in /home/appveyor/gap/pkg/io-4.5.4/tst/testgap.tst:10
# Input is:
for f in files do Read(Filename(d[1], f)); od;
# Expected output:
# But found:
Error, Usage: IO_HasData( f ) with IsFile(f)
Error, Unable to read compressed file correctly: 6
########

...and then lots more errors like this.

Properly recognise ~ in filename strings

Currently on a Mac using ~ (for user directory) in the name of a file for IO_File and chums, results in fail being returned. It would be wonderful if this worked.

gap>  file := IO_File("~/tmp.txt", "w");
fail
gap>

Get rid of SuPeRfail

We should stop using SuPeRfail in io. The only reason it is referenced is to that it can be serialized. We should just remove that code.

A tiny concern is deserializing existing pickled data which contains SuPeRfail. I am tempted to ignore that, as such a thing is unlikely to exist in the first place. Though, just to be sure, we could retain a SPRF unpickler which outputs a warning, and just unpickles it as e.g. a string, or fail or so

GAP code coverage broken by fork()

If using fork with --cover, the resulting profile is broken. This is not necessarily an IO package problem, as the coverage code lives in the main GAP repository, so I will open an issue there, too.

Can't unpickle matrix groups with no generators

In GAP, we can make groups with no generators, so calling Group(GeneratorsOfGroup(g)) doesn't work. The fix is to pass the identity of g as the second argument of 'Group'.

I've fixed this for permutation groups, but not for matrix groups, as nothing simple seems able to fix It:

• The way the pickling is set up, I need the group before I know the dimension and field.

• Passing in a "fake" list of generators which contains the identity is also tricky.

We could just make a fresh matrix group where "GeneratorsOfGroup" won't return nothing, or change how we pickle them so we always pass the identity in (changing the name of course).

CheckForUpdates fails

Due to a change on the gap website, that gap-system.org does not accept anymore http requests, the command "CheckForUpdates" does not work anymore.

In fact, it would also be nice to implement an https client, since http seems to be less and less used.

m4/ac_find_gap.m4 do not support system GMP

This was reported to GAP Support by Bill Allombert:

The IO package include a file m4/ac_find_gap.m4 which does (inter alia)

 AC_MSG_CHECKING([for GAP's gmp.h location])
 if test -r "$ARCHPATH/extern/gmp/include/gmp.h"; then
   GAP_CPPFLAGS="$GAP_CPPFLAGS -I$ARCHPATH/extern/gmp/include"
   AC_MSG_RESULT([$ARCHPATH/extern/gmp/include/gmp.h])
 else
   AC_MSG_RESULT([not found, GAP was compiled without GMP])
 fi;

This is incorrect:
If gap is configured with the option '--with-gmp=system', the test fails even though it should succeed.

source ball contains doc/_entities.xml

My guess is that doc/_entities.xml should not be distributed.

Test failure with memory-checking enabled

After 4d38b74, the test suite fails if GAP is built with memory-checking enabled:

testing: /var/lib/portage/tmp/portage/dev-gap/io-4.8.2-r1/temp/pkg/io-4.8.2/ts\
t/sendstringbackground.tst
########> Diff in /var/lib/portage/tmp/portage/dev-gap/io-4.8.2-r1/temp/p\
kg/io-4.8.2/tst/sendstringbackground.tst:2
# Input is:
IsBound(HPCGAP) or ForAll([1..1000], x -> IO_SendStringBackground(f, "cheese")\
);
# Expected output:
true
# But found:
Error, List Element: <list> must be a list (not an object map)
########
   -1573 ms (0 ms GC) and 19.4KB allocated for sendstringbackground.tst
testing: /var/lib/portage/tmp/portage/dev-gap/io-4.8.2-r1/temp/pkg/io-4.8.2/ts\
t/testgap.tst
Panic in src/gasman.c:2077: incorrectly marked DEAD bag
/var/lib/portage/tmp/portage/dev-gap/io-4.8.2-r1/temp/environment: line 1199:    25 Segmentation fault      ${gapcmd}

I noticed this while testing 4.13.0-alpha1 but it looks like it happens in 4.12.2 as well.

please provide a tst/text.txt.gz without timestamp

Hello,
Lintian emits an package-contains-timestamped-gzip [1] issue for tst/text.txt.gz'. It can be easily fixed by recompressing with the option--no-name'.
Doing so will simplify the maintenance of the gap-io Debian package [2].
Thanks in advance,
Jerome
[1] https://lintian.debian.org/tags/package-contains-timestamped-gzip.html
[2] https://packages.qa.debian.org/g/gap-io.html

Fix SIGCHLD handling

SIGCHLD needs to be revised. There is an attempt to that in the code (see commits 48f0f1f and 3905afa), but it currently does not compile with stock GAP, and should probably reverted for the time being.

A proper fix requires changes to GAP itself. Note that other packages (e.g. libsing, which integrates Singular into GAP), will also need to deal with that. So this would be a useful thing anyway.

Max N., Reimer Behrends and Steve Linton already had a discussion about this at some point, so if anybody is interested in working on this, then talking to them might be a good starting point.

IO_Unpickle removes data from pickled file

The Python pickle module can serialize data as following:

import pickle

# An arbitrary collection of objects supported by pickle.
data = {
    'a': [1, 2.0, 3+4j],
    'b': ("character string", b"byte string"),
    'c': {None, True, False}
}

with open('data.pickle', 'wb') as f:
    # Pickle the 'data' dictionary using the highest protocol available.
    pickle.dump(data, f, pickle.HIGHEST_PROTOCOL)

❯ ls -lh data.pickle
Permissions Size User   Date Modified Name
.rw-r--r--   130 kiryph  7 Sep 12:08  data.pickle

Now when I want to read in the data again, I can do this as following:

import pickle

with open('data.pickle', 'rb') as f:
    # The protocol version used is detected automatically, so we do not
    # have to specify it.
    data = pickle.load(f)

This leaves the file data.pickle untouched:

❯ ls -lh data.pickle
Permissions Size User   Date Modified Name
.rw-r--r--   130 kiryph  7 Sep 12:08  data.pickle

So I can do this again and again.

I would have expected that IO_Unpickle(f) from the gap package IO would behave similarly.
However, when I unpickle gap data, the data is removed from the pickled file (despite f := IO_File("data.guck"); with the mode being "r" read-only).

I did not find a comment about this in the documentation:

• https://docs.gap-system.org/pkg/io/doc/chap5.html
• https://docs.gap-system.org/pkg/io/doc/chap10.html#X81BD8400832EE20B

As a workaround, I could duplicate the file before reading in. However, this increases the number of steps and is less convenient for interactive gap sessions, compared to gap> data1 := IO_Unpickle(IO_File("data.guck"));.

Could there be a flag that says "leave the pickle file unchanged"?

Making documentation on system with case sensitive filesystem fails

If I try making the docs for this package on linux the following happens:

➤ gmake doc
(echo 'Read("makedoc.g");' | /home/makx/dev/gap/bin/gap.sh -A -q)
Generating documentation in Directory("./doc/")
#I Composing XML document . . .
Error, Cannot include file ./doc/IO.xml.

This clearly is because the file is called io.xml, not IO.xml. I assume AutoDoc uses the package name as filename, so this comes down to the question: Is the official name of the package io or IO.

One fix is to rename io.xml to IO.xml, another is to rename the package in PackageInfo.g, yet another would be for @gap-packages/autodoc to lowercase package names for purposes of making a "canonical name".

io-4.4.2 build fails

I can see that on at least several Jenkins slaves:

checking for kill... yes
checking for gethostname... yes
checking for getsockname... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating src/pkgconfig.h
config.status: executing depfiles commands
config.status: executing libtool commands
+ make
 cd . && /bin/sh /mnt/disk2/hudson-slave/workspace/GAP-update-release-test/GAPCOPTS/64build/GAPGMP/gmp/GAPTARGET/packages/label/64bit/gap4r7/pkg/io-4.4.2/cnf/missing automake-1.14 --foreign Makefile
/mnt/disk2/hudson-slave/workspace/GAP-update-release-test/GAPCOPTS/64build/GAPGMP/gmp/GAPTARGET/packages/label/64bit/gap4r7/pkg/io-4.4.2/cnf/missing: line 81: automake-1.14: command not found
WARNING: 'automake-1.14' is missing on your system.
         You should only need it if you modified 'Makefile.am' or
         'configure.ac' or m4 files included by 'configure.ac'.
         The 'automake' program is part of the GNU Automake package:
         <http://www.gnu.org/software/automake>
         It also requires GNU Autoconf, GNU m4 and Perl in order to run:
         <http://www.gnu.org/software/autoconf>
         <http://www.gnu.org/software/m4/>
         <http://www.perl.org/>
make: *** [Makefile.in] Error 127 

Do not try to guess "best" compiler flags"

This is output when running configure:

********************************************************
* WARNING: Don't know the best CFLAGS for this system  *
* Use ./configure CFLAGS=... to specify your own flags *
* (otherwise, a default of CFLAGS=-O3 will be used)    *
********************************************************

It doesn't make sense to try to guess best compiler flags. First off, it just won't work in general (much better to leave it to the user to figure out the best flags), secondly we hardly need this for io

Fixing this is quite easy (remove some configure checks), I'll do it next week.

Pickle FpGroups

I miss the possibility to store a larger list of FpGroups to a file.
As far as I know, only permutation and matrix groups are implemented:

io/gap/pickle.gi

Lines 1155 to 1209 in a911c88

	InstallMethod( IO_Pickle, "for a permutation group",
	[ IsFile, IsPermGroup ],
	function( f, g )
	if IO_Write(f,"PRMG") = fail then return IO_Error; fi;
	if IO_Pickle(f,GeneratorsOfGroup(g)) = IO_Error then return IO_Error; fi;
	if HasSize(g) then
	if IO_Pickle(f,Size(g)) = IO_Error then return IO_Error; fi;
	else
	if IO_Pickle(f,fail) = IO_Error then return IO_Error; fi;
	fi;
	if HasStabChainImmutable(g) then
	if IO_Pickle(f,BaseStabChain(StabChainImmutable(g))) = IO_Error then
	return IO_Error;
	fi;
	elif HasStabChainMutable(g) then
	if IO_Pickle(f,BaseStabChain(StabChainMutable(g))) = IO_Error then
	return IO_Error;
	fi;
	else
	if IO_Pickle(f,fail) = IO_Error then return IO_Error; fi;
	fi;
	return IO_OK;
	end );
	IO_Unpicklers.PRMG :=
	function(f)
	local base,g,gens,size;
	gens := IO_Unpickle(f); if gens = IO_Error then return IO_Error; fi;
	g := GroupWithGenerators(gens);
	size := IO_Unpickle(f); if size = IO_Error then return IO_Error; fi;
	if size <> fail then SetSize(g,size); fi;
	base := IO_Unpickle(f); if base = IO_Error then return IO_Error; fi;
	if base <> fail then
	StabChain(g,rec(knownBase := base));
	fi;
	return g;
	end;
	InstallMethod( IO_Pickle, "for a matrix group",
	[ IsFile, IsMatrixGroup ],
	function( f, g )
	return IO_GenericObjectPickler(f,"MATG",[GeneratorsOfGroup(g)],g,
	[Name,Size,DimensionOfMatrixGroup,FieldOfMatrixGroup],[],[]);
	end );
	IO_Unpicklers.MATG :=
	function(f)
	local g,gens;
	gens := IO_Unpickle(f); if gens = IO_Error then return IO_Error; fi;
	g := GroupWithGenerators(gens);
	return
	IO_GenericObjectUnpickler(f,g,
	[Name,Size,DimensionOfMatrixGroup,FieldOfMatrixGroup],[]);
	return g;
	end;

For example:

gap> L24fp;
[ Group([ g1, g2, g3, g4 ]), Group([ g1, g2, g3, g5*g4^-1 ]), Group([ g1, g2, g3, g4^-2 ]), Group([ g1, g2, g3, g4^3 ]), Group([ g1, g2, g3, g4^-1*g5^2*g4^-1 ]), 
  Group([ g1, g2, g3, g4^-4 ]), Group([ g1, g2, g4 ]), Group([ g1, g2, g4^-2, g5*g4^-1 ]), Group([ g1, g2, g5*g4^-1, g3*g5*g4^-1*g3^-1 ]), 
  Group([ g1, g2, g5*g4^-1, g3*g4^-2*g3^-1 ]), Group([ g1, g2, g5*g4^-1, g3*g4^2*g5^-1*g3^-1 ]), Group([ g1, g2, g5*g4^-1, g3*g4^3*g3^-1 ]), 
  Group([ g1, g2, g5*g4^-1, g3*g4^-2*g5*g4^-1*g3^-1 ]), Group([ g1, g2, g5*g4^-1, g3*g4^-4*g3^-1 ]), Group([ g1, g2, g5*g4, g4^3 ]), Group([ g1, g2, g5*g4, g4^-4 ]), 
  Group([ g1, g2, g5*g4, g4^5 ]), Group([ g1, g2, g5*g4, g4^-6 ]), Group([ g1, g2, g5*g4, g4^7 ]), Group([ g1, g2, g5*g4, g4^-8 ]), Group([ g1, g2, g4^-2 ]), 
  Group([ g1, g2, g3*g4^-2*g3^-1, g4^-4 ]), Group([ g1, g2, g4^-2*g5^-1*g4^-1 ]), Group([ g1, g2, g3*g4^-2*g3^-1, g4^-2*g5*g4^-1 ]), Group([ g1, g3, g4 ]), 
  Group([ g1, g3, g5*g4^-1 ]), Group([ g1, g3, g4^-2 ]), Group([ g1, g3, g4^-1*g5^2*g4^-1 ]), Group([ g1, g3, g4^3 ]), Group([ g1, g4 ]), 
  Group([ g1, g4*g2^-1, g4^-1*g2^-1 ]), Group([ g1, g4*g2^-1, g3*g4^-2*g3^-1 ]), Group([ g1, g4*g2^-1, g3*g4^3*g3^-1 ]), Group([ g1, g4*g2^-1, g3*g4^-4*g3^-1 ]), 
  Group([ g1, g4^-2, g5*g4^-1 ]), Group([ g1, g5*g4^-1, g3*g5*g4^-1*g3^-1 ]), Group([ g1, g5*g4^-1, g3*g4^-2*g3^-1 ]), Group([ g1, g5*g4, g2*g4^2 ]), 
  Group([ g1, g4^-2, g5^-2, g2*g4*g5^-1 ]), Group([ g1, g2*g4*g5^-1, g3*g4^-2*g3^-1 ]), Group([ g1, g5*g4, g4^3 ]), Group([ g1, g5*g4, g4^3*g2^-1 ]), 
  Group([ g1, g5*g4, g4^-4 ]), Group([ g1, g5*g4, g2*g4^4 ]), Group([ g1, g2*g4*g5, g2*g4^-2 ]), Group([ g1, g2*g4*g5, g2*g4^-1*g5^-1, g4^-4 ]), 
  Group([ g1, g2*g4*g5, g2*g4^-1*g5^-1, g4^-2*g5*g4^-1 ]), Group([ g1, g2*g4^2, g2*g4^-2 ]), Group([ g1, g4^-2 ]), Group([ g2, g3, g4 ]), Group([ g2, g3, g5*g4^-1 ]), 
  Group([ g2, g3, g4^-2 ]), Group([ g2, g3, g4^2*g5^-1 ]), Group([ g2, g3, g4^3 ]), Group([ g2, g3, g4^-1*g5^2*g4^-1 ]), Group([ g2, g4, g5 ]), 
  Group([ g2, g4, g5^-2, g1*g3^-1*g5^-1 ]), Group([ g2, g4, g5^-2 ]), Group([ g2, g4, g5^3 ]), Group([ g2, g4, g1*g3^-1*g5^3 ]), Group([ g2, g4, g5^-4 ]), 
  Group([ g2, g5*g4^-1, g1*g4^2 ]), Group([ g2, g5*g4^-1, g3*g5*g4^-1*g3^-1 ]), Group([ g2, g5*g4^-1, g4*g3^-1*g4^-1*g3, g4^3*g1^-1 ]), 
  Group([ g2, g5*g4^-1, g3*g4^-2*g3^-1 ]), Group([ g2, g5*g4^-1, g1*g3*g4^2*g3^-1 ]), Group([ g2, g5^-2, g1*g3*g5^-1, g4^-4 ]), Group([ g2, g4^-2, g5^-2 ]), 
  Group([ g2, g4^-2, g3*g4^2*g1^-1 ]), Group([ g2*g1^-1, g3, g4 ]), Group([ g2*g1^-1, g3, g5*g4^-1 ]), Group([ g2*g1^-1, g3, g4^-2 ]), 
  Group([ g2*g1^-1, g3, g4^-1*g5^2*g4^-1 ]), Group([ g2*g1^-1, g3, g4^3 ]), Group([ g2*g1^-1, g4 ]), Group([ g2*g1^-1, g4*g1^-1, g4^-1*g1^-1 ]), 
  Group([ g2*g1^-1, g4*g1^-1, g4^-3*g1^-1 ]), Group([ g2*g1^-1, g4*g1^-1, g4^-5*g1^-1 ]), Group([ g2*g1^-1, g4*g1^-1, g4^-7*g1^-1 ]), 
  Group([ g2*g1^-1, g4*g3^-1, g4^-1*g3 ]), Group([ g2*g1^-1, g4^-1*g3, g3*g5^2 ]), Group([ g2*g1^-1, g4^-1*g3, g4^2*g5^-1*g3^-1 ]), Group([ g2*g1^-1, g4^-2, g5*g4^-1 ]), 
  Group([ g2*g1^-1, g5*g4^-1, g3*g5*g4^-1*g3^-1 ]), Group([ g2*g1^-1, g5*g4^-1, g4*g3^-1*g4^-1*g3, g4^3*g1^-1 ]), Group([ g2*g1^-1, g5*g4^-1, g3*g4^-2*g3^-1 ]), 
  Group([ g2*g1^-1, g5*g4, g4^3 ]), Group([ g2*g1^-1, g5*g4, g4^-4 ]), Group([ g2*g1^-1, g1*g4*g5, g4^-4 ]), Group([ g2*g1^-1, g4^-2 ]), Group([ g3, g4 ]), 
  Group([ g3, g5*g4^-1 ]), Group([ g3, g4^-2 ]), Group([ g4, g5 ]), Group([ g4, g5*g2^-1, g5^-1*g2^-1 ]), Group([ g4, g5*g2^-1, g1*g3^-1*g5 ]), 
  Group([ g4, g5*g2^-1, g5^-3*g2^-1 ]), Group([ g4, g5*g2^-1, g1*g3^-1*g5^3 ]), Group([ g4, g5^-2, g1*g3^-1*g5^-1 ]), Group([ g4, g1*g3^-1*g5^-1, g5^-4 ]), 
  Group([ g4, g5^-2, g2*g3*g4^-1*g1^-1 ]), Group([ g4, g5^-2 ]), Group([ g4, g5^-4, g1*g2*g3^-1*g5^-2 ]), Group([ g4*g2^-1, g5*g2^-1, g1*g2*g4 ]), 
  Group([ g4*g2^-1, g5*g2^-1, g3*g4^-2*g3^-1 ]), Group([ g4*g2^-1, g5*g2^-1, g1*g2*g4^3 ]), Group([ g4*g2^-1, g4^-1*g2^-1, g5^-2 ]), 
  Group([ g4*g2^-1, g4^-1*g2^-1, g3*g4^2*g1^-1 ]), Group([ g4*g3^-1, g4^-1*g3, g5*g3^-1 ]), Group([ g5*g4^-1, g1*g2*g4^-1, g3*g4^-2*g3^-1 ]), 
  Group([ g5*g4^-1, g1*g4^2 ]) ]

Documentation on Extending the pickling framework can be found here:
https://gap-packages.github.io/io/doc/chap5_mj.html#X7B1C9A9C7D3C0312

`IO_WriteLine` question

Is there a possibility that lines get mixed up when IO_WriteLine is called from several processes on the same file? I guess IO_Flush does not do any locking, so on the operating system level things can go wrong. The order of the lines does not matter.

Is there any locking mechanism available/planned?

HTTP request with unwritable destination

Accidentally stumbled across this while playing with AtlasRep:

gap> SingleHTTPRequest( "brauer.maths.qmul.ac.uk", 80, "GET", "/Atlas/spor/J1/mtx/J1G1-f2r20B0.m1", rec(), false, "/tmp/J1G1-f2r20B0.m1");
rec( body := "/tmp/J1G1-f2r20B0.m1", closed := true, 
  header := rec( ("accept-ranges") := "bytes", connection := "close", 
      ("content-length") := "441", 
      ("content-type") := "text/plain; charset=UTF-8", 
      date := "Tue, 28 Jul 2015 16:39:28 GMT", 
      etag := "\"1d45b1-1b9-34b9ec0c23140\"", 
      ("last-modified") := "Mon, 24 May 1999 18:22:21 GMT", 
      server := "Apache/2.2.3 (Scientific Linux)" ), protoversion := "1.1", 
  status := "OK", statuscode := 200 )
gap> SingleHTTPRequest( "brauer.maths.qmul.ac.uk", 80, "GET", "/Atlas/spor/J1/mtx/J1G1-f2r20B0.m1", rec(), false, "/unwritabledir/J1G1-f2r20B0.m1");
Error, Usage: IO_Write( f ,things ... ) with IsFile(f) called from
IO_Write( out, chunk ); called from
HTTPRequest( conn, method, uri, header, body, target ) called from
<function "SingleHTTPRequest">( <arguments> )
 called from read-eval loop at line 77 of _stdin_
you can 'quit;' to quit to outer loop, or
you can 'return;' to continue
brk> Print(out);
fail

This also happens when trying to write to a directory that exists, but for which the user does not have write permission. Can something friendlier be done, like in the conn.sock = fail case?

HTTP: please honour http proxy

As Debian maintainer of the gap-io Debian package, I got a bugreport that claims that the system's http proxy config is not respected by the current implementation.

Supporting ~ in paths

IO package does not support ~ in paths, for example:

gap> ChangeDirectoryCurrent("/");
true
gap> Exec("pwd");
/
gap> ChangeDirectoryCurrent("~");
Error, <expr> must be 'true' or 'false' (not a boolean) in
  if IO_chdir( path )  then
    GAPInfo.DirectoryCurrent := Directory( path );
    return true;
else
    return fail;
fi; at /Users/alexk/gap4r8p3/pkg/io-4.4.6/gap/io.gi:1710 called from 
<function "ChangeDirectoryCurrent">( <arguments> )
 called from read-eval loop at line 3 of *stdin*
you can replace <expr> via 'return <expr>;'
brk> 

I have heard this from @james-d-mitchell who may have another example.

IO_WriteLine fails for certain long strings

I have found that IO_WriteLine is unable to write certain long strings successfully. On two different computers I have been able to reproduce the following, where the first 2^17 characters of a long string can be written to a single line, but any longer string returns "fail":

gap> f := IO_File("bug.txt", "w");
<file fd=4 wbufsize=65536 wdata=0>
gap> s := List([1..2^18], x->CharInt(Random([63..126])));;
gap> s{[1..30]};
"DbRyD[cnrB^PKuohlarEsYG_gP}[h_"
gap> IO_WriteLine(f, s{[1..30]});
31
gap> IO_WriteLine(f, s);
fail
gap> IO_WriteLine(f, s{[1..2^17]});
131073
gap> IO_WriteLine(f, s{[1..2^17+1]});
fail
gap> LastSystemError();
rec( message := "no error", number := 0 )

I investigated IO's documentation, but couldn't find anything about a line-size limit. Furthermore, I constructed a different example which writes a much longer string successfully:

gap> a := ListWithIdenticalEntries(2^18, 'a');;
gap> a{[1..10]};
"aaaaaaaaaa"
gap> IO_WriteLine(f, a);
262145

Perhaps I have made an oversight, but for the moment I'm unable to write certain long strings to files.

Add `setenv` wrapper

Originally requested at gap-system/gap#3166

Avoid using EvalString in IO_Unpickle

IO_Unpickle using EvalString means that a malicious data file can perform arbitrary code execution. Part of me feels this is a bad thing, and it might be worth putting the work in to remove the evals. However GAP might already be such a leaky sieve that this ship has sailed!

Interested in opinions.

Intermittent failure in `sendstringbackground.tst` in GitHub Actions macOS job

Example 1, Example 2:

+ /Users/runner/gap/bin/gap.sh -l '/Users/runner/work/io/io/gaproot;' --quitonbreak --cover coverage/mMgjCe.coverage tst/testall.g
 ┌───────┐   GAP 4.12dev built on 2022-01-25 16:44:56+0000
 │  GAP  │   https://www.gap-system.org
 └───────┘   Architecture: x86_64-apple-darwin20.6.0-default64-kv8
 Configuration:  gmp 6.2.1, GASMAN
 Loading the library and packages ...
 Packages:   AClib 1.3.2, Alnuth 3.1.2, AtlasRep 2.1.0, AutoDoc 2020.08.11, 
             AutPGrp 1.10.2, CRISP 1.4.5, Cryst 4.1.24, CrystCat 1.1.9, 
             CTblLib 1.3.2, FactInt 1.6.3, FGA 1.4.0, Forms 1.2.6, 
             GAPDoc 1.6.4, genss 1.6.6, IO 4.7.2, IRREDSOL 1.4.3, 
             LAGUNA 3.9.3, orb 4.8.4, Polenta 1.3.9, Polycyclic 2.16, 
             PrimGrp 3.4.1, RadiRoot 2.8, recog 1.3.2, ResClasses 4.7.2, 
             SmallGrp 1.4.2, Sophus 1.24, SpinSym 1.5.2, TomLib 1.2.9, 
             TransGrp 3.3, utils 0.72
 Try '??help' for help. See also '?copyright', '?cite' and '?authors'
Architecture: x86_64-apple-darwin20.6.0-default64-kv8

testing: /Users/runner/work/io/io/gaproot/pkg/io/tst/all.tst
       6 ms (0 ms GC) and 198KB allocated for all.tst
testing: /Users/runner/work/io/io/gaproot/pkg/io/tst/bugfix.tst
     431 ms (12 ms GC) and 24.1MB allocated for bugfix.tst
testing: /Users/runner/work/io/io/gaproot/pkg/io/tst/children.tst
     555 ms (0 ms GC) and 4.25MB allocated for children.tst
testing: /Users/runner/work/io/io/gaproot/pkg/io/tst/funcs.tst
       3 ms (0 ms GC) and 65.9KB allocated for funcs.tst
testing: /Users/runner/work/io/io/gaproot/pkg/io/tst/sendstringbackground.tst
########> Diff in /Users/runner/work/io/io/gaproot/pkg/io/tst/sendstringb\
ackground.tst:2
# Input is:
IsBound(HPCGAP) or ForAll([1..3000], x -> IO_SendStringBackground(f, "cheese")\
);
# Expected output:
true
# But found:
#E Overflow in table of ignored processes#E Overflow in table of ign\
ored proce\
sses#E Overflow in table of ignored processes#E Overflow in table of ignored p\
\
rocesses#E Overflow in table of ignored processes#E Overflow in table of ignor\
\
ed processes#E Overflow in table of ignored processes#E Overflow in table of i\
\
gnored processes#E Overflow in table of ignored processes#E Overflow in table \
\
of ignored processes#E Overflow in table of ignored processes#E Overflow in ta\
\
ble of ignored processes#E Overflow in table of ignored processes#E Overflow i\
\
n table of ignored processes#E Overflow in table of ignored processes#E Overfl\
\
ow in table of ignored processes#E Overflow in table of ignored processes#E Ov\
\
erflow in table of ignored processes#E Overflow in table of ignored processes#\
\
E Overflow in table of ignored processes#E Overflow in table of ignored proces\
\
ses#E Overflow in table of ignored processes#E Overflow in table of ignored pr\
\
ocesses#E Overflow in table of ignored processes#E Overflow in table of ignore\
\
d processes#E Overflow in table of ignored processes#E Overflow in table of ig\
\
nored processes#E Overflow in table of ignored processes#E Overflow in table o\
\
f ignored processes#E Overflow in table of ignored processes#E Overflow in tab\
\
le of ignored processes#E Overflow in table of ignored processes#E Overflow in\
\
 table of ignored processes#E Overflow in table of ignored processes#E Overflo\
\
w in table of ignored processes#E Overflow in table of ignored processes#E Ove\
\

[...etc, omitted...]

########
     198 ms (0 ms GC) and 288KB allocated for sendstringbackground.tst
testing: /Users/runner/work/io/io/gaproot/pkg/io/tst/testgap.tst
    9827 ms (562 ms GC) and 1.26GB allocated for testgap.tst
testing: /Users/runner/work/io/io/gaproot/pkg/io/tst/timeout.tst
      85 ms (79 ms GC) and 111KB allocated for timeout.tst
testing: /Users/runner/work/io/io/gaproot/pkg/io/tst/waitpid.tst
      74 ms (72 ms GC) and 55.5KB allocated for waitpid.tst
-----------------------------------
total     11179 ms (725 ms GC) and 1.29GB allocated
              1 failures in 1 of 8 files

#I  Errors detected while testing

Error: Process completed with exit code 1.

Make testsuite actually test something

At the moment the testsuite as run by travis does not test anything of value. Even with a small patch that runs testgap.g, there is no output, and the tests run quite long, so they will fail in travis.

pickling to strings?

Sometimes it is better to have a pickle to a string, or to a memory area, e.g. in libgap setting, where GAP objects might be parts of a bigger structure that needs to be pickled, e.g. a Python one, cf https://docs.python.org/2/library/pickle.html

I imagine it should not be too hard to provide, as the existing in IO code surely has all the needed parts available, but is these a chance such a change is accepted and merged?

Issue with old(er) binary files

Hi,

for gap package simpcomp we use the io package to save simplicial complexes in binary format. Saving them and then reading them in is not a problem (so I assume our code works, at least in some instances).

But some binary files we produced with an older version of io (and which we still use in our complex library) now produce the error Found a self-reference to an unknown object!. As soon as this happens something breaks severely and reading other files does not produce an error anymore but loaded data is corrupted.

Here is what happens in detail:

gap> SCLibLoad(SCLib,548);
Found a self-reference to an unknown object!
#I  SCIntFunc.GeneralUnpickler: Error while unpicking attribute value of 'ComputedSCSkelExs'
Found a self-reference to an unknown object!
#I  SCIntFunc.GeneralUnpickler: Error while unpicking attribute name
Found a self-reference to an unknown object!
#I  SCIntFunc.GeneralUnpickler: Error while unpicking attribute value of '<object>'
#I  SCIntFunc.GeneralUnpickler: Error while unpicking attribute name
Error, no method found! For debugging hints type ?Recovery from NoMethodFound
Error, no 1st choice method found for `InputTextString' on 1 arguments at /home/jspreer/GAP/gap-repository/lib/methsel2.g:250 called from
InputTextString( a ) at /home/jspreer/GAP/gap-repository/lib/string.gi:652 called from
EvalString( name ) at /home/jspreer/GAP/gap-repository/pkg/simpcomp/lib/io.gi:1734 called from
SCIntFunc.GeneralUnpickler( f, c ); at /home/jspreer/GAP/gap-repository/pkg/simpcomp/lib/io.gi:1757 called from
up( f ) at /home/jspreer/GAP/gap-repository/pkg/io-4.5.4/gap/pickle.gi:275 called from
IO_Unpickle( fh ) at /home/jspreer/GAP/gap-repository/pkg/simpcomp/lib/io.gi:785 called from
...  at *stdin*:26
type 'quit;' to quit to outer loop

Our methods to communicate with io are implemented in io.gi. See, for instance, functions SCIntFunc.GeneralPickler or SCIntFunc.GeneralUnpickler.

Setup is the following:

 ┌───────┐   GAP 4.10dev-1285-g44f1a0c of today
 │  GAP  │   https://www.gap-system.org
 └───────┘   Architecture: x86_64-pc-linux-gnu-default64-kv5
 Configuration:  gmp 6.1.0, GASMAN, readline

Happy to provide you with more details.

Directory name changes from io to IO

I've noticed that the name of the directory was changed from io-X.Y to IO-X.Y, since my package building scripts tripped over it. Please confirm if this change is deliverate, so I will update them. If this is the case, may be worth to update README and other places where the path is mentioned.

io doesn't load in its required version of GAP

The current master version of io doesn't load in GAP 4.8.* because IsHPCGAP (used in read.g) is not declared in this version; IsHPCGAP was added in 4.9.0, I think. The version of GAP required by io is currently 4.8.0.

Add Windows tests to CI setup

IO_open blocks for named pipes although mode O_NONBLOCK is passed

Create a named pipe with bash called test-pipe via mkfifo test-pipe.
Then start GAP, load the package and try

IO_open( "test-pipe", IO.O_RDONLY, IO.O_NONBLOCK );

This operation blocks, although I would expect it not to.
Using IO_open( "test-pipe", IO.O_RDWR, 0 ); works fine though, it only blocks when using IO_ReadLine.

My GAP banner:

 ┌───────┐   GAP 4.8.6, 12-Nov-2016, build of 2017-01-12 17:56:36 (CET)       
 │  GAP  │   http://www.gap-system.org
 └───────┘   Architecture: x86_64-pc-linux-gnu-gcc-default64
 Libs used:  gmp, readline
 Loaded workspace: /home/sergio/.gap/emptyWorkspace
 Components: trans 1.0, prim 2.1, small* 1.0, id* 1.0
 Packages:   AClib 1.2, Alnuth 3.0.0, AtlasRep 1.5.1, AutPGrp 1.6,
             CRISP 1.4.4, Cryst 4.1.12, CrystCat 1.1.6, CTblLib 1.2.2,
             FactInt 1.5.3, FGA 1.3.1, GAPDoc 1.5.1, IO 4.4.6,
             IRREDSOL 1.3.1, LAGUNA 3.7.0, Polenta 1.3.7, Polycyclic 2.11,
             RadiRoot 2.7, ResClasses 4.5.0, Sophus 1.23, SpinSym 1.5,
             TomLib 1.2.6, Utils 0.43

Get rid of global variable ourDIR

I was wondering how IO wraps POSIX functions returning pointer values. Turns out, it doesn't really. Instead, IO_opendir returns a boolean, and stores the actual return value of opendir in a global variable ourDIR. This could lead to a problem if one was not aware of this and tried to read from two directories at once. In the future, with HPC-GAP this might become a real problem.

So, we should try to get rid of such global state. In this particular case, one might return the pointer as a GAP integer (perhaps wrapped inside a record, or even component object or positional object to "hide" it from the user), then rely on the user not messing with it. This would be vaguely similar to what IO_stat does.

There are several more global variables but they are all only used to temporarily hold data; as such, they should be safe for now, but they will become a problem in HPC-GAP. These are ourdirent, ourstatbuf, ourfstatbuf, ourlstatbuf, argv, envp and of course all the globals related to signal handling.

Strings as list don't work, maybe clarify the documentation

In gap-system/gap#4585 @mfarrokhidg asked about using Strings as list of characters as arguments. This does not work. The documentation tries to address this ("arguments must be GAP Strings"), but perhaps the documentation should be more clear that the functions require IsStringRep not just IsString.

I gave the following workaround. A similar example might be useful in the manual.

gap> x:=['t', 'e', 's', 't'];
"test"
gap> IO_mkdir(x,0);
fail
gap> ConvertToStringRep(x);
gap> IO_mkdir(x,0);
true

Add tests for IO_Write and IO_WriteNonBlocking

Document dependencies of functions such as ParDoByFork

In a discussion in the GAP Forum, it turned out that the function ParDoByFork relies on the fact that the return values can be serialized with IO_Pickle.
(Thanks to @frankluebeck for inspecting this.)

This fact must be documented in the IO package manual, for all functions in question.

Please update data at gap-system.org

Hello:
Can you please update the data on the following webpage ?
http://www.gap-system.org/Packages/io.html

Inputs not in IsStringRep are not handled correctly by unbuffered IO_Write and IO_WriteNonBlocking

As pointed out by @ChrisJefferson in his comment on #20:

The same issue exists in the unbuffered writing in IO_Write and IO_WriteNonBlocking. Might be
worth thinking about if there is a global fix for all these cases.

I do think we should use #21 for the buffered case and possibly a seperate fix for the unbuffered case.

We might want some GAP core support, as I can't find a method to make a copy of any string as a
IsStringRep (there is a method to change a string's representation but i don't want to change user's
input).

Test failure with kernel debugging enabled

After building GAP with --enable-debug, the io test suite (from git master) fails for me:

io $ ln -s ../ pkg
io $ gap -A -R --nointeract --roots $(pwd)/\; -c 'ForceQuitGap(TestPackage("io"));'
 ┌───────┐   GAP 4.12.2 of 2022-12-18
 │  GAP  │   https://www.gap-system.org
 └───────┘   Architecture: amd64
 Configuration:  gmp 6.3.0, GASMAN, readline, KernelDebug
 Loading the library and packages ...
 Packages:   GAPDoc 1.6.6, PrimGrp 3.4.4, SmallGrp 1.5.3, TransGrp 3.6.4
 Try '??help' for help. See also '?copyright', '?cite' and '?authors'
─────────────────────────────────────────────────────────────────────────────
Loading  IO 4.8.2 (Bindings for low level C library I/O routines)
by Max Neunhöffer ([email protected]).
maintained by:
   Max Horn (https://www.quendi.de/math).
Homepage: https://gap-packages.github.io/io
Report issues at https://github.com/gap-packages/io/issues
─────────────────────────────────────────────────────────────────────────────
Architecture: amd64

testing: /home/mjo/src/io/pkg/io/tst/all.tst
# line 122 of 282 (43%)gap: src/integer.h:66: SIZE_INT: Assertion `IS_LARGEINT(obj)' failed.
Abort
/usr/lib64/libgap.so.8(+0x7e270)[0x7f9b5707e270]
/lib64/libc.so.6(+0x39bb0)[0x7f9b56e8bbb0]
/lib64/libc.so.6(+0x886e4)[0x7f9b56eda6e4]
/lib64/libc.so.6(raise+0x12)[0x7f9b56e8baf2]
/lib64/libc.so.6(abort+0xd3)[0x7f9b56e7422f]
/lib64/libc.so.6(+0x22157)[0x7f9b56e74157]
/lib64/libc.so.6(+0x32462)[0x7f9b56e84462]
/usr/lib64/libgap.so.8(+0xab065)[0x7f9b570ab065]
/usr/lib64/libgap.so.8(QuoInt+0x397)[0x7f9b570b09d7]
/home/mjo/src/io/pkg/io/bin/amd64/io.so(+0x8dd0)[0x7f9b56d01dd0]
/usr/lib64/libgap.so.8(IntrFuncCallEnd+0x513)[0x7f9b570b6653]
/usr/lib64/libgap.so.8(+0x142b7b)[0x7f9b57142b7b]
/usr/lib64/libgap.so.8(+0x144ca5)[0x7f9b57144ca5]
/usr/lib64/libgap.so.8(+0x145258)[0x7f9b57145258]
/usr/lib64/libgap.so.8(+0x1454d9)[0x7f9b571454d9]
/usr/lib64/libgap.so.8(+0x145899)[0x7f9b57145899]
/usr/lib64/libgap.so.8(+0x1459f8)[0x7f9b571459f8]
/usr/lib64/libgap.so.8(+0x145bb8)[0x7f9b57145bb8]
/usr/lib64/libgap.so.8(+0x145e76)[0x7f9b57145e76]
/usr/lib64/libgap.so.8(+0x146016)[0x7f9b57146016]
/usr/lib64/libgap.so.8(ReadEvalCommand+0x67e)[0x7f9b5714b55e]
/usr/lib64/libgap.so.8(+0x15fba0)[0x7f9b5715fba0]
/usr/lib64/libgap.so.8(+0x9f6af)[0x7f9b5709f6af]
/usr/lib64/libgap.so.8(+0x9fb18)[0x7f9b5709fb18]
/usr/lib64/libgap.so.8(+0x15bae4)[0x7f9b5715bae4]
/usr/lib64/libgap.so.8(+0x15d162)[0x7f9b5715d162]
/usr/lib64/libgap.so.8(+0x15bae4)[0x7f9b5715bae4]
/usr/lib64/libgap.so.8(+0x15b889)[0x7f9b5715b889]
/usr/lib64/libgap.so.8(EXEC_CURR_FUNC+0x2d)[0x7f9b5715db6d]
/usr/lib64/libgap.so.8(+0xa049b)[0x7f9b570a049b]
/usr/lib64/libgap.so.8(+0xa0825)[0x7f9b570a0825]
/usr/lib64/libgap.so.8(+0x9f6ca)[0x7f9b5709f6ca]

make doc stays stuck in gap + fix

My guess is that a terminal quit; is missing in makedoc.g.
hth,
Jerome

Compile error due to undeclared MAX_PTYS

io master currently does not compile:

src/io.c:195:19: error: use of undeclared identifier 'MAX_PTYS'
  for (i = 0; i < MAX_PTYS; i++) {
                  ^
src/io.c:196:11: error: use of undeclared identifier 'PtyIOStreams'
      if (PtyIOStreams[i].inuse) {
          ^

It seems the new SIGCHLD handler code needs to access some internal table of GAP, but both the macro constant MAX_PTYS and the array PtyIOStreams are currently only defined and usable from within GAP's src/iostream.c

I guess the fix will be to move those definitions in GAP to src/iostream.h, then once a GAP with those changes has been released, we can update IO... Slightly nicer would be to conditionally compile the SIGCHLD code, namely only if MAX_PTYS etc. are available.

Calling IO_opendir twice without IO_closedir leaks memory.

We can fix this by checking whether ourDIR is zero before calling opendir. If not, either generate an error or silently call closedir. I'd tend to favor the former, for while that might break some existing code, it hints at the user that something is wrong. But I could live with either.

exitcheck.g and tee: is it hanging ?

I am on my way to write a Debian Continuous Integration [1] simple test based on the `run_tests.sh', its two last line more precisely.

The following command hangs:
$ echo "Read("/usr/share/gap/pkg/io/tst/testall.g"); quit;" | gap | tee testlog.txt

The following command does not hang but does not finish with a proper end-of-line
$ echo "Read("/usr/share/gap/pkg/io/tst/testall.g"); quit;" | gap

This one does not hang but the last line does not have a proper end-of-line (according to cat -A)
$ echo "Read("/usr/share/gap/pkg/io/tst/testall.g"); quit;" | gap > testlog.txt

Are those behaviours expected ?

Thanks,
Jerome

[1] https://ci.debian.net/doc/

Provide a standard test

Provide a standard test, specified in PackageInfo.g to test as a part of GAP test suite.

io/PackageInfo.g

Line 119 in 76dad92

#TestFile := "tst/testall.g",

Opening nonexisting file in append mode

Is this the right behaviour?

gap> IO_CompressedFile("T3.gens", "a");
fail
gap> IO_File("T3.gens", "a");                                       
fail

Checked documentation (up to man 2 open), but no clear answer.

Documentation of `IO_WaitPid`

The documentation of IO_WaitPid claims that it returns fail or a record, but it can at least return fail, False, or a record.

Easy support for compressed files

There are some functions in the semigroup package which automatically invoke gzip, bzip2 or xz as appropriate when users read or write files with extensions gz, bz and xz. It would be useful to have this support built into IO rather the require repeating it everywhere.

The only question (in my mind) is if this should be done in IO_File automatically, or if there should be a new IO_CompressedFile. Personally I'm tempted to add it to IO_File, would anyone ever really want to read raw compressed binary files in gap? :) (of course such people can also use IO_Open).

I'm happy to work on this, just thought I'd check on interest first.

how to GET from a site which redirects http to https?

getting statuscode=301 (i.e. Moved Permanently).

gap> uri:="/api/gps_smallhash/?_format=json&order=1536&hash=651420946779&_offset=0";;
gap> SingleHTTPRequest("beta.lmfdb.org",80,"GET",uri, rec(), false, false);
rec( 
  body := "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>301 Moved Permanently</title>\n</head><body>\n<h1>Moved Permane\
ntly</h1>\n<p>The document has moved <a href=\"https://beta.lmfdb.org/api/gps_smallhash/?_format=json&amp;order=1536&amp;hash=651420946779&amp;_offse\
t=0\">here</a>.</p>\n<hr>\n<address>Apache/2.4.29 (Ubuntu) Server at beta.lmfdb.org Port 80</address>\n</body></html>\n", closed := true, 
  header := rec( ("cache-control") := "max-age=0", ("content-length") := "393", ("content-type") := "text/html; charset=iso-8859-1", 
      date := "Thu, 02 Mar 2023 13:22:36 GMT", expires := "Thu, 02 Mar 2023 13:22:36 GMT", 
      location := "https://beta.lmfdb.org/api/gps_smallhash/?_format=json&order=1536&hash=651420946779&_offset=0", 
      server := "Apache/2.4.29 (Ubuntu)" ), protoversion := "1.1", status := "Moved Permanently", statuscode := 301 )
gap> 

is there a simple way out here, or is this due to lack of ssl, etc?