maxim2266 / strit

Package strit introduces a new type of string iterator, along with a number of iterator constructors, wrappers and combinators.

strit

Package strit (STRing ITerator) assists in development of string processing pipelines by providing a simple iteration model that allows for easy composition of processing stages.

Motivation

Suppose we want to develop a function that reads a file line by line, removes leading and trailing whitespace from each line, selects only non-empty lines that also do not start with the # symbol, and stores those lines in a slice of strings. Using the Go standard library one possible implementation of the function may look like this:

func ReadConfig(fileName string) ([]string, error) {
	file, err := os.Open(fileName)

	if err != nil {
		return nil, err
	}

	defer file.Close()

	var res []string
	src := bufio.NewScanner(file)

	for src.Scan() {
		line := bytes.TrimSpace(src.Bytes())

		if len(line) > 0 && line[0] != '#' {
			res = append(res, string(line))
		}
	}

	if err = src.Err(); err != nil {
		return nil, err
	}

	return res, nil
}

Using strit package the implementation can be simplified down to:

func ReadConfig(fileName string) ([]string, error) {
	return strit.FromFile(fileName).
           Map(bytes.TrimSpace).
           Filter(strit.Not(strit.Empty).AndNot(strit.StartsWith("#"))).
           Strings()

Features

• A number of iterator constructors for reading text from a variety of sources:
  • io.Reader: FromReader FromReaderSF
  • io.ReadCloser: FromReadCloser FromReadCloserSF
  • []byte: FromBytes FromBytesSF
  • string: FromString FromStringSF
  • []string: FromStrings
  • Disk file: FromFile FromFileSF
  • Directory listing: FromDir
  • Recursive directory listing: FromDirWalk
  • External command output: FromCommand FromCommandSF
• Mapping and filtering primitives: Filter GenMap Map
• Sequence limiting functions: Skip SkipWhile Take TakeWhile
• Search function: FirstNonEmpty
• Piping iterator output through an external command: Pipe PipeSF
• Iterator chaining (sequential combination): Chain
• Iterator merging (parallel combination): Merge
• Output collectors that invoke the given iterator and write the result to various destinations:
  • string: String Join
  • []string: Strings
  • []byte: Bytes JoinBytes
  • io.Writer: WriteTo WriteSepTo
  • Disk file: WriteToFile WriteSepToFile
• Predicates and predicate combinators for use with Filter: Empty StartsWith EndsWith Not And AndNot Or OrNot
• Basic parsing supported via Parse function.

More examples:

• Naïve grep:

func main() {
	_, err := strit.FromReader(os.Stdin).
			Filter(regexp.MustCompile(os.Args[1]).Match).
			WriteSepTo(os.Stdout, "\n")

	if err != nil {
		os.Stderr.WriteString(err.Error() + "\n")
		os.Exit(1)
	}
}

• Recursively find all the filesystem entries matching the given regular expression:

func selectEntries(root string, re *regexp.Regexp) ([]string, error) {
	return FromDirWalk(root, nil).Filter(re.Match).Strings()
}

• Build a list of .flac files in the given directory, annotating each name with its corresponding track number from FLAC metadata:

func namesWithTrackNumbers(dir string) ([]string, error) {
	return strit.FromDir(dir, func(info os.FileInfo) bool { return info.Mode().IsRegular() }).
		Filter(strit.EndsWith(".flac")).
		GenMap(prependTrackNo).
		Strings()
}

func prependTrackNo(file []byte) ([]byte, error) {
	name := string(file)

	no, err := strit.FromCommand(exec.Command("metaflac", "--list", "--block-type=VORBIS_COMMENT", name)).
		FirstNonEmpty(func(s []byte) []byte {
			if m := match(s); len(m) == 2 {
				return m[1]
			}

			return nil
		}).
		String()

	if err != nil {
		return nil, err
	}

	if len(no) == 0 {
		return []byte("???: " + filepath.Base(name)), nil
	}

	return []byte(no + ": " + filepath.Base(name)), nil
}

var match = regexp.MustCompile(`tracknumber=([[:digit:]]+)$`).FindSubmatch

Project status

The project is in a beta state. Tested on Linux Mint 19.1, with Go version 1.12. Should also work on other platforms supported by Go runtime, but currently this is not very well tested.

License: BSD