For the back burner: Create .shuf files in a temporary directory during training about marian-dev

[emjotde]

Good point. So, temporary file then? I think kenlm does something that deletes the file right away, and we should specify the temporary directory with T assuming "." by default.

[kpu]

Assuming you don't care about Windows support (which I got from LGPL code), making a temporary file is pretty easy.

// This function has a windows port, hence the separation.
int
mkstemp_and_unlink(char *tmpl) {
  int ret = mkstemp(tmpl);
  if (ret != -1) {
    UTIL_THROW_IF(unlink(tmpl), ErrnoException, "while deleting delete " << tmpl);
  }
  return ret;
}
int MakeTemp(const std::string &base) {
  std::string name(base);
  name += "XXXXXX";
  name.push_back(0);
  int ret;
  UTIL_THROW_IF(-1 == (ret = mkstemp_and_unlink(&name[0])), ErrnoException, "while making a temporary based on " << base);
  return ret;
}
// If it's a directory, add a /.  This lets users say -T /tmp without creating
// /tmpAAAAAA
void NormalizeTempPrefix(std::string &base) {
  if (base.empty()) return;
  if (base[base.size() - 1] == '/') return;
  struct stat sb;
  // It's fine for it to not exist.
  if (-1 == stat(base.c_str(), &sb)) return;
  if (
#if defined(_WIN32) || defined(_WIN64)
    sb.st_mode & _S_IFDIR
#else
    S_ISDIR(sb.st_mode)
#endif
    ) base += '/';
}

[jgwinnup]

I've run into this issue already - took me a bit to figure why those two systems decided not to converge...

[emjotde]

It's high up on my TODO list.

[kpu]

I took a look at the code. It's easy to get a FILE * or an int fd but harder to get an iostream as a temporary deleted file. That said, it's just readlng and writing lines so it doesn't need iostreams. Would you be happy if this changed to FILE * apis?

[kpu]

For what it's worth, it's faster if you size the input (or read everything into a buffer with binary increasing size) and sort string_view.

#include <iostream>
#include <boost/utility/string_view.hpp>
//#include <string_view>
#include <vector>

void SplitLines(boost::string_view in, std::vector<boost::string_view> &out) {
  out.clear();
  while (!in.empty()) {
    boost::string_view::size_type found = in.find('\n');
    out.push_back(in.substr(0, found + 1));
    in.remove_prefix(found + 1);
  }
}

int main() {
  const std::streamsize kBuf = 1048576;
  char buf[kBuf];
  std::cin.read(buf, kBuf);
  std::streamsize got = std::cin.gcount();
  if (got >= kBuf) {
    std::cerr << "Too big\n";
    return 1;
  }
  std::vector<boost::string_view> strings;
  SplitLines(boost::string_view(buf, got), strings);

  std::random_device rd;
  std::shuffle(strings.begin(), strings.end(), std::mt19937(rd()));
  for (const boost::string_view &i : strings) {
    std::cout << i;
  }
}

[emjotde]

I use the filestreams because they are there, not too attached to them in that particular case. I would like to keep them around due to the filtering stream functionalty.

[emjotde]

@kpu Kenneth, what are the includes for this?

    void NormalizeTempPrefix(std::string &base) {
      if (base.empty())
        return;
      if (base[base.size() - 1] == '/')
        return;
      struct stat sb;
      // It's fine for it to not exist.
      if (-1 == stat(base.c_str(), &sb))
        return;
      if (S_ISDIR(sb.st_mode))
        base += '/';
    }

The stat struct and S_ISDIR cause compilation errors, I assume missing headers?

[kpu]

#include <string>
#include <sys/types.h>
#include <sys/stat.h>
#if defined(__MINGW32__)
#include <windows.h>
#include <unistd.h>
#elif defined(_WIN32) || defined(_WIN64)
#include <windows.h>
#include <io.h>
#else
#include <unistd.h>
#endif

[emjotde]

@kpu thanks.