The astir from lexected

Faulty documentation link in README

The link inside the README to the documentation is faulty.

Is it possible to switch tokenizers in a parser?

I'm trying to write a parser when the syntax is generally whitespace delimited tokens; however, in certain contexts, the whitespace is significant; examples such as quoted strings, but I also have instances where a block is delimited by the tokens:

action ... {
  /* start of whitespace significant block
   * where we read everything until the closing brace
   */
}

Would I perhaps need two tokenizers? One that basically looks for these special conditions, and then another that feeds these in and generates a more specific token stream?

Am trying to rewrite a parser using yacc, and it has three "parsing modes" in what appears to be a character-by-character scanning function.

Compilation Errors

First, std::exception doesn't have a constructor taking const char *; I fixed this by subclassing std::runtime_error instead, as suggested at https://en.cppreference.com/w/cpp/error/exception/exception:

The Microsoft implementation includes non-standard constructors taking strings thus allowing instances to be thrown directly with a meaningful error message. The nearest standard equivalents are std::runtime_error or std::logic_error.

Running into (added flag to treat as warning):
MyTokenizer.cpp: warning: ISO C++ forbids taking the address of a bound member function to form a pointer to member function. Say &MyTokenizer::MyTokenizer::actionRegister105 [-fpermissive]

And then, trying to use the example documentation to make my stuff run...

	TextFileStream tfs("Testfile");

	MyTokenizer::MyTokenizer myTokenizer;
	auto tokenizedStream = myTokenizer.processStreamWithIgnorance(tfs);

	auto processedStream(tokenizedStream);
	
	MyParser::MyParser myParser;
	auto parsedStream = myParser.parseStream(processedStream);

gives

main.cpp: In function ‘int main()’:
main.cpp:77:44: error: cannot convert ‘std::__cxx11::list<std::shared_ptr<MyTokenizer::MyTokenizerTerminal>, std::allocator<std::shared_ptr<MyTokenizer::MyTokenizerTerminal> > >’ to ‘ProductionStream<MyTokenizer::MyTokenizerTerminal>&’
   77 |  auto parsedStream = myParser.parseStream(processedStream);
      |                                            ^~~~~~~~~~~~~~~
      |                                            |
      |                                            std::__cxx11::list<std::shared_ptr<MyTokenizer::MyTokenizerTerminal>, std::allocator<std::shared_ptr<MyTokenizer::MyTokenizerTerminal> > >
In file included from MyParser.h:15,
                 from main.cpp:15:
Parser.h:71:133: note:   initializing argument 1 of ‘std::__cxx11::list<std::shared_ptr<_Tp> > Parser<InputStreamType, OutputProductionType>::parseStream(InputStreamType&) [with InputStreamType = ProductionStream<MyTokenizer::MyTokenizerTerminal>; OutputProductionType = Production]’
   71 | inline std::list<std::shared_ptr<OutputProductionType>> Parser<InputStreamType, OutputProductionType>::parseStream(InputStreamType& rs) {
      |                                                                                                                    ~~~~~~~~~~~~~~~~~^~

It seems like perhaps ListProductionStream template class is meant to be used somewhere, but I haven't been able to figure out what needs changing.

Building with g++ 9.3.0 and current master.

lexected / astir Goto Github PK

astir's People

Contributors

Stargazers

Watchers

astir's Issues

Faulty documentation link in README

Is it possible to switch tokenizers in a parser?

Compilation Errors

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

Jobs