NyquistEffect::Tokenizer Struct Reference

Collaboration diagram for NyquistEffect::Tokenizer:

Public Member Functions
bool	Tokenize (const wxString &line, bool eof, size_t trimStart, size_t trimEnd)

Public Attributes
bool	sl { false }
bool	q { false }
int	paren { 0 }
wxString	tok
wxArrayStringEx	tokens

Detailed Description

Definition at line 185 of file Nyquist.h.

Member Function Documentation

Tokenize()

bool NyquistEffect::Tokenizer::Tokenize	(	const wxString &	line,
		bool	eof,
		size_t	trimStart,
		size_t	trimEnd
	)

Definition at line 1998 of file Nyquist.cpp.

{
   auto endToken = [&]{
      if (!tok.empty()) {
         tokens.push_back(tok);
         tok = wxT("");
      }
   };
 
   for (auto c :
        make_iterator_range(line.begin() + trimStart, line.end() - trimEnd)) {
      if (q && !sl && c == wxT('\\')) {
         // begin escaped character, only within quotes
         sl = true;
         continue;
      }
 
      if (!sl && c == wxT('"')) {
         // Unescaped quote
         if (!q) {
            // start of string
            if (!paren)
               // finish previous token
               endToken();
            // Include the delimiter in the token
            tok += c;
            q = true;
         }
         else {
            // end of string
            // Include the delimiter in the token
            tok += c;
            if (!paren)
               endToken();
            q = false;
         }
      }
      else if (!q && !paren && (c == wxT(' ') || c == wxT('\t')))
         // Unenclosed whitespace
         // Separate tokens; don't accumulate this character
         endToken();
      else if (!q && c == wxT(';'))
         // semicolon not in quotes, but maybe in parentheses
         // Lisp style comments with ; (but not with #| ... |#) are allowed
         // within a wrapped header multi-line, so that i18n hint comments may
         // be placed before strings and found by xgettext
         break;
      else if (!q && c == wxT('(')) {
         // Start of list or sublist
         if (++paren == 1)
            // finish previous token; begin list, including the delimiter
            endToken(), tok += c;
         else
            // defer tokenizing of nested list to a later pass over the token
            tok += c;
      }
      else if (!q && c == wxT(')')) {
         // End of list or sublist
         if (--paren == 0)
            // finish list, including the delimiter
            tok += c, endToken();
         else if (paren < 0)
            // forgive unbalanced right paren
            paren = 0, endToken();
         else
            // nested list; deferred tokenizing
            tok += c;
      }
      else {
         if (sl && paren)
            // Escaped character in string inside list, to be parsed again
            // Put the escape back for the next pass
            tok += wxT('\\');
         if (sl && !paren && c == 'n')
            // Convert \n to newline, the only special escape besides \\ or \"
            // But this should not be used if a string needs to localize.
            // Instead, simply put a line break in the string.
            c = '\n';
         tok += c;
      }
 
      sl = false;
   }
 
   if (eof || (!q && !paren)) {
      endToken();
      return true;
   }
   else {
      // End of line but not of file, and a string or list is yet unclosed
      // If a string, accumulate a newline character
      if (q)
         tok += wxT('\n');
      return false;
   }
}

References make_iterator_range(), paren, q, sl, tok, tokens, and wxT().

Referenced by NyquistEffect::Parse(), NyquistEffect::ParseChoice(), NyquistEffect::ParseFileExtensions(), NyquistEffect::ParseFileType(), NyquistEffect::ParseFileTypes(), and NyquistEffect::UnQuoteMsgid().

Here is the call graph for this function:

Here is the caller graph for this function:

Member Data Documentation

paren

int NyquistEffect::Tokenizer::paren { 0 }

Definition at line 188 of file Nyquist.h.

Referenced by Tokenize().

q

bool NyquistEffect::Tokenizer::q { false }

Definition at line 187 of file Nyquist.h.

Referenced by Tokenize().

sl

bool NyquistEffect::Tokenizer::sl { false }

Definition at line 186 of file Nyquist.h.

Referenced by Tokenize().

tok

wxString NyquistEffect::Tokenizer::tok

Definition at line 189 of file Nyquist.h.

Referenced by Tokenize().

tokens

wxArrayStringEx NyquistEffect::Tokenizer::tokens

Definition at line 190 of file Nyquist.h.

Referenced by NyquistEffect::Parse(), NyquistEffect::ParseChoice(), NyquistEffect::ParseFileExtensions(), NyquistEffect::ParseFileType(), NyquistEffect::ParseFileTypes(), Tokenize(), and NyquistEffect::UnQuoteMsgid().

---

The documentation for this struct was generated from the following files:

• Nyquist.h
• Nyquist.cpp