NyquistBase::Tokenizer Struct Reference

Collaboration diagram for NyquistBase::Tokenizer:

Public Member Functions
bool	Tokenize (const wxString &line, bool eof, size_t trimStart, size_t trimEnd)

Public Attributes
bool	sl { false }
bool	q { false }
int	paren { 0 }
wxString	tok
wxArrayStringEx	tokens

Detailed Description

Definition at line 200 of file NyquistBase.h.

Member Function Documentation

Tokenize()

bool NyquistBase::Tokenizer::Tokenize	(	const wxString &	line,
		bool	eof,
		size_t	trimStart,
		size_t	trimEnd
	)

Definition at line 1960 of file NyquistBase.cpp.

{
   auto endToken = [&] {
      if (!tok.empty())
      {
         tokens.push_back(tok);
         tok = wxT("");
      }
   };
 
   for (auto c :
        make_iterator_range(line.begin() + trimStart, line.end() - trimEnd))
   {
      if (q && !sl && c == wxT('\\'))
      {
         // begin escaped character, only within quotes
         sl = true;
         continue;
      }
 
      if (!sl && c == wxT('"'))
      {
         // Unescaped quote
         if (!q)
         {
            // start of string
            if (!paren)
               // finish previous token
               endToken();
            // Include the delimiter in the token
            tok += c;
            q = true;
         }
         else
         {
            // end of string
            // Include the delimiter in the token
            tok += c;
            if (!paren)
               endToken();
            q = false;
         }
      }
      else if (!q && !paren && (c == wxT(' ') || c == wxT('\t')))
         // Unenclosed whitespace
         // Separate tokens; don't accumulate this character
         endToken();
      else if (!q && c == wxT(';'))
         // semicolon not in quotes, but maybe in parentheses
         // Lisp style comments with ; (but not with #| ... |#) are allowed
         // within a wrapped header multi-line, so that i18n hint comments may
         // be placed before strings and found by xgettext
         break;
      else if (!q && c == wxT('('))
      {
         // Start of list or sublist
         if (++paren == 1)
            // finish previous token; begin list, including the delimiter
            endToken(), tok += c;
         else
            // defer tokenizing of nested list to a later pass over the token
            tok += c;
      }
      else if (!q && c == wxT(')'))
      {
         // End of list or sublist
         if (--paren == 0)
            // finish list, including the delimiter
            tok += c, endToken();
         else if (paren < 0)
            // forgive unbalanced right paren
            paren = 0, endToken();
         else
            // nested list; deferred tokenizing
            tok += c;
      }
      else
      {
         if (sl && paren)
            // Escaped character in string inside list, to be parsed again
            // Put the escape back for the next pass
            tok += wxT('\\');
         if (sl && !paren && c == 'n')
            // Convert \n to newline, the only special escape besides \\ or \"
            // But this should not be used if a string needs to localize.
            // Instead, simply put a line break in the string.
            c = '\n';
         tok += c;
      }
 
      sl = false;
   }
 
   if (eof || (!q && !paren))
   {
      endToken();
      return true;
   }
   else
   {
      // End of line but not of file, and a string or list is yet unclosed
      // If a string, accumulate a newline character
      if (q)
         tok += wxT('\n');
      return false;
   }
}

References make_iterator_range(), paren, q, sl, tok, tokens, and wxT().

Referenced by NyquistBase::Parse(), NyquistBase::ParseChoice(), NyquistBase::ParseFileExtensions(), NyquistBase::ParseFileType(), NyquistBase::ParseFileTypes(), and NyquistBase::UnQuoteMsgid().

Here is the call graph for this function:

Here is the caller graph for this function:

Member Data Documentation

paren

int NyquistBase::Tokenizer::paren { 0 }

Definition at line 204 of file NyquistBase.h.

Referenced by Tokenize().

q

bool NyquistBase::Tokenizer::q { false }

Definition at line 203 of file NyquistBase.h.

Referenced by Tokenize().

sl

bool NyquistBase::Tokenizer::sl { false }

Definition at line 202 of file NyquistBase.h.

Referenced by Tokenize().

tok

wxString NyquistBase::Tokenizer::tok

Definition at line 205 of file NyquistBase.h.

Referenced by Tokenize().

tokens

wxArrayStringEx NyquistBase::Tokenizer::tokens

Definition at line 206 of file NyquistBase.h.

Referenced by NyquistBase::Parse(), NyquistBase::ParseChoice(), NyquistBase::ParseFileExtensions(), NyquistBase::ParseFileType(), NyquistBase::ParseFileTypes(), Tokenize(), and NyquistBase::UnQuoteMsgid().

---

The documentation for this struct was generated from the following files:

• NyquistBase.h
• NyquistBase.cpp