Audacity  3.0.3
Public Member Functions | Public Attributes | List of all members
NyquistEffect::Tokenizer Struct Reference
Collaboration diagram for NyquistEffect::Tokenizer:
[legend]

Public Member Functions

bool Tokenize (const wxString &line, bool eof, size_t trimStart, size_t trimEnd)
 

Public Attributes

bool sl { false }
 
bool q { false }
 
int paren { 0 }
 
wxString tok
 
wxArrayStringEx tokens
 

Detailed Description

Definition at line 166 of file Nyquist.h.

Member Function Documentation

◆ Tokenize()

bool NyquistEffect::Tokenizer::Tokenize ( const wxString &  line,
bool  eof,
size_t  trimStart,
size_t  trimEnd 
)

Definition at line 1856 of file Nyquist.cpp.

1859 {
1860  auto endToken = [&]{
1861  if (!tok.empty()) {
1862  tokens.push_back(tok);
1863  tok = wxT("");
1864  }
1865  };
1866 
1867  for (auto c :
1868  make_iterator_range(line.begin() + trimStart, line.end() - trimEnd)) {
1869  if (q && !sl && c == wxT('\\')) {
1870  // begin escaped character, only within quotes
1871  sl = true;
1872  continue;
1873  }
1874 
1875  if (!sl && c == wxT('"')) {
1876  // Unescaped quote
1877  if (!q) {
1878  // start of string
1879  if (!paren)
1880  // finish previous token
1881  endToken();
1882  // Include the delimiter in the token
1883  tok += c;
1884  q = true;
1885  }
1886  else {
1887  // end of string
1888  // Include the delimiter in the token
1889  tok += c;
1890  if (!paren)
1891  endToken();
1892  q = false;
1893  }
1894  }
1895  else if (!q && !paren && (c == wxT(' ') || c == wxT('\t')))
1896  // Unenclosed whitespace
1897  // Separate tokens; don't accumulate this character
1898  endToken();
1899  else if (!q && c == wxT(';'))
1900  // semicolon not in quotes, but maybe in parentheses
1901  // Lisp style comments with ; (but not with #| ... |#) are allowed
1902  // within a wrapped header multi-line, so that i18n hint comments may
1903  // be placed before strings and found by xgettext
1904  break;
1905  else if (!q && c == wxT('(')) {
1906  // Start of list or sublist
1907  if (++paren == 1)
1908  // finish previous token; begin list, including the delimiter
1909  endToken(), tok += c;
1910  else
1911  // defer tokenizing of nested list to a later pass over the token
1912  tok += c;
1913  }
1914  else if (!q && c == wxT(')')) {
1915  // End of list or sublist
1916  if (--paren == 0)
1917  // finish list, including the delimiter
1918  tok += c, endToken();
1919  else if (paren < 0)
1920  // forgive unbalanced right paren
1921  paren = 0, endToken();
1922  else
1923  // nested list; deferred tokenizing
1924  tok += c;
1925  }
1926  else {
1927  if (sl && paren)
1928  // Escaped character in string inside list, to be parsed again
1929  // Put the escape back for the next pass
1930  tok += wxT('\\');
1931  if (sl && !paren && c == 'n')
1932  // Convert \n to newline, the only special escape besides \\ or \"
1933  // But this should not be used if a string needs to localize.
1934  // Instead, simply put a line break in the string.
1935  c = '\n';
1936  tok += c;
1937  }
1938 
1939  sl = false;
1940  }
1941 
1942  if (eof || (!q && !paren)) {
1943  endToken();
1944  return true;
1945  }
1946  else {
1947  // End of line but not of file, and a string or list is yet unclosed
1948  // If a string, accumulate a newline character
1949  if (q)
1950  tok += wxT('\n');
1951  return false;
1952  }
1953 }

References make_iterator_range(), paren, q, sl, tok, and tokens.

Referenced by NyquistEffect::Parse(), NyquistEffect::ParseChoice(), NyquistEffect::ParseFileExtensions(), NyquistEffect::ParseFileType(), NyquistEffect::ParseFileTypes(), and NyquistEffect::UnQuoteMsgid().

Here is the call graph for this function:
Here is the caller graph for this function:

Member Data Documentation

◆ paren

int NyquistEffect::Tokenizer::paren { 0 }

Definition at line 169 of file Nyquist.h.

Referenced by Tokenize().

◆ q

bool NyquistEffect::Tokenizer::q { false }

Definition at line 168 of file Nyquist.h.

Referenced by Tokenize().

◆ sl

bool NyquistEffect::Tokenizer::sl { false }

Definition at line 167 of file Nyquist.h.

Referenced by Tokenize().

◆ tok

wxString NyquistEffect::Tokenizer::tok

Definition at line 170 of file Nyquist.h.

Referenced by Tokenize().

◆ tokens

wxArrayStringEx NyquistEffect::Tokenizer::tokens

The documentation for this struct was generated from the following files:
make_iterator_range
IteratorRange< Iterator > make_iterator_range(const Iterator &i1, const Iterator &i2)
Definition: MemoryX.h:551
NyquistEffect::Tokenizer::tok
wxString tok
Definition: Nyquist.h:170
NyquistEffect::Tokenizer::paren
int paren
Definition: Nyquist.h:169
NyquistEffect::Tokenizer::q
bool q
Definition: Nyquist.h:168
NyquistEffect::Tokenizer::tokens
wxArrayStringEx tokens
Definition: Nyquist.h:171
NyquistEffect::Tokenizer::sl
bool sl
Definition: Nyquist.h:167