Jtdylktuy

Assume I have a macro like

defmyMacro#1{<some stuff>}

And I am calling it like this

myMacro{There are some arguments in here g}

How can I iterate over each single token in the #1 argument inside myMacro? So basically I want to know how I can iterate over a list whose delimiter is not a comma or any other character but rather the token boundaries as applied by TeX. Note that the argument may contain control sequences that are undefined so they must not be expanded.

Example of what I mean:

defmyMacro#1{<iterate over all tokens>|string<current token>|}

myMacro{A test}

which should result in

|A|| ||test|

It is important to note that I also care about spaces so they shouldn't be gobbled away. Also I don't want to execute any code outside of mymacro in order for this to work (e.g. changing the catcode of spaces before calling myMacro).

As I am really into understanding how such a thing works I'd appreciate if you could also explain how and why your provided code works :)

My attempt at this was

defiterate#1{%

    tokenGrabber#1relax<!;!>%

}

deftokenGrabber#1#2<!;!>{%

    |string#1|%

    noexpandarg%

    IfStrEq{#2}{relax}{%

    }{%

        tokenGrabber#2<!;!>%

    }%

}

But this gobbles away spaces and it produces an error for empty inputs or inputs ending with a space.

documentclass{article}



makeatletter

defendtest{test!!!!}

deftest#1{%

par bigskiptextbf{TESTING:} texttt{detokenize{#1}}par

testzz#1endtest}



deftestzz{afterassignmenttestzzzlettmp= }



deftestzzz{%

ifxtmpendtest

else texttt{meaningtmp}par

expandaftertestzz

fi

}



begin{document}



test{123}



test{There are some arguments in here g}



test{ a+ {x sqrt{frac}}}



end{document}

Note that this mechanism consumes the supplied list using let (the form with = and exactly one space is important so it does not drop spaces or = in the input) that makes it easy to detect spaces and braces but for example it only captures the meaning of the token, it can not distinguish { from bgroup nor can it distinguish between any undefined commands, or show which name was used, zzzfoo undefined etc will all appear the same in the loop, as undefined.

For similar reasons, you can not re-construct anything equivalent to the original input from within the loop. given frac{a}{b} you get essentially fracbgroup aegrupbgroup begroup from which it isn't possible in general to reconstruct a working fraction.

So... whether these restrictions matter depend on the intended use of the loop.

If you need it for debugging, it's a one-liner:

documentclass{article}

usepackage{xparse}



ExplSyntaxOn

NewDocumentCommand{test}{m}

 {

  tl_show_analysis:n { #1 }

 }

ExplSyntaxOff



begin{document}



test{123}



test{There are some arguments in here g}



test{ a+ {x sqrt{frac}}}



end{document}

If you run it with pdflatex -interaction=nonstopmode, the console will show

The token list contains the tokens:

>  1 (the character 1)

>  2 (the character 2)

>  3 (the character 3).

<recently read> }



l.13 test{123}



The token list contains the tokens:

>  T (the letter T)

>  h (the letter h)

>  e (the letter e)

>  r (the letter r)

>  e (the letter e)

>    (blank space  )

>  a (the letter a)

>  r (the letter r)

>  e (the letter e)

>    (blank space  )

>  some (control sequence=undefined)

>  a (the letter a)

>  r (the letter r)

>  g (the letter g)

>  u (the letter u)

>  m (the letter m)

>  e (the letter e)

>  n (the letter n)

>  t (the letter t)

>  s (the letter s)

>    (blank space  )

>  in (control sequence=mathchar"3232=12850)

>  here (control sequence=undefined)

>  g (the letter g).

<recently read> }



l.15 test{There are some arguments in here g}



The token list contains the tokens:

>    (blank space  )

>  a (the letter a)

>  + (the character +)

>    (blank space  )

>  { (begin-group character {)

>  x (the letter x)

>    (blank space  )

>  sqrt (control sequence=macro:->protect sqrt  )

>  { (begin-group character {)

>  frac (control sequence=macro:#1#2->{begingroup #1endgroup over #2})

>  } (end-group character })

>  } (end-group character }).

<recently read> }



l.17 test{ a+ {x sqrt{frac}}}

Taking the expandable code from l3tl and recoding in classical style, we might do something like

catcode`@=11 %



chardeftl@exp@end=0 %



longdef@firstoftwo#1#2{#1}

longdef@secondoftwo#1#2{#2}

longdef@secondofthree#1#2#3{#2}

longdef@gobble#1{}



longdeftl@if@empty#1{%

  expandafterifxexpandafterrelaxdetokenize{#1}relax

    expandafter@secondofthree

  fi

  @secondoftwo

}



longdeftl@if@head@N#1{%

  ifcat

    iffalse{fitl@if@head@N@aux?#1 }%

    expandafter@gobbleexpandafter{expandafter{string#1?}}%

    **%

    expandafter@firstoftwo

  else

    expandafter@secondoftwo

  fi

}

longdeftl@if@head@N@aux#1 {%

  expandaftertl@if@emptyexpandafter{@gobble#1}{^}{}%

  expandafter@gobbleexpandafter{iffalse}fi

}



longdeftl@if@head@group#1{%

  ifcatexpandafter@gobbleexpandafter{expandafter{string#1?}}**%

    expandafter@secondoftwo

  else

    expandafter@firstoftwo

  fi

}



defq@act@mark{q@act@mark}

defq@act@stop{q@act@stop}



longdeftl@act#1#2#3#4#5{%

  ifnumiffalse{fi`}=z@fi

  tl@act@loop#5q@act@markq@act@stop

  {#4}#1#2#3%

  tl@act@result{}%

}

longdeftl@act@loop#1q@act@stop{%

  tl@if@head@N{#1}

    {tl@act@normal}

    {%

      tl@if@head@group{#1}

        {tl@act@group}

        {tl@act@space}%

    }%

  #1q@act@stop

}

longdeftl@act@normal#1#2q@act@stop#3#4{%

  ifxq@act@mark#1expandaftertl@act@endfi

  #4{#3}#1%

  tl@act@loop#2q@act@stop

  {#3}#4%

}

longdeftl@act@group#1#2q@act@stop#3#4#5{%

  #5{#3}{#1}%

  tl@act@loop#2q@act@stop

  {#3}#4#5%

}

expandafterlongexpandafterdefexpandafter

  tl@act@spacespace#1q@act@stop#2#3#4#5{%

    #5{#2}%

    tl@act@loop#1q@act@stop

    {#2}#3#4#5%

  }

longdeftl@act@end#1tl@act@result#2{%

  ifnum`{=z@}fi

  tl@exp@end

  #2%

}



longdefiterate#1{%

  unexpandedexpandafter{%

    romannumeraltl@act

      tl@iterate@normal

      tl@iterate@group

      tl@iterate@space

      { }

      {#1}%

  }%

}

longdeftl@iterate@normal#1#2{tl@iterate@action{string#2}}

longdeftl@iterate@group#1#2{tl@iterate@action{{detokenize{#2}}}}

longdeftl@iterate@space#1{tl@iterate@action{ }}

longdeftl@iterate@action#1#2tl@act@result#3{%

  #2%

  tl@act@result{#3|#1|}%

}



catcode`@=12 %



iterate{There are some arguments in here g}



bye

(This uses e-TeX, but that can be avoided.)

The basic idea is to grab the token list and examine the first token before branching and handling as required. I've not done it, but recursion inside groups is doable. Notice that all brace groups become { ... } (unavoidable in expandable code).