WMLxGettext 2.x¶

–domain=DOMAIN_NAME¶

With the option --domain, wmlxgettext will know wich is the # textdomain used by your wesnoth add-on. For example, if your _main.cfg will have:

[textdomain]
   name="wesnoth-xyz"
   path="data/add-ons/xyz/translations"
[/textdomain]

This is what you have to write into the --domain parameter:

--domain=wesnoth-xyz

–directory=ADDON_DIRECTORY¶

With the option --directory, wmlxgettext will know the starting path where all following files/scandirs should be searched. This is a fake example for windows:

--directory=c:\games\wesnoth\userdata\data\add-ons\YOUR_ADDON_DIRECTORY

–recursive¶

If --recursive option is used, wmlxgettext will scan recursively the directory typed on the --directory option and collect all .cfg and .lua files automaticly:

./wmlxgettext --domain=domain_name --directory=/home/user/games/wesnoth/userdata/add-ons/Invasion_from_the_Unknown --recursive -o ./file.po

In the example showed above, infact, wmlxgettext will watch the directory /home/user/games/wesnoth/userdata/add-ons/Invasion_from_the_Unknown and it will collect, recursively, all .cfg / .lua files inside that directories (and subdirectories).

-o [OUTPUT_FILE]¶

If you use this option, wmlxgettext will actually create a .po file, saving it as [OUTPUT_FILE].

The -o options accepts:

• either a file name with absolute path
• or a file name with relative path (for example: ./output.po)
• or it could setted to “-” (wmlxgettext -o - ...) to write output to stdout

Also the parameter --directory discussed before can accept both an absolute path or a relative path starting from the current working directory (for exaple: --directory=. will assign to the --directory option the path of the current working directory).

Storing translatable strings¶

When a translatable string will be found, it will be added in the current node.

Each time a new translatable string found, a new WmlNodeSentence object will be added to the current WML node (but only if the current domain is equal to the addon domain).

A WmlNodeSentence will have those properties:

• sentence: the translatable string (text)
• ismultiline: True if it is a multi-line string
• lineno: line number where the translatable string was located. (if multi-line, the line number where the the string started).
• lineno_sub: The name is ambigous. This parameter (integer value) is a progressive value, expecially useful when more than one string was stored in the same line of the same file.
• overrideinfo: None or ‘string’. If # po-override: overrideinfo directive was found, the overrideinfo will be stored here.
• self.addedinfo: None or list of strings. If one or more # po: addedinfo directive(s) found, the info will be added in this list.

PoCommentedString data infos¶

Now it is time to explain all data infos contained in a PoCommentedString:

• sentence = translatable string text.
• wmlinfos = list of wmlinfos.
• addedinfos = infos added with # po: something directives
• finfos = list of files and line number where any occurence of the string was found.

• orderid = it is an (unique) tuple of three values:

  • fileno: the file where the string was found the first time (file with lower fileno id value.
  • lineno: the line numeber, in fileno, where the string was found the first time.
  • lineno_sub: line_sub is a progressive value. It will be helpful to assign the correct order of the sentences, when two or more sentences were stored in the same file and in the same line.

• ismultiline = True if it is a multiline string.

The orderid tuple is very important, becouse, when wmlxgettext must write down all PoCommentedString objects from the “postring” dictionary to the .po file, it must print them in the right order (and not randomly):

# ./wmlxgettext:196
for posentence in sorted(sentlist.values(), key=lambda x: x.orderid):

When converting a WmlNodeSentence object to a PoCommentedString object, WmlNode.assemble_orderid create the tuple of three values to pass to PoCommentedString.orderid parameter:

• fileno (first value) –> comes from the WmlNode object containing the WmlNodeSentence.
• lineno and lineno_sub –> comes from the single WmlNodeSentence.

PoCommentedString and WmlNode both have a wmlinfos list, but they are conceptually different:

• WmlNode.wlinfos contains single pieces of infos captured on the WML node (example: speaker=speaker_name or id=value).
• Those single pieces must be assembled (with WmlNode.assemble_wmlinfo) to create a single PoCommentedString.wmlinfos element.
• So, when converting a WmlNodeSentence to a PoCommentedString, all wmlinfos contained in the WmlNodeSentence will add a single PoCommentedString wmlinfo.

About overrideinfo and addedinfos¶

A WmlNodeSentence object can contain an override info. This will happen if # po-override: overrideinfo directive was found in the WML/Lua file.

The override info, if exists, will be written directly on PoCommentedString as a PoCommentedString.wmlinfos element. WmlNode wmlinfos list will be ignored for that WmlNodeSentence and assemble_wmlinfos will not executed on that single conversion.

“Addedinfos”, instead, behave in the same way both in WmlNodeSentence and in PoCommentedString objects. Those are additional infos to display to translator. If a WmlNodeSentence object contains elements in addedinfos list, those elements will be added in the PoCommentedString-addedinfos list. This will happen if one or more # po: addedinfo directive(s) was found in WML/Lua file.

Create a new dictionary key or update an existing one?¶

So, when closing a WML node, all WmlNodeSentence objects contained in that WmlNode object will be converted to temporary PoCommentedString objects.

Those temporary PoCommentedString objects will be not immediately stored in the dictionary, since the dictionary must contain one instance only of any sentence.

This why all temporary PoCommentedString objects created by WmlNode.nodesentence_to_posentence function will be “scanned”.

• If a temporary PoCommentedString objects contains an instance of an already existing translatable string, the dictionary key will be updated (no new key will be added). The function update_with_commented_string of the PoCommentedString object contained in the dictionary key will be executed to update that PoCommentedString object.
• If a temporary PoCommentedString object contains a new translatable string not previously stored in the dictionary, this object will be simply added in the dictionary

# ./pywmlx/nodemanip.py:15
def _closenode_update_dict(podict):
    if nodes[-1].sentences is not None:
        for i in nodes[-1].sentences:
            posentence = podict.get(i.sentence.lower())
            if posentence is None:
                podict[i.sentence.lower()] = (
                    nodes[-1].nodesentence_to_posentence(i) )
            else:
                posentence.update_with_commented_string(
                    nodes[-1].nodesentence_to_posentence(i) )

As you can see this check is actually performed inside the ./pywmlx/nodemanip.py module, explained in the next paragraph.

Storing a new WML node¶

nodemanip stores all WML nodes in a list, and not in a real tree structure. This becouse, as explained in the very beginning of this chapter, WML language is structured by nested tag, where any new child tag must be closed before its parent tag. Coming back to the WML sample code showed on the beginning of this chapter (with added comments):

# [scenario] is the first tag encountered in the WML.
# [scenario] tag is the parent of all following (nexted) WML tags and
#            it will be closed after all its child tags
[scenario]
   id=scenario_id
   name= _ "scenario name (translatable)"
   map_data = ...

   # [objective] tag does not have childs, so it will be closed immediately
   # after its opening
   [objective]
       description= _ "objective text (translatable)"
   [/objective]

   # again... [event] tag will have a child: the tag [message].
   # the tag [message] must be closed before the parent [event] tag.
   [event]
     name = "start"

     [message]
        message = _"I am saying something (translatable)"
        speaker = id_of_the_speaker
     [/message]

   [/event]
[/scenario]

So why WML nodes can be stored in a list:

• everytime a new node is added, we can simply add an element in the list. The last item in the list is the last WML node opened.

• the last node in list, is the current node and it is the node we will must close before all other nodes in memory

• when the current node (last node in list) is closed, it will be removed by the list, so the last item on the list (the new current node) will be the parent node, for example, look at the WML sample code above:

  • when [event] tag is opened a new [event] node is added in node list.
  • when [message] tag is opened, a new [message] node is added in node list.
  • when [/message] found, then the [message] node is removed from list and the [event] tag will be now the last node in list (current node)

Coming back to the already displayed flow chart, we could notice that there is a special ROOT node that it will be created by nodemanip. It is a fake node required to avoid memory leaks and it will store all translatable strings stored outside any tag (for example a translatable string inside a macro definition). All captured wmlinfos in ROOT node will be ignored, since autowml is setted to False.

ROOT node is also special becouse, when created, cannot be deleted until the end of the WML file reached.

Deleting a WML node from memory¶

Clearing a WML node is the most important work performed by nodemanip module since, before actually clearing the node from memory, we must verify if the WML code is correctly written:

• the closing tag [/tagname] must be equal to the last [tagname] in list (current WML node). Else, a critical error must be returned (calling wmlerr function - wmlxgettext should stop execution)
• a critical error should be also returned when a close tag is unexpected at all, since no tags are openend (the list of WML node is still empty or the current WML node is the ROOT node).

All those checks is done by the closenode function on nodemanip module:

# ./pywmlx/nodemanip.py:73
def closenode(closetag, mydict, lineno):

But, even if the closing tag [/tagname] is the expected one, nodemanip module does not immediately clear the node from the nodes’ list.

# ./pywmlx/nodemanip.py:15
def _closenode_update_dict(podict):
    if nodes[-1].sentences is not None:
        for i in nodes[-1].sentences:
            posentence = podict.get(i.sentence.lower())
            if posentence is None:
                podict[i.sentence.lower()] = (
                    nodes[-1].nodesentence_to_posentence(i) )
            else:
                posentence.update_with_commented_string(
                    nodes[-1].nodesentence_to_posentence(i) )

As previously explained in Converting WmlNodeSentence to PoCommentedString and all its subparagraphs, infact, nodemanip, before closing the node:

• it will convert all WmlNodeSentence objects contained into the pending WmlNode object, before removing it from the list.
• all the PoCommentedString temporary values created by the conversion will be used to update the dictionary (more details about this process can be found at Converting WmlNodeSentence to PoCommentedString and all its subparagraphs).

Adding a new translatable string into the current WML node¶

Every translatable string found inside a WML file must be stored in the current WML node as a WmlNodeSentence object.

Every time a new WML file is opened, the node list _nodes on nodemanip module is empty (or better, is None).

Usually, a ROOT WML node is created before creating the first actual WML node. This allows to store translatable strings located outside any tag.

But it could happen that a translatable string is found when node list is still empty (and when ROOT node does not still exist).

This why the nodemanip.addNodeSentence function, before trying to add the translatable string in current WML node, checks if the node list is not empty (or better, is not None). If the node list is empty, it creates the ROOT WML node and add the translatable string into that node.

What nodemanip does when end of WML file reached¶

When end of WML reached, nodemanip module will run its own closefile() function. There are three possible cases, as showed in the following flow chart:

Since the root node is not a standard WML node, and since it cannot be closed by any tag, nodemanip needs to explicitly explore it when the end of the WML file reached (otherwise the translatable strings stored in root node will be not added in dictionary).

Standard States¶

Standard states works exactly as previously explained:

The regexp is verified through re.match, so it maches only if the rule is True at the very start of the line. If it matches, than run() is executed.

Run() returns a pair of values (tuple):

• xline: the non-consumed part of the line. If the line is to be considered consumed, then xline will be setted to None
• nextstate: label of the next state to go. Usually it is ‘wml_idle’ or ‘lua_idle’ since the parsing is recursive.

If the regexp does not match, the iffail state will be reached. Usually the iffail is equal to the “next state”. See State Sequence

Always-Run States¶

Always-run states are special states with regexp = None

Unlike standard states, an always-run state will always execute its own run() function. An example of always-run state is ‘wml_idle’ state.

An always-run state does not actually require the iffail parameter. This is why always-run states have iffail = None

IDLE and FINAL States¶

Both IDLE and FINAL states check if there is a pending string, and if it is so, pending string will be stored in memory.

• WML: checks pymlx.state.machine._pending_wmlstring. If pymlx.state.machine._pending_wmlstring is None there is no pending WML string to store
• Lua: checks pymlx.state.machine._pending_luastring. If pymlx.state.machine._pending_luastring is None there is no pending Lua string to store

Both Lua and WML pending strings, before actually storing its own value, perform some cheks:

• verify if it is a translatable string
• verify if the current domain is the same of the addon domain name
• if so, it fixes the string for storage, and then store it

However WML pending string is stored in a very different way then Lua pending string:

• Lua pending string is directly stored, as a PoCommentedString, in the “posentence dictionary”.
• WML pending string is, instead, stored in the current WML node as a WmlNodeSentence. Only when the current WML node will be closed, all WmlNodeSentence objects contained in the node will be stored in the “posentence dictionary”. (See: The nodemanip module and Converting WmlNodeSentence to PoCommentedString)

WmlFinalState always return the pair (xline, 'wml_idle') while LuaFinalState always return the pair (xline, 'lua_idle'), where xline is setted to None in both cases. As previously explained, infact, when xline is None, the line is considered completely consumed and the statemachine will read the next line of the file.

Finally, the ‘lua_final’ state perform another action, but it will be explain later. See About storing the last Lua function name.

Capture String States¶

When a string (translatable or not) is found, then the regexp of the proper “Capture String” state matches. Captured string will be stored as pymlx.state.machine._pending_wmlstring (WML string), or as pymlx.state.machine._pending_luastring (Lua string).

Now it is the time to discuss deeply about those capturing string states.

Capture String: WML States¶

WML accepts only one syntax:

_ "translatable_string"

Only two states, then, required to capture strings:

# ./pywmlx/state/wml_states:161
class WmlStr01:
    # ...
# ./pywmlx/state/wml_states:190
class WmlStr10:

More in details:

• WmlStr01 (‘wml_str01’): This state capture a single-line string and also capture the FIRST LINE of a WML multiline string.

  • If it is a single line string, the string will be stored in pymlx.state.machine._pending_wmlstring. (Change to ‘wml_idle’ state).
  • If the closing quote " does not exist (multiline string) , then the matched string will be saved in pymlx.state.machine._pending_wmlstring. Following lines will be added to the pending string by the WmlStr10 State (change to ‘wml_str10’ state)

• WmlStr10 (‘wml_str10’): All following lines of the multiline string will be added to pending string by this state until the closing quote " will be finded. This states recursively come back to itself, and, when the string ends, state will be changed again to ‘wml_idle’

Capture String: Lua States¶

Unlike WML, Lua accepts three different syntaxes:

"string: type 1"

'string: type 2'

[[string: type 3]]

The third way (mostly suggested for multi-line lua strings) is even more flexible than showed in the sample code above, as you can type any number of equals symbols (from 0 to n) between the two brackets [[ and ]]

Note

In the example above, we wrote [[string: type3]], since it is the most common way of defining a bracketed lua string, but we could also put any number of equals symobols between brackets.

For example, we could have printed [==[string: type3]==] placing the equal symbol two times. In that case, both opening and closing delimiter must use the same amount of equal symbols.

Coming back to wmlxgettext, we shoud now notice that all this flexibility allowed by the lua language (three ways to identify a string) means “more states are required”. There are, infact, seven states this time:

# ./pywmlx/state/lua_states:71 (syntax "1": single-line or start multiline)
class LuaStr01:
    # ...
# ./pywmlx/state/lua_states:173 (syntax "1": multiline)
class LuaStr10:
    # ...
# ./pywmlx/state/lua_states:99 (syntax "2": single-line or start multiline)
class LuaStr02:
    # ...
# ./pywmlx/state/lua_states:193 (syntax "2": multiline)
class LuaStr20:
    # ...
# ./pywmlx/state/lua_states:127 (syntax "3": single-line ONLY)
class LuaStr03:
    # ...
# ./pywmlx/state/lua_states:149 (syntax "3": start multiline)
class LuaStr03o:
    # ...
# ./pywmlx/state/lua_states:211 (syntax "3": multiline [from line 2])
class LuaStr30:
    # ...

This time the flow chart is not so easy to understand at a first sight, so it requires a little explaination:

• Boxes/Ellipses:

  • green -> always-run states (green arrow rule applied)
  • orange -> used for “Next State”, for a better look
  • red (LuaStr10 and LuaStr20): LuaStr10 and LuaStr20 are recursive standard states. They can go back to theirself, until the end of the multi-line string is matched (when the multi-line string ends, ‘lua_idle’ state will be reached) (no arrow rule: all arrows are black)
  • red (LuaStr30): LuaStr30 is indeed an always-run state, but it acts like a recursive standard state. The regular expression evaluation is moved in the run() function since the regexp rule is calculated on runtime. If the regexp doesn’t match (current line of code does not end the multiline string) than LuaStr30 comes back to itself (recursive state). If the regexp does match, the multi-line string finished, and LuaStr30 goes to LuaIdleState.
  • grey -> standard states (black, blue and dotted blue arrow rules applied)
  • purple (ellipse) -> LuaStr30 can find (or not) the ]==] symbol. Purple ellipses shows what happen if ]==] was found and if ]==] was NOT found (see where the black arrows will go).

• Arrow rules (when applied):

  • green -> LuaStr31 is an always-run state. LuaStr31 will always come back to LuaStr30 state
  • blue -> when the state finds what it is searching, go to the state pointed by blue arrow
  • blue (dotted) -> LuaStr01 and LuaStr02 regex rule can match both a single-line string AND the start of a multi-line string. If the a multi-line string matched, than go to the state pointed by the dotted blue arrow instead of the standard blue arrow
  • black -> When the regex rule of the state fails (the state does not find what it is searching). [except for red boxes]

About storing the last Lua function name¶

Unlike the WML states, there isn’t any Lua state that captures lua infos. The only extra info that could be auto-cached inside a lua code is the name of the last function opened / defined.

This kind of search required to use a specific regexp search, using re.search instead of re.match. Unlike all other searches, infact, we need to capture function name at any point of the line we are parsing, or the regexp will not work properly.

But, as explained at the beginning of this page, the state machine relies on re.match (best performance) to verify the regexp rule of every state. For this reason, LuaFinalState searches by itself if there is a function name somewhere, and, if so, stores the value into pywmlx.state.machine._pending_luafuncname.

WML IDLE State¶

m = re.match(r'\s*$', xline)

If the line is actually empty (only contains tabs/spaces) it will be consumed immediately. It is equal to that regular expression:

^\s*$

WmlCheckdomState¶

self.regex = re.compile(r'\s*#textdomain\s+(\S+)', re.I)

this is equal to that (case insensitive) regex:

^\s*#textdomain\s+(\S+)

At the start of the string will search for:

• spaces/tabs (from 0 to n)
• the character #.
• the word textdomain followed by one or more spaces
• one or more NO-SPACE characters, captured into group 1

WmlCheckpoState¶

rx = r'\s*#\s+(wmlxgettext|po-override|po):\s+(.+)'
self.regex = re.compile(rx, re.I)

wich is equal to that (case insensitive) regex:

^\s*#\s+(wmlxgettext|po-override|po):\s+(.+)

At the start of the string will search for:

• spaces/tabs (from 0 to n)
• the character #
• one or more space before an actual word
• one of those words: wmlxgettext, po-override or po captured into group 1.
• followed by the character : and one or more spaces/tabs.
• followed by any number of any characters (at least 1) captured on group 2.

WmlCommentState¶

self.regex = re.compile(r'\s*#.+')

At the start of the string will search for:

• spaces/tabs (from 0 to n)
• the character # followed by any character

WmlTagState¶

rx = r'\s*(?:[^"]+\(\s*)?\s*\[\s*([\/+-]?)\s*([A-Za-z0-9_]+)\s*\]'
self.regex = re.compile(rx)

Before explaining what the regex searches, we need to explain why the regexp was written in this way.

We must take mind that a WML tag (we now focus on open tag, but the discussion is the same also on close tags) can appear in two different ways; this is the first one:

# first way: tagname can be defined at the start of the line
  [tagname]

In this case, the WML line we are parsing may have an arbitrary number of spaces (or tabs) before the tagname, but nothing else must appear before the [tagname]. This is the most common case where a tag is defined, but it is not the only one; a tag can be added also in the body of a WML macro call as a part of WML code passed as parameter to the macro.

So why a WML tag can also apper inside the body of a macro call, like showed in this example:

{MACRO ([foo]
             bar = "baz"
        [/foo])}

So, wmlxgettext had to face two corner problems:

• it should record the [foo] open tag inside the macro call, or it will return an error when closing [/foo] tag will be found

• it should, however avoid to collect array indexes, thinking they are tags, for example:

  # [$i], here, is not a tag, but it is an index value of the array my_array
  value = my_array[$i]
  

So… how to distinguish tag from an array using a regexp? Well… a tagname, when placed inside a WML macro call, should be ALWAYS immediately preceded by (; nothing else than spaces can be putted before the parenthesis and the tag definition.

After all those explainations we have almost all the informations required to understand why the regexp used on WmlTagState is:

^\s*(?:[^"]+\(\s*)?\[\s*([\/+-]?)\s*([A-Za-z0-9_]+)\s*\]

As usual, at the start of the string, an arbitrary number of spaces or tabs (^\s*) can be found.

After that the regexp will consider two different scenarios:

• first scenario: [tagname] is defined inside a macro call
• second scenario: [tagname] stays alone (most common case)

On the fist scenario, the [tagname] is contained into a MACRO CALL, so we must verify that the [tagname] definition immediately follows a parenthesis (, except for spaces or tabs that can separates ( and [tagname]:

(?:[^"]+\(\s*)?

This check is performed by the non-capturing group written above, wich can occur one-time only (when tagname is contained in the macro definition) or it can occur zero times (when the tagname stays alone in the line, second scenario).

The non-capturing group will search for the last opening parenthesis encountered (and following spaces) that satisfies the remaining part of the regexp (explained later) wich search for [tagname].

This is, in particular, made by the second part of the non-capturing group:

\(\s*

But the non-capturing group will verify that no quote symbols (") were found in the meantime:

[^"]+

The reason of this exclusion is related to the wmlxgettext state machine design: the WmlTagState, infact, is evaluated before the WmlStr01 state (wich will search WML strings, translatable or not).

Wich means: if we allowed WmlTagState to match a line containing a quotation, we would let WmlTagState to consume all the matched line, including the WML string, wich will never been evalated by WmlStr01 State. But we don’t want that this event could happen.

So, coming back to the regexp:

^\s*(?:[^"]+\(\s*)?\[\s*([\/+-]?)\s*([A-Za-z0-9_]+)\s*\]

We said:

• ^\s* will search for arbitrary number of spaces (or tabs) at the start of the line
• (?:[^"]+\(\s*)? is the zero or one time non-campturing group that verifies if the tag is included inside a macro call. Wmlxgettext will search for a [tagname] wich is directly preceded by an opening parenthesis and an arbitrary number of spaces (or tabs). In the meantime it will verify that no quotations symbols (") can be found in the meantime. If a quotation symbol will be found, the regexp will be fail, so the WmlStr01 state can do its work (see the flow chart here above).
• \[\s*([\/+-]?)\s*([A-Za-z0-9_]+)\s*\] is the final part of the regexp (valid both for tags placed alone and for tags placed inside a macro call) that actually identify the tag. It will discussed here now.

The final part of the regular expression will search for [tagname], [/tagname], [+tagname] or [-tagname] where any number of spaces can be placed between [, tagname and ].

If +, - or / symbol is used, any number of spaces can be placed between the symbol, the [ and the tagname.

The regular expression, in this final part will also do those tasks:

• it will store, on group(1), the symbol +, - or /. If no symbol will be used, the group(1) will be an empty string.
• it will store, on group(2), the tagname. Characters allowed are only letters, numbers, or underscore, so why the expression ([A-Za-z0-9_]+) is used there (note that tagname must contain at least one character, this is why the + quantifier was used).

WmlGetinfState¶

rx = ( r'\s*(speaker|id|role|description|condition|type|race)' +
       r'\s*=\s*(.*)' )
self.regex = re.compile(rx, re.I)

This case-insensitive regex will be search, start of the string, for:

• spaces/tabs (from 0 to n)
• one of the following words: speaker, id, role, description, condition, type or race. The word will be captured into group 1.
• spaces/tabs (from 0 to n)
• the = character
• spaces/tabs (from 0 to n)
• any number of any character, captured by group 2. (this will be the value assigned to the parameter captured by group 1).

if '"' in match.group(2):
    _nextstate = 'wml_str01'
    pywmlx.state.machine._pending_winfotype = match.group(1)

State WmlStr01¶

This is the state wich will capture a wml string type 1 ( “quoted string” )

rx = r'(?:[^"]*?)\s*(_?)\s*"((?:""|[^"])*)("?)'
self.regex = re.compile(rx)

the regexp used here is a bit complex, so it will be atomized:

^(?:[^"]*?)

without creating group ((?:) creates a non-capturing group), any number of characters different than " will be found. But the search will be less greedy than possible (thank the very last ? putted after *). The “less greedy than possible” rule is necessary, othewhise the following rule will be ignored:

\s*(_?)\s*"

we need, infact, to know if a string is translatable or not. We must see if a _ sign was found before opening the quote. But the _ sign is different than " sign, so if the previous rule was greedy, the regexp could never capture on group 1 the _ sign. Instead, since the non-capturing group (?:[^"]*?) is “less greedy than possible” it will stops as soon the following rule \s*(_?)\s*" will be true.

Since the rule \s*(_?)\s*" will check:

• spaces/tabs (from 0 to n)
• zero or one _ sign, captured on group 1, followed by spaces/tabs (from 0 to n)
• followed by " sign

this means that the regexp, until now:

• is true even if something was found before _ "translatable string"
• will see if _ is used (group 1). Group 1 will be _ if the _ will be found, or it will be an empty string if the _ will not be found (string is not translatable)
• it will check for opening quote " where the string actually starts.

Finally, the regexp continues with:

((?:""|[^"])*)("?)

This part of the regexp must be explained a bit. A WML string can contain two following " signs if you want to use the " character inside your string (for example, using a " sign in a message). For this reason, if you find "" into a WML string, the string is not yet finised.

So, this part of the regexp:

• create a new group 2 (with the outer parenthesis on ((?:""|[^"])*) )

• that group 2 will capture any number of the things captured by the inner parenthesis, wich doesn’t create any additional groups (thank of the starting ?:).

• the “things” that can be captured on group 2, so, can be:

  • either “”
  • or any character different than "

• finally checks if there is the enclosing " sign and capture it to group 3.

This is how this complex regexp works.

State WmlStr02¶

This is the state wich will capture a translatable wml string type 2 ( _ <<translatable capitalized string>> )

rx = r'[^"]*_\s*<<(?:(.*?)>>|(.*))'
self.regex = re.compile(rx)

WmlStr02 is evalued after WmlCommentState (so it is evalued before WmlStr01):

[^"]*_\s*<<

Unlike before, WmlStr02 will match ONLY if the string is translatable (so non-translatable <<string>> will be ignored by regex). The regex will also mach only if no quotes found before the underscore marker followed by the << marker.

We said that WmlStr02 is evalued before WmlStr01, and that is the reason why no quote should be found before WmlStr02 (the WmlStr01 must be evalued and not skipped; so the WmlStr02 regex will fail, and the WmlStr01 state can be reached to collect the WmlStr01).

(?:(.*?)>>|(.*))

The second (and last) part of the regex is a non-capturing group wich contains two alternatives:

• (.*?)>> the first alternative matches if the close marker >> is found (single line translatable string). The capture ends when the first >> occurrence is found (non-greedy capture). Text is captured on group 1.
• (.*) the second alternative matches all the text until the end of the line (multi-line translatable string). Text is captured on group 2.

State WmlStr10¶

This is the state wich will capture multi-line wml “quoted” string (type 1) from line 2 to the end

self.regex = re.compile(r'((?:""|[^"])*)("?)')

The regexp is musch more simplier than the one used by the state WmlStr01 even if it works in a very similar way.

The basic idea of this regexp is: <<we are parsing a multi line string and this is NOT the first line of the string, so the starting part of the file line must be contained into the string until the ending ``”`` will be found>>.

It will save, on group 1 and group 2, what the regexp used by WmlStr01 capture on group 2 and group 3.

State WmlStr20¶

This is the state wich will capture multi-line wml <<capitalized>> string (type 2) from line 2 to the end

WmlStr20 is a very particular state: it is structured as an always-run state, but it works like a standard state.

There is a regex inside the run function wich is very simple:

(.*?)>>

This is a solution that allows WmlStr02 to stay there until the >> end marker will be found somewhere. Infact:

• If the regex fails, WmlStr20 will recursively change to itself (it stays to WmlStr20)
• If the regex matches, WmlStr20 will capture the text into group(1) and then the state will be changed to wml_idle

WmlGoluaState¶

self.regex = re.compile(r'.*?<<\s*')

It will be check, from the start part of the string, any number of any character (less greedy then possible) until << found (followed by any number of spaces/tabs - from 0 to n).

If the regexp will mach, the State will consume the line until the last space of the << symbol, and than switch to lua_idle state (parse Lua language).

LuaCheckdomState¶

rx = (   r'\s*(local)?\s+_\s*=\s*wesnoth\s*\.\s*textdomain\s*'
       r'''(?:\(\s*)?(["'])(.*?)\2''')
self.regex = re.compile(rx, re.I)

The regular expression used by LuaCheckdomState is very long, and it is very different from the one used by WmlCheckdomState. Changing the current domain in lua code, infact, requires a very different syntax:

-- after executing the following line, the current domain
-- will be changed to: wesnoth-xyz
local _ = wesnoth.textdomain('wesnoth-xyz')

It is now the time to explain deeply the regexp used by LuaCheckdomState:

^\s*(local)?\s+_\s*=\s*wesnoth\s*\.\s*textdomain\s*(?:\(\s*)?(["'])(.*?)\2

The regexp can be dived in this way:

• ^\s* –> Arbitrary number of spaces or tabs at the start of the line.
• (local\s+)? –> Optional local keyword. It is captured (if exists) in group(1). If local keyword is not used and the --warnall command line option is used, than a warning message is displayed.
• _\s*=\s* –> the underscore symbol (_) followed by equal (=). Any number of spaces or tab can be placed between underscore and equal; any mymber of spaces or tab can be also placed after the equal symbol.
• wesnoth\s*\.\s*textdomain\s* –> look for wesnoth.textdomain. Any number of spaces can be placed before and after the point symbol that divides wesnoth and textdomain ; any number of spaces can be placed after the textdomain word.
• (?:\(\s*)? –> This is a very important part of the regexp. This non-capturing group will ensure that the regexp will match when zero or one open paranthesis will follow after wesnoth.textdomain. The open parenthesis is, infact, optional.
• (["']) –> Then a single or a double quote is expected, and it will captured on group(2)
• (.*?) –> The actual textdomain will be captured on group(3)
• \2 –> the closing quote (what it was captured on group2, wich opened the quote, must match be the same one that closes the quote)

Lua “Comment” States¶

LuaCheckpoState and LuaCommentState use regexpes very similar to the ones used on WmlCheckpoState and WmlCommentState.

Here the differences:

• You can also use -- po: and -- po-override: or you can use -- # po: and -- # po-override: (both forms are allowed).
• # wmlxgettext: is not supported on lua code (it is useless)
• lua comment starts with -- and not with #

LuaStr01 and LuaStr02 States¶

We will display the LuaStr01 python code

rx = r'''(?:[^["']*?)(_?)\s*"((?:\\"|[^"])*)("?)'''
self.regex = re.compile(rx)

wich is equal to the following regexp:

^(?:[^["']*?)(_?)\s*"((?:\\"|[^"])*)("?)

The regexp used by LuaStr02 is more or less the same, infact it is equal to the following regexp:

^(?:[^["']*?)(_?)\s*'((?:\\'|[^'])*)('?)

The basic logic of those regexp is more or less the same as the one used by State WmlStr01.

As the regexp used by State WmlStr01, it can be divided in three parts:

• things before the strings starts
• check if the string is translatable, searching for _ sign rigtly before the string starts (followed by any number of spaces-tabs). (group 1 = _ or empty string)
• check for start quote (" for LuaStr01, ' for LuaStr02).
• check for text (group 2)
• check for quotation end (group 3) (if empty, is a multiline string).

The actual difference from the regexp used by State WmlStr01 is the first part of the regexp rule:

(?:[^["']*?)

Instead of searching of all characters different than only the " symbol, this regex will search all characters that will be neither ", nor ', nor [.

This will avoid conflicts from the three possible syntaxes and it will ensure that, if any of the regexp match, it will really match the first string, avoiding that a lua string will be skipped.

Another difference is that the “non enclosing quote” is not "" like WML, but it is escaped in a different way (\" or \'), this is why the rule is a bit different also in the third part of the regexp rule.

LuaStr10 and LuaStr20 States¶

The basic idea is the same as the one used by State WmlStr01.

(See also: State WmlStr01 and LuaStr01 and LuaStr02 States).

LuaStr03 State¶

LuaStr03 regexp can is equal to the following regexp rule:

^(?:[^["']*?)(_?)\s*\[(=*)\[(.*?)]\2]

The first part of regexp (^(?:[^["']*?)) is already explained in LuaStr01 and LuaStr02 States.

The second part of regexp((_?)\s*) captures _ on group 1 and collect any following spaces/tabs (without storing them in groups).

The third part of regexp (\[(=*)\[) captures all equal symbols placed between the two brackets and store them into group 2.

The fourth part of regexp ((.*?)) captures all characters contained between the lua bracketed string delimiters (ending delimiter is defined by the last part of the regexp). It captures the less charcaters than possible until the end delimiter found

The last part of regexp (]\2]) will search the right lua bracketed string end delimiter, checking how many equals symbols were captured on group 2 (\2 will search exactly what group 2 matched). So, if the group 2 is an empty string, than ]] will be the end delimiter searched by regexp. If the group 2 is === (3 equals symbols) than the end delimiter will be ]===]… and so on.

LuaStr03o State¶

LuaStr03o State will match when the beginning of a lua multiline bracketed string is found:

^(?:[^["']*?)(_?)\s*\[(=*)\[(.*)

The state LuaStr03o will capture:

• on group 1: the _ symbol (if is used)
• on group 2: how many equal symbols where placed in the starting string delimiter (for example the delimiter [=[ will contain one equal symbol between the two brackets)
• on group 3: the text of the first line of the string. This time the group 3 use greedy rule, capturing all following characters. This is why, this time, the regexp will be True (will match) even if nothing follows the [=[ marker (multiline string).

State LuaStr30¶

The LuaStr30 is a very particular state, wich is structured as an always-run state, but it works like a standard state.

The regexp definition, infact, is not placed (as usual) in the State.regexp parameter, defined in the __init__ function. This becouse all states are stored in the state machine during the setup phase, before starting to parse WML and Lua files. Wich means that all State.regexp values can be defined only on the setup phase itself and they cannot change anymore.

But, this time, we require to use a regexp rule that search exactly wich is the end delimiter for that one lua bracketed multiline string started on the previous LuaStr03o state.

This why the regexp is defined directly in the run() function, wich explicitly performs all actions usually done by statemachine when evaluating a State.regexp.

This is the regexp that will be evaluated in the run() function:

^(.*?)]={n}]

where n is the exact number of equals symbols stored in the PendingLuaString.numequals variable by LuaStr03o.

So, for example, if LuaStr03o.regex previously matched == on group 2 (wich means that [==[ was the opening delimiter used), then the regexp searched by the run function will be:

^(.*?)]={2}]

Now it is the time to actually explain the regexp. We will focus the explaination around this last concrete example (end delimiter must have exactly two equal symbols between close brackets). So why, from now on, we will explain the regexp:

^(.*?)]={2}]

This regexp will match if the line contains somewhere the ]==] delimiter. the final part of the regexp (]={2}]), infact, means:

• litteral ]
• followed by = (two times)
• followed by ]

If the delimiter ]==] will be found, the regexp will match, the last part of the string will be stored on group 1, than it will be added to the pending string. LuaStr30 will go to LuaIdleState (parsed line will be not completely consumed. Only what it will be matched will be removed from the parsed line.

If the delimiter ]==] will not be found, than the regexp will not mach. LuaStr30 will store all the parsed line into the pending lua string and consume it at all, so the statemachine will be able to read the next line of code. LuaStr30 will come back again to itself (it acts like a recursive state, in a very similar way like the LuaStr10 and LuaStr20 states).

LuaFinalState¶

Lua Final States checks if the current parsing line contains a function name:

rx_str = ( r'function\s+([a-zA-Z0-9_.]+)|' +
                r'([a-zA-Z0-9_.]+)\s*=\s*function'
         )
rx = re.compile(rx_str, re.I)
m = re.search(rx, xline)

So it use re.search and not re.match as usual. This mean that we don’t have a sort of an implicit caret symbol at the start of the regexp rule, so the resulting regexp rule is:

function\s+([a-zA-Z0-9_.]+)|([a-zA-Z0-9_.]+)\s*=\s*function

the regex will search:

• function <name_of_function>: where <name_of_function> will be stored on group 1.

or it will search:

• <name_of_function> = function: this time <name_of_function> will be stored on group 2.