More explicit comments and ...

add ## comments, and document better module and method information

Author: rocky

HOW-TO-USE.rst

@@ -11,7 +11,7 @@ skeletal. Then use ``pydisasm --xasm`` to convert to the Python source
 code assembler format. From this then modify the results and run
 ``pyc-xasm``.
 
-In normal python disassembly code (and in the bytecode file), the main
+In normal Python disassembly code (and in the bytecode file), the main
 function appears first; it contains constants which contain code to
 other functions and so on. In Python disassembly these are linearized
 so that from top down you have a topological sort of the dependencies.
@@ -32,39 +32,66 @@ preferable right now.
 Format of assembly file
 =======================
 
-Again, easiest to consult the ``pydisasm --xasm`` output ``.pyasm``-file that is 
-produced. Even easier, just to look in the test directory_ for files that end 
+Again, easiest to consult the ``pydisasm --xasm`` output ``.pyasm``-file that is
+produced. Even easier, just to look in the test directory_ for files that end
 with ``.pyasm``.
 
-In general, lines that start with "#" in column one are comments or code or function 
-objects other than bytecode instructions.
+Lines that start with '##" are comments.
 
-Necessary fields that are in Python code object and function objects
-are here. These include stuff like the Python "magic" number which
-determines which Python bytecode opcodes to use and which Python
-interpreter can be used to run the resulting program.
+Examples:
+
+    ## This line is a comment
+    ## Method Name:       GameSheet
+
+Lines that start with ``#``" in column one are used to indicate a code
+or function object that is not a bytecode instructions. However this
+is only true if the rest of the line matches one of the code of
+function objects mentioned below. If instead the the rest of a line
+does not match a function or code object, it line too will be
+treated tacitly as a comment.
 
 Module-level info
 ------------------
 
 
+The only necessary mdoule-level inforamtion that is needed is the
+Python "magic" number which determines which Python bytecode opcodes
+to use and which Python interpreter can be used to run the resulting
+program.
+
+Optional information includes:
+
+* Timestamp of code
+* Source code size module 2**32 or a SIP hash
+
 Here is an example of the module-level information:
 
 ::
 
    # Python bytecode 2.2 (60717)
    # Timestamp in code: 1499156389 (2017-07-04 04:19:49)
+   # Source code size mod 2**32: 577 bytes
 
-The bytecode is necessary. However the timestamp is not. In Python 3
-there is also a size modulo 2**32 that is recorded.
+Again, the bytecode numberf is necessary. However the timestamp is not. In Python 3
+there is also a size modulo 2**32 that is recorded, and in later Python this can be a
+SIP hash.
 
 ::
 
-   # Source code size mod 2**32: 577 bytes
 
 Method-level info
 ------------------
 
+Method-level information starts with ``#`` in column one. Here is some
+method-level information:
+
+* The method name of the code object (``Method Name``)
+* Number of local variables used in module or fuction (``Number of locals``)
+* A filename where the file (``Filename``)
+* Maximum Stack Size needed to run code (``Stack Size``)
+* Code flags which indicate properties of the code (``Flags``)
+* Fine number for the first line of the code (``First Line``)
+
 Here is an example:
 
 ::
@@ -73,9 +100,10 @@ Here is an example:
    # Filename: exec
    # Argument count: 2
    # Kw-only arguments: 0
-   # Number of locals: 2 # Stack size: 3
+   # Number of locals: 2
+   # Stack size: 3
    # Flags: 0x00000043 (NOFREE | NEWLOCALS | OPTIMIZED)
-   # First Line: 11 
+   # First Line: 11
    # Constants:
    # 0: ' GCD. We assume positive numbers'
    # 1: 0
@@ -105,14 +133,14 @@ the last sentence means is that
 
    LOAD_CONST 3
 
-would be invalid if the size of the constant array is less than 4, or `constant[3]` wasn't defined by adding it to the `Constants` section. However when you put a value in parenthesis, that indicate a value rather than an index into a list. 
+would be invalid if the size of the constant array is less than 4, or `constant[3]` wasn't defined by adding it to the `Constants` section. However when you put a value in parenthesis, that indicate a value rather than an index into a list.
 So you could instead write:
 
 ::
 
    LOAD_CONST (1)
 
-which in this case does the same thing since `1 = constant[3]`. If the value 1 does not appear anywhere in the constants list, the assembler would append the value 1 to the end of the list of the constants list. When writing the final bytecode file an appropriate constant index will be inserted into that instruction. 
+which in this case does the same thing since `1 = constant[3]`. If the value 1 does not appear anywhere in the constants list, the assembler would append the value 1 to the end of the list of the constants list. When writing the final bytecode file an appropriate constant index will be inserted into that instruction.
 
 Line Numbers and Labels
 -----------------------
@@ -187,7 +215,7 @@ parenthesis. For example:
 ::
 
     LOAD_CONST (3)    # loads number 3
-    LOAD_CONST 3      # load Constants[3] 
+    LOAD_CONST 3      # load Constants[3]
     JUMP_ABSOLUTE 10  # Jumps to offset 10
     JUMP_ABSOLUTE L10 # Jumps to label L10
     LOAD_CONSTANT (('load_entry_point',)) # Same as: tuple('load_entry_point')

xasm/assemble.py

@@ -62,12 +62,12 @@ def get_opname_operand(opc, fields):
         return opname, None
 
 
-class Assembler(object):
+class Assembler:
     def __init__(self, python_version, is_pypy):
         self.opc = get_opcode(python_version, is_pypy)
         self.code_list = []
         self.codes = []  # FIXME use a better name
-        self.status = "unfinished"
+        self.status: str = "unfinished"
         self.size = 0  # Size of source code. Only relevant in version 3 and above
         self.python_version = python_version
         self.timestamp = None
@@ -144,6 +144,9 @@ def asm_file(path):
     while i < len(lines):
         line = lines[i]
         i += 1
+        if line.startswith("##"):
+            # comment line
+            continue
         if line.startswith(".READ"):
             match = re.match("^.READ (.+)$", line)
             if match:
@@ -197,7 +200,9 @@ def asm_file(path):
                     asm.timestamp = int(time_str)
             elif line.startswith("# Method Name: "):
                 if method_name:
-                    co = create_code(asm, label, backpatch_inst)
+                    co, is_valid = create_code(asm, label, backpatch_inst)
+                    if not is_valid:
+                        return
                     asm.update_lists(co, label, backpatch_inst)
                     label = {}
                     backpatch_inst = set([])
@@ -336,7 +341,7 @@ def asm_file(path):
 
             if match:
                 line_no = int(match.group(1))
-                linetable_field = "co_linotab" if python_version_pair < (3, 10) else "co_linetable"
+                linetable_field = "co_lnotab" if python_version_pair < (3, 10) else "co_linetable"
                 assert asm is not None
                 linetable = getattr(asm.code, linetable_field)
                 linetable[offset] = line_no
@@ -348,27 +353,31 @@ def asm_file(path):
             if num_fields == 1 and line_no is not None:
                 continue
 
-            if num_fields > 1:
-                if fields[0] == ">>":
-                    fields = fields[1:]
-                    num_fields -= 1
-                if match_lineno(fields[0]) and is_int(fields[1]):
-                    line_no = int(fields[0][:-1])
-                    opname, operand = get_opname_operand(asm.opc, fields[2:])
-                elif match_lineno(fields[0]):
-                    line_no = int(fields[0][:-1])
-                    fields = fields[1:]
+            try:
+                if num_fields > 1:
                     if fields[0] == ">>":
                         fields = fields[1:]
-                        if is_int(fields[0]):
+                        num_fields -= 1
+                    if match_lineno(fields[0]) and is_int(fields[1]):
+                        line_no = int(fields[0][:-1])
+                        opname, operand = get_opname_operand(asm.opc, fields[2:])
+                    elif match_lineno(fields[0]):
+                        line_no = int(fields[0][:-1])
+                        fields = fields[1:]
+                        if fields[0] == ">>":
                             fields = fields[1:]
-                    opname, operand = get_opname_operand(asm.opc, fields)
-                elif is_int(fields[0]):
-                    opname, operand = get_opname_operand(asm.opc, fields[1:])
+                            if is_int(fields[0]):
+                                fields = fields[1:]
+                        opname, operand = get_opname_operand(asm.opc, fields)
+                    elif is_int(fields[0]):
+                        opname, operand = get_opname_operand(asm.opc, fields[1:])
+                    else:
+                        opname, operand = get_opname_operand(asm.opc, fields)
                 else:
-                    opname, operand = get_opname_operand(asm.opc, fields)
-            else:
-                opname, _ = get_opname_operand(asm.opc, fields)
+                    opname, _ = get_opname_operand(asm.opc, fields)
+            except Exception as e:
+                print(f"Line {i}: {e}")
+                raise
 
             if opname in asm.opc.opname:
                 inst = Instruction()
@@ -688,13 +697,13 @@ def create_code(asm: Assembler, label, backpatch):
     is_code_ok(asm)
 
     # Stamp might be added here
-    # if asm.python_version[:2] == PYTHON_VERSION_TRIPLE[:2]:
-    #     code = asm.code.to_native()
-    # else:
-    code = asm.code.freeze()
+    if asm.python_version[:2] == PYTHON_VERSION_TRIPLE[:2]:
+        code = asm.code.to_native()
+    else:
+        code = asm.code.freeze()
 
     # asm.print_instructions()
 
     # print (*args)
     # co = self.Code(*args)
-    return code
+    return code, is_valid

xasm/pyc_convert.py

@@ -142,7 +142,7 @@ def transform_33_32(inst, new_inst, i, n, offset, instructions, new_asm):
 
 
 def transform_asm(asm, conversion_type, src_version, dest_version):
-    new_asm = Assembler(dest_version)
+    new_asm = Assembler(dest_version, is_pypy=False)
     for field in "code size".split():
         setattr(new_asm, field, copy(getattr(asm, field)))
 
@@ -177,10 +177,10 @@ def transform_asm(asm, conversion_type, src_version, dest_version):
             i += 1
             pass
 
-    co = create_code(new_asm, new_asm.label[-1], new_asm.backpatch[-1])
+    co, is_valid = create_code(new_asm, new_asm.label[-1], new_asm.backpatch[-1])
     new_asm.code_list.append(co)
     new_asm.code_list.reverse()
-    new_asm.finished = "finished"
+    new_asm.status = "finished" if is_valid else "invalid"
     return new_asm