Merge branch 'master' into hdtn

Author: T-recks

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,11 +1,11 @@
-<!-- This is just a checklist to guide you. You can remove it safely. -->
+<!-- This is just a checklist to guide you. Please remove it if you check all items. -->
 
 **Checklist:**
 
 -   [ ] If you are new to Scapy: I have checked [CONTRIBUTING.md](https://github.com/secdev/scapy/blob/master/CONTRIBUTING.md) (esp. section submitting-pull-requests)
 -   [ ] I squashed commits belonging together
 -   [ ] I added unit tests or explained why they are not relevant
--   [ ] I executed the regression tests (using `cd test && ./run_tests` or `tox`)
+-   [ ] I executed the regression tests (using `tox`)
 -   [ ] If the PR is still not finished, please create a [Draft Pull Request](https://github.blog/2019-02-14-introducing-draft-pull-requests/)
 
 <!-- brief description what this PR will do, e.g. fixes broken dissection of XXX -->

doc/scapy/build_dissect.rst

@@ -649,7 +649,7 @@ look to its building process::
           def post_build(self, p, pay):
             if self.len is None and pay:
                 l = len(pay)
-                p = p[:1] + hex(l)[2:]+ p[2:]
+                p = p[:1] + struct.pack("!B", l) + p[2:]
             return p+pay
 
 When ``post_build()`` is called, ``p``  is the current layer, ``pay`` the payload,

doc/scapy/layers/dcerpc.rst

@@ -1,7 +1,7 @@
 DCE/RPC & [MS-RPCE]
 ===================
 
-.. note:: DCE/RPC per `DCE/RPC 1.1 <https://pubs.opengroup.org/onlinepubs/9629399/toc.pdf>`_ with the `[MS-RPCE] <https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-rpce/290c38b1-92fe-4229-91e6-4fc376610c15>`_ additions
+.. note:: DCE/RPC per `DCE/RPC 1.1 <https://pubs.opengroup.org/onlinepubs/9629399/toc.pdf>`_ with the `[MS-RPCE] <https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-rpce/290c38b1-92fe-4229-91e6-4fc376610c15>`_ additions.
 
 Scapy provides support for dissecting and building Microsoft's Windows DCE/RPC calls.
 

doc/scapy/layers/dcom.rst

@@ -0,0 +1,128 @@
+[MS-DCOM]
+=========
+
+DCOM is a mechanism to manipulate COM objects remotely. It is in many ways just an extension over normal DCE/RPC, so understanding DCE/RPC concepts beforehand can be very useful.
+Before reading this, have a look at Scapy's `DCE/RPC <dcerpc.html>`_ documentation page.
+
+Terminology
+-----------
+
+- In DCOM one instantiates 'classes' to get 'object references'. A class implements one or several 'interfaces', each of which has methods.
+- ``CLSID``: the UIID of a **class**, used to instantiate it. This is typically chosen by whoever implements the COM object.
+- ``IID``: the UIID of an **interface**, used to request an IPID. This is chosen by whoever defines the COM interface (mostly Microsoft).
+- ``IPID``: a UIID that uniquely references an **interface on an object**. This allows to tell DCOM on which object to run the request we send.
+
+There are other IDs such as the OID (a 64bit number that uniquely references each object), and the OXID (a 64bit number that uniquely references each object exporter), but we won't get into the details.
+
+Per the spec, a DCOM client is supposed to keep track of the IPID, OID and OXID ids. In this regard, Scapy abstracts their usage.
+On the other hand, the calling application is supposed to know the ``CLSID`` of the class it wishes to instantiate, and the various ``IID`` of the interfaces it wishes to use.
+
+General behavior of a DCOM client
+---------------------------------
+
+1. Setup the DCOM client (endpoint, SSP, etc.)
+2. Get an object reference: Instantiate a class to get an object reference of the instance (``RemoteCreateInstance``), **OR**, get an object reference towards the class itself (``RemoteGetClassObject``).
+3. Acquire the IPID of an interface of the object.
+4. Call a method of that interface.
+5. Release the reference counts on the interface (delete the IPID).
+
+Step 3 can be done manually through the ``AcquireInterface()`` method, but Scapy will also automatically call it if you try to use an interface that you haven't acquired on an object.
+
+Using the DCOM client
+---------------------
+
+General usage
+~~~~~~~~~~~~~
+
+1. Setup the DCOM client and connect to the object resolver (which is by default on port 135).
+
+.. code:: python
+
+    from scapy.layers.msrpce.msdcom import DCOM_Client
+    from scapy.layers.ntlm import NTLMSSP
+
+    client = DCOM_Client(
+        ssp=NTLMSSP(UPN="Administrator@domain.local", PASSWORD="Scapy1111@"),
+    )
+    client.connect("server1.domain.local")
+
+.. note:: See the examples in `DCE/RPC <dcerpc.html>`_ to connect with SPNEGO/Kerberos.
+
+2. Instantiate a class to get an object reference
+
+.. code:: python
+
+    import uuid
+    from scapy.layers.dcerpc import find_com_interface
+    from scapy.layers.msrpce.raw.ms_pla import GetDataCollectorSets_Request
+
+    CLSID_TraceSessionCollection = uuid.UUID("03837530-098b-11d8-9414-505054503030")
+    # The COM interface must have been compiled by scapy-rpc (midl-to-scapy)
+    IDataCollectorSetCollection = find_com_interface("IDataCollectorSetCollection")
+
+    # Get new object reference
+    objref = client.RemoteCreateInstance(
+        # The CLSID we're instantiating
+        clsid=CLSID_TraceSessionCollection,
+        iids=[
+            # An initial list of interfaces to ask for. There must be at least 1.
+            IDataCollectorSetCollection,
+        ]
+    )
+
+3. Call a method on that object reference
+
+.. code:: python
+
+    result = objref.sr1_req(
+        # The request message (here from [MS-PLA])
+        pkt=GetDataCollectorSets_Request(
+            server=None,
+            filter=NDRPointer(
+                referent_id=0x72657355,
+                value=FLAGGED_WORD_BLOB(
+                    cBytes=18,
+                    asData=r"session\*".encode("utf-16le"),
+                )
+            ),
+        ),
+        # The interface to send it on
+        iface=IDataCollectorSetCollection,
+    )
+
+4. Release all the requested interfaces on the object reference
+
+.. code:: python
+
+    objref.release()
+
+5. Close the client
+
+.. code:: python
+
+    client.close()
+
+
+Unmarshalling object references
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Some methods return a reference to an object that is created by the remote server. On the network,
+those are typically marshalled as a ``MInterfacePointer`` structure. Such a structure can be "unmarshalled" into a local object reference that can be used in Scapy to call methods on that object.
+
+.. code:: python
+
+    # For instance, let's assume we're calling Next() of the IEnumVARIANT
+    resp = enum.sr1_req(
+        pkt=Next_Request(celt=1),
+        iface=IEnumVARIANT,
+    )
+
+    # Get the MInterfacePointer value
+    value = resp.valueof("rgVar")[0].valueof("_varUnion")
+    assert isinstance(value, MInterfacePointer)
+
+    # Unmarshall it and acquire an initial interface on it.
+    objref = client.UnmarshallObjectReference(
+        value,
+        iid=IDataCollectorSet,
+    )

scapy/arch/windows/native.py

@@ -222,7 +222,7 @@ def recv_raw(self, x=MTU):
 
     def close(self):
         # type: () -> None
-        if not self.closed and self.promisc:
+        if not self.closed and self.promisc and hasattr(self, 'ins'):
             self.ins.ioctl(socket.SIO_RCVALL, socket.RCVALL_OFF)
         super(L3WinSocket, self).close()
 

scapy/config.py

@@ -1146,6 +1146,8 @@ class Conf(ConfClass):
     )
     #: Dictionary containing parsed NSS Keys
     tls_nss_keys: Dict[str, bytes] = None
+    #: Whether to use NDR64 by default instead of NDR 32
+    ndr64: bool = True
     #: When TCPSession is used, parse DCE/RPC sessions automatically.
     #: This should be used for passive sniffing.
     dcerpc_session_enable = False

scapy/fwdmachine.py

@@ -86,9 +86,10 @@ class ForwardMachine:
     Methods to override:
 
     :func xfrmcs: a function to call when forwarding a packet from the 'client' to
-        the server. If it returns True, the packet is forwarded as it. If it
-        returns False or None, the packet is discarded. If it returns a
-        packet, this packet is forwarded instead of the original packet.
+        the server. If it raises a FORWARD exception, the packet is forwarded as it. If
+        it raises a DROP Exception, the packet is discarded. If it raises a
+        FORWARD_REPLACE(pkt) exception, then pkt is forwarded instead of the original
+        packet.
     :func xfrmsc: same as xfrmcs for packets forwarded from the 'server' to the
         'client'.
     """
@@ -372,7 +373,7 @@ def cb_sni(sock, server_name, _):
                     # Load result certificate our SSL server
                     # (this is dumb but we need to store them on disk)
                     certfile = get_temp_file()
-                    with open(certfile, "wb") as fd:
+                    with open(certfile, "w") as fd:
                         for c in certs:
                             fd.write(c.pem)
                     keyfile = get_temp_file()