back2source: Extend ScanCode.io pipeline for Python and C/C++ source and the python bytecode and native binaries

[pombredanne]

Create a new set of pipelines steps for the binary to source analysis for Python and C/C++ source and the python byte code and native binaries using strings and symbols

[pombredanne]

An example of package: https://pypi.org/project/scikit-learn

[pombredanne]

And see also:

• back2source: Run pipeline for Python and C/C++ source and the python bytecode and native binaries #1474

[JonoYang]

@pombredanne I've been looking at the d2d pipeline for elf symbol matching and seeing how I can use that for binaries in Python packages.

I run the d2d pipeline with the source archive and wheel of scikit-learn. The binary sklearn/cluster/_dbscan_inner.cpython-313t-x86_64-linux-gnu.so has the following symbols:

elf_symbols:
  - PyBaseObject_Type
  - PyBuffer_Release
  - PyBytes_AsString
  - PyBytes_FromString
  - PyBytes_FromStringAndSize
  - PyBytes_Type
  - PyCFunction_Type
  - PyCMethod_New
  - PyCapsule_GetPointer
  - PyCapsule_New
  - PyCode_NewEmpty
  - PyDict_DelItem
  - PyDict_GetItemRef
  - PyDict_GetItemString
  - PyDict_GetItemWithError
  - PyDict_New
  - PyDict_Next
  - PyDict_SetDefault
  - PyDict_SetItem
  - PyDict_SetItemString
  - PyDict_Size
  - PyErr_Clear
  - PyErr_ExceptionMatches
  - PyErr_Fetch
  - PyErr_Format
  - PyErr_GivenExceptionMatches
  - PyErr_NoMemory
  - PyErr_Occurred
  - PyErr_Restore
  - PyErr_SetObject
  - PyErr_SetString
  - PyErr_WarnEx
  - PyErr_WarnFormat
  - PyEval_GetBuiltins
  - PyExc_ArithmeticError
  - PyExc_AttributeError
  - PyExc_BufferError
  - PyExc_DeprecationWarning
  - PyExc_IOError
  - PyExc_ImportError
  - PyExc_IndexError
  - PyExc_MemoryError
  - PyExc_NameError
  - PyExc_NotImplementedError
  - PyExc_OverflowError
  - PyExc_RuntimeError
  - PyExc_RuntimeWarning
  - PyExc_StopIteration
  - PyExc_SystemError
  - PyExc_TypeError
  - PyExc_UnboundLocalError
  - PyExc_ValueError
  - PyExc_ZeroDivisionError
  - PyException_GetTraceback
  - PyException_SetTraceback
  - PyFloat_Type
  - PyFrame_New
  - PyGC_Disable
  - PyGC_Enable
  - PyGILState_Ensure
  - PyGILState_Release
  - PyImport_AddModuleRef
  - PyImport_GetModule
  - PyImport_GetModuleDict
  - PyImport_ImportModuleLevelObject
  - PyIndex_Check
  - PyInit__dbscan_inner
  - PyInterpreterState_GetID
  - PyList_Append
  - PyList_AsTuple
  - PyList_GetItemRef
  - PyList_New
  - PyList_SetItem
  - PyList_Type
  - PyLong_AsLong
  - PyLong_AsSsize_t
  - PyLong_FromLong
  - PyLong_FromSsize_t
  - PyLong_Type
  - PyMem_Free
  - PyMem_Malloc
  - PyMem_Realloc
  - PyMethod_New
  - PyMethod_Type
  - PyModuleDef_Init
  - PyModule_GetDict
  - PyModule_GetName
  - PyModule_NewObject
  - PyNumber_Add
  - PyNumber_InPlaceAdd
  - PyNumber_InPlaceMultiply
  - PyNumber_Index
  - PyNumber_Multiply
  - PyNumber_Remainder
  - PyOS_snprintf
  - PyObject_Call
  - PyObject_CallFinalizerFromDealloc
  - PyObject_CallMethodObjArgs
  - PyObject_ClearWeakRefs
  - PyObject_Format
  - PyObject_Free
  - PyObject_GC_Del
  - PyObject_GC_IsFinalized
  - PyObject_GC_Track
  - PyObject_GC_UnTrack
  - PyObject_GenericGetAttr
  - PyObject_GetAttr
  - PyObject_GetAttrString
  - PyObject_GetBuffer
  - PyObject_GetItem
  - PyObject_GetIter
  - PyObject_GetOptionalAttr
  - PyObject_HasAttr
  - PyObject_HasAttrWithError
  - PyObject_Hash
  - PyObject_IsSubclass
  - PyObject_IsTrue
  - PyObject_Malloc
  - PyObject_Repr
  - PyObject_RichCompare
  - PyObject_RichCompareBool
  - PyObject_SetAttr
  - PyObject_SetAttrString
  - PyObject_SetItem
  - PyObject_VectorcallDict
  - PyRun_SimpleStringFlags
  - PySequence_Contains
  - PySequence_Tuple
  - PySlice_New
  - PySlice_Type
  - PySys_GetObject
  - PyThreadState_Get
  - PyThreadState_GetUnchecked
  - PyThread_allocate_lock
  - PyThread_free_lock
  - PyTraceBack_Here
  - PyTuple_GetItem
  - PyTuple_GetSlice
  - PyTuple_New
  - PyTuple_Pack
  - PyTuple_Type
  - PyType_Modified
  - PyType_Ready
  - PyUnicode_Compare
  - PyUnicode_Concat
  - PyUnicode_CopyCharacters
  - PyUnicode_Decode
  - PyUnicode_Format
  - PyUnicode_FromFormat
  - PyUnicode_FromOrdinal
  - PyUnicode_FromString
  - PyUnicode_FromStringAndSize
  - PyUnicode_InternFromString
  - PyUnicode_New
  - PyUnicode_Type
  - PyUnstable_Code_NewWithPosOnlyArgs
  - PyUnstable_Module_SetGIL
  - PyVectorcall_Function
  - Py_EnterRecursiveCall
  - Py_LeaveRecursiveCall
  - Py_Version
  - _ITM_deregisterTMCloneTable
  - _ITM_registerTMCloneTable
  - _PyObject_GC_New
  - _PyType_Lookup
  - _Py_DecRefShared
  - _Py_FalseStruct
  - _Py_FatalErrorFunc
  - _Py_MergeZeroLocalRefcount
  - _Py_NoneStruct
  - _Py_TrueStruct
  - _ZNSt6vectorIlSaIlEE17_M_realloc_insertIJRKlEEEvN9__gnu_cxx17__normal_iteratorIPlS1_EEDpOT_
  - _ZdlPvm
  - __gmon_start__
  - _fini
  - _init

The corresponding source file sklearn/cluster/_dbscan_inner.pyx has the function dbscan_inner. Just making sure that I am correct in thinking that we want to create a relation between sklearn/cluster/_dbscan_inner.pyx and sklearn/cluster/_dbscan_inner.cpython-313t-x86_64-linux-gnu.so because the binary has the symbol PyInit__dbscan_inner

There would need to be a step to parse the pyx file for function names etc

[JonoYang]

This was completed in #1703

See the related pipe in SCIO documentation: https://scancodeio.readthedocs.io/en/latest/built-in-pipelines.html#scanpipe.pipelines.deploy_to_develop.DeployToDevelop.map_python

This is available as a new optional step in the deploy_to_develop pipeline, Python, when using the map deploy to develop pipeline. Select the Python option when running the deploy to develop pipeline to use it: