Plugin dont work after update netbox to 2.9.1

[SerhiiZahuba]

Environment

Python version: 3.6.8
NetBox version: 2.9.1
vCenter version: 6.7.0 -->

• vCenter NetBox Sync version:

Steps to Reproduce

1. After run plugin i see

[root@netbox]# /bin/python3.6 /opt/vcenter-netbox-sync/run.py >/var/log/vcentersync.log
2020-08-23 15:56:17,690 [INFO] Created new HTTP Session for NetBox.
2020-08-23 15:56:17,785 [INFO] Detected NetBox API v2.9.
2020-08-23 15:56:17,786 [INFO] Verifying all prerequisite objects exist in NetBox.
2020-08-23 15:56:17,838 [INFO] NetBox manufacturers object 'VMware' match current values. Moving on.
2020-08-23 15:56:17,866 [INFO] NetBox platforms object 'VMware ESXi' match current values. Moving on.
2020-08-23 15:56:17,895 [INFO] NetBox platforms object 'Windows' match current values. Moving on.
2020-08-23 15:56:17,919 [INFO] NetBox platforms object 'Linux' match current values. Moving on.
2020-08-23 15:56:18,032 [INFO] NetBox sites object 'vCenter' do not match current values.
Traceback (most recent call last):
File "/opt/vcenter-netbox-sync/run.py", line 1601, in
main()
File "/opt/vcenter-netbox-sync/run.py", line 244, in main
nb.verify_dependencies()
File "/opt/vcenter-netbox-sync/run.py", line 1540, in verify_dependencies
self.obj_exists(nb_obj_type=dep_type, vc_data=dep)
File "/opt/vcenter-netbox-sync/run.py", line 1105, in obj_exists
set(vc_data["tags"] + nb_data["tags"])
TypeError: unhashable type: 'dict'

Expected Behavior

Observed Behavior

[Djf2884]

Same here, even after a cleanup, collect seem to work and stop with unexpected behavior.

also it seem not working properly with vcenter/esxi 7.0 which have been updated on API level.

[gstorme]

Probably related to netbox-community/netbox#3703
Can you paste a debug run log + results of GET /api/dcim/sites/?name=vCenter?

[SerhiiZahuba]

2020-08-24 10:00:02,237 [INFO] Verifying all prerequisite objects exist in NetBox.
2020-08-24 10:00:02,268 [INFO] NetBox manufacturers object 'VMware' match current values. Moving on.
2020-08-24 10:00:02,304 [INFO] NetBox platforms object 'VMware ESXi' match current values. Moving on.
2020-08-24 10:00:02,329 [INFO] NetBox platforms object 'Windows' match current values. Moving on.
2020-08-24 10:00:02,358 [INFO] NetBox platforms object 'Linux' match current values. Moving on.
2020-08-24 10:00:02,426 [INFO] NetBox sites object 'vCenter' do not match current values.
2020-08-24 12:00:01,821 [INFO] Created new HTTP Session for NetBox.
2020-08-24 12:00:01,866 [INFO] Detected NetBox API v2.9.
2020-08-24 12:00:01,868 [INFO] Verifying all prerequisite objects exist in NetBox.
2020-08-24 12:00:01,898 [INFO] NetBox manufacturers object 'VMware' match current values. Moving on.
2020-08-24 12:00:01,932 [INFO] NetBox platforms object 'VMware ESXi' match current values. Moving on.
2020-08-24 12:00:01,969 [INFO] NetBox platforms object 'Windows' match current values. Moving on.
2020-08-24 12:00:01,999 [INFO] NetBox platforms object 'Linux' match current values. Moving on.
2020-08-24 12:00:02,069 [INFO] NetBox sites object 'vCenter' do not match current values.

`HTTP 200 OK
Allow: GET, POST, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
"count": 1,
"next": null,
"previous": null,
"results": [
{
"id": 12,
"url": "http://netbox..com/api/dcim/sites/12/",
"name": "vCenter",
"slug": "vcenter",
"status": {
"value": "active",
"label": "Active"
},
"region": null,
"tenant": null,
"facility": "",
"asn": null,
"time_zone": null,
"description": "",
"physical_address": "",
"shipping_address": "",
"latitude": null,
"longitude": null,
"contact_name": "",
"contact_phone": "",
"contact_email": "",
"comments": "A default virtual site created to house objects that have been synced from vCenter.",
"tags": [
{
"id": 10,
"url": "http://netbox..com/api/extras/tags/10/",
"name": "Synced",
"slug": "synced",
"color": "9e9e9e"
},
{
"id": 9,
"url": "http://netbox.**.com/api/extras/tags/9/",
"name": "vCenter",
"slug": "vcenter",
"color": "9e9e9e"
}
],
"custom_fields": {},
"created": "2020-05-13",
"last_updated": "2020-05-13T20:34:40.272068+03:00",
"circuit_count": null,
"device_count": null,
"prefix_count": null,
"rack_count": null,
"virtualmachine_count": null,
"vlan_count": null
}
]
}`

[gstorme]

Yes so this script is incompatible with Netbox >= 2.9.0 due to changes in the way tags are created.
Best to stay on 2.8 until a PR is submitted by someone with the necessary changes.

[SerhiiZahuba]

@synackray can you help ?

[gstorme]

Not just the tags, but also the API changes in 2.9.0 for ipam.IPAddress cause the mapping between IP & virtual-machine to not work anymore.

ipam.IPAddress: Removed interface field; replaced with assigned_object generic foreign key. This may represent either a device interface or a virtual machine interface. Assign an object by setting assigned_object_type and assigned_object_id.

[bb-Ricardo]

this will fix the tag issue: https://github.com/bb-Ricardo/vcenter-netbox-sync/commit/2d3052fd14bc3947b24e1da3927e93e4e44b6355

Will have a look at ip ipam/ipaddress next week.

[bb-Ricardo]

Well, we got some progress: https://github.com/bb-Ricardo/vcenter-netbox-sync/commit/9447109a12e18703954cd843c7bc2a6adb936285

Now all my items sync again as before.

Please check it out. All changes are also in #132

[SerhiiZahuba]

I have a problem after updating the plug-in, all virtual servers were transferred to Standalone ESXi Host, they were previously synchronized with their server, for example, if the virtual machine is on the 192.168.50.170 server, then in the Cluster it will also be synchronized there. Now all VMs are in Standalone ESXi Host.

[SerhiiZahuba]

also, as you can see in the screenshot, some virtual servers are in their clusters, but the main part has moved to Standalone

[bb-Ricardo]

That's very interesting as I didn't touch this part of code: https://github.com/bb-Ricardo/vcenter-netbox-sync/blame/9447109a12e18703954cd843c7bc2a6adb936285/run.py#L557

"Standalone ESXi Host" is used if the hosts cluster name is the same as the hosts name:

                    cluster = obj.parent.name
                    # We throw the cluster away if it matches the ESXi host
                    # name as it is standalone.
                    if cluster == obj_name:
                        # Store the host so that we can check VMs against it
                        self.standalone_hosts.append(cluster)
                        cluster = "Standalone ESXi Host"

and Line: https://github.com/bb-Ricardo/vcenter-netbox-sync/blob/develop/run.py#L647

                elif vc_obj_type == "virtual_machines":
                    # Virtual Machines
                    log.debug(
                        "Collecting info for virtual machine '%s'", obj_name
                        )
                    # Cluster
                    cluster = obj.runtime.host.parent.name
                    if cluster in self.standalone_hosts:
                        log.debug(
                            "VM is assigned to a standalone ESXi host. Setting "
                            "cluster to 'Standalone ESXi Host'."
                            )
                        cluster = "Standalone ESXi Host"

In my opinion the whole sync needs a complete rewrite. Unfortunately this code is only used in a test environment. It was a POC and after the last NetBox API FU I'm not sure if I'm willing to spend my time on changing code and adapting APIs every time a new NetBox major release is issued.

I have loads of ideas but (https://github.com/bb-Ricardo/vcenter-netbox-sync/issues) but not sure if time allows it.

[MarcSN311]

@bb-Ricardo I keep getting errors like this: [DEBUG] NetBox 400 status reason: {"tags":[["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: Synced"],["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: vCenter"]]}
I am on your develop branch