Instagram embed not working on live server

[Bargsteen]

Hi,
I am experiencing some problems with the embedded Instagram posts.
They work locally when running a server using ddev, but not when put on my live server.

The error occurs here because the JSON returned from Instagram on the live server does not contain the correct info:

grav-plugin-static-social-embeds/shortcodes/InstagramShortcode.php

Line 80 in 212ba4c

if (!$post

This is the JSON received on the live server. It seems like it is not logged in correctly.

{
   "config":{
      "csrf_token":"icXKscTWwx55bVviGZEzlR2guniTM52u",
      "viewer":null,
      "viewerId":null
   },
   "country_code":"NL",
   "language_code":"en",
   "locale":"en_US",
   "entry_data":{
      "LoginAndSignupPage":[
         {
            "captcha":{
               "enabled":false,
               "key":""
            },
            "gdpr_required":true,
            "tos_version":"eu",
            "username_hint":""
         }
      ]
   },
   "hostname":"www.instagram.com",
   "is_whitelisted_crawl_bot":false,
   "connection_quality_rating":"EXCELLENT",
   "deployment_stage":"c2",
   "platform":"web",
   "nonce":"+gDQAv0v7jtoYpKVAo7Y8Q==",
   "mid_pct":10.78018,
   "zero_data":{
      
   },
   "cache_schema_version":3,
   "server_checks":{
      
   },
   "knobx":{
      "17":false,
      "20":true,
      "22":true,
      "23":true,
      "24":true,
      "25":true,
      "26":true,
      "27":true,
      "28":true,
      "29":true,
      "30":true,
      "31":true,
      "32":true,
      "34":true,
      "35":false,
      "36":false,
      "38":25000,
      "39":true,
      "4":false
   },
   "to_cache":{
      "gatekeepers":{
         "10":false,
         "100":false,
         "101":true,
         "102":true,
         "103":true,
         "104":true,
         "105":true,
         "106":true,
         "107":false,
         "108":true,
         "11":false,
         "112":true,
         "113":true,
         "114":true,
         "116":true,
         "119":false,
         "12":false,
         "120":true,
         "123":false,
         "126":false,
         "128":false,
         "13":true,
         "131":false,
         "132":false,
         "137":true,
         "14":true,
         "140":false,
         "142":false,
         "146":true,
         "147":false,
         "149":false,
         "15":true,
         "150":false,
         "151":false,
         "152":true,
         "153":false,
         "154":true,
         "156":false,
         "157":false,
         "158":false,
         "159":false,
         "16":false,
         "160":false,
         "162":false,
         "18":true,
         "19":false,
         "23":false,
         "24":false,
         "26":true,
         "27":false,
         "28":false,
         "29":true,
         "31":false,
         "32":true,
         "34":false,
         "35":false,
         "38":true,
         "4":true,
         "40":true,
         "41":false,
         "43":true,
         "5":false,
         "59":true,
         "6":false,
         "61":false,
         "62":false,
         "63":false,
         "64":false,
         "65":false,
         "67":true,
         "68":false,
         "69":true,
         "7":false,
         "71":false,
         "73":false,
         "74":false,
         "75":true,
         "78":true,
         "79":false,
         "8":false,
         "81":false,
         "82":true,
         "84":false,
         "86":false,
         "9":false,
         "91":false,
         "95":true,
         "97":false
      },
      "qe":{
         "app_upsell":{
            "g":"",
            "p":{
               
            }
         },
         "igl_app_upsell":{
            "g":"",
            "p":{
               
            }
         },
         "notif":{
            "g":"",
            "p":{
               
            }
         },
         "onetaplogin":{
            "g":"",
            "p":{
               
            }
         },
         "felix_clear_fb_cookie":{
            "g":"",
            "p":{
               
            }
         },
         "felix_creation_duration_limits":{
            "g":"",
            "p":{
               
            }
         },
         "felix_creation_fb_crossposting":{
            "g":"",
            "p":{
               
            }
         },
         "felix_creation_fb_crossposting_v2":{
            "g":"",
            "p":{
               
            }
         },
         "felix_creation_validation":{
            "g":"",
            "p":{
               
            }
         },
         "post_options":{
            "g":"",
            "p":{
               
            }
         },
         "sticker_tray":{
            "g":"",
            "p":{
               
            }
         },
         "web_sentry":{
            "g":"",
            "p":{
               
            }
         },
         "0":{
            "p":{
               "9":false
            },
            "l":{
               
            },
            "qex":true
         },
         "100":{
            "p":{
               "0":true
            },
            "l":{
               
            },
            "qex":true
         },
         "101":{
            "p":{
               "0":false,
               "1":false
            },
            "l":{
               
            },
            "qex":true
         },
         "102":{
            "p":{
               "0":true
            },
            "l":{
               
            },
            "qex":true
         },
         "103":{
            "p":{
               "1":false
            },
            "l":{
               
            },
            "qex":true
         },
         "104":{
            "p":{
               "0":true
            },
            "l":{
               
            },
            "qex":true
         },
         "108":{
            "p":{
               "0":false,
               "1":false
            },
            "l":{
               
            },
            "qex":true
         },
         "109":{
            "p":{
               
            },
            "l":{
               
            },
            "qex":true
         },
         "111":{
            "p":{
               "0":false,
               "1":false
            },
            "l":{
               
            },
            "qex":true
         },
         "113":{
            "p":{
               "0":true,
               "1":false,
               "2":true,
               "4":false,
               "5":false,
               "7":false,
               "8":false
            },
            "l":{
               
            },
            "qex":true
         },
         "117":{
            "p":{
               "0":true
            },
            "l":{
               
            },
            "qex":true
         },
         "118":{
            "p":{
               "0":false,
               "1":true,
               "2":false
            },
            "l":{
               
            },
            "qex":true
         },
         "119":{
            "p":{
               "0":false
            },
            "l":{
               
            },
            "qex":true
         },
         "12":{
            "p":{
               "0":5
            },
            "l":{
               
            },
            "qex":true
         },
         "121":{
            "p":{
               
            },
            "l":{
               
            },
            "qex":true
         },
         "122":{
            "p":{
               "0":false
            },
            "l":{
               
            },
            "qex":true
         },
         "123":{
            "p":{
               "0":true,
               "1":true
            },
            "l":{
               
            },
            "qex":true
         },
         "124":{
            "p":{
               "0":false,
               "1":true,
               "2":false
            },
            "l":{
               
            },
            "qex":true
         },
         "125":{
            "p":{
               "0":true
            },
            "l":{
               
            },
            "qex":true
         },
         "127":{
            "p":{
               "0":true,
               "1":false,
               "2":true
            },
            "l":{
               
            },
            "qex":true
         },
         "128":{
            "p":{
               "0":false,
               "1":false
            },
            "l":{
               
            },
            "qex":true
         },
         "129":{
            "p":{
               "1":false,
               "2":true,
               "3":false,
               "4":false
            },
            "l":{
               "2":true
            },
            "qex":true
         },
         "13":{
            "p":{
               "0":true
            },
            "l":{
               
            },
            "qex":true
         },
         "131":{
            "p":{
               "2":true,
               "3":true,
               "4":false
            },
            "l":{
               
            },
            "qex":true
         },
         "132":{
            "p":{
               "0":false
            },
            "l":{
               "0":true
            },
            "qex":true
         },
         "135":{
            "p":{
               "0":false,
               "1":false,
               "2":false,
               "3":false
            },
            "l":{
               
            },
            "qex":true
         },
         "137":{
            "p":{
               
            },
            "l":{
               
            },
            "qex":true
         },
         "140":{
            "p":{
               
            },
            "l":{
               
            },
            "qex":true
         },
         "141":{
            "p":{
               "0":false,
               "1":false,
               "2":false
            },
            "l":{
               
            },
            "qex":true
         },
         "142":{
            "p":{
               "0":false
            },
            "l":{
               
            },
            "qex":true
         },
         "143":{
            "p":{
               "1":false,
               "2":false,
               "3":false,
               "4":false
            },
            "l":{
               
            },
            "qex":true
         },
         "146":{
            "p":{
               
            },
            "l":{
               
            },
            "qex":true
         },
         "147":{
            "p":{
               
            },
            "l":{
               
            },
            "qex":true
         },
         "148":{
            "p":{
               
            },
            "l":{
               
            },
            "qex":true
         },
         "16":{
            "p":{
               "0":false
            },
            "l":{
               
            },
            "qex":true
         },
         "21":{
            "p":{
               "2":false
            },
            "l":{
               
            },
            "qex":true
         },
         "22":{
            "p":{
               "10":0.0,
               "11":15,
               "12":3,
               "13":false,
               "2":8.0,
               "3":0.85,
               "4":0.95
            },
            "l":{
               
            },
            "qex":true
         },
         "23":{
            "p":{
               "0":false,
               "1":false
            },
            "l":{
               
            },
            "qex":true
         },
         "25":{
            "p":{
               
            },
            "l":{
               
            },
            "qex":true
         },
         "26":{
            "p":{
               "0":""
            },
            "l":{
               
            },
            "qex":true
         },
         "28":{
            "p":{
               "0":false
            },
            "l":{
               
            },
            "qex":true
         },
         "29":{
            "p":{
               
            },
            "l":{
               
            },
            "qex":true
         },
         "31":{
            "p":{
               
            },
            "l":{
               
            },
            "qex":true
         },
         "33":{
            "p":{
               
            },
            "l":{
               
            },
            "qex":true
         },
         "34":{
            "p":{
               "0":false
            },
            "l":{
               
            },
            "qex":true
         },
         "36":{
            "p":{
               "0":true,
               "1":true,
               "2":false,
               "3":false,
               "4":false
            },
            "l":{
               
            },
            "qex":true
         },
         "37":{
            "p":{
               "0":false
            },
            "l":{
               
            },
            "qex":true
         },
         "39":{
            "p":{
               "0":false,
               "14":false,
               "8":false
            },
            "l":{
               
            },
            "qex":true
         },
         "41":{
            "p":{
               "3":true
            },
            "l":{
               
            },
            "qex":true
         },
         "42":{
            "p":{
               "0":true
            },
            "l":{
               
            },
            "qex":true
         },
         "43":{
            "p":{
               "0":false,
               "1":false,
               "2":false
            },
            "l":{
               
            },
            "qex":true
         },
         "44":{
            "p":{
               "1":"inside_media",
               "2":0.2
            },
            "l":{
               
            },
            "qex":true
         },
         "45":{
            "p":{
               "13":false,
               "17":0,
               "29":false,
               "31":false,
               "32":false,
               "33":true,
               "35":false,
               "36":"control",
               "37":true,
               "38":false,
               "40":"control",
               "45":"control",
               "46":false,
               "47":false
            },
            "l":{
               
            },
            "qex":true
         },
         "46":{
            "p":{
               "0":false
            },
            "l":{
               
            },
            "qex":true
         },
         "47":{
            "p":{
               "0":true,
               "1":true,
               "10":false,
               "11":false,
               "2":false,
               "3":false,
               "9":false
            },
            "l":{
               
            },
            "qex":true
         },
         "49":{
            "p":{
               "0":false
            },
            "l":{
               
            },
            "qex":true
         },
         "50":{
            "p":{
               "0":false
            },
            "l":{
               
            },
            "qex":true
         },
         "54":{
            "p":{
               "0":false
            },
            "l":{
               
            },
            "qex":true
         },
         "58":{
            "p":{
               "0":0.25,
               "1":true
            },
            "l":{
               
            },
            "qex":true
         },
         "59":{
            "p":{
               "0":true
            },
            "l":{
               
            },
            "qex":true
         },
         "62":{
            "p":{
               "0":false
            },
            "l":{
               
            },
            "qex":true
         },
         "67":{
            "p":{
               "0":true,
               "1":true,
               "2":true,
               "3":true,
               "4":false,
               "5":true,
               "7":false
            },
            "l":{
               
            },
            "qex":true
         },
         "69":{
            "p":{
               "0":true
            },
            "l":{
               
            },
            "qex":true
         },
         "71":{
            "p":{
               "1":"^/explore/.*|^/accounts/activity/$"
            },
            "l":{
               
            },
            "qex":true
         },
         "72":{
            "p":{
               "1":false,
               "2":false
            },
            "l":{
               "1":true,
               "2":true
            },
            "qex":true
         },
         "73":{
            "p":{
               "0":false
            },
            "l":{
               
            },
            "qex":true
         },
         "74":{
            "p":{
               "1":true,
               "12":false,
               "13":false,
               "15":false,
               "2":false,
               "3":true,
               "4":false,
               "7":false,
               "9":true
            },
            "l":{
               
            },
            "qex":true
         },
         "75":{
            "p":{
               "0":true
            },
            "l":{
               
            },
            "qex":true
         },
         "77":{
            "p":{
               "1":false
            },
            "l":{
               
            },
            "qex":true
         },
         "80":{
            "p":{
               "3":true,
               "4":false
            },
            "l":{
               
            },
            "qex":true
         },
         "84":{
            "p":{
               "0":true,
               "1":true,
               "2":true,
               "3":true,
               "4":true,
               "5":true,
               "6":false,
               "8":false
            },
            "l":{
               
            },
            "qex":true
         },
         "85":{
            "p":{
               "0":false,
               "1":"Pictures and Videos"
            },
            "l":{
               
            },
            "qex":true
         },
         "87":{
            "p":{
               "0":true
            },
            "l":{
               
            },
            "qex":true
         },
         "93":{
            "p":{
               "0":true
            },
            "l":{
               
            },
            "qex":true
         },
         "95":{
            "p":{
               "0":false,
               "1":false
            },
            "l":{
               
            },
            "qex":true
         },
         "98":{
            "p":{
               "1":false
            },
            "l":{
               
            },
            "qex":true
         }
      },
      "probably_has_app":false,
      "cb":true
   },
   "device_id":"BE440014-5DC8-45A2-838F-E34838A48EB8",
   "browser_push_pub_key":"BIBn3E_rWTci8Xn6P9Xj3btShT85Wdtne0LtwNUyRQ5XjFNkuTq9j4MPAVLvAFhXrUU1A9UxyxBA7YIOjqDIDHI",
   "encryption":{
      "key_id":"25",
      "public_key":"bf9024c7337c48d9ef2ee64525143105404c2c451685477f8b2cbc137d2f1c03",
      "version":"10"
   },
   "is_dev":false,
   "signal_collection_config":null,
   "rollout_hash":"894dd5337020",
   "bundle_variant":"metro",
   "frontend_env":"prod"
}

Any clue as to why this problem occurs and how I could fix it?

Please let me know if you need more information. I am not experienced in PHP at all, so there might be something obvious that I've not included.

And thanks for creating this plugin. It is exactly what I need ;)

[AmauryCarrade]

Hi and thanks for this report!

Well, that's weird. We're reverse-engineering Instagram here so there's no good solution. You're trying to embed the same posts right? Because embeds won't work with private posts as requests are unauthenticated.

Another possible reason would be that your server's IP is recognized by Instagram as some kind of automated IP, or an IP with lots of requests to Instagram, so they are blocking them or presenting them a captcha challenge (that the plugin is unable to solve, naturally). If you have CLI access to your webserver, can you post somewhere (i.e. in a gist) the output of these two commands, executed from your live server?

$ curl -v URL_TO_THE_INSTAGRAM_POST
$ curl -vH "User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0" URL_TO_THE_INSTAGRAM_POST

(Warning, the output will probably be quite long!)

[Bargsteen]

Hi Amaury,

I am trying to embed the exact same posts, yes. I also tried using the website with embeddings after logging out from Instagram and clearing cookies, but they still worked. So that is not the issue.

These are the outputs:

Regular curl

jelinstr@eu12 [~]# curl -v https://www.instagram.com/p/CEMH7uZHSo6/
* About to connect() to www.instagram.com port 443 (#0)
*   Trying 157.240.20.174...
* Connected to www.instagram.com (157.240.20.174) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=*.www.instagram.com,O="Facebook, Inc.",L=Menlo Park,ST=California,C=US
* 	start date: Aug 18 00:00:00 2020 GMT
* 	expire date: Nov 16 12:00:00 2020 GMT
* 	common name: *.www.instagram.com
* 	issuer: CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
> GET /p/CEMH7uZHSo6/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: www.instagram.com
> Accept: */*
>
< HTTP/1.1 302 Found
< Content-Type: text/html; charset=utf-8
< Location: https://www.instagram.com/accounts/login/?next=/p/CEMH7uZHSo6/
< Vary: Accept-Language, Cookie
< Content-Language: en
< Date: Thu, 01 Oct 2020 18:09:06 GMT
< Strict-Transport-Security: max-age=31536000
< Cache-Control: private, no-cache, no-store, must-revalidate
< Pragma: no-cache
< Expires: Sat, 01 Jan 2000 00:00:00 GMT
< X-Frame-Options: SAMEORIGIN
< content-security-policy: report-uri https://www.instagram.com/security/csp_report/; default-src 'self' https://www.instagram.com; img-src https: data: blob:; font-src data: https://*.fbcdn.net https://*.instagram.com https://*.cdninstagram.com; media-src 'self' blob: https://www.instagram.com https://*.cdninstagram.com https://*.fbcdn.net; manifest-src 'self' https://www.instagram.com; script-src 'self' https://instagram.com https://www.instagram.com https://*.www.instagram.com https://*.cdninstagram.com wss://www.instagram.com https://*.facebook.com https://*.fbcdn.net https://*.facebook.net 'unsafe-inline' 'unsafe-eval' blob:; style-src 'self' https://*.www.instagram.com https://www.instagram.com 'unsafe-inline'; connect-src 'self' https://instagram.com https://www.instagram.com https://*.www.instagram.com https://graph.instagram.com https://*.graph.instagram.com https://*.cdninstagram.com https://api.instagram.com https://i.instagram.com wss://www.instagram.com wss://edge-chat.instagram.com https://*.facebook.com https://*.fbcdn.net https://*.facebook.net chrome-extension://boadgeojelhgndaghljhdicfkmllpafd blob:; worker-src 'self' blob: https://www.instagram.com; frame-src 'self' https://instagram.com https://www.instagram.com https://staticxx.facebook.com https://www.facebook.com https://web.facebook.com https://connect.facebook.net https://m.facebook.com; object-src 'none'; upgrade-insecure-requests
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 0
< x-ig-push-state: c2
< x-aed: 20
< Access-Control-Expose-Headers: X-IG-Set-WWW-Claim
< X-FB-TRIP-ID: 1679558926
< Alt-Svc: h3-29=":443"; ma=3600,h3-27=":443"; ma=3600
< Connection: keep-alive
< Content-Length: 0
<
* Connection #0 to host www.instagram.com left intact

User-agent curl

jelinstr@eu12 [~]# curl -vH "User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0" https://www.instagram.com/p/CEMH7uZHSo6/
* About to connect() to www.instagram.com port 443 (#0)
*   Trying 157.240.20.174...
* Connected to www.instagram.com (157.240.20.174) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=*.www.instagram.com,O="Facebook, Inc.",L=Menlo Park,ST=California,C=US
* 	start date: Aug 18 00:00:00 2020 GMT
* 	expire date: Nov 16 12:00:00 2020 GMT
* 	common name: *.www.instagram.com
* 	issuer: CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
> GET /p/CEMH7uZHSo6/ HTTP/1.1
> Host: www.instagram.com
> Accept: */*
> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0
>
< HTTP/1.1 302 Found
< Content-Type: text/html; charset=utf-8
< Location: https://www.instagram.com/accounts/login/?next=/p/CEMH7uZHSo6/
< Vary: Accept-Language, Cookie
< Content-Language: en
< Date: Thu, 01 Oct 2020 18:10:51 GMT
< Strict-Transport-Security: max-age=31536000
< Cache-Control: private, no-cache, no-store, must-revalidate
< Pragma: no-cache
< Expires: Sat, 01 Jan 2000 00:00:00 GMT
< X-Frame-Options: SAMEORIGIN
< content-security-policy: report-uri https://www.instagram.com/security/csp_report/; default-src 'self' https://www.instagram.com; img-src https: data: blob:; font-src data: https://*.fbcdn.net https://*.instagram.com https://*.cdninstagram.com; media-src 'self' blob: https://www.instagram.com https://*.cdninstagram.com https://*.fbcdn.net; manifest-src 'self' https://www.instagram.com; script-src 'self' https://instagram.com https://www.instagram.com https://*.www.instagram.com https://*.cdninstagram.com wss://www.instagram.com https://*.facebook.com https://*.fbcdn.net https://*.facebook.net 'unsafe-inline' 'unsafe-eval' blob:; style-src 'self' https://*.www.instagram.com https://www.instagram.com 'unsafe-inline'; connect-src 'self' https://instagram.com https://www.instagram.com https://*.www.instagram.com https://graph.instagram.com https://*.graph.instagram.com https://*.cdninstagram.com https://api.instagram.com https://i.instagram.com wss://www.instagram.com wss://edge-chat.instagram.com https://*.facebook.com https://*.fbcdn.net https://*.facebook.net chrome-extension://boadgeojelhgndaghljhdicfkmllpafd blob:; worker-src 'self' blob: https://www.instagram.com; frame-src 'self' https://instagram.com https://www.instagram.com https://staticxx.facebook.com https://www.facebook.com https://web.facebook.com https://connect.facebook.net https://m.facebook.com; object-src 'none'; upgrade-insecure-requests
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 0
< x-ig-push-state: c2
< x-aed: 20
< Access-Control-Expose-Headers: X-IG-Set-WWW-Claim
< X-FB-TRIP-ID: 1679558926
< Alt-Svc: h3-29=":443"; ma=3600,h3-27=":443"; ma=3600
< Connection: keep-alive
< Content-Length: 0
<
* Connection #0 to host www.instagram.com left intact

These outputs definitely differ from what I get when I run curl locally. Locally I get all the HTML etc.

[AmauryCarrade]

Hum, I replied via email but the answer's not there… anyway.

Okay, thanks for this data! These outputs reveal the problem: Instagram redirects the requests to an authentication page, even if the post is not private.

I can reproduce this, accessing the post in my browser or using curl on my locale machine, but being redirected if requesting from a server I own (with an IP from a hosting provider, in my case Scaleway). Instagram may block or check requests incoming from known hosting providers…

I'll try to investigate but for now I don't know how to circumvent this Instagram behaviour.

[Bargsteen]

Okay, glad to know it's not just me!
Thanks for investigating the issue.
Let me know if I can help :)

[therealcode]

Just wanted to bump this issue up. Looks like I'm experiencing the same issue with Instagram embeds. My response using CURL on my shared hosting provider returned similar results to @Bargsteen. The error on my page when viewing on the live server looks like the following.

Please let me know if you end up finding a fix or way to circumvent this. Maybe a way to proxy the request using our local machines? Sure, it may break and need updating, but maybe an interim solution?

[AmauryCarrade]

For the moment, I don't have any proper solution, but I'll try to investigate more. I'm short on time to be honest, so it's kinda difficult to work on everything—I apologise for the delays. If anyone want to try things on this and report their findings, feel free! It would be considered.