Public scraping methods fail due to login screen on Instagram on production builds about gatsby-source-instagram

[johnkavanagh]

@oorestisime

Sorry for the trouble this is causing, I made time for working on this tonight. give me a few hours please :)

Just in case nobody else says it: thank you! I know you're not in any way obligated to fix everybody else's broken builds. Your plugin is awesome (when it's working), thank you for looking at it again for us now.

[LarsBehrenberg]

I seem to have the same issue. Just a week ago it seemed to have worked fine.
I am deploying my site with Netlify. With the gatsby develop or build command locally I don't run into issues and even with the NetlifyCLI running the local netlify build command everything is alright. But as soon as I push to Netlify the build fails.

7:18:17 PM: $ yarn build
7:18:17 PM: yarn run v1.22.4
7:18:17 PM: $ gatsby clean && gatsby build
7:18:18 PM: 
7:18:18 PM: info Deleting .cache, public
7:18:18 PM: info Successfully deleted directories
7:18:20 PM: 
7:18:20 PM: success open and validate gatsby-configs - 0.067s
7:18:22 PM: 
7:18:22 PM: success load plugins - 1.694s
7:18:22 PM: 
7:18:22 PM: success onPreInit - 0.016s
7:18:22 PM: success delete html and css files from previous builds - 0.018s
7:18:22 PM: 
7:18:22 PM: success initialize cache - 0.012s
7:18:22 PM: 
7:18:22 PM: success copy gatsby files - 0.047s
7:18:22 PM: 
7:18:22 PM: success onPreBootstrap - 0.010s
7:18:22 PM: 
7:18:22 PM: success createSchemaCustomization - 0.012s
7:18:23 PM: 
7:18:23 PM: warning
7:18:23 PM: Could not fetch instagram posts. Error status TypeError: Cannot read property '0' of undefined
7:18:23 PM: success source and transform nodes - 1.476s
7:18:24 PM: 
7:18:24 PM: success building schema - 0.578s
7:18:24 PM: 
7:18:24 PM: success createPages - 0.240s
7:18:24 PM: success createPagesStatefully - 0.162s
7:18:24 PM: 
7:18:24 PM: success onPreExtractQueries - 0.001s
7:18:24 PM: success update schema - 0.084s
7:18:25 PM: error There was an error in your GraphQL query:
7:18:25 PM: Cannot query field "allInstaNode" on type "Query".
7:18:25 PM: If you don't expect "allInstaNode" to exist on the type "Query" it is most likely a typo.
7:18:25 PM: However, if you expect "allInstaNode" to exist there are a couple of solutions to common problems:
7:18:25 PM: - If you added a new data source and/or changed something inside gatsby-node.js/gatsby-config.js, please try a restart of your development server
7:18:25 PM: - The field might be accessible in another subfield, please try your query in GraphiQL and use the GraphiQL explorer to see which fields you can query and what shape they have
7:18:25 PM: - You want to optionally use your field "allInstaNode" and right now it is not used anywhere. Therefore Gatsby can't infer the type and add it to the GraphQL schema. A quick fix is to add a least one entry with that field ("dummy content")
7:18:25 PM: It is recommended to explicitly type your GraphQL schema if you want to use optional fields. This way you don't have to add the mentioned "dummy content". Visit our docs to learn how you can define the schema for "Query":
7:18:25 PM: https://www.gatsbyjs.org/docs/schema-customization/#creating-type-definitions
7:18:25 PM: not finished Generating image thumbnails - 1.265s
7:18:25 PM: failed extract queries from components - 0.823s

[oorestisime]

#158 fixed the likes issue. i ll release in 2 days

[oorestisime]

I just published 0.8.0-beta.0 with the change. My netlify build passed that stepped but failed because hashtag scraping still doesn't work. I would like some more feedback from you people though.

In that version public scraping of hashtags and profile information is not working. If any of you knows of a way to public scraping hashtags then i am all eyes to get it to work.

Public scraping for public posts is supposed to work though.

In order to use this you need to pass an instagram id instead of a username. I used https://codeofaninja.com/tools/find-instagram-user-id to get an id. (You pass it in the username config, i want to make sure this work before i jump on API changes).

Another change is on the instagram node the username is also going to be the instagram id you provided. Unfortunately instagram doesn't pass in the username when querying like that.

I managed to 50 posts instead of 12 so this is good news.

[oorestisime]

Ok folks, i ll wait another couple of days for some more feedback and then release 1.0 with the list of breaking changes and updated doc section! thank you all for your patience :)

[pul87]

I'm having the same problem on Netlify, seems an issue related to the platform, on development it works.

[oorestisime]

I am seeing this now on netlify as well. locally works fine. I am investigating right now

[oorestisime]

Well now it went through on netlify. I think maybe network issue? I can't find any way to reproduce locally so not sure what i can do here :/

[ashishterp]

I'm having the same issue on Netlify as of last week. I forked a copy of the plugin and added some logging and it looks like Instagram is returning a login page for some reason:

2:11:52 AM: { LoginAndSignupPage:
2:11:52 AM:    [ { captcha: [Object],
2:11:52 AM:        gdpr_required: false,
2:11:52 AM:        tos_version: 'row',
2:11:52 AM:        username_hint: '' } ] }

I tried faking the User Agent on the axios request and that didn't work either. What's weird is that it works fine locally, so maybe it is something on the IG side about where the request is coming from that is triggering it generating a login page.

[oorestisime]

I went through the steps again. here's what i found and let me know if this works for you and if it is clearer:

1. You need to have a Facebook page (I know... :/)
2. Go to your site settings -> Instagram -> Login into your Instagram account
3. Create a app
4. Go to the Graph API Explorer
   1. Make sure you are using v7 as api version
   2. Select your facebook app
   3. Click "Generate Access Token"
   4. Add the following permissions (pages_manage_ads, pages_manage_metadata, pages_read_engagement, pages_read_user_content, pages_show_list, instagram_basic)
   5. Make a GET request at me/accounts
   6. copy the access_token in the response (we call this temporary_token)
   7. click on the id to change the explorer url and append ?fields=instagram_business_account&access_token={access-token}
   8. save your instagram_business_account.id, this is your instagram_id
5. Access Token Debugger:
   1. Paste your temporary_token and press "Debug"
   2. You should see this token now expires in 3 months
   3. Press "Extend Access Token" and press the new debug that appears next to the token
   4. You should see this token now never expires
   5. Copy this new token (we will call this access_token)

With these two information you can now use the plugin as:

{
  resolve: `gatsby-source-instagram`,
  options: {
    username: username,
    access_token: access_token,
    instagram_id: instagram_id,
  },
},

[oorestisime]

i think this explains it good https://github.com/oorestisime/gatsby-source-instagram#public-scraping-for-posts need to pass the concerning username id. You can find it https://codeofaninja.com/tools/find-instagram-user-id
if this is not clear feel free to PR something that would have helped you :)

[xanderjl]

I've also been running into this issue. Noticed a string of failed builds on the deploy dashboard with the same error. I hope your Netlify post brings attention to the problem!

[oorestisime]

I am happy to accept any PRs on that end. It's been a year i did those steps myself and i remember it was quite tedious.

[LarsBehrenberg]

If you could join the netlify post and let people know you have had the same issue, maybe it will gain more attention. Would really appreciate it! Thanks so much here already for all the help!

[hzburki]

The solution provided in this comment worked for public posts. Netlify build succeeded but I had to remove likes from the query.

[oorestisime]

I just released 0.8
https://github.com/oorestisime/gatsby-source-instagram/releases/tag/v0.8.0

Make sure to change your instagram id.

[oorestisime]

Yeah it crossed my mind right after i hit release :/
i ll do this in 1.0 which is as soon as i figure out how to properly get a token for the graph api!

[oorestisime]

Sorry for the trouble this is causing, I made time for working on this tonight. give me a few hours please :)

[oorestisime]

Hey folks thanks for the kind words. I released 0.7.1 with the new endpoint modification but it seems problem persist. If any of you could test as well to confirm this 🙏

Not sure what next steps are if this persists :(

[oorestisime]

@sjelfull Please let me know anything confusing or inconsistent i ll try to help you generate the token and update the docs accordingly :)
Same for anybody else!

[oorestisime]

For the permissions you will need to add the pages_manage_ads, pages_manage_metadata, pages_read_engagement, and pages_read_user_content,pages_show_list, instagram_basic (basically the ones listed in step 2 without manage_pages but with the ones it got replaced with). Hope this gets you unblocked.

I ll try tomorrow to go through all the steps and leave an update here. Thanks this was very helpful!

[LarsBehrenberg]

I started using this approach for public scraping instead. I wonder wether the way images are being fetched with that approach could also be implemented in gatsby-source-instagram?

[zamson]

0.8.0-beta.0 has worked for me both locally and through Netlify. I'm going through the steps to authenticate now for a more long term fix.

Worked how, with public scraping?

[MilosJo]

I went through the steps again. here's what i found and let me know if this works for you and if it is clearer:

1. You need to have a Facebook page (I know... :/)

2. Go to your site settings -> Instagram -> Login into your Instagram account

3. Create a app

4. Go to the Graph API Explorer

   1. Make sure you are using v7 as api version
   2. Select your facebook app
   3. Click "Generate Access Token"
   4. Add the following permissions (pages_manage_ads, pages_manage_metadata, pages_read_engagement, pages_read_user_content, pages_show_list, instagram_basic)
   5. Make a GET request at me/accounts
   6. copy the access_token in the response (we call this temporary_token)
   7. click on the id to change the explorer url and append ?fields=instagram_business_account&access_token={access-token}
   8. save your instagram_business_account.id, this is your instagram_id

5. Access Token Debugger:

   1. Paste your temporary_token and press "Debug"
   2. You should see this token now expires in 3 months
   3. Press "Extend Access Token" and press the new debug that appears next to the token
   4. You should see this token now never expires
   5. Copy this new token (we will call this access_token)

With these two information you can now use the plugin as:

{
  resolve: `gatsby-source-instagram`,
  options: {
    username: username,
    access_token: access_token,
    instagram_id: instagram_id,
  },
},

Regarding step 1. Do you mean a business account or my personal page? I'm trying to setup an fb app for a client but the instagram capabilities does not show up. The Instagram account is linked to their business account. Puh.. wish public scraping was working.

Following this steps my gatsby site is finally working and fetching instagram posts.
Only thing I didn't manage to change is to extend 3months token expiration.
Can confirm that these steps are legit.

Thanks @oorestisime

[yone920]

Any good news on this problem?

[andregmoeller]

First off, thank you for all the work you put into developing and maintining this plugin! Probably it is too late now, but I feel that the configuration parameter / option 'username' should have been renamed to 'instagram_id' – the semantic meaning of the option username has changed. It is not any longer the username.

[maxsteenbergen]

@ngerbauld Are you sure you're using the Instagram ID (all numbers) instead of the username? Confusing, I know, but it should be

{
  resolve: gatsby-source-instagram,
  options: {
      username: 1234567,
   }
}

with the instagram_id and access_token not needed, if I'm not mistaken

[oorestisime]

This is odd.
I pushed a new yarn lock file in the example. i forgot to do so. can you pull yarn install and try again?

[wjx0820]

Thanks for the quick reply but it still not work...
Wait a long time and said could not fetch instagram posts...
Is it a temporary situation?

[oorestisime]

It works for me. I am not sure what is happening locally for you. Are you able to go on instagram? was it working for you and just broke or never worked?

[wjx0820]

I can go to instagram in the browser. But still not work.

Error message like these:
Could not fetch instagram posts. Error status Error: write EPROTO 4472038848:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:252:

warning The gatsby-source-instagram plugin has generated no Gatsby nodes. Do you need it?

My location is China, it must be the issue with GFW...I tried use proxy in the terminal, then npm run develop, but still wait for a long time then failed...😭

[oorestisime]

:( I am really sorry but i am not sure how i can help here :( You will need to try with a vpn or something.

[calpa]

@Jexxie Instagram maybe blocked in China according to the "law", so you may need to find other way.

This should not be an problem of this plugin.

[oorestisime]

Closing this! feel free to reopen if you think there's something to be done in the scope of the plugin!

[tinoguti]

I am having trouble with this same description when I try to build my Gatsby app on AWS but when I try locally somehow it works. I thought I solved the problem by updating npm packages and I had 2 successful builds but today it is not building. Any help on how to solve this problem?
Here's a copy of my log with the part where everything seems to start failing:

2020-05-10T16:23:33.187Z [INFO]: success createSchemaCustomization - 0.073s
2020-05-10T16:23:33.560Z [WARNING]: warning
                                    Could not fetch instagram posts. Error status TypeError: Cannot read property '0' of undefined
2020-05-10T16:23:33.708Z [WARNING]: warning The gatsby-source-instagram plugin has generated no Gatsby nodes. Do you need it?
2020-05-10T16:23:33.708Z [INFO]: success source and transform nodes - 0.521s
2020-05-10T16:23:33.966Z [INFO]: success building schema - 0.257s
2020-05-10T16:23:34.017Z [INFO]: success createPages - 0.049s
2020-05-10T16:23:34.082Z [INFO]: success createPagesStatefully - 0.065s
2020-05-10T16:23:34.082Z [INFO]: success onPreExtractQueries - 0.000s
2020-05-10T16:23:34.111Z [INFO]: success update schema - 0.028s
2020-05-10T16:23:34.446Z [WARNING]: error There was an error in your GraphQL query:
                                    Cannot query field "allInstaNode" on type "Query".
                                    If you don't expect "allInstaNode" to exist on the type "Query" it is most likely a typo.
                                    However, if you expect "allInstaNode" to exist there are a couple of solutions to common problems:
                                    - If you added a new data source and/or changed something inside gatsby-node.js/gatsby-config.js, please try a restart of your development server
                                    - The field might be accessible in another subfield, please try your query in GraphiQL and use the GraphiQL explorer to see which fields you can query and what shape they have
                                    - You want to optionally use your field "allInstaNode" and right now it is not used anywhere. Therefore Gatsby can't infer the type and add it to the GraphQL schema. A quick fix is to add a least one entry with that field ("dummy content")
                                    It is recommended to explicitly type your GraphQL schema if you want to use optional fields. This way you don't have to add the mentioned "dummy content". Visit our docs to learn how you can define the schema for "Query":
                                    https://www.gatsbyjs.org/docs/schema-customization/#creating-type-definitions
2020-05-10T16:23:34.449Z [INFO]: failed extract queries from components - 0.337s
2020-05-10T16:23:34.514Z [WARNING]: npm
2020-05-10T16:23:34.515Z [WARNING]: ERR! code ELIFECYCLE
                                    npm ERR! errno 1
2020-05-10T16:23:34.515Z [WARNING]: npm

[tinoguti]

Now I am intrigued. I've just redeployed my app and it worked. Any idea why this seems to randomly fail? I wouldn't like to have a few failed builds hoping the next one will be the one every time I need to deploy.

[oorestisime]

Hey there, kind of hard to know without reproduction :/

Is it with public scraping or with the graph api?
Where does this run? amplify?

[tinoguti]

Hi. Yes it's running on amplify and with public scrapping, with just an username added to the plugin config parameters.

GraphQL query looks like this:
`
allInstaNode(limit: 8) {

  edges {
      node {
          id
          username
          caption
          localFile {
            childImageSharp {
              fixed(width: 500, height: 500) {
                ...GatsbyImageSharpFixed
              }
            }
          }
      }
  }
}

`
It works well with gatsby devleop and gatsby build locally. But having "random" build fails on amplify.

[oorestisime]

Yeah query isn't the issue. the error you showed above is when it couldn't get the instagram posts. maybe something wrong with instagram during that particular time? does it still happening?

I just triggered another rebuild on netlify for the example app and seems to be working fine

[LarsBehrenberg]

I am really not sure what is happening here, but it seems like this is not the plugins fault? I deployed the same built a couple times on Netlify every time clearing cache before building. The first two times everything works fine and then the 3rd time it breaks.

warning
11:15:23 AM: Could not fetch instagram user. Error status TypeError: Cannot read property '0' of undefined
11:15:23 AM: 
11:15:23 AM: error "gatsby-source-instagram" threw an error while running the sourceNodes lifecycle:
11:15:23 AM: Cannot read property 'id' of null
11:15:23 AM:   74 |   return {
11:15:23 AM:   75 |     type: params.type,
11:15:23 AM: > 76 |     id: datum.id,
11:15:23 AM:      |               ^
11:15:23 AM:   77 |     full_name: datum.full_name,
11:15:23 AM:   78 |     biography: datum.biography,
11:15:23 AM:   79 |     edge_followed_by: datum.edge_followed_by,
11:15:23 AM: 
11:15:23 AM: 
11:15:23 AM: 
11:15:23 AM:   TypeError: Cannot read property 'id' of null
11:15:23 AM:   
11:15:23 AM:   - gatsby-node.js:76 createUserNode
11:15:23 AM:     [repo]/[gatsby-source-instagram]/gatsby-node.js:76:15
11:15:23 AM:   
11:15:23 AM:   - gatsby-node.js:91 processDatum
11:15:23 AM:     [repo]/[gatsby-source-instagram]/gatsby-node.js:91:49
11:15:23 AM:   
11:15:23 AM:   - gatsby-node.js:125 
11:15:23 AM:     [repo]/[gatsby-source-instagram]/gatsby-node.js:125:16
11:15:23 AM:   
11:15:23 AM:   - Array.map
11:15:23 AM:   
11:15:23 AM:   - gatsby-node.js:123 Object.exports.sourceNodes
11:15:23 AM:     [repo]/[gatsby-source-instagram]/gatsby-node.js:123:29
11:15:23 AM:   
11:15:23 AM:   - task_queues.js:97 processTicksAndRejections
11:15:23 AM:     internal/process/task_queues.js:97:5
11:15:23 AM:   
11:15:23 AM: 
11:15:23 AM: not finished source and transform nodes - 1.120s

Btw, I am not using an Instagram API, maybe this makes a difference?

[LarsBehrenberg]

Anything we can do about it? Some workaround?

[oorestisime]

Well the issue is not with the plugin afaict. it is happening when scraping the page doesn't work. But i can't find out whether this is an issue on Netlify (much more likely since it never happens in dev) itself or if Instagram is testing things out. After all the public scraping is extemely dependent on their raw html code :)

Only thing i can susggest if use the API. it worked the whole time i was testing.

Has anybody tried to contact Netlify support to see if anything is off in the network?

[oorestisime]

I am re-opening this for visibility among folks checking the issues.

[dmcreis]

Same problem happening here. I tried a couple of times today and the build is failing. is there any workaround for this? @oorestisime thanks for all your replies :)

[tonilaukka]

I'm having the same issue although my builds fail randomly and hitting redeploy helps.

This is the error message:

11:55:31 PM: error There was an error in your GraphQL query:
11:55:31 PM: - Unknown field 'allInstaNode' on type 'Query'.
11:55:31 PM: failed extract queries from components - 0.464s

[LarsBehrenberg]

@oorestisime thanks for reopening this issue!

I have posted this on the Netlify community as well.
https://community.netlify.com/t/deploying-a-gatsby-site-on-netlify-using-the-gatsby-source-instagram-plugin-failing-on-build-time-issue-with-netlify-fetching-data-from-instagram-cannot-reproduce-locally/15957

[oorestisime]

So from #152 i am reposting the same link here https://stackoverflow.com/questions/57624387/dont-have-profilepage-index-but-i-have-loginandsignuppage
The issue is Instagram over the last week or so has restricted their unlogged-in (guest) access (based on IP address).

This is why we have a login now.

[oorestisime]

I am sorry this is failing for you folks, i ll try over the next days to work something out using this #131 (comment)
hopefully that won't get a login screen. I don't have much time though in the next 2 days so if anyone wants to come up with a PR, more than welcome :)

[tinoguti]

I would just like to add that I am also having this issue on my Amazon Amplify app. I went through 8 failed builds to get one right.

[LarsBehrenberg]

@oorestisime
Yes, have to agree with @johnkavanagh! Thanks so much for the plugin in the first place and all the effort to keep it updated!

[xanderjl]

Can confirm the issue is still persisting on 7.1

10:41:42 AM: error There was an error in your GraphQL query:
10:41:42 AM: Cannot query field "allInstaNode" on type "Query".
10:41:42 AM: If you don't expect "allInstaNode" to exist on the type "Query" it is most likely a typo.
10:41:42 AM: However, if you expect "allInstaNode" to exist there are a couple of solutions to common problems:
10:41:42 AM: - If you added a new data source and/or changed something inside gatsby-node.js/gatsby-config.js, please try a restart of your development server
10:41:42 AM: - The field might be accessible in another subfield, please try your query in GraphiQL and use the GraphiQL explorer to see which fields you can query and what shape they have
10:41:42 AM: - You want to optionally use your field "allInstaNode" and right now it is not used anywhere. Therefore Gatsby can't infer the type and add it to the GraphQL schema. A quick fix is to add a least one entry with that field ("dummy content")
10:41:42 AM: It is recommended to explicitly type your GraphQL schema if you want to use optional fields. This way you don't have to add the mentioned "dummy content". Visit our docs to learn how you can define the schema for "Query":
10:41:42 AM: https://www.gatsbyjs.org/docs/schema-customization/#creating-type-definitions
10:41:42 AM: failed extract queries from components - 1.171s

Thank you for your quick response! Appreciate all of the work you've put into this :)

[oorestisime]

I am trying to think of next steps here:

• if anyone manages to reproduce locally i would love the html of the page to see what's included in that new screen. Or if you are using the new url with ?__a=1 in it then just that.

• for anyone not midning to create an access token and go through https://github.com/oorestisime/gatsby-source-instagram#instagram-graph-api-token then this is an alternative.

[marcusps11]

Same problem here can not produce locally only when i try and build - at least i know i am not going mad.

[cbaucom]

I can also confirm this is an issue for me when trying to build on Netlify. Failed when I was at version 0.5.0 and also after updating to 0.7.1.

12:33:39 AM: Could not fetch instagram posts. Error status TypeError: Cannot read property 'user' of undefined
12:33:43 AM: warning The gatsby-source-instagram plugin has generated no Gatsby nodes. Do you need it?
12:33:43 AM: success source and transform nodes — 4.429 s
12:33:43 AM: success building schema — 0.400 s
12:33:43 AM: success createPages — 0.027 s
12:33:43 AM: success createPagesStatefully — 0.043 s
12:33:43 AM: success onPreExtractQueries — 0.002 s
12:33:43 AM: success update schema — 0.024 s
12:33:44 AM: error GraphQL Error Encountered 1 error(s):
12:33:44 AM: - Unknown field 'allInstaNode' on type 'Query'.
12:33:44 AM:       file: /opt/build/repo/src/components/Instagram.js

Does not fail for me locally on either develop or build

[wjchat]

Does your plugin use Instagram's newer Graph API or the legacy API? BTW, it is a really great plugin. I'm having similar issues with my app deployed on AWS amplify. Reading about how the legacy api is being disabled by Instagram. maybe this has something to do with it?

[oorestisime]

The api uses the newer Graph api and afaiu it is not affected.

[oorestisime]

I logged the output, and its a plain login wall. I can't find anything usefull there. Not sure what i can do next folks. seems like i need to remove public scraping methods.

I d love your input here.

[DevanB]

I've been trying to follow along with the instructions to setup an access token, but they seem outdated and/or not thorough enough to follow. If you do remove public scraping, perhaps someone can give a good walk through of setting up and getting an access token with screenshots and/or videos.

[luchoster]

I've been thinking about making a port from this plugin, to a React component that we can just pull data on page load. Finally last night I decided to start it, and today I encounter this problem with one of the sites I maintain.

Anyway, long story short I published: react-ig - It displays a grid of the posts, at the moment I only had the hashtag working, but I just added the username prop:

<InstagramPosts username="vegas" />

And you'll get the latest 12 posts, this will hit the ig user url, on every page load, so you'll get fresh posts every time. Or use the hashtag, as it says on the readme.

I'm only posting it here, in case anyone needs a quick fix on their build and this could help.

[luchoster]

@luchoster Thanks for this, is the UI configurable?

Not really, but if you want to create an issue https://github.com/luchoster/react-ig/issues I'll see what I can do.

[diogocapela]

I'm having the same problem. The development and build instances both work locally but they fail when deployed on to Netlify. Here is the Netlify deploy error log:

1:13:09 AM: error There was an error in your GraphQL query:
1:13:09 AM: Cannot query field "allInstaNode" on type "Query".
1:13:09 AM: If you don't expect "allInstaNode" to exist on the type "Query" it is most likely a typo.
1:13:09 AM: However, if you expect "allInstaNode" to exist there are a couple of solutions to common problems:
1:13:09 AM: - If you added a new data source and/or changed something inside gatsby-node.js/gatsby-config.js, please try a restart of your development server
1:13:09 AM: - The field might be accessible in another subfield, please try your query in GraphiQL and use the GraphiQL explorer to see which fields you can query and what shape they have
1:13:09 AM: - You want to optionally use your field "allInstaNode" and right now it is not used anywhere. Therefore Gatsby can't infer the type and add it to the GraphQL schema. A quick fix is to add a least one entry with that field ("dummy content")
1:13:09 AM: It is recommended to explicitly type your GraphQL schema if you want to use optional fields. This way you don't have to add the mentioned "dummy content". Visit our docs to learn how you can define the schema for "Query":
1:13:09 AM: https://www.gatsbyjs.org/docs/schema-customization/#creating-type-definitions
1:13:09 AM: failed extract queries from components - 0.220s
1:13:09 AM: npm
1:13:09 AM:  ERR! code ELIFECYCLE
1:13:09 AM: npm ERR!
1:13:09 AM:  errno 1
1:13:09 AM: npm ERR! [email protected] build: `gatsby build`
1:13:09 AM: npm ERR! Exit status 1
1:13:09 AM: npm ERR!
1:13:09 AM: npm ERR! Failed at the [email protected] build script.
1:13:09 AM: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
1:13:09 AM: npm
1:13:09 AM:  ERR! A complete log of this run can be found in:
1:13:09 AM: npm ERR!     /opt/buildhome/.npm/_logs/2020-06-05T00_13_09_340Z-debug.log

[zamson]

Same problem here. So is it only public fetching that is not working?

Public scraping fails locally and on Netlify build

Using [email protected]

[oorestisime]

Yes public scraping fails due to their login screen but the graph api works.

if you are having issues generating a token then this might help you #156 cc @DevanB

I ll wait a few more days in case something else on that doc section needs modification before i go and update it.

[tinoguti]

Could someone sum up the current status up to this point?
I will need to do a build of my app sometime soon but it seems like the build won't work.
What's the alternative?
Thank you

[sjelfull]

@tinoguti The only alternative is doing authorized calls to the Graph API.

I'm in the middle of generating a access token, and its confusing as hell, especially as the instructions doesn't match what i'm seeing.

[oorestisime]

I know of one inconsistency on the docs already which is the permissions because last month they changed. https://developers.facebook.com/docs/facebook-login/permissions/#reference-manage_pages

what else is missing? if you folks don't let me know i can't actually update them :/

[sjelfull]

Let me see if I can boil it down - right now i'm so far down the hole that i'm not sure where I went wrong.

I'll add notes inline in the steps:

1. You need to have a Facebook page (I know... :/)
2. Go to your site settings -> Instagram -> Login into your Instagram account
3. Create a app As far as I can tell, after this step, we need to add Instagram below the heading Add a Product on the FB apps dashboard.
4. Go to the Graph API Explorer
   1. Select your App from the top right dropdown menu Its unclear what you mean by the top right dropdown menu here. If you select the app under My apps, which is the only relevant link I can see in the top right corner, you get brough out of the Graph explorer and to the dashboard
   2. Select "Get User Access Token" from dropdown (right of access token field) and select needed permissions (manage_pages, pages_show_list, instagram_basic) Sounds like the UI has changed a bit here. The previous step, where you select the app, is now below the Get Access Token button, in the field labeled Facebook App
   3. Click "Generate Access Token" Which permissions do we need to ask for for this to be able to access the right Instagram account? What fields/path do we use in the explorer?
   4. Copy user access token
5. Access Token Debugger:
   1. Paste copied token and press "Debug"
   2. Press "Extend Access Token" and copy the generated long-lived user access token
6. Graph API Explorer:
   1. Paste copied token into the "Access Token" field
   2. Make a GET request with "PAGE_ID?fields=access_token"
   3. Find the permanent page access token in the response (node "access_token")
7. Access Token Debugger:
   1. Paste the permanent token and press "Debug"
   2. "Expires" should be "Never"
   3. Copy the access token
8. Graph API Explorer:
   1. Make a GET request with "PAGE_ID?fields=instagram_business_account" to get your Business ID The business ID does not show up here. I only get the id of the page and nothing else.

[hvitis]

I´m having the same issue on updated plugin, on local works fine.

Can I help in any way?

[oorestisime]

Unfortunately the public scraping method doesn't work on build in infrastructure. not sure when they decide to add the login screen either IP based or some kind of rate limit.
In any case i think we can't bypass it (but open to suggestions).

You can test generating the access token in case you find more inconsistencies or you might want to open a PR to update them :)

[tinoguti]

I started using this approach for public scraping instead. I wonder wether the way images are being fetched with that approach could also be implemented in gatsby-source-instagram?

This approach is working for me so far. Thanks. Although I wonder for how long it will work because it seems like it depends on a query id that might change in the future.

[oorestisime]

Yeah not sure i udnerstand what query id is. also, have you tried this request from a build such as netlify?

[will-t-harris]

I'm also not sure what that query id is, but I just built that solution with netlify and it works.

I also refactored it into a hook, which I'm considering throwing on npm. At least I was before I realized it was using that query_id. I'd be shocked if that didn't change in the future.

https://gist.github.com/will-t-harris/4b8671315fb77d541b6ad34276acfdb6

[will-t-harris]

It seems that query_id has been the same since 2017...perhaps this approach is more reliable than it looks at first glance.

https://stackoverflow.com/a/47243409/4552841

[tinoguti]

I was able to do a build on Amplify with that query id approach. It's working fine for now. I did try the whole Facebook develop access token process with no luck. It's quite tedious and requires time to set up.

[oorestisime]

OK great thanks folks, i ll add this in 0.7.3 to test. unless somebody wants to PR this :D

[oorestisime]

I am trying this now. i ll keep you posted

[will-t-harris]

Thanks for trying that out so quickly @oorestisime.

That version seems to work for me, locally and on my build in netlify.

[dmcreis]

thanks a lot for this @oorestisime . i will have a go at the new update. when i do npm outdated the only last version that appears available is 0.7.2. any idea why?

[oorestisime]

I think because i published a beta one. can you manually replace the version in package.json and then npm install?

[abepuentes]

Thanks @oorestisime, I've been testing 0.8.0-beta.0 and its works on develop en on netlify. One comment though, searching for this issue I found the following on instagram API docs. The ID you get from https://codeofaninja.com/tools/find-instagram-user-id es whats facebooks its calling "Legacy ID" and will be deprecated on September 30th, 2020. On the new graph.instagram.com API this field is call ig_id more infor here https://developers.facebook.com/docs/instagram-basic-display-api/. I try the new user "ID" field and doenst work with the "query_id" method.

[oorestisime]

Yes indeed its two different ids and i couldn't get it to work with the new id either. Probably that means that this method will work only until then :/ unless there's a way to specify ig_id in the query. i ll test this out tomorrow and get back here.

Still maybe a nice trade-off for a few months giving us time to find new solutions :)

[homearanya]

HI @oorestisime,

Thanks for your efforts. I follow your steps above and it worked great for 2 of 3 Instagram accounts I manage. However, on the 3rd one I get the following error:

Could not download file, error is failed to process https://scontent.cdninstagram.com/v/t51.2885-15/20394100_1101681946599058_7213100928830799872_n.jpg?_nc_cat=106&_nc_sid=8ae9d6&_nc_eui2=AeHWKYmdMLbZjaOXZuGs-eS4D0Tz10hy7CAPRPPXSHLsIDftwoCaG0zGDF4xQrR4-Yw&_nc_ohc=f4ppFsxMW14AX94zd7i&_nc_ht=scontent.cdninstagram.com&oh=bdf4d988d590b0ce1268b3ff9c5d8c09&oe=5F0272A5

TimeoutError: Timeout awaiting 'request' for 30000ms

for every single post of that account and none of the posts are accessible by graphql.

Any idea why this is happening?

Thanks in advance

[oorestisime]

Sorry no idea :(

[dbertella]

Too much pain to go through fb registration and ig apis, Beta version worked for me. Hopefully we will find a solution to make it permanent. Also likes are not supported with the new scraping apparently, my build was failing because I included them in my query.

[jpmarra]

0.8.0-beta.0 has worked for me both locally and through Netlify. I'm going through the steps to authenticate now for a more long term fix.

[zamson]

I went through the steps again. here's what i found and let me know if this works for you and if it is clearer:

1. You need to have a Facebook page (I know... :/)

2. Go to your site settings -> Instagram -> Login into your Instagram account

3. Create a app

4. Go to the Graph API Explorer

   1. Make sure you are using v7 as api version
   2. Select your facebook app
   3. Click "Generate Access Token"
   4. Add the following permissions (pages_manage_ads, pages_manage_metadata, pages_read_engagement, pages_read_user_content, pages_show_list, instagram_basic)
   5. Make a GET request at me/accounts
   6. copy the access_token in the response (we call this temporary_token)
   7. click on the id to change the explorer url and append ?fields=instagram_business_account&access_token={access-token}
   8. save your instagram_business_account.id, this is your instagram_id

5. Access Token Debugger:

   1. Paste your temporary_token and press "Debug"
   2. You should see this token now expires in 3 months
   3. Press "Extend Access Token" and press the new debug that appears next to the token
   4. You should see this token now never expires
   5. Copy this new token (we will call this access_token)

With these two information you can now use the plugin as:

{
  resolve: `gatsby-source-instagram`,
  options: {
    username: username,
    access_token: access_token,
    instagram_id: instagram_id,
  },
},

Regarding step 1. Do you mean a business account or my personal page? I'm trying to setup an fb app for a client but the instagram capabilities does not show up. The Instagram account is linked to their business account. Puh.. wish public scraping was working.

[jpmarra]

0.8.0-beta.0 has worked for me both locally and through Netlify. I'm going through the steps to authenticate now for a more long term fix.

Worked how, with public scraping?

Sorry for the incomplete response. the beta release allowed for public scraping both locally and through Netlify.

I ended up creating an auth token by following the steps listed by @oorestisime, however I could only extend the access token three months, as stated by @MilosJo.

[NeversSync]

Having the same issue. Just attempted to go through the GraphApi setup (tedious and still not confident I did it correcty) and am getting this error both on development and deployment to Netlify.

Netlify build error:

Here is my package, config and query:
<StaticQuery query={graphql
query myQuery {
allInstaNode {
edges {
node {
localFile {
childImageSharp {
fluid(maxHeight: 500, maxWidth: 500, quality: 90) {
...GatsbyImageSharpFluid_withWebp
}
}
}
}
}
}
}
}

Config:
{ resolve: gatsby-source-instagram, options: { username: steadyhandtea, access_token: '{access token}', instagram_id: '{business id}, } },

Thanks for all the work on this! Hope it can all be resolved.

[mattortiz]

Hi @oorestisime ,

I used the instructions provided above. As with prior comments, the instructions are difficult to follow and I'm not sure if I'm doing it correctly as a result. I did in the end get the instagram-id, temp-access-token and extended access-token (but only good through august 12, for some reason).

Still getting the following error (locally):
11:9 error Cannot query field "allInstaNode" on type "Query" graphql/template-strings

My gatsby.config settings with input from @NeversSync above:
resolve: gatsby-source-instagram,
options: {
username: "envyforge",
access_token: access-token, // extended access token from instructions
instagram_id: instagram-id, // instagram id from instructions
},

package.json setting:
"gatsby-source-instagram": "0.8.0-beta.0",

Just providing input. I look forward to the fix and instructions.

Take care,

Matt

[oorestisime]

Can anyone try getting a permanent token using this https://github.com/Bnjis/Facebook-permanent-token-generator ? if this works then probably i could do this inside the plugin directly and spare people some manual steps.

other areas of research someone can help me with is whether there's a way to automate getting even a short lived token. The plugin doesn't really need a permanent token if we manage to automate the process of getting a short lived one because it will get a new token on each build.

In the meantime i ll handle release of 0.8.

once again sorry for the trouble this is causing

[Mike-Huggins]

Sorry not sure I understand the thread here and the docs might not have been updated. My graphql looks like:

      allInstaNode(limit: 5) {
        nodes {
          id
          caption
          username
          localFile {
            childImageSharp {
              fluid(maxWidth: 200, maxHeight: 200, quality: 100) {
                ...GatsbyImageSharpFluid_withWebp
              }
            }
          }
        }
      }
    }

and my gatsby config:  {
      resolve: 'gatsby-source-instagram',
      options: {
        username: 'username',
      },
    },```

what do I need to do to get this working on netlify deploy please? 

[Mike-Huggins]

My apologies, this was entirely my own fault. Thanks for your effort to fix. I thought I had updated to 0.8 but I had not, for anyone following the thread then this is what my code now looks like with version 0.8 and adding in the user id as above:

  const data = useStaticQuery(graphql`
    query {
      allInstaNode(limit: 5, sort: { fields: [timestamp], order: DESC }) {
        edges {
          node {
            id
            caption
            localFile {
              childImageSharp {
                fluid(maxWidth: 200, maxHeight: 200, quality: 100) {
                  ...GatsbyImageSharpFluid_withWebp
                }
              }
            }
          }
        }
      }
    }
  `);

  return data.allInstaNode.edges.map(node => ({
    ...node.node.localFile.childImageSharp,
    id: node.node.id,
    caption: node.node.caption,
    username: node.node.username,
  }));
};```

[mattortiz]

@oorestisime , the 0.8.0 release and updated documentation got me freed up. Thank you very much!

Matt

[maxsteenbergen]

I'm happy to have found this thread! In my case, we only used the hashtag scraping which isn't fixed in 0.8.0 yet. Can we expect that to return some day or has instagram permanently broken that?

[oorestisime]

i haven't found any way to do it yet. i also haven't looked if the Graph Api can return this information.

Happy to receive help on that end :)

[maxsteenbergen]

I'd love to help out, even though my experience is limited. Any ideas as to why cloud builders fail whereas local builds work? (Note: it's not just Netlify, GitLab usually fails too but sporadically succeeds)

[oorestisime]

Instagram is adding a login screen . i suppose it is adding for when something is requesting it a lot without login but we can't know.

[ngerbauld]

Hi @oorestisime !

I am not sure what I am missing!

After updating the gatsby-source-instagram to version 0.8.0 and following the #24 (comment)
when I run on localhost I get:
"The gatsby-source-instagram plugin has generated no Gatsby nodes", and when I check in graphql its not there "allInstaNodes".

Any idea what might be?

package.json ----> "gatsby-source-instagram": "^0.8.0"

gatsby.config --->
{
resolve: gatsby-source-instagram,
options: {
username: 'instagram_username',
access_token: 'access_token',
instagram_id: 'id',
},
}

Thank you for the help :)

[oorestisime]

if you are using an access token then you don't need a username. its intentended to be as a fallback.
now if you don't have a valid access token that would happen indeed.
is this the first time using the plugin? or was it working and you just updated?

[ngerbauld]

I was using before on version 0.7.0 and it was working, but not anymore. On localhost was working great but when try to build in Netlify was giving errors (just like many people reported up in the conversation).

So I updated to version 0.8.0 but couldn't get the query anymore.

The access token that I got was only valid for 3 months, until August. But should work anyway, right?