I'm trying to create an automated process using Google-apps-script for scraping price data from pages like this one:
https://www.barchart.com/stocks/quotes/$AVVN/price-history/historical
The challenging part is, that the data on the web page is 'lazy loaded', so the 'traditional' scaping methods, that I have used on other web pages, don't work here.
I have considered other ways of solving this problem - but:
- Barchart does not provide data for e.g. $AVVN via http:
//marketdata.websol.barchart.com/getHistory
- I don't want to use the 'Download'-button - as this requires
automated login.
- ImportXML() does not work (it works for other tables on the web page,
but not for the one I want).
I found a similar problem in the following post - that received a very detailed and informative reply from omegastripes:
Open webpage, select all, copy into sheet
-but when I run my code:
function test(){
var url = 'https://www.barchart.com/proxies/core-api/v1/historical/get?symbol=%24AVVN&fields=tradeTime.format(m%2Fd%2Fy)%2CopenPrice%2ChighPrice%2ClowPrice%2ClastPrice%2CpriceChange%2Cvolume%2CsymbolCode%2CsymbolType&startDate=2019-04-15&endDate=2019-07-15&type=eod&orderBy=tradeTime&orderDir=desc&limit=2000&meta=field.shortName%2Cfield.type%2Cfield.description&raw=1';
var options = {
"muteHttpExceptions": false
};
var response = UrlFetchApp.fetch(url, options);
Logger.log(response);
}
-then I get the following error:
Request failed for https://www.barchart.com/proxies/core-api/v1/historical/get?symbol=%24AVVN&fields=tradeTime.format(m%2Fd%2Fy)%2CopenPrice%2ChighPrice%2ClowPrice%2ClastPrice%2CpriceChange%2Cvolume%2CsymbolCode%2CsymbolType&startDate=2019-04-15&endDate=2019-07-15&type=eod&orderBy=tradeTime&orderDir=desc&limit=2000&meta=field.shortName%2Cfield.type%2Cfield.description&raw=1 returned code 500. Truncated server response: <!doctype html> <html itemscope itemtype="http://schema.org/WebPage" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="wi... (use muteHttpExceptions option to examine full response) (line 57, file "DS#1")
Basically an "Oops, something's wrong. Our apologies ... there seems to be a problem with this page." ... if you paste the address into your browser.
So my question is: How can data be scraped from this page or has Barchart now succesfully blocked this scraping option?
See Question&Answers more detail:
os 与恶龙缠斗过久,自身亦成为恶龙；凝视深渊过久,深渊将回以凝视…
