Fixing YahooFinance API On Error 401

Since 1st May 2023, YahooFinance API stops giving quotes on US stocks. Apparently, there’re lots of discussion on this topic, where the previous working API is now giving code 401 on authentication error. The root cause is that Yahoo has started forcing cookies in requests and no longer supports v6 URL. This entry summarizes discussion all over the places to arrive at a working fix.

1. Fix YahooFinance API ("yahoofinance.quotes.query1v7.QuotesRequest")

   1.1 Cookie: get cookie from system property, and set URLConnection with it so that the request to Yahoo carries this cookie.

   1.2 Crumb: set crumb to URL link.

    public static final String YAHOOFINANCE_K_CRUMB	= "yahoofinance.crumb";
    public static final String YAHOOFINANCE_K_COOKIE= "yahoofinance.cookie";
    /**
     * Sends the request to Yahoo Finance and parses the result
     *
     * @return List of parsed objects resulting from the Yahoo Finance request
     * @throws IOException when there's a connection problem or the request is incorrect
     */
    public List getResult() throws IOException {
        List result = new ArrayList();

        Map params = new LinkedHashMap();
        params.put("symbols", this.symbols);

//        String url = YahooFinance.QUOTES_QUERY1V7_BASE_URL + "?" + Utils.getURLParameters(params);
        //https://github.com/sstrickx/yahoofinance-api/issues/206
        String url = "https://query2.finance.yahoo.com/v7/finance/quote"+ "?" + Utils.getURLParameters(params);
        if (!CrumbManager.getCrumb().isEmpty()) {
            url = url + "&crumb=" + CrumbManager.getCrumb();
        }
        // Get JSON from Yahoo
        log.info("Sending request: " + url);

        URL request = new URL(url);
//        RedirectableRequest redirectableRequest = new RedirectableRequest(request, 5);
//        redirectableRequest.setConnectTimeout(YahooFinance.CONNECTION_TIMEOUT);
//        redirectableRequest.setReadTimeout(YahooFinance.CONNECTION_TIMEOUT);
//        URLConnection connection = redirectableRequest.openConnection();
        //set cookie for the connection, which should be defined in system properties by how.
        HttpURLConnection connection = (HttpURLConnection)request.openConnection();
        connection.setConnectTimeout(YahooFinance.CONNECTION_TIMEOUT);
        connection.setReadTimeout(YahooFinance.CONNECTION_TIMEOUT);
        if(System.getProperties().containsKey(YAHOOFINANCE_K_COOKIE)) {
        	String cookie = System.getProperty(YAHOOFINANCE_K_COOKIE);
        	connection.setRequestProperty("Cookie", cookie);
        }else {
        	throw new RuntimeException("Cookie must be defined but not found: " + YAHOOFINANCE_K_COOKIE);
        }
        connection.setRequestMethod("GET");
        
        InputStreamReader is = new InputStreamReader(connection.getInputStream());
        JsonNode node = objectMapper.readTree(is);
        if(node.has("quoteResponse") && node.get("quoteResponse").has("result")) {
            node = node.get("quoteResponse").get("result");
            for(int i = 0; i < node.size(); i++) {
                result.add(this.parseJson(node.get(i)));
            }
        } else {
            throw new IOException("Invalid response");
        }

        return result;
    }

2. Package and configure maven

   2.1 Building: download the source codes for v3.17.0 from github and modify the code for step 1; build it with local machine to get 3.17.0.FIX.jar.

   2.2 Amend maven on my application

    2.2.1 The dependency on v3.17.0 should be removed.
    
    2.2.2 add local repository
   

<repositories>
    <repository>
        <id>local-repo</id>
        <url>file:///D:/java/repos-ws/filestorage/lib</url>
    </repository>
</repositories>

    2.2.3 add local dependency

<dependency>
    <groupId>com.yahoofinance-api</groupId>
    <artifactId>YahooFinanceAPI</artifactId>
    <version>3.17.0.FIX</version>
</dependency>		

3. YahooFinanceWrapper

Build a singleton, so that there’ll be only one instance of it, so that we can initialize cookie and crumb globally. Note that both cookie and crumb should be set to system properties before any Yahoo request is invoked.

public class YahooApiWrapper {
	private Logger logger = LoggerFactory.getLogger(this.getClass());
	private static YahooApiWrapper instance = null;
	public static final String YAHOOFINANCE_COOKIE 	= "https://fc.yahoo.com";
	public static final String YAHOOFINANCE_CRUMB	= "https://query2.finance.yahoo.com/v1/test/getcrumb";
	public static final String YAHOOFINANCE_K_CRUMB	= "yahoofinance.crumb";
	public static final String YAHOOFINANCE_K_COOKIE= "yahoofinance.cookie";

	private YahooApiWrapper() {
		String cookie = getCookie();
		String crumb = getCrumb(cookie);
		System.setProperty(YAHOOFINANCE_K_CRUMB, crumb);
		System.setProperty(YAHOOFINANCE_K_COOKIE, cookie);
	}

	/**
	 * Lazy initialization with double lock check.
	 * 
	 * @return
	 */
	public static YahooApiWrapper getInstance() {
		if (instance == null) {
			synchronized (YahooApiWrapper.class) {
				if (instance == null) {
					instance = new YahooApiWrapper();
				}
			}
		}
		return instance;
	}

	/**
	 * Wrappers YahooFinance.get(symbol) so as to fix 401 issue.
	 * 
	 * @param symbol
	 * @return
	 */
	public Optional get(String symbol) {
		try {
			return Optional.of(YahooFinance.get(symbol));
		} catch (IOException e) {
			e.printStackTrace();
			logger.error(e.getMessage());
		}
		return Optional.absent();
	}
	
	public Map get(String[] symbols) throws IOException{
		return YahooFinance.get(symbols);
	}
	
	/**
	 * Build cookie for YahooFinace api, referencing:
	 * https://github.com/sstrickx/yahoofinance-api/issues/206
	 * 
	 * @return
	 */
	private String getCookie() {
		String cookie = null;
		try {
			URL url = new URL(YAHOOFINANCE_COOKIE);
			HttpURLConnection connection = (HttpURLConnection) url.openConnection();

			// Get the cookie from the response headers
			cookie = connection.getHeaderField("Set-Cookie");

			if (cookie != null) {
				// Print the cookie
				System.out.println("Cookie: " + cookie);
			}

			// get crumb
			System.out.println("Crumb: " + getCrumb(cookie));

			// Disconnect the connection
			connection.disconnect();
		} catch (Exception e) {
			e.printStackTrace();
		}
		return cookie;
	}

	/**
	 * Build crumb to fix "401" issue, referencing:
	 * https://github.com/sstrickx/yahoofinance-api/issues/206
	 * @param cookie
	 * @return
	 */
	private static String getCrumb(String cookie) {
		StringBuilder response = new StringBuilder();
		try {
			URL url = new URL(YAHOOFINANCE_CRUMB);
			HttpURLConnection connection = (HttpURLConnection) url.openConnection();

			// Set the cookie
			connection.setRequestProperty("Cookie", cookie);

			// Make the HTTP request
			connection.setRequestMethod("GET");

			// Read the response content
			BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
			String line;
			while ((line = reader.readLine()) != null) {
				response.append(line);
			}
			reader.close();

			// Process the response content as needed
			System.out.println("Response: " + response.toString());

			// Disconnect the connection
			connection.disconnect();
		} catch (Exception e) {
			e.printStackTrace();
		}
		return response.toString();
	}
}

4. Conclusion

In future when YahooFinance API has fixed this issue, I will reference to their maven again. My "YahooApiWrapper" can remain, so that my code has minimum changes w.r.t dependencies on YahooFinance API.

yahoofinance.Stock stock = YahooApiWrapper.getInstance().get(symbol).orNull();

Reference:

1. discussion on github for 401 issue.
2. configure local maven dependency.