resources
Collecting Stock Market Data from Public Financial Websites
09 Nov 2026
Say you want to test a simple trading idea, so you go looking for five years of daily closing prices for a few hundred companies. All of that information is out there and free to look at, but it almost never comes as one clean file you can just download.
Analysts, small fintech teams and students all run into this. The numbers are spread over lots of public pages, and each one has its own layout, update schedule and patience for automated visits.
So finding the data is usually the easy bit, and collecting it properly without upsetting anyone takes a bit more thought.
Where you can find public market data
Most projects end up pulling from a few kinds of sites. Quote pages on Yahoo Finance, Google Finance or MarketWatch give you prices, volume and some basic ratios, and exchange websites like Nasdaq, NYSE or the London Stock Exchange post listings, corporate actions and daily summaries.
Regulatory filings are probably the most useful source and people tend to forget about them. A company's 10-K will honestly tell you a lot more than any quote widget ever could.
Quote portals often load their numbers with JavaScript (so a plain request just gets back an empty page). Some regional exchange sites also start slowing things down after a few hundred requests from the same address, which is why a lot of teams spread the load over IPRoyal's reliable dc proxies rather than sending everything from one office connection.
Datacenter IPs are a good fit here because the work is high volume and pretty low risk. You don't need to look like someone browsing from home to read a public closing price, so fast and stable connections are usually all you need.
Check for an official API before you scrape
Before you write any parser, check whether the source already gives you structured access. The U.S. Securities and Exchange Commission, for example, offers free EDGAR application programming interfaces that send back company filings and XBRL financial data as JSON, and you don't even need an API key.
That saves a lot of work, because revenue, net income and share counts for thousands of companies come back already labelled, so you don't have to scrape HTML tables.
The SEC does ask automated clients to include a User-Agent with contact details and to stay under 10 requests per second, which seems fair enough.
Building a collector that keeps running
For pages that don't have an API, most people use Python with requests or httpx for simple pages and Playwright for the ones that need JavaScript, and then pandas to clean everything up. Scrapy is handy once you have more sources.
But the harder part is making the collector behave politely. If a server sends back a 429 Too Many Requests status, it's asking you to slow down, and when it includes a Retry-After header you should basically treat that number as the wait time you have to respect.
It helps to cache every response so a parsing bug doesn't mean crawling everything again, and to run end of day jobs after the 4 PM Eastern close. Save the source URL and a timestamp with every value too, and compare prices against a second source.
One price saved without its split adjustment can mess up a whole backtest, and usually nobody notices until the returns look a bit too good.
Staying on the right side of the rules
Data being public doesn't mean you can do whatever you like with it. Many financial sites limit automated access or commercial reuse in their terms, and real time quotes are often licensed from exchanges.
The legal side of web scraping has changed quite a bit over the last decade. Cases like hiQ Labs v. LinkedIn suggested that reading public pages in the U.S. isn't automatically unauthorized access, although contract claims and copyright on compiled data can still come up.
In practice you should read the robots.txt file and the terms, and stick to factual data like prices and filings. Don't log into accounts or work around access controls, and if a site clearly doesn't want bots, there's usually another source.
Where this is heading
More regulators are moving filings into machine readable formats, and the EU and UK are heading the same way as EDGAR with XBRL, which should slowly cut down on fragile HTML parsing.
The people who get the most out of this probably won't be the ones with the most aggressive crawlers. It will more likely be the ones with tidy pipelines who check their numbers twice, which sounds a bit dull until one bad data point costs real money.
Share
Ricardo Tomaz
Ricardo Tomaz is VP of Finance & Operations at Ztudium, the 4IR technology company behind Businessabc, Citiesabc, and IntelligentHQ. He owns the group's finance and operations function, spanning financial strategy, budgeting, reporting, and multi-entity operations, and works directly with leadership to align commercial goals with data-driven insight.
Working at the core of a technology group operating across AI and blockchain, Ricardo writes on the practical intersection of finance and emerging technology: how automation and digital transformation are reshaping business operations, what data-driven decision-making looks like in practice, and the questions of governance and ethics that come with AI-led systems.
