As web scraping becomes more and more prominent in business online, we’re beginning to see it being utilised by banks and investors to make big market calls before anyone else. Whilst it should be no real surprise that embracing the ability to parse through large amounts of data results in a better understanding of how consumers are moving, understanding how hedge funds are using web scraping in their decision making can offer some useful insight into how a successful business might need to think to stay ahead of the pack. Let’s dig into some examples of what kind of data hedge funds are looking at when they need to make big market calls.
Possibly the most simple method a hedge fund could implement is simply looking at Amazon data to determine the kinds of products that are being bought, and monitoring for any sharp changes in activity. For instance, just by scraping retail data a hedge fund would be able to tell a sharp increase in demand for a product. Perhaps more importantly they might notice that a product just isn’t selling very well, despite the company offering it assuring their investors they will meet ambitious targets. In that scenario it would be easy to make a prediction call, sell while the stock is still high and avoid a large loss. That’s the kind of call that would not only beat competitors that don’t use web scrapers to stay informed, but would beat the rest of the market reassured by a company full of hot air.
Hedge funds have also seen success in the past by simply examining the page visit numbers of large retail sites and mapping those trends. By doing so they are able to tell if specific online retailers are picking up unusually large amounts of visits and put some money behind their ongoing success. Making a quick call like this is simple if you have access to the right data sets – which hedge funds have the resources to develop themselves – and could potentially result in huge gains. Of course, it’s not just as simple as online retail scraping – if it was, hedge funds would still be relatively reactive even if it’s to data that only they can see. To be truly ahead of the market hedge funds have always tried to be predictive.
Social media is probably the thing that unites us all in terms of online activity. From children to grandparents, it seems that almost everyone is a regular user of some kind of social media platform. Every day we post thoughts, opinions and endorsements – we talk to family, moan about work or show off things we’ve bought or done. In terms of consumer opinion, this is a gold mind for hedge funds. Because they’re in the business of predicting the market, being able to collect huge amounts of opinion data and parse it (in the kind of sophisticated ways that millions of dollars can buy) is an unparallelled tool for getting a proper understanding of what people like and why they like it. From there it’s a relatively simple guess as to where they are going to move next. According to some, looking at large scale Twitter data is an incredibly precise measure of how the market is going to move. For a hedge fund this is, of course, invaluable.
And it’s not just a horde of consumer information that makes scraping social media so important, because it ignores one key aspect of platforms – influencers. Whatever you might think of individual influencers on sites like Twitter, one thing is abundantly clear: they do exert influence over other users actions. This is oftentimes a double edged sword, but as any savvy investor will tell you, often times a negative can be as lucrative as a positive. You need only look to our earlier example to understand how a hedge fund might profit off of a market downtick. If a fund manager had a simple scraper running on the social media presence of the big influencers, they might be alerted to any endorsements or public criticisms of traded companies. Then, they can act on the information immediately for a quick profit. In some instances single tweets have resulted in a stock shifting by a few percentage points, resulting in potentially huge amounts of profit or loss.
This is all just the obvious though, and as the technology becomes more and more common, we are going to start to see some really innovative applications. Scraping job listings might give an insight as to who is being hired and why and offer a window into future projects by large companies. Looking back we can see hiring data for large tech projects that, if predicted, would have made savvy investors enormous amounts of money. Looking to the future with that mindset may yield positive returns.
Ultimately the bottom line is scraping data is big business, and it’s only going to get bigger as hedge funds begin to establish it as an industry standard tool. As time goes by more and more hedge funds are realising that web scraping is not only hugely important, but it’s probably a mandatory practice at this point if they want to keep pace with their more savvy competition. Having access to the online activity of consumers is an absolutely indispensable tool for someone whose job it is to predict where those consumers are going to move their money. As these funds implement web scraping as a core part of their strategy we’re going to start seeing some really smart uses of scraped data, and that’s where businesses should take note right now – think creatively about where and how you can carve out an advantage in your field with big data. You might just stay ahead of the curve.
Hopefully that’s given you some inspiration for alternative data sources. If you have a web scraping job you’d like to talk to us about please contact us and we’ll get back to you within 24 hours. Happy scraping!
