The “Big Three” real estate traffic leaders are Zillow, Trulia, and Realtor.com. But since Zillow and Trulia have joint forces, we can consider them as one entity and pay some attention to another big fish in the US, a residential real estate company Redfin.
- Zillow was created in 2006 by Rich Barton and Lloyd Frink, former Microsoft executives and founders of the leading travel website Expedia.
- Redfin was founded in 2004 by David Eraker, Michael Dougherty, and David Selinger. The company was a pioneer of map based real estate search even before Google Maps appeared.
- Founded back in 1995, Realtor.com is a grandfather of the whole online real estate industry. It’s the official site of the National Association of REALTORS® and operated by Move, Inc., a subsidiary of News Corp, which belongs to Rupert Murdoch.
- Unlike Zillow and Realtor.com, Redfin is an actual brokerage who pays its agents a base salary, and makes money when users buy or sell homes with its real estate agents.
- Realtor.com and Zillow (including Trulia) are real estate aggregators whose customer buys ads on their sites.
Where do Zillow, Realtor.com and Redfin get their data from?
- The real estate data in the US comes from the individual real estate agents and real estate brokers who are members of local associations of Realtors. This data can be accessed through the “Multiple Listing Service” (MLS) which Realtors associations publish for their local communities.
- The good thing is MLSs give permissions to real estate portals like Zillow, Trulia, Realtor.com and others to republish the MLS listing feeds on their media channels.
- The bad thing is MLSs are quite hard to access. There is no public API you can use to just pull the data from there, and they restrict membership and access to real estate brokers and their agents, which means that a person selling his/her own property (For Sale By Owner) cannot put a listing for the home directly into an MLS.
Is MLS the only source of real estate data?
There are lots of sources that real estate aggregator companies can use to get the listings of properties. You would only need to do three things to get access to those sources:
- get licensed in every state,
- join MLS,
- integrate the data which doesn’t follow a consistent standard across the 900+ MLSs.
Anyway, here is where Zillow goes to make a living:
- Zillow is doing fine attracting real estate agents and brokers to their website. After they welcomed Trulia in their online empire, rumor has it, they control more than 70% of online real estate searches.
- Trulia and Zillow get access to local MLS by negotiating data sharing/syndication agreements.
- Redfin, on the other hand, doesn’t have any of that headache. As a brokerage, it’s free to use data from its membership in the Realtors MLS.
Listings syndication platforms
Because there are hundreds of 900+ MLSs all over the US, it’s extremely burdensome to negotiate an agreement with everybody, especially once you operate nationwide. That’s why the largest aggregator websites enter data sharing agreements with big national real estate companies to get their data directly.
But the largest aggregation of real estate listing data (for sale listings) in the US is called ListHub. The company belongs to Move Inc. (also owns Realtor.com). Last year Move acquired Point2, another large listing syndication platform.
How can I access real estate databases?
- The biggest problem with MLS listings isn’t even the license. It’s a different implementation of listings databases across MLSs. In order to enable data exchange between an MLS database and your website you need to implement IDX (Internet Data Exchange). There are a couple of options to do it.
- One is by using an old FTP server, which will force you to download the entire database of listings in bulk each time you want to update your records.
- The other option is to use RETS (Real Estate Transaction Standard), a framework used to give brokers, agents and third parties access to listing and transaction data. RETS is XML based and there’s a wide variety of open source tools and libraries for working with RETS data.
- There are third party services that provide their APIs that normalize data flows from MLSs into a standard model. You can check out Spark, SimplyRets, Optima Express IDX Plugin for WordPress sites, and iHomeFinder.
You will also need to build a robust backend to be able to properly manage the data. Redfin’s main development languages are Java and Python and they use SQL Postgres database on the backend. They also use Eclipse integrated development environment, Git version control system, Bamboo for continuous integration and build delivery, and JIRA for project management.
Judging from the job offers on Zillow’s website, the company uses LAMP stack, as well as Node.js, Python, Java, and C++.
Property listings aren’t the only data that a real estate app needs
Other than properties, a real estate app or a website generally has the data about neighborhoods, schools, points of interest, demographics, past sales history and other info which improves home buying and selling experience.
Zillow provides APIs for neighborhood information for their listings and a dataset that contains neighborhood boundaries. Trulia also provides API derived from the aggregated data on real estate listings, trends and search activity. Here are other important APIs with geo and neighborhood information:
- GeoNames has a very comprehensive dataset of over 10 million geographical names.
- MapBox allows to build custom maps for real estate listings. Here you can read about why StreetEasy, one of Zillow’s companies, switched from Google and went with MapBox.
- Data.gov site in the US provides an entire catalogue with over 8.000 datasets with anything from crimes to demographics to high school graduation rates.