On October 15, 2019, Jeff Barr, the chief evangelist of AWS, posted the title "Migration complete" on the official blog.
During 17 years of working at Amazon, I found that colleagues in the engineering team were never satisfied with the status quo. They regularly evaluate each internal system to improve the scalability, efficiency, performance and security of the system as much as possible. Once they find the way to improve, they will try their best to learn all their lives, thoroughly innovate the existing architecture and implementation methods, sometimes even break up the existing system; if necessary, start again.
The internal database migration work that I'm introducing to you at this moment is an example of the above point of view, which has just ended after several years of efforts. Over the years, we've realized that we've spent too much time managing and scaling thousands of old Oracle databases. The company's database administrators (DBAs) did not focus on high-value differentiated work, but were tired of maintaining the smooth operation of the system under the situation of the increasing transaction rate and the increasing total amount of stored data. They spend a lot of time dealing with complex and inefficient hardware configurations, license management, and many other issues that could have been handled by modern managed database services.
More than 100 teams from Amazon's consumer business unit are involved in the migration, including Alexa, Amazon Prime, Amazon Prime video, Amazon fresh, Kindle, Amazon music, audible, shoptop, twitch and Zappos, as well as Adtech, Amazon fulfillment technology, consumer payment, customer return, catalog systems, delivery experience, number Word devices, external payment, finance, infosec, marketing, ordering and retail systems.
I am pleased to announce that database migration is now complete. Amazon's consumer business has just shut down some of the last Oracle databases (some third-party applications are tightly bound to Oracle and have not yet been migrated).
We migrate 75 Pb of internal data stored in nearly 7500 Oracle databases to a number of AWS database services, including Amazon dynamodb, Amazon Aurora, Amazon relational database service (RDS) and Amazon redshift. The migration process requires no downtime and covers a wide range of our proprietary systems, such as complex procurement, catalog management, order execution, accounting and video streaming workloads. We carefully analyzed the cost and performance and came to the following conclusions:
Data migration gives internal teams the freedom to choose specialized AWS database services that best fit their needs, giving them better control over budget and cost models. Low latency services have been migrated to dynamodb and other highly scalable non relational databases, such as Amazon elasticache. Transactional relationship workload with high data consistency requirements has been moved to aurora and RDS; analytical workload has been moved to our cloud data warehouse redshift.
We witnessed the final shutdown of the Oracle database and had a brief celebration:
DBA career path as I mentioned before, DBAs have spent a lot of time managing and expanding old Oracle databases. After the migration, DBAs freed their hands and focused more on performance monitoring and query optimization, all in order to provide a better customer experience.
As a task of migration work, we strive to create a new career path for Oracle's DBAs, train them to become database migration experts and consultants, including AWS database technology, Cloud Architecture, cloud security, OPEX cost management. They now work as consultants with internal and external customers, giving them the opportunity to share first-hand experience in projects that migrate mission critical databases on a large scale. The migration cases are as follows: