Mark Zuckerberg first experienced a groundbreaking virtual reality helmet Oculus Rift in January 2014. Standing in the rare shuttered Facebook office, a brick-like device hung on his face and was instantly taken to the ruins of a medieval castle surrounded by thick snowflakes. This dazzling virtual adventure immediately made him think that VR will one day become a major computing platform. Two months later, he acquired his belief in the acquisition of Oculus through a $2 billion investment.
However, at the time of Zuckerberg's demonstration, there was another technology that was rarely discussed in the industry: a younger, related technology called Augmented Reality (AR), which has the opportunity to surpass VR. There is the potential to superimpose background information or special effect digital overlays into the physical world only through smart phones. Although Facebook did not make sensational acquisitions, Zuckerberg ordered his engineers to begin building both AR and VR in the future. Given that the two media have a lot of the same underlying technology, from hardware components to complex computer vision software, it makes sense to work together.
In an interview with Forbes, Facebook Chief Technology Officer Mike · Mike Schroepfer pointed out that “Zuckerberg was the person who really pushed us to invest in AR. ” He stated that AR and VR are among Facebook's top three priority development technologies —— together with networking and artificial intelligence. Within Facebook, hundreds of engineers are working on basic technologies such as computer vision to enable mobile phones to track facial actions in real time, identify coffee cups, or recommend image effects based on context. Artificial intelligence is very important for enhancing AR, so Facebook employees often call their in-app camera an "AI camera". Skropov said that the entire project requires "huge capital expenditure."
(Photo note: Instagram's "Stories" feature on the facial filter)
Today, Facebook is engaged in a major battle with two other technology giants Apple and Google (also to some extent include "Read and Burn" communication application Snapchat developer Snap). The winner will become The preferred platform for AR developers to dominate. Although the technology itself is still in its infancy, it is rapidly gaining popularity. This confirms Zuckerberg’s recent instincts that AR is expected to quickly become popular among the public, while VR is still a relatively unwieldy technology. , It's largely appealing to heavy players. The key advantage of AR is that it does not rely on expensive and cumbersome equipment and does not isolate users from the world around them. It applies to smart phones, which are already owned by more than a quarter of the world's population.
"One big realization is that you can use mobile phones to experience AR. We have about 1.3 billion users using Facebook on their mobile phones." "The number of mobile phones is more than 100 times that of VR helmets," said Joaquin Candela, Facebook's head of applied machine learning, who is responsible for providing AI technical support for AR projects. This makes AR very interesting, making it an obvious area to focus on. ”
Obviously, early signs indicate that AR has the ability to attract consumers and sometimes even fundamentally change the way they interact with mobile phones. Look at how popular some groundbreaking AR apps are, such as puppy masks on Snapchat and virtual treasure hunt games like Pokémon Go. They illustrate that AR, not VR, is the next major stage in the future of our mixed reality. AR-based smartphone interactions ——using masks to transform into rock stars, or using mobile phones to capture sprites — are already social behaviors that are acceptable to people.
However, the tech giants are betting on AR because they apply far beyond social media, games, and bizarre special effects. The technology is expected to give birth to practical applications in various fields ranging from navigation to e-commerce, such as virtual fitting rooms that may promote the online purchase of apparel. The IKEA app has helped shoppers put furniture in their homes in a virtual format. Pharmaceutical companies are using AR to show real-time information about drugs. Hyundai uses AR applications to guide consumers to understand the features of some cars. AR is also expected to appear in customer support tutorials integrated into chatbots. Tom Meyer, manager of the Facebook camera team, said that "Smartphones can basically be a magic lens for you to watch the whole world." ”
Inside Facebook, engineers and executives acknowledge that the AR project is of major importance. If it fails to implement the AR correctly, the company will face the risk of falling usage of its various applications. Its competition with Snapchat for young users suggests that loyalty to social applications may be fleeting. Users will quickly turn to social applications with the most attractive tools and social applications that can transform communications (especially through images and video) into an engaging and evolving experience.
Facebook launched its AR special effects feature a few years later than Snapchat, and for several years in the Internet era can be said for a long time. However, Facebook finally completed the arduous task of overcoming the disadvantages of starting out, which was mainly due to the powerful artificial intelligence technology inside the social media giant (supporting large-scale provision of advanced special effects) and strong product design capabilities. Together, these assets allow Facebook to quickly match Snapchat's capabilities and curb the growth of its young rivals, thereby avoiding Facebook's mistakes with other tech giants such as Google Plus. As Google Plus went online too late, Google missed the opportunity to occupy a place in the social media field.
Success on the AR can bring great returns to Facebook. The company's advertising business was mainly driven by the activity of its applications and the length of users' use. In 2016, it brought in $26.9 billion in revenue. AR special effects are increasingly contributing to the user's usage time of the social network. It encourages users to send more messages and spend more time browsing friends' posts and posting. Facebook users spend an average of approximately 50 minutes a day on their main applications, Instagram, and Messenger——Facebook needs continuous introduction of new product features that attract eyeballs and stimulate image communication to keep users’ average hours at a high level. , against external competition, especially when companies seek to minimize phishing content (such as clicking on baits and pranks) in dynamic news.
(Photo: Zuckerberg demonstrates Oculus's virtual reality helmet at a conference in 2016)
Masks and filters are widely used in various Facebook applications, marking the company’s transformation from a “past-camera” for still images and photo albums to “Meyer's” camera of the future. A major first step has been taken ——“future camera” driven by AI to provide attractive results, it is attached to a network of friends and family. The mask function may seem trivial, but it makes sense for Facebook to have a good life. According to Chris Cox, Facebook’s chief product officer, the company’s focus on image-based features is no less important to the company’s future than the transition from desktop to mobile.
"Overall, if we want to play an important role in the next 10 or 20 years, then Facebook must do a great job of AR. "Skoropf said.
From "Past" & ldquo; Camera to "Future & rdquo; Camera
One morning in May of last year, Zuckerberg appeared on a social network with tattered glasses, and the mathematical equations rotated on top of him. Zuckerberg chose to use this bizarre digital accessory to publish the first "face filter" of Instagram Stories features (called "mask" on Snapchat). “This is my favorite one so far. Zuckerberg said with a smile and looked at the camera.
Those special effects, including weddings throwing confetti, bubble-filled underwater scenes, shaky Koala noses, and bunny ears, attracted 300 million people to experiment, from acting like actress Rey & Middot; Witherspoon. (Reese Witherspoon) and models such as Karlie Kloss, a celebrity, lulled into the home of teenagers who use Instagram's ARies-like Stories feature at home every single month. Everyone is involved. Driven by Instagram Stories (a series of photos and video clips that disappeared in 24 hours) and similar features on WhatsApp and Facebook’s main application, Facebook is now the world’s largest social AR ecosystem, at this time from Zuckerberg. The Oculus demonstration of Georgia was only four years old, and it took six years before the company began to select this core AI technology.
Hundreds of developers are using software "Ar Studio" opened last December to build applications on Facebook's AR Camera platform. Now, anyone with a Facebook account can create AR effects for the social network, including masks, animations, and 3D objects.
Although Facebook is conquering social AR, it is not the first company to popularize social AR. The title belongs to one of its main competitors, Snapchat. This popular app was originally seen by many as a porn messaging tool. Snapchat first launched the "Stories" feature (including various AR effects), which is 3 years earlier than Facebook, which had nearly the same features as it did in 2016. Until more than a year ago, Snapchat's AR functionality was far ahead of Facebook. However, Facebook has been focusing on preparing this core technology behind the scenes in recent years, thanks to Zuckerberg’s early warning that AR will one day become a mainstream communications tool.
Snapchat's first AR effects were full-screen overlays for photos and videos. It then launched "geofilters" location-based artistic effects, such as the famous Rainbow Mask, from 2014 to 2016. Lens ”, as well as a custom avatar named “Bitmojis”. The effects of Snapchat's AR have been a hit, helping the app get 150 million daily active users. Adolescents, perhaps the most valuable and elusive group, see it as an essential chat tool every day. Now, Facebook executives have avoided talking about this young competitor in an interview, but before, this social network has been worried about tracking the growth of Snapchat.
While Facebook was monitoring every move of its new competitors, it also learned a trend in its own applications. Posts on this social network are increasingly turning to photos, GIFs, and videos, with fewer text posts. What is even more surprising is the user's response to the live video broadcast feature in April 2016. Senior management, including Zuckerberg, was shocked by its rapid adoption. It clearly shows that people want to speak through images and animations on Facebook. They want to do it in real time.
Increasing investment in image-based sharing is unavoidable. When Facebook launched a live video feature and noticed the application of the Belarusian self-portrait mask called Masquerade (MSQRD), Facebook's product roadmap was a major turning point. Masquerade has already taken off in Eastern Europe and it has begun to become popular in the United States with a total of 16 million users. The mask tool is almost exactly the same as the Snapchat lens. It is based on the 3D graphics rendering technology developed by Eugene Zatepyakin, the co-founder of MSQRD and developed by Eugen Zatepyakin. Julie Zhou, vice president of Facebook Design, recalls that when the company tried the app at its headquarters in Menlo Park, Calif., the company’s executives quickly became fascinated by it.
MSQRD Filter preview
“People try to change their tools, there is some natural attraction —— you want to be someone else, want to dress up. Julie · Zhou said, "It makes people feel like becoming children again and having fun, but it will not always be a toy." ”
In March 2016, Facebook acquired the startup company, the amount was unknown, and began frantically making up for lost time and chasing Snapchat. Soon after, Zuckerberg published a video of his own Iron Man mask using MSQRD on the social network, and quietly launched an important internal project to quickly build a technical foundation to support more complex in-app cameras.
Since 2015, Facebook has been working on the development of artificial intelligence, such as the computational photography technology that later supported AR. However, until he called for the formation of a dedicated "camera department" in the summer of 2016, Zuckerberg launched the open AR project. The team started with only a handful of engineers and researchers, belonging to the Facebook Applied Machine Learning Department, and was located in Zuckerberg's glass-wall office adjacent to the "Building No. 20" office.
Over the next year and a half, the camera division expanded to more than 100 people. Its members include designers from Hollywood and game companies. They worked with camera product executives of independent Facebook applications such as Messenger and Instagram to help them publish functionality. And iterate quickly. The department has been busy with a variety of tasks, from advancing underlying visual identity technologies and deep learning technologies to organizing focus groups outside of Facebook to experiment with AR tools.
(Photo note: "Style Transfer ” Style Transfer Technology Preview" Facebook powered by Caffe2go
In one of the camera team's research studies, Dantley Davis, former head of product design for the machine learning team and head of Netflix's mobile design group, recalled the interviewers talking to them about what they care about. People's "Happy Birthday" experience. The team provided a series of tools for sending special effects and information to those involved in the study. Dantley pointed out that a man in their research team was relieved that he could wish his wife a happy birthday through an animation because it was easier and quicker to express himself or herself in words or words.
In communication with his wife, he created a very intimate and personalized experience based on the tools we provided. "Dantley said," AR tool allows him to express himself in a way that he thinks is very valuable. They allow people to convey their emotions in a more quirky manner. ”
Shortly after the camera team was established, the explosion of an application by game developer Niantic proved to a large extent that AR can attract mass consumers. Ninetic's game Pokemon Go players grew to 50 million in 19 days after being launched in July 2016. Players walk out of the house and use smartphones to grab AR Wizards. Zuckerberg himself is also a fan of the game. In Facebook's quarterly earnings conference call that month, he mentioned that he and other people "like to play." For Facebook executives, the game confirms that AR is not limited to just a few communication applications like Snapchat.
At the same time, Facebook's camera team engineers are immersed in the development of internal AR processing software, and they hope that the software will become the cornerstone of future AR effects. Competitors like Snapchat rely on external servers to provide AR functionality, so they are limited in speed and complexity. And Facebook is trying to invent a system that handles AI directly on smartphones, which was eventually named Caffe2Go. Facebook wants to fully build infrastructure to launch AR effects, because it will ensure that its AR effects can be extended seamlessly, with better rendering, face tracking, and speed than Snapchat.
While Caffe2Go was at the preparation stage, Facebook’s product team experimented with AR tools with the help of MSQRD. In August 2016, for example, before the opening of the Summer Olympic Games, Facebook first tested its flagship application in full-screen camera mode (the Snapchat style) in Brazil and Canada, and launched an Olympic-themed animation framework and avatar for personal profiles. Face painted mask. That month, Facebook launched its first version of "Stories" on Instagram (almost cloning Snapchat's iconic "Reading and Burning" feature) to lay the foundation for Instagram's AR effects.
In the fall of that year, Facebook completed the development of Caffe2Go and created the first system that can capture and analyze pixels in real time by directly processing AI on smartphones. After testing the technology in the fall of 2016 in the "style transfer" function (transforming a photo or video to an artist like Picasso or Van Gogh), Facebook is ready to use Caffe2Go to support AR functionality on its various applications. , first implemented on Messenger in December 2016. Shortly afterwards, through a "Stories" that named "My Day" for masks and filters for photos and videos, Messenger looks very similar to Snapchat. In March 2017, Facebook extended AR effects to its flagship app, allowing users to scroll to the screen to enter full-screen camera mode, with special effects buttons overlayed around masks and animations.
(Photo: ‘ Stories' preview on the Facebook main app)
In April 2017, at the Facebook F8 annual developer conference, Zuckerberg's major release was the release of the first "camera platform", which allows some developers to develop AR functionality on this social network. Zuckerberg first explained the future work from a philosophical point of view before releasing it. Zuckerberg believes that in the end, science and technology will release people's time, allow people to socialize more, do creative things, and make more works of art.
"In the future, more of us will contribute to culture and society in a way that is not measurable by traditional economics or GDP," Zuckerberg said before displaying the big screen of the company's 10-year roadmap. Many of us will be doing what we consider to be art today, which will be the foundation of many of our communities. ”
"This is why I feel so excited about Augmented Reality," Zuckerberg continued, "and this will allow us to create a variety of things that we could only realize in the digital world today." Will be able to interact with them and explore them together. ”
Zuckerberg marked Facebook’s in-app camera as the core of Facebook’s platform communications for the first time. Zuckerberg predicts that while AR glasses and contact lenses may become the first AR wearable devices in the future, people can now start enjoying AR on their smartphones.
Since Facebook began launching AR special effects at the end of 2016, Snapchat’s growth has been stagnant. With the rise of Facebook's most popular AR platform, Instagram Stories, Snapchat users continued to decline in growth. Today, only about 190 million people use Snapchat every day, while Instagram uses 500 million people every day.
broader AR competition
Although AR's largest consumer-level use is on social media, almost every tech giant is scrambling to introduce AR functionality into its products while creating an ecosystem for a developer community that is generally unfamiliar with AR development. Two months after Facebook launched its AR camera platform, Apple introduced its own developer toolkit, ARKit, for its mobile operating system, iOS 11, to allow developers and marketers to integrate AR into their existing applications. According to Forrester, a market research firm, Apple's ARKit-compatible devices are estimated at 400 million units. In September last year, Apple also launched "Animoji" for iMessage, using face recognition technology on iPhone X, which allows people to customize the emoticons with their facial expressions.
(Photo: Zuckerberg suggested last year that his company is building wearable AR devices.)
However, Google may be the longest-running tech company to study AR. It launched Google Glass as early as 2013. The product has become a major failure. Although it is popular among technology enthusiasts, it has not been popular among consumers because of privacy and social acceptance issues. Google subsequently launched the AR platform called Tango in 2016, which uses depth sensors to draw the interior space, but is only compatible with a few devices that have low penetration rates. In order to push Tango’s capabilities to more mobile phones without the need to add cameras and sensors specifically, Google launched its own ARKit version of ARCore last August. The developer kit aims to cover existing and future Android devices, including Samsung Galaxy 8.
A Google spokesman said that by the end of the winter, ARCore is expected to cover 100 million Android devices. Like Facebook, Google naturally is interested in using AR for search. In May of last year, Google announced its computer vision tool, Lens”, to let people organize albums in Google Photos and let Pixel and Pixel 2 users point their cameras at stores and other objects to get real-time information. Apple and Google not only benefit from having their own operating system (developers can develop a large number of AR applications on these operating systems), they also have a resume of successful hardware product development that Facebook has so far failed to match. Facebook is not on the level of device development with them, so it is difficult to implement hardware ambitions.
Forrester analyst Thomas & Middot; Thomas Husson pointed out that "Facebook's camera platform will allow developers and marketers to reach more audiences over time, but the limitation of Facebook is that it is not hardware." controller. To truly bring about an amazing augmented reality experience requires integration between software and hardware. ”
Facebook, Apple, and Google all focus on smart phones, and Microsoft has always focused on providing wearable devices for corporate customers. The company launched the AR helmet Hololens for $3,000 in 2016. Hololens runs on an operating system called Windows Mixed Reality that can be used by other VR and AR hardware manufacturers. However, AR equipment may take a few years to become cheap enough and win the favor of consumers.
Facebook may never have its own operating system or the next AR device (though it is trying), but it does have some unique and important assets: First, it covers a much wider audience than Snap, Kakao, Line, and WeChat etc. There are many social or communication applications. (Data about people's interests and network of relationships is helpful for personalizing AR effects). Second, Facebook has one of the largest computer vision teams in the world, which will help it develop better features than later. Even if Facebook has never been able to make its own AR hardware popular, it will still be one of the largest AR content ecosystems in the coming years.
The next trend of advertising marketing
In the last weekend of the final episode of the seventh quarter of last July’s drama “Game of Thrones,” fans around the world ran to social media as usual to share their perceptions of the post-study arrangement and the fate of the characters in the drama. But in addition to traditional status updates, a more compelling type of post has also spread virally: more than 1 million people have created on Facebook a nightmare that they have gradually turned into a terrifying, blue-eyed night, accompanied by a sound Converter and video clips on snow background. The face-tracking mask adjusts in real time on their faces as people roar or sing loudly, and those videos are widely circulated among dynamic messages and messages between friends.
The mask's ability to transform is not the only bright spot. It was made by HBO, not Facebook (the "GOT" logo in the upper left corner). It quickly became one of the most successful AR advertising campaigns on Facebook and demonstrated that high-quality AR effects can become ads that prompt people to choose to send to their friends. As the Night King mask shows, AR is an attractive ad format for many marketers because it brings with it new ways to maintain the attention of smartphone users for longer. Facebook is positioning itself as the default platform for AR marketing campaigns designed to encourage users to play games with friends or publish self-timer animations.
(Photo: HBO's "Game of Thrones" Night King AR Mask was used more than 1 million times on Facebook)
Ebily Giannusa, director of digital media and marketing at HBO, told Forbes that "The game of power" fans would not be happy to have new interactions every day for 165 days a year. The Facebook Camera Platform is very attractive to us because it is a simple and advanced tool that allows fans to bring themselves directly to the world of "Game of Thrones." Fans have a smart phone on the line. ”
It is estimated that only about 5% of marketers are using AR technology. According to a recent Forrester study, 17% of marketers plan to use the technology this year. For marketers, AR will be more important than VR for at least the next three years. Forrester's Husson pointed out that while the VR experience makes more sense for ambitious brands that are intended to attract customers for a long time, there are currently many more marketers who benefit from experimenting with AR.
As AR extends from smart phones to wearable devices, the technology may become an almost always online tool to enhance people's sensory experience. It may also replace the traditional search bar and present us based on our location, interests, and social networks. Need real-time information. However, consumer-grade use of AR is still in its infancy. In the next few years, the tech giants will be busy improving core technologies and creating ARs for smart phones. Facebook will focus on its areas of expertise — communication and personal expression. Over time, AR may make Facebook and its dynamic news completely different ——more immersive, with more video and interactivity, although the details are still vague.
Facebook will need to continuously improve its AI to increase its speed and accuracy, such as identifying objects in video, understanding how the scene is pixelated when viewed from different angles, and how to draw relationships between objects in the scene when. In the long term, to reach a wider range of users, including in developing countries, Facebook will need to make its AR functionality more compatible with relatively weak cellular networks and older handsets.
"We are still in the basic exploration phase. We are still building the basic technology," Facebook's Candela pointed out, "So I am now focusing on implementation." At the same time, we will also have some crazy exploratory projects. ”
At the same time, according to Skopf, Facebook is “investing heavily in hardware” to support AR and build more social tools. In addition to defending Facebook’s industry position, there are other benefits to improving the underlying AR, such as helping to combat spammers and problematic content through better visual recognition and language understanding tools, and promoting game and robotics development.
“Thought that ‘this is exactly creating a mask for a space cat, is wrong. "Candela said," You use our social infrastructure and the technologies we promote, and what applications you develop are unimaginable. But I'm convinced that this will open things that we haven’t thought of yet.——Your meaningful AR experience will be social, and in the experience you are yourself. ”(Lebang)