Professors Karim R. Lakhani and David A. Garvin and Research Associate Eric Lonstein prepared this case. HBS cases are developed solely asthe basis for class discussion. Cases are not intended to serve as endorsements, sources of primary data, or illustrations of effective or ineffectivemanagement.Copyright © 2010, 2011, 2012 President and Fellows of Harvard College. To order copies or request permission to reproduce materials, call 1-­800-­545-­7685, write Harvard Business School Publishing, Boston, MA 02163, or go to www.hbsp.harvard.edu/educators. This publication may not bedigitized, photocopied, or otherwise reproduced, posted, or transmitted, without the permission of Harvard Business School.KARIM R. LAKHANIDAVID A. GARVINERIC LONSTE INTopCoder (A): Developing Software throughCrowdsourcingIn December 2009, Jack Hughes, CEO and founder of TopCoder Inc., entered his ????????????????????headquarters in Glastonbury, Connecticut, eager to review a particularly complex softwaredevelopment project for an ????????????????????????????dynamic power pricing system. Eight years after foundingTopCoder, Hughes still enjoyed detailed project reviews. He was particularly proud that hiscompany could produce high-­quality software solutions for which his own employees did not haveto write a single line of code. Instead, the firm nurtured a global community of more than 225,000programmers who competed to design and create software modules for TopCoder clients, a processthat the popular press called crowdsourcing.1??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????-­free and operationalon its first day, a rarity in the software industry. Especially impressive to Hughes was that in fourmonths, 65 participants from 11 countries on six continents had competed in 57 contests to create thiscritical pricing system for the client (see Exhibit 1). As of 2009, TopCoder routinely producedsoftware solutions for over 45 clients, including AOL, Best Buy, Eli Lilly, ESPN, GEICO, and theRoyal Bank of Scotland.??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????changes in the software industry, while also pursuing its unique competition-­based softwaredevelopment approach. He had transitioned his business from a model that helped other software??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????traditional IT consulting services and competitions, mobilizing developers world-­wide to solveclients?? problems.The shift to a greater emphasis on competitions, encompassing all aspects of softwaredevelopment, however, meant that project volume was a growing issue for TopCoder. Hughes had tothink through how a competition-­based business model, which increasingly stressed contests as anorganizing as well as money-­making approach, could handle increases in numbers of competitions,clients, and participants. Hughes considered his own goal: attaining $200 million in revenue from ahigh of just over $18 million in 2008. He fundamentally believed that contest demand would spur thesupply of TopCoder participants, who would in turn create high-­quality software solutions. But, was1 ????????????????????????????????????????????????????????????????????????????????Wired Magazine 14.06, June 2006.For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .610-­032 TopCoder (A): Developing Software through Crowdsourcing2$200 million in revenue a reasonable goal? Did his assumptions make sense? If so, what would it taketo increase revenues by over an order of magnitude?Background and Current OperationsBefore he founded TopCoder in 2001, Hughes had built a custom software development2company, Business Data Services, in 1985;; the company name changed to Tallan in 1991. Tallanemployed some 600 people before being sold to CMGI in 2000.3 As he was completing thetransaction, Hughes reflected on what he had learned from his experiences at Tallan??the experiencesthat would inspire the core tenets of the TopCoder business model. Although Hughes enjoyed histime at Tallan, the company struggled in some areas. For example, recruitment was an expensive andfrustrating process because finding qualified programmers was time-­consuming and talent wasdifficult to assess???? ???????? ?????? ?????????????????????? ?????????????????? ???????????????????????????? ?????????????????????????? ???????????? ?????????? ???????????? ??????????????obsolete after only a few years of productive service, leading to high levels of employee turnover.?????????????????????????? ???????????????? ?????????????????? ?????????? ?????? ?????????????????????? ?????????????????? ?????????????? ?????????????? ?????????????????? ???????????? ?????????????????????????????????????? ?????? ?????????? ???????????????? ?????????? ???????? ???????????? ???????? ???????? ?????????????????? ???????????????? ?????????????????? ???????????????????? ????????????components instead of building each application from scratch.Drawing upon these and other insights, Hughes set about creating a new kind of organization thatwould build a ??community?? of programmers to help address the issues he had identified. Theseprogrammers would compete??as well as affiliate??by building and using components that hadalready been tested and found workable. The idea of reusing software components for new projectswould become the core of the solutions the new company, called TopCoder, provided. Hughes?????????????????????? ???????? ???????????????? ?????? ???? ??????o-­???????????? ???????????????????? ???????? ?????????????????? ???????????????????????? One side of theplatform would be clients, firms that needed software developed, who would work with his staff tospecify programming challenges. The other side would be community members who would competein contests to create solutions to the challenges for money and skill ratings. TopCoder would be in themiddle as the platform host, designing and enforcing the rules of engagement between clients and???????? ???????????????????? ?????????????????? ?????????? ?????????????????? ?????????????????????? ?????????? ????????????????d that the company needed toexcel at five core tasks: breaking down large client software projects into components, taking in andprocessing client project specifications, determining appropriate contest prizes, having a consistentand unbiased way of selecting contest winners, and fixing bugs at the back end of development.Setting out to amass an initial collection of highly skilled programmers, from 2001 to 2003TopCoder asked established software development companies to sponsor world-­wide web-­basedp????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????competition platform and provided the company with access to talented programmers from aroundthe world. In return, the sponsors, including Sun Microsystems and Google, used the contests to??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????vice president of finance, explained thatduring the sponsorship phase, TopCoder offered unusually large prizes??as much as $5,000 to$10,000 per match for tournament winners??to attract competitors and expand the community. Inaddition, every contestant that participated received an objective numerical rating for their2 Custom software development by specialist firms in the global IT consulting and services sector (for example, Accenture andIBM) was an over $500 billion segment in 2008. (Source: ???????????????? ?????? ?????????????????????? ???? ???????????? ???????????????????? ?????????????????? ???????????????????? DataMonitor, March 2009.)3 ???????????? ?????????????????? ???????????? ?????????????????? ?????????????? ???????? ??????????M???????????????? InternetNews.com, February 14, 2000, http://www.internetnews.com/ec-­news/article.php/303771/CMGI-­Acquires-­Tallan-­for-­920-­Million.htm.For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .TopCoder (A): Developing Software through Crowdsourcing 610-­0323performance against the global talent pool, providing a clear signal to TopCoder and others about thetalent in the community.By the end of 2004, the TopCoder community was 50,000 members strong. In its early efforts touse the community to generate revenue, TopCoder acted as a placement firm, matching top-­ratedcommunity members with firms seeking new talent. Hughes, however, was ????????????????????????????????????????????????????????the idea of TopCoder becoming a placement firm. That was not my end ????????????????????????????????????????????????In 2005, TopCoder began to use its community to develop software components and applications.Hughes first tested this model by having highly rated community members compete to redesign and???????????????? ???????? ?????????????? ???????? ?????????????????? ???????????????????? ???????? ???????????????????? ?????????? ???????? ?????????????? ???????????????? ???????? ?????????? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????itive proof thatcomplex software systems could be built through competitions.Initially, TopCoder adopted a model to create solutions for clients by contracting with communitymembers, running competitions, and providing consulting services. The company broke down thesoftware development process into seven distinct but interrelated tasks: 1) conceptualization, 2)specification, 3) architecture, 4) component production, 5) application assembly, 6) certification, and7) deployment. Most revenue came from consulting services: TopCoder billed clients for the time the???????????????????? ?????????????????? ?????????????????? ???????????? ???????????????????????????????? ???????? ?????????????????????? ?????????????? ???????????????????? ???????????????? ??????component design and development competitions, assembling components, and delivering finishedsolutions.Shortly after TopCoder started developing software for clients, the company identified reusablecomponents from the software it was creating and collected the components in a catalog. These????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????ue proposition to its clients. Manyof the custom applications could be produced by combining existing catalog components with newcomponents built through competition. TopCoder had also received eight U.S. patents for variousaspects of running online programming contests in a distributed community setting and had otherpatents pending domestically and internationally.?????????????????????? ?????????????? ?????????????????????? ???????????? ???????? ?????? ???????????? ???????????????????? ?????? ???????????????????? ?????????????????? ?????????????? ??????????????????????????????????????????????????????????????????????????????????????-­??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????2007 and 2008, TopCoder produced nearly $20 million in revenue, but platform manager costsremained high (see Exhibit 2 for information on revenue and platform manager costs). Attempting toalleviate costs, in 2007 TopCoder introduced competition tracks for component architecture andassembly. With these new competition tracks in place, the work traditionally done by platformmanagers would now be done by the community. In 2008, the company also added competitions insoftware development tasks, such as conceptualization and specification, as well as deployment andbug fixing.By early 2009, TopCoder had moved increasingly away from the hybrid consulting model. It nowfocused on completing all tasks in software development through competitions. Instead of paying fortime and materials for TopCoder platform managers, clients paid a monthly platform fee based onthe complexity of their software requirements and the estimated number of competitions they wouldrun through the TopCoder platform each month. The platform fee also provided clients withunlimited access to the ?????????????????????????????????????????????????????????????????????????????????????? Roughly 60% of most clients??projects could be accomplished through reusing components from the catalog. The company coupledthe move from the hybrid consultancy model to a competition model with the reduction of manyplatform manager positions, leaving the company with 16 project managers servicing 35 clients bythe end of 2009.For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .610-­032 TopCoder (A): Developing Software through Crowdsourcing4As of late 2009, TopCoder ran two different types of competitions on its platform: algorithm andclient software development. Algorithm competitions served as the primary means for attracting newmembers and retaining existing members. These competitions required members to develop creativesoftware solutions to relatively difficult programming challenges. All members were assessed againsteach other through an automated computer scoring system;; they then received a TopCoder rating fortheir performance. Some algorithm competitions also had cash prizes for winners.The second type of competition targeted developing software applications for specific clientneeds. A TopCoder platform manager initially worked with the client staff to develop a ??game plan????(see Exhibit 3 for a representative game plan) or a project road map for building the software. Thefirst step typically involved a contest where the general client problem was presented to theTopCoder community in a conceptualization contest. Here contestants publicly cross-­examined theclient staff as to their actual needs and then submitted a business requirements document and high-­level use cases. The client chose the submission or submissions that best represented ???????? ??????????????????needs as the basis for further development. Then a series of specification contests was held to createthe application??s requirements documents, application wireframes (i.e. the logical flow of theapplication), and storyboards (detailed cases of the user experience). Next, the output of thespecification contests was fed into several architecture contests to create the overall system andcomponent level designs. At this point, the TopCoder platform manager would work with the clientto either select components from the catalog or commission the creation of new components throughdesign and development competitions. After the component production phase, all the relevantcomponents were put together through an assembly competition with the objective of creating aworking system. Assembly was then followed by certification and testing contests and then,eventually, deployment. Throughout the execution of the game plan, TopCoder retained flexibility in???????????????????????? ?????? ???????????????? ??bug r???????????? ?????? ???????????????????????? ?????????????????? ?????????????? ??????????????????????????s or unforeseenerrors.To determine winners and assess quality in client software development, TopCoder used acommunity-­based peer-­review system. In particular, expert and experienced TopCoder communitymembers were paid to grade and comment on all contest submissions using detailed scorecards,ultimately picking the contest winners. The winning competitors for each contest then receivedmonetary prizes, and all participants received updated ratings for their performance. TopCoder alsoran studio contests if an application required logos or graphics;; in those cases, clients chose thewinners.Evolution of the TopCoder CommunityGrowth and CompositionFrom 2001 to 2009, TopCoder added an average of 25,000 new computer programmers to itscommunity each year. After filling out a short online registration form, anybody in the world couldparticipate in a software development competition;; by spring 2009, the TopCoder community hadover 200,000 members (see Exhibit 4 for community growth). Although the size of the overallcommunity was large, the number of people within that community who actively participated incontests and posted in forums was much smaller. The majority of community members at TopCoderregistered as members of the community but never competed in any contests. In fact, by 2009, only35,000 unique individuals had competed in contests. To Mik???? ?????????????? ?????????????????????? chief technologyo???????????????? ???????? ???????????????????? ???????????? ?????? ???????? ???????????????????? ???????? ???????? ???????????????? ?????????????? ?????????????? ???????? ?????????? ??????????????????????For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .TopCoder (A): Developing Software through Crowdsourcing 610-­0325enough in the TopCoder platform to register and had the potential to provide TopCoder withincreased development under the right conditions.A second group within the TopCoder community comprised those members who at one timeparticipated in TopCoder contests but then stopped participating. Lydon noted that, after TopCoderdecreased prize values in 2008, many competitors from the United States and Canada left theTopCoder community. Yet another group included people who participated in TopCoder contestsbut did not win. TopCoder saw those competitors as the ???????????? ????????????people who primarilycompeted for the sake of learning????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????less-­skilled competitors could improve over time and increase their levels of contribution. Lastly,?????????????????????? ?????????? ?????????????????? ???????????? ?????? ???????????????????????? ?????????????????? ???????? ?????????????????? ?????????????????? ???????? ?????????????? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????for 0.5% of the total TopCoder population.The ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? in their 20s.According to Michael Paweska, a six-­???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ?????? ???????? ???????? ?????????? ?????????? ???????? ???????????????????????? ?????? ?????????? ?????????? ?????????????? ?????????????????? ?????? ???? ?????????????????????? ?????????????? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????TopCoder attractedcompetitors from developed nations such as the United States, Canada, South Korea, and Japan, aswell as from emerging economies such as China, Russia, Poland, India, and Ukraine. Wu Yanbo, aChinese TopCoder community member studying abroad in Australia, explained that mostcompetitors in the lower-­paid contests were from developing countries. According to Wu, the prizeswere not large enough for many individuals from developed countries to compete, since they couldspend their time better elsewhere.Justin Gasper, a member since 2001, began experimenting with the TopCoder platform whileworking for a traditional software engineering company. After winning significant money withTopCoder, Gasper decided to quit his job in 2005 and devote 40 to 50 hours a week to TopCoder.Gasper explained: ???????????????????? ?????? ?????? ????????-­?????????? ?????????? ???? ???????????? ?????????? ???? ???????? ?????????? Gasper was one of?????????????????????? ???????????????? ?????????????????? ???? ?????????????? ?????? ???????? ???????????????? ?????????????? ?????? ?????????????????????????? ?????? ??????????????????????????competitions, Gasper won at least second place 95% of the time and had a win percentage of 69.23%.Competitors at TopCoder could choose which contests and what type of contests to join (seeExhibit 5 for participation and prize data by contest type).Profiles and RatingsEach programmer in the TopCoder community maintained a public profile that displayed his orher user name, contest history, and basic personal information. Another part of the member profile??????????????????????????????????????????????????numeric rating for each type of contest. The rating system was modeled onthe one used to rank grandmaster chess players engaged in worldwide competition. A ?????????? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? within the community and a high skill level.Yellow, blue, and green color ratings represented descending skill levels. ????????????????????????????????????countryrank, total community rank, success rates for contests, and reliability??or percentage of times thecontestant joined a competition and submitted a passing solution??were featured in their profiles.TopCoder members could also choose whether or not to display their total earnings on their profiles(see Exhibit 6 for an example member profile).For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .610-­032 TopCoder (A): Developing Software through Crowdsourcing6Motivating MembersBetween 2001 and 2009, TopCoder paid out over $20 million in prizes and peer review money toits community of developers. However, prize money was not evenly distributed throughout theTopCoder community. The top 5% of prize earners received approximately 80% of the total prizepool, while the majority of TopCoder community members earned little or no money fromcompetitions. Some competitors were extremely successful. For example, from 2006 to 2008, Paweskaearned $200,000 to $300,000 per year, while Gasper averaged over $100,000 annually. Wucommented: ??????????????????????????????????????????is the most attractive thing. The prize is very good compared to theincome of my friends who are working in some local companies in China. Even though the economyis not very good and TopCoder reduced its prizes, I can still earn around $1000 per month in my??????????????????????????TopCoder typically awarded prizes to the top two submissions in each contest, with the??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????Besides prizes awarded on a contest-­by-­contest basis, another main source of income for memberswas the Digital Run. In the Digital Run system, the top five ranked competitors for each contest wereawarded points based on contest rank and performance. At the end of each month, TopCoder tallied?????????????????????????? ???????????? ?????????????? ???????? ???????????????? ???????? ???????? ???????????? ??arners thousands of dollars in bonus prizes.Paweska explained that success in the Digital Run was not all about who was the best programmerbut more about who could handle the most all-­nighters. Other competitors, such as Gasper, alsomade money through contracted projects that TopCoder assigned.In addition to their cash earnings, many community members reported that their TopCoder ratingwas very important because it provided an objective assessment of ability. Wu commented that it wasnot easy to maintain a very high rating as it required familiarity with many kinds of technologies,quick thinking, the ability to learn independently, a strong work ethic, and attention to detail.According to W?????? ???? ?????????????????? ?????????????? ???????????? ?????? ???????????????????? ???????? ???? ?????????????????????????? ?????????????? ???????????????? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? Gasper noted thatTopCoder ratings were also symbols of status and prestige for many programmers: ??If you have red???????????????????????????????????????????????????????????????? Indeed, many prestigious software firms asked potential recruits toget a TopCoder rating before applying for a job. To others, however, the rating system was lessimportant. Gasper, for example, explained that winning and making money meant more to him thanratings.Although there were differences of opinion regarding the importance of ratings, almost allcommunity members agreed that competing at TopCoder provided numerous opportunities to learnand improve. In fact, for many programmers, a TopCoder career often began with failure, but post-­contest evaluation and peer review of each submission helped them grow and improve. Gaspernoted: ?????????????????????????????????? in my first competition. But the reviewers were really good at pointing me inthe right direction???????????????????????????????????????????????????????????????????????????? ?? ??????????????????????????????????????????????????????????????????????????????????peer reviewed by people who are better at programming ????????????????????????????????????????????????????????????????????????????????????they hurt your feelings;; they are ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????that getting feedback from reviewers was crucial and added that community members could alsolearn from acting as a reviewer for contests. For scientists and developers, Wu believed thatalgorithm contests were particularly helpful at sharpening research skills and improving criticalthinking abilities. In all cases, continual learning opportunities from peers were an important reasonfor participation.Gasper described the appeal of working at home on a web-­based platform instead of in a?????????????????????????????????????????????????????????????????????? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .TopCoder (A): Developing Software through Crowdsourcing 610-­0327a half an hour to work each day and can do the same work at home. If I want to take off a day to playgolf, I just do i?????? ???? ?????????? ???????????? ?????????? ?????? ???????? from 2:00 to 6:00 p.m.???? ?????? ?????????? ?????????????????????????? ??????????????Sharing similar sentiments, Paweska liked that while working at TopCoder he did not have asupervisor looking over his shoulder.???????????????????????????????????????? ?????? convenient but also challenging, as competitors had to actively managetheir individual levels of participation. Gasper constantly balanced effort and reward to maximize?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? too much work for???????????????????????????????????????????????????? . . . ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????worthwhile to solve???????????????? skill that ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????Wu noted that, although the firm was competitive in sprit, competition at TopCoder was neverdisrespectful or nasty and that people liked to help each other, even when they competed in the samearena. TopCoder forums were the main source for collaboration. In the forums, less-­experiencedcommunity members asked for assistance on certain problems and received instant feedback frommore-­experienced competitors.At TopCoder, conversations and relationships extended beyond the scope of softwaredevelopment. Hughes reflected on a particularly remarkable exhibit of communal strength and?????????????? ???????? ?????????????? ???????????????????? ???????????????? ???????????????? ?????? ?????????????????? ?????????????????????????? ???????????? ???????? ?????? ???????????????????????????? ???????????????? ?????????????? ?????? ???????????? ?????????? ?????????????????????? ?????? ???????????????? ???????? ?????????? ?????????? ???? ?????????????? ?????? ????????community members took ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????Once a year, TopCoder paid for all of the best talent from the community to travel to Las Vegas,Nevada, to compete in the TopCoder Open (TCO). In addition to serving as a proving ground for thebest programmers in the world, the TCO provided community members with the opportunity tonetwork professionally and socially.The TopCoder community had a distinctive culture, with identifiable personalities. Wu explained:?????? ???????????????? ?????????? ?????????????????????? ?????????? ???????? ???????????????? ???????? ???????? ???????? ?????????????????? Clearly, the members built it upcontinuously. When I joined the community, there were already some leading members who wereactive in competitions and forums, brought out good suggestions, and started up interesting and???????????????????? ???????????????????????????? ?????? ?????????? ?????????????? ???????? ?????????? ?????? ???????????????????? ???????????????? ?????????????????? well beyondTopCoder. For example, Tomasz Czajka??????????????????????????????????????????????????rock star???????????????????????????????????????? pictureplastered on billboards throughout Warsaw after he won the TopCoder Open in 2006.????????????????????????????????????????????????Clients came to TopCoder to have high-­quality software developed in a cost-­effective and time-­efficient manner. TopCoder positioned itself to serve both large firms and medium-­ to small-­sizedbusiness that wanted to see systems developed. Keith Moore, a TopCoder client and former seniorvice president at LendingTree.com, believed that, regardless of size, any company could takeadvantage of TopCoder, whether it was a five-­person operation or large outsourcing vendor. Formany CIOs, the process of software development and talent recruitment was a major headache, andmissed deadlines and large cost overruns were common worries. According to Stephen Laster, the???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? over 48% of itsemployees every three years. This process is very costly. The same problem exists with ourFor the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .610-­032 TopCoder (A): Developing Software through Crowdsourcing8outsourcing consultants. When selecting consultant teams, we tried out 60 programmers beforefinding our team of 20. With TopCoder, I pay for performance and the CIO sees Nirvan????????As of 2009, TopCoder had developed a strong relationship with existing clients for deliveringhigh-­quality software solutions and superior customer service. After completing their first projectwith TopCoder, 82% of clients signed up for a second round of contests. These clients cited severaladvantages.BenefitsBetter Ideas Before sinking thousands of dollars into a project, a client could run aconceptualization contest through which TopCoder members helped identify bad ideas and generatebetter approaches early in the development cycle. When the client introduced a business problem tothe community, members asked hundreds of questions. Nic Perez, a former technical director atAOL, ?????????????????????????????????