Follow-Up to World of Workcraft

Here is the accompanying report we had to go along with our scenario of World of Workcraft. This report outlines some of the technical and research challenges. As before, credit should be shared with Rick Wash and Michael Bernstein. Blame should go to me.

Introduction

We envision a future comprised of a highly flexible workforce, where people with appropriate skill sets can be quickly and easily brought together on demand into flash teams to complete projects of varying sizes, difficulties, and time scales. These projects will also function as apprenticeships for people, helping workers to continually learn useful skills that will enable them to actively participate and adapt to changes in the dynamic knowledge economies of the twenty-first century.

We are starting to see early glimpses of this kind of future work force. For example, many graphic designers and programmers are hired on a consulting basis, working with short-term project teams that are dissolved once the project is completed. There are also numerous temporary work agencies that place workers into a wide range of positions. For example, Manpower Inc. is an agency for placing temporary workers, claiming over 400,000 clients per year. Finally, we are seeing increases in the number of people who telecommute, with over 17.2 million people occasionally telecommuting in 2008. As more and more media is digitized, and as the quality of collaboration tools increase, the number of people who can potentially work from anywhere in the United States will only increase.

Our vision is to vastly expand the number of domains and the number of people who can participate in this kind of knowledge economy. However, there are currently numerous technical, legal, and economic barriers in place, introducing transaction costs that create significant frictions and inefficiencies. Below, we outline several technical challenges that must be overcome before this vision can come to fruition.

There are many societal benefits if we achieve this vision. One is a flexible workforce that is continually adapting to changes in the world economy. Another benefit is far lower barriers for finding work that matches their skills and interests. A third benefit is greater innovation, as people can try many more ideas quickly and cheaply. People can also share knowledge and skills to more people. Furthermore, just as cloud computing vastly reduces one's front-end capital costs and space requirements and thus lower initial startup costs for new companies, crowd computing offers the same for the personnel needed for startups. A fourth benefit is a reduced need to commute to a central workplace, helping to reduce traffic congestion and pollution as well as improving quality of life.

Electronically Mediated Skilled Worker Marketplace

One of the major technical challenges we face in bringing forth this vision is the creation of an electronic marketplace for skilled labor and flash teams. Currently, finding and hiring skilled labor has high transaction costs: finding skilled workers, negotiating legal and HR contracts, and putting together effective teams.

We envision a new form of Internet-based electronically mediated work marketplace that can dramatically reduce the transaction costs of short-term skilled labor by efficiently matching skilled workers with market needs. The first challenge is specification: potential workers can specify their skills and their interests, and employers need to specify their needs and the projects they want completed. Currently on systems, this specification is usually very vague: specifics might be negotiated over time between employers and employees. In current microtask markets like Amazon Mechanical Turk, specifics might be left out entirely. As we push toward smaller tasks and automated matching, we need to make these specfications more specific. A worker isn't just "a programmer" but rather, someone with active skills in Python, Perl, rusty skills in Java, and recent experience with web development for Facebook apps. With sensors and computer vision technologies, these skills could also include those outside the computer, including for example mechanical skills or knowledge of a geographic area. Additionally, we need to have a way of warranting, or attesting to the validity of these skills.

Another related challenge for the electronic marketplace is efficient task-worker matching. Currently, most matching is done via an expensive search process: workers linearly scan through an enormous list of jobs looking for ones they might be interested, and then they apply for those jobs. (See for example, Chilton, L., et al. Task Search in a Human Computation Market). This search is a very inefficient method of matching skilled workers to employer needs. Instead, we propose centralizing the matching process and having computers assist the process of forming flash teams. But, what is the best role for computers to take, and how do we do this? One example where this has been successful is the National Resident Matching Program, where newly minted MDs go to get their first residency job after medical school. This is a centralized match where MDs submit where they would liek to work, and hospitals submit who they want to hire, and then the NRMP tells everyone who will work where.

When both workers and tasks are much more heterogeneous -- creating an entire job market -- we will need new sociotechnical designs to facilitate matching. This intelligence could incorporate elements of economics and computer science. Economics expertise can help ensure that the resulting matches are efficient and suggest appropriate incentives for workers and employers to honestly reveal their skills and needs. Computer science will be needed to help people specify their skills and tasks (HCI) and to build this system so that it can appropriately scale. Workers need a way of (semi-automatically) specifying their resume and warranting their skills and experience, but still having the flexibility to retain agency. Employers will need a way of clearly specifying job parameters, including specifying the work that needs to be done and any additional contracts -- financial, legal, intellectual property, benefits, etc. -- that are associated with the job.

The biggest challenge will be developing appropriate algorithms for doing this matching. There are both strong technical and economic constraints on this algorithm. The match is of high dimensionality; workers need to be matched to jobs across multiple skillsets, location constraints, financial/legal constraints, worker interests, task domains, and learning/development/long-term goals. Understanding and specifing the tradeoffs inherent in this type of match is very difficult, and using a computational algorithm -- such as a maching learning algorithm or other AI system for creating this matching -- is an open research problem. Additionally, there are considerable economic constraints on this algorithm. The resulting matches have to be satisfactory to both the workers and the employers. And the process -- the way the algorithm creates the match -- should be seen as fair, working best when both sides of the market are honest. This economic constraint, inducing honest revelation, is really important because if people are deceptive in their resume or their job descriptions, then the computer will not have the information it needs to effectively create matches.

Lifelong Learning

We hope that this type of system will support lifelong learning among the workers in the system. Being an ad-hoc worker in a system like this allows people try and do many small tasks quickly. Workers who want to learn new skills can work on projects outside their current skillset but within their zone of proximal development. One of the technical challenges for this system is to figure out how to specify the zone of proximal development and incorporate it into the matching process. We want people to receive matches near their current skill level so that they can learn. Alternatively, workers might be matched with a mentor who can oversee their work and help them learn a new domain. However, including these learning goals in a matching algorithm means that, to some extent, dishonest workers might be able to game the system and use it to get tasks that they should never be matched with. Allowing for mentorship and training and development should be supported, while simultaneously preventing people from gaming the system.

A second technical challenge incorporating learning into explicit feedback systems. Consider, for example, a reputation system like eBay's: a worker's early attempts may not be very skilled, but over time and with learning they will improve. However, if the reputation system still includes that early work as an important part of their formally specified reputation, that may artifically hold the worker down and prevent them from receiving appropriate work. We need to better design reputation and resume specification to allow for learning and change over time; a person's skills and reputation is not static but is constantly changing. (However, spammers may take advantage of such a system to appear more reputable than they actually are.)

A third technical challenge is helping workers figure out their zone of proximal development, and suggesting what types of "stretch" tasks might be the best way to expand skills and continue lifelong learning. The marketplace might be able to use a recommendation system to suggest new skills and projects to undertake to support lifelong learning: "other workers like you undertook these projects to develop their skills." However, those recommendations should include both what they are likely to agree to learn, but also include information about the market and what kinds of skills are needed and valuable.

Worker Skill Specification

To faciliate effective matching between workers and tasks, the system needs a strong reputation and skill system. Traditional resumes are concise but may not help with matching workers to tasks because they are fairly high-level. We envision a much richer range of skills in our system, enabling workers to fluidly move between tasks that exercise multiple aspects of their skill sets. Workers can build up an automatic resume of work history and skills: this includes traditional skills like programming in Python, as well as situational skills like living in Boston, hobbies like beer brewing, or familiarity with the Appalachian Trail, all of which could be useful depending on the specific task on hand. Potentially, managers could provide feedback to workers in ways that build up expertise profiles (or levels). Workers can then apply for more lucrative tasks that require high-level workers, like a "Level 5" Python programmer. This framework would also enable workers to take qualification and training tasks to "level up" and learn new skills, opening up new jobs and making re-training in a difficult economy much more feasible.

In technical terms, this goal requires a framework for workers to explicitly author and manage their skill set, as well as for an automatic reputation system like eBay or Amazon. Other challenges include accurately capturing what skills a person has, verifying those skills, and helping workers manage what skills are disclosed to others. These are all challenging from a sociotechnical design perspective, as workers' reputations are on the line.

Better Collaboration Workspaces

Workers and requesters require new interfaces to support complex, ad-hoc work structures. How can requesters (managers) control the process and ensure good results? How can workers learn more about the task they are performing, communicate with the requester, or even rewire the entire workflow? In this section we describe some research challenges for interfaces supporting workers and managers.

Managers need tools for task specification, awareness, and coordination of their distributed flash teams. They need to author workflows. Authoring should be as easy as describing the job to another person: "I need these ten interface designs mocked up in a vector art program. Then get another designer to iterate on them, and fifteen potential users to try each. I want a usability researcher to run that study and report the outcome to me." The manager can convey this request to a meta-worker or an algorithm, which transforms it into a workflow, or the manager might use a template workflow for specific kinds of tasks. Once the task has begun, the manager needs awareness and coordination support for the team. Can the manager inspect progress and ask workers at particular stages how the work is going, ensure effective communication flow between stages, and do quality control checks?

Workers need interface support to ensure that they complete the correct task and do it effectively. Currently, workers on systems like Mechanical Turk see only a thin slice of the problem, so it is very easy to misinterpret instructions. Workers need to be able to inspect the history of an artifact they are working on: who worked on this job previously, and what were they supposed to do? They need to be able to communicate with the manager, getting clarification or asking for in-progress feedback. They should also be able to suggest alternate workflows: what if the user experience engineer sees an early usability problem and wants to send the mockup back to the graphic designer for a quick fix, or wants to consult an accessibility expert before making a recommendation?