The following is likely the final chapter in the story of Replica in Portland, as told by yours truly, Kate Kaye. I have conducted deep research and reporting on the project since it began in earnest in 2019, for this site as well as for Fast Company and Geekwire. This story is published here on RedTail in part because I have a staff reporting gigthat precludes me from reporting for other publications.
What you’ll learn in this article:
Replica’s project in the Portland-area, led by regional government entity Metro, was fraught with disagreements and frustrations.
Following a year-and-a-half of working together without getting the project off the ground, client and tech vendor eventually agreed it was in their “mutual best interests” to end the partnership.
Metro appears to have been a demanding client and in the end the agency’s project lead expressed distrust in the company.
Replica complained that Metro wanted information that either was not feasible or deviated from the firm’s approach to privacy.
Documents and email records obtained via FOIA request for this and other Replica stories are available within.
It’s not often when city governments dig deep into software products before they’re deployed. But after year-and-a-half of testing and customizing Replica — the mobility analytics product spawned by Google’s city tech sibling Sidewalk Labs — Portland, Oregon’s regional government agency Metro pulled the plug on the project. Records obtained via freedom of information requests as well as direct communications with Metro and Replica reveal project roadblocks at every turn, along with a series of he-said-she-said disputes over data quality and transparency, access, privacy and equity.
“Replica’s conduct has not only prevented us from having the transparency that we need to rely on Replica’s data; it’s damaged our trust in the project and in Replica as a partner,” wrote Metro Technology Strategist Eliot Rose in an October 2020 email sent to Replica CEO Nick Bowden and Caroline Ahern, the firm’s customer success manager.
Replica is software created to help city governments understand how people move around a given region. It uses artificial intelligence and machine learning to build a synthetic populous that can be tracked and analyzed. The idea is to replace old-school, relatively expensive approaches like surveys to gauge people’s travel behavior in a way that’s intended to protect privacy.
In the end, the government staff who launched the project officially in May 2019, eager for privacy-safe, real-time data on how people move around the Portland area, decided the tech was more a unique data product than something that met their city planning needs. “At best, Replica is a novel experimental data source that we can internally compare to our travel model and other data to improve our methods over the long term,” Rose wrote in that email sent to the Replica team. “That’s not without value, but it’s not what we agreed to pay for, nor is it worth the resources that we’ve dedicated to this project.”
“Replica’s conduct has damaged our trust in the project and in Replica as a partner.” – Eliot Rose, Metro in Portland
Bowden fired back in the October email exchange: “You stated you do not have enough transparency about how accurate and representative Replica’s data is to use with confidence. This statement feels unfounded on many levels. Over the last two years, we have created, refined, and provided substantial documentation about our technical methodologies, data sources, data coverage, and our approach to privacy.”
Quality Fish and a Federal Smart City Contest It was way back in 2016 when Portland government staff would have first gotten a glimpse of the Replica concept in its nascence. The U.S. Department of Transportation announced its plan then to work with Sidewalk Labs and seven finalists for its Smart City Challenge grant — Austin, Columbus, Denver, Kansas City, Pittsburgh, Portland and San Francisco — to develop a data platform called Flow, which appears to be an early version of Replica. Columbus got the grant. But the seed was planted for a partnership in Portland.
“This statement feels unfounded on many levels.” – Replica CEO Nick Bowden
There was excitement in the air when talks between Metro staff and Replica began in earnest in late 2017, emails show. The software spin-off was still just a product that Sidewalk Labs expected to pilot in Toronto, home to its “city of the future” Sidewalk Toronto project, which imploded in 2020.
Portland isn’t the only city to stop working with Replica, either. Protocol about a year ago reported that Kansas City found value in a trial of the software, but ultimately, “didn’t have enough staff ‘to take advantage of all of [Replica’s] capabilities.’”
Replica was spun out from Sidewalk Labs in 2019 as a separate firm (Bowden displayed tongue-in-cheek humor when he established Replica in California as a “quality fish reproductions” business). The company still has deep ties to Google parent company Alphabet though. A 2020 Alphabet investor filing shows Sidewalk Labs and Innovation Endeavors III, an investment fund affiliated with former Google CEO Eric Schmidt, together have 47% ownership of Replica.
For what it’s worth, Replica appears to be in building mode. The company lists several job openings on its site.
A Sputtering Process Throughout the span of the relationship, the Replica project in Portland seemed to spin its wheels, always close to fully operable and implemented, but never quite there.
Replica isn’t exactly plug-and-play. Metro knew going into the project that some level of customization was required, and vendor and client worked closely throughout 2019 and the better part of 2020 to calibrate and test the system. In order for Metro to ensure Replica’s digital population avatars truly reflected where, when and in what modes people in the region typically move around, the agency gave the company a set of acceptance criteria it would have to meet. If the acceptance criteria were not satisfied, the parties agreed they would not proceed.
An amended contract between Metro and Replica contains the agreed upon acceptance criteria for the project as of late 2019.
“We don’t feel like we can rely on Replica…” A couple things really got Rose and the Metro crew excited about Replica. One was the ability to gauge movement patterns of people on bikes, scooters or on-foot rather than more-readily tracked cars and public transportation. The other: the promise of predicting how traffic or public transportation decisions might affect the BIPOC community or other vulnerable residents. For my May 2019 Geekwire report about the project, Rose told me, “We haven’t seen any tools that have offered the kind of information Sidewalk Labs is offering regarding pedestrian and cyclist mobility.”
But by August 2020, Metro decided Replica had failed to meet its acceptance criteria. So, Replica proposed to alter that criteria. But Metro didn’t like the idea much.
“Based on what you’re proposing, we don’t feel like we can rely on Replica as a source of bike/ped [pedestrian] data, as an input to our travel model, or as a tool for detailed equity analyses – all of which are central to our use cases and to the reasons why we chose to pursue this partnership in the first place,” Rose told Ahern, the Replica customer rep.
Ya Gotta Give Data to Get Data In Replica’s defense, Ahern, in an email to Rose at the time, pointed to Metro’s failure to supply the company with adequate ground truth data, particularly of the very sort Metro had hoped to obtain through its relationship with Replica: data showing bike and pedestrian movement in the region. The company uses this type of ground truth bike and walking data as a control group to calibrate its models.
“We don’t feel like we can rely on Replica as a tool for detailed equity analyses.” – Eliot Rose
Essentially, said Ahern, Replica didn’t get enough of that data from Metro. “While Portland is unfortunately not unique in its limited ground-truth for bike and pedestrian data, it’s worth noting that we were only supplied with seven (7) bike sensors and pedestrian counts that only included 2 hours worth of counts per location/day,” wrote Ahern at the time to Rose and other Metro staff, cc’ing Bowden.
For the record, when I first inquired about the Replica project back in 2019, sources in Portland told me no sensors or physical tracking devices would be involved.
“There is no requirement for customer agencies to provide this information; however, it can ensure higher levels of measurability when provided,” a Replica spokesperson told me in an email exchange this month. “Metro has always had the ability to use Replica data to better understand bike and pedestrian movements.”
A Pileup of Allegations A couple months later, the relationship had deteriorated even further. Rose in an October 2020 email to Replica ran through a litany of additional reasons Metro decided to sever ties with the company. In addition to concerns that the company “did not conduct appropriate due diligence” when evaluating the data Metro supplied to Replica to help customize its system for the region, Rose said the firm failed to provide some other important information.
“For us, ‘transparency’ goes [sic] means that we can share quantitative, objective information with the public on whether the data we use is representative, accurate, unbiased, and protective of privacy,” wrote Rose.
Namely, Rose said Metro asked for but did not receive:
Margins of error In order to assess accuracy, Metro wanted quantitative information showing margins of error associated with Replica’s travel behavior estimates.
A full privacy audit Metro said it only received an overview of a privacy audit Replica conducted, but the agency wanted the full report. Rose had expressed frustration over the company’s apparent stonewalling regarding the issue since as early as February of 2019, as I reported in Fast Company.
“Metro staff unfortunately never took the time to review.” – Replica spokesperson
Replica denied these allegations. A company spokesperson told me via email this month, “Replica submitted all deliverables with a detailed report of accuracy measures. Metro staff unfortunately never took the time to review.” As for the full privacy audit, the company said, “A complete report was provided. Our records show that Metro never opened it.”
Disputes over Privacy and Data Equity But as contention over privacy swirled, another data ethics dispute had Metro on the defensive.
Metro wanted data showing home and work locations gathered from mobile devices Replica uses to build its synthetic dataset. The agency wanted this information to evaluate whether the system adequately represents BIPOC communities and rural residents. “We planned to use demographic testing to ensure that the data we use is representative, and it’s only because you changed your data sources that we are requesting additional information on representativeness,” wrote Rose in his October email detailing problems with the company. “You continue to provide aggregate calibration reports rather than disaggregate reports, which is inconsistent with our agreement and which makes Replica appear to perform better than it actually does.”
“Metro continued to insist that Replica deviate from our methodology and privacy principles.” – Replica spokesperson
This bit about disaggregate data had been a sticking point. Supplying that data would have presented its own privacy risks, as I reported in the Fast Company piece. The Replica spokesperson said as much in our email exchange this month: “Metro wanted Replica to provide demographic data that had a high level of detail about individuals, but that goes against Replica’s philosophy on privacy as well as our methodology. We didn’t feel comfortable sharing the high-fidelity data on individuals they were requesting, which frustrated the Portland team. In that sense, it is not a question of mismanagement of the project but rather that Metro continued to insist that Replica deviate from our methodology and privacy principles.”
End of the Replica Road Trip A contract termination document was attached to that October message but not made available in the records haul. This more recent version from November appears to be the final version.
Although the FOIA’d emails indicate initial disagreement and legal wrangling over whether the contract would be terminated as a result of breach of contract or COVID-19 related force majeure, the November termination document states, “The Parties have determined that it is in their mutual best interests to terminate the Software Agreement.”
So ends the saga that was Replica in Portland. As a reporter with an interest in tech and data ethics issues living in Portland, the project became a bit of a journalistic side-obsession for me. However, I believed it was important for someone to research and report the city’s potential use of this algorithmic system, one that could inform very consequential decisions about how people get around this place.
What I’ve presented in this story is merely an overview of what’s evident about the project and how it played out. If you peruse the full stack of FOIA’d emails I used to report this story, you’ll find there are several other points of contention over the data – who asked for what and whether they got what they needed. I invite [bored, nerdy] readers to dig into that.
For lack of time, this story does not tackle other aspects of this situation such as whether any of the $457,000 Portland agencies including Metro were contracted to pay — had the project officially launched — ever changed hands. I invite other reporters to pick up the mantle.
For now, that’s the end of the line for this Replica-in-Portland road trip.
The following is likely the final chapter in the story of Replica in Portland, as told by yours truly, Kate Kaye. I have conducted deep research and reporting on the project since it began in earnest in 2019, for this site as well as for Fast Company and Geekwire. This story is published here on RedTail in part because I have a staff reporting gig that precludes me from reporting for other publications.
What you’ll learn in this article:
It’s not often when city governments dig deep into software products before they’re deployed. But after year-and-a-half of testing and customizing Replica — the mobility analytics product spawned by Google’s city tech sibling Sidewalk Labs — Portland, Oregon’s regional government agency Metro pulled the plug on the project. Records obtained via freedom of information requests as well as direct communications with Metro and Replica reveal project roadblocks at every turn, along with a series of he-said-she-said disputes over data quality and transparency, access, privacy and equity.
“Replica’s conduct has not only prevented us from having the transparency that we need to rely on Replica’s data; it’s damaged our trust in the project and in Replica as a partner,” wrote Metro Technology Strategist Eliot Rose in an October 2020 email sent to Replica CEO Nick Bowden and Caroline Ahern, the firm’s customer success manager.
Replica is software created to help city governments understand how people move around a given region. It uses artificial intelligence and machine learning to build a synthetic populous that can be tracked and analyzed. The idea is to replace old-school, relatively expensive approaches like surveys to gauge people’s travel behavior in a way that’s intended to protect privacy.
In the end, the government staff who launched the project officially in May 2019, eager for privacy-safe, real-time data on how people move around the Portland area, decided the tech was more a unique data product than something that met their city planning needs. “At best, Replica is a novel experimental data source that we can internally compare to our travel model and other data to improve our methods over the long term,” Rose wrote in that email sent to the Replica team. “That’s not without value, but it’s not what we agreed to pay for, nor is it worth the resources that we’ve dedicated to this project.”
Bowden fired back in the October email exchange: “You stated you do not have enough transparency about how accurate and representative Replica’s data is to use with confidence. This statement feels unfounded on many levels. Over the last two years, we have created, refined, and provided substantial documentation about our technical methodologies, data sources, data coverage, and our approach to privacy.”
Quality Fish and a Federal Smart City Contest
It was way back in 2016 when Portland government staff would have first gotten a glimpse of the Replica concept in its nascence. The U.S. Department of Transportation announced its plan then to work with Sidewalk Labs and seven finalists for its Smart City Challenge grant — Austin, Columbus, Denver, Kansas City, Pittsburgh, Portland and San Francisco — to develop a data platform called Flow, which appears to be an early version of Replica. Columbus got the grant. But the seed was planted for a partnership in Portland.
There was excitement in the air when talks between Metro staff and Replica began in earnest in late 2017, emails show. The software spin-off was still just a product that Sidewalk Labs expected to pilot in Toronto, home to its “city of the future” Sidewalk Toronto project, which imploded in 2020.
Portland isn’t the only city to stop working with Replica, either. Protocol about a year ago reported that Kansas City found value in a trial of the software, but ultimately, “didn’t have enough staff ‘to take advantage of all of [Replica’s] capabilities.’”
Replica was spun out from Sidewalk Labs in 2019 as a separate firm (Bowden displayed tongue-in-cheek humor when he established Replica in California as a “quality fish reproductions” business). The company still has deep ties to Google parent company Alphabet though. A 2020 Alphabet investor filing shows Sidewalk Labs and Innovation Endeavors III, an investment fund affiliated with former Google CEO Eric Schmidt, together have 47% ownership of Replica.
For what it’s worth, Replica appears to be in building mode. The company lists several job openings on its site.
A Sputtering Process
Throughout the span of the relationship, the Replica project in Portland seemed to spin its wheels, always close to fully operable and implemented, but never quite there.
Replica isn’t exactly plug-and-play. Metro knew going into the project that some level of customization was required, and vendor and client worked closely throughout 2019 and the better part of 2020 to calibrate and test the system. In order for Metro to ensure Replica’s digital population avatars truly reflected where, when and in what modes people in the region typically move around, the agency gave the company a set of acceptance criteria it would have to meet. If the acceptance criteria were not satisfied, the parties agreed they would not proceed.
An amended contract between Metro and Replica contains the agreed upon acceptance criteria for the project as of late 2019.
“We don’t feel like we can rely on Replica…”
A couple things really got Rose and the Metro crew excited about Replica. One was the ability to gauge movement patterns of people on bikes, scooters or on-foot rather than more-readily tracked cars and public transportation. The other: the promise of predicting how traffic or public transportation decisions might affect the BIPOC community or other vulnerable residents. For my May 2019 Geekwire report about the project, Rose told me, “We haven’t seen any tools that have offered the kind of information Sidewalk Labs is offering regarding pedestrian and cyclist mobility.”
But by August 2020, Metro decided Replica had failed to meet its acceptance criteria. So, Replica proposed to alter that criteria. But Metro didn’t like the idea much.
“Based on what you’re proposing, we don’t feel like we can rely on Replica as a source of bike/ped [pedestrian] data, as an input to our travel model, or as a tool for detailed equity analyses – all of which are central to our use cases and to the reasons why we chose to pursue this partnership in the first place,” Rose told Ahern, the Replica customer rep.
Ya Gotta Give Data to Get Data
In Replica’s defense, Ahern, in an email to Rose at the time, pointed to Metro’s failure to supply the company with adequate ground truth data, particularly of the very sort Metro had hoped to obtain through its relationship with Replica: data showing bike and pedestrian movement in the region. The company uses this type of ground truth bike and walking data as a control group to calibrate its models.
Essentially, said Ahern, Replica didn’t get enough of that data from Metro. “While Portland is unfortunately not unique in its limited ground-truth for bike and pedestrian data, it’s worth noting that we were only supplied with seven (7) bike sensors and pedestrian counts that only included 2 hours worth of counts per location/day,” wrote Ahern at the time to Rose and other Metro staff, cc’ing Bowden.
For the record, when I first inquired about the Replica project back in 2019, sources in Portland told me no sensors or physical tracking devices would be involved.
“There is no requirement for customer agencies to provide this information; however, it can ensure higher levels of measurability when provided,” a Replica spokesperson told me in an email exchange this month. “Metro has always had the ability to use Replica data to better understand bike and pedestrian movements.”
A Pileup of Allegations
A couple months later, the relationship had deteriorated even further. Rose in an October 2020 email to Replica ran through a litany of additional reasons Metro decided to sever ties with the company. In addition to concerns that the company “did not conduct appropriate due diligence” when evaluating the data Metro supplied to Replica to help customize its system for the region, Rose said the firm failed to provide some other important information.
“For us, ‘transparency’ goes [sic] means that we can share quantitative, objective information with the public on whether the data we use is representative, accurate, unbiased, and protective of privacy,” wrote Rose.
Namely, Rose said Metro asked for but did not receive:
Margins of error
In order to assess accuracy, Metro wanted quantitative information showing margins of error associated with Replica’s travel behavior estimates.
A full privacy audit
Metro said it only received an overview of a privacy audit Replica conducted, but the agency wanted the full report. Rose had expressed frustration over the company’s apparent stonewalling regarding the issue since as early as February of 2019, as I reported in Fast Company.
Replica denied these allegations. A company spokesperson told me via email this month, “Replica submitted all deliverables with a detailed report of accuracy measures. Metro staff unfortunately never took the time to review.” As for the full privacy audit, the company said, “A complete report was provided. Our records show that Metro never opened it.”
Disputes over Privacy and Data Equity
But as contention over privacy swirled, another data ethics dispute had Metro on the defensive.
Metro wanted data showing home and work locations gathered from mobile devices Replica uses to build its synthetic dataset. The agency wanted this information to evaluate whether the system adequately represents BIPOC communities and rural residents. “We planned to use demographic testing to ensure that the data we use is representative, and it’s only because you changed your data sources that we are requesting additional information on representativeness,” wrote Rose in his October email detailing problems with the company. “You continue to provide aggregate calibration reports rather than disaggregate reports, which is inconsistent with our agreement and which makes Replica appear to perform better than it actually does.”
This bit about disaggregate data had been a sticking point. Supplying that data would have presented its own privacy risks, as I reported in the Fast Company piece. The Replica spokesperson said as much in our email exchange this month: “Metro wanted Replica to provide demographic data that had a high level of detail about individuals, but that goes against Replica’s philosophy on privacy as well as our methodology. We didn’t feel comfortable sharing the high-fidelity data on individuals they were requesting, which frustrated the Portland team. In that sense, it is not a question of mismanagement of the project but rather that Metro continued to insist that Replica deviate from our methodology and privacy principles.”
End of the Replica Road Trip
A contract termination document was attached to that October message but not made available in the records haul. This more recent version from November appears to be the final version.
Although the FOIA’d emails indicate initial disagreement and legal wrangling over whether the contract would be terminated as a result of breach of contract or COVID-19 related force majeure, the November termination document states, “The Parties have determined that it is in their mutual best interests to terminate the Software Agreement.”
So ends the saga that was Replica in Portland. As a reporter with an interest in tech and data ethics issues living in Portland, the project became a bit of a journalistic side-obsession for me. However, I believed it was important for someone to research and report the city’s potential use of this algorithmic system, one that could inform very consequential decisions about how people get around this place.
What I’ve presented in this story is merely an overview of what’s evident about the project and how it played out. If you peruse the full stack of FOIA’d emails I used to report this story, you’ll find there are several other points of contention over the data – who asked for what and whether they got what they needed. I invite [bored, nerdy] readers to dig into that.
For lack of time, this story does not tackle other aspects of this situation such as whether any of the $457,000 Portland agencies including Metro were contracted to pay — had the project officially launched — ever changed hands. I invite other reporters to pick up the mantle.
For now, that’s the end of the line for this Replica-in-Portland road trip.
Share this: