Project Overview
The US Digital Service started as an Obama White House team responding to the dramatic failure of healthcare.gov (where we spent $700 million building a website that wouldn't turn on). We bring private sector tech expertise into high-stakes federal projects. Stephen Levy profiled us in the article “Inside Obama’s Stealth Startup.”
The Quality Payment Program (QPP) is the largest payment reform effort in Medicare history. Our analytics platform will award approximately $178 billion in financial incentives to shift Medicare from a pure fee-for-service model (if a doctor runs ten tests, they'll get paid ten times) to value-based care, where we incentivize quality care at efficient costs.
Despite fears that QPP would become “the next healthcare.gov,” our incredible team built the service using open-source components, launched the platform before our required start date, maintained 100% uptime, and supported hundreds of vendors who submitted millions of API requests. Back-of-the-envelope calculations indicate that our novel approach saved at least $100 million in implementation costs for QPP. It was the greatest honor of my life so far to work with such incredible federal public servants on such important goals.
For comparison, over 94% of federal IT projects are either over budget or behind schedule, despite the fact that we spend $86 billion on them, and 40% of projects are completely scrapped or abandoned.
Technical Work
As the Chief Technology Officer, I helped build and lead a team to tackle the challenge of gathering data from 2 million providers and groups, calculating risk scores and outcome metrics for 34 million beneficiaries, adjusting provider's payments based on billions of claims, and allowing all interested parties to login and understand their data in an intuitive user interface.
As CTO, I led our technology strategy, architecture, implementation, and hiring/contracting efforts. We built a technology team of over 200 people, including a rotating cast of 26 other US Digital Service teammates. We used human-centered design to drive our product roadmap, made many of our products open source, and ran a crowdsourced “bug bounty” where the bug bounty team from Synack noted that we were the fastest federal client they've ever worked with in fixing the security vulnerabilities their researchers identified. We integrated Okta’s identity-as-a-service platform with our legacy identity systems to dramatically improve our user login experience.
To fulfill our “API-first” strategy, we built an active community with external developers to submit to our API, with frequent communication through a public Google Group.
We moved our entire infrastructure to scalable AWS cloud resources and exceeded our very high targets for provider participation. Our most important feature was delivering users with real-time scoring and feedback on the complex scoring algorithms used on their claims, compared to a previous 3-year-long feedback cycle for Medicare metrics. All of our deployments were zero downtime.
I also spent several months working with and ensuring a smooth transition of the CTO role to my replacement. Although to be honest, I learned far more from the very wise David Koh, former director of engineering for OkCupid, than he learned from me during that period.
Recognition
During two years of government service, I was named one of the 100 most important people in federal health IT and one of the fifteen "Disruptors of the Year" in the $86 billion federal software industry.
According to one healthcare analytics firm, the QPP platform represents "a quantum leap forward compared to what CMS has done before.” One user called the help desk to tell us, “This is almost too easy–it’s scary!”
QPP received several awards, including the FedHealthIT Innovation Award and an API World Award.
Tech stack
The following tools were used building the QPP platform. The platform was generally designed as a variety of separate services often run by separate teams, communicating mostly through RESTful HTTPS APIs.
The user interfaces were built in JavaScript (Angular 4), with visualizations mostly in D3.js. Front-end services are Node.js. ETL is written in Python, connecting to databases in RDS for MySQL. We also used DynamoDB for session management. Kong was the backbone of our API gateway. Due to the hard work of many talented people, we also were able to clear a variety of hurdles to connect directly to externally-managed databases in CMS’s existing Teradata, Oracle, and mainframe (!) data warehouses. Analysis is run in Python and SQL, as well as Apache Spark, with workflow management by Apache Nifi. APIs are publicly documented in Swagger as well as our own narrative API overviews. We integrated Okta’s identity-as-a-service platform with a legacy login system at CMS. Akamai is used for content distribution and DDoS protection.
Web analytics were provided by Google Analytics, NewRelic, and ChartBeat. Other DevOps tools we relied on heavily include Splunk for logging & alerting, PagerDuty for alert escalation, Nessus vulnerability scanning, SonarQube & various linters for static code analysis, CloudFormation for configuration management, AWS Trusted Advisor for resource use & security monitoring, Cloudbees Jenkins for continuous integration and Selenium for browser-based automated testing. We had a quarterly “bug bounty” with the crowd-sourced security team from Synack for additional red-team penetration testing. We used Docker containers to improve the consistency of our environments (especially local development matching production), but we have yet to move to full Docker clusters, partly due to security compliance constraints in the federal sector. We used hand-rolled feature flags for feature management but were transitioning to LaunchDarkly when I left.
Infrastructure is pretty basic AWS resources: EC2, S3, ELBs/ALBs, KMS, auto scaling groups, VPCs, and some experimentation with Lambda and SQS, as well as RDS and DynamoDB as previously mentioned.
Despite some difficulty securing compliance approvals, we pushed very hard to acquire basic collaboration tools on our project, such as JIRA, Confluence, Github, and HipChat.
Gallery
Two videos of our live software platform are below. (They’re pretty deep in the weeds, since they’re designed as tutorials for healthcare providers using the system.)
Here’s a video tutorial of one of our APIs. According to one developer we interviewed about her experience, “Seeing the swagger docs was psychologically amazing... that’s when I knew things were going to be ok.”
For comparison, here’s a video showing the user interface of one of the previous measurement programs that we replaced.