Role of product manager in Fairness-aware Machine Learning (ML) product

Human biases are well known from the historical events to current day-to-day activities. From the past few years, the inherent biases presented in society are finding their ways into the artificial intelligence (AI) systems amplifying such biases. With the explosion of data, the AI/ML systems are becoming increasingly powerful, taking important and sensitive decisions affecting millions. There are several applications in which AI/ML systems have surpassed human performance. On the other hand, there are case studies like COMPAS, Amazon’s earlier hiring algorithm where the system produced undesirable outputs.

Fairness means the decisions made by the AI/ML systems are not biased due to any sensitive information such as race, gender, age, disability status etc. Eventhough fairness can be measured through some metrics, they are not exhaustive and cannot be satisfied simultaneously in all situations. One way to define unfair behaviour by AI/ML systems is by its harm or impact on people. Some articles on fairness in AI/ML [1,2] help us to understand key concepts and how to apply these in our systems.

Following fairness in any ML product gives us a better product at the end of the day serving a broader population. It provides competitive advantage and good branding for the product. The social impact such products create is huge. It creates inherent responsibility as a product team to follow ethics while building any product. There are legal norms and policies such as the European Union’s General Data Protection Regulation (GDPR) and California Consumer Privacy Act (CCPA) imposing not only privacy rights but also on fairness, accountability, confidentiality and transparency in AI/ML systems. Adopting fairness enhances user’s trust and long-term engagement with the product.

Bias can creep into any part of the ML pipeline from an unrepresented dataset to learnt model and beyond. Errors that result from these biased systems can impact some users disproportionately more than others. It is important for the product team to be mindful of adopting fairness while designing an end-to-end ML pipeline.

Any ML product life cycle follows design, data, model and application. At each stage of the product, the human bias element is added into the system. As a product manager, identifying the key opportunities of bias in the system and communicating them with the cross-functional team becomes important. Let us discuss some key opportunities for bias to creep into the system and best practices for handling fairness in each stage of the ML life cycle.

  • Framing the problem: Bias can be created by ambiguous task definition. As a product manager, clearly defining the task along with the model’s intended and unintended effects and biases is the first step towards fairness. Involve diverse stakeholders and multiple perspectives while defining the task. Iterate it through multiple cycles to refine the task definition. Defining fairness at the early stage of the product always saves time, effort and cost.
  • Collecting the data: Data acquisition using different platforms, querying and filtering are some entry-points for data bias. Apart from the above, societal bias and skewed samples corrupt the data sources. Thinking critically before collecting the data and choosing the right data source becomes extremely important. As a product manager, communication with the development team to check for biases in technology used to collect data, humans involved in data collection, sampling strategy, demographic parity, cultural context, source selection etc. ensure the collection process is fair and ethical. This will result in a fairness-aware data collection for the product.
  • Labelling and preprocessing data: Apart from data acquisition, labelling either by human or software introduces bias in the system. While preprocessing the data techniques like choosing the attributes, cleaning using software, discarding data, bucketing values, missing features, unexpected feature values, skewness in samples also introduce bias into the system. As a product manager, communication with the involved stakeholders about the labelling bias and preprocessing data ensure that data is fair and inclusive of the demographics.
  • Model building: Bias can creep into the algorithm using the model building assumptions, process of splitting the train or test sets, loss function definition etc. As a product manager, checking with the development team on the bias in the objective function, assumptions about the model can reduce unintended effects. Ensuring to add selected “fairness criteria” in the objective function helps in reducing bias.
  • Deployment and feedback: Fairness-aware testing using test data representative of all the demographics can be communicated to the team. Involving the diverse stakeholders can help to identify bias in the system. Continuously monitoring the match between training, test and instances in deployment helps to reduce the bias. As a product manager, checking whether the fairness metrics is giving reasonable results ensures good results. For example, checking the results using inclusive confusion matrix to isolate results for every group or sub-population of the demographics gives the actual picture of the ML model’s performance. It is also good to monitor the user report, user complaints and users interactions with the system. Auditing the system involving multiple stakeholders will also reduce bias in the product.

Practically ML-based models rarely operate with full perfection when applied to real-time data. If an issue comes in live product, it is important to check if it aligns with existing societal disadvantages and how it might impact the long/short term solution. Product teams owning data-driven platforms are continuously faced with decisions about data collection, maintenance and modelling. Following the three complementary steps in any organization, the algorithmic bias can be assessed.

  • Research and analysis — translating both existing research into the organizational context, as well as case studies into specific products
  • Develop process — to integrate into existing product and educating organization about it
  • External communities — exchange lessons learnt and ensure work done internally keeps up state-of-the-art

Developing methods alone will not be enough to address bias, it needs to be educated and shared to the organization as a whole. Conscious efforts need to be taken to educate, iterate on potential tools or shared framework to teams.

  • Aligning on priorities and inter-dependencies: Organizations have to agree on the trade-off between different demographics, stakeholders, biases and error in terms of algorithmic bias efforts.
  • Communicate the minimal viable product steps: Bias is easier to tackle with new products than existing products and processes. Unintended biases are much harder to eliminate if not looked at from the beginning. Addressing algorithmic bias in product development requires short term narrow steps with continual improvement path forward in a time-taking manner.
  • Education: Assessing and addressing bias needs to be part of the product team’s goal setting process. Rather than adding it as a special requirement, it can be educated along with general machine learning courses for internal audiences. Companies have to add algorithmic bias efforts as part of baseline expectation while building any product. Step-by-step approach is crucial with iterative feedback for handling algorithmic bias.

Fairness and accountability go hand-in-hand in any ML product. Accountability means the possibility to identify and assign responsibility for a decision made by the AI system. As a product manager, identifying the bias at each stage of product life cycle. Assigning ownership within the team and between teams for algorithmic accountability helps to mitigate bias in the product.

Some best practices to be checked by the product team are:

  • Following fairness in each step of ML product life cycle
  • Creating and monitoring fairness metrics to assess fairness in ML product
  • Designing UX/UI with fairness aware concepts, like providing better messages from the product to the customer
  • Using Fairness tools like FairML, Lime, AI Fairness 360 (IBM), What-If-tool by Google, SHAP, Fairlearn by Microsoft across cross-functional teams for uniformity to mitigating bias in the product
  • Creating a checklist specific to domain and product with questions related to fairness across organization for cross-functional team to follow
  • Using custom made fairness-aware datasheets for datasets, model cards for model reporting, auditing, dash boarding, case studies, possible types of biases and outcomes expected for every product and communicating the same across cross-functional team

The following process can be followed for the product life cycle to ensure fairness and can be repeated for every new change in the system. Some sample questions that can be asked to the teams involved at every step in the process are also provided.

  • Identify product goals: Be specific with the product goals. Some questions to be asked are: What are you trying to achieve in your product? For what population of people? What metrics are you tracking?
  • Get the right people in the room: Different domains require different expertise and decision makers to be involved. Internal people include product teams, legal, policy, user research, design, social scientist, domain experts, machine learning experts. External people include academics, consultants, advocacy groups, government agencies etc.
  • Identify stakeholders: Each product might involve multiple stakeholders. Some questions to ask here: who has a stake in the product? Who might be harmed? And how?
  • Select a fairness approach: Identifying the fairness approach specific to your product domain and fitting company values/goals is important. Some questions to ask are: what type of fairness? At what point? What distribution?
  • Analyse and evaluate your system: Take the complete system along with people, technology and processes. Break the large system into smaller components. Analyze each component to understand the decision made and their impact and how well it matches the selected fairness approach.
  • Mitigate issues: As a product manager, decide if you need to change your design, data or metrics. Consider all types of intervention with chosen balancing metrics and how it impacts the product and customers.
  • Monitor continuously and escalation plans: Build monitoring and testing for all metrics you are tracking. Develop response and escalation plans for the product or at each stage of product life cycle. Some questions to ask are: How to respond when some issue happens? What blocks product launch? Who decides what to be done?
  • Auditing and transparency: Important to consider who else needs visibility into the product, process and system. Some questions to be asked here are: Do you need to prove that your system meets regulations? Do you want outside experts to certify your system? Do users need to understand fairness in the system?

There are practical challenges even after considering the above factors. Different stakeholders can have different perspectives on fairness. For building/designing any AI/ML product, someone must decide on the fairness definition/criteria/metrics and it will depend on the product, company, laws, geography, culture etc. It is always good to start with better decision making rather than fixing things. But the luxury of doing it might not happen every time, since people join the organization at different times, so existing systems need fixation. Some major needs for industry practitioners related to fairness in machine learning are:

  • Fairness aware data collection and curation
  • Application- and domain- specific tools and resources
  • How to support fairness auditing given only partial demographics information
  • Useful and usable tools for fairness debugging
  • Standardized processes during ML life cycle related to fairness
  • Standardized definition of fairness for ML products


Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store