What you'll learn on this podcast episode
Does learning actually occur as a result of ethics and compliance training, or are employees just paying lip service when they take courses? How can you tell the difference? Today, the E&C community is focused on program impact and effectiveness rather than checking boxes—in part because regulators have made it clear that E&C programs must show impact from their activities. On the season 9 finale of the Principled Podcast, Susan Divers discusses how compliance teams can ensure they’re getting the right insights to improve their programs with Kristi Kevern, the senior managing director at Dell Technologies. Listen in as Kristi shares how her team collects and analyzes data to better manage and enhance Dell’s E&C program—particularly in the training area.
Where to stream
Be sure to subscribe to the Principled Podcast wherever you get your podcasts.
Guest: Kristi Kevern
Kristi Kevern is an innovative thought leader with 20+ years of experience in internal controls design, implementation, management, and assurance. At Dell Technologies, Kristi drives enterprise-wide risk management and governance activities, conceptualizes and implements global programs aimed at mitigating FCPA, AML, SOX, ESG and other key risks, turns findings into fixes with post-investigation remediation, and experiments with AI/ML for further prevention and insights using data. Prior to Dell, Kristi served as a founding member of the Coca-Cola Company’s Ethics Office, where she investigated allegations of fraud and served as ethics advisor to the credit union. As a former Big 4 manager at PricewaterhouseCoopers LLP, Kristi led assurance and attestation engagements for Fortune 500+ clients. Kristi is a recipient of TRACE International's Innovation Award, and she has led Dell Technologies to an Ethisphere World’s Most Ethical Company designation 10 times. She is membership chair of the Conference Board’s Global Business Conduct Council and a frequent speaker at conferences and universities. Kristi graduated with honors from Auburn University and is a Certified Public Accountant residing in Austin, Texas.
Host: Susan Divers
Susan Divers is a senior advisor with LRN Corporation. In that capacity, Ms. Divers brings her 30+ years’ accomplishments and experience in the ethics and compliance area to LRN partners and colleagues. This expertise includes building state-of-the-art compliance programs infused with values, designing user-friendly means of engaging and informing employees, fostering an embedded culture of compliance and substantial subject matter expertise in anti-corruption, export controls, sanctions, and other key areas of compliance.
Prior to joining LRN, Mrs. Divers served as AECOM’s Assistant General for Global Ethics & Compliance and Chief Ethics & Compliance Officer. Under her leadership, AECOM’s ethics and compliance program garnered six external awards in recognition of its effectiveness and Mrs. Divers’ thought leadership in the ethics field. In 2011, Mrs. Divers received the AECOM CEO Award of Excellence, which recognized her work in advancing the company’s ethics and compliance program.
Mrs. Divers’ background includes more than thirty years’ experience practicing law in these areas. Before joining AECOM, she worked at SAIC and Lockheed Martin in the international compliance area. Prior to that, she was a partner with the DC office of Sonnenschein, Nath & Rosenthal. She also spent four years in London and is qualified as a Solicitor to the High Court of England and Wales, practicing in the international arena with the law firms of Theodore Goddard & Co. and Herbert Smith & Co. She also served as an attorney in the Office of the Legal Advisor at the Department of State and was a member of the U.S. delegation to the UN working on the first anti-corruption multilateral treaty initiative.
Mrs. Divers is a member of the DC Bar and a graduate of Trinity College, Washington D.C. and of the National Law Center of George Washington University. In 2011, 2012, 2013 and 2014 Ethisphere Magazine listed her as one the “Attorneys Who Matter” in the ethics & compliance area. She is a member of the Advisory Boards of the Rutgers University Center for Ethical Behavior and served as a member of the Board of Directors for the Institute for Practical Training from 2005-2008.
She resides in Northern Virginia and is a frequent speaker, writer and commentator on ethics and compliance topics. Mrs. Divers’ most recent publication is “Balancing Best Practices and Reality in Compliance,” published by Compliance Week in February 2015. In her spare time, she mentors veteran and university students and enjoys outdoor activities.
Principled Podcast transcription
Intro: Welcome to the Principled Podcast brought to you by LRN. The Principled Podcast brings together the collective wisdom on ethics, business and compliance, transformative stories of leadership and inspiring workplace culture. Listen in to discover valuable strategies from our community of business leaders and workplace change makers.
Susan Divers: Does learning actually occur as a result of ethics and compliance training, or are employees just paying lip service when they take online courses? How do you know? Today the ENC community is focused on program impact and effectiveness rather than just checking boxes, in part because regulators have made it crystal clear that ethics and compliance programs must show impact from their activities. That can be very hard in the training area. Hello, and welcome to another episode of LNS's Principled podcast. I'm your host, Susan Frank Divers, director of thought leadership and best practices at LRN.
Today I'm delighted to be joined by Kristi Kevern, senior managing director at Dell Technologies. Kristi is uniquely suited to discuss this topic as she drives enterprise risk management and governance activities at Dell. She conceptualizes and implements global programs aimed at mitigating key risks, and I love this, turns findings into fixes with post investigation remediation, and she in her spare time experiments with artificial intelligence and machine learning for further prevention and insights using data. Kristy, thanks so much for coming on the podcast.
Kristi Kevern: Susan, thank you so much for having me. It's always wonderful to speak with you.
Susan Divers: Well, let's dive in and if you can start with an overview of Dell's Innovations in the ENC area. I know there are many, and that you're a leader in innovation, what do you see as the biggest accomplishments of your ENC program in the past 17 years, and then maybe we can talk about some of the challenges?
Kristi Kevern: Absolutely. So this is the most exciting time in ethics and compliance at Dell Technologies. We are focusing on our digital transformation journey. And at Dell, our purpose is to create technologies that drive human progress. We do that for our customers and we also do that internally. We have the same drive for innovation within our ethics and compliance office, and we focus on two components. It's a simple and straightforward strategy. We want to have leading and proactive tone, as well as digital and predictive tools. And I'll talk about both of those focus areas. On the tone side, the greatest shift I've seen in my 17 years at Dell is the dedication, support, and commitment of business leadership. So we've all seen the challenges in getting leadership support and commitment, and I think what really is striking to me here at Dell is that our executives are engaged in every aspect of our program.
So whether it's making difficult disciplinary action decisions, whether it's standing up ethics committees and region, or allocating investment and resources to make our program better, and the tone is not just at the top, it's in the middle, it's around the edges, and this really truly provides a culture of integrity. And you mentioned the digital innovation. This is my spare time, but it's also my passion. And I think on the tool side, it really does help to be a technology company at our core. We have an advantage I think that other companies just don't have and aren't able to provide to the ethics and compliance program. So we've had the opportunity to experiment and we've had some really significant wins. The first win is the My Ethics app, and this is our mobile solution for team members to complete assigned training. And you mentioned the regulator expectations for increasingly using analytics and data, and we're experimenting in this as well.
We've actually invested in data science training for some of our investigators, some of our ethics investigators, and they've graduated from programs with proven skills in Python and related tools and separately, and this is a statistic that I absolutely love, 50% of my team have patent applications filed with the US Patent and Trademarks office, and this is pretty much as innovative as you can get. I'm sure you agree. So we have two patents filed currently, one for what we call Dell Image Analysis. It's a fraud prevention machine learning tool, and the tool analyzes images to detect falsification. The second patent pending tool is what we call corruption and risk detection system or cards for short. And it's an algorithm that we've embedded into our business systems and processes designed to detect public customer bribery.
Susan Divers: Kristi, that was a joke about your spare time. I know that you do a lot, but I really do find what you've been doing in the AI and machine learning area, just astonishing and the fact that you've got that number of patents out there. Wow, I hope you're getting enough sleep. At any rate, let's talk about challenges now. If you can tell us about some of those, because everybody has them, of course.
Kristi Kevern: Absolutely. Absolutely. I think the greatest challenge for us, and I think this probably really resonates well with the audience here, and it's remaining current with the increasingly vast and complex regulations that require compliance. And so it's keeping up with the volume of regulations, but also keeping up with the evolving expectations that we have as compliance professionals, not just in data analytics, but in a number of other areas inclusive of training effectiveness, which I know we're talking about more today.
Susan Divers: That makes total sense. And it's interesting, we had Stephane Bernardeau on a webinar recently that we did, and he's the French chief ethics and compliance officer for Auchan, a major European retailer, and he said exactly the same thing. I think it's a phenomena that everybody's struggling with, but let's turn to a good topic to zero in on, which is when you focus on ethics and compliance training at Dell, how do you determine if learning has occurred? What are you measuring when you ask that question and how does it differ from what I think is still unfortunately the standard in this area of how many completions course completions do you have?
Kristi Kevern: Absolutely, and that's a great question and I think that we've really grappled with that over the years. As compliance training professionals, how do you in fact determine whether learning has occurred? I think it's not just looking at whether or not training has completed. You need to determine whether the training that you're providing is in fact digested and effective. And I think that's the most important objective for us. It's not just a check the box exercise, and it goes back to what you measure is what matters. If you only measure the number of courses completed, then that's what you are implying is the end goal. And I'm not saying that completion does not matter.
And at Dell we do achieve 100% completion for over 130,000 team members, which I know is astonishing. And every year we hear from our audit committee and other stakeholders how impressive that is, but we don't just stop there. What we look at in addition to that metric is what is some insights, some data insights that can point to whether or not learning has actually occurred?
Susan Divers: That is incredibly impressive. I don't know of too many companies, especially of your size that can make that statement. So let's talk about sort of how you get there, because clearly that's very successful. Let's start out with how do you plan your training curriculum?
Kristi Kevern: Well, for 130,000 team members, we have a multidisciplinary approach. So it's not just a one size fits all. We want to make sure that we have a tiered approach and we layer general training with heightened awareness as well as maximum awareness. So each of the team members really do have a tailored and customized strategy. We start with new hires, all new hires, including acquired team members, complete not just online courses within the first 45 days, but they are also engaged in role-based education respective to their particular job responsibility at Dell.
We layer that on with annual training. So we have a 100-day window in which team members complete required general and targeted courses. And in the interim, in between the new hire and the annual training, we supplement with targeted microburst learnings. And these microburst learnings are developed following investigations or other challenges that we face, and we disseminate the training within a short period of time to higher risk geographies or functions. And this is how we meet the goal of closer to real time training.
Susan Divers: Again, that's terribly impressive and it maps very precisely to the Department of Justice guidance on effective ethics and compliance programs because you're using real examples, even probably suitably sanitized, but as examples to drive awareness and as you say, it's both relevant and occurs within a short space of time after you've had an issue. So you mentioned identifying target audiences and then mapping the training to the specific audience. How do you actually do that?
Kristi Kevern: So to achieve not just risk-based, but also role-based training, we look at the risk assessment output. So our risk assessment, our component of ERM is the main driver for our courses and our content. We look at investigation trends, like I'd mentioned. We look at regulatory developments, so that's all of the input into the what, but the who is just as important. We match the content to roles using a matrixed approach. And we found this to be successful over the years. And the reason being, when we have such a vast population of team members, we're not just able to look at one component, one side of the coin. We want to look at where a team member sits within the functions. So for example, we look at whether a team member rolls up to the chief marketing officer, but we also look at Workday job families.
We have over 250 Workday job families at Dell, and we'll look at the job families to make sure that that team member is in a particular job family that may have a need to know for the information. So for example, a team member may not be sitting in the chief marketing organization, but have a marketing related job family and have a need to know for privacy notice and consent rules.
And we also want to ensure that content is fresh, so content is updated every year to ensure we address the most current guidance and expectations. So for example, in 2020, we created a new standing Strong Together course. In 2021, we had content that addressed COVID precautions, and this year we are rolling out content that addresses artificial intelligence governance when the content is relevant and addresses the current environment, this resonates best with team members.
Susan Divers: That makes perfect sense. And again, it illustrates how adaptable and flexible your program is because you can adapt to changes for example, such as Russia sanctions or concerns about AI. I could talk about this all day, but let's talk about use of mobile devices. You mentioned My Ethics app and I'd love to hear more about that.
Kristi Kevern: Absolutely. This is probably one of the greatest wins that we've had in the past couple years in our training program, and it really is truly designed to give team members flexibility in how and when they take their assigned training. We developed the My Ethics app in partnership with LRN, and this is designed to bring ethics and compliance tools and knowledge to team members wherever they are located, located and on any device. And what's really interesting, Susan, is that the app is embedded in our mobile repository. So at Dell, if team members have access to mobile email, they have ease of access to complete assigned training to navigate the code of conduct, to speak up to access policies and managers can also use the app to track and encourage progress of training. So it truly is keeping the team member front and center in what can we do to make the team members easily access all the tools necessary for the role.
Susan Divers: Well, and that's illustrative of another best practice we see strongly emerging, particularly in the wake of the pandemic, which is to make ethics and compliance programs much more user-friendly and employee focused. And also, you failed to mention that you rolled out the enhanced capabilities for the app during the pandemic, but I'll note that for the record because it was a way to further help employees keep abreast of ENC. So let's go back to a question that we sort of started out with and explore it in a bit more depth. And that question is how do you determine whether learning has or hasn't occurred? And then what do you do if it hasn't occurred according to your measurement? And I'd also like to talk about some of the research you've done on what is the ideal length of an online training course?
Kristi Kevern: Absolutely, absolutely. So we look at data insights primarily to determine whether or not learning has occurred. So I mentioned the consumption or the completion metric, which I think all of us here use. We've double clicked into some interesting data using the LRN Reveal dashboard, there are five components that I look at. The first one is what I call focused engagement, and this is the amount of time team members spend on a particular course, and it's compared to what we know as the true course duration. My team has completed all of the courses within our curriculum and we know it takes 20 minutes versus 25 minutes for a particular course. And we also have engagement along the way, so we know team members are not just simply opening the course and having it run in the background. So that's the first element focused engagement.
What is the time spent on a particular course? We also look at knowledge retention. So it's the number of attempts to pass a particular knowledge check in its entirety and at a per question level, and this is important, so you can determine whether or not the content leading up to the question is digestible, is understandable, and we can also see where team members are challenged in absorbing a particularly complex topic. We use this to not just determine whether team members have retained knowledge at the course, but we'll also go back at a later point in time and test that particular question or maybe supplement a communication that gives you a better understanding of that particular topic. The third item we look at is learner feedback, and it's important to provide the best possible learning experience, Susan, as you know, and that helps aid retention as well.
This past annual cycle, we had close to 12,000 team members provide feedback on overall design of the course on the digestibility of the content and relevance to role. So this is how we get that feedback from team members. Did we hit all of those key points? What's really exciting about the feedback from our learners is our score is pretty high. Overall our score is a 4.8 out of five, and that's for the 19 courses that we provided last year to the full audience. The fourth data element that we look at is language usage, and that's essential to ensure maximum comprehension. We look at what team members completed the course in English, and that is Susan. The majority, we see probably about 95% of our team members complete courses in English, but we do have seven other core languages, and we look at those core languages to see what are team members completing to ensure that they fully understand the content in a particular country or locale.
And then the most important element is that qualitative feedback. That's the fifth item. Each year we gather and review thousands of text comments, so open field comments that are aimed to make our courses better. The way I look at it is if we ask for that feedback and then we better show the team members that we've listened and we actively respond to the feedback. And one of the feedback elements that I am most proud of over the years is the feedback that we received from our true abilities team members. They let us know that we needed to make our courses more friendly for team members that needed accessibility considerations. And so now all of our courses meet the web content accessibility guidelines 2.1 AA standard, and that is a leading practice. Thanks to the feedback from our team members, we're able to make our courses better and better each year.
Susan Divers: Well, that's not only a picture of best practices in terms of training programs, but that's an example of good moral leadership in terms of responding to the needs of people and meeting them where they are. We're certainly proud to be your partner. I do want to ask one last question before we wrap up, given the time, and that's that I believe you came upon sort of the golden mean for how long a training course ought to be based on employee comments, and that's a question we get asked pretty frequently by other clients. So if you could talk just a little bit about that.
Kristi Kevern: Absolutely, absolutely. So I mentioned the overall score, the rating rather, a 4.8 out of 5 majority of team members gave us 4 and 5s. We had a handful of team members, close to 300 team members who gave us very poor ratings. And it turned out it was all linked to the same course. We had a course that was relatively dry it talked about patents and information protection concerning patent filings and the pro related processes, and the course was 45 minutes. The team members told us it's too long and it contains duplicative content. I think the team members might have felt slightly offended for the repeat language and the repeat content in that particular course, and they wanted us to know it.
On the flip side, the courses that were shorter, the courses around 10 to 15 minutes, maximum of 20 minutes, those are the courses that receive the highest feedback. We also know that team members prefer highly interactive courses. So what I say is the ideal length is no longer than 20 minutes. If we're looking at a word count, it's about 7,500 words and it's the same amount of time to read and digest a blog post or an article. And what we found is if we go any longer than that, that's where we have risk of disengagement. And if you have disengagement, you're simply not getting the right message and you're simply not allowing the learners to absorb the message that you are intending to deliver. Now, we still do have courses that are at that 45, 35 minute length, and we are actively shortening to ensure we can get down to what we think is that golden timeframe for our courses.
Susan Divers: That's a great example and it's very helpful, particularly in the approach you took to understanding what laid behind the concerns that the employees were raising. Kristi, this has been absolutely inspiring as well as enlightening, but we're running out of time. It's my pleasure to have hosted you today, and I hope you'll come back again in the future.
Kristi Kevern: Thanks so much, Susan. I've really enjoyed it and appreciate it.
Susan Divers: My name is Susan Divers and I want to thank everyone for tuning in to the Principled Podcast by LRN.
Outro: We hope you enjoyed this episode. The Principled Podcast is brought to you by LRN. At LRN, our mission is to inspire principle performance in global organizations by helping them foster winning ethical cultures, rooted in sustainable values. Please visit us at LRN.com to learn more. And if you enjoyed this episode, subscribe to our podcast on Apple Podcasts, Stitcher, Google Podcasts, or wherever you listen. And don't forget to leave us a review.
Be sure to subscribe to the Principled Podcast wherever you get your podcasts.
