Microsoft 365 App Compliance: what eight years taught us

For eight years, Claranet has delivered auditing as part of the Microsoft 365 App Compliance Programme: the framework behind the 'certified' status you see awarded to apps in the Microsoft marketplace. We designed the standard with Microsoft in 2018 and have delivered it ever since, certifying hundreds of applications, add-ins, and integrations a year. 

Here are five lessons from building and running it, for anyone responsible for a security or assurance programme of their own. 

1. A credible standard is more than a penetration test

For most compliance programmes, the penetration test is treated as the proof. We designed this one around a compliance framework instead. When we started in 2018, the App Defence Alliance did not exist. Microsoft and Claranet built the standard on established frameworks such as ISO 27001, PCI-DSS, and NIST CSF, combined with our own cyber and governance, risk, and compliance (GRC) expertise, to address a specific risk: protecting Microsoft customer data from weak integrations with independent software vendors (ISVs) looking to use or extend Microsoft 365. 

The pen test is closer to the final 20% of the work. The rest is a compliance framework covering technical and procedural controls, business continuity and risk management, and data handling and privacy controls, including GDPR. That can amount to 70 controls, depending on the hosting environment. 

Since then, the Alliance, Slack, and other vendors with ISV marketplaces have emerged with a similar intent: giving customers confidence that third-party applications meet an appropriate security bar. Where a vendor already holds ISO 27001, SOC 2, or FedRAMP, we map those across so they count towards the total. Every control demonstrates one thing: that the application meets the security standard for protecting the Microsoft data held within it. 

2. Scaling the programme is a people problem first

As the volume of apps grew, the real constraint was people: enough skilled assessors to apply the standard consistently. We built a training approach to solve it. Every new associate gets three days of intensive onsite training, then structured buddying and shadowing until they can deliver on their own, which gets a new analyst contributing in weeks rather than months. 

Growth also needs clearer ownership. We moved from a flat team to defined roles, from associates through to consultants and team lead, each with a clear remit across scoping, delivery, and quality assurance. 

3. The standard has to move with the technology

A good starting point is not enough. The original standard was built on sound technology, compliance, and data protection principles, but applying it at global scale is different. The ISV ecosystem stretches across regions and cultures, with each contributor bringing different operating models, regulatory expectations, and levels of security maturity. 

The programme has to stay vigilant and pragmatic. The goal is to secure connections to an acceptable level for Microsoft, while avoiding controls so onerous that vendors can never realistically achieve acceptable risk mitigation. That means keeping the certification current, investing in learning and awareness, staying close to emerging risks, and working with Microsoft and its partners as the landscape changes. Underlying this is the deep and trusted working relationship we have developed with Microsoft over eight years. Open communication, shared expertise and a strong understanding of Microsoft’s priorities enable us to work as one collaborative team, continually evolving the programme while supporting customers in achieving their security, compliance and business aims. 

4. One process has to flex from a dozen people to thousands

The vendors we assess range from well over 5,000 people to three-person teams, each with different pressures, experience, and resources. It taught us not to assume knowledge, to adapt to what is in front of us, and to collaborate pragmatically. 

We identify complexity early, during scoping, then match the right people and the right amount of time to each engagement. Some need genuine hand-holding, others are more independent. Reading that correctly up front is what keeps quality consistent. 

5. Responsiveness is part of the service

Security assessment has a reputation for being slow. At this scale, it can't be. We keep the programme moving with dedicated coordination, and give priority applications the extra support they need. We recently shifted the assessment process to live calls and screensharing, which lets us support vendors more responsively and move them to an outcome quicker. 

For vendors weighing up certification, we built a gap analysis service and follow-on consultancy, so they understand what's required before they commit. 

Claranet has been a valued partner in helping us build and evolve the Microsoft 365 App Compliance Program. Their combination of technical expertise, rigorous assessment and practical support helps Independent Software Vendors strengthen their security practices and gives customers greater confidence in the apps they use. Our collaboration demonstrates the importance of keeping security standards both credible and achievable as technology evolves

Tony Balkan, Compliance Program Manager, Microsoft

What eight years shows

Running a single security programme for eight years, at this volume and to this standard, is not something many security organisations can point to. It shows what we do well: designing a security standard, building the team to deliver it, and keeping both fit for purpose as the technology and the demand change. 

If you're a larger organisation that needs a security programme or framework built and run at scale, that's the experience we bring.