Democratizing the Bazaarvoice API

This post is a transcript from John Banning’s presentation at
The Austin Homegrown API Meetup in September 2019.

My name is John Banning, I’m going to talk about myself in a second, but I didn’t change the title, I’m at Bazaarvoice and this is the title of my presentation. So who am I? A senior developer advocate of Bazaarvoice. I’ve been there for five years.

So, hopefully most of you know what developer advocates and developer evangelists do in a typical job. I work with the API at Bazaarvoice. We handle escalation points and support. We write tutorials and documentation for the developer portal. So developer.bazaarvoice.com, my coworker Frederick and I own all of that content. We manage API keys and we also solicit feedback from our users on the API and drive new products and new features.

We also monitor quotas, and the rate limits on the API. Hopefully people know something about Bazaarvoice. Can I just get a show of hands? Has anybody heard of Bazaarvoice and know what they do. Really? Okay, that’s great.

So Bazaarvoice was founded in 2005. Primarily they are a ratings and review platform for anything you buy online. You read the ratings and reviews before you buy them. This is the marketing spiel I was told to put in my slides. The clear market leader and providing the world’s top retailers. We have over 6,000 clients and a third of them, roughly a third are on the API or API-exclusive. Everyone uses our API if you’re a hosted display client or an API client, Enterprise client, but primarily a third of them we consider Enterprise and are API-only.

Some of the traffic numbers we hit, we have 13 million requests during Black Friday that hit our origin server, so it’s pretty significant. We provide 3,700 requests per minute. We satisfy those and some of the bigger clients include Home Depot, you might’ve gone there to buy some lumber, Walmart, you might’ve gone there to buy whatever, Best Buy or electronics stuff. Those are just three of some of the top ones and we also have, we’re also global at the same time.

So there comes to the question of, “how do we satisfy all these requests, because it’s a significant number of requests?” We use Apigee.

Apigee was recently acquired by Google or Alphabet a couple of years ago or so. They’re our third-party API gateway, our CDN and proxy to our API requests. These are just some of the APIs that more public-facing. What else about Apigee? All of these, our APIs are not RESTful

We have some that are more considered bulk APIs, but Apigee handles everything. Rate limits, again, our CDN because we can piggyback off global, Google’s global kind of network. It’s been a good solution. We recently migrated about two years ago from Mashery to Apigee. If anybody has questions about Apigee or why we’re using them, feel free to talk about that.

So I’m on what is now known as the app API edge team. We were API infrastructure (APII) and it consists of QA developers, developer advocates, and also product owners, and product managers. And at the time when we decided to democratize the API, we were responsible for the external APIs and the internal APIs. So really what this meant was we were a giant bottleneck for the whole company. If you want an API in the company, you came through us.

We had to set it up, set up the rules, kind of learned the logic and it was a bottleneck. People couldn’t get their work done in a timely manner. So we came up with this idea. This is just a quick overview of what we owned and decided not to own. So this is kind of an overview of some of the APIs. The green and blue were the ones we claimed ownership and the red and yellow were the ones that, we didn’t want to touch. Someone else needed to own the logic and all that infrastructure, but it all fell in our laps.

So at the time we were supporting, these are the ones we did not want to support so much, the privacy API for GDPR, client response, personalization, product sentiments. We acquired a company called Ad Structure that does natural language processing to understand product sentiments on reviews. These were the ones that we didn’t really know the logic, how are they making, creating the response and the data to respond to the API.

We could set up the end points, but other than that we didn’t want to own it. So we decided to come up with this idea of democratizing the API. Let’s give all the developers for the different end points and the different domains of knowledge, access to Apigee and let them set things up. So allow anyone at BV to start, put up an API and work it through Apigee.

But we needed to put some rules in place first. We wanted to give every team the ability to own what they develop in a way, all the way to the API edge. So, we sat down and came up with the guiding principles. It’s this Holy Grail document we have on our confluence page, an internal documentation of 11 bullet points. And you can’t read them, they’re all too small. But I’ll go through most of them, of what all external and an internal API should adhere to.

So just like traffic rules in sporting game rules, you need to have rules and regulations and then contract that a common language that people speak. So I’m going to go through some of them and highlight them. So, it’s intended for Enterprise clients only. So as I mentioned earlier, Enterprise clients, those are the only ones accessible who have access to our API.

So those are some of the Walmart’s, the Best Buy’s, as I mentioned. And in higher ASF people who are spending more money with Bazaarvoice. So another rule is endpoints should be intuitive and be based on the intended use. So if you hit the reviews end point, you shouldn’t get back consumer information about that person. You should get a review, pretty intuitive. They should be easy to use and appropriate for the client developers, makes sense.

Features should be uniformly available across the platform if it’s available in the UI than the API should have it also.

So this was a big sticking point where our hosted display team, which two thirds of our clients use, had features that no API clients could use at the time. So one of them was called progressive submission. This came out, a couple of years ago probably, not quite on a hosted display. So we just released this, probably in the last couple of weeks on API. So it wasn’t feature parity at all.

Use well known and industry standards over in-house custom non-standards.

But wait, I’ve got more. They must be stable and reliable in the established service level agreements. Internal and external documentation is important as the application code itself. So when developing this we had the must, should may, I was kind of upset this didn’t have the must cause I had write documentation, so we’ll get back to that one. Provide a level of security consistence with their purpose.

So GDPR we put O off on top of that and client response, we put O off on top of that, but for the reviews’ endpoint, it’s publicly-facing hosted on most websites so we did not put it on that. Supported services. So our service team, our product team and our go to market team must be aware of it and support this API when it comes out.

Public APIs must follow a standard set of guidelines. So the interfaces are similar, so similar call pattern, I’ll show you that in a second. And traffic must be accessible through whatever log file system we’re using at the time.

Some additional considerations we needed to conform to hypertext transfer protocol (HTTP), use the correct methods. Everything’s not a GET anymore, which is great. You’ve post submissions, you post reviews, you don’t get them correct headers and we also returned correct statuses, 200s, 400s, and 500s. And this is what I was saying, the scheme should follow this, the host, the service name, version, and point and then any query string parameters.

Other considerations: Bazaarvoice public API should implement cores, any change management we’re deprecating. You’re either one of those states for any of the APIs, deprecated, sunsetted, active… or the latest version.

Security. We also go through a security audit so we don’t expose any public PII, public identifiable information. It’s personal. Limits and offset, every API endpoint has limits, offsets, pagination, localization, and then rate limiting. So we came up with this giant document.

In addition, we provide people who are going to use Apigee as their API gateway. I trained on how to Apigee. It’s an onboarding exercise and there’s documentation on how the shared workflows work. The proxy, the trace sessions in the CICB. Think of this as the training wheels for anyone coming up, we walk through the program with them. So then they’re off and running. So, now, anyone at Bazaarvoice.

Yeah, everybody gets an API. It’s easy, everybody gets an API check under your chairs. There’s not an API—I’m just kidding. So the stated benefits of when we started this was increased uniformity between Bazaarvoice public APIs, reduce ambiguity, and improve the experience of API consumers.

So, the winds from the democratization process where it freed up our team to do the work we need to do and not have to be the bottleneck anymore. We migrated to a new API key management system.

Rate-limiting was a big thing and Black Friday, as you can imagine when I mentioned all of the traffic we get, is now being fully worked on and supported and scoped. And we also have innovation coming up the next version of API. We can actually talk about it and not be full of shit anymore. This is coming! And we are also creating client facing tools that are helping expose a client’s API usage.

So beforehand when Best Buy, I wanted to know, “Hey, how many API calls did we go over a great limit last week?” I would have to say, “I don’t know. I have to go into the Kibana logs and look it up. Call me back in a week.” Now, we’re going to expose that and let them search it for themselves.

But we also had some issues, as I mentioned earlier, the documentation as an afterthought. It should be a must, we must document that—I’m looking at my boss Ryan. And we also had some inconsistent parameters where, one parameter was hidden equals true.

Another API might just say hidden. It’s not consistent, so it’s a bad user experience. I think we need to consider the developer experience much more.

So these are my suggested improvements. Whoever did the mock server talk a couple of weeks ago or a couple months ago with this last one, I’m interested in that. Setting up a mock servers to have that used, it was Stoplight, wasn’t it?

Then also we need to go back and do work to retrofit our APIs. But the mock server, you get it up and running and your developer experiences. I can test this thing day one when they set it up.

View the video from John’s presentation

Want to learn more about APIs?
Join us for our next meetup!