Netsertive's Herb Brittner - Going Beyond The Cookie Cutter

SPEAKERS
Liran Haimovitch, Herb Brittner

Liran Haimovitch 00:02
Welcome to The Production - First Mindset. A podcast where we discuss the world of building code from the lab all the way to production. We explore the tactics, methodologies, and metrics used to drive real customer value by the engineering leaders actually doing it. I'm your host, Liran Haimovitch, CTO and Co-founder. Today, we're going to be discussing running Java and PHP side by side in modern cloud environments. With us is Herb Brittner, Senior Director of Technology at Netsertive. Thank you for joining us, and welcome to the show.

Herb Brittner 00:43
Thank you for having me.

Liran Haimovitch 00:44
So, Herb, what can you tell us about yourself?

Herb Brittner 00:47
I am Senior Director of Technology at Netsertive. I lead all software development product and IT teams. I have a background as a software engineer, and I'm very passionate about solving problems for customers.

Liran Haimovitch 00:59
So, speaking of solving problems for customers, what kind of problems does Netsertive solve for its customers?

Herb Brittner 01:05
Netsertive is the digital marketing solution trusted by multi-location businesses across the country. We were founded in 2009, and have over 1500 retailers, franchises, auto dealers, and media companies that use our award-winning technology and team to run localized digital advertising, search engine optimization, and Web Solutions at scale.

Liran Haimovitch 01:27
So, what kind of challenges do multi-location businesses face when it comes to digital advertising?

Herb Brittner 01:34
There's actually a lot. They face consistency, they face budgeting problems, they want to make sure that everybody has the same message across the network. And then, when you're creating the campaign, you want to be able to put your budget in certain places, some markets require more than others. And then, at the end of the campaign, you want to be able to compare them. How did everybody do? Who were my high performers, my low performers, what can I do to help get everybody on the same path?

Liran Haimovitch 02:02
So in a way, you're kind of trying to run lots of community campaigns, that has some things in common, but some things are differentiated between locations.

Herb Brittner 02:11
Yeah, so the biggest challenge we have is bringing all that together, and keeping the localized feel, and making sure everything's not cookie cutter along the way.

Liran Haimovitch 02:22
And how long has Netsertive been around?

Herb Brittner 02:24
We started in 2009, I actually joined as a software engineer in 2012. So not quite the beginning, but I'm going on 10 years in October.

Liran Haimovitch 02:33
Cool. So the company's been around for a while. And I'm sure you have many generations of technology, many pieces of code written over the years. Kind of, what does the service looks like on the inside?

Herb Brittner 02:44
Yeah, we are using a combination of mostly Java and PHP applications. Our newer websites are written in React. Everything's running in Kubernetes in the cloud, we have a lot of database optimizations and database ETLs as well that run every day, we pull a lot of data into our data warehouse for reporting and analysis on a daily basis. So, there's a lot of different stuff in there, from MySQL to Postgres and some snowflake as well.

Liran Haimovitch 03:14
So what kind of data are you pulling into those databases, and where's the data coming from?

Herb Brittner 03:18
There's kind of two sides to the data in our databases, the first would be the configuration side. So, setting up all the campaign's, all the ad groups, and all the ads, and keywords that go along with them. On the other side, there's the performance data. So, that's the reporting we get from people like Google and Microsoft, and Facebook and the trade desk. We pull that all together so that we can report on everything at different granularities to our customers.

Liran Haimovitch 03:46
And then when you're putting all this data and you're customizing all of those reports. How much of the operation of the services of the customization of the ads is done by Netsertive employees, versus how much of it is being done by the end customers themselves?

Herb Brittner 04:01
We have some of both, we have some self-serve customers in certain areas that can log in and make some changes to their campaigns themselves. And probably a majority of our customers are more of the…where they work with our account managers to determine the media strategies and our team will actually use the tool?

Liran Haimovitch 04:21
And how do you differentiate the product offering and the engineering processes between those two kinds of folders?

Herb Brittner 04:29
That kind of goes into a broader 'How do you determine what you work on?' in general, and at the end of the day, you have to align your product roadmap with your company's goals. And those goals can change year to year, quarter to quarter maybe. So just keeping a pulse on what's important to you, and plan accordingly.

Liran Haimovitch 04:32
Speaking of which, what's important to you? Over the past few years, you've migrated most of your infrastructure to Kubernetes. In fact, you're running quite a lot of PHP servers on Kubernetes. And that's fairly unique in the industry.

Herb Brittner 05:04
I suppose it is. PHP is what we started using back in the day. PHP for websites and APIs and Java for automation applications that go around it. As we've grown, we've started moving more of those APIs into Java, different frameworks in Java, like Spring Boot, we still have some of the PHP APIs around. We aren't really writing any new applications in PHP. But until we get all of them migrated, we'll have them around. And if there are small changes, we need to make here and there we'll do that

Liran Haimovitch 05:38
Over the past few years, what kind of changes? What kind of efforts have you made to make your PHP servers easier to maintain?

Herb Brittner 05:45
I think you hit the nail on the head earlier, we use Docker and Kubernetes to spin up and down test environments, deploy different version of containers and applications, and APIs, essentially, using the microservice framework. Everything's kind of separated into their different duties. So, when you're talking HTTP at API, CRUD operations, it doesn't really make much difference if you're using PHP or Java from the API layer. So that's definitely helped us out and then deploying to production is the same, it's just deploying that container to the production cluster.

Liran Haimovitch 06:19
Yeah, containers are great to kind of isolate away how the application is being built and packaged and what technologies are being used. At the end of the day, you just have an image that trends, an HTTP server, and it's fairly opaque, what's happening on the inside.

Herb Brittner 06:37
Yeah, and that's a great way for things to kind of work as they work, you know, we don't have to upgrade everything in one fast swoop. We can do things as we get the capacity to do them, and plan accordingly. It also lets developers kind of have a way to learn new things, try new technologies, and try new patterns over time as well. And as long as the contract doesn't change, yeah, it's pretty opaque, that doesn't make much difference.

Liran Haimovitch 07:04
Have you faced any unique challenges, trying to modernize those PHP environments? I mean, most Kubernetes users are using Java, or Golang, or node or ROS, and all of a sudden, you're running PHP. Are there any, you know, sharp ends to that approach?

Liran Haimovitch 07:22
And yet. you've kind of ended up saying, we're going to move as much as possible to Java. So kind of, why not stick with PHP?

Herb Brittner 07:22
It's been a little while since we actually did that work. But we invested in that with our DevOps team to streamline as much as possible around containerizing everything and getting things into Kubernetes. It's been a few years and there are definitely some road bumps along the way and different things that you run into with PHP specifically. But once you get them all figured out and automated. Today, it's not even a question I believe, nobody asked that. It just works.

Herb Brittner 08:00
It's actually a trend I've seen with developers, in general, it's hard to find anyone that wants to work with PHP these days. Those people exist, obviously. But more people want to do things like React and Node and Java. So, I think people are just getting more vocal about their preferences and the market is in such a place where they can find whatever they want to work on, and whatever language they want to work in. So trying to just keep modernizing our stack is a good way for us to stay competitive in the market.

Liran Haimovitch 08:31
That's a refreshingly honest answer. Yeah. Not saying that PHP is superior in anyway technologically, not that it is or isn't. But sometimes different factors come into the decision saying that my employees are happier rocking for Java, and my candidates are easier to hire. It is very uncensored, refreshing, I guess, to a certain degree, it's also interesting, because it's not necessarily because it's only foreigner to, you know, develop in PHP or Java, but people perceive it as more modern, more fun, more interesting. And the truth doesn't matter that much.

Herb Brittner 09:07
Right. I mean, there are a few things like, PHP has its quirks. Every language has its quirks. And you know, you've learned love and you get through it, and then you know, what you need to do going forward. But yeah, there's definitely enhancements coming to PHP, but it doesn't seem as quick to evolve as other languages. And when people learn languages, they generally start with a Java or a C sharp, maybe.

Liran Haimovitch 09:31
So why not go with C sharp or Node? Why did you decide to go with Java in spring for your new web API's?

Herb Brittner 09:40
We started with Java because we already did Java. And there's a lot of familiarity with Java and object-oriented programming and the libraries that spring boots specifically. It simplifies and you can kind of create a template of an API right off the bat. I mean, when you're creating a CRUD API, it's pretty standard. You're going to have your creates, you're going to have your updates, you're going to have your reads. And it kind of gives you that out of the box, you just have to fill in the unique parts of what that API needs to handle and not worry about the boilerplate. Yeah, faster development and happier developers because they don't have to focus on the mundane stuff. They're focused on the business logic and the uniqueness of what they're building.

Liran Haimovitch 10:22
What kind of challenges did you find around spinning up those Java servers, running them in Kubernetes, running them in production?

Herb Brittner 10:29
It was a lot of work just ahead of time to get things to a point where they could run successfully in Kubernetes. When you know, historically, you have things on the…and certain apps run at different times, you try to think of different ways to do that. Kubernetes has its own cron built-in. There are also other tools we use, like Argo for workflow management. So we can say, we're going to run this application to pull data. And then there are different stages that happen in the workflow after that. So figuring out all of those things, and how to modernize things is the most time-consuming part of that.

Liran Haimovitch 11:07
I have to admit, I had some bad experiences with the Kubernetes cron APIs. They're fairly cumbersome. And we found that we didn't have much observability into what was going on in there, kind of how did you fare with the Cron API?

Herb Brittner 11:22
Exactly. And that's why we moved to that, that Argo workflow manager instead of the Kubernetes Crons. Because, like time zones, actually were a problem, whenever the time zone shifted, and daylight savings took effect. Now everything's an hour off unless we updated all the different Helm charts to fix that. So you come up with your solution to fix it right away and then you build a longer-term solution, that hopefully resolves some of those problems.

Liran Haimovitch 11:50
So how are you using Argo? For workflow management for ETL? For Crohn's? What's your experience around that?

Herb Brittner 11:57
It's actually really positive. I really like that tool specifically, because it gives you the, like the tree view. So you can see the red or green status, you can see what stage of the workflow It's on. It's really similar to airflow. I know, that's another common one that we investigated when we made that decision. And we went with Argo at the end of the day because it was Kubernetes native. So it'll spin up and down containers and jobs and everything as it goes. And all of the observability was there, you could also see the logs. And you can see all the different runs that happened today, scheduling things for, you know, every day, every hour, every five minutes if you wanted to, like it's very flexible.

Liran Haimovitch 12:40
And what is it like troubleshooting? I mean, you're running all those ETL jobs, you're ingesting tons of data, you're-- I'm guessing you're doing various aggregations and fancy calculations around the data. So how do you monitor the data flows? How do you troubleshoot when things go wrong?

Herb Brittner 12:58
So Argos actually really helpful with that, because if something fails, it'll be read, and it puts some messages into our team chat. So, we'll know immediately if something happened. And then, we can dig into the logs after that to help resolve it. But I knock on wood, because the Argos handling the workflow, and it has different stages of 'I can go to the next step only if this succeeds, or if this fails I can stop'. It actually makes things pretty easy and it's very reliable, in my opinion so far.

Liran Haimovitch 13:34
That's great to hear. Now I'm wondering, you're running all these fancy new Java technologies, you're using Kubernetes, you're using Argo. You have Java Springwood APIs, you have Java ETL jobs. And then, you have those PHP servers side by side, the old stuff, nobody wants to work on, the legacy technology. What are the challenges? What are the unique elements of operating both of those generations side by side?

Herb Brittner 13:59
I think day-to-day, there are actually very few challenges, because all of these technologies built around them make it more seamless. So, we're not necessarily handling the PHP servers ourselves day to day, they spin up and down as needed. They're all load balanced in Kubernetes, automatically. And if we don't need to make any changes to them, they just keep doing what they're doing. So, it's actually pretty minimal. It's nice that the technology these days allows you to do that without as much overhead as you would have had in the past.

Liran Haimovitch 14:33
Yeah, I mean if you're getting the concept of pets versus cattle, right. As long as you have your service or cattle, as long as they can be managed as a group. You can relegate most of that day to day to machines, to the-- You know, the ship running in the day-to-day heavy lifting and just kind of sail on, focus on the business logic, on the value, on the future of the business.

Herb Brittner 14:57
Exactly. And that's the part that's exciting, right? Like I mentioned earlier, I'm really passionate about solving problems for customers. I'm not really passionate about troubleshooting, why PHP server doesn't want to turn on right? Or it doesn't want to work correctly. I mean, there are people that want to focus on that. But yeah, in the business framework, it just needs to work. And we need to be focusing on solving those problems, the right problems,

Liran Haimovitch 15:20
We're definitely seeing that the rise of platform as a service, the rise of Kubernetes of managed databases, are taking away much of the grind of day-to-day operations of setting the IT infrastructure. So, you can get container as a service, you can get databases as a service, you can get all the building blocks for application, so you can just focus on the code itself, and what it's going to do.

Herb Brittner 15:43
Yeah, I think that's the biggest improvement we've made, even in the last couple years, right? You don't have to worry about how to get everything set up and running locally anymore. There are basically services to do almost everything. So there's a learning curve, that you have to get over to figure out how to do that, how to troubleshoot things, but it's definitely nicer, and at least a faster development at the end of the day.

Liran Haimovitch 16:08
Yeah. Now, you've been at Netsertive for a decade now or you're going to be at the end of the year. So I have to ask, what's the most memorable bug from your time at Netsertive?

Herb Brittner 16:18
My most memorable bug, probably, likely didn't make it to production. But when I first joined the team, right? That's not fun. Oh, well, you know, try not to have any super memorable production bugs, but you know, things go wrong, and you have to fix them. But the one I remember... actually didn't make it to production. But it was a good learning experience for me and it's something that I kind of kept with me forever, is working with databases, and creating queries, you can really get into the mindset that, oh, it's just, it's easy. You're just selecting star from a table, right? And then, you get the more complex joins, and you get to more filtering, and all of those things, and you end up with a result set that's got this Cartesian product in there, where everything is multiplied by however many joins you pulled in, and all the numbers are wrong. And sometimes it can be hard to spot, because some things look correct. But maybe for this customer, they had a couple of different records, that ended up compiling on top of each other. So again, luckily, that wasn't one that made it into a production environment. But it was a good learning for me early on in my career. So always very careful writing queries after that, because even the simple queries can be, yeah, deceivingly complex.

Liran Haimovitch 17:43
Yeah, queries or code, you can get them wrong, especially if you don't know what data is going to be running against, especially if it's hard to predict data that's going to end up in production, that you're going to be running the queries against.

Herb Brittner 17:56
Yep. 100%. And then, of course, you know, memorable bugs, maybe things that we weren't in control of, are things when your cloud provider has an outage. What do you do? How can you debug? How can you get back up and running as fast as you can, or, in some cases, maybe data from Google is delayed, and they have a bug where they can't send reports out. Now we have to figure out how to backfill things whenever they do get things back up and running, how do we notify our customers that there's a delay there? And then even all of our automation after that writing will automate all of our bidding and budgeting based on reporting? And if it doesn't exist? We've got to wait.

Liran Haimovitch 18:37
Yeah, exactly. Herb, any last word for our audience?

Herb Brittner 18:41
I think the important thing about your technology stack is that you're solving the right problem. So the technology you use doesn't necessarily matter as much as the outcomes. And if you're using a legacy technology, and you want to modernize it, you've got to come up with a plan and a migration strategy to get there. And ideally, get there without trying to do everything at once. So, trying to do things in stages is incredibly important. It'll help you succeed, and it'll put a lot less stress on your team.

Liran Haimovitch 19:16
Makes perfect sense. So, it was great having you on the show. Thanks, everyone for listening in. And if you're looking to get some PHP timing or Java time, probably want to check out Netsertive.

Herb Brittner 19:27
Heck yeah. Happy to have you.

Liran Haimovitch 19:35
So, that's a wrap on another episode of The Production-First Mindset. Please remember to like, subscribe, and share this podcast. Let us know what you think of the show. And reach out to me on LinkedIn or Twitter at @productionfirst. Thanks again for joining us.