Behind all the simplicity and functionality of Stellar lies a complex and intricate C++ program, Stellar Core. Maintaining global consensus in near-real time across a global network of nodes is no small feat, and it doesn't happen by magic: it happens because of advanced programming designed and built by some of the world's smartest and most dedicated engineers. One such engineer, Jonathan Jove, joins us today to talk about the artful task of building and maintaining such a system. This is a two-part episode, so be sure and tune in next week to catch the second half of the show.
Jon Jove (00:00):
... somebody says something to you and asks you, "Hey, why did you make this decision?" and you're like, "Actually, I didn't have to make that decision, and I could've made a better one, and once I make that better decision, there's a whole world of opportunity there sitting beneath the surface." It's like a goldmine, but you're standing on top of dirt, and you can't see it right beneath your feet.
Tyler van der Hoeven (00:30):
Hello, Stellar community. Welcome to Episode 14 of the Stellar podcast. Today we are joined by Jon Jove, a member of SDF's Core team. Core, for those who may be unfamiliar, is the counterpart to Horizon. There is where all of the consensus happens. It's a C++ program that allows the nodes of the Stellar network to share this common ledger, ratify changes to it. It's the core of Stellar, that's why it's called that, and then most of the rest of us work with Horizon. Right? So, that's the ingested version of Core, taking the XDRs and decoding those into a JSON format.
Tyler van der Hoeven (01:14):
But behind the scenes, behind Horizon, there's this Core program, which I don't necessarily understand all that well. So, very excited to talk to Jon today and get a better sense of what goes into making this thing work. To get things started, though, we'll just start with a little bit of background. Maybe, Jon, if you can cover a little bit of your history, how you became a Core Stellar developer. I assume you were born somewhere, started out as an infant child, just like myself, but here you are, years later, as a Core developer. How in the world did that happen?
Jon Jove (01:51):
Yeah. So, a common misconception, I actually wasn't born. No. No, I'm kidding. I was, just like everybody here, I hope. Sometimes when I think about it, I feel like I've been programming since then. I've been programming since I was 10 years old and a little bit. My brother was programming, my older brother, when he was in high school, and when you're little you always want to do what your brother's doing, do what your sister's doing, whatever. So, I had him teach me a bit, and in a while the student had surpassed the teacher. Literally, my brother's a teacher nowadays.
Jon Jove (02:34):
When I joined SDF, I really didn't know anything about blockchain. I'd be lying if I said I did. My previous career or my previous job was in... I was a quantitative trader. That's what I used to do. So, when I joined SDF, it was kind of like, I'm looking for something new to do. I want to do something where I can write software that people are actually going to use, and it'd be great if it was something financial because I'm familiar with it. So, I was just looking around, looking for a job where I could be a C++ engineer doing that, and came across the SDF, and I met Jed and David, and I thought that this was going to be a really good experience for me, and it really turned out to be the right job, so I've been really happy doing this.
Tyler van der Hoeven (03:23):
That's super interesting. I too have an older brother who's a programmer, but he also was a musician, sort of, I guess. He played guitar, and I thought that was cool, and he told me I should play banjo, so that's how I followed my brother. I played banjo, which was a terrible decision. It's hard to make money as a banjo player, and if anyone ever asks me to play banjo, I will say no. But what is that like, trying to synchronize with something as important as Core? Do you guys have a very specific workflow? How do you break up the Core work? Is it broken up by CAP? How does that work?
Jon Jove (04:10):
So, there's a lot of different work going on on Core, and this is kind of a benefit for having a kind of small team. Stellar Core is this... It's a big piece of software. It has a lot of parts that are pretty clearly separated. There's the SEP module that actually runs Consensus, and there's the overlay module that runs the network, and there's the transaction processing system, which is where most... When you talk about a CAP, you're probably making changes there. There's other parts too. There's the data layer that actually talks to be database and makes sure that we have the right data that we're pulling up and writing back.
Jon Jove (04:49):
I'm not going to say that these are completely independent systems, but they're pretty separated, and this is kind of a boon because I can work on something... Lately, I've been working on this implementation for a different CAP, CAP 15, which is... I don't think it's the topic of today, but it's about fee bumps, and we've been working on doing this correctly for a long time, and I can just work on it and work on it and work on it, and make changes and changes and changes and changes as we're refining the implementation and sorting everything out, and I'm not really likely to conflict with anybody or block anybody or anything like that.
Tyler van der Hoeven (05:26):
So, let's talk a little bit about what a CAP is. So, we've talked about SEPs before on this show, the Stellar Ecosystem Proposals, but maybe we can talk a little bit about a Core Advancement Proposal. That's what a CAP is, right?
Jon Jove (05:39):
You've got it.
Tyler van der Hoeven (05:40):
All right. So, what is a CAP? How does it differ from a SEP, and ultimately, what is the purpose of having that differentiation?
Jon Jove (05:51):
So, at its core, a CAP is a... It's a protocol-level change to Stellar Core. You could think of Stellar Core as having two kinds of parts. It has parts that are just implementation, and this is, for example, the data layer. It doesn't matter how you write the data layer. As long as you talk to the database and pull in the right data, you're good to go. You could design it in any way you want as long as it has that property. On the other hand, a protocol-level change is something like what happens when you send a payment. Every single node in the network has to do it the exact same way. Nobody can disagree.
Jon Jove (06:31):
CAPs are in this domain. So, when we make a change to one of the subsystems, and it's not a protocol-level change, we don't write a CAP. We just think about it as a pure engineering problem. How can we make the implementation better? When we're writing a CAP, there's a lot more that goes into it. Part of it is the commitment, that you can't easily change it. Stellar Core contains the history of every version of the protocol, and so all the code to run the old protocol versions is in there too. So, you don't really ever get to go backwards in the sense of, oh, I want to remove this change. All you have to do is change it again. So, when we're making a protocol change, we're focused on getting the best possible properties that we can in the most surgical way possible.
Tyler van der Hoeven (07:27):
That's a good description. What are some of the reasonings behind making a change? Stellar exists. Isn't it perfect? Why don't we just leave it alone? You can all go home. You don't have to do any more work. Why do we need CAPs?
Jon Jove (07:43):
So, Stellar is great, but I wouldn't say it's perfect. So, for me, the first question is, does this need to be a CAP? That sounds like a stupid question. It's like, I suggested a CAP. Does it need to be a CAP? Well, I already suggested it, so maybe it should be. But it's actually a kind of profound question. You're basically asking the question, do we need to fix this in Stellar Core, or can we fix this by building on top of Stellar Core? That's the difference between a CAP and a SEP. When you're building a SEP, what you're kind of saying is, "I want some people who I associate with to do this in a standardized way." When you're building a CAP, what you're saying is, "Everybody, without exception, has to work this way. That's the only possible way to do this." This is a hugely more impactful change. You can ignore a SEP. You cannot ignore a CAP.
Tyler van der Hoeven (08:43):
That's a good way to put it, because it's the difference between the interoperability between like service providers, so if it's a wallet, you might have a wallet implementation, and you want all wallets to operate like you, but if you enforce everyone who's building on top of Stellar to operate like a wallet, is that what we want Stellar to be, or is that just what we want Stellar wallets to be like? Maybe, what are some of the guiding principles behind making that decision between a SEP and a CAP? When is it SEP time?
Jon Jove (09:18):
Yeah. So, for me, one of the most compelling things that would convince me to write a CAP would be we tried to do this as a SEP, and it didn't work. That's a very compelling argument because then it's like, okay, I saw what you did. I can see what happened, and it's not the desired goal, so we need to do something stronger. Another thing that would convince me is if you can show me that it can't possibly be done without a CAP. This is a harder ask. You need to know a lot about the protocol to know that something cannot possibly be done, because lots of things that seem like they might be impossible you can actually build out of the different components that we put into the protocol.
Tyler van der Hoeven (10:02):
Do you have an example of anything like that?
Jon Jove (10:04):
Yeah. So, Stellar Core, the Stellar protocol, it doesn't have any native mechanisms for options, but if I want to write an option, I can write something that looks just like an option using preassigned transactions, or preauthorized transactions, for that matter, depending on what I'm trying to achieve. So, there's nothing built into the protocol that says... There isn't a make option operation, but that doesn't mean that you can't build something yourself that looks just like an option. That would be something that... If you suggested that you wanted to have options built in to the protocol, I would respond, well, you can already do this. So, why isn't the existing solution satisfactory to you? Have you tried? If you can then show me, oh, well, it doesn't have the following properties that I would require, or it's deficient in this manner, then we can talk about, oh, actually, the protocol isn't good enough, versus, oh, you just didn't realize that there were these components that you could combine in this clever way.
Tyler van der Hoeven (11:10):
Yeah, absolutely. That's a good way to put it too, because there's a lot of... Stellar is built for something in the documentation that might point to a specific use case, but you can get away with a whole lot of finagling if you're clever about it, with preassigned transactions, with time bounds, with preauthorized transactions as signers. There's all kinds of interesting ways you can get to the goal that you might have, in a less-than-obvious way, I guess, and that just requires some more deep thinking around the functionality that exists, and thinking about it less in terms of what we say you should do with it, and more around what actually happens when you do it, because that functionality you could twist to be exactly what you want, even if it wasn't what it was designed or the words we use to describe it. The words that we use to describe it ultimately don't matter. It's what it actually does that matters, and that might allow much more freedom to accomplish the business goals that you might have.
Jon Jove (12:12):
So, you hit on a lot of the things that I think about when we're doing these engineering decisions. For me, when we're making decisions about a CAP, there's a lot of vision and mission that goes into it. Every CAP has a section like, what Stellar network goals are you serving by doing this? That's right at the top. By the time you've written 100 words, you've probably had to address that question. So, we are really internalizing that. We are making decisions that are about the mission, about the vision, and they have a big impact on what architecture people will need, what architecture we need in Stellar Core, and we never stop thinking about that.
Jon Jove (13:05):
So, a lot of times, these questions come into usability. If you have a feature, but it's not really that usable, do you have the feature at all? Well, if making it more usable would serve the mission, then we might make a cap to facilitate that. That's actually a lot of what CAP 23 is about, making something that you can already do, but with great pain, into something that you can do easily, that you don't need to write gigantic flow diagrams out to try to understand, okay, what state am I in? Oh, no. If I transition to this state, I'm going to have a problem. No. Nobody wants to do that. That's not usable. So, we are thinking about these kinds of questions a lot.
Tyler van der Hoeven (13:55):
I think that touches... I mean, that's a lot of the... Maybe not quite as a danger deep level. That's with a lot of the SEP processes as well, but it's that much more important when you're at a core level of this is going to impact every single entity that is trying to build onto Stellar, and is that something we want for the whole ecosystem? When you are looking at CAPs and CAPs are coming in, how... A lot of the CAP discussion happens on the Google developer email list. How do you view community involvement in the CAP discussion? Is it essential? Is it semi-helpful? Is it core to the formation of this thing? How do you view community involvement in the CAP discussions? How is it building up what that CAP is becoming?
Jon Jove (14:52):
To me, the community involvement, that's essential. In an ideal world, we'd have more of it, but I think the fact that we don't is kind of our fault, my fault for not talking about it enough. That's why we're here talking about it. But when we're making a CAP, there are two questions. We try to be as backward-compatible as possible, but normally when you change something, you might make things globally much better, but you might make some things slightly worse. Can we accept to make something slightly worse? Well, community feedback helps a lot to know, oh, no, we depend on that working this way. You can't change it. So, that's one thing. But also, we want to know that we're doing things that are impactful.
Jon Jove (15:42):
There's lots of interesting theoretical ideas that I have. I spend a lot of time thinking about the protocol, and from time to time I come up with interesting concepts that they never make it to CAPs, because while they're interesting, they're useful conceptually, nobody's asking for them. They're not going to move the needle today. So, one example of the way that we get community involvement, actually, other than just on the mailing list, we actually have held these working groups as we've developed these CAPs, and that has been very helpful, and I think as we do it more, it'll become more helpful as people become more familiar with what it means to sit down and really dive into the exact technical specifications of a proposal. What are the implications? Is it right? But we've done this a few times. A bunch of people have joined me for these, and it's been really cool.
Tyler van der Hoeven (16:40):
That's neat. Y'all just get in a Google Chat and hash it out?
Jon Jove (16:45):
Yeah, exactly.
Tyler van der Hoeven (16:46):
That's cool. Yeah. That's neat, and I think I agree 100%, that having community... Not just having random people talking about these things, but people that are... Their money is invested. Their business is invested in the decisions that are being made, what otherwise would be behind the scenes, to get this feedback loop of would this be helpful, how would this change so that you can get ahead of the curve a little bit, but also make an impact in the decisions that are made so that the thing that gets updated, the changes that are made, you're not just aware of, but you helped form in a way that you understand the global scope of this thing and how it's maybe a little bit worse for you, but it helps 300 other companies, or you'll understand how that change will affect your business, and how either maybe new functionalities that are opened up to you, or you may have to change some code, but ultimately, you'll be more performant.
Tyler van der Hoeven (17:37):
You understand and have a voice in those things. It makes those changes much easier to consume, and I think, again, that that perspective is so important to have, particularly in the CAP process, because like you already mentioned, SEPs are a little bit more fluid. They can be updated. They can be changed. They can eventually become CAPs if they're not successful. So, there's a little bit more fluidity there to say, "Let's try something that will help make the network more interoperable for this particular type of entity, this type of project that's being built on Stellar, for wallets or exchanges or what have you."
Tyler van der Hoeven (18:09):
So, there is a little bit more specificity I think you can have within a SEP that starts to explore, how can we make this thing interoperable between specific entities, but when you're doing something in CAP, you are addressing the entire ecosystem, so there has to be a lot more care put into what's the mission of SDF. What is Stellar trying to do, and is any proposal lining up with that, and at the most, like you said, the smallest level, and then building that out, because the smallest level still might be something really big. You're trying to accomplish something, and if your small thing just added something that didn't accomplish the goal that you initially had, then it shouldn't be included, no matter how small it is.
Tyler van der Hoeven (18:50):
But if that thing kind of grows to be something a little bit larger that does include some new operations or even some breaking changes where there are going to need to be major updates, that still is the smallest version of the thing that we have agreed is important enough to warrant the friction that will be caused by including this as an update, but it still is the smallest version of that thing. But you are always considering what's the mission of the whole network, not just a specific sector, which is maybe how a SEP would differ from a CAP.
Tyler van der Hoeven (19:23):
All right. Well, Jon, thanks so much for jumping in. We're going to cut it off here. We'll jump in next week for CAP 23, but thanks so much for the discussion so far. It's been fascinating, and look forward to digging in a little deeper into a specific CAP and how some of these thought processes make their way into a specific CAP. For more information about Stellar and the future of decentralized finance, visit stellar.org. Get involved in the discussion in one of your active communities on Keybase stellar.public, or Stellar Stock Exchange. Until next time, I'm your host, Tyler van der Hoeven. Catch you all later.