A Developer’s View Into Blockchain Network Architecture

• 

  Konstantin RichterCo-Founder & CEO
  Blockdaemon
• 

  Jesse RobbinsGeneral Partner
  Heavybit

This Blockdaemon hosted panel “A Developer’s View Into Blockchain Network Architecture” looks at blockchain through a devtools lens. The group focuses on topics including decentralization and technical debt. Hosted by Konstantin Richter, CEO & co-founder of Blockdaemon, you’ll hear from experts including:

• Brian Behlendorf, Executive Director of Hyperledger
• Jed McCaleb, Co-founder of Stellar
• Jake Craige, Lead Developer at Coinbase
• Jesse Robbins, Founder of Orion Labs

Introduction

Konstantin Richter: I’m Konstantin Richter. I’m the emcee host of the night. I’m the CEO and co-founder of Blockdaemon. We are here today with a fairly eclectic group of people who have a lot of experience in cloud computation formation, blockchain infrastructure, protocol management, etc.

The purpose of today is to talk a little bit more about the current state of blockchain, the infrastructure challenges and the tooling that is available to actually build real things. And one of the things we’ve tried to do is to blend some people who have a lot of experience in building cloud solutions, with people who build amazing blockchain technology.

We at Blockdaemon– just a quick introduction. I don’t want to shill our services, but we’re a venture backed company and we describe ourselves as the “Heroku for blockchain,” if you know what Heroku is. This is a software-centric discussion, but it’s a cloud configuration tool and Heavybit was actually started by the founder of Heroku, who is also an investor.

We are part of the Heavybit developer community and accelerator. This is why we’re here today. We’ve also had a couple of interesting announcements today, as Blockdaemon. We’ve launched our Stellar deployment which went live today, as well as Quorum.

But we’re very happy to have Jed here. He’s the founder of Stellar and a few other things. I’m going to go quickly so everybody gets a chance to introduce themselves, but I want to say a couple of words about everyone.

Jed is one of the people that inspired me a lot as a true developer, who probably is one of the most influential people in the space that you know the least about. He started and founded Ripple, he worked with Mt. Gox. He had a bunch of different experiences and he’s been in the space for a long time.
So we’re super happy to have him here.

To my left, Brian, who is the executive director of Hyperledger. A man who has an illustrious background in the open source community, with Apache and a bunch of other very exciting projects. They’ve been a VC, they’ve been in the space for a long time.

At the end, Jesse Robbins. The myth, the man, the legend. But Jesse is a partner at Heavybit, and Jesse is also the founder of Chef. He was instrumental in actually setting up the AWS cloud infrastructure business, as well as a bunch of other things.

And Jake here is one of the lead engineers at a small blockchain startup called Coinbase. They’re doing pretty well. And so the reason why he’s here is because these guys actually have to manage infrastructure and nodes where, “If things don’t work, shit breaks.”

A Developer’s View Into Blockchain Network Architecture

I feel we have a good section here. This isn’t too scripted. The goal is to start reminiscing a little bit about the open source community and how all you guys started becoming developers and engineers in the space, and what got you excited around open source systems.

Maybe we start that way and we go down the line while you introduce yourself. Say who you are, If I’ve missed anything substantial, and then talk a little bit about what inspired you about open source and what you currently do in blockchain. And then we’ll riff on that.

Brian Behlendorf: Sure. So once again, Brian Behlendorf, executive director of Hyperledger. I call myself the “Nerd Diplomat” though because at a Hyperledger we’re part of the Linux Foundation. So our job is to try to pull together two kinds of communities.

Communities of developers building underlying technology for distributed ledgers and smart contract systems, the second community are the companies. Like Blockdaemon, IBM, Baidu and Tencent. All these startups around the world who are building and want to build products and services on top of these technologies are just incorporated into their products.

And so the Linux Foundation has pioneered a certain model about around how to get companies to work together as a consortium. Not so much a charity, but recognizing the fact that most open source development is done by people working for a living, working to fix a bug or to add a feature for their employer. And how do you harness that? How do you channel it?

How do you try to keep the amount of unnecessary disputes between the companies and between developers to a minimum? By providing air traffic control.

I don’t actually employ any developers, nor do I write any open source code these days, and the world is much better off for that. But I got into it when I got onto the internet in 1991 as a freshman at UC Berkeley, and it was amazing to me that this thing had been built.

The internet felt like a secret universe, this portal to a place where you could send messages to people on the other side of the planet for free. You could access FTP sites or talk to people over IRC or read Usenet or a mailing list.

All of this infrastructure seeming to have emerged out of people writing protocols and white papers and standards. But not only writing the standards, but writing and giving away software that implemented those standards.

It was software you could inspect. You could crawl inside and see how it worked.

And if you tried running it yourself, and I quickly got a job as a sysadmin for the business school at Berkeley. Again, just goofing off, but it gave me a chance to crawl inside this code. And when you find a bug, and if you can come up with a fix maybe submit it.

Maybe it’s the right fix, maybe it’s completely the wrong fix but it provokes somebody else into actually doing the right fix. And so that culture of people sharing code long before the term open source was invented, which was roughly 1998. Long before Apache got started. which I fell into again because it felt like we were continuing this tradition of people working together to build common technologies.

But long before all that, this is how we got internet software. And I daresay it probably preceded proprietary software in the earliest days of how people figured out how to make computers do what they needed. So in continuing that tradition, when the web started to grow up, there were a bunch of us running websites built on top of the preexisting NCSA server.

This is the same team of student intern developers working at the University of Illinois who built the Mosaic browser.
Some of whom went off to start VC firms like Andreessen Horowitz, but the server side of that,
the students all left.

All of us using that software got a note one day that said, “Good news! All these students went off and got a job at this new startup company called Netscape.” And we were like, “Congrats. but we’re still using this old software.” And so because we had been exchanging patches with each other anyways, we just decided to keep doing that and called ourselves Apache, and eventually started a non-profit around it.

But it felt the most natural thing in the world. It’s like you’re riding a bicycle and you just keep riding that bicycle.

Jesse Robbins: How many people in this room have contributed to, or built a product on an Apache licensed product? OK. Almost all of you, regardless of whether you use a Mac or the internet, or an Apple product or a Google product, depend on the fact that the Apache Software Foundation exists and produced a license model that allowed us to write and share code freely to deal with issues like patents and other things. all really hard and complicated, now-solved problems.
At least in one general category.

And it’s interesting because the history that Brian just shared is often lost when you think about how things get built now, how easy it is to build things now. We depend on a foundation that many men and women built and that they built over a period of many years, that came from the same energy that’s starting to bring you into this room.

Brian: And I’ll give due credit to the Free Software Foundation, which actually started in ’84-’85. Something like that. So that also helped lay a lot of the tracks that we were able to drive on. But I think right around when Apache got started in ’95 there was this recognition, it wasn’t about charity necessarily,

It wasn’t even necessarily about ideology, about, “The world has to be free software.” It was instead a very pragmatic sense that by this underlying infrastructure stuff we all have to use, that hopefully we can all have the luxury of ignoring.

You don’t worry about the plumbing in this building, for example. Maybe we should. I don’t know. But you don’t have to worry about it because somebody else is dealing with it. Well, some software is infrastructural, some software is in your face.

The more underlying infrastructure software we can develop as communities, and the more time and money we can all spend on the truly differentiating stuff.
And that gets lost sometimes in the world of ICOs and a world of everyone building really bespoke blockchain infrastructures, that there’s a lot of commonality between these technologies and we should be working more. That’s what Hyperledger, at the end of the day, is about.

Konstantin: Jake, do you want to say hi?

Jake Craige: Yeah. I’ll try to follow that. I grew up with the internet–

Jesse: And you use Apache-based softwares, among other things I imagine.

Jake: Yeah, sure. My first foray into open source was when I started doing Rails development.
And the fact that this framework existed in this space and it was open source and you could see what is going on, and you could see this huge community around that space, and all this stuff that they provided that I could just go in and use.

I worked primarily in consulting before joining Coinbase, and so I’ve worked with a lot of different companies.
And most of them that we worked with are using Rails and are able to spin up these things very fast and have all these features they want really easily by just adding this gem here and there, and being able to do that.

It’s just a really cool thing, and that’s what got me interested in it.
From there my first open source contributions that have some value was with the Ember CLI project.
There was a cool thing that happened there where I was working on Ember.js applications, and I was building this thing to integrate with this thing called Cordova, which I think was Apache, or is now?

So we were using Cordova and we needed some tooling that integrated better with Ember CLI, so I was building this separate library. And what happened was Stefan Penner from the core team saw the repo and noticed and said, “How about you add this as an add-on?” Or, “How about you add some hooks and CLIs? You can actually do this yourself with the toolchain.”

It was this reaching out of him on this repo, he just opened an issue, that led to being able to actually contribute to Ember CLI and put in the first steps of this add-on system that they have today. That’s a really powerful thing. The fact that I was able, as somebody who worked at this small little company in Houston, be able to contribute in this big way and now the software is used by thousands of thousands of people around the world. I think that’s powerful.

The fact that you can see this, and that you can learn from it, learning was huge. Being able to work with these other developers who knew more than I did.

To be able to learn and see how they did things, the tradeoffs they made and to be able to have those conversations with them. To be this community together that pushes the world forward, and shares the software and says, “You can use this and you don’t need to worry about this part of it. You can just build this thing that you want and that your customers need.”

Konstantin: Great. Cool. Jed?

Jed McCaleb: Let’s see. My first big open source projects were definitely in the blockchain space, and prior to that we would always have this decision whether we should open source or not. And one of the cool things I like about the blockchain space is there’s no decision. It definitely has to be open source.

Because, like Brian has pointed out, it’s this open infrastructure that is this underlying thing, and people want to be able to see it. It’s used by many parties and so everyone needs to have access to it.
So that’s been great. And one of the other cool things I think about this space in general is that what you’re building is not just software, but it’s these open platforms.

It’s not just people contributing to the code, but they’re contributing into the actual ecosystem.

So there’s just a lot of different players that are involved, contributing all kinds of ways, which has been pretty exciting. Because it’s kind of like you’re in a big team not just with your own organization,
but with organizations at large. So that’s been pretty inspiring and fun.

Konstantin: Jesse, tell us, what is your role in the tech world?

Jesse: Well, I am motivated by a belief that developer technologies and tools have the power to make a better world. So in high school, my start, I became a sysadmin and I wrote a paper in in high school which was before open source was a thing. It was, “I want to work with a big group of people that are going to make the internet technologies and protocols for this thing called ‘the internet.'” And I described a vision for wanting to participate and help build organizations that make things that make things better.

My role at Heavybit is I’m one of the part time partners, so I have the good fortune of now helping other founders and teams avoid many of the problems and challenges that I learned as a first time CEO at Chef, and often provide a lot of the experience that comes from building what is now a big company and having worked on building big infrastructure in other environments.

So, that’s what I do. I’m hoping that everyone is here to learn how to embrace the next wave that is coming, and either build on, in, around or contribute to a set of tools and platforms that have a potential to change the way and improve lives for the next decade.

Demystifying Decentralization

Konstantin: Great. Thanks guys. Maybe one way to talk a little more about open source is the concept of decentralization. I’m curious what that means. We’ve got two guys who build protocols that are open source and blockchains, where the word “decentralization” is supposedly meaning.

It seems blockchains either can be built for speed or consensus mechanisms,
and there’s this tradeoff in-between.

So I’m curious what you think about the sharing of infrastructure that’s needed in order to actually have a decentralized network, and how important that is for you. And what that means for the security of the network. And then actually, I’d be really curious to hear, Jesse, what you think about the degree to how that translates into cloud formation and that world that we came from. I’d be curious to hear about that from both of you guys. Brian, why don’t you start?

Brian: I’ll start, sure.

Konstantin: What is decentralization?

Brian: “What is decentralization?” That’s a pretty contentious argument now.
There are people who take fairly binary points of view on this. “You’re not decentralized.” There’s a lot of tribalism–

Konstantin: “I’m a single point of failure.”

Brian: I do think a lot of people have felt that the last 15 years on the net, there’s been less decentralization of the internet as a whole. There’s fewer places where you can get an e-mail account, fewer places to go get caught up on news or to build a group for your family to share messages.

It feels like a lot of this has started to centralize or fall into a small number of parties. And a lot of very smart, very active people who for 10 years have been trying to build ways to re-decentralize the net.

I think a lot of people read in Satoshi’s original white paper, “Wow. Here’s a way that helps counter some of the scale advantages that you get from centralization.” Because centralized systems are incredibly cheap, incredibly scalable.

We know now, thanks to a lot of the hard work of open source developers, how to build central infrastructures to support things at the scale of a Facebook or a Twitter.

It’s actually harder to build more decentralized message systems, or decentralized databases. Those sorts of things. And so in that vein, a lot of people are trying to ask, “How do we tease this apart and build more decentralized systems?”

I think some of the cryptocurrency community had taken the perspective of, “If you go all the way to centralized and say that the only source of validity in verifying transactions is the proof of amount of CPU time that you’ve been able to successfully burn on a trivial problem, then that’s perhaps the most fair way that we could think of to build a broad network without anybody in the center.”

But I think they forget that there’s still a governance involved in deciding what software to run on the nodes and how that software is built. The debate for example, in the block size.
Or in the Ethereum community, between bailing out the DAO hack and not bailing out the Parity hack, is basically a sign that there is still governance at work.

You can either recognize that upfront and have a structure for having decision making inside of a community, or you can try to pretend that it doesn’t exist, and you end up with Lord of the Flies.

Konstantin: How do you do it for Fabric?

Brian: So there’s that side of the world.
And then there’s another side of the blockchain world which has been about, let’s say, you have a formal recognition for the role of a governing entity in a network.

The metaphor I’ve been using is it’s like having a referee on a football field or a soccer field. Somebody who basically grants certificates to participants to join a network. Then once you’re on the network you can participate in the consensus, you get a full copy of the ledger that’s being created, you can write transactions.

Frankly, that ledger could also be public read-write.
But you have these trustworthy nodes that are basically able to submit these transactions to a common pool. I think that’s still decentralized. If you have the ability to fire the referee if they start making bad calls, if the referee can basically make up rules as they go. And they can throw yellow flags because they don’t like you or they can charge you $100 dollars as a player to get on the field, that’s a bad referee and you should be able to fire them.

My hope is that we can use blockchain technologies to take a lot of business networks out there that today are very centralized in operation.
Where you do have to trust some party at the center, who charges a 40% fee for the service, by the way.
And they keep all of the systems of accounts clear and you have to trust them.

They are the ones at the end of the day in the point of authority.
And instead, tease those apart so that we’re much more decentralized, more individual competing players in a field but still able to play a game and hold that governing entity to check.

And that’s very much like the degree of decentralization you have in the open source community.
Open source projects, the one defining characteristic of an open source license is something called the right to fork. Which is the right to say, “Here’s a team building code. I disagree with them. I’ve got a better idea. They might disagree with me, that’s fine. I can take this code and go in another direction.” Or if you and I have a dispute,
we can part ways.

That’s actually a way of lowering the risk of working together on a piece of code because it means one of us doesn’t have to throw away the investment that we’ve made. And I think blockchain networks, certainly the public chains work the same way. That’s how we have a Bitcoin classic and Ethereum classic, and all these others. But I think we’ll see that phenomenon on both the permission and the unpermission.

Konstantin: Jed, I’m curious, because you’ve been instrumental in starting Ripple. Ripple is one of those protocols that gets a lot of stick about, “Is it a blockchain? Is it decentralized?” You took the open source components of Ripple and created Stellar, that actually was an enhancement of that. What was the thinking there? And what’s your view on decentralization, and what’s a blockchain and not?

Jed: Sure. I think there’s a spectrum of decentralization. It’s not binary, for sure. Not only is there a spectrum but there’s lots of different axes.
There’s, “Are the people contributing to the code, is that a set group of people? Are the people running the servers, is that a set group of people?” Like the governance, there’s lots of different aspects to it. And you can vary on all of those.

It’s really hard to be fully decentralized. I don’t even know what it means to have nobody control the thing, because ultimately at the end of the day there are people running this software. It is software, so somebody has to run it.

It’s always a trade-off, and it also depends on what your goal of your system is. In many cases decentralization doesn’t even make sense. At the end of the day if you’re having to trust– like if you want to issue an equity in this decentralized system, at the end of the day you’re trusting some company to redeem that equity and turn that equity into something.

There’s a party that you’re trusting, so does it make sense for it to be a fully decentralized system? So there’s a lot of weirdnesses like that, that you see in the blockchain world.
Where people are trying to make something way more decentralized than it will be at the end of the day.

Let’s see, what else. I mean, that’s it. I guess in terms of what’s a blockchain and what’s not a blockchain, to me the defining characteristic is there’s some public record out there that people can see but that you can’t change arbitrarily. So it becomes essentially this trusted third party where–

Konstantin: What is “arbitrarily?” If you’re in the DAO hack, for example.

Jed: That’s a problem. What it means is if there’s some database, and Google is running a database for all their e-mail. Google can arbitrarily go and change your e-mail. They probably don’t, but they have that power. Whereas with Bitcoin no one can arbitrarily change your balance because it requires all this proof of work and there’s this chain of hashes. So that’s what I mean.

There’s this public record that people can see, but there are certain rules for how it gets permeated. And once you start varying those rules then you deviate from what it means to be this blockchain idea.

I think that’s the important characteristic, rather than the actual underlying consensus mechanism.

Konstantin: How about Stellar? What was the thinking there in terms of enhancements around Ripple? It’s a foundation, there’s structurally somewhat decentralization in build.

Jed: The idea is that when I first learned about Bitcoin I was super excited about it. It was like, “I never thought this was possible, to solve this problem before.” But one of the things that has always bothered me is just how wasteful it is. So just trying to think of other ways to solve this consensus problem, and again make this thing where there is this public record that people can see but it’s not just arbitrary to change.

And so that was the idea. How do you come up with another system that can solve that same problem, but maybe in a more efficient way? Or a way that’s tailored for a different use case. And then along with that is, once you realize you have this,there’s a lot of other problems you can solve besides being a new internet currency. You can do lots of different things. And this is what Stellar is trying to do.

Konstantin: That’s exciting. Jesse, what do you feel as a guy who comes from cloud services, enterprise tooling, Chef.
What do you think about all this stuff? These decentralized, open source–

Jesse: “All this stuff!”

Konstantin: All this stuff. In two to three words.

Standardization of Distributed Systems

Jesse: The internet is built on a distributed system called BGP. And it’s a routing protocol, that’s how packets find each other, and it’s a consensus-oriented model. It requires global scale. It requires also the participation of multi parties and actors.
There’s hostile actors from time to time that will change internet routing tables.

How a packet gets from point A to point B is a complex thing that requires humans and machines in the loop doing stuff in order to agree on how we send data around.

Another version of this is DNS, the way that we find what a hostname is, that turns into an IP address, that routes through the BGP routing tables.

And all of these things are systems that were born from a decentralized early model. They have tended toward centralization, as I find ironically that humans tend to centralize.

We are anti-entropic, whereas the rest of the world tends to work the way the rest of the world works. We seem to be organizing machines.

There’s a couple of things that I have noticed over the years and trends that I see. One thing to know is that I am a firefighter EMT by training, that has been a thing I’ve done throughout my career. And speaking as a firefighter in this room I want to talk about some things that have emerged over time.

Jed: Fire safety issues?

Jesse: Yes. So here we are, and we have assembled in this room. This is a converted warehouse space. If you look up above you, you will see that there are sprinklers. and if you look over there you will see that big red hydrant stand that comes out. that is part of a decentralized fire suppression system that has emerged through a system of codes that are required for buildings that have occupancies with a certain type of risk. Now, the fire department did not install this.

There is no central installation agency for this sprinkler system. This is installed because there are safety codes that have emerged from one thing and one thing only, which is a loss of life. No part of our fire safety code has emerged and been widely adopted like sprinklers without a significant loss of life event that then causes people to go, “We need a regulation that says if there’s this many people in a space with this type of construction, you’ve got to have sprinklers, you’ve got to have exit doors that work in a particular way.” That is a part of how occupancy models work.

And as a systems engineer, and a person that has built a lot of infrastructure, I have observed over the years and in fact been responsible very early– my title at Amazon was “Master of Disaster” and I owned website availability for every property that bore the Amazon name. So I know a little about this stuff.
And the big thing to understand is that as we start to build these models in a distributed mode, we see the evolution of what Brian described.

Which is you have groups of people coming together with rough consensus and running code, and over a period of time we build systems that become things that people depend on. And then every once in a while, we have events that cause a loss, and the greater the loss the more likely it is that you’ll see calls for regulation.
You’ll see calls for these other types of changes. So one of the things that I look to is thinking about the world often like we think about fire codes.

Sprinklers are expensive and that means that if you want to open an artist space, for instance, in Oakland. And you want to be hackers who form a collective and you get together and you build beautiful artwork and other things.
And you do that in a building that doesn’t have sprinklers and then there’s a big fire, that results in tragedy. And so that’s why when we go, “Some things need to be regulated, some things need to have safety standards emerge.”

When you’re thinking about it with your hacker mindset, you begin to understand that the more people depend on you and the more that people depend on a system working in a particular way, the less they can really be responsible for their own safety in depending on that system.

So what I expect will happen over time, is that we will see the emergence of safety standards and safety protocols that become common whether we’re using a ledger for storing and sharing data publicly, as you’re describing. Or whether we’re using it as part of a financial system or transactional model.

What I believe is that we will see a distributed system emerge very similar to the way that that BGP routing has emerged and standardized, the way that DNS has emerged and standardized.
Unfortunately it is now centralized far more than BGP has.

And as cloud standards and blockchain related standards start to emerge, they will be based around the idea of simple concepts, “This is how you can audit. This is how you can see.
This is how we can make sure that systems work, that they have minimum understandable risk models that are mapped appropriately to the work and the value that they are doing.”

You’re saying, “In a tent you don’t need a sprinkler system. You don’t need sprinklers when you’re camping.
You need them when there’s a concentration of risk, and the people or the things that are exposed to that risk don’t have a method of evaluating it for themselves.”

So that’s my longitudinal perspective on how we will see both standardization and centralization occur over a period of time.

Konstantin: Great. I want to stir us briefly, and we’re going to open up for questions. We’ll have time to dive into different subject areas that we’ve touched on. But let’s talk briefly a little bit about the current technological performance of the infrastructure, and where we currently are in blockchain land. Jake, we were speaking a little bit about how Coinbase obviously runs noDes. There’s a lot of stuff happening on Coinbase. I assume.

The question is, you probably have the firsthand experience of the edge case of running nodes and keeping a ledger up to date, and running queries. How does that infrastructure look, and how much do you lean on centralized tools or existing developer tools to help with your current infrastructure? Do you have nodes for each currency that you run? Do you outsource any of that? What does that look like?

Nodes and Currencies

Jake: Yeah, I will talk about that. We run all our own nodes. We build them from source. We don’t rely on external releases. We check caches and all that stuff. We use Docker, and Docker powers basically everything we do.

It’s a very core open source component to our stack and allows us to build these in a really good way and get them shipped.
For the running of the nodes themselves, there is still tooling we have to build on top of just running a node. It can be just downloading a single node and then starting it up and then letting it sync.

The problem with that in our system is you sometimes need more nodes and sometimes you need less nodes, and so you’ve got to be able to start those up and you can’t wait a week or a day or whatever it may take to get the node up. You’ve got to get nodes up in a reasonable amount of time to be able to handle the increased amounts of traffic you may see.

So for that you have to build it in these systems that will take the entire database of the node which, in Ethereum, worst case to be fair, of this archive node that stores all the state history. Right now we’re on 1.2 terabytes or so. And so storing that and then shipping that around your cloud platform for each instance you need to do is a lot of work.
It’s a lot of data transfers. It’s a lot of storage that—

Konstantin: What cloud do you use, by the way? I’m just curious.

Jake: AWS.

Konstantin: How much do you pay a month?

Jed: He’s going to get you a deal.

Jake: Who knows? Probably a lot.

Konstantin: I’ll give you 10%–

Jake: I’d say it’s a lot probably. I mean at least on the crypto side of things, probably a lot for just transferring backups from one to the other. Any time we spin up a node it’s like, “All right, transfer a terabyte over S3,” or something.

We’ve got some new things in place that we’re working on to make that a little bit better. So it’s not just a raw transfer, but using some really nice Amazon features. Like elastic block storage and EBS stuff, which would be really cool.
But that’s one of the core things that delays getting nodes up and getting them available. The second thing that I think is critical in our world is we want redundancy.

We don’t just run a single node and expect that to work the whole time, because if it doesn’t then that causes problems.

People can’t send their payments anywhere and they can’t receive them.
There’s a lot of problems. So we need redundancy for that, you throw a load balancer in front of it. But now all these nodes are completely isolated from each other.

They may be in different regions, they may be in different AZs.
And because of that you have this eventual consistency problem.
This is a really interesting thing because we have to take all the data from the blockchain from the nodes, and basically process it ourselves because we have a much bigger set of data to look at.

We need to take hundreds of millions of addresses and take a look at those and say, “All right, do any of these addresses that this is getting sent to, are they ours?”
And the nodes aren’t built to do that. So we have to do that ourselves.

Konstantin: So, running queries across that network, how do you do that?

Jake: Rather than running queries for most things we actually just pull all the blocks. So all of our nodes right now are block base.
You can take every block, pull all its transactions and just inspect them yourself. So we’re doing duplicate work here.

The node has already done this and updated its own systems, but now we need to take that and implant into our system.
Because the nodes are not meant to handle hundreds of millions of addresses. They’re not built for our use case.

Jesse: They aren’t built for centralization.

Jake: Yeah.

Jesse: You are the central part of a decentralized system.

Jake: I agree. But The other side of that is my laptop doesn’t have 1.3 terabytes of storage.

Jesse: You need a new laptop.

Brian: Which you can get on Amazon.

Jake: Versions get cheaper, I guess.

Jesse: They’ve got 6T now.

Jake: Yeah, I know. So that’s a big thing for us is being able to pull this data, and it’s all separate. they’re not meant to run in these centralized worlds. But I think the future is there’s going to be a lot that are running in these worlds. There’s going to be a lot of companies that get big and that they need to run these nodes, and they need to run them in a way that’s efficient. And I think there will be specialized software for this. That’s built for this case.

And I think we always want to have this base case. Where you can run it on your laptop, where you can spin up your own thing.
When you can take what y’all are doing and run it on your own AWS world. I think that’s a really important thing to maintain. But I do think that for this scale, it just seems inefficient the way that it works in a centralized system, and being able to process that and manage that for people.

Konstantin: Makes sense. And we were talking earlier, we’re part of a Skype group of 100 execs of exchanges in wallets where we talk exclusively around Ethereum nodes not syncing and stuff.

Jesse: How do I do not get added to that group?

Konstantin: I’ll add you. You’ll have fun. But what’s fascinating for us as a solution provider in that space is to also learn that the largest centralized agents obviously don’t want to outsource any of that.
They’d like to maintain the centralization there and that creates an enormous amount of costs and stuff. How many people work in the infrastructure, on that side in Coinbase? I’m just curious. It’s not just you.

Jake: No, there is a team. We have a full infrastructure team dedicated to keeping infrastructure stuff healthy, SRE team for that stuff. But outside of that it’s mostly the crypto team which I think is 10 to 15 people or so right now, and we keep an eye on that.

As we grow we are figuring out how to reorganize the teams to have specialized features.
Where we might have this world where we have our internal team for spinning up nodes, and that we can do that efficiently, and they’re able to maintain that and provide good metrics.

That’s another thing that I don’t think is ideal in the current state of node software, is getting insight into what’s happening within the node.
We might see some weird case that seems odd to us that doesn’t match our understanding of what the blockchain is supposed to do and we don’t know what’s happening. We might log data out, and we’re going to log things from the nodes. But what the node logs and what they mean aren’t usually documented.

They might be vague.

They also just log straight to standard DOS which doesn’t make it as searchable, there’s no formatting you can do on the software that I’ve seen or at least the software we run. And so there’s a lot there.
And insight into how the node is performing, what’s happening there, metrics and being able to actually have a clear picture.

Because for the most part it’s a black box. We trust these things to power our entire business, and without knowing that it makes it harder to run these things and know when things are wrong and know what’s wrong.

Other than trying to see, “It’s not syncing blocks. Why? Here’s some logs. We hope somebody can help out with that.”

Jesse: I’m chuckling over here. When we were getting ready for the panel I was talking to Brian. And basically Heavybit exists to help developers build developer toolsets. And that means that you have to walk through whenever there is a major change in how infrastructure works, and the tools that we rely on, there’s going to be a bunch of technologies or companies or both that emerge that have to instrument the entire stack.

Here you are describing her orchestration layer, “We used these Docker containers in order to.” So that sounds familiar, “And we do that on AWS,” that sounds familiar, and then “It’s hard to instrument and understand and inspect what’s actually going on. If only we had…”

So what’s interesting is that there is this perfect repetition of every cycle that we go through where pretty soon they’ll be the Splunk for blockchain and the Heroku for–

Konstantin: We’ve got Splunk, too. We got all this stuff.

Jesse: The whole thing. And so what’s interesting and important is the problems that Coinbase has at scale, which are the problems that happen when a platform wins, are different than the problems that many of you when you’re starting and building a first project or maybe in an entirely different space are going to face.

The advice I can give people, and you can give people, about operating at large scale systems are interesting.
But there’s this other side of it which is how do you think about the space as a whole. One of the things I’m curious about is, as people think about the toolchain.

What are the problems that you see that you can take from systems management and you can apply to decentralized systems management?

What is the toolchain that you’re looking for? Where’s the white space where you’re like, “If only I had block I would get way more sleep.” What’s waking you up at 2 o’clock in the morning on your rotation?

Jake: Geth, currently. It’s been a little weird for us. But I think the stuff that Blockdaemon is doing is pretty cool, because you mentioned, “What if you are starting some new business, and you’re building it on a blockchain and you need nodes?” Yeah you can just go manually configure AWS to do something and hope it stays up, and hope you don’t have to restart or re-sync the blockchain or change configuration and get a new version, and you need to auto scale it up. And so you need tooling in place for that.

And I think y’all are doing stuff to make that better and I think there’s probably a lot of tooling that can be built how to automate that solution and to do it for you, by just wrapping your process in this thing and it will automatically take the DataDear and pull that off somewhere else.

Metrics and Monitoring

Jesse: Do you build your own metrics? Do you have your own metrics stack? Or how do you do your monitoring, once you have deployments up, how do you operate the platform that you’re running?

Jake: These days most logging and metrics logging goes into Kibana, ElasticSearch, all that world. And then we also use Datadog for specific metrics and tracking that. And then we do some of our own custom instrumenting for what we can via RPCs on these nodes, and take them and say, “What is the block height?”

So that we can graph that and see, “It is freezing at this period of time,” or, “What’s the memory use, what’s the disk space? How many peers does it have?”

The information we can pull from the RPCs we can take out and try and use that to do some monitoring and alerting and such on top of it.

Jesse: Is that something that you think of as being proprietary to Coinbase? Let me know if this is out of scope. I’m interested because everyone in this room hopefully will be building technologies that operate at your scale. And so there is a certain amount of operating experience and exposure that you uniquely have in this room.

So the question is, do you think that toolchain itself is part of the value of building a blockchain business? Separate from the fact that you’re running an exchange the way that you are.

But is that table stakes, or is that something special?

Jake: I think it’s an important thing to build a business unless you want to reinvent the wheel. And I think a lot of this stuff can be built in an open source world. And we would love to do more open source things as needed. And it’s one of those things where you’re going to need it if you want to run your nodes.

The other alternative is to use a company, use Infura to run nodes. And a majority of Ethereum of is probably powered on Infura. And it’s a very centralized entity. And they haven’t done anything bad yet, but they could.

If you want this decentralized world you need to be able to run your own nodes. And right now it’s really hard.
It’s not that easy to do and it’s not easy to do reliably.

If your system goes down, and you’re not tracking that your nodes are not syncing blocks anymore for some reason, and that it has no peers, nobody’s going to tell you that. The nodes don’t have things built in to tell you that.
You have to track that yourself.

So you can totally build tools on top of this that do that stuff for you and just wrap it all up. It doesn’t have to necessarily live in the node. There are some things, the logging stack and all that, that I think are some things that could be improved. To have better analysis of issues when they come up rather than just raw standard out strings.

Konstantin: For Stellar, what was your thinking there? How do you develop that architecture? How do you monitor the network? You have an exchange on Stellar, right?

Jed: Well yeah, it’s built into the network. I think a lot of these issues are because this software is not designed for this use case. It’s not designed to be run at scale, like we were saying before. And the people that made it, it was more, “Someone’s going to put this on their laptop.” It’s a descendant of Bitcoind.
Most of these projects are descendants of Bitcoind.

Konstantin: What’s Bitcoind?

Jed: The first Bitcoin node client. And honestly, it just wasn’t written with this use in mind.
It’s combining the API with the validation and all this stuff. It’s doing all these weird things where it’s not separating the stuff.

When we designed Stellar we tried to keep that more in mind.
Our API server is this whole other piece of software, a Stellar core writes out to a database that now the API can access. So it’s much more easy to put load balancers in front of it and scale it out, and we also emit all these metrics. Stellar core does. So you can see what’s going on.

Like what parts are slow, and all this kind of stuff. We tried to keep the idea that eventually it’d be cool if everyone runs this on their laptop, but that’s just not the way the internet works. Anyone can run a mail server. But in practice, only a few companies do.
And I think the same will be true in the new world.

Jesse: We all used to run mail servers.

Brian: I still do.

Jesse: You’re the only one that does.

Jed: I mean, I did it for a long time. But
now it’s just a hassle. You could run your own DNS server, but you don’t.
You let other people do it. You need to make your core software be able to be run by enterprises.

Konstantin: What are you most excited for? What’s next for Stellar?

Jed: For us?

Konstantin: What are you excited about? Where do you spend your day, thinking about technically–

Jed: Well, we’re hiring a bunch right now. So that’s where I spend my day, unfortunately. But for us we’re really focused on scaling right now. all of these blockchain technologies, none of them could fulfill their total promise yet. Obviously there’s huge scalability issues with Ethereum right now and we’ll eventually hit them.

We have a lot of room to run,
but I think they’re all aspiring to be this global distributed system and none of them could really handle that load at this point.

That’s our number one concern, is how to actually make it get there.
That’s what we spend most of our time on.

Brian:
We take a different approach. I mean, Hyperledger, even before I joined, staked its flag in this space that said, “Maybe it’s not one giant ledger that everybody lives within and writes transactions to. Maybe it’s not one global computer. Maybe it’s actually a patchwork of a lot of different ledgers that exist out there, that are fired up easily between a set of parties.” You realize they have a common set of use cases and that one ledger might differ quite a bit from the other.

And choice of technology, but also very prosaic things like, “How frequently do you upgrade?” You might have one set of customers and one set of applications where they want ultimate stability.
They don’t want things to change more than once a year, and be heavily tested. Others, you want to update every week when there’s a new release.

One who wants full, Turing complete programmable environment for their smart contracts, others say, “No there’s just three kinds of contracts to have on here. And we’ll audit the code to that thoroughly and we’ll just lock those down.”
This idea, I would wager, is actually more decentralized than a notion that says everybody’s on one set of rails.

And I think you do need that when you want payments, because you want payments to have as large an audience as possible and be as portable as possible. You want things that operate at the scale of a Bitcoin, Ethereum, and Stellar. Those sorts of things.
But for a lot of these other kinds of applications out there for blockchain technology and supply chains, in health care, in all sorts of bespoke financial markets.

It’s perfectly fine to have a blockchain comprised of 20 to 100 nodes,
processing a couple of hundred transactions per second. That’s the vast majority of use cases out there would be more than satisfied by those, as long as you have ways to tie transactions across ledgers to do discovery of which ledgers are out there. And I’d say, as long as it’s always easy for that 21st Bank to join the network for that 101st health care organization that joins the network.

So there’s a mix of tech and politics in those kinds of questions. But I think we’re going to end up with most of the blockchain world looking these kind of easy to spin up, more manageably sized blockchain networks that might have lifespans.
They might even be rolled up at the end, and some net settlement, and become basically a historic thing and then restarted again partly to address some of these scalability concerns.

Konstantin: That’s great. Well, let’s roll this panel up. I do want to open up for questions.

Q&A

Visions of Blockchain Integration

Jesse: You mean a database, right? What we have is, “I’m a Fortune 500 company and what I want to do is write some data, and do some work on data, and have that have high integrity and trust.” It’s a database.

Brian: Well, to a short ledger, right? There will be clients that run on each of these different companies that are on a shared ledger, the interface between that and the rest of the organization will be web services interfaces. It’ll be a local SDKs. It’ll be stuff that’s familiar to the communities there.
They’ll still be an integration challenge, and I think the biggest question is migration.

Because for any of these blockchains to be useful they have to become the system of record.

That can’t be an echo of a transaction record happening in some other system, they have to, if there’s a dispute they have to be the final arbiter. And so migrating from a bespoke in-house ancient database system, our accounting system, to use something like that as the reference could be a big leap.
Both technologically and politically inside of many large companies.

Jake: The database is definitely a big thing. Having a database so you can actually query and do analysis is huge. I don’t know much about how Stellar stuff works, but I know you support something with Postgres, and you may be able to actually query that and do things. This is the problem we have today at Coinbase, because our data team wants to look at the blockchain data and match that up with our internal accounting and make sure everything’s kosher.

And we can’t without building custom tooling that takes all the data from the blockchain and pulls it into a sequel database that they can actually use. And then on top of that, with moving everything over, I think another big thing is transaction fees.

You have to be able to trust that this thing is going to be constant. When you’re in the world of Ethereum, or Bitcoin or really any of the popular chains today.

The fees may change out from under you with no fault of your own. Somebody comes in with the new ICO, they come in with a new app and all of a sudden, the fees are 10-40 times more expensive. And that’s not something you can use in this public world.

So maybe private blockchains are one of the ways that you can do that, to make that more of a consistent experience. But that’s another huge barrier to adoption. You don’t know what it’s going to look like tomorrow, and you have no assurance of that.

Jesse:
Just to be clear though, a blockchain that has a sequel interface to it is a database, and it’s just a cool database. It’s really cool and it has a bunch of interesting benefits–

Brian: A shared, multi-master database resilient to hostile actors, which is the novel part.

Jesse: That’s fantastic. That’s great.

Jed: Well, I mean all of these are databases. Bitcoin’s a database if you want to go that far.

Jesse: That’s what a Fortune 500 company, when they’re not looking at it for financial reasons but instead are looking at, “What are the benefits to a system like this? To a shared one, to a centralized, to a private model,” they are looking at, “What are the benefits of having an enhanced ledger, where you get away from the many restrictions of a relational database that you can’t understand and audit and do other things too?”

But fundamentally, if you’re in this room and you’re trying to build a startup that’s building for Fortune 500 companies, and you’re trying to think, “How can I help them?” The way that they’re going to approach you initially and actually pay you money, is they’re looking for something that’s like a database but has a bunch of these benefits.

So if that’s you in this room, just remember that this is a chance to make a better thing that people understand a little bit,
and do so in a way that they understand a lot.

Looking Ahead 5 Years

Jed: “In five years, how will I spend a day of my life?” I don’t know. I feel like it will be a robot holocaust. I don’t know.

Jesse: Apositive sentiment there.

Konstantin: Probably some fancy diet? But one thing to that though, what we found interesting when we started to deploy nodes and networks, and trying to iterate around the infrastructure components, and orchestration services around blockchain. Is how early we still are. I mean there’s just so many components missing in order to build anything remotely consumer-centric, or anything that hits scale on a level that you’re used to.

From my perspective, it’ll be five years before you see something even remotely relevant. It’s still very early.
From that level, and that was interesting for us when we started Blockdaemon last year because we started just before the boom, where token prices and valuations and where blockchain suddenly became the hottest subject. Because the valuations were so crazy.

We started raising money with infrastructure scaling issues, all that type of stuff, which people really didn’t want to hear at the time. And now it’s a very popular subject, so we’re doing better.

But it’s been really interesting, that dawn of understanding on how complex decentralized systems really are.

And how much work it takes in order to shepherd open source communities to agree on consensus models, and build tools that can live on a blockchain that with time becomes more and more and more complex? And also, an interesting question would be, “How large do we think Bitcoin or Ethereum or any of those chains will be in five years? How do you do that?”

Jake: Don’t know.

Jesse: I have a phrase that I wrote once, a mentor of both Brian and mine, Tim O’Reilly. I said to him, “You become what you disrupt.” And Tim likes to throw that back in my face every once in a while, because I’ve done some industry disruption, and then suddenly you find yourself in that model.

I feel at the moment, where we are, if we succeed and suddenly we see mass adoption in finance in a mainstream way, if we see mass adoption by the major enterprises and they’re using blockchain, what you’re going to find is that you look like the entities that you disrupted.

If you’ve built something that looks like a database, congratulations.
You run a big database company. And, “It’s got these other advantages.” Cool,
so maybe you disrupted Amazon, or Microsoft, or Google and their cloud environment.
You look just like them.

The interesting thing is that decentralization has a potential to change that fabric in the same way that it did the first time when we launched and began to commercialize the internet.

And so what I hope is that we see a wave of innovation and new services that emerge, that five years from now look like what we started to see in the internet in ’99, 2000.
Where it was a whole new true amazing world. And so that’s what I’m looking for. But I do believe you become what you disrupt.

Brian: The Who said it better: “Say hello to the new boss, same as the old boss.”

Jesse: Yeah, exactly.

Private Vs. Mainnet

Konstantin: Great question. On the public node for us, it’s more of an aficionado product. I do urge people, like, Bitcoin, I’m not nearly as savvy as any of these guys.

But I do like the concept of easy node deployment and making it easy for nodes to be deployed and run. And I think that’s very important for Bitcoin and Ethereum and any of the larger public networks. The use case very often is connect your own wallet to it, have a complete history of the ledger.

You can run some query, you pull some data out of it. But it’s not something you need to do. You can go on Coinbase and have him worry about it. And so the use case on the private stuff is the way we productize it. It’s really a network management tool. What that means is, any protocol and project that has a native token that needs outside nodes to validate transactions, needs a system in order to make it easy for people to deploy a node and connect it.

Because it is actually fairly tedious. Nodes are the gremlins of software. they tend to be a little idiosyncratic based on the protocol and configuration. And then just because you know how to do an Ethereum node, it doesn’t mean that you know how to do a Stellar node. Or any other type of node. They are all specific and there’s so many of them now, that as a developer you have no time educating yourself and all these different very specific use cases you need in order to connect to a network.

So customers of ours are companies like Aeon, for example, which is the more popular protocol. The Gold Network which is a Quorum version that we’ve just launched. We announced it today. These guys need outside people to run transaction validation nodes that are banks and commodity traders, and they have no idea how to deploy a complex form node. And so we have a White Label product that they can use. They send these guys an invite.

It takes them to a back end where there’s a big gold button, and you just click that, and it deploys a node and you can select the infrastructure of your choice. And so that’s the thing where we make the most money with. Ultimately, I would urge everyone to. I do feel that a node is your voice in the consensus mechanism.

And I think on the larger networks it’s fun to run them. But obviously I’m biased and I like that stuff.

Brian: You’ll bootstrap a network locally on Blockdaemon with the presumption that at some point the network will grow, and there will be nodes on that network on other infrastructure providers.

Konstantin: Absolutely. Totally. And we work with everyone. We’re trying to be very open. We’re trying to open source our version of it, where individual protocols can actually add their own nodes to our servers for people to deploy. So we don’t even have to do that anymore. That would be my dream.

Best Decentralized Governance

Konstantin: I don’t know how to answer that.

Because, I mean, it depends on the use case of the consensus that’s needed. So there’s a lot of popular ones. I haven’t seen a blockchain where consensus is applied in a way that we maybe think of it. Even Bitcoin is fairly centralized in a lot of ways, if you think about it. So I don’t have a good answer, I don’t know. Jed, do you have an answer?

Jed: Yeah. I think people just don’t know how to govern anything. Like society. Governance is an open question still. There’s no easy answer, I don’t think. It’s ultimately boiled down to personality politics.

Brian: There’s a temptation in this space to try to say, “Can we automate these through having rules around governance that we can encode in software?” Things like, if two thirds of the network agrees that there’s a bad actor node, that they can kick them out, or something like that.

Jed: But then it’s just a popularity contest.

Brian: There’s a risk. You set up a set of rules–

Jesse: He has all the nodes. So he just changes the nodes.

Brian: But even the creation of rules, and we were all nine years old once. Somebody tells you a set of rules, your brain goes to, “How do I game those? How do I hack those to my advantage?”

And so I think coming up with algorithmic rules, the prevention of double spend, is a really cool thing.
It’s a great form of governance rule.

That there’s probably other logic that we can implement into a blockchain and make sure the right kinds of things happen, and the wrong kinds of things don’t. And the more that we can encode, the less arbitrary that enforcement can be. But at some point, at the end, you need some human level of conversation between the people who are running the nodes, running the validators.

To say, “Should we evolve the rules in some way? Should we fix that bug that somebody used? And by the way, they exploited that bug, and they stole assets from somebody else.” We need to correct for that. And so some adjudication process. I mean, you talk about DNS, ICANN gets a lot of hits for being an awkward organization, but at least they have a transparent process for when somebody steals a domain name, or somebody feels, “You registered my trademark.”

There’s a dispute process that everybody who’s a domain name registrar had to agree to, to have the right to register domain names. And so we’ll probably see similar things, and other types of blockchain networks out there where it’s a lot of algorithmic, but some small little bit of adjudication process that the parties agreed to as humans. To go in and sort out bugs or broken transactions.

Jed: I think what you mentioned earlier is always the ultimate escape mechanism, that you can always just fork it. If you ultimately, if you disagree strongly enough, then you can just create your own network.

Jake: I think you can, but the network’s effect is so strong. I mean you always see these core developers, who you hear, “They don’t have that much power because you can run your own node,” or, “You can just not run their software.” But when you’re running nodes you end up relying on them for specific things. You rely on specific APIs they may offer. If they implement a feature you don’t like, yes you can choose to not run it, or you can fork it and run your own.

But to actually have a network on that, or to change your infrastructure to be able to support that is pretty powerful. So
you can’t automate everything, which I don’t think anybody has said here yet, but you can’t automate who controls the code and what goes in there because there will always be these gatekeepers there and the people who implement it.

There’s a lot of incentives for them to do the thing. If they do the wrong thing, people aren’t going to want to run their software
and they might actually take that opportunity to switch.

And so there is incentives there for them to do that, but they do still have this power to control it. And if there’s a single node implementation that most people run, they have a lot of power to push the network in the direction that they want to go whether or not people agree with it.

Brian: I think Tezos has put the most amount of mature thinking into this kind of question too. How do you automate protocol upgrades and get consensus by the node operators on the right kinds of upgrades to do?

Konstantin: Not on the legal governance, unfortunately.

Blockchain Roadblocks

Jesse: Massive increase in the cost per watt.

Jed: I would say regulatory concerns are probably the most hampering effect, I think.

Jake: I think major bugs. Major, money losing bugs in some of these implementations. Or massive chain splits that caused havoc on the world of cryptocurrency, and it’s just going to call the entire security and trust of it into question If that happens.

Brian: Does anyone remember the last crypto wars in the late ’90s? When it was officially illegal to export arbitrary encryption technology of arbitrary key length? We used to have rules that pretended that this was a big secret, public e-crypto. And I’m worried that the kinds of things that would set this back would be a trade war, leading to people not able to travel to conferences or not able to collaborate across certain geographic boundaries.

I’d worry about pissing our allies off, I’d worry about– Oh wait. There’s a lot of that happening now. Before we go down that path.

I think it’s macro issues, not technology issues, that put a lot of this at risk.

Konstantin: I think a mixture of that, or Jake getting drunk and hitting the wrong button in Coinbase.

Brian: Publishing the keys to WikiLeaks.

Konstantin: That’s right. Something like that.

Future of Web Services

Brian: We have a slightly different take, which is the reason why no one can run a mail server on their own anymore is spam.
Spam made it too expensive to run mail filters, filtering out the inbound. But it also set a whole large part of the IP address space as blacklisted. You couldn’t send mail from there because it was assumed by the ISPs, if you were sending mail from your home internet connection that you were probably a spammer.

I run my own mail server, but I’ve had the same IP address for 20 years so it’s been white listed everywhere. But it’d be really hard now to run it from my home. And the funny tie to blockchain is proof of work actually came from an anti-spam proposal, which was to stop the spammers by sending a challenge that they would have to solve.

Whether it’s refactoring or reversing a hash, or something like that, to try to slow them down got too expensive to send. The conclusion being e-mail failed because it was too cheap. It was zero cost to send e-mail, so everybody did for everything, and that just swamped the system. And that’s unfortunate.

And it’s possible that public cryptocurrency approaches might help mitigate that, might provide an economic model now for being able to accept messaging again that can’t be so easily DDoSed. But it would be really tragic, in my opinion, to go back to a world where you had to spend a little bit of money to send an e-mail. Because before I got on the internet I was on Prodigy, where it cost 25 cents to send a 280-character e-mail message to somebody else. I’m really glad we got off that.

Jake: Isn’t that what blockchains are doing?

Brian: Kind of. Well, the public ones.

Influencing Regulations

Jake: I’d say one thing, just being up here is helpful. Because outside of this world, it’s all criminals, it’s all terrorists. And just being up here and showing, “We’re trying to actually do good things. We’re not actually those people.” That in itself is helpful to push it forward and bring it more mainstream so It’s something that people want.

If it’s something people want, it’s harder to take it away. They still can. But it’s going to be harder.

And I think that’s just one thing that we can do, as a very small thing.

Konstantin: Is there a specific regulation?

Jed: No, not really. I would echo this sentiment. I think the sooner this stuff can be used widely in the world, it’ll make it much harder for it to be regulated. Because obviously it is being used by some stuff that regulators want to stop. they probably want to stop these really scammy ICOs, I suppose.

And if that becomes the totality of what this space is about, then yes, they’re going to put the kibosh on it. So you really want legitimate use, so it can have even greater legitimate use in the future.