Getting up to speed on Ethereum
https://medium.com/@oneofthemanymatts/getting-up-to-speed-on-ethereum-63ed28821bbe
Last updated
https://medium.com/@oneofthemanymatts/getting-up-to-speed-on-ethereum-63ed28821bbe
Last updated
Aug 2017
You should read this blog post if:
You’re a professional software engineer
You want to have a deep working understanding of Ethereum and the related ecosystem.
Prerequisites:
You understand the concept of a blockchain and how Bitcoin uses it to create a trustless digital currency. No? or . Hint, you should probably watch the long one.
You’re a professional software engineer. Seriously, I mean, nontrivial amounts of real-world software engineering experience. Understanding systems/architecture/math is a learning efficiency multiplier.
Like a week of your free time. I told you we’re skipping the nonsense.
You understand the basic concept of a and how it can be used to quickly verify information correctness.
Note: you don’t need to read the whitepapers, but you need to understand them. But it turns out that reading them is the fastest way to understand them. 😉
I recommend reading the entirety of this post, absorbing the contents at a high level, and then diving into each link individually, over the course of a few days.
You’ll be surprised at how little of this technology is magic (read: none of it), despite the severe case of buzzword-itis the ecosystem has. Everyone is building off of the shoulders of giants; if you make a living as a professional software engineer you have the ability to understand all of these projects and technologies at a deep level. Just don’t get distracted by the bullshit.
is a distributed computer; each node in the network executes some bytecode (hint: Smart Contracts), and then stores the resulting state in a blockchain. Due to the properties of the blockchain representing application state, this results in “applications that run exactly as programmed without any possibility of downtime, censorship, fraud or third party interference”.
Obviously an acceptable explanation won’t fit into a paragraph. Go ahead and read the . Or one of the other trillion “how do I Ethereum/Blockchain/Smart Contract” posts on the internet. Or watch this video titled .
A “Smart Contract” is literally just some code (seriously, that’s it) that’s executed in a distributed environment like the Ethereum platform. The platform on which it’s executed gives this piece of code some properties such as: immutability, deterministic operation, distributed & verifiable state, etc. The state managed by this code is stored, immutably, on a blockchain and anyone can double check that the current state is correct by replaying all of the transactions from the beginning of the chain.
In Ethereum, contracts are given an address to uniquely identify them (it’s a hash of the creator’s address and how many transactions they’ve sent before). Clients can then interface with this address by sending it ether, calling functions, querying the distributed state that it manages, etc.
Smart Contracts, again, are just some code with some distributed state managed by a blockchain. For example, multi-sid wallets that you’re using to receive/send ETH is just a smart contract with a fancy UI on it.
This concept is pretty powerful, and I’m sure you’ve already read all about it. If you haven’t, browse your favorite mildly-technical new source (hey Medium!) and you’ll be inundated with people telling you how much potential there is. Some buzzwords: asset/rights management, decentralized autonomous organizations (DAOs), identity, social networking, etc.
The “price” of gas (i.e. “how much $$$ does adding two numbers together on this distributed computer cost me”) is determined by the market, similar to Bitcoin’s transaction fees. If you pay a higher gas price, nodes will prioritize your transactions for that sweet sweet profit.
In general, it’s way more expensive to compute and store things on Ethereum than it would be to do it in a traditional environment, but Ethereum gives your code all those nice properties we discussed above, which can be just as valuable.
In general, reading state is free and writing state costs gas. Here’s a more in-depth overview of the concept of gas:
A Distributed App is an app where the “server side” is just one or more well-known Smart Contracts that exist on the Ethereum network.
A Distributed App doesn’t need to store all of its state and perform all of its computation on the blockchain, which would be pretty expensive for end-users, but a distributed app does eventually store trusted state on the Ethereum blockchain, which anyone can then read. Many distributed apps also use technologies like IPFS and Golem (discussed later) to handle computation and storage off of the Ethereum blockchain, but in an equally decentralized manner.
Also, we’ve finally agreed on the proper capitalization of “dApp” via twitter poll (so you know it’s statistically significant), so let’s all just write it as “dApp” then, yeah?
These distributed applications are usually accompanied by some sort of user-friendly frontend because nobody wants to manually send transactions to/from contracts using a cli or manually craft requests with hashes and opcodes oh my.
A dApp Client is literally just any “client” or “frontend” as you’d normally use the term in programming, except this client interfaces with the Ethereum blockchain (perhaps in addition to other services) somehow. These clients are frequently written in JavaScript because we haven’t yet finished converting everything in the world to NodeJS.
More seriously, most dApp clients are written primarily in JavaScript because it can be run in a web browser and everyone’s already got one of those. They’re also frequently written in Go due to a superior state of existing tooling available. It’s a viciously positive cycle of improvement, which means unless you really know what you’re doing, you get to choose between JavaScript and Go (and to a certain extent, Rust) for interfacing with the Ethereum blockchain and the protocols being developed on top of it.
* So there’s a little bit of confusion/discussion around the exact terminology/definition of “Distributed App”: is it just the smart contract(s)? Is it the entire backend of a system which, at some point, interfaces with the Ethereum platform to store trust? Or maybe it includes the client code, too, and the user interface so the whole bundle of stuff is called a “dApp”?
* I’ve gone ahead and defined as “the backend of a system that interfaces with the Ethereum blockchain”. This is different enough from “Smart Contract” to warrant its own concept and also implies (correctly) that anyone can create a client to interface with a distributed app
A dApp Browser is exactly what it says on the tin; it’s an application (a normal one that we’re all familiar with) that makes using dApp clients (usually javascript that interfaces with an Ethereum node to communicate with smart contracts) easier.
The primary purposes of a dApp browser are to
Provide a connection to an Ethereum node (either to one hosted locally or remotely) and an easy way to change that connection to point to a different node (which might be connected to a different network),
And provide an account interface (“wallet”) for the user so that they can easily interface with these dApps.
Your node needs to know which blockchain to download and which peers to talk to; see below for a discussion of the different networks available.
If you’re distributing a dApp client to users, you don’t necessarily need to provide access to an Ethereum node as well; dApp Browsers provide the connection to any client that needs it.
So you know how we can write code (a “Smart Contract”) that stores some state on a blockchain? What if, in that state, we stored a map of ethereum addresses to an integer. And called that integer your balance. Your balance of what? Let’s just call them “tokens”. Oh.
Due to some issues with the ERC20 spec (namely that it was made quickly, has small security concerns, and requires two transactions for payments), now we have more proposed token standards.
* there had been some confusion around ERC223 vs ERC23, but they are the same concept; the ERC number is 223, so this standard should be referred to as ERC223.
You interface with your smart contract (aka, executing methods and reading state) by connecting to an Ethereum node and executing functions over a JSON RPC API. There are a variety of Ethereum node clients that perform this in a developer-friendly way. Both geth and parity provide consoles/browsers for interfacing with contracts.
When you “deploy” a smart contract, all you’re really doing is sending a transaction to the 0-address ( 0x0 ) with the contract bytecode as an argument.
Here’s more information about transactions:
Once you start writing smart contracts, you end up doing a lot of the same things over and over again; compiling your source code to bytecode and an abi, deploying it to the network, testing the deployed contract, etc. You probably also want a nice way to play around with new ideas.
Frameworks like Truffle, Embark, Populous, and Perigord standardize and automate a lot of the minutiae. They provide a nice developer experience for writing contracts, deploying contracts, and incredibly importantly, testing contracts.
This post has a lot of good information and uses truffle to deploy and interface with a contract.
Of note, you can also use NPM to provide smart contract code for distribution, which a lot of people do, notably Open Zeppelin.
Mainnet — the main Ethereum network, and generally the default in whatever client or browser you’re using.
An Ethereum account is a private key and address pair. They basically just store Ether, and they don’t cost gas to create (because they exist automatically with every possible address). All transactions on the Ethereum network originate from an account; contracts don’t have the ability to initiate a transaction.
Now that we’ve correctly defined the two, prepare to see people confusing the two terms everywhere and labelling anything that sends/receives Ether as a Wallet and calling anything and everything an account.
The code that is your Smart Contract runs on every full node in the network within the EVM (Ethereum Virtual Machine). It’s your standard virtual machine that executes some bytecode, except the vm is specifically isolated from the network, the filesystem, processes, etc. Nobody wants to write bytecode, so we’ve got some higher level languages that compile down to EVM bytecode.
Here’s what Solidity looks like:
LLL looks something like:
If you’re learning, you probably don’t want to write anything in LLL.
There are a bunch of other high-level languages in various states of usability and development and more will undoubtedly be developed. In order to reach wide adoption, though, a language and compiler must be thoroughly vetted and tested, which will take time.
Once a smart contract is deployed to Ethereum, it is immutable and exists forever. If you write a bug, you don’t get to take down the broken version; you can only fix-forward*.
Because so many engineers developing for Ethereum and other smart contract platforms are coming from web development, this concept might be new and crazy.
There are a variety of different security-related traps your code could fall into, both at a language level and from a high-level logic perspective. When you deploy a production smart contract that handles real money, you need to be 100% confident in the operation of the contract.
* there’s a selfdestruct() function that a contract can use to remove itself from the network.
It uses the protocol shh which is pretty cute. There’s surprisingly little documentation and adopting of this protocol.
Whether or not whisper is under active development is up for discussion.
This is an organization (like, a group of human beings) where, instead of using legal documents to enforce operation, they use a bunch of smart contracts. Your group of human beings then uses these contracts to do all the normal stuff an organization does, like vote on things and determine what color all the local co-op houses should be painted.
A side effect of this is that decision making, governance, and what color the houses should be painted is immutably stored on the blockchain (in the state of those contracts). Cool stuff.
While this is a new protocol, there is an http gateway and a filesystem adaptor, meaning you can fetch IPFS content via http and mount the entire world wide filesystem on your local disk at /ipfs . IPFS also provides a naming system called IPNS (InterPlanetary Name Space), which allows for mutable state (recall that everything in IPFS is immutable). You can even use DNS TXT records to direct your IPNS client, allowing you to generate human-friendly links to data.
There’s also some public backlash about how FileCoin is being implemented and distributed:
Liquidity of tokens is a relatively large problem in the crypto ecosystem. Trading between users requires satisfying both your desire to buy and the other party’s desire to sell (or vise versa).
The value of your arbitrary token managed by the Ethereum blockchain can fluctuate pretty arbitrarily; this sucks if you’re trying to use that token for some sort of actual real-world process (like going to sleep and hoping it’s still worth something). Stablecoins are considered a Big Deal™ for the field of cryptoeconomics.
“What if my Smart Contract needs an external piece of information, like the weather in New York? I’d have to use a decentralized oracle protocol to ask a bunch of people the weather (expensive, slow), but if I wrote a service that provides this information from a central place, I’d be betraying the core idea of a decentralized application.”
Their integration is pretty powerful; you can fetch URLs, parse JSON and XPATHs, query WolframAlpha, etc.
Zeppelin is a technology firm doing some really awesome, professional stuff in the space. Honestly, they’re doing a lot of things and it’s tough to keep track of it all.
There are some other neat planned aspects of zeppelin_os, like the scheduler (async execution of contract functions, since by default contracts don’t do anything until interacted with), a marketplace protocol, and off-chain developer experience tools. You can learn more about them via the whitepaper.
Brave and the Basic Attention Token was started by Brendan Eich, who originally created JavaScript and then co-founded Mozilla.
Obviously this document is going to be out of date in t-minus 2 hours, so if there’s a protocol/platform/technology/team that you’re really excited about, let me know via a comment and I’ll consider documenting it.
The purpose of this document is to create building blocks of understanding rooted in real life. None of that “key and locker” lossy metaphor nonsense. If you’ve found it to be helpful, give it a 💚 (wait no, a 👏). If you found that it did the opposite, let me know via a comment. Or on twitter dot com or whatever.
It also attempts to remove most/all of the buzzwords and high-level magic that so many people want you to believe that this technology is.
Smart Contracts (again, just blobs of code) are executed by every single full node in the network, which is a lot of redundancy (good) but this costs a lot of energy and time (bad). Because it costs money to perform computations, the cost of performing computations is directly tied to which computations your code performs. To say that another way, each low level opcode in the EVM costs a certain amount of “gas” to perform. The word “gas” is totally arbitrary; it’s just an abstract label given to the cost of performing computation. There’s also a network-enforced gas limit, to solve the ; i.e., you can’t write a program that never ends, because you’d run out of gas and the computation would be rejected by the network.
The ethereum organization on github has a that has some references and examples. Make sure you check the activity on the file you’re looking at, though, cause this kind of information gets stale very quickly.
is the official Ethereum dApp browser. It’s really just a pretty web UI for interfacing with an Ethereum node and sending transactions to/from Smart Contracts (dApps!).
is a mobile browser with a unique approach to the UX of using dApps.
is Coinbase’s foray into building an Ethereum wallet and browser. It bets big on the “wechat” + “chatbot” vibes.
is a Chrome Extension that turns Chrome into a “dApps Browser”. Its core feature is that it injects web3, a JavaScript Ethereum client library, into every page, allowing dApps to connect to MetaMask’s hosted Ethereum nodes. The Chrome extension allows you to manage wallets and connect to the different Ethereum networks available.
is a an Ethereum client (as well as a full-node implementation) that integrates with your web browser, turning it into a dApp browser.
Mostly everything you know about Bitcoin nodes applies here. Nodes store a copy of the blockchain and optionally execute all of the transactions to confirm the resulting state. Run yourself a full node or light client with (first-party, Go) or (third-party, Rust).
You should probably go ahead and run all of these node clients with docker and some sort of persistent storage. If you don’t feel like running a node on your own, you can use a third party like . There is also a way to run a local node for testing and development, discussed later.
Yeah, all of the “tokens” that you hear about are just numbers in a distributed hash table with an API (aka protocol) to add and subtract. (that’s 38 lines of code, half of it comments and whitespace).
Go ahead and read ; you’ll see that it’s just a contract (Crowdsale) that interfaces with another contract (MyToken) which is just like the basic token contract linked above. It’s not magic.
People are using tokens for a variety of uses, and you’ll quickly see that imagination has no limits. Tokens are frequently used to incentivize interaction with a protocol, prove ownership of assets, proof of voting rights, etc. .
, which is a good read.
Everyone started defining their own protocol for interfacing with their Token contract and that got old pretty quickly, so some some people got together and . It just says “hey, support these function signatures and we’ll all have a much better time”.
defines a non-fungible token standard. Non-fungibility means that each token is not equal to another; one “token” can be worth more or less than another and have unique properties. The best example of this is something like or .
* there was also but its ideas have been merged via community consensus into ERC721
If you’d like to programmatically interface with contracts, there are various Ethereum client implementations. For JavaScript, , , and are popular. For golang, the abigen executable in provides go packages for interfacing with contracts. At the end of the day, though, it’s just a standard JSON RPC API, so you can always write your own adaptor for the language of your choice if once isn’t available. Some client libraries provide convenience functions as well, beyond simple function execution.
Run a local ethereum node for testing and development with (previously called ).
Truffle (written in Node) has the most developer adoption and seems to have the most active development; follow the guide to get up to speed.
(Node) has similar but different ideas for how developers should structure projects.
(Go) is very similar to Truffle.
(Python) is an actively developed python framework that satisfies the same niche.
When you first start playing around with contracts, you should avoid using a framework until you understand the value it provides, much in the same way you shouldn’t start learning how to write HTML with rails new . The easiest thing to do at first is use to play around with the language and ideas.
Sharing is caring, so . Using ETHPM, you can inherit from or link to other well-known contracts and libraries, reducing code duplication and ideally providing a strong foundation of good contracts for future development.
for more information and background.
— The primary Ethereum testnet using Proof of Work. This network, because of low computation volume, is easy to DDOS, split, and otherwise mess with. It was recently revived and is usable again after being temporarily abandoned after a spam attack. It consistently has a very low gas limit, for no good reason, so many contracts will fail to deploy on this network.
— A parity-client only testnet using Proof of Authority which advertises immunity to spam attacks and a consistent 4 second block time.
— A geth-client only testnet using Clique Consensus, which is therefore more resilient to malicious actors, despite the low computation volume. (my personal favorite)
You can also run your own private Ethereum network. The go-ethereum team built to configure a full network, complete with custom bootnodes, genesis block, and consensus rules, which is what powers the Rinkeby network. You can also run your own infrastructure, perhaps using or . But you probably won’t need to run a private network any time soon.
A wallet is one of two things: it can be 1) a fancy interface for creating and sending transactions using your account (i.e. MyEtherWallet) or 2) just a smart contract (which, again, is just some code) that sends and receives ether; it’s not a native concept. . They can come in many flavors like multisignature, paper, etc.
is the first-party language for describing smart contracts. It’s the most popular and therefore has the most examples, references, and tutorials. You should probably learn this one unless you know what you’re doing.
Play around with Solidity in , a web-based playground for Solidity contracts.
LLL, which stands for Low-Level Lisp-Like Language, is exactly what it sounds like on the tin. While LLL isn’t advertised as the primary language supported by the Ethereum team, it still receives updates and is hosted in the .
.
is a high-level python-esq language that compiles to EVM bytecode. It has been effectively deprecated due to the numerous critical-severity bugs found by Zeppelin. For a similar approach to the language, see Vyper.
is also python-inspired and developed with a focus on security, simplicity, and no-surprises. It is still in development.
is a language designed to represent a smart contract as a finite state machine; your contract is a function of state and transaction and produces a new state. It is still in development.
Decompile Ethereum smart contract bytecode with or disassemble it with . This is a constantly evolving area, and new tools are being developed pretty quickly. The repo will contain more information.
ConsenSys has a beautiful repository of that you should understand at a deep level.
Before you deploy a smart contract that’ll handle real cash money, you should probably and have it pentested. If you’re handling RealMoney™, you should have your code professionally audited.
is a messaging system build into Ethereum. It allows dApps to publish small amounts of information for the purpose of communicating with each other in non-real-time.
Although it hasn’t been updated in a while, .
You’ve probably heard about the “” and how big of a deal it was. The exploit occurred due to a now highly-recognized re-entrancy bug; a malicious contract could cause a non-infinite, recursive call to the contract, causing it to update internal state incorrectly. In the case of the DAO contract, this meant lots of money being sent to someone that should have only gotten small amounts of money. The community voted as a super majority to hard-fork the network and restore funds to the DAO contributors. This resulted in the birth of , where no funds were returned; the code that was written is the law of the land, and cannot be reverted.
is also tackling the challenge of designing a company that operates according to smart contract logic, with a focus on creating a company, rather than an organization, that can accept investment, handle accounting, pay employees, distribute equity, and generally do business as we know it today. They’re also building a beautiful dApp client to make using their protocol as easy as possible.
Check out the for a better understanding of how it operates.
Swarm is a decentralized storage network being developed within the Ethereum ecosystem as a first-party project. Read , but the gist is that they’re basically the same except they have slightly different philosophies and use different protocols under the hood.
(InterPlanetary File System) is a protocol (among other things) for distributing files. Think of it as a filesystem using the ideas behind bittorrent and git where data is content-addressable and immutable. IPFS stores information using a data model called , which you can learn about via the links below.
is Protocol Labs’ effort to create a distributed market for storage on IPFS; aka, an incentive layer for providing storage to the network. The FileCoin consensus protocol (mostly) does away with wasteful Proof of Work and uses Proof of Replication and Proof of SpaceTime (yes, really) to ensure that a piece of data is replicated a certain number of times and is stored for a specific amount of time.
You should , the , and the .
While FileCoin isn’t yet released, you can use the existing IPFS storage network to host the html/css/js of your dApp client as well as use it as a database using something like .
is a decentralized prediction market (well, two markets) that lets users wager on real-world event outcomes. On one side, you have the prediction market where users trade tokens to indicate belief in a specific outcome; once the outcome is realized, the winning tokens have full value. To facilitate this, you have the decentralized oracle protocol, which creates a market for providing knowledge of real-world events,
is also a decentralized prediction market with a lot of the same ideas and concepts as Augur. and here’s a comparison:
is a distributed market for computing power in the same way that IPFS + FileCoin creates a distributed market for storage.
Skip the marketing talk and for a better understanding.
The is creating a protocol for trading tokens and a dApp that uses that protocol. Developers can build an exchange (technically a “relayer”) on top of their distributed application (aka smart contract collection) and users don’t have to worry about trusting your app to settle trades; settlement is handled on the blockchain. The 0x protocol is designed to use off-chain third-parties (those “relayers”) to broadcast trades and manage order books (so orders can be created/updated/deleted without sending a slow & costly transaction to Ethereum) but use Ethereum for settlement.
They’ve started by implementing (previously 0x OTC), a dApp that uses their protocol to transfer tokens directly between users. You can . They launched the contracts to the mainnet and are working with the community to build out Relayers.
Skip the buzzwords and .
ConsenSys is working on , which is very similar, but focused on communicating intents (rather than signed promises to trade) over “indexers” and then having the order be brokered p2p. .
is a protocol (and a set of smart contracts implementing that protocol) that lets you create a token that 1) prices itself based on orders and 2) provides instant liquidity by holding another token (like Ether or any ERC20) in reserve as collateral.
The is a DAO that is managing the . Dai was recently released and is currently stable; managing to keep its value during two notably turbulent periods. There are various other Stablecoins being attempted, namely Tether, Fragments, Basecoin.
attempts to solve this issue by 1) delivering the data to your smart contract from an external source and 2) providing a proof that the information is from that source and unchanged. So if you trust random.org, you can use Oraclize to provide your smart contract with a random number.
functions as an oracle for transactions on the Bitcoin blockchain, meaning you can write smart contracts on Ethereum that respond to events on the Bitcoin blockchain. For example, you can let people pay for your service in BTC and a smart contract can verify that the payment has gone through and then perform your service.
They manage , a set of vetted smart contract best practices that you can inherit from and use in your own dApps. Check out their for a great learning resource. Honestly you should probably just read every contract in there.
They’re taking that code reusability concept a step forward and creating . Ignore the term “OS”; it’s not an operating system in a classic sense. zeppelin_os is a collection of features, tools, and services that aim to provide a solid developer experience while maximizing smart contract security.
One part of the zeppelin_os is the “zeppelin_os Kernel” which is not a kernel, but is actually a set of well-known smart contracts that act as . They’re and can be independently upgraded in case of security patches. Because you’re including less code in the contract itself, deployment costs less gas and developers cut down on code repetition.
* sidenote: I’m really not a fan of people adopting these already well-defined terms (like OS and Kernel) and using them to describe things that aren’t what we already understand to be an OS and a Kernel. It just adds more confusion to an already ridiculously definition-saturated space. Call your project what it is and don’t fall into the that requires blog posts like this to accurately describe what you’re working on.
is a decentralized registry of human-readable names to addresses. Plus one for a descriptive project name. Various projects integrate with it, allowing you to pay .eth addresses or otherwise use it as a convenient lookup tool. You can also create DNS records that resolve to your .eth address.
is an attempt to decentralize digital advertising by monitoring “user attention” and distributing value between publishers, advertisers, and users, cutting out the middlemen.
and are tackling the problem of a decentralized identity system.
is a higher level approach to decentralized markets and communities. At its core, it’s a bunch of smart contracts governing how people post listings, search and filter listings, manage reputation within the community, manage payments, etc. It can be used to build markets like and .
ConsenSys (like, the word consensus, but as a company name; it’s pretty clever but I was pronouncing it as “con-SEn-SIS” for the longest time so don’t make that mistake) is a “venture production studio”. They’re an (honestly surprisingly huge) umbrella group that’s sponsoring development of a bunch of and . Of note, they sponsor truffle, Infura, MetaMask, Gnosis, and uPort.
was mentioned above regarding OpenZeppelin and zeppelin_os. They also perform smart contract audits and perform consulting work. .
is an impressive group of people working on IPFS, FileCoin, libp2p and IPLD, among other projects.
has a great overview of the landscape around Tokens, ICOs, and VCs.
