version 1.0 2011.10.12 NANOG53 weds morning session notes Donni starts the session off at 0634 hours Pacific time. Set your pagers/phones to vibrate, turn your laptop volume down! Survey giveaways--you must be present to win! Thank you for filling those out, they really do Ben Mason,...nope. $500 credit towards any MRV product! John Christoff? Yes, he's present! Second survey is newcomer survey Raphael...is present! yay! Thank you again to JI solutions for the gifts! Over to David Meyer who wings the next part of the presentation. ^_^; Cathy Aaronson, takes a minute to remember Abha, who left us 10 years ago. There's a scholarship, and one of the winners of the scholarship is here. She was a huge contributor, and we miss her terribly. And Cathy's the netgrrl person, so contact Donni introduces Geoff Huston, and reminds people who come to the mics to identify themselves! Geoff talks about v4 exhaustion and v6 adoption. This is not a technical presentation; it dives around topics, and is as useful for ARIN members as it is for NANOG members who try to make it all work. mainstream telecommunications industry has a rich history, starting in 1836 or 1837, when wireless telegraphy started off. 1860, Australia, express boat took 90 days to bring news. When telegraph arrived, news only took 2 days; it transformed socienty. Each telegram cost 3 weeks wages; and were staffed by people who spoke Pashduan, so bit error rate was around 40%. By 1984, ATT in US was worlds largest corporation; this industry is massive, it dominates the world; we have created a massive industry. By and large, we've had rich history of success. Mostly. Remember ISDN? And ATM? Example of why design by committee is hopeless. If you're running it still, you have his sympathy. And the stuff that succeeds, did so by design. SMS was never meant to happen...140 characters was never enough (look at twitter!) And IP was just an attempt at making OSI better; it was never meant to take over. So, looking at challenges we face today, how are we going to face them? Running out of v4, and making bridge to v6, it's not clear we'll do this with any accuracy, precision, or success. Do we need to worry? Surely it'll happen; when v4 runs out, pressure to move to v6 will "just happen". All we have to do is wait, says the mailing lists. Or is that a meaningless mantra, as an optimistic hope without substance? "inevitability of technological evolution" First success; "wires" -- new york was almost blocked out for wires for every user. move to virtual circuits was inevitable; there was no more room in the air for more copper. From virtual circuits, we moved to packets; each time the inevitability and acceptability seem obvious; it makes everything cheaper. Any packet switching protocol would have worked; packets are cheaper than circuits (well, other than appletalk). This was a progress of economics, making it cheaper for consumers each time. We need to move from IPv4 to IPv6, it's simple, right? Well, to get from here to there, there's a bit of an air gap. Normally our industry takes planning to excess; look at y2k! But in this case, we've let v4 run out and dribble away; and our thoughts of slowly moving from one to the other in parallel has vanished into the dust. There's no more v4 to run in parallel. So, now we need to get creative; but asking software developers to get creative is going to get ugly. We're going to have NATs, Carrier Grade NATs, etc. We're going to have to do new things with content delivery networks, when end-to-end no longer exists; the network won't help anymore. When we really run out, the invention of the 1980s will come back, and application level gateways will come back. The dream is that this is temporary; we'll build it, and then rip it out on Thursday when it's all done. This is the first time we're requiring the carriers to make massive investments in their infrastructure. Revenue per user in DSL is $2/month; if they call your helpdesk, you've lost revenue for the year. If you buy equipment, you're not going to rip it out; you're going to keep it around, keep it working, and try to realize the value of investment over as long a period of time as possible. It won't be a transition; once you spend the millions of dolllars, it'll be part of your business planning. It goes from being a transition to a black hole, with radiation spewing out at random. Risk is that we may come out in a completely different direction. Is this the case that v4 in some way could be made to last forever in some crippled way? Could you really make IPv4 last forever, if your arm was twisted? How much v6 is out there? Really out there? If you look at ASNs that transit and originate v6; in v4, 16,000 are transit, 29,000 are stubs. Almost half, 39% of those folks are dual stack capable; transit industry has the message, and it's actually easy; you have technical capability, your users are ISPs who are technically competent. Other side of equation...most of you have a v6 stack in the silicon ready to go; only ones who don't in the room are the ones running v6. Half of the internet today, when given a v6 literal, with no DNS, really do their best to connect to it; rest are running XP still. :-P Google graph over 2 years of v6 users; barely moving to the up and right; .1% to .35% during v6 day. He sees same thing from his APNIC measurements. Between transit and the device, there's a massive air gap, and only .3% can jump that gap. Where's the brokenness in the structure here? It's the last mile access. The operating system people have it right; auto tunnelling sucks, so it only gets turned on when it actually can reach v6. Excuses why the access networks don't support IPv6 break down to that we operate in an economic and business regime that makes provisioning v6 an unattractive investment option; and we're too broke to just do it for a lark, given the revenue from a user is $2/month; your revenue margin won't let you take a support call from your customers if you turn on IPv6. How could you make v4 last forever? seven layer stack in 1970s...won't find it in Google, since it's before 1980. :-P telcos discover that people only pay for services that they use. You pay for a service not based on what it costs, but on what it's worth to you. So, telcos used mountains of money from vertical integration (900 numbers, faxes, etc.) to subsidize the fundamental costs. We deregulated; so the money isn't there anymore to subsidize. But since the money is in content now, none of the money can be shuffled around to cover it via structural cross subsidization. ISPs aren't ready yet to be commodity carriers yet; ISPs don't have money to invest in v6, but boy do they want it. In economy, people pay you because you're an access provider. You can't bias the picture due to net neutrality; you just get money from the user. Now, with address exhaustion, you're running a constriction point; you're rationing access based on ports. You get one port...unless you pay for more, in which case you can get 20 ports. If you don't like that, you can buy rack space in their network to host content. Suddenly, you're the toll gate, and you can start charging for passage across it. In a world of scarcity, net neutrality is a myth; you have no addresses, it's no longer possible. Now, everyone has to pay to get access to my customers; otherwise, that aperature will work against you. Will access providers do this? It's not just feasible, it's happening now. That's the current play that's taking open networks, and seal them, so even devices aren't open, let alone the networks behind them. We are recreating the world of the 1970s with depressing accuracy. :( We're looking at the air gap; transit is there, end users are ready for v6; how can we end up with one network that is open, and doesn't require permission from every NAT gateway along the path? How do we get to a network that doesn't regard addresses as the new aperature and currency? This is the really hard question; in looking at our efforts as a community, we're failing; right now, 0.3% isn't a figure to celebrate. Right now, we're moving at geological rates. We aren't doing this very well, are we? We focus our efforts on patching v4 along. Challenges: internet is child of deregulation, and the environment has been competive, and fostered incredible changes. Rapid, fast change due to deregulation and competion. Internet as industry; there are many different views, content, carriage, asic makers, etc. everyone is competing against each other, trying to optimize their own perspective. Every approach will be explored by the industry, inevitabley. there is no monopoly driving it, there is no plan, and no regulator to tell us what to do; there is only market pressure, and nobody knows how they will play out. that's the problem. v4 exhaustion timeline...second challenge; we're not all running out at the same time. APNIC graph is pretty much flat at the last /8. other regions have 3-6 /8s left; if you do various graphs; RIPE, greeks have done you a favour, moved runout from March to June. ARIN is surprising; 350 million people, those six /8s will run until Oct 2014 at current rates. Rest from world, other than Europe, will last a while; if you can persuade greeks to default in the right way, that might even move to the right. This industry has a problem with reality; network world talks about v4 exhaustion as a "future posibility"; for APNIC, it's history, not future. people don't seem to believe it until it's actually *on you*. It's a network; we're all connected; what affects me, impacts you, and vice versa. Industry has a massive credibility problem; it's very good at telling myths to itself, and then believe them. Regional diversity is about to rear its ugly head. If you want more addresses in Asia region, answer is simple. "No". there aren't many sellers, even if you want to buy. The silicon industry doesn't care; they're going to sell hundreds of millions of iphones, android boxes, etc. So, CGNs will be installed by truckloads. Then next year, Europe will start to see the same thing happen, and they'll start investigating that dark, icky, IPv4 forever track. Each of us will explore this problem at different speeds, in different places. Things will start to look different in different market pressures; no longer one network. We'll solve our problems, each on our own; but those different trajectories will result in different solutions in each region. Human myth; we can make long term plans. We keep talking about long-term transition plans. Think about your long term plan in 2006; are you still committed to it? What about your long term plan 10 years ago? Can you even remember it? The longer the transition, the more likely you're going to forget it was a transition; you'll lose the plot, and get distracted by the current issue; we don't maintain trajectory over long periods of time. How will network fare? Badly. We'll head in different directions, and v6 will fade into the distance; and regional diversity means we're not going to come out of this in the same direction. The last 10 years were great; NANOG and ARIN are succeeding because we've been having fun; it's exciting, fun, we like it, and customers like what we've done for them. We want more, as do our customers; but we need open, acceptable networks to make that continue; and IPv4 isn't an option for that; IPv4 is just going to give carriers a way back to 1970, with addresses as the aperture. So, how do we do this? No idea. Only 0.3% is going anywhere useful; the rest is jammed in that transitional grey bottleneck. no matter where you look, there is confusion, not clarity. We don't know where we're going. 3 thoughts: solving your problems alone isn't enough in networking; you're connected. You have to think about a common interest. There are 7 billion others out there; you can't solve a local problem at expense of everyone else, or there will be no more network. We built the 1970s intentionally, and the 1980s were a nightmare. We have to figure out where common interest lies; because otherwise we won't have an open, neutral network for us or our children; it's everyone's problem. Too many men in this industry. No woman would ever want pain prolonged. Pain is terrible; scarcity produces pain and undertainty. If you prolong scarcity, you prolong pain and uncertainty. Scarcity and fairness are not synonyms; the longer you increase the levels of uncertainty, the more wacko solutions people will build just for themselves. The longer we prolong this, the lower the probability we come out with one network. Stop trying to ration out the last few addresses; you're working against your long term interests. The sooner we get on with it, the better. The longer people are diverted by the thought that there's more IPv4 addresses, the more wacko solutions will get put in place. The longer we prolong the illusion, the worst the outcome will be for us all. Q: Bill Norton--at what point would you say IPv6 has failed, and we need to work on something to replace it? A: That is not an option. We don't have time left. If we lose momentum of open network now, the only thing that will recreate our current model is if something comes along that is a thousand times cheaper. If we end up with walled gardens again, quantum networks and neutrino networks won't save us. Q: John Curran, ARIN--final couple of slides; it's not worthwhile to try to conserve those final addresses; we issue in 3 month blocks right now; if we issue in 12 or 24 month blocks, would that help? addresses would still get used for prolonging v4, in the ISP control, rather than ARIN/RIPE control. A: his view is that this is prolonging the agony. rationing is a strange outcome; once you ration, everyone gets less than what they want; and you end up pissing everyone off. this is industry self regulation. think about the entire picture; don't perfect scarcity. bring it on fast, let it happen. Q: michael sinatra, ES.net; when we circle in transition, and go into CGNs; van jacobson, getting rid of addresses entirely? could we converge onto something else? A: architecturally, yes; it's the business side you have to think about. Akamai got into racks for free, because at the time providers couldn't sell it; but once we hit scarcity, you won't be able to get into those racks at all. Q: vint cerf, google; you mention massive expenditures to implement v6; google hasn't seen huge amounts of cost, it's just taken some time; why would a massive investment be needed? A: help desk calls of customers; $60 to $65 to answer a call; if you take 3 calls to set up v6, you've lost everything; it's not in CPE, it could be elsewhere in the network. Q: economic arguments are the only ones that are going to win here; scarcity creates opportunity for networks. what about routing table slots, do we charge for those? What economic arguments can be mounted that persuades networks that it is in their interests to build the network we want, vs the one they think they want? A: That part of the puzzle is what is called a market failure; market forces are not pusing in a direction that is in the best public interest. So, we may need a force to push the market in the direction of public interest; seeing legislation around network neutrality; seeing that it doesn't operate in a complete vacuum. So, we may need persuade the regulators that even though ATT and MSFT weren't great successes, this time around, we really do need some help. We're going to need more stakeholders than just industry. Q: Mike, Dyn, at his house he runs v6 in parallel with v4; could we get more consumers educated on why v6 is better, and have consumers voice to their ISPs? A: don't think it's plausible, and don't think it'll work; look at analog to digital conversion; ability to make entire world change cut over was a failure on the technical side; finally came down to just "it won't work anymore, get a new one." Same thing for v6; for geeks maybe, but not general public. Q: Owen Delong; there's a runout dispersion that happens; the longer it lasts, the worse it gets, is that right? A: it's not a case of being worse; it's that the solutions that come up in different regions get wackier and wackier; china is coming up with different solutions from what europe is asking for, and will be different from what folks in ARIN region will want when their turn comes up. And since each area asks for different things, the chances of interoperability drop lower and lower. Q: so, what do we do to fix that? issue small blocks over longer period doesn't work. Issue smaller blocks over same period of time don't seem to work. A: he offered observations, not solutions. Q: Alain, Juniper, CPE question; content provider has v6, his laptop has v6; but at $100 per CPE, and a billion CPE, that's $100 billion we need; should we ask for government bailout? A: GFC mark 1, GFC mark 2 came along; if we'd asked in 2006, we could have maybe done it. but now, governments are broke; you'll get all sorts of things from them, except a check. Andrew Dul is up next to talk about economics of IPv4 market and IPv6 deployment why talk about economics at a NANOG? he took some time off, and did some sudies in economics; there's not a lot of literature about how economics underlies this, so he's trying to shed some light on how these principles affect us. The money that flows around this industry is large, and if we push against the economic forces behind it, we're going to meet resistance. brief history; we used need as measure of who should get address space over time, morphed into RFCs and RIR policies RIRS saw need for transfers after exhaustion transfer policies were created to allow economics to effect transfers pros and cons of a market? pros: bring economic forces into arena, let money decide where address space goes; could put addresses into organizations that will use them to generate revenue from people. con: could delay or deter v6 adoption as Geoff noted; if we don't think about how market will affect us longterm, the mindset that we can solve this using a market means we don't get the outcome we desire. chopping into little deaggregated blocks has implications for routing tables and network engineers. v4 market to date; what do we know? large transation, MSFT spent $7.5M on 660K addresses. ARIN released list of directed transfer policy blocks there's about 3 /16s beyond MSFT that have been traded since policy went into effect; not a lot of movement. some address brokers are showing up, and are listing their prices on their website. 15 years of address consumption graphs; it's up and to the right, it's increasing every year. APNIC has really driven usage of address space in past 5 years. what underlying effects are in the marketplace driving forces? world GDP vs address consumption; people get more mobile devices, need more addresses. As people get richer, we need more addresses. v4 vs v6 deployment cost chart; reliability, hardware cost, people cost, training, suppport, and address space cost. cost to deploy v6 is much larger, need newer hardware, more training, etc. Harold Hotelling did work in 1930s (oil reserves); won't transition until cost of old resource exceeds cost of new resource. v4 vs v6 costs over time; start with v6 at 2x v4, and 10% delta, down on v6, up on v4 per year as a theoretical model, get to a point where v6 makes sense. substitutes: coke vs pepsi; items that are swappable. IPv4 vs IPv6; IPv6 is not a perfect substitute for IPv4; need transition technologies to make it work. network effect; the more people there are somewhere, the more valuable it is. We're all on v4, so that's the place where the value is. NAT is other issue; from economic perspective, it's a good substitute in small scale, as homes have shown for years. what about dynamics of market? v4 market will never be perfect competitive market; there are transfer/switching costs. demand grows, supply is always limited. efficiency graph showing RIR runout, we're not completely out, we can reshuffle some space. If we're 70% efficient in 2012, that's still like 800 million addresses we could reclaim, which would take us to 2016 to hit 100% utilization. when v4 hits 100% efficient utilization, we start having to deply v6. In 2021, 50% of all addresses would have to be v6 if we stick with current demand numbers. Game theory; how organizations make decisions; coordination games with different payouts depending on decisions made. 6rd vs dual-stack, write 4x4 matrix showing choice each makes, and the outcome. it's a problem with v6 transition; no clear path forward for transition; too many options; what is the plan that we have? Can we all agree on one that's the best way forward? too many standards to choose from. need to coalesce on a solution that's best for everyone. Can't find a solution everyone finds palatable from a long term persistence. Economic committment; large players choosing one protocol and putting stake in ground may get everyone on bandwagon with them. 3 hypothesis large scale v6 won't happen until v4 exhaustion is complete transfer prices in different regions will vary based on different rules and restrictions inter-regional transfers could change that How much is IP address worth? $150-1100 per v4 address, for guaranteed unique address based on models. like it or not, economics are shaping the decisions industry is making; those forces are with us, and need to be accounted for. what can we do as engineers and operators to change the economic landscape to encourage IPv6 adoption? Questions? Q: david farmer, UofMinn; like that he focuses on econimics; he focused on cost only; but that's not only economic driver; for enterprises, yes. But for SP networks, revenue is also a factor. cost may be less of a factor than revenue; so how does that play into the model? A: right now, we're in an area where revenue is really restricted, which limits how you can model this out; could perhaps change model of how this works if we look at how revenue could offset costs of transition. Richard Barnes from BBN is up next not a vendor, not an operator; hangs out in IETF; talks about emerging tech about geolocation, as a bookend to Dave Ward's talk about exposing information from network to application could result in good outcomes. Applications want to know where users are; foursquare, yelp, weather, restaurant finding guides, etc. content localization, advertising; Google, Yahoo, want to know which version of the site to serve; it can be critical for content providers to make sure right content is displayed. Hulu, others have to also be aware of legal restrictions on where content can be shown. Fraudulent use of credit cards online could be detected this way, depending on circumstances. Right now, these depend on scavenged data; they halfway work, but cause some operator headaches. Can we provide ways for SPs to privide the information in a better way? Currently, 3rd party boxes reside off the network, IP geo databases, scrape data out of whois. Try to present data based on region, but registration can be from different areas. Network triangulation helps, but network geography seldom maps to real geography. Other case are first-party location services; GPS running on a cellphone, for example. wifi, cell databases have shown up, where device can report on what it sees, that is looked up in a database, and tells you where it is. final option is to just ask the user where they think they are. Most 911 services for VoIP work that way; ask user where they are. that's pretty much state of the art at the moment. IP/geo databases have a lot of variability, from 1km to 100s of km, to entirely wrong country. worse in mobile, fiber and TDM a bit better. And databases have little v6 data. mobile space; if you have GPS, you're in good shape; but database lookups are iffy at best; no central registry of wifi lookup. fidelity in more open regions goes down drastically. fixed networks require user registration. when things go wrong, how do we fix them? right now, mail to NANOG mailing list seem to be the most successful way to fix them. http://nanog.cluepon.net/nindex.php.GeoIP has some contact points; but we still see 1-5 emails to the list about it. Is there a better way to do this? Provide a channel to ISP who knows where their infrastructure is, to provide that information to geoIP systems. some maturing technologies, but nothing standard yet. couple of avenues to consider; Feeva/Bering Media transparent proxy in router chassis inject encrypted location into HTTP header users can only request privacy out of band users can't detect when location is set only works for HTTP; only sent with request, nothing back in reply showing it's in use. not a good long term model. GSMA OneAPI in mobile community suite of REST HTTP APIs, provides geolocation, charging, SMS gateway resources. uses authorization through OAuth delegated system. no knowledge of which service provider an app needs to look the user up in more useful for static items like phone numbers. Few GSM items in API that would need to be squared away. ITEF goal to provide IP geolocation GEOPRIV create a location service for the internet simple to wrap around existing location resources scalable to the Internet (dynamic discover) explicit considerations for user privacy need to be built in. discovery: DHCP+NAPTR FRC5986 fallback on NAPTR in reverse DNS (in draft) protocol: http-enabled location delivery XML over HTTP to discovered URL first party HELD ask the device itself. third-party HELD request of a database, or to headend for device, etc. as a web service, you can see who is asking; is it the device itself? If it's the device itself asking, tell it everything; random third party, not so much. RFC6280 talks about managing privacy for user location information. You can provide the user whatever information you have about them; it's the person themselves, they should already know; others, not so much. interaction between subscribers, providers, and regulators; technology is geared to provide hooks for all three to interact. deployment and next steps; typical new protocol chicken and egg problem. clear interest from application side: google, maxmind, quova, mozilla potential FCC/regulatory interest from NG911 several open-source HELD implementations perl, php, android initial work being done on geo-whois store URL for geolocation contact for an SP prototyping in RIPE, proposed in ARIN, some talk about ARIN during the meeting RIPE geo-whois prototype has coordinates directly, rather than URL, might have privacy issues. act now prototype a location service look at what location-relevant resources are in your network MAC address to device, SNMP tables, etc. info is probably there, just needs to be pieced together. Will talk about adding this to whois database during open policy hour. rbarnes@bbn.com if you have question. Q: Sandy Murphy, Sparta--Geoff talked about balkanization of internet; this provides geolocation information; would this work as well in cases where network itself needs to know? A: there are some bits of information that would be relevant to CGNs, like the source port as part of the data tracked. Haven't thought about providing network sservices. Q: Welty, Switch: as a user, would like to be able to change the location of something; doesn't always want his web pages localized based on his geolocation; that way he doesn't have to call support when he dosn't see what he wants. A: would need a way to express privacy controls, on what to tell or don't tell; either hide, or provide something completely different. Q: Mike Joseph, Google--problem with getting data from user is whether you can trust it; filtering for regulatory data means that you can't trust the user to put the right information in themselves. A: goal here is to provide a channel for the operator to provide input; nothing says this would be the only source of information, and Quova, MaxMind, would still be around to vet the data provided by user and service providers, in anti-fraud system. Q: Andy newton, ARIN staff; RIPE demo puts coordinates into whois; is that coordinates of network, endpoint, ? A: on a network, so it's an approximation. Q: for a URL in whois, would the goal be to give it a prefix, an IP, etc? A: XML gives richer specification, so you could give network, prefix, polygon, or other data to that query. Some concluding comments from Dave Meyer and Betty. Dave Meyer's first NANOG was NANOG 4, in Ann Arbor at uMich. It's worth going back to look at agendas for early meetings, many topics are still the same. :) Thanks to Comcast for hosting, they deserve a huge hand! number of great other sponsers for breaks, bear and gear, etc. We just did nanog elections; board of directors of newnog/nanog did a wonderful job supporting us; please keep doing what you're doing, it's working great. Steve is terming out as chair of the board, he guided us through the transition; he's been around for longer than Dave has. We also have a program committee; the people term'ing out; CJ, Jim Cowie Barry Greene Mohit, Chris Morrow Donni Sonia those are the people who brought this to you; consider running for one of these slots!! It's definitely worth doing! Thanks to AMS, the company that is helping us put this on; Chris, Kristen, and Anabelle. Steve and Verilan crew doing the network, thanks! Speakers, thank you for taking the time to develop the content and give the talks. And thanks to everyone for coming out and making this a huge success!! Call for presentations will be up for San Diego meeting after this; think about talks and tracks you'd like to see!! Over to Betty now; thanks again to everyone who helped make this a success!! And thanks again for Comcast for hosting us for the second time. Statistics: 581 registrations 162 newcomers 494 from US 15 germany 12 UK 10 canada 26% ISPs this time around. Prepare for NANOG54 in San Diego We remember Abha Ahuja who passed away 10 years ago in October. There are two scholarships in her name, one for NANOG, and one for AFNOG; information about those is available. Election results should be posted on the page as well. All 4 board candidates have been elected to the board. Bylaw corrections and language cleanup has been approved. Section of bylaws talking about committee structuring was also approved. program committee selection, and program for NANOG 54 is next order of work for them. And Dave Meyers is back up to give a HUGE thank to Betty for all the work she's been doing to make the meeting a success! And with that, the meeting really *is* done. :) video stream ends at 0841 hours pacific time.