version 1.0 2011.10.10 NANOG 53 notes -- morning session Dave Meyers kicks off the morning meeting at 0930 hours eastern time; hope everyone had a fun time at the social last night. Betty is up next to talk about Steve Feldman, who has been a dear friend, and extremely critical in bringing NANOG from where it was years ago, to where it is today, during the evolution, from being a part of Merit, to be where it is now, as a community owned organization. During the transition, both a Merit appointed board leader, and a community appointed leader were needed. He has been selfless in his time, putting in 8-12 hours a week in making the transition a success. The award is presented to Steve in recognition of his years of service. Steve says thank you to everyone on the board, and everyone else out there, he couldn't have done it without us. And on that note, he puts another call out for participation; please do become a member. Real reason he's up there is to introduce the host, Comcast; the host role is huge, not just the party (which was great!), but also the connectivity, the people to run the registration desk, etc. So, Kevin Mclearney is brought up to the podium, and presented with an award as the host; Kevin also calls out to Ren Provo, who was key in making it all happen. Kevin thanks John B for the lab, Betty Burke, Ren Provo, Dave Meyer and the program committee, and Tony for leading the network connectivity; two years ago, they did 100G at the Marriott, it was pre-standard, and caused a lot of challenges. Today, still trying to do 100G, and still having problems; it's a sign we're moving too slowly. Comcast carrying 4Tb across their core; they really need to be at 100Gb to carry it. In 4-5 years, their city pairs will be at 5-6Tb. Look at your five year plan and understand where you need to be, and push your vendors to be there for you. So again, welcome, thanks again to Ren, and now onto the main event. Dave Meyers encourages people to sign up for one of the committee leadership slots! And now over to Dave Ward to kick off the morning. Usually talks about router design, but this time will be talking about programmable networks. He does note that large interfaces *are* available from vendors, and perhaps customers could increase their uptake of them. Programmable Networks are SFW application developers are wholly interested in the services they can develop for their users; their interest is to get the best experience for their users. application developers don't have a good understanding of the weather report for the network; they don't know what the bottlenecks are, and what the experience actually is. The first developer who connects the application to the network paths involved will have a winning lottery ticket. applications today just blindly probe and guess the network; either using pings, probes, whois information to look at targets; looks for either the best place to deliver content, or best ways to deliver it. The network has the information, if only they had the way to look at it. network has fundamental information on where the end user is, what they want, how much they need, etc. It's not just programming the network; it's a bidirectional information sharing that's needed. The application needs to inform the network, as well as the network informing the application. [the video stream breaks up for a few moments] What is possible in this new world? understanding the access link lets you understand the quality of service you can get out there. How do they actually work together? a number of high level conversations going on; they all attempt to solve all problems, whether it's virtualizing datacenters, building service profiles for customers, populating content caches, they're all attempting to duplicate connectivity between two different points. But remember we're not going to reinvent the internet by doing this. signalling PVCs ala X25, ATM, etc. isn't going to scale. [video stream drops again] so, we're looking to augment what we have, not recreate it. the challenge to doing this is there is a zillion different stats out there. openflow, etc. there's been a trend over past 10 years to communicate the information within the protocols we have. First is path computation elements; first done for interAS traffic engineering Once you have a server you can program, with endpoints, you've implemented soft PVCs. By using technology we're familiar with, we Generic application extensions to IGP; put app data into IGP; not wanting to overload routing protocols; they want routing to converge faster than application; so, like multi-instance IGP, you can pass application information separate from routing information, to keep network converging quickly. Signal best location for application via proximity in the network. BGP with traffic engineering extensions; you can pass information in multi instance mode so it doesn't interfere with the network layer. Arguments about which IGP or which protocol are the right one to use aren't helping; we don't have the resources as a community to do them all. Going with BGP and its hierarchy makes sense. We put more information into BGP; we put CDN information in, both users that are reachable in a geographic area, or content that is reachable in a given geographic region. We have the fundamental building blocks to pass that information up to the application. How do we make it happen without breaking the system? We can do it without breaking the internet; at the bottom are the network APIs, they come from fundamental constructs of what is available today; fundamental idea of radius and diameter of the network; it's straightforward to take data from the network protocol and make it visible via an API. Current applications are thinking fundamentally differently than the network side; our goal is to get the data out onto orchestration platforms for the application developers to use. New attrributes will need to be carried, such as delay. The mere fact that we aren't passing delay data for our calculations...the time has come where we need to talk about this again. Delay, variance, loss on a path. Realtime topologic and weather needs to be available to the orchestration platforms. Openflow is the only platform so far that allows this; it's a current shiny object being discussed; it has high utility.. [stream breaks up again] How do I get from where I am, to where I want to go; Where is it, where is my object, how do I get there so I don't have compete with everyone else 4 fundamental building blocks Orchestration and development platforms traditional IT emerging network function specific emerging service specific new provider based development platforms OSS, VSS, network functions, service functions, and provider functions are the basic categories Something about reaching half the world's population and 2/3 of the world's GDP using these. networked application examples... what is a service engineered path? system for providing pre-engineered traffic pathway between source and destination with known characteristics [video stream goes unusable] Example: content request routing 3 datacenters. application layer using plugin to try to find the closest location to get the data, the shortest path, the fastest set of servers to use. Next example bandwidth calendaring; don't want to push all the network state up and down the stack. Use the pieces that are available to pass the information you need. As an operator, you want to be able to pick loop free paths that meet your SLAs; you have network APIs being exposed to the platform, Cloud bursting, openflow most common use case, last example is social networking; advertising, gaming, Am I close to my limit? By giving this information, you get active information that improves the experience. [audio stream drops out again] Unleash the potential application can help the network drive a better user experience. Q: V. Georgia Tech; can you comment more on money flows, who would pay whom, to scavenge bandwidth? A: Zynga would want to get into collaboration with networks; if they can deliver a better experience, they would want to have money flow in that direction; but he's not an economist. Q: Patrick, Akamai When will 100G interfaces cost less money than Nx10G? A: He's an engineer, no idea Q: back to talk; this rings gigantic alarm bells about network neutrality; services that can pay for better performance, or a telco could prioritize their video offering over others. A: It can be done today already with existing tools; this just makes it easier. Q: Jeff Stacks, blue ridge networks. This is really useful information from the network layer; why do you think the network would want to provide this information to the application? A: this is assuming the application already wants to do something, this lets the network tell the application do so in a more intelligent manner. This lets the application find out if what they want to do is even feasible; currently, applications use multiple TCP streams to try to get as much as they can Q: Randy Bush, IIJ, he gets the william jennings bryan silver toungue award; adding more complexity to the network to give more money to the vendor. :P what about security, and privacy implicatoins? active networking, we tried this 3 times before, and they died on the vine in the security and privacy realms. this needs to be thought of at the base of the architecture. A: security issue, no doubt it's clear he didn't fundamentally address it. Deep packet inspection that's out there already; he's completely sidestepping the issue, he's trying to avoid all the deep packet inspection fubar and let the application and network talk more directly. Q: Sue Hares, BGPer, Huawei; Dave you poopoohed this when doing application specific networking; what's changed his mind, did it come from SDOs, from customers; what caused him to take a switch from his previous position? A: The bolts in his...he teases his friends... PCIngnite, quick and dirty hack of BGP to pass information around. Q: Bett Watson, neustar A: VPN gateways have ifmap information, a way to spew information about what data is traversing a given VPN gateway; Q: geolocation presenter from wednesday, something about IETF, some technical specifics that will be covered weds at 11am. privacy, there has been some discussion, RFC6280, there has been some thinking and discussion about this already. A: Thanks! Thank you very much to Dave! Greg Hankins is up next; router architectures; haven't talked about hardware since the FIB limit discussion at nanog 49 100G, one packet every 6 nanoseconds, is very challenging port density vs complexity, covers basic router forwarding architecture DRAM vs packet lookup memory, and packet buffer memory. 150 million packets? chip-to-chip interfaces, serdes links, 25-28G links coming soon. packet memory access rates, lookup 1ns random reads needed yesterday buffer; 1ns random read/write needed yesterday today, 48ns random for off chip DRAM 20ns random for RDLRAM We want fast and big. He covers the various types of memory, and speeds and lookups involved with them. Tough challenge to get that much data through the read/write cycle of memory. Need better buffer technology than commodity DRAM. Key ASIC challenge; we're trying to do a whole lot of different things right now, looking at different labels, timestamping, OAM, [audio stream drops again] The more transistors they can put onto a gate, the more you can do; we're coming up on 5 billion transitors on ASICs now ASICS traditionally 2D packages; moving to silicon interposer, adds multiple chips on same package, eventually 3D packing, which gets much shorter interconnects, down to the 1ns lookup speeds needed. Q: linecard pitch for 3D packaging? A: very specific question, he doesn't know. Gives much lower power consumption and head dissipation if you can do this. chip interconnect signaling ASIC technology integration current process technology optimized for memory or logic. new packaging allows for both. right now, boards are completely packed, there's just no more realestate. Alternatives to FIB scaling optimize fib, only put what you need into the hardware. software FIB; put a default, and a few necessary routes in hardware, rest keep in software. openflow, lisp, other techniques. want to design hardware with a minimum of 7+ years of lifetime that supports multiple protocols, high density cfp2 100G Questions? Q: they're trying to fix the wireless, a note was just sent to the list about it. Break time! Back, Jason Keiburg, T-Mobile USA involving backhaul to meet bandwidth needs of new handsets, etc. mobile ethernet backhaul lessons learned why ethernet in the backhaul? customers use lots of data, TDM doesn't scale ethernet is everywhere (underlying tech might not be) Ran architecture moving to all IP (kind of) Radio Access Network (RAN) architecture how do you get from radio tower to MSO speeds and feeds in last mile TDM needs; few T1s for 2G, GSM 3G/UMTS, 4G/HSPA, multiple T1, IMA, ATM but they don't scale. So, replace middle with ethernet, you lose intelligence, less OAM, push intelligence out to the edge, so they have to do... less tunnelling, overhead build or buy? they're an over the top company; buying it either as ethernet or wave. He has 30 ethernet providers today, and trying to scale that is a challenge. will use licensed or unlicensed microwave if necessary, 1-5% of access. proactive and reactive challenges interface handoff specifications [audio stream drops out] 12-18 month buildout for TDM, ethernet builds go considerably faster. understand what it takes for each technology, don't get stuck with bottlenecks. QoS--do you do it on your end, does vendor do QoS? They aim for fast dumb pipes from vendors. consider domain sizing. Things will break; it's inevitable; know how to troubleshoot it; know how higher layer technologies interact with lower layers; few seconds for IP convergence doesn't help if cell systems take 30 minutes to settle back. Lessons learned use ratified technologies elan technologies; layer 2/layer 3 technologies who do you want to converge? If you have true diversity with dumb pipes, you do the convergence; if it's elan, over VPLS, they're the ones converging. Old pseudowire infrastructure won't go anywhere. will have to support it for next five years. Cost we pay for T1s, with 100mb next to it, pseudowire technology makes some sense. synchronizations local gps feeds, pull synch from it; Europe, synch-e is there; 1588 and GPS are their two sources. build end to end intelligence. BFD is now in hardware, it scales to the moon, support; a lot of big providers out there, parter with them; what are steps of triage, what information do you collect, what is the next steps; don't want both sides trying to learn on the fly. 2G/3G been around for a while. Will stay TDM. 4G has ethernet handoff, own bearer signalling. In the lab, looking at scaling challenges. ARP timers, 4 hours doesn't cut it anymore. directed ARP, keepalives troublesome architectures they've seen; RAN guys have to work with IP guys. Somebody has to support this; can't have different NOCs for TDM, ethernet, IP. Took a couple hundred TDM NOC people, brought them up to understand ethernet and IP so they can do troubleshooting, event correlation, etc. TDM and IP don't always mix; took almost a year to get them comfortable taking phone calls during outages. Training, correlation during events, solve as soon as possible; 10,000 alarms in a minute doesn't help anyone. theorize how things might break; bring it into the lab. kick tires see what services on top see. Device at edge; cell site router, you want a robust, intelligent device; they use BFD, need to be able to handle 1Gps on a 1 gig link at small packet sizes. Integrate OAM, don't put many serial devices. RAN is gospel; trust RAN first and foremost; next is routers, and then finally transport behaviour. Monitor; events aren't always actionable. ethernet statmux, lose visibility. CFMs measuring characteristics; try to paint a picture before the event, was it good/bad? RFC2544, jitter, loss, if you buy 50mbits, you expect 50mbs to the end site. scaling--think about the future; don't want to reengineer it when you need more more bandwidth. usually quite forgiving. Best practice; make sure design meets 3-5 year plan. scaling; if you're doing it 30,000 times, you need it repeatable; authoritative database, key data to index off, unique identifiers. Questions? Q: Anish Matta, comcast--no spanning tree in their network. 2G not going anywhere? Are they capping the traffic? A: 2G isn't growing, finite spectrum on the radios; they need a 2G fallback scenario for 3G fallback; so if they have to nail up T1s for it, that bandwidth is a foregone conclusion; but it can't go away until they sunset the 2G network. Probably won't go away for 5 years, until 4G with 3G fallback is out there. Thanks to Jason for that. About 250 members, with 600 people in attendance, expect to see a long line at the patrick kissing booth. Amie, centurylink to talk about 95th percentile billing. Instead of doing analytical forecasting and statistical analysis, now they can look at the actual data to analyze. Internet usage trends consequences. It's been increasing dramatically; 4 years ago, 3G; now 20G about an hour of netflix, 1G, 30 hours of browsing top 1% of users are downloading over 170G/month; like 5 hours of netflix SD a day. because of dramatic increase in usage, they see a trend in moving towards consumption based billing. smart phone providers are doing it, ATT does it with its 150G cap each month. Over the top video is the killer app now. 2:1, and it keeps increasing, as more and more people are watching. 7-11 peak 95th percentile billing, also called burstable billing, variable rate billing, alternative to flat rate billing or consumption based billing. Traffic measured every five minutes, 8640 measurements in a 30 day month, sorted from lowest to highest; throw away top 5% of values (top 432 values), remaining highest value is the 95th percentile usage for the month. Chart with the data showing raw data, average data, then graph area under the curve is the volume of data. pretty typical for video traffic, spike later in the day; they have to build to peak traffic, but 95th percentile billing is less than that. How close does the peak come? start with 95th percentile measurement divide by 95th percentile/mean ratio X/1.8 is average Mbps for month divide by 8 bits per byte figure out what the price per Gbyte per month is. peak to mean ratio of 1.8-2.0 you may end up paying more per Gbyte as a transit customer. For different peaks, same area under the curve. 95th percentile, in this case 24, mean is 15, goal is to get the two things as close to the same as possible; reduce peakedness, get better efficiency in your network; reduce peak to mean ratio, shift traffic to other interfaces, if spikes occur, keep them to less than 36 hours a month. Erratic, unpredictable traffic has an impact on costs; the more variable the traffic, the more expensive it is to design to. Area under the spike curve should be about 2% Internet is two sided market; centurylink is market with two user groups; content and users. Price per mb; think of your service; 33cents/Gb, but content provider may be paying 0.5cents/Gb; so consumer pays for bulk of the cost. by converting from Mbs to GB/month, you can see who is paying for what portion of the cost of the bits. Do the pricing structures we have in place accomodate paying for the infrastructure? Are they appropriate? Q: Geoff Huston; problem is as old as the internet. More traffic should equal more money; if you want more resource, you should have more money to buy it. US is one of the unique countries that ignored volumetric pricing, and went to 95th percentile; Mike O'Dell from UUnet started it, it took off in this country only, everyone went to it. 95th percentile assumes a certain peak to mean ratio, which volumetric doesn't. other blinding problem with 95th percentile is that it's a lie. Users can't replicate measurement from the provider; billing disputes can't be resolved. Both consumer and producer can deploy exactly the same devices, but if their time basis for their measurements, you can come up with two different, legitimate 95th percentile rates. Get over it, charge by volume! A: Yes, but backbone providers need some indicator of peak, so we can build to the peak. Q: no you don't, just charge more for more volume; your business models are structurally broken, and can't bill for "more usage costs more money". A: On the consumer end, providers play chicken, offer flat rate billing to get customers; it's difficult to design to that, and make profit. Q: Tom, have you done any similar studies on bit cost delivery? it's not static, it has been declining. A: Costs aren't coming down at same rate as increase in usage, which is going up 70% per year per customer. Q: Do you mean cost of bit service delivery for lifetime of gear? A: Yes, costs are not coming down as fast as the usage is increasing; there is a point at which they will cross. Q: So one side, the viable side, and other side is invisible to anyone but the provider. A: Elephant in the room is net neutrality, has their hands tied. Q: Jason, rackspace, Centurylink customer; they do shift traffic around as it gets close; do they provide incentives like utility grids to shift traffic around when it comes close to peak? A: good idea; that would let providers reduce or shift the peaks around. Q: Geoff Huston; why are hands tied with network neutrality? If your job is delivering packets, regardless of content, why do you care? A: metered billing has been accepted by FCC; two options are flat rate and metered rates. They could charge different rates for different types of traffic, and make more revenue, were it not for network neutrality. Q: If you didn't have network neutrality, they could start to charge people with whom they currently have no business relationship A: We'll have to talk afterwards... Thanks to Amie Next up John Curran, ARIN, who has decided it's easier to reject all your requests at once, rather than one at at time! [much laughter ensues] Quick talk about what's happening at ARIN the second half of the week. We have the PPML public policy mailing list; these policies will affect you, possibly years later when you go to make requests. If it's important to you, speak up! We have 5.7 /8s in ARIN region of IPv4. Substantial portion is from Interop block returned; but that's under discussion whether that should go back to IANA; so closer to 4.7. Also a proposal in coordination with IETF, an internet draft, to set aside a block of addresses for transition technologies; which could eat a /10. The rate of request to ARIN has actually dropped off; it looks like we'll have space available through end of 2012. Geoff Huston will have a longer talk about that on Wednesday. Of course, that's not terribly far off. Remember, if you're an ISP, you get space based on 3 months, you talk to ARIN a lot more frequently. Specified transfer; if you want more than 3months, or if you have particular address block you've been using, but want to clean up records. You can transfer address space if the other party needs it, up to 12 months of need. They don't match people up; there's an emerging market matching up buyers and sellers; if you want to make sure they are the address holders, and will pass a transfer with ARIN, there are ways of doing that. Parties who are interested in transferring space can document it, ARIN can validate the space, make sure they're the holders of record; again, this is an information service, not a brokerage service. There is still space via normal process. 9 draft polices, 3 active. global coordinate transfer policy, needs based transfer to other regions. need based transition block, coordinating with IETF, find out if that's technically useful. ARIN 2011-7, compliance, require accurate info. 2011-8, M&A and specified transfer can happen at same time. global policy, post IPv4 alllocation mechanisms. Show what you need, get more space; never a policy for giving less than /8s from IANA. So, if something less than /8 goes to IANA, what would they do it it? 2011-10, remove the single aggregate from specified transfer 2011-11, extend 12 month to all specified transfers 2011-12, set need to 24 months? 2011-13, IPv4 number resources within region; if they're registered in region, can only be used within the region; as the region changes, what happens? www.arin.net has the policies; participate remotely, on the mailing list, etc. Questions on any of the slides? Q: Sam Weiler...question on abstract...RPKI, want to tell us a story? A: ARIN has been working on RPKI; region it operates in is north america; problem with US is that it's a very litgious society; when you do something, you'll probably get sued, so you need to prepare to be sued. RPKI gives digital certificates assuring that someone has rights to a resource. If someone typos an ASN, and due to it, their routing goes off line, and they come back to ARIN and sue them because the sytem generates a bad token and blocked their network access. So, ARIN has to demonstrate non-repudation to handle lawsuits like that. Need nonrepudiation protection, bad actor detection and recovery; they are doing hosted RPKI, ARIN issuing certs on behalf of blocks registered to them, targetted for Jan 1st; later, will support 3rd party hosted CA, up/down RPKI, will be later next year. Going a bit slower due to unique aspects of north america. New LRSA is out there for comment under suggestion page; also a new policy development process, the PDP; both are in the ARIN comment and suggestion process. Q: Cathy Aaronson, there is a voting process; people at the companies should be prodded to vote A: good point; NANOG is doing voting right now; you should also participate in ARIN elections; advisory council works on policy proposals, lead discussions, advise board. for each ARIN member, there is a designated ARIN representative who can vote; that designated member is the only one who can vote. Need people to find their designated member, and have them vote! You can put statements of support for candidates on the web site. Q: someone from LINX--in RIPE community, RPKI policy put through proposal process; is it good for RIR to host cert authority? Within ARIN region, what opportunities are there for operators to speak up about whether it's right for ARIN to run a cert authority? A: open mike time at ARIN meeting, you can stand up and speak about RPKI, and speak at the member meeting; but it vests with board of directors to make the decision. Take time to fill out survey, right side of Agenda, one lucky winner wins seven minutes in heaven with RAS. LUNCH time!!