(version 1.1) 2010.06.16 NANOG49 day 3 notes Dave Meyer calls the room to order at 0935 hours Pacific time. Applause for the party from last night, thanks to Hurricane and ?? Over to Tom and Matt for the first talk of the day long-distance wireless to the farallon islands. http://nanog.org/meetings/nanog49/abstracts.php?pt=MTYwOCZuYW5vZzQ5&nm=nanog49 If NANOG would like to see a longer tutorial on how to do this, maybe a Sunday session, let them know. They've been doing network builds in crazy places for a while. stakeholders: Point Reyes US Fish and Wildlife Service California Academy of Sciences etc. About 50km outside of the golden gate, 30m to 5hrs transit time to get there, depending on how you go; helicopter or boat. They are right on edge of continental shelf. contain 30% of california's nesting seabirds. Getting access to the site is challenging; no pier, dock, or beach; you take a boat out around 4am to get there, then take a zodiac, which is grabbed with a crane to pull you up 50 feet and then swing you onto the rocky island. Islands are effectively in the middle of the shipping lanes, 3 lanes, north, middle, and south. Only 370 feet, but a good landmark to get you going on the hawaii/SF run. You can see top of lighthouse from SF, and original lens has been preserved at fishermans' wharf. coast guard replaced the lighthouse with a new establishment. Island is staffed 365 days a year; it's a critical point. Only way cell phones work is to climb to top of island, and only legacy i-band nextel or sprint service works. They have some early legacy 2.4Ghz data links, more down than up. Wanted a webcam, which needed more upstream bandwidth. PRBO, point reyes bird observatory is moving more into digital realm, the old paper diary is moving digital. Data is taken very seriously, the researchers document every interaction with wildlife. Design criteria requirements affordable (stakeholders are non-profits) weather/rust proof -- salt water, sun, lots of bird poop reliable--on-island IP & RF knowledge is limited Easy to fix--see above need simple LEDs showing status, clearly labelled cables, etc. site survey obstructions--pretty much clear line of sight access and security island and mainland infrastructure Island limitations 100 watts limit in lighthouse equipment had to be low power (donate retired 'carrier' quality gear would exhaust budget) limited antenna mounting space "right of way" US coast guard property management if they mounted gear, they would 'own' it Mainland tower space and fiber to 200 paul from city of SF internet transit donation from internet archive RF no trees, buildings sniff to make sure nobody else on frequencies RF and path modeling path engineering software (>$50k to free) EDX good path modelling software pathloss does path profiles RadioMobile made in Canada, radio community uses it More cash gives you more accurate modelling Link Budget will there be enough signal at far end? Path Profile anything in the way? Atmospheric modelling rain fade interference to others or path from others antenna issues Link budget Gain transmit and receive amps output power Loss transmission line loss free space loss obstruction and diffraction atmospheric loss Nice schematic of loss issues picture of signal level through the link, with losses and gains along the way 5.8Ghz Path Budget lns.com, small perl script, lets you calculate rudimentary path budget. As you crank in the numbers, it turns out that reality matched pretty well. Looking at Fade Margin to make sure receiver still has enough signal to decode signal in the face of atmospheric changes. Path loss caused by: refraction thermal ducting marine evaporation boundry layer (signal gets bent too high or too low) solve it with multiple antennas, or power through it. atmospheric Attenuation rain/snow (not so much at 2.4Ghz) trees (spring/summer vs fall/winter) typical solutions--just have lots of signal Fresnel Lens somethingorrather Atmospheric calculations are built into the more expensive modelling software packages. uh oh. stream freezes up. :( stupid corp network. when it comes back, he's on the 2.4Ghz path profile slide, showing microwave path, the fresnel zone they need to clear, the curve of the earth, and a small hill on the SF end of the world. Antenna and transmission line selection what types of connectors, coax, where will it be mounted, use materials that won't degrade quickly, etc. Will put radome on to protect antenna, for example. Electrical specs operating frequencies how much gain needed preferred radiation pattern power capability of the antenna polarization used to prevent multipath use vertical polarization when going across horizontal water (reflections would be horizontal) in downtown, reverse is true. rocket antenna, mimo, two chains on it, one on each axis. Antenna polarity slide with lots of details. Physical and network diagram slide. 2 links to island, 2.4 and 5.8, it's .n gear; they've gone through 3 or 4 iterations on the island, looking at best reliability and failover. standard cisco switch, to soekris board, then to get connection from top of lighthouse down, to buildings below, they have a number of fiber connections from the top of the island down. 3-4 feet of coax, antenna under a radome, to small boards powered by power of ethernet. coax is difficult to handle, you have to worry about loss, and cables are difficult to prepare. They generally get less than 12 hours of notice before boat goes out; small fiberglass enclosure, was an example case with clear cover, not meant to be clear. 2.4Ghz, and rocket radio 5Ghz. weatherproof connectors from box to rest of gear. Lighthouse network is somewhat messy, they have minimal notice, and out of a 12 hour day, they get maybe 3 useful hours of work. 100mb trendnet ethernet converters, 2950 12-port switch, soekris board. runs down to staff house MPOE, voip, wifi, routers. Ubiquiti BulletM2 and Rocket5 based on linux, you can tweak it as you see fit. Under $100 per radio single N connector for most; for 5.8, since it's dual polarized, they have dual RP-SMA connectors. h11n based, pre-standard, so really works well with other ubiquity gear. POE not well standardized for it; voltages are all over the map; make sure everything matches first! Pacific Wireless radome antenna sub-500, dual polarity soekris net5501 500mhz AMD process (they just work) OpenBSD4.5 pf is great, easy syntax, handles NAT tricks well secure flashrd--strip down base OS, sample kernels limits writes to flash for longer life dhcpdump--sniff VoiP phone DHCP vendor tags ngrep--mainly for -W byline support, debug slow social network site loading Ubiquiti 802.11n MiMo signal level jumping all over disable AirMAX (proprietary polling) disable auto-negotiated data rates; frequent changes lose sync use smallest channel width as possible long distance links have interference, avoid QAM use BSFK tune link length settings lessons learned pf "scrub" can bite you in asymmetric routing P-MTU like problems, some HTTP sites not loading always have techies buy the cabling try crimping RJ45 on 22G sticky wires Bird poop is extremely foul/fowl and wet loaned tools get lost never enough time future directions would love to avoid unlicensed bands upgrade to HD webcam increase bandwidth and reliability (diversity) on-island local Nagios weathermap pozar at lns.com matt at peterson.org Q: Jay Hannigan, impulse they used both polarizations simultaneously; was that for more bandwidth, better signal/ A: Wanted to exploit the dual chains on the radio, to do that with one antenna requires dual polarization; gave 3dB more. Q: tony kapela what was final power? A: Radios, 2-3watts, soekris like 5 watts, cisco was big one, 50watts. Q: are there other systems out there that would be appropriate? A: dragonwave makes nice carrier-grade radios that push more bw, but they cost a lot more. Q: final performance numbers? A: Could probably do MCS16, fastest rate; they locked at MCS8, like 12mbit, and latency is relatively low. They've been watching the world cup out there, watching hulu, and a real telephone out there is nice, really enjoying 20th century. Fast in the download direction, nobody else out there. Upload direction is more challenging, only a couple of mbits back up, but like 20mb to the island. Uptime--running into marine evaporation boundry layer; most days in SF is windy, fog; about 1 week out of the year, absolutely still clear day. about 30meters above the water, you get a lot of evaporation, and the signal bends enough it hits the water; driving across the bridge, he saw the optical reflection of the island, he could tell there would be no signal to the island. There's a nice graph showing correlation of nice clear weather to loss of signal. Inspect before you connect, layer 0 troubleshooting. Tyler Vander Ploeg product line manager at JDS http://nanog.org/meetings/nanog49/abstracts.php?pt=MTYxOSZuYW5vZzQ5&nm=nanog49 he's talking about best practices about maintaining fiber optic connections. maintain interface quality and cleanliness of fiber ends before connecting will cover connectors, contamination issues, sources of contamination, simple solution, and summary Some kinds of tools; 125 micron means you need a microscope. Probescopes are most common out in the field, something that lets you inspect both sides. it lets you inspect any type of connector, either a patch connector, or behind a bulkead, even with multifiber ribbon cables. Inspection points--you do it at any point when fiber terminations are handled. fiber optic connectors; network managers love connectors, it gives you flexibility for moves, adds, changes; but with it comes the downsides of attenuation, insertion loss, etc. why is inspection more important now than it was 10 years ago? higher bandwidth needs, increasing PHY demands fiber penetration and popularity One of the newest things seen in industry is new technicians; it's a good time to get them started on the right foot. The more experienced people may bring a lot of bad habits with them. Anatomy of SC connector is shown; housing body shields the ceramic ferrule; the fiber is housed in the very middle of the ceramic ferrule. The part that transmits the light is very, very tiny; you won't see any contamination on it with the naked eye. Next up is the microscope shot; only the very, very center core (9 micron on single mode) out of the 125 micron actually conducts the light; rest is cladding. two more shots, one of simplex connector, another one of a ribbon connector. Next up, focus on the connection. The connection is just two butt ends rubbing up against each other; they're the weakest link in the chain. Need 3 Ps for a good connection Perfect core alignment physical contact pristine connector interface Manufacturers solved first two; the endface being clean, though, is up to us. Contamination--single particle between two endfaces can cause significant back reflection, loss on the signal. Next up is OTDR trace over distance, every connector shows a spike due to reflective activity. Slow, gradual slope due to fiber loss...then you hit a dirty connector, with almost 5dB loss. Usually connector loss should be 1.5 to 2dB loss. with almost 5dB loss, it's likely to be outside your budget. Only thing causing the loss is the few fragments on the actual core. But particles migrate over time, as vibrations shift the end faces against each other, and each impact crushes and spreads debris. The other downside is that dirt ends up damaging the end of the fiber itself, like slamming gravel between two drinking glasses. People often don't inspect until after they see loss; that's too little too late, you've probably created pits or embedded dirt into your glass itself. Exponential impacts due to dirty connectors in a datacenter can ripple onward. types of contamination dirt, oil, pits, scratches dirt and oil can easily be removed proactively. Putting a dust-cap on the end isn't always a sure-fire cure; condensation can develop inside dust caps, and mold compounds can develop inside the dust cap; need to clean that off before you try to mate the ends and embed it into the glass. People are often not careful about their test leads in their kit; if it gets contaminated, it can damage the gear you plug it into, or damage the connectors on your test device. Solution to this is fairly simple. Inspect the fiber, determine if it's clean before you connect. inspect; if dirty, clean; re-inspect *again* before plugging it in. It's an easy best practice to have. use your probe microscope to check, then clean, then re-check, and only then connect. Make sure you inspect *both* sides of the connection. probe microscopes come with multiple tips for inspecting into bulkheads as well as cable ends. proactive vs reactive inspection; inspect and clean *before* problems show up; by the time loss shows up, the dirt is likely to have been embedded into the fiber. dirt is everywhere. Whenever you handle fiber, inspect and clean! reduce network downtime, optimize signal performance Choose the right tools for the job; combining inspection tool with power meter, to incorporate inspection with checking signal levels. Where inspection should occur? At every point where fiber terminates and interfaces with another. summary get a good probe microscope do proactive inspection and cleaning always inspect before you connect!! Break time, thanks to GoGrid; will come back at 1110 am, few more survey giveaways, you must be present to win. OK, welcome back from break! Lots of giveaways, prizes, etc, and this time is no different! Todd, Andre, Michael Sinatra, come to the front to be ready for lightning talks. 3 Rokus and a proxy box are all offered up. winners are called... small or medium-sized hoodies, mail to dtemkin at netflix.com, and he'll see what he can do. Raffle for a VCR? vendor collaboration room? apple TV ooh, winner of that is called as well. not present, they loose. ^_^; GoGrid is doing raffle for an iPad now. they don't actually have it now, they'll send it later. ^_^; and everyone who entered will be eligible for a $250 credit for trying out their virtualization thingie. http://nanog.org/meetings/nanog49/presentations/Wednesday/DNSSEC-20100616-OF-Nanog.pdf DNXSEC.CZ, top level domain for Czech Republic about 14% of domains signed that's 99,000 out of 690k domains www.nic.cz has numbers. Before deploying it, heard lots of complaints no business case registrars don't want it it is too expensive too complicated philosophy somebody must do it, get it started security is not a special service it's a natural feature, part of domains registry responsibility to get it started needed allies--ISPs, registrars, etc. Communication with registrars seminars before and after DNSSec launch nice conditions--no fee DNSSec training positive discrimination in co-marketing programme technical support for bulk DNS record registration--no DS record, but keyset Changed scheme a bit; name points to nsset, which contains nameservers, etc. This way, one object can be created, and thousands of domains pointed to it. The keyset is another object, it can also include several different keys, for a domain, DS records can point to that keyset. One key can sign all of them, so doing key rollovers is less painful. Co-marketing registrar and cz.nic together cost split 50-50 maximum limit is based on registrar performance DNSSec bonus 10% of price given back End user education increase awareness lots of conferences, articles in papers presented and explained attackes against DNS communication with important players czech EU presidency eu2009.cz was signed DNSSec hardware tester; test your DNS resolvers, reports DNSSec compatability with device and network open source http://www.dnssectester.cz/ if you want it in other languages, send them a request and a translation. ^_^ DNSSec validator firefox add-ons; shows icon similar to 'https' validates domain name in address bar No DNSSec, broken DNSSec, functional DNSSec download from http://www.dnssec-validator.cz/ or search in mozilla add-ons still in beta, but it's close to final release. After launch some registrars started to offere DNSSec but as a bundle in "secure domain" product for small additional fee 2008, few hunded people start doing DNSSec Then in 2009, two registrars turned on DNSSec by default, for free, no extra charge (unlike earlier fee) well communicated, very good media coverage gave them marketing advantage and competitive advantage. Large market share, converted a large number of zones at once. DNSSec can be deployed at large scale it is not so complicated registry can start it up as pioneers, found some bugs and small implementation problems so, Czech Republic is the most secured DNS in the world at this point. :) Now using NSEC3 done for enum0 in August, will do it for CZ after root zone is signed. As is tradition in Czech republic, they made the important decisions in a pub. picture of Steve Crocker, others at a pub drinking and making decisions. No time for questions. Michael Sinatra is next up DNSCurve vs DNSSec http://nanog.org/meetings/nanog49/presentations/Wednesday/sinatra_lightning_nanog49.pdf Most people know about DNSSec; few people know about DNSCurve; it flares up on one or two mailing lists. Some people argue we should deploy DNSCurve vs DNSSec. He's just an ops person, not an IETF person. bug nick weaver if you don't like the way standards body is going. Will use argument by analogy try to systematically understand and perhaps even explain a concept via analogy. Isomorphism: components of the analogy that map to the components of the real thing DNSSec RFC 4033-4035 draft-ieft-dnsext-dnsse... DNSCurve provides link-layer security for records, by encryping them, making them difficult to eavesdrop on. In DNSSec world, records are created and signed cryptographically, but then are handed across an insecure medium, where signatures can be inspected and archived. In DNSCurve, you create records, but don't sign them; instead, you encrypt them at the transport layer before handing off to the next hop on its journey. For DNSSec, you can validate the authenticity of the original record, but people can see the signed data. For DNSCurve, nobody can inspect the data going past, but you don't sign the record itself. Why not do both? There's no reason that both can't be done; both sides argue theirs is the one true way, but they solve different things, so you can sign the records before encrypting them along the wire. This is really a false dichotomy; we should finish standardizing *both*, and let ops teams decide which pieces they want to use. http://burnttofu.net/dnscurvevsdnssec.html Next up is Todd Underwood, not representing anyone. Prefixes as Probabilities http://nanog.org/meetings/nanog49/presentations/Wednesday/Prefixes_as_Bundles_of_Probability%20(1).pdf modest proposal to radical extend useful life of IPv4 Problem as we know it. v4 prefix space running out quickly v6 is not operationally equivalent to v4 yet Time is running out and mechanisms are needed to extend the useful life of v4 possibilities so far: address market/trading rationing flat out runout resource allocation now RIRs allocate resources on a 0/1 scale you either do or do not get it. odds that someone else has been allocated the same resource are very, very low (mistakes only) (story of working at upstream for a power-plant that had a /24, ran out, and just kept allocating more and more, and skipping addresses that happened to conflict with things they needed to get to) disadvantages extremely discrete, yes/no allocation requires external factors (price change, e.g.) to tune allocation/depletion rate Market/transfer policy does some of this, but it has high transactional friction; it's not like eBay Thought experiment allocate unique addresses at high price allocate non-unique addresses at much lower price, based on probability that someone else will be on same space. Will massively extend lifespan, by allowing multiplicative reuse of remaining prefixes. Allows for great derivative market. There's dirty space...let people who don't mind using dirty space use it. Modelling the space advantage C-count of re-use Oop -- operational overhead on a prefix not just collisions, but any operational impact Ap association domain for a given prefix p(oop) is obviously a function of Cr & Ap collision avoidance and detection time limited addresses, people sleep at times geographically limited scope (europe only) provider isolation/selection system You can choose addresses that are in use by someone else, if you never want to reach people at an ISP you're fighting with. Many conflicting uses would be for things like P2P, to keep traffic from moving off your network. Would need some instrumenting, like doing flow inspection of the internet core to tell when a resource is being overused. toddunder at gmail dot com Questions: line closed after martin levy Q: Dave Temkin, netflix derivatives, sounds like oil market--do you want to be the BP of the IP market? A: We will need to fund instrumentation somehow. Q: Mark Kosters, ARIN Truly fascinating and mind-numbing work. This is fraught with peril; would like to see ARIN DC to take up policy implications of this and decide how we can move forward with this. :D Q: Martin Levy HE has been doing IPv6 for a number of years... in case there is a need for future employment, send him a resume. :D Q: Leo Bicknell he things v6 thing is like Y2k; he thinks it's a lot of panic over nothing. So he's going to take Todd seriously. 1) you're in the wrong forum; should be talking at ARIN/RIPE about this 2) there are ISPs that offer probabiliistic IP space and packet delivery today! If you chose the right provider, you're already able to this. Q: Joe (Odd Tunderwood) If it's good enough for really small particles, it's good enough for us! Q: What types of pharmaceuticals lead to this? A: this is just one of the many great ideas he comes up with, day in and day out. Q: Anton Kapela Simpler solution, RFC3514, we can use it to take remainder of header, with one bit to designate whether it's to or from Google, and we get lots of address space back that way. Q: Brandon Ross Have you thought about applying these concepts to v6 as well, rather than trying to avoid v6? If we apply this to v6, and have multiple companies around the world Q: Lee Howard Had a similar idea, but chose to keep it to himself. There are great ways to create pockets of unreachability in v4; this is but one of them. People use squatted address space internally already. There is something here that is sane, actually. There is potential room for work here, but he's not actually going to ask Advisory Council to follow up on this yet. Q: Michael Sinatra This proposal provides all the benefits of Carrier Grade Nat, but none of the complexity; and with less complexity comes greater reliability, so this has the potential to greatly increase the reliability of the internet. Thanks to Todd, now get back underneath your bridge! Nicholas Weaver Measuring Access Connectivity Characteristics with Netalyzr http://nanog.org/meetings/nanog49/abstracts.php?pt=MTU5MCZuYW5vZzQ5&nm=nanog49 Network transparency and network debugging how do you know what the network actually is? and when it breaks, how does it break? Java applets can perform a lot of activity by default can do arbitrary TCP and usually UDP if user accepts, can really do anything. User clicks on the icon, accepts the tool, it evaluates the network. Is it NAT'd? are ports renumbered? is it on any blacklists? does it block ports? what about UDP? can't send, but can receive fragmented MTU. Finds path MTU gets path MTU too big latency to server, bandwidth buffer capacity (why bittorrent kills your net) hiddent HTTP proxies? are they vulnerable to a bug (BlueCoat boxes have a vulnerability) HTTP Cache--does it work right or not? Is it v6 capable? DNS behaviour DNS properties DNS glue check the clock Nice big test suite. Beta 2009 released Jan 2010 non-beta release. 110,000 unique sessions so far. Look at a year's worth of data. bias in dataset, geeks like using the tool. openDNS in 10% of results, more than average. 90% of sessions are behind NATs 2/3 of NATs have DNS proxies in them. 4.4% of NATs respond to external DNS requests. (lots of DNS reflectors out there) protocol filtering tests (connect to echo server, kill proxies and firewalls) Local proxies are very common (AV shimming into stack) NATs include FTP proxies Most proxies don't like these communications SIP awareness is very common Surprised about how little SMTP outbound filtering there is; dynamic blocking rather than static blocking. Run your SSH server on port 443; that's one port that is *not* molested. Slammer block, 1434, turned on 7 years ago, never turned off. About 11% of DNS ports reject non-DNS traffic. captive DNS proxies are seen. Check for large DNS responses. About 1.3% of sessions can't handle EDNS at all, and 5% fail on DNS larger than 512 bytes, and 10% more can't handle fragmented traffic. lots of problems for DNSSec for client. Fingerprint DNS servers based on glue policies. 10% of resolvers that advertise ability to accept large responses actually can't, they're behind firewalls that kill it. DNS wildcarding is growing very common three ways wildcard everything www, like comcast and verizon charter, quest, wildcard everything. wildcard servfail is rest. 28% show wildcarding; exclude comcast and quest still 20%. don't count on SERVFAIL anymore. DNS man in the middle 3 variants open DNS is proxy for google, they do tell you about it. wide open west, few other ISPs, MithM google, point to proxy they control, not sure what proxy does. Malicious DNS resolvers; redirect ad.doubleclick.com to serve up ads for ViMax male enhancement. All issues are due to recursive resolvers; so end host will need to do DNSSec validation on the client, but the clients can't get data. Fragments and PMTU-D are broken. Linux sets DF on UDP packets, packet gets dropped silently, or ICMP message is sent back up to app which treats as exception and terminates the connection. Need to turn off PMTU for all of linux or fix apps. 13% of PPOE ICMP reporting is unreliable Network has decreed that fragmentation doesn't work, and ICMP replies are unreliable, so fallbacks are needed. HTTP 8-9% proxies in corporate environments, captive locations. Often mandatory. In-path caches, often wrong. transcoding, not often. IPv6, none on their systems, so they load the ipv6.google.com in hidden frame, report back. About 6% of users are able to fetch it. It's starting to get out there, but slow adoption still. What are our questions? Netalyzer is an ongoing project. Want to know what else to add to it. Conclusion--it actually works! when measurement tool makes it on slashdot, you know you've done something. Q: Brandon Ross You alluded to different brokenness by different regions, different providers; that data would be very good to know from support side. A: For ISPs, they have top 10 issues; how different ISPs do wildcarding, who has PPOE, who blocks SQL slammer, etc. Don't want to release data, potentially private data about people's networks included in it. Do they do regionalized reporting? They do capture bandwidth numbers. About 10% of hosts intercept and kill port 80 traffic they don't like. Q: Tony Hain, Cisco Path MTU is broken; fine to make that statement for v4; what about brokenness of v6 vs v4? Right now, can't measure it. A: yes, way up on their short list to add in, they're very aware of the importance of that, once they have good v6 connectivity in place. Q: Chris Woodfield, Yahoo Any thought about including non-web-specific protocols, like RTSP, which may get captured by proxies which do bad things to them? A: Limited by java (1.4), so UDP and TCP are pretty much all they can do. Q: What about flash? A: same origin policy in flash is weird, and he's not a flash programmer, sorry. Q: Dan Mahoney, ISC Any checking on whether ISPs allow proper filtering? A: Can't craft raw packets, so can only do PMTU from server to client, unfortunately. The spoofer project is a more appropriate resource for that. Q: Google person If they'd like access to real v6 cloud, they can host it for them; glad to know they were able to use the v6 image. A: They'll be hosting at lbl up the hill real soon now, but thank you for the offer! Wow. Pretty cool NANOG. Nick was impressive, following Todd. :) Thanks, and some statistics. 609 total attendees, largest since october 2001, NANOG 23. Of those, 200 were at their first NANOG 26 countries 45 speakers top numbers so far. :) Thanks to Dave Temkin and Netflix as hosts, they totally rocked. Parties--Netflix, Google, Silent Partner, Coresite, HE, wow, big thanks! Thanks to all the sponsors, heck if I'm going to copy them all down here. ^_^; Thanks to Merit for all the logistics support. Many great speakers, so much content, couldn't accept it all. Steering Committee and Program Committee for putting it all together. And thanks to everyone for showing up and making it such a great event! Next NANOG is Telx in Atlanta, GA in October. Thanks again to everyone! You're now excused--you may go home. :) NANOG49 wraps up at 1221 hours Pacific time.