(version 1.2) 2010.06.14 NANOG 49 notes day 1 http://nanog.org/meetings/nanog49/abstracts.php?pt=MTU4OCZuYW5vZzQ5&nm=nanog49 Welcome to NANOG49, hosted by Netflix, let's have a hand for them. And for Silent Partner and Dyn, for the social last night. Some reminders for everyone. Check out the vendor collaboration room, it's open all the time, but the demos will be from 4-6 each afternoon so you can see things working together, in California East. Oh. Georgian, on the way in. Survey each day, with giveaway, so do the survey! Beer and Gear tonight, starts at 6pm, lots of sponsors for the Beer and Gear. [reads boring list of companies] Thanks to all of them. Social tonight will be at Ruby Sky tonight, thanks to those sponsors. Next up, Netflix to talk about what will be happening today. Don thanks everyone who worked hard to bring the program to fruition; thanks to the Merit staff, the volunteers, Netflix, and especially Dave Temkin for coordinating all the hosting for this. We're up to 599 attendees, so we beat AOL NOG in Reston...but still below the all-time high. This has been worked on since early 2009; many thanks to his boss at Netflix who gave him the support to make this happen. Thanks to the PC for the wonderful program we have lined up this time around; big hand to the presenters from yesterday, it's the best set of Sunday tutorials we've seen in a long time. The evolution of NANOG and the transition to the new organization was discussed at the community meeting last night, and will continue today at the BOF at 4pm; please do participate in that transition planning! There was a survey from yesterday, do fill that out before the first break for the giveaway! Ruby Sky Social starts at 8:15 and runs to midnight, after Beer and Gear. Ken Florence up next to say a few words, and then it's Vijay to tell us about the end of the world. Ken does the esoteric presentation. There was a decision about trying to do streaming of hidef video to devices with no hard drive and small memory buffers, based on such scientific evaluations as 'my home bandwidth seems to double about every 18 months, so it seems to follow Moore's law'. Thanks to everyone who put this together; the internet is a giant cooperative, and it's the efforts of people like you who keep it all running. David Meyer thanks the Netflix team again, it's Monday, it's going smoothly, they've done a fantastic job. OK, with that, it's over to Vijay. http://nanog.org/meetings/nanog49/abstracts.php?pt=MTU5NSZuYW5vZzQ5&nm=nanog49 He's giving his presentation on behalf of the excellent set of network architects at Google; it's really their work, he's just doing the presenting. His basic philosophy is that he's not really sharp; if he can't understand it, then he can't run it. So, he cut his screed from a text on programming complexity, and changed it to fit the network arena. You can do 8 things really well, but the system will be complex. But, take 2 or 3 long tail items out, and the system becomes much simpler. Keep it to the 80% you can get done with 20% of work. VLANS in a datacenter. Build vlans, instantiate service on a set of machines in racks, firewalled off from other systems through the vlans. Better solution, don't use vlans, but provide a library that does lightweight encryption, so you can only talk to other apps that have your key. One flat layer 3 network, do bin packing for machines in clusters is much denser. Don't imagine what might come along, build for what you have, and then rebuild later; what you have won't last that long anyhow. Google's goal is to organize the world's information and make it universally accessible. Crawl system pulls in about 10Tb of data every few hours, and Youtube puts out 1billion views a day. When you do a Google search, you're getting data back in a few hundred ms; that can't go to disk, which means most of the internet index is in RAM. They build warehouse scale computers to hold that in memory. "datacenter as a computer" is a book that talks about it. Shows picture of large warehouse datacenter near hydroelectric plant, to get lowest cost of electricity with fewest losses. Evaporative chillers take up a large chunk of space, as does power substation. Computer inside is built of racks of processors, clustered together. If you're on one server, you have access to some amount of storage locally, in about 4ms. In a rack, you have about 1TB of RAM, and a few hundred TB of disk, within about 40ms As you go to cluster, 30TB of RAM you can access in about 400ms. You gain more resources, but access time drops down by an order of magnitude. Why not go with blade server for optimized communication to get faster communication? Amdahl's law; as you distribute your programming, your communication will still dominate over the wins you get from adding more cores. And on the internet scale, nothing really works on less than 100,000 cores; so there's no value in buying integrated systems. Instead, buy cheapest components, and interconnect with cheapest network possible. So, no VLANs because those require processors that support spanning tree, etc. In simplifying, you lose features, but the work cycles are dominated by communication, so those features don't help at large scale. Reliability and Availability the bigger you build things, the more likely it is that something will fail. Since things will fail every day when you have that many devices, you might as well write the software to support failover from the start, let the software deal with the failures, and not worry about any specific reliability. At least one item will fail or crash a day. Vijay tells dead horse story. Had one box in the pop, and other was backup unit, company was contracted to bring the standy unit from the office to the pop. 4 hours, 12km...seemed simple. But 12 hours later, no backup had arrived. The tuktuk who was hired never showed up. So office people hired a donkey cart to carry the router, but the donkey was hit by a truck and killed; and when replacement donkey showed up, and they unhooked the cart, the router tipped over and fell into a puddle of water, and died. This is not unique; in US, we have drunken hunters. We have lots of cheap alcohol available, and lots of firepower and ammunition available to them. In the Pacific Northwest, drunk hunters shoot at insulators carrying fiber, causing fiber outages. So, bad things happening to your internet whether it is in emerging markets or the US. And when bad events happen, your correlation tends to go to 1. Rail derailments tend to take out fiber on both sides of railroad tracks. Most reliable fiber tends to be in gas pipelines, because the penalty for cutting those tends to be very high. About .5% of machines tend to overheat in clusters. PDU failure about .5% Rack move (1000 machines down) Network rewiring (rolling 5%) Racks go wonky; fowarding asics in swiches go bad router reloads, maint gone bad. Slow disks, and every so often identical disks will be 5-10x slower, which causes GFS to cycle as masters go to slave and back again. Hard drive failure bathtub curve... In a given search run, you'll have one disk fail... ...and then I lose the stream. darn it. OK, everybody lost the hidef stream. I cut over to the Flash stream. Layer Cake Showing the levels of intelligence and complexity at each layer. network is dumb, only has what it needs for MPLS style resiliency. Traffic Classes Two types of traffic User ("I") is user initiated Machine to Machine ("M"), bulk, predominantly machine to machine. When you run datacenters without chillers, you need to be able to migrate jobs to other datacenters, to prevent extreme temperature excursions. To handle those failovers, you can either do converged network, and carry it all within common transport infrastructure, or even with common routing and transport. The overlay model is separate transport, separate spectrum for datacenter to datacenter traffic. converged L1 network, your datacenter traffic is carried along with your egress traffic. that's typically how networks get built. Converged L1 + L3, common spectrum and common layer 3 routers carrying traffic both for datacenter and external traffic. converged model is seductive, you can try to fill in the troughs with machine-to-machine traffic. You can build a transport/packet box with ITU or without ITU integration, you do electrical/optical conversion two times at each end. to do the packet processing, and then the transport muxing. If you do tightly integrated DWDM optics, you can cut 20% of the electrical work. Overlay L1/L3 network; you build optimized, long-distance network, coherent 100G or 40G system, where you can run long haul without regen and add-drops. Paul came up with this system; you can put LSRs, depending on where they are, which allows you to extend the life of your routers, by transiting your traffic through the simplest boxes possible. They built a model, which ran through the models with levers to adjust the options, and out comes a cost structure based on those inputs. eBay, grey market ports to build the network model, normalized 10G numbers Those depend on traffic and load; the model mainly shows you what the really bad ideas are. So, layer 3 converged, for example, the junction points dominate, all the traffic hits OEO junctions, and all the cross-country traffic has to go through same OEO as the peering traffic that gets dropped off. Ran the model against actual spend, it's within 3% of their actual capex spend. so converged is bad when 1:1 traffic on egress and datacenter. For low volumes, integrated converged and overlay are cheapest. When you double machine-to-machine traffic, L3 with no ITU integration is bad. So, you need ITU integration first off. But most optimal is I scale and G scale with ITU integration, two separate networks. But this is sensitive to traffic and topology. There's some really stupid ideas, and L3 converged is one of them. For warehouse scale computers, you really don't want that. This doesn't hold true for telcos as much, but where you have a lot of M-to-M traffic it does make sense. Conclusions: at this scale, more internal than external traffic volumes make cost efficiency a primary metric cost-effective LSRs provide the simplest way to effectively stat-mux traffic and add/drop (but only at large metro level routers that will do interconnection with other people) However, DWDM has to be integrated with LSR function Depending on add/drop locations and relative scale of I and M traffic...[lost image at this point] Simplified unified end-to-end control play e at highest network layer (IP/MPLS) is a MUST No need for additional control plane at optical layer other layers provide necessary functionality Questions? [drat--lost audio for the Question, but heard Vijay say "you are absolutely correct!" to the person, so it must have been a good question, whatever it was] Don't forget to put your phones on mute! Analysis of 1/8 and dark net traffic. http://nanog.org/meetings/nanog49/abstracts.php?pt=MTU3NyZuYW5vZzQ5&nm=nanog49 Down to the last 16 /8's of the in the IPv4 free pool. Water at the bottom of the barrel is pretty...murky. IANA allocated 1/8 to APNIC in January. RIPE noticed that when they announced their de-bogonization routes to test it out, they saturated their uplink, which indicated something fishy was going on. Worked with APNIC, announed 1/8 for 10 days, collected about 8TB of full packet captures, and observed the traffic. About 170Mb sustained traffic. Some high events, peaking at 200,000pps. They looked at 35/8 during same period, compared same time period. 15-25Mbits on 35/8 when announced, vs 130-150Mb on 1/8 On 35/8, TCP dominates, on 1/8, UDP dominates. [matt has to jump on vendor call, will be back when finished] Fill out your surveys, they will be due by 11:20 to be counted. Survey URL has been corrected, they will hand out 3 top-of-the-line Roku XL boxes. Survey is still anonymous, when you enter your name in, it does not tie in any way to the survey that is submitted. Oh, I get back in time for break. Lorezo Colletti, IPv6 at Google http://nanog.org/meetings/nanog49/abstracts.php?pt=MTU3NiZuYW5vZzQ5&nm=nanog49 Collateral damage geolocation happens on broad swaths of IP space all at once. Carrier-grade NAT, users share IP space, run out of ports. set-top boxes, mobile networks, Verizon, T-Mobile, they're out of v4 already, they have overlapping net-10 space, hard to manage, they're already moving to IPv6. They'll put NAT64 if you aren't on v6 yet. To maintain end-to-end, v6 is the only way to go. Started as 20% project, and then people came out of the woodwork to help. Started with pilot, lab network, then scaled up from there. Did it in stages. Traffic isn't there to start with, so you can do it smaller. Work from outside, move in. First the load balancer, then the frontend, then move towards backend. As long as you keep track of IPv6 source, you can pass to v4 hosts. take v6 address, remove user-modifiable bits, hash into 224/3 sometimes not perfect "your last login as from 238.1.2.3" this shim layer allows you to put V6 in, without having to audit all the code in the network. Took about 18 months to have most google services available over IPv6. Youtube was one of the last pieces, due to hardware issues. Not blocking on dual-stack backbone; just need technology that works. You can't do your www.google.com record, or about 0.1% of users won't reach google anymore, which is considered unaccepable. But it's done on demand; if your DNS server is in whitelist, you can get there. If your DNS server meets the criteria, you can just start carrying traffic over v6 for your users. Residential ISP networks are a big challenge; that's where the biggest support challenges are. Testing and iteration implementations mostly work, but will have bugs. don't expect things to just work If you find a bug, report it, keep testing, there's many more bugs to find. work around it in the design if you can. If a firewall filter term has a 1-bit match in bits 32-64, and then a term with a 2-bit match on bits 64-96, the second term will not match. He lists lots of other fun bugs they hit. He shows a tcpdump, a traceroute, and scrolls around on maps, showing the v6 packets just flowing happily. Mostly works, just one in a thousand users that don't work. Most applications, including web browsers, get list of addresses back, use first that works, with v6 tried first. If failure is local to host, you get local error, it's fast. network error, routers can send back unreachables but stack may not give up on unreachables. spoofing RST packets works better to make app give up OS behaviour local failure, RST, fast network, timeouts, if host listens to unreachables, fast, otherwise, slower. www.google.com can have up to 6 AAAA records. but can cause longer timeouts. what goes wrong? home gateway behaviour, routers may turn on 6to4 and go through broken relays at best, it's latency increase; at worst, just won't work. may try 6to4 with private addresses will never work, but implementations still try. May prefer 6to4 router over native router. Host may prefer a 6to4 address, if it uses private addresses. If you have public from 6to4, and v4 private address, it'll prefer the public 6to4. Teredo is nightmare for short-lived connections, huge setup cost. host firewalls may block icmpv6, may drop v6 entirely. Home gateway sending out an RA of 0/64 (not valid at all) Host ignoring the unreachables 24 second timeout Measuring brokeness ask browser to connect to v4 and dual stack hosts using invisible elements use long-lived websites, like gmail, youtube use javascript to make multiple requests for same session, test-ipv6.com does similar setup MTU, glue have a sentinel request after a set time good if user disconnects use one-time hostnames reasonable data set currently about 10M samples per day web requests only, no DNS yet not stat analysis yet, but daily numbers are stable v4 also has non-zero failure Results vary per network and per OS Large ISP A 0.064%, better than average, still not good enough Large whitelisted ISP, 0.03% Spread with v4 is less significant than above whitelist masks brokenness different OSes have different numbers; for large ISP A all clients, 0.064% excluding Macs, 0.014% Mac prefers 6to4 over IPv4. :( Router problems need router upgrade at home users don't upgrade home gateways firmware not upgradable hard to know what problem is Host problems workarounds in individual applications (Chrome) to fix all apps, need OS support. Host fixes: happy eyeballs draft simultaneous parallel connections (v6 and v4, dup syn packets) Mac OS X plan of record Can't fix MTU holes Probe on attach Fetch a 1280 byte object over HTTP over IPv6 warn user or disable IPv6 OS-wide on failure to fetch similar to what recent versions of Windows do to detect captive portals That's why the can't turn v6 on main hostnames across the board; it'll break users. Q:, Leo Bicknell; the users that break, is it an ongoing study, and has the percentage of brokenness changed over time, up or down? A: It is indeed ongoing, the older data is muddier; the newer version is giving clearer data; they will monitor the data over time to get more trend info over time. [Matt has to run to yet another meeting, and misses the "state of the IPv6 routing table in 2010" talk] Don't forget to fill out your survey! You can still submit lightning talks, there will be votes tonight and tomorrow for the topics to be presented. Don't forget Beer and Gear after this, and the social at Ruby Sky tonight. John will talk about IPv6 for home networks with his panel. http://nanog.org/meetings/nanog49/abstracts.php?pt=MTU2NSZuYW5vZzQ5&nm=nanog49 He organized the panel with some folks from industry, James Woodyatt from Apple, works on Airport and Time Capsule at Apple. Allen Huotari from Cisco, member of home network products team at Cisco (formerly Linksys). Salah Nassar from Netgear, working on wireline products. John works at Comcast, and they are doing IPv6 trials with their customers. He gets a *lot* of email about the trials. What does the home network have to look like, what products, what RFCs does it need to follow, etc. So this is a chance for people to ask questions about the trials. If you haven't signed up for the trials, you should, go to http://www.comcast6.net/ and sign up to be part of the trial. They have announced several different trial technologies, from dual stack commercial to home broadband with 6rd. Comast thinks 6rd is a good enabler for access networks that can't natively support IPv6. There are still some issues, esp. from the DHCP point of view with it. Native dual-stack to residence over Docsis; they simply add IPv6 over same connection. They'd like to use that model for as long as they can; the network can largely support that mode of operation. Will have some trials late 2010, early 2011 for dual-stack lite, which they look at as an insurance policy of sorts. They're looking at it, but hope they never need to actually use it. Now over to James from Apple. He writes software for Airport, Airport Extreme, and Time Capsule. Apple's experiences doing IPv6 on products, and how engineers view it, and what they tell their managers; don't ask questions about future apple directions, he can't answer them. July, 2004, Airport Express, wifi range extender, USB print server, itunes streamer, access point, does a lot in small package...and is IPv6 enabled. Can be configured over the network using link local IPv6 addresses, with new protocol called Bonjour. It works everywhere, no matter the topology, and you really can't screw it up. They did the Apple Mobile Me, "Back to My Mac", rather than storing your data in the cloud, it does a virtual private IPv6 connection tunnelled over IPv4 network; it acts as though it's a local connection. IPSec encapsulated over UDP port 4500, use dynDNS to push traffic over WAN links, use uPIMP to punch through firewalls. There's no IPv4 on these tunnels; all the applications have to work seamlessly over IPv6. 10.6 added more, and even more will come; user doesn't need to know what stack they're running. But IPv6 in single domain isn't the same as having global IPv6 connectivity. Some typical home network diagrams are shown as slides, from the telco perspective into the home network. The hosts share public address, and can share that; but the host itself doesn't know how to share that address implicitly. port forwarding web site services for doing port forwarding through firewalls portforward.com has 41 pages of application to port mappings. IPv4 addressing is starting to run dry. ISPs either stop growing networks, or start doing network address translation. Stopping growth isn't going to happen, so NAT is going to happen. If you have more than 18 million subscribers, you won't fit in private address space. Revolutions are not made, they come. The IPv6 driver is here; further growth can only happen in a sane fashion by moving to the dual-stack network of tomorrow. It will again provide end-to-end connectivity for the IPv6 home connected devices. residential gateways will get delegated prefixes to hand to other internal gateway devices. Hosts can use privacy addresses to do somewhat private communications. For private, internal only devices like itunes streaming devices, they can use link-local addresses. Current iPhone OS releases don't support IPv6, but hopefully when iPhone OS 4 comes out, people will hopefully be reassured. They like 6rd, which is a great way to roll out v6 without having to touch your access networks. Free is an ISP in france that did this, it works great for them. IETF going to standardize this soon. DS lite, good protocol for figuring out how to to provide v4 connectivity as you move to IPv6 only infrastructure. Think of it as an exit strategy. Apple has been delivering IPv6 to customers for a while without them even realizing this. Delivering services on their timeframe, not on the ISP timeframe. If your app is in the app store, chances are it is already v6 capable. We're already past the point of no return on this!!! Allen, Cisco person talks about 6rd Why would you think of tunnelling at all? Retains end-to-end semantics. two tunnel models; IPv6 (DS) lite and 6rd (rapid deployment) Why tunnel IPv6 in IPv4? It does provide a transition with minimal touch points. IPv6 via IPv4 SP networks using 6rd Method for deploying IPv6 on an Ipv4 infrastructure. It's like 6PE; there's CE device, and BR (border relay). Border Relays designed to work as anycast to make load balancing easier. v6 traffic follows v4 routing. [Matt has to go to yet another meeting, so misses the rest of the v6 home network panel, and subsequent talk.] I return just in time to see people wandering off for break. NANOG transition BoF Steve calls it to order at 1635 hours Pacific time. http://nanog.org/meetings/nanog49/abstracts.php?pt=MTYyMiZuYW5vZzQ5&nm=nanog49 Randy Bush goes to the mic and makes a joke about transitors. Randy Bush brings a check for $250, as the first membership, asks if they can go about doing the business stuff, so he can go do some technical stuff. Thanks to everyone who is here. Raise your hands if you weren't here for the meeting yesterday... ok, a few people, so he'll zip through the bits from yesterday, and then let the business people dive into it. So, same slides as yesterday, pretty much; will refresh the two who weren't here yesterday. Then will spend a lot of time on the budget stuff, since that was new since yesterday. Lots of time for QA and check-giving. outline: Nanog NOW Nanog of the future other stuffs Now, mission, conference organizer, operational communications service. Now, status--Merit controls the finances, takes the risks, tells us what we can do. Now, governance--program committee, communications committee, marketing working group. But Merit can still tell us what we can and can't do. Why change? We have working self-governance over the past five years, but we still can't control what we do and don't do. So, looking to make a legal entity that is owned by the membership, controlled by the membership, and removes a level of indirection. flow chart, move from activity of merit to independent organization. The back-room bits, contracting, system operations, etc. is all done by Merit; we think same thing can be done with small paid staff and outsourcing. Proposal: Mission stays the same; hold conference, mailing list, do education, outreach; if we want to do more, we can. Proposal: status and governance; new BoD have been picked for short term. New Bylaws being worked on now, will be voted on within next six months. Once bylaws are written, board can be adjusted. membership should be inclusive of people who can't always make it to meetings. Expand funding sources; right now, funds come from direct meeting sponsorship and meeting attendance. Aim to make funding level less dependent on meeting attendance. Paid membership, people empowered to vote in elections. Shouldn't be terribly expensive, discounts for students, etc. proposal: conference and mailing lists; would like to increase sponsorship to keep attendee costs down. outsource some elements, but keep management aspects internal. Timeline and budget oh, that was end of recap from yesterday, rest of this is new stuff. Q: Olie Jacobson, Cisco. Why membership model, instead of IETF, where there is no membership; there is an umbrella model which is well funded. A: Member dollars don't hurt, but they aren't there for survival of organization. This is to allow membership to participate more actively in voting and governance of the organization. A: It does bring a little more funding in that doesn't exist purely hand-to-mouth, meeting to meeting. Q: Some thoughts on membership front; mail thread with Tom Scholl in 2004; some good thoughts on this. Provides selection bias into NANOG. Some people care a lot about governance, and some who do. People right now vote when begged, once a year, when they don't really know or care about people. Network engineering and architecture right now lacks any type of professional membership organization; others have ACM, SPIE, IEEE, etc. Would be nice to be able to belong to a professional organization. Q: Bill Woodcock from back row. Can they talk about scope creep, institutionalization, and budget? Once this exists, and has staff, and has a reason to look for more money and staff, do we set limits? Can executive director simply look for more budget, or do we set limits on what budget they can have? A: Honestly, just been focused on trying to make it all work so far. There are limits on what a 501(c)3 organization can do, so within those parameters, there is room to innovate. Board will need to think about how decisions will affect budgets. Q: RS, steering committee; thanks to Woody for making the meta-point; they've been working on this for about three months, and are pretty far along compared to many startups. There are many seats at the table, and much stuff that still needs to be done; there are lots of people in the community... [stream goes away...] This is to everyone in the room; please contribute your time and talents to make this a success. Thanks to Sylvie, Joe, Mike, and Dwayne, for putting the hours in last night for this so it will make sense; Oh, it was Joy, ok. Oh, Joe is up actually. Timeline from yesterday is put up, it's delineated by the normal election cycle in October. ARIN has said they will help with doing the parallel elections, and make sure the current enfranchised NANOG voting world can ratify what we're doing here. We have a sharp delineation between Merit activities based on budget cycles, contracted hotel for 52, for 53, thanks to ARIN and JC we have that lined up. Two sets of numbers, based on past performance, and based on pessimistic assumptions in case we trip all over ourselves, which is also the break-even point. Bill Norton asked how many meetings per year, this is a three year pro-forma, with meetings under Merit under year 1, mixed year 2, year 3 is all newNOG, stick with three meetings until we know what we're doing. Q: Todd Underwood; many loudmouths want fewer meetings per year; Merit squashed that, saying they couldn't cut meetings. If we have an outsourced model, do things scale more linearly, and can we change how many meetings a year we do? A: It does require additional sponsorship model elements if we change it; our current funding is hand-to-mouth, where every meeting pays for the set of meetings. Within the realisitic assumptions, based on past performance, there are assumptions about meeting sponsorships, there had been questions about memberships; assumed $100 for membership for sake of budget; year 1, pick up the interested participants, Randy is a good sign. Nominal growth rate, based on attendance, 1/3 new members each time. Category for early contributors, like Spectra who helped underwrite some contracts; solicit moderate number of corporate contributors, and decouple those memberships from actual memberships. Dave Temkin, netflix, marketing working group corporate, 5,000, makes sense; if we look at merit model, people expect something for that at the meeting; vendors set up for breaks, or beer and gear; getting 5 companies to give 5,000 each is doable, but that's a pretty far departure from where we are today. $50,000 is an optimistic assumption, not realistic. One thing that works well is that vendors can get their name out for $4k or $5k. At $50k will a company have a sense of ownership over the organization? Number of smaller bronze sponsorships vs one or two big gold sponsorships? Currently, Merit gets that from ARIN for the operation of videocasting and archiving, etc. And that's an annual item, not a repeat. Instead of annual sponsors and then meeting sponsors, why not break up current model, and make gold/silver/bronze sponsors, and tie them to... ok, patrick recruits him into budget working group. ;) Fundamentally, there should be plenty of money. Q: Do we think the current sponsorship model is actually broken? Is it sustaining in the current model, or does it need to change now? A: yes, it does need to change; what we have today is someone who pays for three beer and gear sponsorships a year, and we're moving to someone who can do different items, or participate in only two nanogs. So, it's taking existing programs and trying to make them more efficient for nanog sponsorships. Q: Be wary of making too many changes around that model right now; let's keep current model until the new org is up and stable. Sponsors may be gun shy about making a big committment to a new org. A: jumping in at $20,000 isn't just about getting a bumper sticker, this is about supporting the community. But not every company wants to work with the inner workings of nanog; they may just want to get their name out, and may have different ideas about the level of involvement. A: Valid points, and would like sponsorships to be active participants, vs just giving us money and moving on. A: We are engineers, and we have a habit of falling into the trap of discussing the how, rather than the what. This set of slides is a working model based on a certain set of assumptions. Let's not getting to point of not seeing the forest for the trees. It may not be worth changing the model for this. Q: Jay Hennigan, there was a 300,000 budget that came close to balancing exactly at yesterday's meeting. We are engineers, not bankers. banking should be boring. This is a proposal for a relatively large organization. Would be good to see how this relates to what we have today, for this meeting. from a participant view, things seem to be working now; would be interesting to see what would change from what we have now. A: Sylvie notes we need a certain amount of cash to keep going. Under Merit, they advanced money out of their own pocket to keep things going. If we're on our own, we don't have those pockets to dig into. We need to build up those pockets on our own. We have paid down hotel for Denver; it will be in a year's time, so the runway isn't that long. Remember, as you look down the columns, these come in threes. Q: Todd Underwood, Sylvie made a good point; this needs to be skewed to make initial cash quickly. Companies to be able to buy unnamed headcount to meetings in advance, give the check in advance. A: That didn't make it on the pro-forma, but it has been bantered around. They wanted to introduce as little change as possible, but they do want to discuss it in the working groups. Q: We must find a way to completely insulate the program committee from the marketing committee. We can see a bit of it in the room, where there was more marketing talk and less technical talk, people get angry. Keep jackass serious engineers on the program committee who will piss all over the marketing presentations. Q: Bill Norton; of revenue items, which ones are new, and which exist today? The meeting sponsors exist, and down exist today; assume 350 out of 400 attendees pay; each column represents a year. 48, 49, 50, $110,000 oCF $220,000, $330,000 for OCF for next two years. Q: Neil, CW person can we have a graph of cash flow in pure picture form? That would be month by month, in vs out graph. A: will go onto website at some point, good suggestion. Expense slide. Would assume an executive director as staff, in the $80-$100k range to have the experience needed; would need them for half the year this year. Numbers will need to be tuned. legal fees, front loaded contracts to get things started. pretty straightforward estimates, based on small footprint operating organization. Promotional materials hotel costs, worked out from historical data, took high/low, took average as baseline number to work with. There's no travel listed there. As contracted outsource model, the...oh, the paid staff will need to travel, good point. Summary income side of things slide. People went over gold/silver/bronze already. If you disagreed before, you'll still disagree. Cash out slide now; previous slides were in number of units; this puts it in dollars, so you can see annual cash flow Q: Neil, CW, can we do a deal with Merit, for $30,000 they can become a member (for the domain name?) A: they just want to document a fair value transfer, terms to be worked out. They don't want people to grumble about the name being given over; this is listed as "brand acquisition". Q: Bill Norton, trying to understand, there's an entire year of bookkeeper and accounting, $1200, and $5000--this jumps out at him, where do these numbers come from? A: Some are from research like conference cancellation, and some are SWAG; this is pro-forma, if you have specifics, please contribute! accountants and legal fees are well within range for a small company. Currently, we get legal for free, but first year of a 501(c)3 are generally higher. Then later, a bucket for legal fees, just in case. Q: That means Tom Daily and Dyn corporation generously donated them (the legal fees, that is!) Q: Todd Underwood notes there's a different sense of involvement that happens with a member owned organization, like write checks, and donate services. He'd like to see this encouraged, not just through Patrick badgering people, but by having a way for people to help donate those resources. We'll have lots of technical volunteers; but what about non-technical things, like legal resources, etc. So think about how to volunteer and accept those resources. Q: Patrick notes we have Beer and Gear sponsors who paid $10,000 each for sponsorship in 30 minutes, so let's speed up. Even in pessimistic assumption, they still made the break even numbers, or within $10k of it. Q: Bill Norton, do corp sponsorships go up 42% in the pessimistic mode? A: no, that's a bullet, not an up. It's actually only 42%, period. The seriously pessimistic slide is exactly that, the membership doesn't go well, attendance is bad, sponsorship goes down, and the expenses maintain, so that's worst model; book big hotel with small attendance. Q: Charles Guker, open access working through numbers, based off past numbers, trying to understand based on numbers given from Merit; there's numbers in here for travel, equipment, etc. But what are the non-salaried expenses? Hotel costs, equipment costs, supplies, travel. Equipment is recurring; do we keep buying new equipment each time? A: About $15,000 to $20,000 listed in previous meetings for equipment; that covers the leasing of the A/V gear, etc. Those are numbers that were carried forward from what Merit had presented on their budgets. Offered to sell equipment through the university office; newNOG can decide if they want to buy that from Merit or not. The letters are posted on the newNOG for people who care. They are being careful to not underestimate. Q: Mike Hughes; stuff wears out, you want to refresh it, people want draft n wireless, etc. Especially equipment that is shuffled around. Q: Bill Norton--with staffing, are those fully loaded costs A: Those are base comps; benefits are a separate line item, not included. The bigger slide includes 30% of salary for loaded cost. And yes, they will be in an office, or a contract basis. There's been gear donated to merit for benefit of NANOG; will that gear get moved to new org; Merit will sell it to newNOG based on the assessment of what it is worth. Q: Tim Pozar, credit card transactions, 2% is low, should be 3-4%. Conference manager, contract negotiator; there is $30-40k that is aggregated to make the spreadsheet fit on the slide. The full thing would be four pages long. A: Patrick signs Tim up for budget committee. :) Q: Union labour for power distribution? Is that in the $150k? Can't be done by volunteers in NY, unlike SF. A: In current model, hosts pays those costs, so it's in the $110 already. Someone mumbles from back. Q: Charles; there are some professional orgs that do professional conference organization and scheduling; will professional do that, or will they disseminate that out to contrators? A: Yes, that'll be set by paid staff, then worked by contractors. Q: Will, quibbling over sub 1% items out of a million dollars is a waste of time. Q: Steve Gibbard, we have spreadsheets and models for running meetings, but there's a lot that needs to happen; he pitched in his $200 yesterday, Randy put his in today, if more people pitch in, we can make this happen. Q: Kevin Oberman, There's a lot of questions about what's rolled up where; would be good if the right people can make available online the detailed spreadsheets so everyone can be sure there's no major items that got missed. A: They'll do it as soon as possible. Give Sylvie a week or so. They can either publish the working sheet, with rolled up categories (but won't satisfy the deeply curious), but would convey some info. Give her another week for the breakout version. Q: Dan, this is a directionally accurate budget, in 90 days we'll have a much more accurate budget, and as we iterate through, we'll be closer to having a good, accurate annual budget good for a year afterwards. But taking his pro-forma budget as a first pass as being accurate isn't valid. For people who are in homeowners association, you get those budget reports year after year; they're boring, and they build them by copying other associations. So we need to talk to other orgs, and other associations, and learn about what budget items to include. Someone else mumbles from in back. Many people involved with this have been doing event planning in the past, and who know accounting, so it's not entirely a bunch of amateurs. Q: Denesh, would like to know how he can send in a check for $200-$250 if he's in the UK. Email board@newnog.org, and they'll send instructions on how to send money. Duane is treasurer, he will gladly take your money. Q: Bill Norton notes this is the first time many of us have seen these numbers, so we're told this is a done deal, we'll vote on this in a couple of months. There's a lot of turmoil, what it is that's really done. He's got an uncomfortable feeling we don't know what the heck we're doing. A: This isn't so much a done deal, as much as that there's timers on contracts that need to be hit. Merit has a responsibility to their shareholders to be cautious, so we had to pony up. Q: But contracts need to be signed, so this is going forward; he doesn't care about numbers, but the assumptions are highly questionable. The sponsorships, and other sources of money are based on a lot of assumptions. Maybe everything will be fine; but it seems only one group has done any ongoing conferences, and that's Merit. A: That's interesting fear mongering, but it's not productive. Q: Bill Norton suggests we pay Merit to do this for the first couple of years. A: That would be a case of outsourcing, which we are considering. Q: Ren thanks everyone on stage for answering the questions as well as on the mailing list. This is taking a swag for the brainstorm for the collective audience. We voted for these people as our steering committee to do the right thing for us; we don't need to wait another four years. A: Bill thinks that if there were more documents from both sides, there would be more agreement about this. Q: Patrick asks Bill to take it up on the side so we can keep the meeting moving. Neil from CW notes that he's worked on these types of things before, and feels that we're better along than many others; and his organization is very proud about how far along we are; and his organization would like to be one of the first corporate sponsorships. A: Note, the 501(c)3 status has not been approved, so donations now are *not* deductable for individuals; corporations are expensed as normal. Another note; to get this rolling, people up on stage had to put some money into this to get things going, if you vote against it, they'll be out a bunch of money, but that's about it. Every single member of the transition team has put some money into this. Q: someone from telecordia if you can put a paper address for those who use billpay on the website, that would be much appreciated. A: Yes, that will go up, do not be afraid it is in moscow, idaho. [from IRC, it seems to be: send checks payable to "NewNOG Inc." to: NewNOG, Inc PO Box 8726 Moscow, ID 83843 ] How can you shape the vision? join the working groups; finance, budget, communications, marketing, membership Q: John Curran, ARIN ARIN does have a position with respect to the transition; they do consider a viable operators org in the region to be necessary, so they helped with one of the conference registrations. They support this, this is too an important an item to fail. Patrick notes ARIN has been amazingly supportive about this; they have promised they are not aiming to cherry pick the conference, they do not want this to become ARIN-NOG. OK, Beer and Gear sponsors are waiting to have you drink beer and mingle with them. Go get your drinks and tee shirts now. BOF wraps up at 1756 hours Pacific time.