IPv6 Working Group
23rd May 2019
At 9 a.m.:
CHAIR: Good morning, welcome to the IPv6 Working Group. If you were thinking it would be abuse, just go downstairs, it's there.
Once you decide doing to the microphone and say something, please state your name and affiliation and so that everyone can know who you are and what you are doing.
The last meeting's minutes, I checked just before the meeting, there were no comments yet. Is there anyone in the room who had something to tell or should we just say thank you you Massimo for doing these again. Very effective minutes.
So the agenda, we are going to show you very soon, and this afternoon we have a combined session with the routing Working Group, it's actually more or less a tutorial. And we are going to see some things about v6 and routing together.
So, the other thing is that all the presentations can and should be rated, and you can actually win some prizes with that.
So, first on is Marco Hogewoning with an update on the ITU S G‑20 stuff.
MARCO HOGEWONING: Good morning. For those of you who don't know, I work for the RIPE NCC. I have presented on this before, I am going to say I have been asked to keep it brief, apologies, my schedule means I have to be at in two places at once so I will be run out of here but as soon as I am done, I will be here all week if you have any questions or want to discuss this.
These slides have a lot of text, if you want full historic overview, download them and read them in your own pace.
ITU study group 20 which is said to study Internet of things, smart cities and communities started its work in 2015 and shortly after, January 2016, already, that's more than three years ago, decided to open a few work items regarding IPv6. At that stage, we weren't involved but several of our friendly Member States and sector members objected on the spot so we even managed to walk out of the meeting when they made this decision as a form of protest. But yet, these work items started.
So, kind of after that in 2016 we got a liaison statement from the ITU saying we are starting working on IPv6 are you willing to help us, which we responded to you shouldn't be doing this, IPv6 work belongs to the RIRs and operational community. ITU doesn't really have a space in here, also we feel you don't have the expertise.
Well, that is kind of went on, nothing really happened so we decided let's join the meeting. So we went to it meeting in Dubai, watched what was going on, then came back in September, yes ITU thinks thinks two meetings a year about six months apart, together with ARIN made a formal contribution as we are ITU sector members we can do that, saying that IPv6 work you are doing, it doesn't belong here, it's not progressing, there is tons of mistakes being made, you clearly don't have the expertise, we recommend stopping this.
Again, that fell to deaf ears. So in turn, in discussing this, people said like, well if you are searching ‑‑ such an IPv6 expert why don't you help us. So in coordination with the chairs here we decided let's them take up on that. So impossible for us of course to bring the whole RIPE community to an ITU meeting so it's much easier if you bring the ITU to RIPE so we have invited them to Marseille, come over, explain what you are doing and show your work and have the experts take a look at it. Surprisingly, they said yes, even more surprisingly they send us the whole draft. And I must ‑‑ it's okay it's sometimes happens that ITU get transmitted out of the study group that is working on them, usually that is ITU internal. It's relatively uncommon for ITU to send uncompleted work out of the organisation to others. Let alone that they send it out to others with sort of a carte blanche saying here a is what we are doing, please tell us what you think.
Well, the community review happened and it was very clear that the current work of ITU wasn't actually up to par when it comes to IPv6 and numbering plans. The proponents did an attempt of explaining in Marseilles what they were doing so as your secretariat we submitted those reviews back in a nice compiled document as a liaison and then we were told informally like this is going to be discussed at the next meeting in China in December 2018, mind you so we jumped straight away from February when you started to review, basically to December, not much happened.
So we went in to China with your review on the table, but also as sector member, we made a contribution. ARIN made a contribution. The United States made a contribution, all saying we have looked at the RIPE review, we think you should stop this.
There was also contribution from the proponent with his story from the floor we got backup from Canada, UK, Germany, GMS I all saying stop this. There is where ITU turns geopolitical because we nearly had consensus and one Member State objected, quite possibly for geopolitical reasons, no we want could keep this. Can you explain. No, we just want to keep it. Okay. This escalated into the closing plenary where the chair, the chairman of SG20 made a comprise proposal saying let's give this one more shot, one more meeting, if people really aattach to this work they should come back at the next meeting with new text that addresses the concerns of RIPE. Otherwise it's gone. We took that as a compromise, sure we will see what happens.
So last meeting in April, in Geneva, we again together with ARIN made contributions saying nothing really happened, let's stop this. There is a long list of issues. Surprisingly, it went very quickly, no more geopolitical issues and the meeting quickly reached consensus. So as of last meeting, the work item on developing a reference model for an addressing architecture is no more.
Job done, so you say.
(Applause)
Hold on, hold on. Because looking forward, yes, the work item is gone, but it doesn't mean the problem is gone away. The fact that people started this work item was based on concerns with regards to the IPv6 deployment and the speed at which people, you, the industry, are deploying IPv6.
Monitoring, unless the whole world is going to move over to IPv6, we will see this keep popping up, whether it's in SG20 or elsewhere in ITU or in other fora, people are still concerned this isn't speeding up enough.
So we will be there, we have also been discussing over avenues with the development sector in ITU to do a bit more training on IPv6 to help them get along with developing a addressing plan according to you and our standards.
So that's basically it for now.
The work item is gone, we will remain to SG20 and to other work in ITU as my colleague Trish yesterday explained in cooperation. This isn't done yet.
Yes the US would like to thank the RIPE meeting and so I would and everybody who helped, the UK, Canada, Germany, everybody else who helped us and me personally along for that.
As I said, happy to take short questions now, otherwise find me in the hallway or over lunch or tonight at the dinner.
CHAIR: Any questions? Don't seem to be a very big rush on the mics. Thank you.
(Applause)
Next up is Geoff Huston and IPv6 Internet measurements.
GEOFF HUSTON: Thanks. I work with APNIC. A number of times I have presented to you on measuring v6 deployment from the end user, part of the reason why I did this is, I don't know about you but I was sick and tired about all the claims of so much v6 without substantiation or measurement of the way we did this has been relatively unique, we utilised online advertisements and the whole idea is to measure eyeballs, not infrastructure, not volume, not bytes, eyeballs. And the whole idea is with ads, ads actually allow scripting, you probably all knew this. What we do inside the script is actually send the user to a number of URLs. Some of these URLs are only reachable in 6, one of them is reachable in either protocol, v4 or v6. All of those URLs have unique domain names, all of them resolve to servers that we run so when the ad is running inside your machine, whatever it might be, a sequence of DNS queries and then an https query comes back to us. So we see you. And when we offer you, because we know we did, a v6 only URL, we are looking to see if there is contact. If there is, you can do v6. Routing is working, DNS is working everything is working, you can do 6. If we don't see you doing 6 but we see you doing 4, obviously you can't do 6. And also in the dual stack, look very hard at which one you preferred. Because if we are all running happy eyeballs, you should prefer 6 if it's available. Caveat sometimers and we are interested in that.
We do about 7 million of these ads a day, we see everywhere, even the Faroe islands, but the ads saturate the Internet. Everybody watches YouTube and gets ads. But there is more than that. Because we do a full packet capture. Full packet capture. So we understand a lot, lot more about what's going on with you and v6 end‑to‑end. And this is prompted this talk to actually look at reliability measurements.
How reliable is v6? If you are offering v6 is it the same standard and quality as v4? Or do users get wedged?
So, as I said, two URLs in this case, someone V had and run in v6, same end point on our side (4) so same you, same me, different protocol paths.
So, we are doing full packet capture and looking at these packets. And one of the things I'm really interested in is the SYN exchange, that initial TCP handshake is actually really informative. It's not a user mode operation. It's down in the kernel. What that means is, I don't get, if you will, browser noise interfering with this measurement. What's going on is the app is saying open, the kernel is going fine, send an SYN, ACK, it's as fast as you are ever going to get.
Now, I don't see what the client is doing, that is a mystery to all of it, that is running a script I can't see inside your machine, I probably shouldn't. Who knows what I will find. Anti‑abuse and all that. I'm the serve, I see everything. So I know the roundtrip time to you because the difference in time between the first received SYN and the closing ACK is one roundtrip time. It's the roundtrip time of sending the SYN/ACK back to you and getting the ACK. How do I know if you are broken? Well, if you can't send me a SYN, I have no idea. You keep your SYNs to yourself. But if I see your SYN, sorry, I just don't resist it, if I see your sin, I am going to send you a SYN ACK, I have got a packet capture, if I don't get the ACK you are not receiving v6 packets, you can send them, can't receive them.
So it's broken.
So let's start measuring and looking at that. Globally, I am looking at the percent of v6 connections (measuring) that don't complete. Just don't complete. Worldwide at this particular point, assuming there is a laser pointer here, around 1.5%. Now, when you look at v4, you get fractions of a percent and it's noise. 1.5% of v6 connections that fail is really, really bad. It just shouldn't happen. So, it's been worse, it's been 4%, things are getting better. 2017 was a bad year. But, you know, it's still not good.
Now, we can do a bit more here but the first thing is to actually understand what's going on and why. That's not 100 percent and it's not 0, it's in between. And when you get measurements that are in between, it's normally not the service provider's fault. There is more than likely end points doing firewalls. Now, sometimes they are sending me an IPv6 address that is unreachable, very, very small number of folk are doing that kind of inventions; they send me a v6 that isn't routed as a source address. More often it's a firewall. And sometimes it's a failure of transition mechanisms. Sometimes and I have talked to a number of ISPs possibly in this room, it's actually failure of your CPE, that you might find clusters where you have deployed the same CPE giving you failure whereas other parts of your network are just fine. So there are a number of reasons for this that sort of lie behind that failure rate.
But as I said, it's a lot to do with the transition mechanism. So, this is T Mobile. The average in the US is just under 1%. T‑Mobile's v6 failure rate is basically 0. Why? 464 XLAT which I actually thought was pretty weird at the time is actually brilliant because the underlying mechanism is always 6, it's 4 that is the the add‑on because 4 is in the handset and NAT and everything else but if it's 6, it's native from the end point, you know, your pocket, all the way through my server. So that's what you can do with a transition mechanism which is based on underlying v6 connectivity where v4 is the overlay. The v6 performance is absolutely brilliant. So yes this is why it's 464 XLAT. There is no processing and no client handling and state. So if you are looking to maximise reliability and you are looking at transition mechanisms you shouldn't discount 464 XLAT it's actually a nice idea.
Reliance geo in India, this is scale, this is ‑‑ God knows there are a lot out there, about the largest ISPs. Their mobile roll‑out is v6, v6 native using 464 XLAT and it scales like crazy, the blue line is still down there at the bottom. The earlier experimentations in 2016 managed to clean up their ‑‑ it's rock solid and just works. In Vietnam, it just doesn't work. Obviously, they are not using 464 XLAT. The failure rates up around 7% which is completely unacceptable. And it's been sustained like that for two years. If that was all of their servers and 7% of connections always failed, they wouldn't be in business two years later.
So what's holding them up, what is managed to make all this work? Happy eyeballs. If it wasn't for v4 coming along and going on hang on, all bets are off, we will go down the AAAA path this would be a disaster. And the only reason why they don't fix it is happy eyeballs. Because it delays the connection by a mere 250 milliseconds, what is the problem? So oddly enough, happy eyeballs both saves them and stops them fixing it at the same time. Which is a weird contradiction it.
On the balance of probability doing the right thing by the customer is the best thing. So in some ways happy eyeballs is a good thing, it saves them from themselves. Having a connection failure rate of 7% of all users running v6 in Vietnam is really horrible engineering and doing nothing about it for more than two years is just, I don't understand. I don't understand how you could tolerate that.
But if you are a customer of the cooperation for finance promoting technology Vietnam who are doing such a great job of promoting v6 their connection failure rate is one in four and it's getting worse.
Again, happy eyeballs is saving them but what are they doing so wrong that they are getting one in four v6 connections that's not completing a SYN handshake? By the way, they are going to Singapore, and if you know your math, it's just the next country down, it's not far away. This is not a massive transit with huge delay. How are they getting it wrong beats me. Why is it getting worse. Because they are professionals, I suppose.
(Wrong).
Vietnam is not the only one that has this kind of problem. If you are living in Panama, it's really small in front of me here. Holly crap. That's 50%. If you are living in Panama, v6 is a joke. Half of the time it fails, it just means that doesn't work and it hasn't got any Bert. If you are living in Turkey it grinds around 10% on average. It has been worse, things getting Bert but still pretty pathetic. In China, now this does surprise me a lot, huge amount of work in China, a lot of people and a massive government programme, we need v6, and a whole lunch of their largest providers, China telecom, etc., have all been doing a great amount of work but their overall connection failure rate is still sitting at somewhere around 5%. This is not a good job, it's a really shocking job.
So, for users in those countries, Bangladesh as well and it's broken, I believe it's dual stack transition technologies that are failing them, happy eyeballs makes it all go away.
So as long as they are running dual stack there is no problem I need to fix. And interestingly, it's their problem, it's not v6, because why do ISPs in the US, Sweden, Thailand, Korea, do such a good job, in India? So, what are they doing so right that these guys are getting so wrong? I actually think it's the transition technology they chose. The one I think that the IETF finds really hard to do is to make decisions.
So when confronted with A and B and the Working Group is sitting there do we publish A and B, some bright spark says let's do both and the transition technology debacle, they managed to do 30. Little wonder that we gave no clear guidance. Little wonder that ISPs deployed really quite crappy technologies because the IETF didn't really say do this and your users will hate you, do this and you stand fighting chance. No, we just published all 30. Some of them are abysmal. DNS 6, 4 doing lies in the DNS to make your transition work, oh, my God. The first person that deploys tunneling DNSSEC out of their own. Any kind of doing DNSSEC is out on their own. These transition technologies that orchestrate the ‑‑ were a fine academic research paper. Full points. Any fool that tried to implement them was just that, a fool. It's ridiculous. You can't orchestrate these pieces and expect it to work seamlessly because this happens. Don't do it.
And don't forget, too, that happy eyeballs is saving you today, but the aim of this entire exercise is not to make v4 persist for your great, great great‑grandchildren. That's not the goal. V4 is not meant to save your bay con, it's meant doing away. V4 is meant to disappear. So the goal is, that happy eyeballs doesn't save, happy eyeballs is meant to make v4 irrelevant, bus if happy eyeballs really works, when given the choice you always choose 6, no one chooses 4, hell I might as well turn it off. That was the goal, not oh, my God, v6 has failed let's try v4. So, this is the whole thing. Dual stack is not meant to save you. And what we are doing right now is using dual stack as the crutch for really poor implementation of transition technologies. Part of it was the testify IETF's fault and part of it was industry's fault that no one looked, users wouldn't complain because spotting at 250 millisecond in TCP handshake is kind of technical bullshit. Okay so it's a little slow today, it's a little slow yesterday, what's the difference? Users don't complain, you don't care.
So, I think that just says all of this. Oh, happy eyeballs, though, has one severe twist, and let's look at that a little bit harder. Previously before happy eyeballs, what we did is we'd launched a connection, open, using the AAAA address and waited for the operating system doing couldn't do it, couldn't do it. If you are on Windows, the operating system takes 21 seconds and I think four packets. Oh, well that didn't work. 21 seconds is an eon. But if you think that's slow do it on your Linux box and try something that doesn't use happy eyeballs like SSH, try and make an SSH connection to something that doesn't respond, and start your watch, tick, tick, tick, 108 seconds later, I have died and disappeared, I can't get there. So the original model was, try v6, wait for an eternity, 108 seconds, pre ‑‑ was so much faster at 75 and only then try v4, loudy idea. Applications said your operating system guys are living in the wrong century, this is no way to could it. So it was the application that started happy eyeballs. They send down DNS resolution requests independently, get host by name calls. And if they get back a v4 address first, they will wait for up to 50 milliseconds for a AAAA for a v6 address. So even if the DNS resolution is slightly slower, there is a 50 millisecond sort of patience time to get back both addresses. And then, you start unconditionally a v6 handshake. You send the SYN out in v6. You wait for 250 milliseconds. If the TCP call comes back, you are done, it's a v6 session let's run. If 250 milliseconds elapses and nothing has happened, open up the v4 connection. So 250 milliseconds is a really interesting number. You are coming to me, and let's say you are in New Zealand, and let's say you are load sharing ‑‑ Hi Sebastian ‑‑ you are Vodafone and load sharing and got two ways out of that country, there is a cable running to Australia where everything is sane and reasonable and to the US where everything is broken, if you send your v6 packets and don't forget you are going to Singapore, out through the US, and your v4 packets to Australia to get to Singapore, what's the difference in RTT between a straight shot to Singapore and a double shot across the Pacific? It's more than 250 milliseconds. All of a sudden Vodafone New Zealand are going why is there no v6 in your experiment chair? Because routing, happy eyeballs, because of that magic number. So the assumption is in happy eyeballs and this is the magic assumption that most of you don't quite grock, to get to the same end point you need to use the same path for this to work.
Now, if all of you were singularly homed, if you are load balancing you are in hell, because if you are load balancing across quite diverse paths, New Zealand, and you are going literally east and west, then there is a distinct possibility that for some destinations you are going to get it horrendously different and as soon as that kind of deviation happens, you get India. Who for some insane reason, maybe it was British commonwealth, God bless the queen, decided to send all their v6 packets to the UK because you can, all the v4 packets went straight to Singapore. It's the straightest path. All the v6 packets went to England. Instantly, the v6 preference fell to nothing, because the 250 millisecond timer went no, you are going to use v4. It wasn't just India. India noticed and cleaned it up. They basically put their v6 routing where v4 went. But Vodafone, this is today's graph, well done Vodafone, they started mucking around at the end of last year and, you know, yes, they are not getting the v6 they wanted out of all of this because their asymmetric paths are so asymmetric that sometimes it just prefers v4 all the time.
So this is why when we look at the average RTT difference, is 6 faster than 4 or 4 faster than 6, it doesn't matter. Because of happy eyeballs if you diverge too far it really matters because the way you make v4 die is to always prefer 6 if it's available. But no users is going to give you 108 seconds to fulfil that goal. You have only got a quarter of a second and that is the margin you have got to engineer into, if you are going to do load balancing and diverse paths you need to understand don't make them too diverse or you are going to fall into this issue for some popular sites you might actually not be doing 6.
So here is China. Singapore is just down there on equator. 100 millisecond difference from v4 to 6. Across the Pacific. So they are going north/south in for v4 and east west for v6, it just doesn't work properly. A lot of people in China, a lot of things to stuff up. They do it so well sometimes.
So, I am getting close, I have got five minutes. Three suggestions:
Really, really careful about your state transition choice. Really careful. Avoid stateful, avoid pulling in the DNS, avoid intricate multi‑stage hops, it isn't going to work. Keep it simple. 464 XLAT is about the best I can same. Don't do v6 in v4 encapsulation. Your customers will hate you, nothing will work.
And keep your routing congruent, don't let it split. But that's not quite all, and I have just got a couple of minutes left so as we do this let's remember one other thing that is completely broken in v6 and we don't know how to fix it. V6 changed where the extension headers live, they are a shim that's inserted between the v6 header and the transport protocol header which means if you are a switch that wants to find the transport head you are you have got to unravel a chain. If you have got a small CPU you are going to spend cycles you haven't got, what you do is drop the packet. If there is an extension header including a fragmentation header you drop the packet.
So, there is a lot of talk about v6 in the DNS, there is a lot of talk about DNSSEC in the DNS, there is a lot of talk about large UDP packets in the DNS which means there is a lot of talk about fragmentation in 6 in the DNS. 38% of the time users can't receive a fragmented v6 packet in the DNS. If you think of v6 only network is in the immediate future, that number has to drop to zero. Somehow. Either avoid sending fragmented packets or clean up everything that drops it. One way or the other, you have got to change that.
End‑to‑end, TCP works, well it doesn't because if you ever get fragmentation and you send a fragmented packet in TCP 22% of users all around the world go didn't hear. Again, if you think v6 is going to be everywhere and there is going to be no v4 that can't be 22% it's got to be 0. You can go and change all those really cheap switches deployed in places you have never heard of, you can, you can try, good luck. Or you can avoid using any form of extension header in v6 in your apps, the choice is yours, not mine.
So, there really are ongoing with v6 reliability and quite frankly, some of it is right near the edge; old equipment doing stupid things. Darwinian evolution will kill them, the power supply is going to blow up, over time that will improve. But the whole issue of pat MTU management and fragmentation they are still pushing out equipment that drops extension headers in packets, you want cheap switches, they don't do v6 properly. Stop buying cheap gear. You are all going to listen to me, aren't you? Change the protocol oddly enough is easier, oddly enough changing the protocol is easier. So, if you are looking at protocol design for God's sake don't frag the packet, don't put in extension headers, you are joking, you are on a path to failure. Thank you.
(Applause)
JAN ZORZ: Geoff, thank you very, very, very much for these measurements. I may add one possible reason why this SYN ACK is happening, actually we documented this in RIPE 690 best operational practice, where we said don't do, with the current end host, don't do dynamic prefixes, don't change the prefix because the user needs to change the prefix behind the CPE and that doesn't work. Exactly what you described here, this is exactly what we documented there. And I will talk at the end of this session about the follow‑up from RIPE 690 because Gert was going through the roof, we can't go dynamic and, you know, we went back to the IETF and said can you please change the slack to actually follow the topology and actually don't keep all addresses that are not given to you any more, preferred for one month and don't keep addresses that you don't own any more for three months, so the more operators deploys IPv6 in a way that they are deploying IPv4 and change the prefix every 24 hours or every hour or, you know, the more frequent they change the prefix, the more breakage you will see there because you have prefixes that cannot route back. And I think ‑‑ and thank you for ‑‑ thank you for proving what we documented then, are trying to do in the future. Thanks.
GEOFF HUSTON: Thanks.
(Applause)
GEOFF HUSTON: A popular suggestion.
JEN LINKOVA: Google. I am a bit confused when you mention DNS 6, 4 here, so those failure rate I presume it means v4 succeeded but 6 didn't, you eliminate when v4 did not work as well?
GEOFF HUSTON: Well, the ad itself is either 4 or 6 at some point but the ones that are failing, I assume the ad was delivered in 4, there is a certain amount of handshake between me and the browser to get the experiment up and running, that's in 4, normally. But can be in 6 but it's probably in 4 for the failure cases. The first time they really exercised v6 and that's it when they get a v6 only URL so 4 has been working all the time and then they presented with a v6 problem.
JEN LINKOVA: Why I am confused is because DNS 6, 4 and using transition technology which rely on it would assume if your v6 does not work you would not even get to v4 only resources so I am not sure how ‑‑ why will you see v4 working but v6 not and v6 only network
GEOFF HUSTON: Would every DNS resolver send me detail as to its state, would that I could see inside the state of end client and I could see this in great detail, I can't. What I can see is that where I know your transition technology and the cases of 464 XLAT we know who is deploying it, I see rock solid performance in v6, it always connects. In other cases which seems pretty obvious they are not running 464 XLAT I see great unreliability. Now I know there is DNS 6, 4 out there, I know what was the Chinese one, IVI or something? There is all this goop out there and my general impression is as soon as you have shared state, as soon as you try and sort of orchestrate a number of actions, you are in a place that it's not rock solid, and I will put DNS 6, 4 into that bucket of it's not rock solid.
JEN LINKOVA: What I am thinking here if ‑‑ it just means if you deploy v6 only however do you it, it doesn't matter, and your v4 relies on v6 then you shouldn't see those networks with a high failure rate because nothing would work with them.
GEOFF HUSTON: You have got to solve the extension header problem so you have taken the rocket ship to Pluto already right now, you have got to solve other issues before v6 only is actually a credible statement.
AUDIENCE SPEAKER: Although there are many times in life when a multiitude of options is really nice, I agree with you that in transition mechanisms this is not the case. So, I would like to ask, do you have actual data of which transition mechanisms cause breakage so if you see breakage in various ASs do you have any information of what kind of transition mechanism they used or is it just a hand ‑‑ that introducing stateful mechanisms ‑‑
GEOFF HUSTON: I have no idea what you do inside your network, it's your network and your packets and your rules. I do know of certain folk who have published their transition mechanism, T Mobile and GO are classic. And when they say I'm using this and I see a result here, it's easy to put that together. But if I see something else, I have no idea what you are doing, I really don't, unless you tell me. Other than that, I am just guessing.
JAN ZORZ: I was thinking about it with your measurement do you have an option to try to understand if the same machine on the other end is changing prefixes over time? Some sort of cookie?
GEOFF HUSTON: The ad runs for precisely ten seconds or all the answers are finished. You might be watching the video for two hours, you who cares, the ad is really quick, I have only got ten seconds
JAN ZORZ: You cannot identify the same machine on the other end, if they do the test twice?
GEOFF HUSTON: Well, unless you are doing privacy address cycling every two seconds which seems a bit weird, I won't see it. Nice try.
CHAIR: Thank you.
CHAIR: Next up we have NRA and large scale development ‑‑ deployment of IPv6 enabled hotspots.
ENNO REY: It's not just me giving this, I am giving this together with Christopher with whom the two of us do most of ERNW IPv6 projects and this has a background say in the project but also in the fact that we have a conference running like 12 years and in that conference, called troopers, the conference network, the default SSID is v6 only since 2016, so we have some insight, well, it's a few 100 users, but we have insight what is happening say once you do IPv6 only with DNS 6, 4 in that case, I think in wi‑fi you can't do anything else, as 464 XLAT is not available for most common desktop operating or laptop operating systems.
In any case, background here is twofold: Our own conference network and the fact that a customer organisation brought us in late 2018, they planned to run v6 in hotspots, several, mainly throughout Germany but there might be more in other spaces. And they ask us okay, can you help us defining a strategy and help us with an implementation plan to do v6 there. So, the talk is split in three pieces. We will shortly discuss strategic elements, v6 only was going to a dual stack, is one, and that is the most important one. Then we will look at, say, typical adjustments, what you do once you run v6 and wi‑fi networks. And there will be some additional considerations, what about the monitoring infrastructure, what about telemetry, stuff like this.
That is the kind of background, as I mentioned, next to the fact that we run out own conference network.
Several hotspots and the thing is those are planned say ‑‑ these are not the exact details, I can't give the customer name right now, there might be another talk in a year in the have, 6 Working Group where we will be able to discuss more details. Right now, we can't. Let's imagine at say wi‑fi deployment in public spaces like supermarkets, like malls, so you have a specific user population there. And the thing is, as I mentioned one of the questions v6 only was a dual stack, and another thing to keep in mind is here it's a free offering, from that angle, one could take a stance like, well, we don't really care if it breaks or if something is not working as what could be the expectations of users, given it's free and in case ‑‑ most of them will have mobile over data plan anyway. But on the other hand, say when it comes to like management, when it comes to the purses being the project sponsor kind of, there is concerns, well, we ‑‑ even if it's free, we want to avoid bad press in the sense of, well, you know, they sell crappy products and their wi‑fi doesn't work. So there is a bit of an interesting decision space.
Actually, I think one of the reasons why they brought us in, besides we have some technical expertise in this space, is that they didn't really want to take the decision v6 only plus DNS NAT 64 ‑‑ was a dual stack, say on their own it's always easier to bring in some consulting company to rely on. This is what the experts recommended.
We noted that in the beginning, there was lots of, say, concerns floating which ran really ‑‑ weren't based on technical insight, what could break and people had found there is this three‑year‑old block port of somebody describing that in a wireless, Android didn't work, v6 only, stuff like this. That was the reason why we decided to build a test lab which I described in the lightning talk on Monday. What we learned and tried to, I wouldn't say educate them on but it was like okay think about ‑‑ think about who is the users, what type of applications do they use, what are the expectations, what can you assume for operating system platforms, say for example at troopers we have very, very few problems, issues actually in v6 only, people don't even know, it's a default SSID, but they are the user population is technical people very ‑‑ very technically savvy which have a certain set of devices, usually quite modern ones which would not be what you find, say, in a mall in certain places in Germany there is all H groups, there is lots of ‑‑ there is a lot of heterogenerality when it comes to users and the platforms so we try to, let's find out what actually is there and what are the conclusions when it comes to, say, a technology stack.
Another thing to keep in mind and what we learned as, or what we tried to convey as a message, say time is your friend here. It's planned to have the first pilot implementations in the second half of this year, and start with a deployment maybe in 2020, so this can be taken into account as well.
As I mentioned to factors coming into the decision, is like, well, what is the expectations, how to handle those from a communication level, what to tell, say, the users in case something doesn't work, how to community with management, device platforms and all this, and we strongly recommend, once you think about doing such a thing in your organisation, do a lot of testing. This really helped to have a solid database for a decision.
As of today, May 2019, there are two main, say, types of applications where you should expect problems in a v6 only NAT 6, 4 setting is actually games, multi‑player games as mentioned on Monday, fortnight, just performing DNS resolution for an A record. That is one of these types of things. And VPN clients. For the type of, say, deployment I talk about here, VPN client running say on corporate desktops or corpse laptops are not really a concern. This is not ‑‑ they don't expect many of such say users being in the overall population using the service, and they expect like, well, those users can connect in case there is problems they have a mobile again with a data plan.
But, for example, you might know that at Microsoft, there was a huge effort which, the lady responsible for this talks about publicly a lot, and they had a bit of a setback, they tried to implement v6 only in their guest wi‑fi, and then say VPN clients not working, that is an issue there. But from what we can see and there is a lot of discussion about this in many circles there is a lot of progress so apparently problems are going away right now very quickly.
One last slide ‑‑ or thing to consider, at FOSDEM ‑‑ it has the same approach with NAT 6, 4 since I think two years as well, and their network infrastructure is kind of sponsored by Cisco so they are using their gear and they have some insight what is happening, some telemetry data, and they looked at what the differences in a traffic profile in dual stack versus in the v6 only network and it turned out in dual stack network they see much more E SP, IPsec than in the v6 only, and their conclusion, and I think this is a sound conclusion, and it's ‑‑ they have an interesting, there is an interesting blog post on the Cisco blog, their conclusion was apparently, people who want to use a VPN clients are ‑‑ switch from the v6 only to the dual stack network, to make it work. So this points in the direction there is a specific type again of applications which might have problems as of today. This is just three months ago.
I already mentioned time or the trend might be your friend. That was one part of the message that we communicated in the project. We were like, okay, there might be problems as of today, but fastforward six or 12 or even 18 months those problems will be gone. So again we strongly considered, and actually we recommended go with v6 only. That is say the message in the project and that is the message we want to give to you here as well.
In 2019, as I mentioned on Monday, going with v6 only plus transition in a wi‑fi network is actually doable for the vast majority of things.
Quick remark here: Once you do this, you should still distribute DNS resolvers by a combination of, put it into route advertisement option 25 or for stateless DHCP 6. At Cisco live they had a NAT 64 setting but they provisioned the DNS resolver only by route advertisements, which was a bold move actually, and it meant, for example, I myself, I run window server 2016 on my laptop, I couldn't connect to that one as I didn't getDNS information. We think, going with dual pronged DNS resolver provisioning approach is a good idea. Must be done, once you consider going with such a strategy.
Then there is second part on the what type of adjustments have to be actually done in the wi‑fi network.
Christopher: As enknow has already said we are running the IPv6 only SSID was in our conference network and when I started to have a look what does this mean in terms of deployment, there are a couple of things you immediate to consider when you are deploying IPv6 in general in wi‑fi networks. The first one is wi‑fi is shared media, and just for the record, yes, even with a dot 2 or wi‑fi 6 how they call it, don't get fooled by the marketing guys, it's still shared medium and maybe they tried to tell you it's a wireless switch, it isn't. We will still ‑‑ we are still having the shared media even with the 8 O2 .11A X.
As most of you might know IPv6 involves a lot of multicast‑based communication. We have router solicitations and advertisements and duplicate address detection, neighbour solicitations, and what effect does this have on the shared media?
Well basically, as it's a shared media, media can only use by one client at a time so the basically means if a client is sending a multicast packet the access point needs to physically transmit this multicast packet to every connected client. And as we do not have any acknowledgment mechanism for broadcast on multicast packets within the wi‑fi medium the access point is constrained to transmit this packet on the lowest supported transmission rate. This can often translate to one end bit, 6, 11, so this basically means that the physical medium is locked for a longer period of time, depending on the supported data rates. And this of course wastes pressure air time which you want of course to optimise to ‑‑ to optimise and increase the general performance.
So, another thing you need to consider, especially for large scale deployments is the interaction between the control and the access point as well, let's say for example from the side the controller receives a route advertisement and the controller needs to duplicate this packet and transmit it to the actual access point to the access points over the Unicast tunnel. So if you are just having 50 or 60 access points this shouldn't be much of a problem, as soon as you are having more than 100 or 200 access points it might place some burden on the CPU of the controller so there is another case where you might consider switching from a Unicast tunnel to a multicast tunnel.
Another thing to consider is battery power of wi‑fi clients. As the wi‑fi clients typically go into a radio sleep mode but they have to wake up when they have to process the multicast packets, and how this is done typically, the wi‑fi clients signals the access point, I am going to sleep now, please inform me if you have any multicast packets which I need to process.
So, actually the multicast packets are stored on the access point and that signals the client, hey, please wake up, you need to process some multicast packets which of course then drains the battery life and power of the devices, so it might make sense to get some form of optimisation so that not every multicast packet from every IPv6 enabled client will get transmitted to everyone. As I said, this might have a negative impact on the performance. There have been some discussions within the IETF regarding this space, there are a couple of drafts and RFCs floating around which are covering this topic more in detail.
So, at the end of the day, what does this mean in practice? Well, you should consider to perform some tuning, tuning basically once on the controller level is there any possibility to proxy, throttle or block some form of traffic? What about the inter AP communication or the communication between the access point and the controller and then optimising the layer 3 infrastructure, for example, properties of route advertisements, of neighbour discovery or other multicast‑based protocols, for example, MLD, do we have the possibility to tune some parameters to make IPv6 just less noisy or chatty, to optimise the air time.
Actually, one thing I forgot to mention here regarding the multicast‑based transmission is, the usage of link local multicast neighbour resolution protocols in basically every modern operating system, be it MDNS, be it bonjour or stuff like that. You should take this into consideration when you are planning your IPv6 deployment and wi‑fi so you need to answer for yourself. Do I need these packets, actually? And if the answer is no, at least I have chosen to just drop all these packets because they are really, really chatty and they are really hurting your efficiency especially in wi‑fi environments.
And as I was curious, just a quick anecdote:
I think it was on Monday, I ‑‑ started on the dual stack ID here to get an idea how much traffic and chatty traffic is there and I just captured time span of around about five minutes and in these I captured around 100,000 packets and 50,000 packets were only MDS packets which is had no use case, at least for this conference network and were just wasting precious air time.
Of course in this case as I was in a dual stack SID I am in this case for v4 and v6.
Coming back to the tuning, what can be done? Actually, one thing you should consider and this heavily depends on the actual controller you are using, here this case and within our deployment and our conference network we just happen to use some Cisco stuff. So I am referencing here the Cisco WLC. What you can see here better or worse is actually to optimise the multicast distribution or the distribution of multicast packets, is that the WLC implements some form of proxy, so if a client sends a multicast‑based neighbour solicitation, this is not transmitted to all wireless clients. Instead, the WLC is answering for the actual receiving client so that the multicast packet does not have to be transmitted to all over the wi‑fi medium. So this is one form of optimisation which can be performed just to ensure that the ‑‑ that this multicast based neighbour solicitation is not ‑‑ does not need to be transmitted to all wireless clients.
Another thing to consider is the throttling of route advertisements. So these are typically sent out either periodically or in response to a received router solicitation and at least for our deployment as the majority of the deployment is wi‑fi, but we also have a wired side and within the wired side I have chosen a pretty aggressive route advertisement interval from about, I think, four or five seconds and I just wanted to make sure not every five seconds the route advertisement gets transmitted to all wi‑fi clients. So what have I done? In this case, the WLC also includes a mechanism to throttle the router advertisement just to ensure that the router advertisements are just transmitted over the wi‑fi connection or link in a specific amount of time.
And also, as we are running security conference, I am also want to make sure that at least from an IPv6 perspective, I have done everything I could from a security point of view to implement some form of IPv6 security features. Luckily, at least in this case, there are some features readily available on the most recent controller version, which are also enabled by default, for example the most two important ones route advertisement guards and THCP guards.
That is the stuff we have done on the controller.
On the gateway configuration on layer 3, here again as I said, we just happen to use a Cisco box for our network. What we have done to reduce the amount of multicast packets, was in our network we have increased the router lifetime to 9,000 seconds which is the maximum value which you can configure and specified within the RFCs, and we have increased the reachable lifetime to 900 seconds so this means that the entry on the receiving client, the entry in the neighbour cache for the default gateway stays in reachable state for 900 seconds, so just to reduce the neighbour unreachability detection mechanisms implemented with an IPv6.
Unicast solicited RAs, this means if a client sends router solicitation and the router is answering it does not answer to the link local multicast address but instead he is answering to the link local address of the actual client who sent the router solicitation. And these have been some values which has been inspired by Andrew Yourtcheno, he used within the Cisco wi‑fi deployments as well.
And here just for your reference, this is the configuration of the actual interface, as you can see here link local address, global address, here the reachable time to set to 900 seconds here it's 900,000 because wants to accept or have milliseconds so I had to configure 900,000 milliseconds. We had ‑‑ here we do have the actual provisioning of the DNS service, one for over the router advertisements and binding DHCP pool for the actual provisioning of DNS.
So, that's regarding the technical intricacies, and what you should consider by ‑‑ to optimise. Another point which enenknow will discuss is what about supporting infrastructure.
ENNO REY: There is mainly two things you have to keep in mind in addition to the technical configuration of the wireless and the infrastructure side. That is there is more infrastructure in wireless, in such wi‑fi hotspots it's usually there is a captive portal. At some point it evolved. In this case, the guys reached out to the vendor, like okay what about IPv6. The vendor is one of the main providers of captive portals. The answer was, well, we will have it by mid‑2019, so we haven't got an update yet but I would be optimistic it will be there soon but that's a component you have definitely to keep in mind, not only from like can be say, can you talk ‑‑ communicate with the NTTs, the components of the captive portal but also whatever radios ‑‑ what about radios, what about locking the things a bit happening in the bag. And there is say monitoring and telemetry. Once you do this, you definitely or we from the say organisers' perspective at a conference we definitely want to see what is happening, how much of that is used, what is the number of translations and stuff, and here actually, it turned out that at least the infrastructure we are running, Cisco, as the main gate way, there is no easy way to get some counters on actually NAT 64 translations so we had to do a bit of work around which in that case was the with the embedded event manager. There is a way to get, I won't cover this in detail for time constraints, feel free to get back to us if you are interested to learn about ‑‑ more about this. Actually, there is a bit of a clutch to get hold of the number of translations happening, and talking about telemetry, there is nothing thing we were very interested in to get an idea what about DNS resolver ‑‑ how many do use the one provided by route advertisements as opposed to those via the DHCPv6, we set up two DNS instances running on different IP addresses. To get an idea. This raised within the organisation team a number of interesting questions is this privacy invasive, how do we make sure that we don't capture any data that could be, say, regulated. That is a thing that must definitely be kept in mind once you think about hotspots, wi‑fi hotspots, what type of data is there by default and what type of data are you interested in foretell em tree and for understanding what is happening reasons and how are these data, say, regulated. We had a number of discussions around this in our own space, and with the customer of the project. As right now they already have some, there is a predecessor of what they plan to deploy, and they already have some visibility as for client platforms and stuff but there is meg addresses, it becomes debatable, are those covered or not. So this is not just a technical thing. Here is another mention which definitely has to be closely say evaluated.
That's it. There is one last thing for the conference network, we are in control, how do we get the feedback loop and we find out if something doesn't work and how do we incentivise users if there is two offers offerings. In the case of troopers, the conference, there is a so‑called challenge going on, people can get points. It's CTF is part of that but there is all other types of, say, things to be done to get points and we actually, we were like, okay, dear people, you can get points once you use the v6 only and you can report anything that doesn't work, show us, and in case it doesn't work and you ‑‑ say in a user forum that gives extra points. And I think in that week, a lot of was done for the ‑‑ for v6 global deployment as there were lots of window tickets as people wanted to get those points.
So, concluding:
Once you think about deploying wi‑fi hotspots with v6, some decisions have to be taken, v6 only, what is dual stack. We strongly recommend go with v6 only, it's 2019. It works. Specific configuration has to be done, this is a space where we can have a check, it can be done it's technically possible. When it comes to telemetry that is a space where you might encounter most of the problems. Thank you.
(Applause)
CHAIR: Thanks. Any questions?
AUDIENCE SPEAKER: Into the question but a little comment. You mentioned that you saw huge amounts of NDNS traffic here in the conference network, at least in NAT 64 network. I just wanted to mention quickly that people at the IETF are quite aware of that problem, and for example for service discovery we are thinking of mechanisms to shift that to Unicast so that we are actually using less of the multicast, at least in this problem domain.
AUDIENCE SPEAKER: Blake with eyebrows. Thanks for putting this together it's really helpful. It's really cool.
CHAIR: Thanks a lot.
(Applause)
MASSIMILIANO STUCCHI: Good morning everyone. I am Max, my name is a bit longer but Max is fine. I am the IPv6 programme manager at the RIPE NCC and I wanted to show you, I will be brief, I only have three slides, I wanted to show you what we are working on and ask for a little bit of cooperation, if possible, if someone wants to chip in and start to help us.
So, what happens: We have seen a presentation from Geoff. Networks are trying to move to IPv6 and some of them are not doing it properly. Actually, many of them. So, we have received via our external relations group at the RIPE NCC but also when we go around and give training courses we find out that people have ‑‑ don't have a good ideas on how to run an IPv6 network, or they can't find exact documentation, they cannot find good guidance. So, especially for mobile networks where things are more strict and a bit more complicated than they are for fixed networks, there is ‑‑ there are very few parts of documentation available, and vendors are a bit tricky in that sense. So we identified this as an issue and together especially with my colleague we started working on ‑‑ started thinking about what can we do, can we think about writing a document to help these people, document the best practices for mobile networks for moving over to IPv6. And then we started. So we contacted some operators, we built a list, started talking to them, and asking them, divided into two groups, like the ones that actually made the step to move to IPv6 and the ones who haven't done it yet, to ask them both what went wrong in your approach and if you can tell us what did you do to fix that, but also if you haven't moved to IPv6, what is keeping you from doing so.
And then we started building a list of issues, so we have a list of topics that we put together, we already have document structure that we put together back in February and then we have, we don't have much text yet but we are working on it. So the next steps are to try to talk to more and more operators. What we ask to you do is maybe if possible have a one hour chat with us on Skype, any other system, so that we can address directly what the issues were and then we keep everything confidential but we will use this to fill up the document with more and more important and interesting parts.
And then the goal is to have very, very soon a draft for review so that we can put ‑‑ push it to the community and see if there is any guidance we can get about that.
So. This is it. Just as soon as they showed me the "stop" sign.
JEN LINKOVA: It's not fair but I want to make a suggestion at the mic. I have seen situations where people were writing documents regarding v6 deployment and mobile operators and also were not able to reviewers because everyone who could review, because I think there is a kind of ‑‑ gap between don't know how IPv6 works it's not the same people who know how mobile networks works. Could you find someone who knows how mobile networks works and give us normal network engineer's tutorial.
MASSIMILIANO STUCCHI: We are talking to both people who have done it and those who know what went wrong and there are a few of them who have already given us the permission doing ahead with them, and the others we want to see if, with time we can Mcthe document better and better by also following what they are doing and seeing what issues they find.
JEN LINKOVA: Shall we expect tutorial on next RIPE meeting, could you tell how 4G or 5G works?
MASSIMILIANO STUCCHI: I cannot promise that but we can discuss about that.
CHAIR: Any further questions? Thanks.
(Applause)
And last today up is Jan Zorz.
JAN ZORZ: So, you have heard Geoff, right? Yeah. It's a problem. I work for Internet Society but in this case I am here in my personal capacity and the views does not necessarily reflect the views of my employer. Get that on the table.
Who of you remember RIPE 690 BCOP? Hands up. Not many hands. All right. So, we wrote this RIPE 690 best cooperational practice document that deals with the sizes of the prefix allocations that ISPs should assign to users, to end users, and also how to deal with this. Should they be dynamic or persistent or should they be ‑‑ should they be dynamic or not or how we call them, persistent or not. This means should the ISP change the prefix every 24 hours or change the prefix every time the CPE reboots and we figured out that, in real cases, this is not a good idea because the end host, SLAAC, does not follow the topology change properly.
When we documented this and people started deploying it in a way that they keep the prefix as stable as possible, actually Gert, one of the co‑authors e‑ went through the and we had a long discussion and he was saying this needs to be dynamic, there should be no static address on IPv6, this should be doable. We documented the problem, it clearly is a problem as Geoff also proved. And we went to the IETF, and we said, okay, if the end host, the SLAAC mechanism, cannot follow the topology changes properly, this needs to be changed somehow.
So I will go through the presentation that Fernando gave where we submitted the draft with some ideas, and let's see, I would like to, at the end, I would like to have some feedback from the operators in the room, what should we do about it, because when you will hear the feedback that we got from the IETF, yeah.
Okay. Simple scenario: We have CPE router and we have SLAAC on the LAN and ISP ‑‑ ten minutes left ‑‑ that's going to be tough ‑‑ ISP changes prefix every so often for example CPE reboots, and all of a sudden when CPE comes back it requests prefix deigation and it gets different prefix because it was rebooted. But does not understand and remember which prefix it was before so it does not send the packet with lifetime 0 to remove all the addresses from the network because the CPE has no idea what was before.
The problem is that same, that you have seen in Geoff's slides, SYN/ACK, never comes back ‑‑ okay, SYN/ACK, ACK doesn't come back because there is no back to the network and to the CPE that had the old addresses.
And there is breakage, and as mentioned, happy eyeballs is currently today saving our skin but this will not go forever, so we need to fix this.
The result:
Old addresses are maintained and quite frequently such addresses are preferred and we did the calculations that the old addresses are preferred, they can be preferred for a month, and they can stay on the machine for three months, if nobody tells that this prefix is removed from the network, and this is clearly not good.
What to do:
There was lots of discussion. We recommended for the sites to use stable prefixes, for the ISPs to not change prefixes but some of them are clearly not very happy. There is lass suggestion that the CPE implements the prefixes information in their non‑volatile memory that can be preserved throughout reboots. Some CPE vendors are doing this and actually when the CPE comes back, gets different prefix, compares and starts sending RA packets with lifetime 0 to deprecate the old prefix on the network, but not all of them does that.
And then CPE vendors went like, yeah, but, you know, this memory is expensive and we don't want to right anything to the CPE other than configuration, we don't want to change anything.
So, another idea was to actually fix the IPv6 on the end host. How? Get rid of stale addresses and routes on in a timely manner. So if the same router advertises a new prefix, so all of a sudden you get an RA with the new prefix and count the number of consecutive RAs from the same router that does not include the previous prefix after several attempts you start, for example, after two RAs, unprefer the address and that after two additional ones remove the address and the routes. This may ‑‑ this may lead in some cases in the condition where we have multiple prefix on the network, sometimes yes and no, but this is up to discussion. There was also another idea from general that I quite like, to actually create for each prefix, create additional link local address on the router and send RA packets in this address and if the prefix changes, this link local address is removed from the router and consequently, with the neighbourhood discovery also the default router, this link local is removed and therefore the IPv6 address on the other end is actually the preferred. This is also one possible solution.
Well, we went to the IETF, we gave this presentation, we submitted the draft. I think in one week it was 300 or 400 emails, it was crazy. When people tried to identify what is going on, if there is a problem, what is the problem and things like this, that week I couldn't read anything else, I was like, it was crazy. But then we discovered that the problem is that, for example, also what people does is that the ISPs does not keep the promise they do, so if the ISP sends the prefix delegation to a user with the lifetime of ten years and that should be the case the, they cannot change the prefix in 24 hours, in this case you are not keeping your promise, you said to your client, I am giving you this prefix for ten years, and then I'm changing it. It's not fair. So, please, keep the promise that you put in the prefix delegation lifetime. There are also other things that the CPE vendors should consider, for example if I receive a prefix delegation that is the lifetime is set for one hour, one hour, I should not send out the RA packets on the inside of the CPE with the lifetime higher than one hour because I don't have the promise from my uplink that this addresses will last for more than one hour so I can't promise to the internal clients that these prefixes will last for more than one hour. So this is usually all the problem because the CPE vendors don't actually care about this and they don't implement mechanisms that would transfer the lifetime from the uplink to the internal network.
So, there are many things that we can do. We can keep our promises as an ISP, we can fix the CPE. We can fix the SLAAC to react to topology changes in more timely manner, but all these things are causing an ugly graph that Geoff was presenting before so that clearly is a problem.
And my question is, where do we go from here? What do we want to do as a community, as operators, as people that are deploying IPv6. Don't be shy, I want to hear from you. Thank you.
JEN LINKOVA: I think we all agree there is a problem, right? My draft about ‑‑ just default address selection to solve exactly this, was like two years old. I would like to make few comments for the audience on your slides because I think the slides probably do not necessarily emphasise for few important things. First of all, your CPE which you do not deprecate the prefix when it changed doing absolutely wrong things ‑‑ so we are talking about broken CPEs. Secondly, this problem actually not specific to the DHCP prefix delegation. I am seeing this with the network manager on Linux machine, on wired network. Oh, my God I got another prefix but I am going to keep it all the one. So, that's why it could not be fixed on the CPE level so I think it should be joint effort from the host stack and we also definitely need to fix broken CPEs but just to repeat what I said, it's probably not worth doing seriously ‑‑ so it would be nice to use existing mechanism as much as possible to make them more robust. That is why I came with suggestion of using existing address selection and using domains, if anyone is willing to do hackathon we can trial and I will had been a happy to trial.
JAN ZORZ: Yeah, I agree something needs to be done. And basically, the feedback that we got from the IETF and also from co‑chairs was there is nothing wrong with IPv6, you are configuring it wrongly, you are deploying it wrongly, you are ‑‑ you should not give out the dynamic addresses, you should not change the topology and you should keep the promise that the keep to the CPE. I was like okay. I will take this back to the operational community and let's see if they Lynch me.
AUDIENCE SPEAKER: Costas. I want to share our experiences in my previous employer. We have been running dynamic prefixes in IPv6 for seven years, eight years. I do not remember the case. I believe it goes back many, many years that we solved this problem on the CPE side, we just depicated everything, on each PPP disconnection and that's it. We didn't have the option to use static IPv6 prefixes because our B and G vendor did not support it in its DHCP implementation and we would need to he engineer our entire deployment and introduce a central waives big project. So, we didn't take this effort of introducing centralised DHCPv6 to give static prefixes which by the way our view as engineering team back then would be that we should give static prefixes. Although I must say that for many years we didn't have any problem with dynamic IPv6 prefixes via DHCPv6
JAN ZORZ: But you fixed the CPE.
AUDIENCE SPEAKER: Yes.
JAN ZORZ: Any CPE vendor in the room.
AUDIENCE SPEAKER: We saw the problem and communicated it to our CPE vendor
JAN ZORZ: You understood there is a problem, many operators don't understand there is a even a problem.
AUDIENCE SPEAKER: I don't think there is anything wrong in the IPv6 specifications, we shouldn't address that.
CHAIR: We are in coffee break. I think we should continue this discussion on the list. People will have suggestions or things to share, please do so. And this afternoon we have a combined session with Routing Working Group, and Jen wants to say something.
JEN LINKOVA: Nobody leave the room until you rate the talks.
CHAIR: I was going to say that, some people are always faster. So thank you and see you in the afternoon.
LIVE CAPTIONING BY AOIFE DOWNES, RPR
DUBLIN, IRELAND
