[af-ix-discuss] Renseignements, Interconnexion de deux POP

[geier at geier.ne.tz (Frank Habicht)]

Hi Benny,

What the difference between "interconnection of two different IXPs" and
"interconnection of two POPs of the same IXP" depends very much on the
definitions, so we probably have to clarify these first.

And the effects of the differences - feasibility etc - then also depend
on local circumstances.

Definitions:
First not that some things that call themselves "Internet Exchange" are
not really what we here mean by "IXP". One example is the old incumbent
operator in South Africa ("SAIX") and another is "EMIX" in the UAE, and
there are and were several others.
When you get at one place service to reach the "whole internet" - that's
an internet transit service and that's what the above mentioned do. A
(possibly big) ISP.

An IXP will allow many different ASNs to connect directly with each
other, and does not control the policies or the commercial aspects
between them. Typically, an IXP will also not charge depending on the
transferred traffic or volume, and also not on a bandwidth subscribed
to, other than the physical limitations of the port on the IXP ethernet
switch fabric (eg 100M/1G/10G/... or 2x1G/2x10G/..).

If one organisation extends the IXP switch fabric to new location(s),
that is in the vast majority of cases within the same city. From one
data centre to another data centre. This can most often be because there
are some more peers at the second data centre, who want to peer with
your existing peers at location 1. And since it would be more expensive
for the 10 new peers to each get a fiber to your location 1, it's better
the IXP gets 2 (for redundancy) fibers to a seconds switch in location 2.
And then the new switch 2 would also be operated by the IXP operator.
Because the 10 peers at location 2 are competitors to each other and
trust the IXP operator more.

So this makes it still one IXP, with multiple "POPs" or "locations"  -
almost always in the same city.

If the same IXP would start a new POP in another city, and connect it to
the same switch fabric, and transport packets between the cities for the
peers, that would make it more like a network operator, they would have
to have more support staff, and would be offering services more similar
to normal ISPs.

This is a dangerous thing. IXPs depend on the network operators (ISPs,
present and prospective peers at the IXP) to view the IXP as neutral and
beneficial for the general internet development, and connectivity -- but
NOT to compete in the same business and possibly take customers away
from the ISPs!

So the distance between those POPs matters.

For two different IXPs to connect to each other...
a) if we assume they are run by the same organisation.
that organisation will in my opinion effectively become a network
operator, need support staff, will need to charge differently for
traffic staying in city 1 vs traffic crossing the distance to city 2.
It will have to obtain infrastructure or capacity between the cities.
And then what if it will not be fully used? You employ Sales people? You
risk paying for a link that is not getting used?

I think it is fair to say that if it will be sustainable, then it's like
an ISP business.

Unfortunately in many places in Africa, a (high capacity) link between
cities is often more expensive than a Internet transit link of the same
capacity at any of the 2 locations. And that gives advantages to content
hosted elsewhere, but .... let's not go too fast astray.

Some people argue that with current relatively low traffic levels
between locations 1 and 2 we have to build the link, and then traffic
will increase, for instance because more content will be hosted locally.
This is the eternal chicken-and-egg problem between local content and
local interconnection. And now we are stretching the word "local" very
much already - geographically.

Some people think a temporary trial can be useful to find out more;
some think someone else can support this as a one-time cost;
Would you not better avoid being dependent on that?

So connecting 2 IXPs of the same organisation is like extending one IXP
switch fabric to another city. often bad.

b) connecting 2 IXPs of different organisations:
same problems, plus: the intentions and incentives of the 2
organisations might not always stay aligned as much as they are (or
seem) right now.

Is one of them more dominant than the other?
Does the bigger want for the other to pay all the cost of the
connection? After all, the smaller gets to enjoy more (all the peers at
the bigger IXP).


If there is a border between the 2 locations, it all becomes much more
difficult again. Because chances are the 2 regulators do not intend to
"enable" (support) connectivity, but to rather restrict and control it :-(


If the border is in a river [ :-) ] then you get more challenges of
physical safety and installation of the fiber, but maybe fewer problems
with securing right-of-way. If you're not thinking of fiber, than i wish
you that the capacity of your physical link will not be enough, and
you'll have to upgrade to fiber soon.


If the 2 IXPs try to put the own outdoor plant - fiber or microwave -
they will need to get the expertise to do that.
And the expertise is found with staff of telecommunications and ISP
companies. Maybe these (at least 2 or 3 of them) should make links
across the river, ehm... border, ehm... to the other city ?

And then the ISPs that invested in this can use this as a benefit (and
sales pitch) to their customers? and they can also re-sell this to other
ISPs. Maybe even at the 2nd IXP location ;-)

If we could get these ideas to the ISPs, if should mean more long-term
neutrality and trust in the IXPs.

Are any network operators present at both IXPs?
Are they using an efficient, relatively direct route for their own
traffic between these locations?


Thanks,
Frank



On 1/10/2018 7:05 PM, Benny.MBOKO at arpce.cg wrote:
> Hello Nishal,
> First of all, thank you for all the time you have devoted to bring me all this light.
> Could you tell me about the interconnection of two different IXPs and the interconnection of two POPs of the same IXP knowing in the first case that there is only the interconnection link that is shared, whereas in the case of POPs of the same IXP, there is a whole network to build?
> 
> 
> Best regards
> 
> Benny MBOKO
> ARPCE- Congo
> 
> -----Message d'origine-----
> De?: Nishal Goburdhan [mailto:nishal at inx.net.za] 
> Envoy??: mercredi 10 janvier 2018 15:40
> ??: Benny MBOKO <Benny.MBOKO at arpce.cg>
> Cc?: af-ix-discuss at af-ix.net
> Objet?: Re: Renseignements, Interconnexion de deux POP
> 
> On 4 Jan 2018, at 15:07, Benny.MBOKO at arpce.cg wrote:
> 
>> Bonjour ? tous et tous mes v?ux les meilleurs,
>>
>> Je viens vers vous pour un peu plus de lumi?re sur l?interconnexion de 
>> 2 POP.
>> Comment peut-on interconnecter deux Points de Pr?sence d?un IXP 
>> sachant que chaque POP a ses propres ressources (ASN et IP) Les 
>> ressources de management de chaque POP interviennent-elles dans cette 
>> interconnexion ?
>> Quelles sont les ?tapes ? franchir et quelles sont les pr?requis ?
>> Merci pour votre coup de pouce.
>>
>> Cordialement
>> Benny Sh?rif MBOKO OTOKA
>> Administrateur r?seau et syst?me
>> Au Service du tr?s haut d?bit
>> [cid:image001.png at 01D04223.BE54C720]
>> Immeuble ARPCE ,91 bis Avenue de l?amiti? 
>> Centre-ville-Brazzaville-Congo BP : 2490 T?l mobile : (+242) 06 653 29 
>> 38/044525539 T?l bureau : (+242) 05 510 72 72 Site internet : 
>> www.arpce.cg<http://www.arpce.cg/>
>> Email : Benny.MBOKO at arpce.cg<mailto:Benny.MBOKO at arpce.cg>
> 
> 
> hi benny,
> you had sent this message to the mailing list owner in error.  i?m redirecting this to the mailing list itself, (and i took the liberty of replying as well :-)
> 
> happy 2018 to you too!
> 
> it is not a good idea to interconnect internet exchange points.  it is better to allow internet exchange points to run and develop independently.  over time some internet exchange points may fail.  
> that?s not necessarily a bad thing, and simply shows how the market, or policy, changes in a region.  in cases where you hear of IXPs interconnecting, it is generally the same IXP that has multiple PoPs within a city.  this usually happens when there are multiple colocation facilities in the city, and when peering participants (ie. the networks that want to peer) want to be able to connect to the same IX fabric, but from different facilities.
> 
> but, even this is not easy to accomplish, since the IXP administrator would have to take the effort of building infrastructure to connect across the city.  this infrastructure will not be free, and now places the burdens of capital expenditure (ie. equipment) and operational costs (ie. payments to keep the fibre running) on the shoulders of the IXP admin.  generally, IXPs are non-profits that work to improve interconnectivity (and not ISPs that are working to make money) so, increasing the cost of the IXP?s operations is something that you always want to avoid!
> artificially increasing the cost of running the IXP, means that you would need to find creative ways of funding this, and that would quickly distract you, the IXP administrator, from the basic operations of the IXP - which is to run a stable fabric!  so, even if you are thinking of building, or extending your IX to multiple sites, i?d strongly suggest doing this, *if, and only if*, the cost of the fibre between the different locations is very cheap, or free!
> note the first very big constraint here - the cost.
> 
> i started with cost, since, it is important to realise what an IXP is meant to do.  a well-managed IXP will reduce the average-per-bit-delivery-cost (ABPDC) of the networks that connect to it.  ie.  peering makes the network cheaper to run.  if this is not true, networks will stop peering, and continue/start purchasing IP transit instead.  so, if you start to introduce costs into the IXPs operations, the IXP will have to find ways to fund this, and, that goes against the what is probably the first rule of the IXP - which is to make interconnection cheaper!
> of course, the costs go beyond just the recurring costs of the connectivity between the locations;  as the IX operators, if you did decide to build a distributed exchange, you?d need to have a 24x7 NOC to manage this.  and you can?t have just a single person, so that means staff costs, HR costs, etc.  all of which needs to be funded, and all of which makes it even more difficult to get networks to the IX to start.
> 
> there are other reasons as well;  if you do go ahead, and make this happen, then your participants - networks that would otherwise be peering - may complain that you are competing with them for transmission, and may withdraw from your IX.  that is *absolutely* to be avoided, since a network removing itself, is never a good thing for the peering environment.  and, in the long term, this is a direct disincentive to invest in telecoms infrastructure by the private sector;
>   infrastructure, which otherwise, would lead to more competition, lower pricing, and a more stable overall network.
> 
> i?ve mentioned three reasons, and explained just one in some detail.  
> but i?m hoping you start to see that this isn?t a good idea!
> 
> there only a few niche caches where an IXP interconnects directly with another;  these *always* come with limitations;  rules like you are not allowed to use more than X %bandwidth, etc.  in the long run, this situation does not scale, and is generally avoided.  of the more than
> 500 IXPs globally today, there are fewer than ten cases where this happens, and i would strongly encourage you to not follow this model.  
> interconnecting IXPs may sound like a grand idea, and it serves a short-term political goal, but at a practical, and long-term economic level, it?s quite a disastrous thing to do to your economy.
> 
> there are many cities in the world that have multiple IXPs, that do not interconnect with each other.  multiple IXPs in a city work best, when they are considered separate (ie. different infrastructure) and generally only work well, when there?s already a rich mesh of interconnectivity.  in other words, get an IXP up and running;  try to get as many participants connected and peered, and when these participants start to feel like there is a need for resiliency, then it?s a good idea to consider getting another IXP up within the same 
> city.   it?s a *terrible* idea to start two competing IXPs from day1, 
> as this will likely fragment the peering market, and that?s something that you want to avoid at all costs.
> 
> i?ll pause here to give other list folk a chance to express their opinions, but i?m happy to explain this in greater detail if 
> necessary.   ;-)
> 
> ?n.
> 
> ______________________________________________________________________
> This email has been scanned by the IT101 Email Security System.
> For more information please visit http://www.it101.be ______________________________________________________________________
> 
> 
> ______________________________________________________________________
> This email has been scanned by the IT101 Email Security System.
> For more information please visit http://www.it101.be
> ______________________________________________________________________
> 
> _______________________________________________
> af-ix-discuss mailing list
> af-ix-discuss at af-ix.net
> http://af-ix.net/mailman/listinfo/af-ix-discuss_af-ix.net
>